Identical or Similar? AI-Powered Product Matching with Amazon Bedrock and S3 Vectors

A few weeks ago, someone asked me a question:

“How can I use AI to improve my inventory if I have similar or duplicate products, but their names aren't written exactly the same way?”

To explore how to solve this problem, we'll use video games as an example. ( ⚠️ First spoiler: that's the type of data we'll be working with in this article.)

So, what does it really mean for two products to be duplicates?

⚠️ Second spoiler: AI isn't going to solve everything. We could send all the information about each product to an AI model and ask it to determine whether it's a duplicate. But processing an entire catalog this way can be expensive, and we can't guarantee that the responses will always be consistent. In other projects, I have worked with text normalization, regular expressions, and similarity techniques. These remain useful tools, and in many cases, they are sufficient. For this MVP, I wanted to explore a different approach: using rules to handle straightforward cases, while reserving AI for more ambiguous scenarios that cannot be resolved by a rule.

That's why, before we start building the solution, there are three questions I want to answer: 1. How can we distinguish truly duplicate products from those that are simply similar? 2. When does AI add value, and when is it better to use traditional rules? 3. Can we build an MVP on AWS using a serverless architecture while keeping costs under control?

To answer these questions, we'll build a solution that combines AI with traditional rules. We'll walk through the process, from data analysis and normalization to architecture design, comparison logic, and evaluation of the results. Because detecting duplicates isn't simply about finding similar names. First, we need to understand what we're comparing and what it actually means for two products to be the same.

Poor product data quality is not an isolated problem. Various studies highlight the difficulties companies face in keeping information consistent, up to date, and accessible across their systems. For example, in 2026, GS1 UK published a report on product information management in the construction sector, based on research involving more than 300 professionals and 15 prominent figures from the industry. Some of its findings include:

When a company sells products through multiple channels, keeping information up to date and consistent can become a challenge. Product Information Management (PIM) systems are designed to manage this process.

A PIM allows companies to centralize, enrich, organize, and distribute product information from a single location. It can manage product names, descriptions, technical specifications, images, and other attributes needed to sell a product. Adobe explains here how these systems work.

Consider, for example, a company that sells video games through its own website and marketplaces such as Amazon or eBay. Product information may come from different suppliers and in different formats, so it needs to be organized before being published across each channel. Although this article introduces the concept of PIM, depending on the organization, product information may be managed in an Excel spreadsheet, a database table, enterprise systems such as a CRM or SAP MM, or specialized solutions like a PIM. The idea is to explore a solution that can be accessed through an API, for example, to help perform data quality checks and detect potential duplicates, regardless of where the product information is managed.

Before starting to build the solution, I had to make several decisions: which data to use, which part of the catalog to process, and how to combine AI with traditional rules without driving up costs. These are the five principles that defined the scope and design of the MVP.

I used the product metadata from Amazon Reviews 2023, in the Video Games category. I chose this category because it covers many different scenarios: the same game can be sold for different platforms (Nintendo, PlayStation, Xbox or PC), in different editions (standard, deluxe or collector’s), in physical or digital format, or as a refurbished product.

For the MVP, I selected a subset of products based on the 20 best-selling video game franchises according to Wikipedia. To identify the records, I used regular expressions that look for the names of these franchises and their variants in the dataset titles. This keeps the data volume manageable and makes the selection criterion reproducible.

I chose a serverless AWS architecture, with on-demand processing and low-cost AI models. Every technical decision is also evaluated for its impact on cost, from data preparation to model calls.

To avoid unnecessary costs, I first analyze the data and apply deterministic rules to the cases that can be solved simply. The language model is reserved for cases that require interpreting text, recognizing abbreviations or evaluating ambiguous matches.

Deciding whether two products are the same, a variant or different products is not just a technical question: it depends on the business. In this MVP, those criteria are defined in the prompts and in the pipeline rules, with examples that guide the model. They are a proposal for this case study: in a production environment, they must be reviewed and validated with the business teams of each organization.

This MVP is a practical approach to building a data pipeline for detecting duplicate products. The goal is not a perfect system, but to check whether the approach is viable and to measure its technical behavior: how long it takes, how much it costs, which share of the decisions is resolved with rules and which needs a model, and how extraction quality evolves across prompt versions. I also review specific cases to understand where it succeeds and where it fails.

Before going into the details of each step, it is worth looking at the complete solution. The goal of the architecture is to transform a disorganized product catalog into one where each listing is associated with the actual product it represents, making that information available for real-time queries. To achieve this, the solution combines batch processing, which processes the entire catalog at once, with a real-time component that handles new products or search queries as they arrive.

Source: Architecture diagram created by the author.

The image above shows the solution architecture, organized into three parts:

Exploratory Data Analysis (EDA) is a process that helps us better understand a dataset before applying models or making decisions. Through different exploration and visualization techniques, we can identify patterns, discover relationships between variables, detect anomalies, and assess data quality. Although we often associate EDA with the early stages of Machine Learning projects, the concept is not new. Statistician John W. Tukey formalized it in his book Exploratory Data Analysis, published in 1977.

Because it helps us understand the data before working with it, identify outliers, detect inconsistencies, and question our initial assumptions. Instead of assuming that the data has the expected structure or quality, we explore it to discover what it is actually telling us.

It can be tempting to send raw data to a model with a prompt and hope for the best. But every unnecessary field adds tokens that we pay for with each call, and every inconsistent data point is a potential error that the model can propagate. EDA helps us decide which data to send, what to correct beforehand using simple rules, and which part of the catalog to process. For this use case, I performed ten checks:

The findings that had the greatest impact on the design were:

With the data analyzed, the next step is to create the resources required by the pipeline. Ideally, we would define the entire infrastructure as code using Terraform, AWS SAM, or CDK and deploy it with a single command. To keep this MVP simple, I created the resources through the AWS Console and included the scripts and AWS CLI commands in the repository to reproduce the deployment.

Before getting started, you will need: An AWS account with access to Amazon Bedrock in us-east-1 and the Nova Micro, Nova Lite, and Titan Text Embeddings V2 models enabled. AWS CLI v2, configured with the permissions required to create the resources. The meta_Video_Games.jsonl file from the Amazon Reviews 2023 dataset. Clone the repository and navigate to the project directory:

git clone https://github.com/RominaElenaMendezEscobar/aws-games-products.git cd aws-games-products Enter fullscreen mode Exit fullscreen mode 📁 Repository Structure The resources are organized by service under aws/. Each Lambda function has its own directory containing the code, permissions policy (policy-role.json), and required configuration.

aws/ ├── S3/ → data bucket ├── S3Vectors/ → vector bucket and index ├── DynamoDB/ → products table ├── lambda/ → 8 functions + deployment script └── step-function/ → pipeline definition and its role Enter fullscreen mode Exit fullscreen mode The deployment follows this order: storage → Lambda functions → Step Functions, as each stage depends on resources created earlier.

This bucket stores the intermediate data generated during the different pipeline stages.

bash aws/S3/create_s3_bucket.sh Enter fullscreen mode Exit fullscreen mode 6.2.1.2 Amazon S3 Vectors game-catalog-vectors / index games-v2 This is where the product embeddings are stored. First, we create the vector bucket and then the index, which uses 1,024-dimensional vectors compatible with the Titan Text Embeddings V2 configuration and cosine distance. The title and base_title fields are configured as non-filterable metadata: they are preserved in the results but cannot be used as filtering criteria in the index.

aws s3vectors create-vector-bucket \ --vector-bucket-name game-catalog-vectors aws s3vectors create-index \ --vector-bucket-name game-catalog-vectors \ --index-name games-v2 \ --data-type float32 \ --dimension 1024 \ --distance-metric cosine \ --metadata-configuration '{"nonFilterableMetadataKeys": ["title", "base_title"]}' Enter fullscreen mode Exit fullscreen mode 6.2.1.3 Amazon DynamoDB · game-catalog-products The table stores products identified by parent_asin and uses on-demand capacity. Two global secondary indexes allow us to query the results by identifier:

bash aws/DynamoDB/create_dynamodb_table.sh Enter fullscreen mode Exit fullscreen mode 6.2.2. 🟥 Permissions and Configuration Each function has its own IAM role with least-privilege permissions. The policies are defined in the policy-role.json file in each directory and specify which services each Lambda can access. Each role also includes the AWS managed policy AWSLambdaBasicExecutionRole, so the function can send logs to CloudWatch. The deployment script (6.2.3) creates the roles and attaches both policies. Before deploying, review the ARNs in each policy-role.json and replace <account_number> with your own AWS account ID. Also check the resource names and the Region.

All eight functions use Python 3.12. To simplify deployment, I prepared a script that creates each function's IAM role from its policy-role.json file, packages the code, and creates the Lambda function or updates it if it already exists. The script receives the function’s folder, its name, the memory and the timeout, and optionally a Lambda Layer ARN and the ephemeral storage size:

./aws/lambda/deploy_lambda.sh <folder> < function -name > <memory-MB> <timeout-s> Enter fullscreen mode Exit fullscreen mode These are the commands to deploy the eight functions:

cd aws/lambda # Usage:./deploy_lambda.sh <folder> <function-name> <memory-MB> <timeout-s> [layer-arn] [ephemeral-MB]./deploy_lambda.sh ETL01-game-catalog-etl game-catalog-etl 2048 900 <pandas-layer-arn> 2048./deploy_lambda.sh SP01-game-catalog-prepare-batch game-catalog-prepare-batch 1024 180 "" 1024./deploy_lambda.sh SP02-game-catalog-extract-ondemand game-catalog-extract-ondemand-access 512 900./deploy_lambda.sh SP03-game-catalog-parse-results game-catalog-parse-results 1024 300 "" 1024./deploy_lambda.sh SP04-game-catalog-embed game-catalog-embed 512 900./deploy_lambda.sh SP05-game-catalog-decide-pairs game-catalog-decide-pairs 1024 900./deploy_lambda.sh SP06-game-catalog-build-clusters game-catalog-build-clusters 1024 900./deploy_lambda.sh RT01-game-catalog-resolve-product game-catalog-resolve-product 512 60 Enter fullscreen mode Exit fullscreen mode ⚠️ The ETL Lambda function uses pandas, which is not included in the default Lambda runtime. It requires the AWS-managed AWSSDKPandas-Python312 layer (in the Lambda console: Layers → Add a layer → AWS layers) and 2 GB of ephemeral storage, as it writes the cleaned catalog to /tmp before uploading it to S3.

Finally, we create the Standard state machine game-catalog-dedup-pipeline, which coordinates the six processing stages. Its role has permission to invoke only the six Lambda functions used by the pipeline ( LambdaInvokeScopedAccessPolicy.json ).

aws iam create-role --role-name game-catalog-dedup-pipeline-role \ --assume-role-policy-document file://aws/step-function/trust-policy.json aws iam put-role-policy --role-name game-catalog-dedup-pipeline-role \ --policy-name LambdaInvokeScopedAccessPolicy \ --policy-document file://aws/step-function/LambdaInvokeScopedAccessPolicy.json aws stepfunctions create-state-machine \ --name game-catalog-dedup-pipeline \ --definition file://aws/step-function/game-catalog-dedup-pipeline.json \ --role-arn <step-functions-role-arn> \ --type STANDARD \ --region us-east-1 Enter fullscreen mode Exit fullscreen mode 6.2.5. 📥 Upload the Raw Data aws s3 cp data/meta_Video_Games.jsonl s3://game-catalog-dedup/raw/ Enter fullscreen mode Exit fullscreen mode 6.3. Data Preparation After exploring the dataset, the next step is to prepare it for the duplicate detection pipeline. The EDA helped us understand its structure, identify data quality issues, and determine which attributes could be useful.

Based on these findings, I divided the preparation process into two stages: first, cleaning and filtering the data; then, selecting the catalog we would use for the MVP.

A Lambda function applies the rules defined during the EDA before any AI model is involved. The goal is to prevent incomplete records or unnecessary fields from affecting subsequent processing stages. The main operations are:

Each stage stores its results in a separate Amazon S3 folder ( raw/, clean/, mvp/ …). In addition, the pipeline stages store their outputs in an execution-specific subfolder, such as attributes/top20-v8/. This separation offers two practical advantages: First, it allows us to reuse the results of a stage without repeating the entire processing workflow. Second, it makes it easier to compare different pipeline executions and identify what has changed between versions. This proved especially useful when refining the prompt: I could rerun subsequent stages using the already prepared data, without repeating the catalog cleaning process every time I modified the model instructions.

Once the data is prepared, the next step is to define how we will use AI. The goal is not to ask a model to find duplicates directly across the entire catalog, but rather to break the problem down into more specific tasks. To achieve this, I designed two prompts: one to normalize the attributes of each product and another to determine whether two candidate products are duplicates, variants, or different products. However, before selecting the models and writing the instructions, there were several requirements the solution needed to meet.

The first requirement was consistency. If two listings describe the same video game, we need the model to generate normalized attributes that allow us to recognize them as the same product, even when their original titles are different. We also needed structured output that followed a predefined schema and could be processed programmatically without requiring manual interpretation.

response = bedrock. converse ( modelId = " us.amazon.nova-micro-v1:0 ", system = [{ " text ": SYSTEM_PROMPT }], messages = [{ " role ": " user ", " content ": [{ " text ": json. dumps ( build_payload ( product ))}]}], inferenceConfig = { " temperature ": 0, " maxTokens ": 300 }, ) attributes = json. loads ( response [ " output " ][ " message " ][ " content " ][ 0 ][ " text " ]) Enter fullscreen mode Exit fullscreen mode The build_payload function sends only the fields I selected during the EDA: title, store, categories, four keys from details, and three product features. This way, we avoid sending unnecessary information to the model. This snippet shows the basic model invocation. The complete implementation also needs to handle response validation and error management.

This prompt uses Amazon Nova Micro and runs on every product in the catalog. Its purpose is to transform an original product listing into a set of normalized attributes that we can use in subsequent stages. For example, a product listing might contain:

{ "title": "GTA: Vice City - platinum [Playstation 2]", "categories": [ "Video Games", "Legacy Systems", "PlayStation Systems", "PlayStation 2", "Games" ] } Enter fullscreen mode Exit fullscreen mode The expected output would have a structure similar to this:

{ "product_type": "game", "base_title": "Grand Theft Auto: Vice City", "platform": "PlayStation 2", "edition": "Standard", "format": "physical", "region": "unknown", "publisher": "Rockstar Games", "confidence": "high" } Enter fullscreen mode Exit fullscreen mode In this example, the model normalizes the title and platform, while other attributes are determined according to the rules defined in the prompt. For instance, the Platinum label is treated as a budget re-release and normalized to Standard for this MVP. Other values may be inferred from the available information, so it is important to consider the limitations of these inferences.

The key is not simply to clean up the title, but to ensure that equivalent products are represented consistently without losing important distinctions, such as the platform. 📍 This prompt is used by the game-catalog-prepare-batch (batch) and game-catalog-resolve-product (on-demand) Lambda functions, both for the initial catalog processing and for resolving subsequent queries.

This prompt uses Amazon Nova Lite and is reserved for ambiguous cases that require additional evaluation. After normalization and candidate retrieval, not every match is straightforward. Two titles may be very similar while referring to different video games. For example: A: Mario & Sonic at the Olympic Games [Nintendo Wii] B: Mario and Sonic at the Olympic Winter Games - Nintendo Wii Although they share many words, these titles refer to different games. The presence of Winter is a meaningful distinction that we should not ignore. The prompt must return a structured decision, for example:

{ "relation": "different", "reason": "The base titles of the games are different." } Enter fullscreen mode Exit fullscreen mode 📍 This prompt is used by the game-catalog-decide-pairs (batch) and game-catalog-resolve-product (on-demand) Lambda functions. By reserving Nova Lite for ambiguous pairs, we avoid using a more expensive model for every record in the catalog.

The SYSTEM_PROMPT was organized into five parts:

Source: Screenshot captured by the author from the Amazon Bedrock console (AWS).

To evaluate the models' behavior and improve the prompts, I organized the tests across several dimensions:

I tested Amazon Nova Micro and Nova Lite with equivalent inputs to evaluate their performance on extraction and comparison tasks. The final decision was to use Nova Micro to normalize all products and reserve Nova Lite for ambiguous pairs. This way, the model with stronger reasoning capabilities is used only when it can provide additional value.

I used a temperature of 0 to reduce response variability during testing. This is useful when comparing prompt versions, although it does not guarantee that every execution will produce identical results.

Grouping ten products into a single call could have reduced the number of extraction calls from approximately 3,740 to 374. However, during testing, I observed that the model could mix information between records, incorrectly modify titles, or return results without properly associating them with their identifiers. For this reason, I kept one call per product and used eight concurrent calls. This decision increases the number of requests but makes it easier to maintain the correspondence between each input and its result.

Although the catalog is primarily in English, I also wanted to test how the system would respond to searches written in Spanish. I tested queries such as pokemon esmeralda game boy advance, fifa 14 ps3 código digital, and el mario kart de la wii. The goal was to evaluate whether the model could interpret these expressions and normalize them to the vocabulary and attributes used in the catalog. These tests are relevant for a system that not only processes existing records but can also receive search queries written by users.

Each new version was tested against previously failing cases and control cases that already worked, to ensure that fixing one issue did not introduce another. These included titles without numbers, titles that already use digits (“FIFA 23”, “NBA 2K17”), and letters that are part of a game's name (“Pokémon X”, “Call of Duty: WWII”, “Mega Man X”).

During testing, I noticed another important detail: the Playground can preserve conversation history. When I tested many products consecutively in the same chat, previous responses could influence subsequent ones. In one test, the model stopped correctly normalizing Roman numerals after several examples. Since each Lambda invocation is independent, this behavior did not reflect the actual conditions of the pipeline. To avoid misleading comparisons, tests needed to use the same message sent by the application, without previous conversational context, starting a new conversation whenever necessary.

The Amazon Bedrock Playground allowed me to observe token consumption and call latency to estimate costs before running the full pipeline. I later compared these estimates with the actual results. I also measured the extraction time for all 3,740 products, which was 8 minutes and 26 seconds, mainly constrained by Bedrock's requests-per-minute quota. These measurements allowed me to evaluate technical decisions beyond response quality, taking both cost and processing time into account.

The prompt went through eight versions. Here are some of the differences between the first run on the 20 franchises (version 6) and the final version (version 8):

The catalog is processed by an AWS Step Functions state machine that orchestrates six Lambda functions. Each function performs a single task, reads the previous stage's output from S3, and stores its results in a separate folder.

Source: Screenshot captured by the author from the AWS Step Functions console (AWS).

I chose Step Functions to coordinate the processing for six reasons:

{ "bucket": "game-catalog-dedup", "run_name": "top20-v8", "title_contains": [ " \\ bmario \\ b", " \\ btetris \\ b", " \\ bpok[eé]mon \\ b", " \\ bcall of duty \\ b", " \\ bgrand theft auto \\ b| \\ bgta \\ b", " \\ bminecraft \\ b", " \\ bfifa \\ b", " \\ bassassin'?s creed \\ b", " \\ bresident evil \\ b", " \\ bfinal fantasy \\ b", " \\ bwii (sports|fit|play|party|music) \\ b", " \\ blego \\ b", " \\ bthe sims \\ b", " \\ bnba 2k", " \\ bzelda \\ b", " \\ bmadden \\ b", " \\ bneed for speed \\ b", " \\ bsonic \\ b", " \\ bmonster hunter \\ b", " \\ bred dead \\ b" ], "model_id": "us.amazon.nova-micro-v1:0", "prompt_version": "v8", "index_name": "games-v2", "decision_model_id": "us.amazon.nova-lite-v1:0", "pairs_run": "pairs-top20-v8", "clusters_run": "clusters-top20-v8", "attribute_keys": [ "attributes/top20-v8/attributes_games.jsonl" ] } Enter fullscreen mode Exit fullscreen mode What each parameter represents:

The central idea behind the design is to separate two questions:

base title | platform | edition | format Enter fullscreen mode Exit fullscreen mode For example:

Grand Theft Auto: Vice City | PlayStation 2 | Standard | physical Enter fullscreen mode Exit fullscreen mode Normalized attributes are used instead of the original title so that two listings written differently can generate the same representation when they describe the same product. This helps reduce naming inconsistencies and improves candidate retrieval through vector similarity search. Next, Amazon Titan Text Embeddings V2 converts this representation into a vector.

Amazon S3 Vectors returns the 20 nearest neighbors for each product in the index, and we keep those with a cosine distance below 0.25. A distance of zero indicates identical vectors; the greater the distance, the lower the similarity between them. This threshold helps reduce the number of comparisons, but it does not determine on its own whether two products are duplicates.

That decision is made in the next stage by evaluating their attributes and applying the defined business rules.

The implementation is divided into two functions: one builds the text from the normalized attributes, and the other invokes the Bedrock embedding model.

def identity_text ( attrs ): parts = [ attrs [ " base_title " ], attrs [ " platform " ], attrs [ " edition " ], attrs [ " format " ], ] return " | ". join ( p for p in parts if p and p!= " unknown " ) def embed ( text ): response = bedrock. invoke_model ( modelId = " amazon.titan-embed-text-v2:0 ", body = json. dumps ({ " inputText ": text, " dimensions ": 1024, " normalize ": True, }), ) return json. loads ( response [ " body " ]. read () )[ " embedding " ] Enter fullscreen mode Exit fullscreen mode The identity_text function omits unknown attributes: if one listing does not specify the format and another does, including "unknown" would make their identity texts different and push apart two products that might actually be the same. The embed function sends the text to Titan Text Embeddings V2 and returns the generated vector. For this MVP, I configured 1,024 dimensions to match the S3 Vectors index.

Once we have identified the candidate products, the next step is to determine the relationship between them. Similar titles are not enough: we need to check whether they represent the same product, a variant, or different games. To do this, we combine business rules with an AI model that intervenes only when the rules are not sufficient. The following decision tree summarizes the logic implemented in the MVP.

Source: Decision diagram created by the author.

First, we normalize the titles to prevent differences in capitalization, apostrophes, or punctuation from affecting matching. For example, “Link’s Awakening” and “Links Awakening” are considered equivalent after normalization.

If the base titles match, we compare the platform, edition, format, and region.

def rule_decision ( a, b, distance ): """ Returns (relation, reason), or None when the pair goes to Nova Lite. """ if normalize_title ( a [ " base_title " ]) == normalize_title ( b [ " base_title " ]): diffs = [ f for f in [ " platform ", " edition ", " format ", " region " ] if not compatible ( a. get ( f ), b. get ( f )) ] if not diffs: return " duplicate ", " same base_title and compatible attributes " return " variant ", " differs in " + ", ". join ( diffs ) if distance < 0.08: return None # gray zone: Nova Lite decides return " different ", " different base_title " Enter fullscreen mode Exit fullscreen mode The normalize_title and compatible functions apply the rules from steps 1 and 2: the first normalizes the titles, and the second checks whether two attributes match or whether one of them is unknown. When rule_decision returns None, the pair is sent to Nova Lite together with the DECISION_PROMPT.

📄 The full code is in aws/lambda/SP05-game-catalog-decide-pairs/lambda_function.py. In the final run, 10,209 of the 10,449 candidate pairs (97.7%) were classified using deterministic rules applied to the attributes extracted by the AI, and only 240 pairs (2.3%) required an additional evaluation with Nova Lite. This result shows that combining AI-based attribute extraction with deterministic rules can significantly reduce the number of additional model calls.

However, these figures describe how the decisions were made, not how accurately the system identified duplicates. In addition, the distance thresholds used in this MVP (0.25 to select candidates and 0.08 for the gray zone) were set empirically, based on cases observed during development. For a production implementation, a labeled reference set would be needed to check whether these thresholds are appropriate for each use case.

Once the pairs have been classified, we need to turn these individual relationships into groups that we can query and use later. In this MVP, we create two levels of grouping: 🏷️ canonical_id: identifies listings that represent the same commercial product. 👨‍👩‍👧 family_id: groups duplicates and their variants, such as the same game available on different platforms, in different editions, or in different formats. This distinction helps us answer two questions that are often confused: how many listings represent the same product, and how many products belong to the same game family.

So far, the pipeline has processed the entire catalog in a single run. But the result is more than just a duplicate detection report: once processing is complete, we have a product database in DynamoDB, where each listing is associated with its corresponding product and family, and a vector index in S3 Vectors containing each game's vector representation. With these two components, we can query the catalog in real time. This also opens the door to integrating the solution with systems used to manage product information, such as a PIM or ERP. Some possible scenarios include:

The query is sent to Nova Micro using the same extraction prompt as the pipeline. The model returns the normalized attributes:

{ "product_type": "game", "base_title": "FIFA 14", "platform": "PlayStation 3", "edition": "Standard", "format": "digital", "region": "unknown" } Enter fullscreen mode Exit fullscreen mode The key transformation has already happened:** “ps3”** has become “PlayStation 3”, and “codigo digital” has been interpreted as format: digital. The user's query now uses the same vocabulary as the catalog.

Using these attributes, we build the same identity text format used for every game in the catalog, and Titan converts it into a vector:

FIFA 14 | PlayStation 3 | Standard | digital

S3 Vectors returns the candidates with the highest vector similarity. Among them is the listing “FIFA 14 - PS3 [Digital Code]”, which has the same normalized identity text as the query and therefore a distance of zero, along with other physical versions of FIFA 14.

For each candidate, we query DynamoDB to retrieve its canonical_id (which commercial product it represents) and family_id (which game it belongs to).

Using the same rules as the pipeline, we compare the query against each candidate: the digital listing is a duplicate because its title, platform, and format match, while the physical copies are variants because the format is different.

{ "decision": "duplicate", "canonical_id": "CAN-B00GGUW312", "family_id": "FAM-B00CD90R7M", "identity_text": "FIFA 14 | PlayStation 3 | Standard | digital", "matched_with": { "title": "FIFA 14 - PS3 [Digital Code]", "distance": 0, "method": "rule" } } Enter fullscreen mode Exit fullscreen mode The response does more than identify the product: the family_id also allows us to retrieve all its variants (other platforms, physical copies, and different editions).

The game-catalog-resolve-product Lambda function allows us to test different queries and see how the system identifies a product and retrieves its variants. The following images show some examples of the responses. Two fields are especially important:

Source: Screenshot captured by the author from the AWS Lambda console (AWS).

Source: Screenshot captured by the author from the AWS Lambda console (AWS)

The pipeline processed 3,740 product listings in 14 minutes and 4 seconds, from preparing the requests to creating the product clusters.

According to the AWS Step Functions execution history, the most time-consuming stages were:

The game-catalog-extract-ondemand-access Lambda function collects the input and output token counts returned by Bedrock. The execution summary, available in the ExtractAttributes step output in Step Functions, recorded:

This MVP allowed me to demonstrate that deterministic rules, embeddings, and language models can be combined to detect potential duplicates without sending every comparison to an AI model. It also showed that cost does not depend solely on the model selected. Data preparation, attribute selection, prompt design, and the rules used to handle straightforward cases all have a direct impact on resource consumption and execution time.

During the development of this MVP, I tested various products, edge cases, and prompt versions using a subset of the catalog. These tests made it possible to refine the extraction rules and evaluate the technical behavior of the pipeline, but they do not replace a systematic assessment of accuracy using labeled data. However, the results should be interpreted within the scope of the experiment.

Before using a similar solution with a real-world catalog, I would focus on several areas:

⚠️ Note: this project is an exploratory MVP and is not intended for production use as is. Keep in mind that:

Before applying a similar approach to a real catalog, it should be adapted to each organization's business rules, the models should be evaluated, and the accuracy of the results should be measured with your own data.

1.** Adobe. (s. f.)**. Product information management (PIM). https://business.adobe.com/blog/perspectives/product-information-management