Produktiver RAG-Chatbot mit Amazon Bedrock: Was im Alltag wirklich zaehlt

How we built a document Q&A system with NestJS, PostgreSQL, pgvector, Hybrid Search (RRF), Prompt Caching, and Claude on Amazon Bedrock. Building a demo chatbot is easy.

Building one that consistently answers correctly after weeks of document updates, changing requirements, and real user traffic is where things become interesting.

Most Retrieval-Augmented Generation (RAG) failures don't look like failures.

Yet answer quality gradually gets worse.

You ask about error code E-4021 and receive documentation for E-4012.

A recently updated document never appears in search results.

Prompt caching is enabled, but costs don't go down.

Hybrid search is implemented, but retrieval quality barely improves.

The chatbot simply becomes less useful over time.

For teams involved in application software development, the real work begins after the first successful response. Retrieval quality, chunking strategy, embedding consistency, prompt caching, document lifecycle management, and hallucination control quickly become more important than model selection itself.

This article walks through a complete production-oriented RAG implementation built with:

More importantly, it explains why each component exists, what problem it solves, and which production issues it prevents.

RAG (Retrieval-Augmented Generation) does not ask the LLM directly. Before each question, the system:

The problem "how does the LLM know my data" becomes a search problem. If retrieval is weak, the chatbot is weak, because no prompt can produce information that isn't in the context.

This distinction is easy to underestimate.

Many teams spend weeks experimenting with prompts while retrieval quality remains unchanged.

In reality, most enterprise application software development projects that incorporate AI succeed or fail based on retrieval architecture rather than prompt engineering.

That is why most of this article is about chunking documents and search, not about prompts.

The three stages of RAG: documents are indexed ahead of time, then each question goes through retrieval and then generation

══ OFFLINE ══════════════════════════════════════════════════════════ data/in/*.md|txt|json │ ▼ LoaderService ──► ChunkerService ──► Bedrock (Cohere Embed) ──► Postgres read files 2-level split + input_type = chunks table header injection search_document (vector + tsvector) ══ ONLINE ═══════════════════════════════════════════════════════════ POST /api/chat { question } │ ▼ SearchService.hybridSearch ├─ embedQuery (Cohere, input_type = search_query) ├─ DenseSearch (HNSW index) ─┐ run in parallel ├─ KeywordSearch (GIN index) ─┘ └─ reciprocalRankFusion ──► top 5 chunks │ ▼ PromptBuilder ── [system + few-shot: CACHED] + [context + question: not cached] │ ▼ Bedrock (Claude, InvokeModel) │ ▼ { answer, citations, usage } Enter fullscreen mode Exit fullscreen mode Folder layout, one folder per stage:

src/ ├── config/ all config, read from env ├── database/ schema.sql + pg.Pool wrapper (no ORM) ├── llm/ LlmService interface + Bedrock implementation ├── ingest/ loader → chunker → repository → ingest.service ├── search/ dense · keyword · rrf · search.service ├── prompt/ system prompt · few-shot · prompt builder └── chat/ wires the three steps into one RAG loop + HTTP API scripts/ db-init · ingest · search (CLI) test/ unit tests for chunker, rrf, prompt builder Enter fullscreen mode Exit fullscreen mode This architecture may look surprisingly simple.

Every stage has a single responsibility and can be tested independently.

When retrieval quality drops, we can inspect retrieval.

When generation quality drops, we can inspect prompts.

When ingest breaks, we can inspect indexing.

The separation becomes extremely valuable once the system grows beyond a proof of concept.

The diagram in section 2 shows the blocks. This section slows down: it takes one concrete document and one concrete question and follows them through each function, each SQL query, and each Bedrock call. Once you have read this section, the code in the later steps is just detail.

The sample data used throughout is data/in/warranty.md:

# Warranty Policy ## Warranty period All products are covered by a warranty of 12 months from the date of purchase. ## Returns Products with a manufacturing defect can be exchanged within the first 7 days. Enter fullscreen mode Exit fullscreen mode 3.1 The ingest flow: from raw files to the index This flow runs offline via npm run ingest, not over HTTP. Re-run it whenever documents change.

npm run ingest [-- <folder>] │ ▼ scripts/ingest.ts │ NestFactory.createApplicationContext(AppModule) (no HTTP port opened) ▼ IngestService.ingestDirectory(dir = INGEST_DIR) │ ▼ LoaderService.loadDirectory ─────────► RawDocument[] │ recursive scan;.md/.txt = 1 document;.json = n documents; other extensions skipped │ │ for EACH RawDocument: ▼ ChunkerService.chunk ────────────────► Chunk[] │ split by heading, split long sections (1200 chars, 120 overlap), prepend heading ▼ BedrockService.embedDocuments ───────► one 1024-dim vector per chunk │ up to 96 texts per call, Cohere input_type = search_document ▼ ChunkRepository.upsert (per chunk) ──► chunks table │ INSERT... ON CONFLICT (chunk_id) DO UPDATE │ Postgres computes search_vector and updates HNSW + GIN indexes ▼ ChunkRepository.deleteStale ─────────► delete leftover chunks of this document │ ▼ IngestReport { documents, chunks, deleted } → printed to console, pool closed Enter fullscreen mode Exit fullscreen mode Read files. LoaderService walks the folder recursively.

A markdown file becomes one RawDocument:

{ "source": "data/in/warranty.md", "title": "Warranty Policy", "text": "# Warranty Policy \n\n ## Warranty period \n All products are covered..." } Enter fullscreen mode Exit fullscreen mode source is the file path, and title comes from the first H1 (or the file name if there is none). For.json files, each element with a text field becomes its own document.

Chunk. splitByHeading reads line by line and starts a new section whenever a line begins with #. The sample file produces 3 sections:

section A heading = "Warranty Policy" text = "" → no chunk section B heading = "Warranty period" text = "All products are..." section C heading = "Returns" text = "Products with a..." Enter fullscreen mode Exit fullscreen mode Section A has a heading but no body, so it is dropped. Sections B and C are shorter than 1200 characters, so they are kept whole. A longer section would be split further by splitLongText: paragraph boundaries first, then sentences, with 120 characters of overlap between consecutive pieces.

After prepending the heading (header injection), we get 2 chunks:

[ { "chunkId": "data/in/warranty.md#0", "title": "Warranty period", "content": "Warranty period \n\n All products are covered by a warranty of 12 months from the date of purchase.", "metadata": { "heading": "Warranty period" } }, { "chunkId": "data/in/warranty.md#1", "title": "Returns", "content": "Returns \n\n Products with a manufacturing defect can be exchanged within the first 7 days.", "metadata": { "heading": "Returns" } } ] Enter fullscreen mode Exit fullscreen mode chunkId has the form <source>#<position>. It is the key that lets the next ingest update the existing row instead of adding a new one.

Embed. Both content strings go to Cohere in a single InvokeModel call:

{ "texts": [ "Warranty period \n\n All products are...", "Returns \n\n Products with a..." ], "input_type": "search_document", "truncate": "END" } Enter fullscreen mode Exit fullscreen mode The result is 2 vectors of 1024 floats each. Because the heading is part of content, chunk #0's vector carries the meaning "warranty period", not just the sentence about 12 months.

Store. Each chunk is one INSERT... ON CONFLICT (chunk_id) DO UPDATE. The application only writes content and embedding. Postgres does the rest:

Clean up leftovers. deleteStale('data/in/warranty.md', ['...#0', '...#1']) deletes every row of this file whose chunk_id is not in the list just written. On the first run there is nothing to delete. If the file later shrinks from 5 chunks to 2, rows #2, #3 and #4 are deleted here.

At the end, the console prints a summary: number of documents, chunks, and stale chunks deleted.

Three things worth knowing about this flow:

This flow runs online, once per request. One question costs 2 Bedrock calls (1 embed, 1 generate) and 2 SQL queries run in parallel.

Client │ POST /api/chat { "question": "How long is the product warranty?" } ▼ ValidationPipe + AskDto ─────────────► 400 if empty, > 2000 chars, or unknown fields │ ▼ ChatController.ask → ChatService.ask │ ▼ SearchService.hybridSearch(question) │ ├─ [2a] BedrockService.embedQuery ────► question vector (Cohere, input_type = search_query) │ ├─ [2b] in parallel (Promise.all): │ ├─ DenseSearch: ORDER BY embedding <=> $vec LIMIT 50 → HNSW index │ └─ KeywordSearch: search_vector @@ websearch_to_tsquery → GIN index │ └─ [2c] reciprocalRankFusion(k = 60) ─► top 5 chunks, with denseRank / keywordRank │ ▼ PromptBuilder.build ─────────────────► { system:, user: } │ system = SYSTEM_PROMPT + few-shot (cached) │ user = <context>..</context> + <question> (not cached) ▼ BedrockService.generate ─────────────► InvokeModel Claude │ cache_control on the few-shot block; retry on throttling ▼ ChatService assembles the result │ answer = all "text" blocks joined │ citations = the chunks that were sent, numbered to match [n] │ usage = input/output tokens + cache write/read tokens ▼ Client receives { answer, citations, usage } Enter fullscreen mode Exit fullscreen mode Guard the input. ValidationPipe runs with whitelist and forbidNonWhitelisted, so the body may contain only the question field. AskDto requires question to be a non-empty string of at most 2000 characters.

Invalid requests get a 400 immediately, before a single cent is spent on Bedrock.

[2a] Embed the question. The question goes to Cohere with input_type: "search_query". This mirrors step of the ingest flow: documents use search_document, questions use search_query, same model. Get either one wrong and question vectors and document vectors no longer live on the same "map".

[2b] Two search branches run at the same time.

The dense branch compares the question vector with every chunk vector by cosine; thanks to HNSW it does not scan the whole table. With our 2-chunk sample corpus:

dense: 1....#0 "Warranty period" highest cosine 2....#1 "Returns" Enter fullscreen mode Exit fullscreen mode The keyword branch passes the raw question to websearch_to_tsquery('simple',...):

'how' & 'long' & 'is' & 'the' & 'product' & 'warranty' Enter fullscreen mode Exit fullscreen mode This function joins words with AND. Chunk #0 contains none of how, long, is, and has products rather than product (the simple dictionary does not stem), so the keyword branch returns 0 results for this question. That is the real behavior of the code, and the reason the dense branch matters: for long natural-language questions the keyword branch usually stays silent. It shines on short queries containing codes or names, like E-4021 or warranty 12 months.

[2c] Merge with RRF. Each chunk gets 1 / (60 + rank) from every branch it appears in:

...#0 rrf = 1/61 = 0.01639 denseRank = 1 keywordRank = null...#1 rrf = 1/62 = 0.01613 denseRank = 2 keywordRank = null Enter fullscreen mode Exit fullscreen mode If the query were warranty 12 months, chunk #0 would appear in both branches, score 1/61 + 1/61 = 0.03279, and pull far ahead of chunks found by only one branch. The list is cut to the top 5 ( TOP_K ).

Build the prompt. PromptBuilder numbers the chunks from 1 and wraps them in tags (the repository uses Vietnamese prompt text and tag names; English equivalents are shown throughout this article, see section 12):

<context> Warranty period Warranty period All products are covered by a warranty of 12 months from the date of purchase. Returns Returns Products with a manufacturing defect can be exchanged within the first 7 days. </context> <question>How long is the product warranty?</question> Enter fullscreen mode Exit fullscreen mode The heading appears twice because title is printed first while content already contains the heading from header injection. It's harmless, but you can drop the title line when rendering to save tokens.

If neither branch finds anything, the context is (no relevant documents found) and the LLM is still called. The system prompt and few-shot example #2 teach the model to say it could not find the information instead of inventing an answer.

Call Claude. The body sent via InvokeModel:

{ "anthropic_version": "bedrock-2023-05-31", "max_tokens": 2000, "system": [ { "type": "text", "text": "You are a Q&A assistant that answers from the provided documents..." }, { "type": "text", "text": "Examples of good answers:...", "cache_control": { "type": "ephemeral" } } ], "messages": [ { "role": "user", "content": [ { "type": "text", "text": "<context> \n Warranty period \n... \n </context>" }, { "type": "text", "text": "<question>How long is the product warranty?</question>" } ] } ] } Enter fullscreen mode Exit fullscreen mode Everything before cache_control (system prompt + few-shot) is identical on every request, so from the second request on it is read from cache. messages changes with every question and is always billed at full price. If Bedrock returns ThrottlingException or ServiceUnavailableException, the request is retried after 500ms and then 1s; any other error is thrown immediately.

Return the result. Claude returns a content array. The code keeps only type: "text" blocks and joins them, ignoring other block types. citations is the list of chunks put into the prompt, numbered to match [n] in the context, so the client can turn in the answer into a link to source:

{ "answer": "All products are covered by a 12-month warranty from the date of purchase.", "citations": [ { "index": 1, "chunkId": "data/in/warranty.md#0", "source": "data/in/warranty.md", "title": "Warranty period" }, { "index": 2, "chunkId": "data/in/warranty.md#1", "source": "data/in/warranty.md", "title": "Returns" } ], "usage": { "inputTokens": 180, "outputTokens": 25, "cacheWriteTokens": 0, "cacheReadTokens": 650 } } Enter fullscreen mode Exit fullscreen mode (The usage values above are illustrative.) Note that citations contains every chunk that was sent, not only the ones the model actually cited. The client should filter by the [n] numbers that appear in answer.

Accuracy doesn't come from one place. It accumulates across both flows:

What these two flows don't do yet, covered in the "Upgrade path" section at the end: Unicode NFC normalization for both documents and questions, rewriting questions using conversation history (the API is currently stateless), a confidence threshold to skip the LLM when results are too weak, and checking in code that the [n] the model cites actually exist.

One design goal guided every technology choice: keep the architecture understandable.

Every component should be replaceable without rewriting the rest of the system.

That's the entire runtime dependency list:

"dependencies": { "@aws-sdk/client-bedrock-runtime": "^3.696.0", "@nestjs/common": "^10.4.0", "@nestjs/core": "^10.4.0", "@nestjs/platform-express": "^10.4.0", "class-transformer": "^0.5.1", "class-validator": "^0.14.1", "dotenv": "^16.4.5", "pg": "^8.12.0", "reflect-metadata": "^0.2.2", "rxjs": "^7.8.1" } Enter fullscreen mode Exit fullscreen mode 5. Prerequisites 5.1 Local environment Node.js 20+ Docker + Docker Compose An AWS account 5.2 Enable models on Amazon Bedrock Open the Amazon Bedrock console and pick your region (this article uses ap-northeast-1 ). Make sure your account has access to two models: Cohere Embed Multilingual v3 and a Claude model.

The first time you use an Anthropic model, AWS may ask you to fill in a use-case form. Open the Model catalog and note the exact model ID for Claude. Bedrock model IDs have the anthropic. prefix, for example anthropic.claude-opus-5-5.

Some regions require calling through a cross-region inference profile; in that case use the profile ID the console shows. 5.3 Minimal IAM policy The application only needs bedrock:InvokeModel:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "bedrock:InvokeModel" ], "Resource": [ "arn:aws:bedrock:*::foundation-model/*", "arn:aws:bedrock:*:*:inference-profile/*" ] } ] } Enter fullscreen mode Exit fullscreen mode In production, narrow Resource down to the exact ARNs of the two models you use.

About credentials: static access keys work, but it's better to leave them empty and let the AWS SDK pick up AWS_PROFILE, SSO, or the EC2/ECS IAM role. The code below supports both.

npm i -g @nestjs/cli nest new rag-chatbot --package-manager npm --skip-git cd rag-chatbot npm i @aws-sdk/client-bedrock-runtime pg dotenv class-validator class-transformer npm i -D @types/pg ts-node Enter fullscreen mode Exit fullscreen mode Add these scripts to package.json:

"scripts": { "build": "nest build", "start:dev": "nest start --watch", "start:prod": "node dist/main", "db:init": "ts-node scripts/db-init.ts", "ingest": "ts-node scripts/ingest.ts", "search": "ts-node scripts/search.ts", "test": "jest", "lint": "eslint \" {src,scripts,test}/**/*.ts \" " } Enter fullscreen mode Exit fullscreen mode 7. Step 2: Config in one place Every setting is read from env, with sensible defaults so it runs out of the box. No per-project profile files.

# ── App PORT = 8000 # ── PostgreSQL + pgvector DB_HOST = 127.0.0.1 DB_PORT = 5432 DB_NAME = ragdb DB_USER = postgres DB_PASSWORD = postgres # Must match the embedding model's dimensions and the vector(N) column in the schema. # cohere.embed-multilingual-v3 → 1024. Changing it = re-run db:init + re-embed everything. EMBEDDING_DIMENSIONS = 1024 # ── AWS Bedrock AWS_REGION = ap-northeast-1 AWS_ACCESS_KEY_ID = AWS_SECRET_ACCESS_KEY = AWS_SESSION_TOKEN = # Leave the keys above empty to let the SDK use AWS_PROFILE / SSO / instance role.

BEDROCK_EMBEDDING_MODEL_ID = cohere.embed-multilingual-v3 BEDROCK_GENERATION_MODEL_ID = anthropic.claude-opus-5-5 # ── Ingest / chunking INGEST_DIR =./data/in CHUNK_MAX_CHARS = 1200 CHUNK_OVERLAP_CHARS = 120 EMBED_BATCH_SIZE = 96 # ── Search TOP_K = 5 # number of chunks finally placed in the prompt CANDIDATE_LIMIT = 50 # candidates fetched by EACH branch before fusion RRF_K = 60 # RRF constant k, 60 is the value from the original paper # ── Generation MAX_TOKENS = 2000 ENABLE_PROMPT_CACHE = true Enter fullscreen mode Exit fullscreen mode src/config/config.ts (abridged; the full version has an AppConfig interface documenting each field):

const num = ( v: string | undefined, fallback: number ) => v === undefined || v === ''? fallback: Number ( v ); export function loadConfig (): AppConfig { return { port: num ( process. env. PORT, 8000 ), database: { host: process. env. DB_HOST || ' 127.0.0.1 ', port: num ( process. env.

DB_PORT, 5432 ), name: process. env. DB_NAME || ' ragdb ', user: process. env. DB_USER || ' postgres ', password: process. env.

DB_PASSWORD || ' postgres ', embeddingDimensions: num ( process. env. EMBEDDING_DIMENSIONS, 1024 ), }, bedrock: { region: process. env. AWS_REGION || ' ap-northeast-1 ', accessKeyId: process. env.

AWS_ACCESS_KEY_ID, secretAccessKey: process. env. AWS_SECRET_ACCESS_KEY, sessionToken: process. env. AWS_SESSION_TOKEN, embeddingModelId: process. env.

BEDROCK_EMBEDDING_MODEL_ID || ' cohere.embed-multilingual-v3 ', generationModelId: process. env. BEDROCK_GENERATION_MODEL_ID || ' anthropic.claude-opus-5-5 ', }, ingest: { dir: process. env. INGEST_DIR || './data/in ', chunkMaxChars: num ( process. env.

CHUNK_MAX_CHARS, 1200 ), chunkOverlapChars: num ( process. env. CHUNK_OVERLAP_CHARS, 120 ), embedBatchSize: num ( process. env. EMBED_BATCH_SIZE, 96 ), }, search: { topK: num ( process. env.

TOP_K, 5 ), candidateLimit: num ( process. env. CANDIDATE_LIMIT, 50 ), rrfK: num ( process. env. RRF_K, 60 ), }, generation: { maxTokens: num ( process. env.

MAX_TOKENS, 2000 ), enableCache: process. env. ENABLE_PROMPT_CACHE!== ' false ', }, }; } export const CONFIG = Symbol ( ' CONFIG ' ); Enter fullscreen mode Exit fullscreen mode Register CONFIG as a global provider in AppConfigModule ( { provide: CONFIG, useFactory: loadConfig } ) so every service can inject it with @Inject(CONFIG).

name: rag-core services: postgres: image: pgvector/pgvector:pg16 # pgvector 0.7+, needed for the HNSW index container_name: rag-postgres environment: POSTGRES_USER: ${DB_USER:-postgres} POSTGRES_PASSWORD: ${DB_PASSWORD:-postgres} POSTGRES_DB: ${DB_NAME:-ragdb} ports: - " ${DB_PORT:-5432}:5432" volumes: - pgvector_data:/var/lib/postgresql/data healthcheck: test: [ " CMD-SHELL", " pg_isready -U ${DB_USER:-postgres} -d ${DB_NAME:-ragdb}" ] interval: 5s timeout: 5s retries: 10 volumes: pgvector_data: Enter fullscreen mode Exit fullscreen mode name: rag-core is there for a reason: without it, Compose uses the folder name as the project name, and two checkouts with the same folder name on one machine will silently recreate each other's containers.

CREATE EXTENSION IF NOT EXISTS vector; CREATE TABLE IF NOT EXISTS chunks ( id BIGSERIAL PRIMARY KEY, -- Logical key "<source>#<index>": re-ingesting the same file upserts the same row. chunk_id TEXT NOT NULL UNIQUE, source TEXT NOT NULL, chunk_index INT NOT NULL, title TEXT, content TEXT NOT NULL, metadata JSONB NOT NULL DEFAULT '{}':: jsonb, embedding vector ( 1024 ), -- Auto-generated full-text column, 'simple' dictionary. search_vector tsvector GENERATED ALWAYS AS ( to_tsvector ( 'simple', coalesce ( title, '' ) || ' ' || content ) ) STORED, created_at TIMESTAMPTZ NOT NULL DEFAULT NOW (), updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW () ); -- Vector search: HNSW + cosine CREATE INDEX IF NOT EXISTS chunks_embedding_idx ON chunks USING hnsw ( embedding vector_cosine_ops ); -- Full-text search CREATE INDEX IF NOT EXISTS chunks_fts_idx ON chunks USING gin ( search_vector ); CREATE INDEX IF NOT EXISTS chunks_source_idx ON chunks ( source ); Enter fullscreen mode Exit fullscreen mode Three decisions worth noting:

scripts/db-init.ts reads the SQL file, swaps in the vector dimension from EMBEDDING_DIMENSIONS, and runs it. Safe to run any number of times (every statement is IF NOT EXISTS ):

const sql = await fs. readFile ( path. join ( __dirname, '../src/database/schema.sql ' ), ' utf8 ' ); await client. connect (); await client. query ( sql. replace ( ' vector(1024) ', `vector( ${ config. database. embeddingDimensions } )` )); await client. end (); Enter fullscreen mode Exit fullscreen mode 8.4 Database service: a thin wrapper around pg.Pool @ Injectable () export class DatabaseService implements OnModuleDestroy { private readonly pool: Pool; constructor (@ Inject ( CONFIG ) config: AppConfig ) { this. pool = new Pool ({ host: config. database. host, port: config. database. port, database: config. database. name, user: config. database. user, password: config. database. password, }); } async query < T extends QueryResultRow > ( sql: string, params: unknown [] = []): Promise < T [] > { const result = await this. pool. query < T > ( sql, params ); return result. rows; } /** pgvector accepts a vector as the literal '[1,2,3]' cast to::vector. */ static toVector ( values: number []): string { return `[ ${ values. join ( ', ' )} ]`; } async onModuleDestroy (): Promise < void > { await this. pool. end (); } } Enter fullscreen mode Exit fullscreen mode 9. Step 4: The LLM layer, calling Bedrock 9.1 Interface first, implementation second The rest of the codebase only knows this interface, not Bedrock:

export const LLM = Symbol ( ' LLM ' ); export interface PromptBlock { text: string; /** true = block belongs to the static prefix and gets a cache_control marker. */ cacheable?: boolean; } export interface GenerateRequest { system: PromptBlock []; // persona, rules, few-shot: the cached part user: PromptBlock []; // retrieved context + question: changes every turn maxTokens?: number; } export interface Usage { inputTokens: number; outputTokens: number; cacheWriteTokens: number; cacheReadTokens: number; } export interface LlmService { embedDocuments ( texts: string []): Promise < number [][] >; embedQuery ( text: string ): Promise < number [] >; generate ( request: GenerateRequest ): Promise < { text: string; usage: Usage } >; } Enter fullscreen mode Exit fullscreen mode // src/llm/llm.module.ts: switching providers = changing this one useClass line @ Global () @ Module ({ providers: [{ provide: LLM, useClass: BedrockService }], exports: [ LLM ], }) export class LlmModule {} Enter fullscreen mode Exit fullscreen mode 9.2 Client and retries @ Injectable () export class BedrockService implements LlmService { private readonly client: BedrockRuntimeClient; constructor (@ Inject ( CONFIG ) private readonly config: AppConfig ) { const { region, accessKeyId, secretAccessKey, sessionToken } = config. bedrock; this. client = new BedrockRuntimeClient ({ region, // Don't pass credentials when there are no static keys, // so the SDK uses its own chain (AWS_PROFILE, SSO, instance role)....( accessKeyId && secretAccessKey? { credentials: { accessKeyId, secretAccessKey, sessionToken } }: {}), }); } private async invoke ( modelId: string, body: unknown ): Promise < Record < string, any >> { let lastError: unknown; for ( let attempt = 1; attempt <= 3; attempt ++ ) { try { const response = await this. client. send ( new InvokeModelCommand ({ modelId, contentType: ' application/json ', accept: ' application/json ', body: JSON. stringify ( body ), }), ); return JSON. parse ( new TextDecoder (). decode ( response. body )); } catch ( err ) { lastError = err; const name = ( err as { name?: string }). name?? ''; const retryable = name === ' ThrottlingException ' || name === ' ServiceUnavailableException '; if (! retryable || attempt === 3 ) break; await new Promise (( r ) => setTimeout ( r, 2 ** attempt * 250 )); // 500ms, 1s } } throw lastError; } Enter fullscreen mode Exit fullscreen mode Only transient errors (throttling, service unavailable) are retried, with exponential backoff. A ValidationException or AccessDeniedException won't change no matter how many times you retry, so those are thrown immediately.

private async embed ( texts: string [], inputType: ' search_document ' | ' search_query ', ): Promise < number [][] > { const out: number [][] = []; const batchSize = Math. min ( this. config. ingest. embedBatchSize, 96 ); // Cohere: max 96 texts per call for ( let i = 0; i < texts. length; i += batchSize ) { const body = await this. invoke ( this. config. bedrock. embeddingModelId, { texts: texts. slice ( i, i + batchSize ), input_type: inputType, truncate: ' END ', }); out. push (...( body. embeddings as number [][])); } return out; } async embedDocuments ( texts: string []) { return texts. length? this. embed ( texts, ' search_document ' ): []; } async embedQuery ( text: string ) { const [ embedding ] = await this. embed ([ text ], ' search_query ' ); return embedding; } Enter fullscreen mode Exit fullscreen mode input_type directly affects results. Cohere only projects documents and queries into the same space when each side declares its role correctly: documents are search_document, questions are search_query. Using the same value for both still runs and reports no error; recall just drops a few percent, and hardly anyone notices.

The body sent to Claude via InvokeModel follows the Anthropic Messages format, with anthropic_version: 'bedrock-2023-05-31':

type TextBlock = { type: ' text '; text: string; cache_control?: { type: ' ephemeral ' } }; /** Put cache_control on the LAST block of the static prefix. */ private toBlocks ( blocks: PromptBlock []): TextBlock [] { const out: TextBlock [] = blocks. filter (( b ) => b. text. trim ()). map (( b ) => ({ type: ' text ' as const, text: b. text })); if (! this. config. generation. enableCache ) return out; const lastCacheable = blocks. reduce ( ( acc, b, i ) => ( b. cacheable && b. text. trim ()? i: acc ), - 1, ); if ( lastCacheable >= 0 && out [ lastCacheable ]) { out [ lastCacheable ]. cache_control = { type: ' ephemeral ' }; } return out; } async generate ( request: GenerateRequest ) { const body = await this. invoke ( this. config. bedrock. generationModelId, { anthropic_version: ' bedrock-2023-05-31 ', max_tokens: request. maxTokens?? this. config. generation. maxTokens, system: this. toBlocks ( request. system ), messages: [{ role: ' user ', content: this. toBlocks ( request. user ) }], }); // Keep only 'text' blocks. Newer Claude models may return other block // types (e.g. thinking); without this filter they leak into the answer. const text = (( body. content?? []) as Array < { type: string; text?: string } > ). filter (( block ) => block. type === ' text ' ). map (( block ) => block. text?? '' ). join ( '' ). trim (); const usage = body. usage?? {}; return { text, usage: { inputTokens: usage. input_tokens?? 0, outputTokens: usage. output_tokens??

0, cacheWriteTokens: usage. cache_creation_input_tokens?? 0, cacheReadTokens: usage. cache_read_input_tokens?? 0, }, }; } } Enter fullscreen mode Exit fullscreen mode Why the cache marker goes on the last block of the static part: see Step 7.

The pipeline has four classes, each with one job:

LoaderService ──► ChunkerService ──► LlmService.embedDocuments ──► ChunkRepository (read sources) (chunking) (batches of 96) (upsert + clean up) ▲ IngestService only orchestrates, no logic of its own Enter fullscreen mode Exit fullscreen mode 10.1 Loader: the only place that knows where data comes from export interface RawDocument { source: string; // file path, URL, CMS id... title?: string; text: string; metadata?: Record < string, unknown >; } Enter fullscreen mode Exit fullscreen mode LoaderService scans the folder recursively:

async loadTextFile ( file: string ): Promise < RawDocument > { const text = await fs. readFile ( file, ' utf8 ' ); return { source: file, title: /^# \s + (.+ ) $/m. exec ( text )?.[ 1 ]?. trim ()?? path. basename ( file ), text, }; } Enter fullscreen mode Exit fullscreen mode To add S3, Confluence, Notion and so on, add one method that returns RawDocument[]. Chunking and embedding stay the same.

This step has a big impact on quality but is often done carelessly.

@ Injectable () export class ChunkerService { constructor (@ Inject ( CONFIG ) private readonly config: AppConfig ) {} chunk ( doc: RawDocument ): Chunk [] { const { chunkMaxChars, chunkOverlapChars } = this. config. ingest; const pieces: Array < { heading?: string; text: string } > = []; for ( const section of this. splitByHeading ( doc. text )) { for ( const part of this. splitLongText ( section. text, chunkMaxChars, chunkOverlapChars )) { pieces. push ({ heading: section. heading, text: part }); } } return pieces. map (( piece, index ) => ({ chunkId: ` ${ doc. source } # ${ index } `, source: doc. source, chunkIndex: index, title: piece. heading?? doc. title, // Header injection: the heading goes into the embedding and FTS with the content. content: piece. heading? ` ${ piece. heading } \n\n ${ piece. text } `: piece. text, metadata: {...( doc. metadata?? {}), heading: piece. heading?? null }, })); } private splitByHeading ( text: string ) { const sections: Array < { heading?: string; text: string [] } > = []; let current: { heading?: string; text: string [] } = { text: [] }; for ( const line of text. split ( ' \n ' )) { const heading = /^ ( # {1,6})\s + (.+ ) $/. exec ( line ); if ( heading ) { if ( current. text. join ( '' ). trim () || current. heading ) sections. push ( current ); current = { heading: heading [ 2 ]. trim (), text: [] }; } else { current. text. push ( line ); } } sections. push ( current ); return sections. map (( s ) => ({ heading: s. heading, text: s. text. join ( ' \n ' ). trim () })). filter (( s ) => s. text || s. heading ); } private splitLongText ( text: string, maxChars: number, overlapChars: number ): string [] { if ( text. length <= maxChars ) return text? [ text ]: []; // Prefer paragraph boundaries, then sentences (Japanese punctuation included). const units = text. split ( / \n{2,} / ). flatMap (( p ) => ( p. length <= maxChars? [ p ]: p. split ( / (?< = [.!?。？！ ])\s +/ ))). filter (( u ) => u. trim ()); const out: string [] = []; let buffer = ''; const flush = () => { if (! buffer. trim ()) return; out. push ( buffer. trim ()); buffer = overlapChars > 0? buffer. slice ( - overlapChars ): ''; // overlap }; for ( const unit of units ) { if ( buffer && buffer. length + unit. length + 2 > maxChars ) flush (); if ( unit. length > maxChars ) { // The smallest unit is still too long, no boundary left to respect. for ( let i = 0; i < unit. length; i += maxChars - overlapChars ) { out. push ( unit. slice ( i, i + maxChars ). trim ()); } buffer = ''; continue; } buffer = buffer? ` ${ buffer } \n\n ${ unit } `: unit; } if ( buffer. trim ()) out. push ( buffer. trim ()); return out. filter ( Boolean ); } } Enter fullscreen mode Exit fullscreen mode If you can make only one chunking improvement, make it header injection: it costs almost nothing (string concatenation) and improves both the vector branch and the keyword branch.

A note on size: CHUNK_MAX_CHARS=1200 counts characters. For the same content, Vietnamese takes roughly 1.5x as many tokens as English, and Japanese roughly 3x. For Japanese documents, lower this limit so chunks stay within the embedding model's limit (anything beyond it is silently cut by truncate: 'END' ).

async upsert ( chunk: Chunk, embedding: number []): Promise < void > { await this. db. query ( ` INSERT INTO chunks (chunk_id, source, chunk_index, title, content, metadata, embedding) VALUES ($1, $2, $3, $4, $5, $6::jsonb, $7::vector) ON CONFLICT (chunk_id) DO UPDATE SET source = EXCLUDED.source, chunk_index = EXCLUDED.chunk_index, title = EXCLUDED.title, content = EXCLUDED.content, metadata = EXCLUDED.metadata, embedding = EXCLUDED.embedding, updated_at = NOW() `, [ chunk. chunkId, chunk. source, chunk. chunkIndex, chunk. title?? null, chunk. content, JSON. stringify ( chunk. metadata ), DatabaseService. toVector ( embedding )], ); } async deleteStale ( source: string, keepChunkIds: string []): Promise < number > { const rows = await this. db. query < { chunk_id: string } > ( `DELETE FROM chunks WHERE source = $1 AND chunk_id <> ALL($2::text[]) RETURNING chunk_id`, [ source, keepChunkIds ], ); return rows. length; } Enter fullscreen mode Exit fullscreen mode Upsert alone is not enough for safe re-ingestion. Say a document used to have 10 chunks and the new version has 6. Upsert updates the first 6 rows, but the last 4 stay there with their old embeddings and keep being retrieved.

The chatbot ends up answering from content that no longer exists.

deleteStale deletes by exclusion after the upsert finishes, instead of "delete everything, then write again". That way search never sees a window where the document is missing.

Abridged version (in the repo, embed + upsert live in a separate embedAndStore method):

async ingestDocuments ( docs: RawDocument []): Promise < IngestReport > { const report = { documents: 0, chunks: 0, deleted: 0 }; for ( const doc of docs ) { const chunks = this. chunker. chunk ( doc ); if (! chunks. length ) continue; // Embed in batches, then upsert each chunk for ( let i = 0; i < chunks. length; i += this. config. ingest. embedBatchSize ) { const batch = chunks. slice ( i, i + this. config. ingest. embedBatchSize ); const embeddings = await this. llm. embedDocuments ( batch. map (( c ) => c. content )); for ( const [ j, chunk ] of batch. entries ()) await this. repo. upsert ( chunk, embeddings [ j ]); } report. deleted += await this. repo. deleteStale ( doc. source, chunks. map (( c ) => c. chunkId )); report. documents += 1; report. chunks += chunks. length; } return report; } Enter fullscreen mode Exit fullscreen mode The CLI script scripts/ingest.ts boots Nest as an application context (no HTTP server) and calls the service:

const app = await NestFactory. createApplicationContext ( AppModule, { logger: [ ' log ', ' warn ', ' error ' ] }); const report = await app. get ( IngestService ). ingestDirectory ( process. argv [ 2 ]); console. log ( `done: ${ report. documents } documents, ${ report. chunks } chunks, ${ report. deleted } stale chunks deleted` ); await app. close (); Enter fullscreen mode Exit fullscreen mode 11. Step 6: Hybrid search (vector + full-text) merged with RRF 11.1 Why you need both branches Each branch fails exactly where the other is strong:

You combine the two because they cover for each other, not because more branches are always better.

async search ( queryVector: number [], limit: number ): Promise < Candidate [] > { const rows = await this. db. query < any > ( ` SELECT chunk_id AS "chunkId", source, title, content, metadata, 1 - (embedding <=> $1::vector) AS score FROM chunks WHERE embedding IS NOT NULL ORDER BY embedding <=> $1::vector LIMIT $2::int `, [ DatabaseService. toVector ( queryVector ), limit ], ); return rows. map (( r ) => ({... r, score: Number ( r. score ) })); } Enter fullscreen mode Exit fullscreen mode <=> is pgvector's cosine distance operator, so 1 - distance is cosine similarity. The query deliberately has just ORDER BY embedding <=> $1 LIMIT n, because that is the shape the planner needs to use the HNSW index. If you add filters, check again with EXPLAIN, since it is very easy to turn this into a full table scan.

async search ( queryText: string, limit: number ): Promise < Candidate [] > { const rows = await this. db. query < any > ( ` WITH tsq AS (SELECT websearch_to_tsquery('simple', $1) AS q) SELECT c.chunk_id AS "chunkId", c.source, c.title, c.content, c.metadata, ts_rank(c.search_vector, tsq.q) AS score FROM chunks c, tsq WHERE c.search_vector @@ tsq.q ORDER BY score DESC LIMIT $2::int `, [ queryText, limit ], ); return rows. map (( r ) => ({... r, score: Number ( r. score ) })); } Enter fullscreen mode Exit fullscreen mode websearch_to_tsquery accepts the syntax users already know ( "exact phrase", a OR b, -exclude ) and never throws a syntax error. Passing it as a bind parameter is safe enough; no extra escaping needed. ts_rank is not BM25 (Postgres has no built-in BM25). With RRF that doesn't matter, as you'll see right below.

11.4 Merging two lists: why not add the scores The "obvious" approach is score = 0.7 × cosine + 0.3 × ts_rank. It's wrong:

Reciprocal Rank Fusion (RRF) throws away the raw scores and uses only ranks:

RRF(d) = Σ 1 / (k + rank_branch(d)) k = 60 branch Enter fullscreen mode Exit fullscreen mode A worked example:

Dense: 1. chunk_A 2. chunk_B 3. chunk_C Keyword: 1. chunk_C 2. chunk_A 3. chunk_D chunk_A: 1/61 + 1/62 = 0.03252 ← rank 1 chunk_C: 1/63 + 1/61 = 0.03226 ← rank 2 chunk_B: 1/62 = 0.01613 chunk_D: 1/63 = 0.01587 Enter fullscreen mode Exit fullscreen mode chunk_A wins without being first in either branch, because it ranks well in both. Agreement between two independent methods is a stronger signal than a high score from one.

The constant k flattens the curve: with k = 60, the gap between rank 1 and rank 2 is only about 1.6%, so appearing in several lists matters more than the exact rank.

export function reciprocalRankFusion ( branches: { dense: Candidate []; keyword: Candidate [] }, k = 60, topK = 5, ): SearchResult [] { const byId = new Map < string, SearchResult > (); const ingest = ( candidates: Candidate [], branch: ' denseRank ' | ' keywordRank ' ) => { candidates. forEach (( candidate, i ) => { const rank = i + 1; const existing = byId. get ( candidate. chunkId ); if ( existing ) { existing. rrfScore += 1 / ( k + rank ); existing [ branch ] = rank; return; } const { score: _ignored,... rest } = candidate; byId. set ( candidate. chunkId, {... rest, rrfScore: 1 / ( k + rank ), denseRank: null, keywordRank: null, [ branch ]: rank, } as SearchResult ); }); }; ingest ( branches. dense, ' denseRank ' ); ingest ( branches. keyword, ' keywordRank ' ); return [... byId. values ()]. sort (( a, b ) => b. rrfScore - a. rrfScore ). slice ( 0, topK ); } Enter fullscreen mode Exit fullscreen mode The result keeps each chunk's denseRank and keywordRank. When debugging, one look tells you whether a result came from semantics, keywords, or both.

async hybridSearch ( query: string, options: SearchOptions = {}): Promise < SearchResult [] > { const topK = options. topK?? this. config. search. topK; // 5 const limit = options. candidateLimit?? this. config. search. candidateLimit; // 50 const queryVector = await this. llm. embedQuery ( query ); const [ dense, keyword ] = await Promise. all ([ this. dense. search ( queryVector, limit ), this. keyword. search ( query, limit ), ]); return reciprocalRankFusion ({ dense, keyword }, this. config. search. rrfK, topK ); } Enter fullscreen mode Exit fullscreen mode Two details decide the quality:

Adding a third branch (multi-query, or real BM25) is just one more array passed to RRF. No score normalization, no weight tuning.

The repository ships this prompt in Vietnamese (with <ngu_canh> / <cau_hoi> tags); the English equivalent is shown here.

export const SYSTEM_PROMPT = `You are a Q&A assistant that answers from the provided documents. Answering rules: - Answer only from the CONTEXT provided. Do not guess, and do not use knowledge outside the context. - If the context is not enough to answer, say plainly that the information was not found. A wrong answer is more dangerous than "I don't know". - Cite sources with [number] matching the position of the context passage you relied on. - Answer in the same language as the question. - Be concise and get straight to the point. Do not start with "Based on the documents...".`; Enter fullscreen mode Exit fullscreen mode Three rules for this file:

The rule "answer in the same language as the question" is essential: when documents are in English and the user asks in another language, without this rule the model tends to answer in the language of the context.

export const EXAMPLES: Example [] = [ { question: ' How long is the product warranty? ', context: ' Warranty policy \n All products are covered by a 12-month warranty from the date of purchase. \n\n ' + ' Returns \n Defective products can be exchanged within the first 7 days. ', answer: ' Products are covered by a 12-month warranty from the date of purchase. ', }, { // Negative example: there is context, but no passage answers the question. question: ' Does the company have a branch in Da Nang? ', context: ' Warranty policy \n All products are covered by a 12-month warranty from the date of purchase. ', answer: ' The available documents do not mention any branch information. ', }, ]; Enter fullscreen mode Exit fullscreen mode Examples are the strongest signal in a prompt: the citation format is easy to misread when described in words, but obvious from a single example.

The refusal example matters as much as the answer example. If every example is a successful answer, the model learns that there must always be an answer. Example #2 teaches it to decline when the context is irrelevant.

Prompt caching is a prefix match: the system hashes the prompt from the start up to the cache_control marker. Changing one byte at position N invalidates the cache for everything from N onward. Consequence: content must be ordered from most stable to most volatile.

┌─ system ─────────────────────────────────── CACHED ─┐ │ SYSTEM_PROMPT persona + rules, never changes │ │ EXAMPLES few-shot, never changes │ ← cache_control └─────────────────────────────────────────────────────┘ ┌─ user ───────────────────────────────── NOT CACHED ─┐ │ CONTEXT retrieved chunks, per question │ │ QUESTION the question, every turn │ └─────────────────────────────────────────────────────┘ Enter fullscreen mode Exit fullscreen mode @ Injectable () export class PromptBuilder { build ( question: string, results: SearchResult []): GenerateRequest { const system: PromptBlock [] = [ { text: SYSTEM_PROMPT, cacheable: true }, { text: renderExamples (), cacheable: true }, ]; const user: PromptBlock [] = [ { text: `<context>\n ${ this. renderContext ( results )} \n</context>` }, { text: `<question> ${ question } </question>` }, ]; return { system, user }; } /** Number from 1 so the model cites,... consistent with the few-shot. */ private renderContext ( results: SearchResult []): string { if (! results. length ) return ' (no relevant documents found) '; return results. map (( r, i ) => `[ ${ i + 1 } ] ${ r. title? ` ${ r. title } \n`: '' }${ r. content } ` ). join ( ' \n\n ' ); } } Enter fullscreen mode Exit fullscreen mode Why is the retrieved context NOT in the cached region? Each question retrieves a different set of chunks. Put it before the marker and every request creates a new cache entry that no later request can read.

You end up paying the cache write premium (about 1.25x the input price) without ever getting the cache read price (about 0.1x or lower, depending on the model).

Cache economics: with a 5-minute TTL, you break even on the second request (1.25 + 0.1 = 1.35 versus 2x without caching). A chatbot with a steady stream of users is the ideal case.

Minimum length: a prefix shorter than the model's minimum cacheable length is not cached, and no error tells you so. The minimum varies by model (from a few hundred to a few thousand tokens). The system prompt + few-shot above is fairly short, so check cacheWriteTokens on the first request. If it is 0, the prefix is too short: add business rules, add few-shot examples, or accept running without cache.

Why <context> and <question> tags: XML tags mark a clear boundary between data and question, which helps the model avoid treating document content as instructions.

@ Injectable () export class ChatService { constructor ( @ Inject ( LLM ) private readonly llm: LlmService, private readonly search: SearchService, private readonly prompt: PromptBuilder, ) {} async ask ( question: string ): Promise < ChatResponse > { const results = await this. search. hybridSearch ( question ); // 1. retrieve const request = this. prompt. build ( question, results ); // 2. build prompt const { text, usage } = await this. llm. generate ( request ); // 3. generate return { answer: text, citations: results. map (( r, i ) => ({ index: i + 1, // matches [n] in the answer chunkId: r. chunkId, source: r. source, title: r. title, })), usage, }; } } Enter fullscreen mode Exit fullscreen mode 13.2 Controller class AskDto { @ IsString () @ IsNotEmpty () @ MaxLength ( 2000 ) question!: string; } @ Controller ( ' api ' ) export class ChatController { constructor ( private readonly chat: ChatService, private readonly search: SearchService, private readonly repo: ChunkRepository, private readonly db: DatabaseService, ) {} @ Get ( ' health ' ) async health () { return { status: ( await this. db. ping ())? ' ok ': ' degraded ' }; } @ Post ( ' chat ' ) ask (@ Body () dto: AskDto ) { return this. chat. ask ( dto. question ); } /** Inspect retrieval results directly, without the LLM. Useful when tuning. */ @ Get ( ' search ' ) search_ (@ Query ( ' q ' ) q: string, @ Query ( ' mode ' ) mode?: string ) { if ( mode === ' dense ' ) return this. search. denseSearch ( q ); if ( mode === ' keyword ' ) return this. search. keywordSearch ( q ); return this. search. hybridSearch ( q ); } @ Get ( ' stats ' ) stats () { return this. repo. stats (); } } Enter fullscreen mode Exit fullscreen mode Besides validation, @MaxLength(2000) stops users from sending huge questions that inflate token costs.

import ' dotenv/config '; async function bootstrap (): Promise < void > { const app = await NestFactory. create ( AppModule ); const config = app. get < AppConfig > ( CONFIG ); app. useGlobalPipes ( new ValidationPipe ({ whitelist: true, forbidNonWhitelisted: true, transform: true }), ); // Let the pool close cleanly on SIGTERM instead of cutting off running queries. app. enableShutdownHooks (); await app. listen ( config. port ); } bootstrap (); Enter fullscreen mode Exit fullscreen mode 14. Step 9: End-to-end test run cp.env.example.env # fill in AWS credentials (or use AWS_PROFILE) docker compose up -d # Postgres + pgvector npm install npm run db:init # create extension, table, indexes Enter fullscreen mode Exit fullscreen mode Put a few markdown files in data/in/, for example data/in/warranty.md:

# Warranty Policy ## Warranty period All products are covered by a warranty of 12 months from the date of purchase. ## Returns Products with a manufacturing defect can be exchanged within the first 7 days. Enter fullscreen mode Exit fullscreen mode npm run ingest npm run start:dev Enter fullscreen mode Exit fullscreen mode Check retrieval before spending generation tokens. Make this a habit from day one when tuning RAG:

npm run search -- "warranty 12 months" # hybrid, with per-branch ranks npm run search -- "warranty 12 months" dense # vector only npm run search -- "warranty 12 months" keyword # full-text only Enter fullscreen mode Exit fullscreen mode Hybrid output prints rrf=... dense=1 keyword=2 <source> for each chunk, so you can see at a glance which branch a chunk came from.

curl -s -X POST http://localhost:8000/api/chat \ -H 'Content-Type: application/json' \ -d '{"question": "How long is the product warranty?"}' Enter fullscreen mode Exit fullscreen mode The response looks like this (illustrative values):

{ "answer": "All products are covered by a 12-month warranty from the date of purchase.", "citations": [ { "index": 1, "chunkId": "data/in/warranty.md#0", "source": "data/in/warranty.md", "title": "Warranty period" }, { "index": 2, "chunkId": "data/in/warranty.md#1", "source": "data/in/warranty.md", "title": "Returns" } ], "usage": { "inputTokens": 180, "outputTokens": 25, "cacheWriteTokens": 0, "cacheReadTokens": 650 } } Enter fullscreen mode Exit fullscreen mode Send the same question twice: the first request has cacheWriteTokens > 0, the second has cacheReadTokens > 0. If cacheReadTokens stays 0 across many requests, something is breaking the cache. It's usually a dynamic value that slipped into the system prompt, or a prefix below the minimum length.

The chunker and RRF are pure functions, so their tests need no database and no AWS:

it ( ' a chunk ranked well in both branches beats a chunk ranked first in only one ', () => { const results = reciprocalRankFusion ( { dense: [ candidate ( ' dense-only ' ), candidate ( ' both ' )], keyword: [ candidate ( ' keyword-only ' ), candidate ( ' both ' )], }, 60, 3, ); expect ( results [ 0 ]. chunkId ). toBe ( ' both ' ); }); it ( ' splits by heading and prepends the heading to chunk content ', () => { const chunks = makeChunker ( 1000 ). chunk ({ source: ' doc.md ', text: ' # Title \n\n Intro. \n\n ## Part A \n\n Content A. \n\n ## Part B \n\n Content B. ', }); expect ( chunks ). toHaveLength ( 3 ); expect ( chunks [ 1 ]. title ). toBe ( ' Part A ' ); expect ( chunks [ 1 ]. content ). toContain ( ' Part A ' ); }); Enter fullscreen mode Exit fullscreen mode 15.2 Multi-stage Dockerfile FROM node:20-alpine AS builder WORKDIR /app COPY package*.json./ RUN npm ci COPY tsconfig.json tsconfig.build.json nest-cli.json./ COPY src./src RUN npm run build RUN npm prune --omit = dev FROM node:20-alpine AS runtime WORKDIR /app ENV NODE_ENV=production COPY --from=builder --chown=node:node /app/node_modules./node_modules COPY --from=builder --chown=node:node /app/dist./dist COPY --chown=node:node package.json./ USER node EXPOSE 8000 HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \ CMD node -e "require('http').get('http://127.0.0.1:'+(process.env.PORT||8000)+'/api/health',r=>process.exit(r.statusCode===200?0:1)).on('error',()=>process.exit(1))" CMD ["node", "dist/main"] Enter fullscreen mode Exit fullscreen mode The npm ci layer is separate so the Docker cache survives source edits. The runtime runs as the node user, not root. The healthcheck uses Node's built-in http, so there's no need to install curl.

15.3 Production Compose: fail fast on missing variables docker-compose.prod.yml layers on top of the dev file, adds the app service, hides the Postgres port, and makes every sensitive value mandatory:

services: postgres: environment: POSTGRES_PASSWORD: ${DB_PASSWORD:?DB_PASSWORD is required} ports:!reset [] # Postgres not exposed outside the network app: build:. environment: DB_HOST: postgres DB_PASSWORD: ${DB_PASSWORD:?DB_PASSWORD is required} BEDROCK_GENERATION_MODEL_ID: ${BEDROCK_GENERATION_MODEL_ID} #... depends_on: postgres: condition: service_healthy Enter fullscreen mode Exit fullscreen mode docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d Enter fullscreen mode Exit fullscreen mode The ${VAR:?message} syntax makes the deploy fail immediately when a variable is missing, instead of silently running with the default postgres password.

name: CI on: push: branches: [ main ] pull_request: jobs: check: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: 20 cache: npm - run: npm ci - run: npm run lint - run: npm test - run: npm run build Enter fullscreen mode Exit fullscreen mode 16. Production Pitfalls That Quietly Destroy RAG Quality None of these throw an exception. The chatbot just answers worse, and the cause is very hard to track down.

By this point, the system is already capable of answering questions over a document collection.

The next challenge is not functionality.

Most production RAG systems evolve through a series of small improvements rather than a single architectural rewrite.

The following upgrades are listed roughly in order of value delivered relative to implementation effort.

The codebase above is Advanced RAG at its minimum: hybrid search + RRF + header injection + prompt caching. When you put it in front of real customers, these are the next steps, ordered by benefit per unit of effort:

For multi-tenant setups (several businesses on shared infrastructure), filtering by tenant_id from the schema up is a matter of right/wrong, not quality. Without it, data leaks between customers.

After building several generations of RAG systems, one lesson keeps repeating itself:

RAG is fundamentally a retrieval problem.

Most production issues are not caused by the language model.

The architecture in this article intentionally avoids unnecessary complexity.

Just PostgreSQL, pgvector, Amazon Bedrock, Hybrid Search, and a clear separation between indexing, retrieval, and generation.

For teams involved in application software development, that simplicity is often an advantage rather than a limitation. A system that can be understood, measured, and debugged is significantly easier to improve than one hidden behind multiple abstraction layers.

A system that can be understood, measured, and debugged is much easier to improve than one that depends on layers of abstraction.

If you're building a RAG chatbot today, invest in retrieval first.

Everything else becomes easier afterwards.