Semantic Caching for LLM Calls in Node.js With pgvector

A semantic cache turns each incoming prompt into an embedding and searches pgvector for an earlier prompt that means the same thing. If the similarity is above a threshold you set, it returns the stored answer and skips the model call. Done well, it makes repeat questions faster and cheaper to answer. Done carelessly, it shows one tenant's answer to another tenant, or answers a question about invoice 1042 with the reply meant for invoice 1043.

This post builds the cache in Node.js on Postgres with pgvector. It covers the table, cache keys scoped to each tenant, TTLs, and how to tune the threshold so that a cache hit really is the right answer.

Exact-match caching (hash the prompt, look up the hash) only helps when users type the same words. They rarely do. "How do I reset my password?" and "forgot my password, what now" should return the same answer.

Skip it when each response depends on live, per-user data such as order status, or when you want the output to vary, as in brainstorming tools.

CREATE EXTENSION IF NOT EXISTS vector; CREATE TABLE llm_cache ( id bigserial PRIMARY KEY, tenant_id text NOT NULL, scope text NOT NULL, model text NOT NULL, prompt_text text NOT NULL, embedding vector ( 1536 ) NOT NULL, answer text NOT NULL, hit_count integer NOT NULL DEFAULT 0, created_at timestamptz NOT NULL DEFAULT now (), expires_at timestamptz NOT NULL ); CREATE INDEX llm_cache_embedding_idx ON llm_cache USING hnsw ( embedding vector_cosine_ops ); CREATE INDEX llm_cache_lookup_idx ON llm_cache ( tenant_id, scope, model, expires_at ); Enter fullscreen mode Exit fullscreen mode Set the vector dimension to match your embedding model. Vectors from different embedding models can't be compared, so if you change models you need a fresh table.

Multi-tenant data has one pgvector catch. An HNSW index finds the nearest neighbours first and applies your WHERE filter afterwards. With many tenants, all the top candidates can belong to other tenants, and the query returns nothing. On pgvector 0.8 or later, turn on iterative scans so the index keeps searching:

SET hnsw. iterative_scan = relaxed_order; Enter fullscreen mode Exit fullscreen mode What goes into the cache key The embedding finds similar prompts. The cache key controls which prompts are allowed to match in the first place. Anything that changes the correct answer belongs in the key:

I combine the versions into one scope string to keep the query simple.

This uses knex raw queries on Postgres. Plain pg works the same way with numbered placeholders. embed and callModel stand in for whatever your provider SDK offers.

import knexFactory from ' knex '; const db = knexFactory ({ client: ' pg ', connection: process. env. DATABASE_URL }); // TTL in seconds, per kind of question const TTL = { faq: 86400, docs: 21600, default: 1800 }; function normalize ( text ) { return text. normalize ( ' NFKC ' ). toLowerCase (). replace ( / \s +/g, ' ' ). trim (); } // Embeddings blur numbers, so IDs and dates must match exactly function hardTokens ( text ) { return ( text. match ( / [ 0-9 ] + (?:[.,:- ][ 0-9 ] + ) */g ) || []). sort (). join ( ' | ' ); } function shouldStore ( answer ) { if (! answer || answer. length < 20 ) return false; return! /i ( can't|cannot|am unable to ) /i. test ( answer ); } export async function cachedCompletion ( opts ) { const { tenantId, kind, model, promptVersion, kbVersion = ' none ', locale = ' en ', prompt, threshold = 0.95, embed, callModel, } = opts; const text = normalize ( prompt ); const scope = [ kind, promptVersion, kbVersion, locale ]. join ( ': ' ); const vector = JSON. stringify ( await embed ( text )); const { rows } = await db. raw ( `SELECT id, prompt_text, answer, 1 - (embedding <=>?::vector) AS similarity FROM llm_cache WHERE tenant_id =? AND scope =?

AND model =? AND expires_at > now() ORDER BY embedding <=>?::vector LIMIT 1`, [ vector, tenantId, scope, model, vector ] ); const best = rows [ 0 ]; const isHit = best && best. similarity >= threshold && hardTokens ( best. prompt_text ) === hardTokens ( text ); if ( isHit ) { db. raw ( ' UPDATE llm_cache SET hit_count = hit_count + 1 WHERE id =? ', [ best. id ]). catch (() => {}); return { answer: best. answer, cached: true, similarity: best. similarity }; } const answer = await callModel ( prompt ); if ( shouldStore ( answer )) { await db. raw ( `INSERT INTO llm_cache (tenant_id, scope, model, prompt_text, embedding, answer, expires_at) VALUES (?,?,?,?,?::vector,?, now() + make_interval(secs =>?))`, [ tenantId, scope, model, text, vector, answer, TTL [ kind ]?? TTL. default ] ); } return { answer, cached: false, similarity: best?. similarity?? null }; } Enter fullscreen mode Exit fullscreen mode Why the code makes these choices:

The lookup query already ignores expired rows, but those rows still use storage and bloat the index. Delete them on a schedule:

DELETE FROM llm_cache WHERE expires_at < now (); Enter fullscreen mode Exit fullscreen mode To invalidate, you don't need to find and delete rows. Bump kbVersion or promptVersion when content or prompts change. Old entries stop matching straight away and expire on their own.

Most semantic caches go wrong here. Treat any threshold you copy from a blog post, including the 0.95 above, as a starting point. Cosine scores shift with the embedding model, the language and the length of your prompts.

Tune the threshold with real data in shadow mode:

Decision rules that hold up in practice:

One trade-off to keep in mind: the embedding call adds latency to every request, including misses. If your hit rate stays low after tuning, the cache can cost more than it saves, and dropping it is a reasonable call.

Caching sits next to retries, streaming, error handling and cost controls in any Node.js backend that calls LLMs. Geminate Solutions covers those pieces in a wider guide.

If you're adding AI features to a Node.js service, the Node.js AI integration guide is a good next read.