LLM Gateway & FinOps Architecture for B2B SaaS: Semantic Caching, Routing & Tenant Quotas

The finance lead forwards you the model provider invoice with one line highlighted. Spend tripled in a month. Revenue did not. When you dig in, a single tenant on the starter plan is running a nightly script that sends the same 40-page contract through your "summarize" endpoint 600 times, every request goes to the most expensive model because that was the default in the SDK call, and nothing in your stack knew which tenant any of those tokens belonged to.

I wrote this with help from AI tools and checked every config, code path and claim myself. I run Loop Origin, where we build custom web applications for B2B SaaS teams, and this is the final part of the series: the gateway layer that turns LLM calls from an unbounded cost into a metered, budgeted product feature.

📚 Series Roadmap — Production AI & SaaS Systems Engineering

Direct answer: An LLM gateway is one internal service that every model call passes through. It identifies the tenant and feature, checks a token budget before the call, serves exact or semantic cache hits, routes each task to the cheapest model that meets its quality bar, and records actual token usage afterwards so costs land on the right tenant's ledger.

In most SaaS codebases, the first AI feature calls the provider SDK directly from a controller. The second feature copies that code. By the fifth, you have five places that pick a model, five retry policies, no shared cache, and API keys in three services. Nobody can answer the two questions finance will ask: which tenant spent this and what would it cost on a cheaper model.

A gateway centralizes four decisions that should never be made inside product code:

Laravel API ──┐ Node workers ─┼─► LLM Gateway ──► quota check (Redis) ──► cache (exact → semantic) MCP agents ──┘ │ │ miss │ ▼ │ router: task class → model tier │ │ └──── usage events (Part 4 meter) ◄── provider A / B / self-hosted Enter fullscreen mode Exit fullscreen mode The gateway can be an open-source proxy you configure or a small service you write. Either way, product code sends a task, not a model name: { tenant_id, feature: "contract_summary", input }.

Reserve an estimated token cost before the call, then settle the real number afterwards. A Lua script keeps the check-and-reserve atomic in Redis or Valkey:

-- KEYS = budget:{tenant}:{yyyy-mm} ARGV = estimate ARGV = monthly limit local used = tonumber ( redis. call ( 'GET', KEYS [ 1 ]) or '0' ) if used + tonumber ( ARGV [ 1 ]) > tonumber ( ARGV [ 2 ]) then return - 1 end redis. call ( 'INCRBY', KEYS [ 1 ], ARGV [ 1 ]) redis. call ( 'EXPIRE', KEYS [ 1 ], 40 * 24 * 3600 ) return used + tonumber ( ARGV [ 1 ]) Enter fullscreen mode Exit fullscreen mode // gateway/budget.ts export async function reserve ( tenant: Tenant, estimate: number ) { const key = `budget: ${ tenant. id }: ${ monthKey ()} `; const res = await redis. eval ( RESERVE_LUA, 1, key, estimate, tenant. plan. monthlyTokens ); if ( res === - 1 ) throw new QuotaExceeded ( tenant. id, tenant. plan. name ); } export async function settle ( tenant: Tenant, estimate: number, actual: number ) { await redis. incrby ( `budget: ${ tenant. id }: ${ monthKey ()} `, actual - estimate ); await meter. emit ({ tenant: tenant. id, metric: " llm_tokens ", value: actual }); // Part 4 pipeline } Enter fullscreen mode Exit fullscreen mode Estimate input tokens with the provider's tokenizer and add the request's max_tokens cap for output. Overestimating slightly is fine because settlement corrects it; what matters is that the 600-request nightly loop hits a wall instead of the invoice.

Exact cache first: hash the normalized request (model tier, system prompt version, input, temperature) and look it up. It is cheap and has zero quality risk. In my experience it catches more than teams expect, because batch jobs and retries resend identical inputs.

Semantic cache second: embed the input, search a vector index scoped to the same tenant and feature, and return the stored answer only above a strict similarity threshold.

const hit = await vectors. search ({ namespace: ` ${ tenant. id }: ${ feature }: ${ promptVersion } `, vector: await embed ( input ), topK: 1, }); if ( hit && hit. score >= FEATURE_THRESHOLDS [ feature ]) return hit. payload; // e.g. 0.95 Enter fullscreen mode Exit fullscreen mode Three rules keep semantic caching from becoming a data-leak or wrong-answer machine:

Most SaaS AI features are not equally hard. Classify them once and map each class to a tier:

# gateway/routes.yaml contract_summary: primary: { provider: provider_a, model: mid-tier, max_tokens: 800 } fallback: { provider: provider_b, model: small, max_tokens: 500 } timeout_ms: 20000 plans: starter: { tier_override: small } ticket_classify: primary: { provider: self_hosted, model: small } fallback: { provider: provider_a, model: small } Enter fullscreen mode Exit fullscreen mode The plans override is where FinOps meets pricing: starter tenants get the cheaper tier, enterprise tenants get the better one, and the difference is visible on your pricing page instead of hidden in your margin.

Validate a cheaper route before switching: replay a sample of real (anonymized) inputs through both tiers and score the outputs with your own rubric. Route changes without an eval are guesses.

Retries are where LLM costs quietly multiply. As Part 3 covered for caches, a retry storm turns one outage into many. In the gateway:

Reuse the OpenTelemetry setup from Part 5. Each gateway call becomes a span with low-cardinality attributes:

span. setAttributes ({ " llm.feature ": feature, " llm.route ": route. name, " llm.model_tier ": route. tier, " llm.cache ": cacheResult, // "exact" | "semantic" | "miss" " llm.tokens.input ": usage. input, " llm.tokens.output ": usage. output, " tenant.plan ": tenant. plan. name, // plan, not tenant ID, to keep metrics cheap }); Enter fullscreen mode Exit fullscreen mode Generate metrics from these spans for cost per feature, cache hit rate and fallback rate. Keep the tenant ID in the usage events and the billing ledger, not in metric labels, so dashboards stay affordable.

Start with an existing open-source LLM proxy if it supports your providers, budgets and logging. Write your own thin service when you need tenant-aware quotas tied to your billing data, which most B2B SaaS products eventually do.

A budget check in Redis and an exact cache lookup are usually a few milliseconds. Semantic cache adds an embedding call, so only enable it for features where hit rates justify it.

There is no universal number. Start strict, around 0.95 cosine similarity, log near-misses, and review them by hand before loosening it per feature.

Count input tokens with the provider's tokenizer library and add the max_tokens cap for output. Settle with the real usage numbers the provider returns.

Yes, if your metering is accurate. Emit settled token usage into the same pipeline as your other usage events and price it per plan, with clear limits shown in the product.

Agents make many calls per task, so they need the gateway most. Give each agent run its own budget reservation and stop the run, with a human approval gate, when it exceeds it.

That closes Production AI & SaaS Systems Engineering. Across six parts we sandboxed MCP agents, migrated databases without downtime, protected caches from stampedes, metered usage for billing, made traces affordable and, here, put a budget and a router in front of every model call. The common thread: put one deliberate control point in front of every expensive or risky operation, measure what passes through it, and let product code ask for outcomes instead of hard-coding infrastructure choices. Questions about your own setup are welcome in the comments.