LLM API Token Count Impact: Reducing Cost in Ordinary Tenant Apps

A practical rule: cap and measure retrieved context per tenant before tuning prompts. Token count drives LLM API cost because providers bill around the amount of text processed, and private-knowledge applications can send the same large documents repeatedly. Keep the answer quality target fixed, then reduce tokens that do not help answer the question.

For a B2B SaaS product answering questions over private knowledge bases, the last two belong together. Retrieve broadly enough to find candidates, rerank them, and only then spend the generation budget. Record every stage against the tenant that caused it.

A request has more than the visible question. It can include system instructions, conversation history, retrieved passages, tool results, and the requested answer. Tokenization turns text into the units used by a model, so longer input or output generally means a larger usage quantity. The exact pricing model is provider-specific and can change; the engineering invariant is the measured input and output token count.

Consider two tenants asking, “What is our retention period?” Tenant A has three tightly matched chunks. Tenant B has thirty because the index returns every document that mentions “retention.” The question is identical. The second request carries a larger input and may make the model's job harder with irrelevant or contradictory passages.

That is the first operational trap: a global average hides tenant shape. A few tenants with verbose documents, long conversations, or repeated retries can dominate usage while the product-wide average looks healthy.

The useful unit is a request ledger, not a monthly total. Attach tenant_id, a privacy-safe request identifier, input/retrieved/output token counts, candidate and final context counts, a configuration identifier, outcome, latency, and retry count. Do not log private document text just to explain cost. Counts, hashes, document identifiers, and redacted metadata can locate the expensive stage without creating a second sensitive data store.

question -> candidate retrieval -> reranking -> context packing -> generation -> usage ledger

A common mistake is putting one large limit at the generation call. By then, retrieval has already done unnecessary work, and conversation history may have copied old text into the prompt. Give each stage a reason to stop.

Candidate retrieval can favor recall. Reranking can reduce a large candidate set to passages with a stronger relationship to the question. The Cohere Rerank documentation describes reranking as a second-stage operation over an initial result set; the same architecture can use any compatible reranker or an in-house scoring step. The boundary is architectural, not a particular product.

Context packing needs a deterministic budget. Sort evidence by score, preserve source boundaries, and stop before the input ceiling. Do not silently truncate a passage in the middle of a table or policy clause. If the best evidence does not fit, return narrower context or ask the application to reformulate the question.

Here is a concrete, illustrative trade-off: a lookup policy with a 1,800-token context and a 300-token output has a 2,100-token generation allowance before history and instructions. An explanation policy in the example uses 4,200 and 700 instead. The larger allowance can preserve evidence for a harder answer, but it also raises the maximum exposure for a single request.

Keep the result visible to the team that owns the retrieval policy. A cost chart without context counts can show movement without showing the cause; a trace without tenant aggregation can show the cause without showing who bears it.

These values are example configuration, not a universal recommendation. Making them explicit makes the decision testable.

type BudgetPolicy = { historyTokens: number; contextTokens: number; outputTokens: number; }; const policyByQuestionKind: Record < string, BudgetPolicy > = { lookup: { historyTokens: 600, contextTokens: 1800, outputTokens: 300 }, explanation: { historyTokens: 1200, contextTokens: 4200, outputTokens: 700 }, }; export function chooseBudget ( questionKind: string ): BudgetPolicy { return policyByQuestionKind [ questionKind ]?? policyByQuestionKind. lookup; } Enter fullscreen mode Exit fullscreen mode Select the policy before retrieval, then emit its identifier with the request record. A cost review can distinguish a larger answer class from a pipeline that ignored its limit.

Start with the largest removable input. In private-knowledge systems, that is often duplicated context rather than the question itself. Normalize repeated headers, avoid inserting the same source in both the system prompt and retrieved context, and trim history to turns that change the current answer. Keep citations or source identifiers compact, while preserving enough provenance for verification.

Then measure retrieval waste. If ten candidates become three final passages, inspect the seven discarded items. If they are usually irrelevant, improve filtering or reranking. If they contain the answer, a smaller cap is a quality regression disguised as savings. Pair tokens with answer acceptance, citation coverage, refusal rate, and a small evaluation set representing each tenant's document style.

A useful before-and-after test is deliberately boring: run fixed questions against the same corpus, then compare input tokens, output tokens, passage count, latency, and answer checks. Change one policy at a time. A prompt that saves input tokens but causes retries has not reduced the real cost.

Do not assume shorter output is always better. A one-sentence answer can trigger a follow-up request, while a complete answer with its source may finish the workflow. The target is fewer unhelpful tokens, not the smallest dashboard number.

A tenant-facing cost view needs two layers. The first is a request-level trace for debugging. The second is an aggregate ledger for allocation and trends. Keep both dimensions: tenant ownership and pipeline stage.

type UsageEvent = { tenantId: string; requestId: string; stage: " retrieval " | " generation "; inputTokens: number; outputTokens: number; candidateCount?: number; selectedCount?: number; retry: boolean; budgetId: string; }; export function recordUsage ( event: UsageEvent ): void { if ( event. inputTokens < 0 || event. outputTokens < 0 ) { throw new Error ( " Token counts must be non-negative " ); } } Enter fullscreen mode Exit fullscreen mode Use bounded metric dimensions. A tenant identifier may fit an internal ledger, but putting unbounded request IDs into a time-series metric can create cardinality problems. Store request IDs in traces or event records, and aggregate metrics by tenant, stage, configuration, and outcome where appropriate.

OpenTelemetry's metrics model is a useful standard reference for separating measurements from attributes and choosing aggregation deliberately.

Alert on behavior, not only dollars. A sudden increase in median context tokens, retry share, or retrieved candidates can signal a broken filter before the financial total catches up. Set thresholds from a baseline per workload. One global threshold treats a short lookup and a long policy explanation as the same operation.

Token budgets cannot solve bad retrieval. If the index misses the relevant passage, adding text can increase cost while reducing clarity. If documents conflict, reranking cannot decide which policy is authoritative without metadata and a freshness rule.

Budgets do not replace access control. Filter documents for the tenant before generation, and carry authorization decisions through retrieval and context packing. Cost attribution must never become a reason to mix tenant data.

The decision rule is straightforward: set a context budget by answer class, use a second-stage selection step when candidates are noisy, and attribute measured input and output tokens to the requesting tenant. Review quality and retries beside the token chart. That produces cost control for ordinary application traffic, not just a clean demo.