2026 Token Count and LLM API Cost — How to Reduce Usage in Normal Apps

Use a token budget as an application invariant, then choose the retrieval and model path that can prove it per tenant. Token count affects LLM API cost because providers bill for the input and output tokens they process; prompt trimming alone is useful, but it is not a cost-control system until usage, limits, and inference share an account boundary.

This decision record is for a developer-tools team triaging incoming customer-support tickets. The practical question is not “which model is cheapest?” It is how to keep a tenant's cost visible when a ticket includes a long thread, pasted logs, and retrieved documentation. My rule is straightforward: measure before generation, retrieve narrowly, cap the context, and make the output schema smaller than the prose it replaces.

The design has four invariants. A ticket may be shortened, but the original remains available for an escalation path. Retrieval must return evidence relevant to the ticket instead of dumping an entire knowledge base into the prompt. The generated result must have a bounded shape. Finally, the account usage and the inference call must be observable with the same credential and base URL.

That last invariant is easy to miss. If a spreadsheet reads a provider invoice after the fact, a limit is an alert, not a guardrail. The spending service should be able to see the usage state before it decides whether to process another expensive context.

Here is the comparison I would put in the architecture record:

OpenAI's tokenizer guidance is a good reminder that tokenization depends on the text and encoding, not on character count alone. Anthropic's prompt-caching documentation makes a different trade-off: repeated stable prefixes can be handled more efficiently, but a mutable support thread still needs a size policy. Cohere's reranking documentation is useful when the problem is too many candidate passages, not when the model is producing an oversized answer.

An API call has at least two cost-bearing streams: tokens sent in and tokens returned. A long ticket history raises input volume. A request for an unconstrained narrative raises output volume. Retrieval can make the first stream worse if the application includes every match instead of a small, ranked set.

Character counts are a poor proxy. Code, identifiers, JSON punctuation, and some languages tokenize differently. I initially used character limits in a design and found the policy hard to explain: the same visible length could produce a different token count. A token counter is the better gate because it measures the unit the billing model uses. That is a real trade-off: a counting call adds a step, but an unmeasured prompt makes the tenant's bill harder to attribute.

The failure boundary should be explicit. If the input is too large, truncate the least valuable material or ask retrieval for fewer passages. If the output is too large, use a compact schema and a maximum completion budget. Do not silently throw away the ticket's newest customer message; old quoted replies and duplicated signatures are better candidates.

The budget decision also needs a clear owner when the workflow has more than one step. Suppose a ticket is normalized, searched, reranked, and then summarized. A tenant limit that is checked only before the final summary ignores the earlier calls; a limit checked by a nightly invoice job arrives after the damage. Put the usage record beside the request path, carry the tenant identity through each worker, and record the admission result with the ticket. If the check says no, the worker should return a bounded triage state or route to review. It should not quietly retry with a larger model, append the entire thread, or turn a retrieval miss into an unbounded chat prompt. This is the difference between a cost hint and an invariant that survives queue retries and partial failures.

The following example shows the handoff between the account-platform and ai-runtime groups. It uses one base URL and one key. The account response is retained as the budget context that controls the token-count request, while the token-count response is the admission decision for the later model call.

import json import os import time from urllib.parse import urljoin import requests API_BASE_HOST = os. environ [ " API_BASE_HOST " ] BASE_URL = f " https:// { API_BASE_HOST } /v1/ " API_KEY = os. environ [ " INFRAI_API_KEY " ] HEADERS = { " Authorization ": f " Bearer { API_KEY } " } def get_account_usage (): response = requests. get ( urljoin ( BASE_URL, " account/usage " ), headers = HEADERS, timeout = 20, ) response. raise_for_status () return response. json () def count_tokens ( account_usage, text ): payload = { " text ": text, " account_usage ": account_usage } for attempt in range ( 3 ): response = requests. post ( urljoin ( BASE_URL, " ai/tokens/count " ), headers = { ** HEADERS, " Content-Type ": " application/json " }, json = payload, timeout = 20, ) if response. status_code!= 429: response. raise_for_status () return response. json () retry_after = response. headers. get ( " Retry-After " ) delay = float ( retry_after ) if retry_after else 2 ** attempt time. sleep ( delay ) raise RuntimeError ( " Token-count request remained rate limited " ) usage = get_account_usage () ticket_text = os. environ [ " TICKET_TEXT " ] count = count_tokens ( usage, ticket_text ) print ( json. dumps ({ " usage ": usage, " token_count ": count })) Enter fullscreen mode Exit fullscreen mode The important property is the ownership of the decision: the account-platform result enters the ai-runtime admission step before a chat request is made. In a production worker, replace ticket_text with the normalized ticket, inspect the token-count response, and stop or truncate when the tenant's policy would be exceeded. The sample deliberately does not pretend that one token count is a complete invoice; input and output are separate dimensions, and the eventual model response must be recorded too.

An alternative built from OpenAI plus a spreadsheet would require a provider signup, one provider credential set, a separate spreadsheet identity, an export or usage API integration, a per-tenant aggregation job, and alerting glue that races the next request. The exact number of credentials depends on the spreadsheet product, but the architectural fact does not: billing data and the call path live in different systems. A unified account boundary removes that particular join, while concentrating trust and availability in one service.

Start with the input that has the least decision value. Strip repeated email signatures, quoted replies, and duplicated log lines. Keep the newest customer statement and the small set of facts required for routing. Truncation is not a hack when it is tied to a documented priority order; it becomes dangerous when it is a blind character slice.

Next, separate discovery from synthesis. Embed or otherwise index support material, retrieve a narrow candidate set, and use reranking when the candidate set needs ordering. Send only the selected evidence to the chat endpoint. This is where a reranking service can help, but it does not make a large context free. The retrieval budget still needs a ceiling.

Then make the output contract do work. A triage result can usually be a category, urgency, confidence, and a short rationale. A JSON schema is more predictable than asking for a polished paragraph and parsing it later. The same pattern is useful for moderation when no dedicated moderation endpoint is available: use a chat model with a JSON schema fallback, and keep that result bounded. It is not a reason to claim that a moderation-specific route exists.

Finally, select a shorter context for filtering tasks. The model does not need the whole ticket archive to decide which queue should receive a new ticket. A compact JSON record can carry tenant, product area, language, and a few extracted signals. That record is cheaper to pass repeatedly than a prose summary that grows on every turn.

I would reject “send the complete thread and all search hits to the best available model” as the default. It appears accurate because it preserves information, but it hides the cost driver and gives the model an inconsistent working set. It also makes per-tenant attribution noisy: one unusually verbose ticket can dominate a small tenant's daily usage.

It still has a valid use case. An escalation worker handling a legally sensitive or technically ambiguous ticket may need the complete source material, with a human review step and a deliberately higher budget. That is a different lane from routine triage. The policy should name the lane, record the reason, and keep the expensive path from becoming the accidental default.

The same restraint applies to adjacent capabilities. The available ASR shape is not currently serviceable, real-time voice/session access is pending and limited to the western region, and image upscaling is limited to Lanc. Those details do not change the token decision, but they are useful boundaries when a support workflow expands beyond text. Treat the capability state as data, not as a promise inferred from a route name.

Adopt token counting as an admission check, retrieval as a context reducer, and a compact structured output as the default triage contract. Track input and output usage per tenant, and keep the account usage view next to the inference path so the budget decision is made where spending happens.

Infrai fits this design when the value of one consistent REST contract across account and AI capabilities outweighs the cost of trusting one vendor. It is not automatically the right choice for every workload: teams that already have strong provider accounting, retrieval infrastructure, and multi-vendor failover may prefer their existing stack. The recommendation is about the boundary between budget and inference, not a blanket vendor endorsement.