Five Ways to See LLM Chatbot Token Costs: Tenant-Aware Sales CRM Operations

Short answer: compare LLM chatbot production cost per token by tenant and by accepted CRM action, then choose the candidate that passes quality and latency constraints with the clearest cost trail.

A chatbot can have the lowest advertised token rate and still be the wrong production choice for an e-commerce team. The operational constraint is per-tenant visibility: the system must explain which tenant created each token, retry, and CRM action before anyone can call a cost comparison useful.

The practical choice is to compare complete work units, not isolated token prices. For a sales-call summarizer, one work unit is the transcript received, the model output, any retry, the resulting CRM action, and the review outcome. Measure that unit with the same replay set across candidates. Then set a tenant budget from observed usage instead of guessing from a price table.

Think of the before and after like this: before, Finance sees one monthly model invoice; after, an operator can trace one tenant's invoice share to a call, a prompt version, a token count, and a successful or rejected CRM update. The second view is what makes a “cheapest” comparison operationally honest.

A sales call does not end when text comes back. The application may add instructions, transcript content, conversation history, a schema, and a follow-up request. It may also retry a timeout. If the CRM write fails after a valid summary, a later worker can send the same work again. A token-only comparison that records just the final response hides those paths.

Start with a small event contract. Keep provider details behind a generic interface, and attach the tenant identity at the boundary where the application already knows it. The fields below are deliberately boring. That is useful: boring fields are easy to query six months later.

type LlmUsageEvent = { tenantId: string; operation: " sales_call_summary "; requestId: string; promptVersion: string; inputTokens: number; outputTokens: number; retryCount: number; cacheHit: boolean; crmAction: " created " | " updated " | " rejected "; reviewRequired: boolean; occurredAt: string; }; function costUnit ( event: LlmUsageEvent, inputRate: number, outputRate: number ): number { return ( event. inputTokens * inputRate ) + ( event. outputTokens * outputRate ); } Enter fullscreen mode Exit fullscreen mode The rates are configuration, not constants hidden in application code. Keep the input and output components separate. A provider can look favorable on one side of the split and unfavorable on the other; collapsing them makes the explanation harder.

Also record the number of attempts. A successful second attempt is still production work.

The trade-off is explicit: more fields make the ledger more useful, but they also create more data to retain and protect. Keep the event small enough to govern.

Do not put raw transcripts in the cost event. A request identifier is enough to join protected tracing data under the team's retention policy. This keeps the cost ledger useful without turning it into a second transcript store.

The useful comparison axis is cost per accepted CRM action. Cost per generated summary is an intermediate signal. A summary that needs manual correction, or never becomes a CRM activity, has not completed the business job.

type TenantLedger = { tenantId: string; receivedCalls: number; acceptedActions: number; inputTokens: number; outputTokens: number; retries: number; estimatedCost: number; }; function costPerAcceptedAction ( ledger: TenantLedger ): number | null { if ( ledger. acceptedActions === 0 ) return null; return ledger. estimatedCost / ledger. acceptedActions; } Enter fullscreen mode Exit fullscreen mode That denominator matters. A tenant with a small transcript and a high rejection rate can be more expensive operationally than a tenant with longer calls and clean actions. Keep both figures in the report: cost per summary and cost per accepted action. They answer different questions.

For a fair run, replay the same sanitized call set, the same CRM action schema, and the same acceptance rules. Pin the prompt version. Record the distribution, not only the average. One long call can distort a mean, while a p95 input-token count can reveal why a tenant's bill keeps surprising the support team.

The comparison should also include the embedding step when the application searches prior calls or account notes. The public embeddings guide describes embeddings as numerical representations useful for search and related tasks, which means those tokens belong to the work unit when retrieval is part of the request. Keep retrieval input and generation input distinguishable so the team can see which stage is growing.

A metric without a tenant dimension cannot answer a tenant question. A trace without a request ID cannot connect a timeout to its retry. A log without a prompt version cannot explain why two otherwise similar calls used different amounts of context.

Use the same low-cardinality fields across logs, metrics, and traces: tenant ID, operation, prompt version, model configuration label, outcome, and environment. Do not turn transcript text, request bodies, or unbounded customer labels into metric dimensions. Those fields belong in restricted trace or log storage, subject to the application's privacy controls.

The flow is easy to picture: the ingestion event carries tenant ID; the summarization span adds token counts; the CRM span adds the action result; the cost event joins them by request ID. When a number looks wrong, follow the line from invoice share to request, rather than opening a dashboard and guessing.

Alert on behavior that needs investigation, not on a provider's name. Examples include a sudden rise in retry count for one tenant, a growing gap between generated summaries and accepted CRM actions, or an input-token distribution that shifts after a prompt change. These alerts protect the decision model even when the underlying model configuration changes.

Use a sealed evaluation record. For each candidate, store the configuration label, the exact input and output counts, response status, retry count, latency, and acceptance result. Apply the same accounting formula to each record. The result is a table your team can audit, instead of a screenshot of a public price page that may omit application behavior.

The decision rule can be stated plainly: reject a candidate if it cannot meet the tenant's quality or latency requirement; among the candidates that pass, compare cost per accepted CRM action and the spread across tenants. This is a constraint, not a ranking. The cheapest aggregate number is irrelevant if one tenant absorbs the retries or review queue.

Use this compact comparison table before collecting quotes:

The table is a decision frame, not a ranking. A hosted boundary can reduce operational work while weakening control over runtime capacity; self-hosting can improve control while adding that work. A router can spread load while making per-tenant attribution more demanding. The right answer depends on which constraint the team can actually support.

No single rate answers the production question.

Keep provider adapters narrow. A generic call boundary is enough for the comparison:

type SummaryRequest = { tenantId: string; transcript: string; promptVersion: string; }; type SummaryResult = { summary: string; inputTokens: number; outputTokens: number; }; type Summarizer = { summarize ( request: SummaryRequest ): Promise < SummaryResult >; }; async function runSummary ( summarizer: Summarizer, request: SummaryRequest, ): Promise < SummaryResult > { return summarizer. summarize ( request ); } Enter fullscreen mode Exit fullscreen mode This boundary keeps the experiment focused on observable behavior. It also prevents a comparison article from becoming a setup guide for one vendor. Do not claim that an adapter is interchangeable until the event contract, error mapping, structured output handling, and token accounting have all been tested.

Start with four views. The first is tenant cost over time, split into input and output. The second is cost per accepted CRM action. The third shows retries, rejected actions, and review-required results together. The fourth shows token distributions by prompt version.

Give the operator a drill-down path from each chart to a request ID. Make the empty and partial states explicit: a missing cost rate is a data-quality issue, not a zero-cost request; an action with no acceptance result is pending, not successful. These distinctions prevent a quiet dashboard from becoming a false budget signal.

Set a review threshold before launch. When a tenant crosses it, the system should preserve the relevant event trail and route the case to an owner. Avoid silently shortening transcripts or dropping CRM actions just to keep a number under budget. That creates a cheaper-looking report and a less trustworthy product.

This method has a limit: it can expose cost and outcome differences, but it cannot decide whether a sales team accepts a particular summary style. That requires a separately defined acceptance rubric and human review sample.

The two common objections are predictable. “Can we just divide the monthly invoice by the number of calls?” Only if every call has the same input, output, retry, retrieval, and review behavior; production systems rarely do. “Can we choose from the lowest token rate?” Only after the candidate passes the acceptance and latency constraints and the ledger shows its tenant distribution. The rate is an input to the decision, not the decision itself.

For an in-app e-commerce chatbot that summarizes sales calls into CRM actions, observability is the comparison method. Record the complete work unit, preserve the tenant boundary, replay identical inputs, and compare accepted outcomes. That gives the team a defensible answer to the cost question without recommending a product or pretending that one public number describes production.