Your chatbot resends the whole conversation on every message. Here's what that costs, measured.

Posted on Oct 15 • Originally published at qubax.ai

Chat APIs are stateless. To answer message 41, your app sends messages 1 to 40 again, and you pay for all of them as input tokens, every single time.

On our own API traffic over the last 7 days (about 411,000 requests), 99.3% of all tokens processed were input tokens. The median request sent 3,782 tokens in and got 147 back. Most LLM bills aren't about long answers. They're about long histories.

Here's how much that costs on a realistic chat, and two simple fixes, measured on real API calls.

A support bot after 40 turns, then one more question. Three strategies:

""" trim_context.py: what does chat history really cost? """ import os from openai import OpenAI client = OpenAI ( base_url = " https://api.qubax.ai/v1 ", api_key = os. environ [ " QUBAX_API_KEY " ]) MODEL, PRICE_IN, PRICE_OUT = " claude-sonnet-5 ", 0.3835, 1.5339 # $/1M history = [{ " role ": " system ", " content ": " You are a concise support agent for an invoicing SaaS. " }] for i in range ( 40 ): history += [ { " role ": " user ", " content ": f " Question { i }: how do I change the due " f " date on invoice # { 1000 + i }?

I also need to know whether the " " customer gets notified and if late fees recalculate. " }, { " role ": " assistant ", " content ": f " Open invoice # { 1000 + i }, click " " Edit, change the due date and save. The customer gets an update " " email, and late fees recalculate from the new date.

" }, ] question = { " role ": " user ", " content ": " Last one: can I bulk-change due " " dates for all invoices of one customer? " } def ask ( messages ): r = client. chat. completions. create ( model = MODEL, messages = messages, max_tokens = 120, temperature = 0 ) return r. choices [ 0 ]. message. content, r. usage def keep_last ( msgs, turns ): return msgs [: 1 ] + msgs [ 1:][ - 2 * turns:] def summarize_old ( msgs, keep_turns ): old, recent = msgs [ 1:][: - 2 * keep_turns ], msgs [ 1:][ - 2 * keep_turns:] note = client. chat. completions. create ( model = " glm-5.3-flash ", temperature = 0, max_tokens = 800, messages = [{ " role ": " user ", " content ": " Summarize this support chat in at most 60 words. " " Keep names, IDs and terms. \n\n " + " \n ". join ( f " { m [ ' role ' ] }: { m [ ' content ' ] } " for m in old )}], ). choices [ 0 ]. message. content return msgs [: 1 ] + [{ " role ": " system ", " content ": f " Summary of earlier conversation: { note } " }] + recent for label, msgs in [( " full history ", history + [ question ]), ( " last 6 turns ", keep_last ( history, 6 ) + [ question ]), ( " summary + last 4 ", summarize_old ( history, 4 ) + [ question ])]: answer, u = ask ( msgs ) usd = ( u. prompt_tokens * PRICE_IN + u. completion_tokens * PRICE_OUT ) / 1e6 print ( f " { label: < 18 } { u. prompt_tokens: > 5 } in $ { usd:.

6 f } { answer [: 70 ] } " ) Enter fullscreen mode Exit fullscreen mode Results (Claude Sonnet 5, Oct 11, 2026) Strategy Input tokens Cost per message vs full Full history (40 turns) 3,690 $0.001599; Last 6 turns 596 $0.000413 −74% Summary + last 4 turns 531 $0.000388 −76% Same model and same output length, with roughly a quarter of the bill. (Exact token counts vary a little between runs, because the summary text changes.)

The summary step costs extra: one call to GLM 5.3 Flash at about $0.00002. You only pay it when the history crosses your limit, not on every message.

Cost is only half the test, so I added a memory check. At the very start of the chat, the user says: "My customer is Northwind, account ID 7731, net-45 terms." After 40 more turns, the bot is asked to repeat the ID and terms. Three runs each:

The plain sliding window is cheap, but it forgets anything older than the window, including the facts your user said only once. The summary keeps them at almost the same price. To make that reliable, tell the summarizer what to keep: "Keep names, IDs and terms."

MAX_TURNS = 8 # keep this many recent turns verbatim SUMMARY_AT = 12 # compress once history passes this def build_messages ( system, turns, summary, new_msg ): if len ( turns ) > SUMMARY_AT: summary = summarize ( summary, turns [: - MAX_TURNS ]) # cheap model turns = turns [ - MAX_TURNS:] msgs = [{ " role ": " system ", " content ": system }] if summary: msgs. append ({ " role ": " system ", " content ": f " Earlier: { summary } " }) return msgs + turns + [ new_msg ], turns, summary Enter fullscreen mode Exit fullscreen mode Store summary next to the conversation, so you only compress each part of the history once.

I measured this through Qubax, an OpenAI-compatible API for 400+ models, often well below OpenRouter prices; the traffic numbers above come from its usage logs. The trimming code works with any OpenAI-compatible endpoint. Live model prices are on the price index.

For short-lived chats, yes. But in our test it forgot an account ID given at the start 3 out of 3 times. If users state facts once and expect them to be remembered, summarize the older turns.

A cheap, fast one. The summary call in this test cost about $0.00002, and you only pay it when the history passes your limit.

Partly. Caching makes the repeated prefix cheaper, but long histories still add latency and can push you past context limits. Trimming and caching work well together.

How do you handle long chats: windows, summaries, or embeddings-based memory? Curious what holds up at scale.

Originally published at qubax.ai. Qubax gives you every top AI model with one API key, cheaper than OpenRouter.