Routing-Stufen bei LLM-APIs: Auto ist eine Preisentscheidung, keine Qualitätsentscheidung
Ein Betreiber eines OpenAI-kompatiblen Chat-Endpunkts mit über 50 Modellen stellte fest, dass Agenten für triviale Klassifikationsaufrufe Tokens im Frontmodell-Tarif verbrannten. Das automatische Routing folgt vor allem Preiserwägungen.
Posted on Oct 15 • Originally published at imapp.blogspot.com
I run an OpenAI-compatible chat endpoint that fronts 50+ models. The original design was simple: one model name, one price, pass-through. Then my own agents started burning through tokens on trivial classification calls, and I noticed something worse — I was paying frontier-model rates for "is this a PDF?" questions.
So I added three routing tiers: eco for cheap fast models, premium for frontier, and auto which picks per-request. The surprise wasn't the tiers. It was what auto actually had to know.
A router can't just look at prompt length. A 40-token request can still need a frontier model ("explain this theorem"). A 4,000-token request can be pure extraction ("give me the dates in this contract"). I ended up building a tiny classifier: task type heuristics (summarize/extract/code/reason/chat), length bands, and a per-caller override. The auto tier is now a small decision engine, not a load balancer.
The failure mode I didn't expect: latency. eco is fast but sometimes wrong on edge cases, and my agents were retrying — which meant two eco calls plus one premium fallback cost more than just calling premium first. I added a confidence threshold: if the eco model's output looks uncertain (short answers to open questions, hedging language), escalate immediately instead of letting the agent retry blindly.
Compared to calling providers directly: I save the integration tax (one API key, one base URL, one billing relationship) but I add a routing tax (the classifier runs on every request, ~2ms). For my own agents, that's worth it. For a human developer who already knows which model they want, the tiers are just noise — they can still pass an explicit model name and bypass auto entirely.
The lesson: "auto" sounds like a convenience feature. In practice it's a cost-control feature wearing a convenience costume. If you're building agent infrastructure, design for the agent that doesn't know what it needs — because that's most of them, most of the time.
It's packaged as an OpenAI-compatible /v1/chat/completions endpoint at https://x402.freeq.one/tools/llm_chat.html — model list and rates at /tools/llm_chat/models.