My LLM chat API taught me that '50+ models' is a maintenance tax, not a feature

Posted on Oct 18 • Originally published at imapp.blogspot.com

I launched my OpenAI-compatible chat endpoint with 50+ models behind one integration. The marketing math was easy: one client, any provider, swap without rewriting code. The engineering math was harder.

The first cost was the model registry. Every provider ships a slightly different /models payload — some include context windows, some don't; some use max_tokens, some use max_completion_tokens; a few lie about which models are actually available in which regions. I ended up maintaining a normalization layer that's basically a translation table between five different vocabularies for "this model exists."

The second cost was routing. I added auto/eco/premium tiers so agents could pick a price-quality tradeoff without knowing provider names. That meant keeping per-model latency and cost data fresh, or the router makes confident wrong decisions. A stale price table sends every "eco" request to a model that got more expensive last Tuesday.

The third cost — the one that surprised me — was deprecation. Providers retire models with 30 days notice sometimes, less other times. My registry had no concept of "this model is going away." Agents hardcoded model names into their workflows, then woke up to 404s. I now treat model IDs as leases, not identities, and the API returns a deprecation header when I know one is coming.

The lesson: a unified model catalog is a product surface, not a convenience wrapper. It needs schema validation, freshness guarantees, and a deprecation policy — same as any other API you'd build. The agents using it were fine. It was me who wasn't.

It's running at https://x402.freeq.one/tools/llm_chat.html if you want to poke the model list endpoint and see the normalization scars.