Building a Rate-Limit-Aware LLM Router in Python: Retries, Backoff & Capacity Scoring

Most "LLM router" projects treat rate limits as an afterthought. They don't. This post walks through the pieces that make a router actually survive a provider's 429s in production.

If you've ever had your app die because OpenAI hit a 429 Too Many Requests mid-request and your only fallback was "try again in 1 second in a tight loop," you know the pain. That pattern burns you twice: you hammer the provider faster, and you have no idea how long to actually wait.

The fix isn't a longer sleep. It's reading the provider's own headers and building a router that's capacity-aware. That's the core engineering problem this post tackles, using SentioGate as the reference implementation.

A rate-limit-aware LLM router is three things glued together:

Each one is small. Together they're the difference between "occasionally 429s" and "never surfaces a 429 to the user."

Providers are inconsistent. OpenAI sends x-ratelimit-limit-tokens, x-ratelimit-remaining-tokens, retry-after. Anthropic sends anthropic-ratelimit-requests-remaining, anthropic-ratelimit-requests-reset. Gemini sends retry-after + its own quota headers. Groq, Mistral, DeepSeek — each has its own flavor.

Your router's provider adapter layer normalizes them:

@dataclass class CapacityState: remaining_requests: int # requests left in current window reset_at: datetime # when the window refreshes remaining_tokens: int | None # if the provider reports token quotas reset_tokens_at: datetime | None is_rate_limited: bool # last response was a 429 Enter fullscreen mode Exit fullscreen mode Each provider adapter's job is to fill in one of these from its response. The router core never knows which provider it's talking to — it only sees CapacityState.

The router's scoring layer already ranks candidates on cost, latency, reliability, quality, and — crucially — capacity. The capacity score is derived from CapacityState:

This is where most naive implementations go wrong. They treat a 429 as "retry in 1s" instead of "this provider is *unavailable until time T, rank it last until then." The circuit-breaker framing is the difference between a router that survives and one that just shuffles the same failing candidate around.

When the router must retry (you're out of alternatives, or the failure was transient), it uses the server's Retry-After as the base wait:

def next_wait ( retry_after_s: int, attempt: int, backoff_base: float = 0.5 ): # Respect the server; floor at the exponential backoff so you don't # trust a 0 from a flaky provider. server_wait = max ( retry_after_s, 0 ) backoff_wait = backoff_base * ( 2 ** ( attempt - 1 )) return max ( server_wait, backoff_wait ) Enter fullscreen mode Exit fullscreen mode Key detail: max(), not +. If the server says "retry in 30s" and your backoff says "wait 2s," you wait 30s. If the server says "retry in 0" (a bug or a 500, not a 429), your backoff is the floor. You never blind-loop.

The router caps total retries ( max_retries ) and enforces a wall-clock deadline on the whole request, so a pathological provider can't hang your app.

This is the part people forget. Retries + backoffs add up. A request that's "30s timeout, 2 retries, 0.5s base backoff" can actually run for over 70s before giving up. Your front-end is long gone.

So the router carries a single wall-clock deadline for the entire logical request (all retries included). Each candidate attempt gets min(candidate_timeout, time_until_deadline). When the deadline passes, the router stops and returns what it has (or a structured error) — it never runs past the deadline.

This is what "hardened transport" means in the intro post.

With all three pieces in, the router's behavior under pressure is:

That last line is the goal: the user never knows a provider failed. The router absorbs it.

Here's the skeleton (abridged) of the core routing loop you'd build around these pieces:

def route ( request, providers, deadline ): scored = rank ( providers, request ) # cost+latency+reliability+quality+capacity for attempt, provider in enumerate ( scored ): if time. time () > deadline: return DeadlineExceeded () if provider. capacity. is_rate_limited: continue # circuit breaker: skip for now try: return provider. invoke ( request, timeout = min ( provider. timeout, deadline - time. time ())) except TransientError as e: update_capacity ( provider, e. headers ) # decode Retry-After / x-ratelimit if attempt + 1 >= max_retries: raise time. sleep ( next_wait ( e. retry_after_s, attempt + 1 )) return NoProviderAvailable () Enter fullscreen mode Exit fullscreen mode The whole system is ~200 lines once the provider adapters exist. The adapters are the real work — and they're the part SentioGate's SDKs handle for you.

httpx / requests retry middleware is connection-level — it retries the same URL with the same body. It has no idea that a 429 means "this provider is capped, rank another one." A router is decision-level — it picks a different provider on the next attempt. That's the entire value proposition, and it's why a plain retry loop isn't enough.

This is the engineering post of the series. The companion posts: intro, semantic compaction, and custom providers. Tell me below what I should cover next.