Not All AI Provider Failures Are the Same: Building a Fault-Aware LLM Fallback System

Not All AI Provider Failures Are the Same: Building a Fault-Aware LLM Fallback System

A common approach to multi-provider LLM integration is straightforward: send a request to a provider, catch an error if it fails, and try the next available key or provider.

That works for some failures, but treating every error identically introduces unnecessary problems.

An invalid API key, a provider-wide outage, a deprecated model identifier, and a malformed application request are not the same failure. They have different causes and require different responses.

For example, if a provider returns an HTTP 503 because its service is temporarily unavailable, cooling down a perfectly valid API key does not solve the underlying problem. It simply reduces the key's availability after a failure that was unrelated to its validity.

In StudyHub, I built a multi-provider LLM routing system that classifies provider failures before deciding whether to cool a key, switch providers, invalidate a cached model, or stop the request.

A fallback system needs to answer two questions:

Without classification, the router can waste valid credentials, repeatedly request models that no longer exist, or continue retrying requests that will fail regardless of which provider receives them.

StudyHub represents provider failures through a custom "ProviderCallError" that carries diagnostic information such as status codes, provider identifiers, and parsed error details.

The system then classifies failures into four categories:

The taxonomy determines how the routing system should respond.

The "KEY_FAULT" category represents failures associated with the credential or account state.

In the current classification rules, these include HTTP statuses 401, 402, 403, and 429.

Examples include invalid credentials, account restrictions, billing problems, and rate limits associated with the account or key.

When the router classifies a failure this way, it cools the affected key and attempts another eligible key or provider.

The intention is to avoid repeatedly sending requests through a credential that is currently unable to serve them.

However, status codes are signals rather than universal explanations. A 429 response, for example, may represent a key-specific rate limit, an account-level quota, or another provider-specific restriction.

For that reason, the classification should be understood in the context of the provider's API behavior and the error information available to the implementation.

The "PROVIDER_TRANSIENT" category handles failures such as HTTP 500, 502, 503, and 504, along with qualifying network-level connection failures.

These failures commonly indicate temporary provider or network problems rather than invalid credentials.

When such a failure is classified as transient, StudyHub advances to another provider without unnecessarily cooling the current key.

This distinction helps preserve credential availability during external outages.

It also illustrates why retry behavior should follow the cause of a failure rather than treating every exception as evidence that the current key is unusable.

The exact handling still depends on the provider's response semantics and the router's classification rules.

Providers change their model catalogues. Models may be renamed, removed, deprecated, or made unavailable to a particular endpoint.

A model identifier that worked yesterday may therefore fail even though the provider and credential remain usable.

StudyHub classifies certain HTTP 400 responses as "BAD_MODEL" when structured error information indicates that the requested model is invalid, missing, or decommissioned.

It then checks for known indicators of an unavailable model.

When a response matches the classification rules, the router evicts the affected model from the shared Redis discovery cache rather than cooling the key.

This prevents other application instances from continuing to select the same cached model identifier after the system has determined that it is no longer usable.

The distinction is important: invalidating a model and invalidating a credential are different operations because they address different underlying resources.

The classification also needs to remain conservative. Not every HTTP 400 means that a model has disappeared; malformed payloads and other client-side errors can produce the same status.

The final category, "NON_RETRYABLE", acts as the conservative default for errors that do not match the known classifications.

The reasoning is that some failures originate in the application's request construction rather than in the provider's availability.

If the application sends a malformed payload, rotating through several providers may simply repeat the same mistake.

In those cases, failing fast avoids wasting requests and obscuring the original error behind a chain of unsuccessful fallback attempts.

This policy depends on the classifier being accurate. An unfamiliar provider error may be misclassified as non-retryable even when another provider could have handled the request.

Unknown errors should therefore remain observable, and the taxonomy should evolve as new provider response patterns are understood.

A practical implementation challenge is that Python's "requests" library exposes failures through different exception types.

An HTTP error may include a response with a status code and structured body. A connection failure or timeout may occur without any HTTP response at all.

StudyHub uses a helper, "_wrap_request_exception", to normalize these failures into a common representation.

For HTTP errors, the wrapper extracts the response status code and attempts to parse the response body as JSON so the classifier can inspect structured error details.

For network failures, the wrapper captures the exception information even when no response was received. A fallback regular expression, "_NETWORK_MESSAGE_FALLBACK_RE", can inspect the error message when necessary.

Normalizing the information allows the classification layer to reason about failures consistently without requiring every routing decision to understand the details of each Python exception type.

The goal is to separate exception handling from the policy that determines what the router should do next.

A failure classification is less useful if every application instance maintains an independent view of provider health.

Suppose one instance identifies a key as temporarily unusable. If another instance immediately retries that same key because it has not observed the first instance's state change, the system can generate unnecessary failures.

StudyHub centralizes runtime state in Redis using dedicated key families for cooldowns and provider blacklists, including prefixes such as:

This gives multiple instances a shared mechanism for observing relevant failure state.

The system also provides an environment-variable-controlled switch, "AI_PROVIDER_REDIS_STATE_ENABLED", that can disable Redis-backed provider-state tracking and fall back to process-local in-memory state.

That fallback can preserve some operation when Redis is unavailable, but it changes the coordination model. Individual processes may then hold different cooldown and blacklist information.

In other words, the fallback favors continued local operation over consistent shared state. It should not be mistaken for equivalent distributed coordination.

The classification system improves the router's ability to respond appropriately, but it also introduces design responsibilities.

Providers do not necessarily use status codes and error bodies in identical ways. A classifier that works for one provider may need additional rules for another.

Fallback is not always the correct answer

A second provider may not support the requested model, modality, or request format. The router must still respect provider compatibility and request constraints.

Shared state introduces its own dependency

Redis allows instances to coordinate cooldowns and blacklists, but Redis availability and the behavior of the local-state fallback must be considered explicitly.

When the classifier cannot confidently identify a failure, preserving diagnostic information and monitoring unknown errors helps prevent silent misclassification.

These trade-offs are part of building a reliable routing layer. Provider fallback is not simply a list of API keys; it is a policy for responding to different failure conditions.

Multi-provider LLM routing becomes more reliable when the system distinguishes the reasons requests fail.

StudyHub's failure taxonomy separates credential-related problems, transient provider failures, invalid model identifiers, and errors that should stop further fallback attempts.

Each category leads to a different response: cool the relevant key, switch providers without penalizing the credential, invalidate a stale model entry, or fail fast.

Combined with normalized Python exceptions and Redis-backed shared state, this design makes provider failure handling more deliberate and observable.

The broader lesson is that resilience does not come from retrying everything. It comes from understanding what failed, choosing a response appropriate to that failure, and preserving enough information to investigate cases the system does not yet understand.