Time to First Token (TTFT): Cut Off Stalled LLM Streams and Fail Over

Time to first token (TTFT) is the time from sending a streaming request to receiving the first token of the response. Bifrost, the open-source AI gateway built in Go, can now enforce a TTFT deadline per routing rule: when a provider accepts a stream but produces no output in time, Bifrost closes that attempt and moves to the next fallback before anything reaches the client. This tutorial configures the cutoff, chooses a value from measured TTFT, and shows how to confirm which provider served each stream.

TTFT covers everything between the request leaving the client and the first piece of real output arriving: network time, provider queueing, prompt processing (prefill), and the generation of the first token. For a streaming chat or coding assistant, it is the moment the user stops staring at a spinner, which is why it matters more to perceived speed than total generation time.

Streams are typically delivered over server-sent events, so a provider can open the connection, send headers and startup events, and still produce no text. Bifrost counts only real output as the first token: text, reasoning, tool call arguments, audio or image data, a finish reason, or usage. Startup events such as response.created or an empty assistant role delta do not count, as described in the routing rules documentation.

For the techniques that lower TTFT itself (prompt size, caching, model choice, region), read the guide to cutting time to first token. This tutorial covers the other half: what to do when one provider's TTFT spikes and you want the request served somewhere else.

A provider that fails fast is easy to handle. It returns a 429 or a 5xx, and the gateway retries or falls back to another provider. A provider that accepts the request and then stalls is harder, because nothing has failed yet. The connection is open, no error has arrived, and the request will only end when the request timeout expires, which is commonly set to 30 or 60 seconds so that long generations can finish.

What you need is a separate deadline for the first token only, short enough to catch a stall and independent of how long the full generation may take. The automatic fallback walkthrough covers fallbacks for outright failures; a TTFT cutoff extends the same chain to slow starts.

You set ttft_timeout_ms on a routing rule. When that rule matches a streaming request, each attempt that still has a fallback after it gets the deadline. If an attempt has not produced its first token in time, Bifrost closes its connection and moves to the next fallback. Because nothing is streamed to the client until the first token arrives, the client never sees the abandoned attempt.

Figure 1: Only attempts with a fallback after them get the deadline, and the client sees nothing until a first token arrives.

Figure 2: Any no answers the attempt to the normal request timeout instead of the TTFT deadline.

The cutoff lives on routing rules because rules already decide the target and fallback chain for a class of traffic. That lets you give a latency-sensitive chat route a tight deadline while batch or agent traffic on the same LLM gateway keeps the default behavior.

Start with a running gateway and at least two providers that can serve the same model family. The gateway setup guide gets Bifrost running with npx or Docker:

npx -y @maximhq/bifrost Enter fullscreen mode Exit fullscreen mode Add the providers in the Web UI at http://localhost:8080 or in config.json. Each provider has its own network_config, which sets retries for errors and the request timeout that the last attempt in a chain still runs under:

{ "providers": { "openai": { "keys": [{ "name": "openai-1", "value": "env.OPENAI_API_KEY", "models": [ "*" ], "weight": 1.0 }], "network_config": { "max_retries": 2, "retry_backoff_initial": 500, "retry_backoff_max": 5000, "default_request_timeout_in_seconds": 60 } }, "azure": { "keys": [{ "name": "azure-1", "value": "env.AZURE_API_KEY", "models": [ "*" ], "weight": 1.0 }] } } } Enter fullscreen mode Exit fullscreen mode The defaults are max_retries: 0, a 500 ms initial backoff, and a 5000 ms cap, as listed in the retries reference. The rule in Step 2 also falls back to Bedrock, so configure it the same way. Azure, Bedrock, and Vertex AI need extra key fields such as deployments or regions; the provider configuration guide lists them.

Retries still matter with a TTFT cutoff in place: a 429 or 503 on the primary is retried according to max_retries before any fallback runs, while a TTFT miss skips straight to the fallback.

Routing rules are CEL expressions with a target, fallbacks, and now an optional TTFT deadline. This rule matches every request for gpt-4o, sends it to OpenAI, and falls back to Azure and then Bedrock, cutting off any attempt with a fallback still ahead of it after 1.5 seconds without a first token:

{ "governance": { "routing_rules": [ { "id": "fast-gpt", "name": "Fast GPT", "enabled": true, "cel_expression": "model == 'gpt-4o'", "targets": [{ "provider": "openai", "model": "gpt-4o", "weight": 1 }], "fallbacks": [ "azure/gpt-4o", "bedrock/openai.gpt-oss-120b-1:0" ], "ttft_timeout_ms": 1500, "scope": "global", "priority": 0 } ] } } Enter fullscreen mode Exit fullscreen mode To add the deadline to an existing rule without editing config.json, update it through the API:

curl -X PUT http://localhost:8080/api/routing/rules/fast-gpt \ -H "Content-Type: application/json" \ -d '{"ttft_timeout_ms": 1500}' Enter fullscreen mode Exit fullscreen mode In the dashboard, the same field appears as Time to first token cutoff (ms) on the routing rule form. The expression language is CEL, so you can narrow the deadline to a tenant, a header, or a team. For example, model == 'gpt-4o' && headers["x-surface"] == "chat" limits the cutoff to traffic your chat front end tags, while background jobs that call the same model keep the normal timeout. The routing and fallback article explains how scopes and priorities decide which rule wins.

Applications do not change. Any client that already calls Bifrost through the OpenAI or Anthropic SDK by changing the base URL, as in the drop-in replacement guide, gets the cutoff as soon as the rule matches:

curl -N http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "gpt-4o", "stream": true, "messages": [{"role": "user", "content": "Summarize the incident report in three bullets."}]}' Enter fullscreen mode Exit fullscreen mode Step 3: Choose a cutoff value from your TTFT data A cutoff that is too tight abandons healthy requests and moves traffic away from your preferred provider. One that is too loose never fires. Pick the value from measured TTFT rather than a guess.

Bifrost exports a bifrost_stream_first_token_latency_seconds histogram through its Prometheus integration, labeled with provider, model, routing_rule_name, and fallback_index. This query returns the 95th and 99th percentile TTFT per provider and model over the last day:

histogram_quantile(0.99, sum by (le, provider, model) ( rate(bifrost_stream_first_token_latency_seconds_bucket[1d]) ) ) Enter fullscreen mode Exit fullscreen mode Change 0.99 to 0.95 for the p95 line. A practical starting point is a cutoff somewhat above the primary provider's normal p99 TTFT, so healthy requests almost never trip it and real stalls do. Long prompts raise TTFT because prefill takes longer, so a route that carries large contexts needs a higher value than a short chat route; give those routes separate rules rather than one compromise number.

Reasoning output counts as a first token for the cutoff, so a model that streams its reasoning is not cut off while it thinks. A model that reasons without streaming any of it can take far longer to emit output, so measure it before putting it behind a tight rule. The AI observability overview covers the other gateway metrics that sit beside this histogram.

After the rule is live, confirm the cutoff fires where you expect and that fallbacks serve the requests it cuts. Figure 3 shows the four places a cutoff is visible.

Figure 3: Headers answer one request, logs explain each attempt, and the histogram tells you where to set the cutoff.

Response headers. On a stream, Bifrost writes its headers before the first chunk. When a fallback served the request, the response carries x-bifrost-routing-info-is-fallback: true, the primary's provider and model in x-bifrost-routing-info-primary-provider and -primary-model, and x-bifrost-fallback-index with the fallback's position counting from 1. The response headers reference lists every header:

curl -sN -D - -o /dev/null http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "gpt-4o", "stream": true, "messages": [{"role": "user", "content": "ping"}]}' \ | grep -i '^x-bifrost-' Enter fullscreen mode Exit fullscreen mode Request logs. Each request's log entry includes an attempt_trail. A cut attempt appears there with ttft_timeout, alongside any key rotations or retries that happened before it. The built-in observability documentation shows where to find the trail in the logs view.

Routing engine log. Each cutoff writes a line in the form TTFT timeout: <provider>/<model> produced no first token within <duration>, which is useful when tailing a single gateway during a test.

Prometheus. Filter the first-token histogram by fallback_index to see how much streaming traffic the fallbacks absorb. A sudden rise in fallback-served streams for one rule usually means the primary's TTFT moved, which is worth an alert. The governance and observability post shows how these metrics feed dashboards.

Bifrost classifies every failed attempt with a failure_class and records it on the attempt trail. A stream cut by the TTFT deadline belongs to the timeout class, keeps ttft_timeout as its reason, and is recorded with status 504. The failure classification table shows how each class drives retries and key rotation:

Two consequences follow. First, a TTFT miss and a provider error take different paths: the error spends the provider's retry budget, while the TTFT miss goes straight to the fallback because retrying a slow provider adds more latency. Second, each fallback runs with its own full retry budget, so a fallback that returns a 429 still rotates keys before the chain moves on.

Load balancing across keys and providers works alongside the cutoff. If you weight a rule's targets across OpenAI and Azure, Bifrost picks one per request and applies the deadline to whichever it picked. In Bifrost Enterprise, adaptive routing shifts traffic away from degraded keys and providers over time, while the TTFT cutoff rescues the individual request that hit the slow one. The adaptive routing deep dive explains what that layer observes.

One cost to plan for: a provider may still bill the prompt tokens of an attempt that was cut off, because it already received the request. A tight cutoff on a route with very long prompts can therefore pay for the same prompt twice whenever it fires.

TTFT is the time between sending a streaming request to a language model and receiving the first token of its response. It includes network time, queueing at the provider, prompt processing, and generation of the first token, and it is the main driver of how fast a chat interface feels. The TTFT reduction guide covers ways to lower it.

It depends on the model, prompt length, and provider, so measure it rather than adopt a fixed target. Compare your own p95 and p99 TTFT per provider and model from the bifrost_stream_first_token_latency_seconds histogram, and set expectations per route, since long-context and reasoning requests have a higher normal TTFT than short chat turns.

No. ttft_timeout_ms applies only to streaming requests. Non-streaming requests ignore it and run under the provider's request timeout, configured as default_request_timeout_in_seconds in the provider's network settings, as shown in the provider network settings.

No. Bifrost streams nothing to the client until an attempt produces its first token, so an attempt that is cut off before that point is invisible to the caller. The stream the client receives comes entirely from the attempt that answered, and the response headers say whether that was a fallback.

No. A TTFT miss goes straight to the next fallback without using the provider's max_retries, because retrying a provider that is slow to start usually adds latency. Errors such as 429 and 503 still use the retry budget before a fallback runs, as described in the retries and fallbacks documentation.

You may be. The provider already received the request, so it can bill the prompt tokens even though Bifrost closed the connection before any output. Keep the cutoff above the primary's normal TTFT so it fires only on genuine stalls, and track fallback-served volume in the Bifrost LLM gateway metrics.

A stalled stream costs users the full request timeout unless something cuts it short. A TTFT cutoff on the routing rule turns that stall into a quick, invisible failover, while retries and the last attempt keep their normal behavior. To try TTFT cutoffs, fallbacks, and routing rules on your own traffic, book a demo with the Bifrost team, or compare gateway latency in the Bifrost benchmarks.