Warum ein 99-Prozent-Uptime-Versprechen bei LLM-API-Relays nicht reicht
Viele Relay-Anbieter für LLM-APIs werben mit hoher Stabilität, im Betrieb zeigen sich aber Ausfälle und Schwankungen. Ein Team beschreibt seine Erfahrungen und wie es die Zuverlässigkeit konkret testet.
*Originally published on the Folkbench blog: https://folkbench.com/en/blog/ai-api-relay-stability-guide?utm_source=luntan&utm_campaign=dev *
A relay site may promise "99.9% fast and stable" and a "dedicated high-concurrency channel". Once you connect, things look different. This is how we went from buying on instinct to running automated checks before we pay for or integrate a relay.
Our early check was a Python script that POSTed to the relay every 5 minutes. HTTP 200 with some text = healthy.
Less than two days into production, users reported the model stopping mid-sentence. The logs showed that some relays still returned 200 OK when the upstream pool was exhausted (429 / 401), when the gateway timed out (504), or when a node crashed. Mid-stream they would quietly insert data: [DONE], or put {"error": "rate limit exceeded"} inside the JSON as if it were the model's answer. A status-code monitor was fully fooled.
Daytime average response stayed near 2 seconds. At the night peak, timeouts and abandonment climbed. The worst case still finished the whole answer in about 4 seconds, but the first token took 12 seconds. Users stared at an empty box and left.
Lesson: one ping, or two casual requests, is not a test of a streaming AI API.
Latency – separate waiting from generating
Reliability – look through the fake success
Stream quality – catch the silent failure
Score = (p̂ + z²/(2n) − z·√(p̂(1−p̂)/n + z²/(4n²))) / (1 + z²/n) Enter fullscreen mode Exit fullscreen mode p̂ = observed success rate, n = sample size, z = normal quantile for 95% confidence. Small samples get penalized.
Folkbench publishes these measurements (P95 latency, availability and per-token price per relay) with no sign-up needed: https://folkbench.com/?utm_source=luntan&utm_campaign=dev