Failover-Route für KI-API-Aufrufe an einem Wochenende selbst gebaut

Saturday, 07:40. My side project's "summarize thread" button returned a 429.

My retry loop made it worse. Every retry burned the same quota I was already out of.

By 08:30 I had a decision to make. Build a full AI gateway, or ship one honest thing by Sunday night.

I shipped the one honest thing. A failover lane with a circuit breaker, a classifier for retryable errors, and five tests that run against fake upstreams. No dashboard. No router config language. No Kubernetes.

Here is the build log: the scope I kept, the code, the tests, and everything I deliberately skipped.

A weekend dies from ambition, not from bugs. So I wrote the cut list first.

Kept: one breaker per provider, retryable-error classification, a half-open probe, and a test harness. Skipped: distributed state, metrics export, prompt versioning, and cost accounting.

The skipped list is not a roadmap. It is a list of problems I do not have yet. One process, one API key, one user: that is my whole deployment.

This is the part most side projects get wrong. A 400 from a bad prompt is not an outage. Failing over on it just hides your bug behind another provider.

So classify first. Only retryable failures should move traffic.

# lanes.py RETRYABLE_STATUS = { 408, 409, 425, 429, 500, 502, 503, 504 } def is_retryable ( exc: Exception ) -> bool: status = getattr ( exc, " status_code ", None ) if status is None: # Network errors, timeouts, connection resets. return isinstance ( exc, ( TimeoutError, ConnectionError )) return status in RETRYABLE_STATUS Enter fullscreen mode Exit fullscreen mode 401 and 403 are configuration bugs. 400 and 422 are prompt bugs. Send those to your logs, not to the fallback lane.

The breaker has three states. Closed means primary. Open means fallback only. Half-open means one probe gets through.

# breaker.py import time from lanes import is_retryable class Breaker: def __init__ ( self, fail_threshold = 3, cooldown_s = 20.0 ): self. fail_threshold = fail_threshold self. cooldown_s = cooldown_s self. fails = 0 self. opened_at = None def state ( self ): if self. opened_at is None: return " closed " if time. monotonic () - self. opened_at >= self. cooldown_s: return " half-open " return " open " def _trip ( self ): self. opened_at = time. monotonic () self. fails = 0 def _reset ( self ): self. opened_at = None self. fails = 0 def call ( self, primary, fallback ): state = self. state () if state == " open ": return fallback (), " fallback_open " if state == " half-open ": try: out = primary () except Exception as exc: if not is_retryable ( exc ): raise self. _trip () return fallback (), " fallback_probe_failed " self. _reset () return out, " primary_recovered " try: out = primary () except Exception as exc: if not is_retryable ( exc ): raise self. fails += 1 if self. fails >= self. fail_threshold: self. _trip () return fallback (), " fallback_error " self. _reset () return out, " primary " Enter fullscreen mode Exit fullscreen mode The second return value is the only observability I allowed myself. It tells you which lane answered, and that is enough for a weekend.

You cannot debug a breaker against a flaky provider. You can debug it against a fake that fails on command.

Five scenarios cover almost everything I care about:

# test_breaker.py import pytest from breaker import Breaker class Boom ( Exception ): def __init__ ( self, status_code ): self. status_code = status_code def primary_raises ( status ): def fn (): raise Boom ( status ) return fn def test_non_retryable_does_not_failover (): b = Breaker ( fail_threshold = 1 ) with pytest. raises ( Boom ): b. call ( primary_raises ( 400 ), lambda: " fallback " ) assert b. state () == " closed " def test_opens_after_threshold (): b = Breaker ( fail_threshold = 3 ) for _ in range ( 3 ): out, lane = b. call ( primary_raises ( 429 ), lambda: " fallback " ) assert out == " fallback " assert b. state () == " open " def test_open_skips_primary (): b = Breaker ( fail_threshold = 1 ) b. call ( primary_raises ( 503 ), lambda: " fallback " ) calls = [] def spy (): calls. append ( 1 ) return " primary " out, lane = b. call ( spy, lambda: " fallback " ) assert lane == " fallback_open " assert calls == [] Enter fullscreen mode Exit fullscreen mode Scenario five needs a short cooldown. Set cooldown_s=0.05 and the half-open test runs in milliseconds.

Run it: pytest -q test_breaker.py. Five green tests, one afternoon, no API key.

A fallback lane only helps if it is cheap and available during someone else's outage. That rules out a second premium key for a hobby project.

This is where MonkeyCode fit my build. The operator states that the project offers free model access and a free server option, and that it is open source; I used it as the fallback lane and as the place to run the cron job. Treat those as operator-supplied claims, not as measured results. As of this writing the operator also states a free token allowance in the millions, so check the current number on the project's own pricing page before you size anything around it. Allowances change.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The important design point is not the vendor. It is that your fallback must be a different failure domain from your primary. Two keys in the same account are one outage, not two.

I skipped distributed state. The breaker lives in one process, so a deploy resets it. For a side project that runs one worker, a reset is a non-event.

I skipped per-endpoint breakers. My app has one model call path. Adding granularity before I had a second path would have been speculative abstraction.

I skipped metrics. The lane string already goes into my request log, and I read that log twice a week.

I also skipped prompt-level retries. If the model returns a wrong answer, a breaker will not save you. That is an eval problem, and it is next weekend's problem.

Free lanes have their own rate limits. A breaker does not buy you capacity.

Half-open sends one probe. Under real concurrency you may want a token bucket there instead.

And an in-memory breaker forgets everything on restart. If you need it to survive deploys, you need shared state, and that is a different weekend.

Skip this approach if your fallback model has never passed your own eval set. Skip it if a regulator, not an engineer, decides where your data goes. Skip it if you are already at high request volume, because a single-process breaker will flap under load.

If you want to try the lane I used, start with a throwaway key and the five tests above. Get them green before you point real traffic at anything.

Sunday, 23:10. The button works, the tests pass, and the only new file is 40 lines long.

The scope cut is what shipped the feature. Not the model.