I built a gateway that retries LLM refusals until I get a real answer

If you work with LLM APIs you know the dance: you ask for working code, and you get a lecture. A disclaimer. Three paragraphs of "as an AI I cannot" followed by a suspiciously detailed explanation of why it cannot.

I got tired of it, so I built armor-gw — a self-hosted, OpenAI-compatible proxy that sits between your app and any upstream model and refuses to accept refusals.

Every response passing through the gateway gets inspected by 13 detector classes. Not just the obvious "I can't help with that" — also the sneaky ones:

When a refusal is detected, the request gets reformulated by one of 62 mutation operators and retried — while a second variant from a different strategy family races it in parallel. First clean answer wins.

The strategy layer is a Thompson-sampling bandit that learns per-model which approach actually works: persona framing, encoding tricks, or inversion techniques. A diagnostic judge labels each refusal with its cause (policy / safety-speak / capability-doubt / evasion) so the next retry targets the actual failure mode.

A normal OpenAI endpoint. Drop-in compatible:

from openai import OpenAI client = OpenAI ( base_url = " http://localhost:8787/v1 ", api_key = " sk-your-key " ) r = client. chat. completions. create ( model = " your-upstream-model ", messages =) Enter fullscreen mode Exit fullscreen mode Multiple upstreams, per-model routing, failover. The console UI shows every retry round live — watching it classify a refusal, pick a strategy, and break through on round 3 is honestly a little addictive.

Detecting refusals is harder than it looks. Keyword lists fail immediately — models rephrase. The most reliable signal turned out to be entity survival rate: if the technical nouns from your request never appear in the response, nobody answered your question no matter how confident the prose sounds.

Output hygiene matters as much as breaking through. The gateway strips chain-of-thought leaks, disclaimers, filler questions, and "want me to elaborate?" endings. The deliverable is the deliverable.

Strategies decay. Upstream models get quietly updated, and an operator that worked for weeks stops working. There's a regression bank (65 seeded test cases) that runs on demand and flags any topic whose success rate drops 20%.

Self-hosted, MIT, one-command deploy, runs on anything with Python 3.9+ (I develop on an aarch64 Alpine box):

Feedback welcome — especially which upstreams you'd want supported out of the box, and what refusal patterns you're hitting that the detectors miss.