How I cut my AI inference bill by 80% - moving from OpenAI to self-hosted models

My OpenAI bill was $400/month. After moving to self-hosted models on a single Hetzner GPU box, it's $80/month and falling. Here's the playbook.

I was using GPT-4o for ~3M input tokens/day for a RAG app and ~500K output tokens/day. About $450/month. The killer was the output tokens.

Worked for ~30% of queries. Quality drop too steep for the rest. Saved ~$100/month.

Cost: $1.50/hour for an A100, mostly idle. About $300/month. Same quality as GPT-4o on most of my tasks. Breakthrough.

Hetzner GPU servers (H100, A100) in Frankfurt. Latency ~10ms from EU, ~95ms from US. Worked for me.

Hardware: Hetzner GPU server, 1× A100 80GB on-demand. Effective cost ~$190/month (shared between workloads).

Software: vLLM serving Qwen 2.5 72B (Apache 2.0), all-MiniLM-L6-v2 for embeddings, LiteLLM as drop-in OpenAI-compatible gateway.

If you have $500+/month OpenAI spend, the math works.

Savings drop as you scale. But at current scale, savings are real.

AI inference is now genuinely cheap to self-host if you have the volume to amortize setup. If you're paying $400+/month to OpenAI, it's worth investigating.