LLM Gateway Compared: LiteLLM vs Portkey vs OpenRouter

Posted on Oct 21 • Originally published at fp8.co

TL;DR: An LLM gateway is a proxy between your application and model providers that unifies the API, routes and retries requests, and tracks spend. LiteLLM and Portkey are open-source gateways you can self-host. OpenRouter, Cloudflare AI Gateway and Vercel AI Gateway are managed services. Self-host when data residency and key custody matter most. Use a managed gateway when you want failover without running a proxy.

The first LLM feature in most codebases calls one provider SDK directly. That works until the second provider shows up. Maybe a cheaper model wins on a subtask, a region needs a different vendor, or an outage takes down the only model you depend on. At that point every call site has to learn a second API shape, a second retry policy, and a second way of counting tokens.

The timing matters in late 2026. Frontier labs now distribute their models through several clouds at once: OpenAI's open-weight models on AWS Bedrock and, per this week's announcement, OpenAI models on Google Cloud, plus Anthropic models on Bedrock and Vertex AI. That makes "which model" and "which provider serves it" two separate decisions. The same model can be reached through three or four endpoints with different prices, rate limits and data-handling terms. A gateway is the layer that makes that choice at request time instead of at code-review time.

Search data from our own readers points the same way. Queries like "pros and cons of model-agnostic llm gateway for small apps" and "agentcore gateway llm gateway" show people trying to work out which kind of gateway they actually need. This guide answers that with a side-by-side comparison of the five gateways most teams shortlist, plus a framework for when you need none of them.

An LLM gateway is a reverse proxy that speaks an LLM API (almost always the OpenAI Chat Completions format, increasingly also the Anthropic Messages and OpenAI Responses formats) on the front and translates to each provider's native API on the back. Everything else is policy that runs on that request path:

┌──────────────────────────── LLM Gateway ────────────────────────────┐ App ───▶ │ auth (virtual key) → budget check → cache lookup → route selection │ ───▶ Provider A (OpenAI │ ↓ ↓ │ ───▶ Provider B format) │ guardrails (pre) retry / fallback / cooldown │ ───▶ Self-hosted vLLM │ ↓ ↓ │ │ guardrails (post) ← response normalization ← usage + cost log │ └─────────────────────────────────────────────────────────────────────┘ Enter fullscreen mode Exit fullscreen mode The capabilities break down into five groups:

None of these are hard to build for one provider. What makes a gateway worth having is getting all five right across every provider at once, including streaming, tool calls and the error-code quirks that differ between vendors.

Two rows decide most evaluations: deployment (who operates the proxy) and data path (whose network your prompts cross). Everything else is a feature delta you can usually work around.

LiteLLM ships in two forms. The Python SDK gives you litellm.completion() with one call signature across providers. The proxy server is the gateway: a standalone OpenAI-compatible service, configured through a config.yaml, that any language can call.

The proxy's strength is how much routing behavior you can express in configuration. Model groups are named aliases. Repeating a model_name across entries load-balances across deployments, and fallback chains map one group to others:

model_list: - model_name: chat-default # alias your app calls litellm_params: model: bedrock/anthropic.claude-sonnet-4-5 aws_region_name: us-east-1 - model_name: chat-default # same alias = load-balanced deployment litellm_params: model: vertex_ai/claude-sonnet-4-5 vertex_location: us-east5 - model_name: chat-backup litellm_params: model: openai/gpt-5-mini api_key: os.environ/OPENAI_API_KEY router_settings: enable_pre_call_checks: true # required for context-window fallbacks fallbacks: [{ " chat-default": [ " chat-backup" ]}] context_window_fallbacks: [{ " chat-default": [ " chat-backup" ]}] litellm_settings: num_retries: 3 # per group, before falling back request_timeout: 60 allowed_fails: 3 # failures/minute before cooldown cooldown_time: 30 Enter fullscreen mode Exit fullscreen mode This example shows the pattern described above: the same model served from two clouds as one load-balanced group, with a different model as the fallback. Your application only ever sends model: "chat-default".

On top of routing, the proxy adds virtual keys, per-key and per-team spend tracking and budgets (backed by Postgres), callbacks to observability tools such as Langfuse, and an admin UI. LiteLLM publishes benchmark figures for proxy overhead (it cites 8 ms P95 at 1k RPS). Treat that as a vendor number to reproduce on your own hardware, not a guarantee.

Choose LiteLLM when you have a platform team, a Python-friendly ops stack, and requirements like "keys never leave our VPC" or "every team gets a budget". Watch out for the operational surface. You own Postgres, Redis (if you enable shared caching or rate limits), upgrades, and supply-chain hygiene. Pin exact versions and verify the signed Docker images LiteLLM publishes rather than pulling latest. A gateway holds every provider credential you own, so it is the highest-value target in your AI stack.

Portkey's open-source AI Gateway (MIT licensed) runs on Node.js: npx @portkey-ai/gateway starts it locally, and it deploys to Docker, a plain Node server, or Cloudflare Workers. Routing is defined in JSON configs that can be attached to a request header or saved on the hosted platform:

{ "strategy": { "mode": "fallback", "on_status_codes": [ 429, 503 ] }, "targets": [ { "override_params": { "model": "@openai-prod/gpt-5-mini" } }, { "override_params": { "model": "@anthropic-prod/claude-sonnet-4-5" } } ] } Enter fullscreen mode Exit fullscreen mode By default, a fallback fires on any non-2xx response. on_status_codes narrows it, which matters more than it looks. You usually want to fall back on a 429 (rate limit) or 503 (overload), but a 400 (your request is malformed) will fail identically on every target. Retrying it just multiplies latency and cost. Targets can themselves be load-balancers or nested strategies, so you can build "load-balance across two keys, fall back to a second provider" as one tree.

Portkey's other differentiator is guardrails. The gateway has a large catalogue of pre-built checks (regex, JSON-schema, PII and others) that can run before or after a model call and can deny, flag or trigger a fallback. Semantic caching, the richer observability UI and team management live in the hosted and enterprise tiers.

Choose Portkey when you want a small self-hosted footprint, a TypeScript-native stack, or guardrails at the gateway rather than in each service. Choose LiteLLM instead when you need the deepest provider coverage and a self-hosted admin and budget system without a paid tier.

OpenRouter flips the model. You don't run a gateway; you buy credits and get one API key for a very large catalogue of models from many inference providers. It has two routing layers:

{ "models": [ "anthropic/claude-sonnet-4.5", "openai/gpt-5-mini" ], "messages": [{ "role": "user", "content": "Summarize this incident report." }], "provider": { "sort": "latency", "data_collection": "deny", "zdr": true, "max_price": { "prompt": 3, "completion": 15 } } } Enter fullscreen mode Exit fullscreen mode The economics are simple. OpenRouter states there is no markup on inference pricing. Its revenue is a fee on credit purchases, plus a 5% fee on bring-your-own-key usage above a monthly allowance. For open-weight models served by many competing providers, sort: "price" can land well below what one provider would charge.

Choose OpenRouter when you need breadth fast: prototyping across many models, evaluating open-weight models without signing up with each host, or a small app that wants failover with zero infrastructure. Avoid it as the only path when your contracts require prompts to stay inside your cloud account, or when you need enterprise discounts negotiated directly with a provider.

Both are managed control planes run by edge platforms. You keep calling a model API, but through their endpoint, and you get logging, caching and failover without deploying anything.

Cloudflare AI Gateway sits in front of Workers AI, OpenAI, Anthropic, Google Gemini and other providers. Its core features (analytics dashboard, caching and rate limiting) are free on all plans, and it supports request retries and model fallbacks. Logs are subject to per-plan storage limits. With unified billing, Cloudflare buys inference on your behalf at the provider's per-token rate and charges a 5% fee on purchased credits. Guardrails are billed as Workers AI inference. It fits best if your app already runs on Workers, because caching and rate limiting then sit on the same network as your code.

Vercel AI Gateway is callable from any infrastructure, not just Vercel deployments, through the AI SDK, OpenAI Chat Completions, OpenAI Responses, or Anthropic Messages formats. It routes across providers with configurable provider order and model fallbacks, records every routing attempt in request logs, and supports budgets per team, project, API key or member. Vercel states it adds zero markup to provider token prices, including BYOK traffic. One detail to read before relying on it: budgets cover spend billed through Vercel's system credentials, while BYOK spend is metered separately. If a BYOK request fails, the gateway can fall back to system credentials, so check that behavior if "only use our provider contract" is a hard requirement.

Choose an edge gateway when your app already lives on that platform and you want observability and failover in an afternoon. Look elsewhere when you need custom routing logic, on-prem models behind your firewall, or a single audit log spanning several clouds you control.

No, and mixing them up is the most common mistake we see in architecture reviews. Both are called "gateways", but they sit on different arrows of an agent diagram:

AgentCore Gateway turns Lambda functions and APIs into MCP tools that agents can discover and call with IAM-scoped credentials. It does not choose which model answers a prompt. A production agent typically needs both: an LLM gateway for the reasoning loop, and a tool gateway for actions, with deterministic authorization on the tool side. Some vendors now market a combined "AI gateway" for both. Evaluate the two halves separately anyway, because their failure modes don't overlap.

This is the question behind the most specific search query we see on the topic, and the honest answer is that a small app often doesn't need a self-hosted one.

The pragmatic path for a small app: start with a provider-agnostic SDK rather than a gateway. The Vercel AI SDK or LiteLLM's Python SDK gives you one call signature with no proxy to run; our TypeScript LLM SDK guide compares the options. Add a managed gateway when you need cross-provider failover or spend visibility, and self-host only when key custody or data residency force it.

Work through four questions in order. Each one eliminates options:

Whatever you pick, test it the same way before production:

Feed the gateway's logs into the same tracing backend as the rest of your agent. Our LangSmith vs Langfuse vs Phoenix comparison covers how to pick one. A fallback that fires silently is a cost and quality change you want to see.

An LLM gateway is a proxy between applications and model providers that exposes one API, usually OpenAI-compatible, and applies routing, retries, fallbacks, key management, caching and spend tracking to every request. It lets you change models or providers through configuration instead of code.

LiteLLM is better when you want the widest provider coverage and built-in virtual keys, team budgets and an admin UI in the self-hosted product. Portkey's MIT gateway is better when you want a lighter Node.js deployment, JSON routing trees with status-code-specific fallbacks, and gateway-level guardrails.

OpenRouter states it adds no markup to inference pricing. It charges a fee when you buy credits, and a 5% fee on bring-your-own-key usage above a monthly allowance. Because it can route to the cheapest provider for open-weight models, total cost can come out lower than a single direct provider.

Not always. Bedrock already gives you many models behind one AWS API, plus cross-region inference. A gateway becomes useful when you also call providers outside Bedrock, need failover to another cloud, or want per-team virtual keys and budgets that span providers. AgentCore Gateway handles MCP tool calls, not model routing.

A well-run gateway adds single-digit to low tens of milliseconds, which is small next to model generation time measured in seconds. The bigger latency risks are misconfigured retries that repeat a non-retryable error, and an undersized self-hosted proxy. Measure P95 with your own payloads before and after adding it.

Yes. LiteLLM, Portkey and Cloudflare AI Gateway can all target any OpenAI-compatible endpoint, which includes vLLM servers. A common pattern is to route routine traffic to a self-hosted open-weight model and fall back to a hosted frontier model on errors or context-length limits.

Originally published at fp8.co. Subscribe for weekly AI engineering analysis at fp8.co/newsletters.