LLM Gateway Reference Architecture For The Enterprise

A single developer calling a model through a gateway needs little more than a base URL and a key. An enterprise running hundreds of AI features across many teams, regions, and regulatory regimes needs something more carefully structured. This article lays out a reference architecture for deploying an LLM gateway at enterprise scale, walking through the split between control and data planes, regional deployment, authentication and access control, budgets, guardrails, observability, and the deployment topologies that determine where your data lives. TrueFoundry is used throughout as the concrete example, shown to make the architecture tangible rather than to make a pitch.

The most important architectural decision in an enterprise gateway is to separate the control plane from the data plane. These are two distinct components with two distinct jobs, and keeping them apart is what lets the system be both centrally governed and fast at the edge.

The control plane is the centralized brain. It holds configuration: which models exist, how routing is defined, who has access to what, what budgets and rate limits apply, which guardrails are active. It is where administrators and platform teams make decisions. It is not in the path of any model request.

The data plane, also called the gateway plane, is the only component that sits in the request path. When an application calls the gateway, it is the data plane that receives the request, applies routing, meters tokens, enforces limits, and forwards to the provider. Crucially, the data plane does not phone home to the control plane on every request. It reads its configuration and cache from memory, so the per-request work stays local and fast. This is the architectural reason a gateway can add typical latency overhead under 5 milliseconds, internally around 3 to 4 milliseconds, at throughput on the order of 200 to 250 requests per second per single CPU: the in-path component is lean, and the heavy coordination happens out of band.

Because the data plane is the only thing in the request path and reads its configuration from memory, you can deploy it regionally. An enterprise with users and workloads in several regions can run a data plane close to each one, so requests do not have to cross the globe to reach a gateway. Each regional data plane pulls the same centrally managed configuration from the control plane, which means policy is consistent everywhere while traffic stays local.

This regional model is also the foundation of data residency control. Where the data plane runs, and where it is allowed to send request and response data, is something the architecture lets you constrain. That matters enormously for organizations bound by rules about where personal or regulated data may be processed and stored.

It helps to trace a single request end to end. An application, authenticated with its credential, sends an OpenAI-compatible chat-completions call to the regional data plane nearest it. The data plane validates the credential and checks the caller's access rights and rate limits, drawing on configuration it already holds in memory.

It resolves the requested model or virtual model group, applies the routing policy to select a target, and if guardrails are configured, inspects the prompt before it leaves. It forwards the request to the chosen provider, streaming the response back as it arrives, and as the stream flows it accumulates token usage. On completion it records that usage against the relevant team, application, and user for budget attribution and observability, applies any response-side guardrails, and returns the result.

If the chosen target had failed, the data plane would have transparently retried against a fallback candidate. Throughout, the control plane was never in the path; it had already supplied the configuration that made every one of these decisions possible.

At enterprise scale, credentials are not an afterthought. The architecture distinguishes two credential types by purpose. A Personal Access Token (PAT) represents a human and suits development and interactive use. A Virtual Account Token (VAT) represents an application or service and suits production workloads. Keeping these distinct means a departing employee's personal token can be revoked without taking down production services, and a compromised application token can be rotated without disturbing developers.

Around those credentials sits role-based access control. Access to models, the ability to change routing, the authority to set budgets, the visibility of usage data: all of these can be granted by role rather than handed out uniformly. A platform team might administer configuration while individual product teams can only call the models they have been granted and view their own usage. This is how a single shared gateway serves many teams without any team being able to affect another.

Because every request flows through the data plane and its usage is metered in tokens, the control plane can express spend and protection as policy. Budgets can be set per team, per application, or per user, and enforced so that once exhausted, further requests are refused rather than silently accruing cost. Rate limits cap request or token throughput per consumer, protecting shared provider quotas from any single noisy service. Guardrails, applied in the data plane where the prompt and response text is visible, screen content on the way in and on the way out. The important architectural point is that all three are defined centrally in the control plane and enforced locally in each data plane, so a policy written once applies everywhere the gateway runs.

Centralizing traffic through the data plane makes the whole fleet observable from one place. Request volumes, latencies, error rates, token consumption, and cost can be broken down by model, team, and application. For an enterprise this is both an operational necessity, letting platform teams spot a failing provider or a runaway workload, and a governance necessity, letting finance and compliance answer who spent what and which models handled which traffic.

The same architecture supports several deployment topologies, and the choice among them is largely a choice about where your request and response data resides. There are four to consider:

These topologies exist precisely because enterprises differ in how much they must control where data flows. A team with light requirements may happily use SaaS; a bank or a defense contractor may need self-hosted or air-gapped. On the compliance side, the platform supports SOC 2, HIPAA, GDPR, and ITAR, which maps onto the range of regulated industries these topologies are meant to serve.

Even in an enterprise deployment, an application's view of the gateway stays simple, which is the point of putting all the complexity in the control plane. An application still just points the standard OpenAI client at its regional data plane endpoint and supplies its production credential. Using environment variables, that looks like this:

Application-side configuration. All routing, budgets, access control and guardrails are defined centrally in the control plane, not here.

export OPENAI_BASE_URL=" https://gateway.truefoundry.ai " export OPENAI_API_KEY="your_virtual_account_token"

Everything that makes the deployment enterprise-grade, the regional data planes, the RBAC, the budgets, the guardrails, the residency rules, is configured once in the control plane and enforced in each data plane. The application developer sees none of it; they see a base URL and a key. That contrast is the whole design in miniature: complexity concentrated where it can be governed, simplicity preserved where work gets done.

A sound enterprise reference architecture for an LLM gateway therefore reads as a set of deliberate separations. Configuration is separated from the request path by splitting the control plane from the data plane. Latency is minimized by letting regional data planes serve requests from in-memory configuration.

Governance is centralized through RBAC, budgets, rate limits, and guardrails defined once and enforced everywhere. Data residency is honored by choosing a deployment topology, from SaaS through hybrid and self-hosted to air-gapped, that places request and response data where regulation requires. And observability ties it together by making the whole fleet legible from a single vantage point.

Each separation buys a specific property, and together they let one gateway serve an entire enterprise without forcing any team to trade speed for control.