Enterprise AI gateway
Enterprise AI Gateway – Technical Guide A control‑plane that authenticates, routes, and governs every LLM request across an organization. 📋 Diagnosis Table Symptom Likely Root Cause When a Gateway Becomes Worthwhile Sprawl of API keys Each team provisions its own provider credentials Multiple inde
A control‑plane that authenticates, routes, and governs every LLM request across an organization.
flowchart TD A["Multiple LLM Consumers"] --> B["Compliance / Cost Constraints"] B --> C["Need Central Auth & Quotas"] C --> D["Enterprise AI Gateway"] D --> E["Implement Shared Control Plane"] A --> F["Few Consumers"] F --> G["Ad‑hoc Direct Provider Calls"] Enter fullscreen mode Exit fullscreen mode If the left‑hand path is taken, a gateway is strongly recommended. The right‑hand path may remain viable for very small deployments.
Teaching‑only – The snippet demonstrates how a service could decide whether to invoke the gateway or fall back to a direct provider, based solely on a configuration file.
# enterprise_ai_gateway_helper.py """ Decision helper for routing LLM requests. - Reads a JSON config that contains: * `allow_list`: approved model identifiers * `quota_limits`: per‑tenant request caps * `fallback_provider`: secondary cloud vendor - Returns a dict with routing metadata; raises AssertionError on mis‑configuration. """ import json import os from pathlib import Path from typing import Dict, Any CONFIG_PATH = Path ( os. getenv ( " GATEWAY_CONFIG ", "./gateway_config.json " )) def load_config () -> Dict [ str, Any ]: """ Load and validate the gateway configuration. """ with CONFIG_PATH. open () as fp: cfg = json. load ( fp ) # Basic sanity checks – deterministic assertions, not runtime heuristics. assert isinstance ( cfg. get ( " allow_list " ), list ), " allow_list must be a list " assert cfg. get ( " allow_list " ), " allow_list cannot be empty " assert isinstance ( cfg. get ( " quota_limits ", {}), dict ), " quota_limits must be a dict " assert isinstance ( cfg. get ( " fallback_provider " ), str ), " fallback_provider required " return cfg def decide_routing ( tenant_id: str, requested_model: str, current_usage: int ) -> Dict [ str, str ]: """ Determine the provider and model to use.
Parameters ---------- tenant_id: str – Identifier of the calling team/product. requested_model: str – Model name requested by the caller. current_usage: int – Number of requests already made this billing period. Returns ------- dict with keys `provider`, `model`, `via_gateway`. """ cfg = load_config () # Enforce allow‑list assert requested_model in cfg [ " allow_list " ], f " Model { requested_model } not approved " # Enforce quota (simple per‑tenant cap) quota = cfg [ " quota_limits " ]. get ( tenant_id, 0 ) assert quota > 0, f " No quota defined for tenant { tenant_id } " assert current_usage < quota, f " Tenant { tenant_id } exceeded quota ( { current_usage } / { quota } ) " # Routing decision – primary provider derived from model name convention primary_provider = " azure " if requested_model. lower (). startswith ( " gpt " ) else " aws " # In a real deployment the health check would be dynamic; here we assume primary is healthy. provider = primary_provider # Example of deterministic fail‑over logic (placeholder) # if health_check(provider) == "unhealthy": # provider = cfg["fallback_provider"] return { " provider ": provider, " model ": requested_model, " via_gateway ": " true ", # All decisions flow through the gateway } # Example usage (for illustration only) if __name__ == " __main__ ": # Simulate a request from team 'risk‑engine' asking for 'gpt-4o-mini' with 12 prior calls. routing = decide_routing ( " risk-engine ", " gpt-4o-mini ", 12 ) print ( routing ) Enter fullscreen mode Exit fullscreen mode 📄 Illustrative Architecture Decision Records ADR 1 – Adopt Central Model Allow‑List Context Multiple engineering squads independently select foundation models.
Procurement data shows several distinct model versions in production, leading to fragmented licensing and security reviews.
Decision Introduce a central allow‑list stored in a managed key‑value service. The gateway validates every request against this list before routing.
Context During a load test, Azure OpenAI latency spiked markedly, breaching internal availability expectations. No automatic reroute existed, leading to request timeouts.
Decision Encode a fail‑over rule in the gateway: if the primary provider’s recent latency exceeds an acceptable threshold, automatically route new requests to the secondary provider defined in fallback_provider.
This guide is intended for architects and platform engineers evaluating the trade‑offs of an Enterprise AI Gateway. Adoption decisions should be driven by measurable compliance, cost, and resilience signals rather than assumed performance gains.