AI control plane: the architecture for AI governance and security

Posted on Nov 5 • Originally published at speakeasy.com

By Sagar Batchu, Co-founder & CEO at Speakeasy. Originally published on Speakeasy.

The AI control plane is the governing layer between every AI agent in an organization and every system they're allowed to reach. It unifies connection, identity, policy enforcement, and observability so that every prompt, response, and tool call flows through a single controlled path.

AI tools are proliferating within companies faster than anyone can keep track of. Teams are adopting Cursor, Claude Code, Copilot, ChatGPT, and dozens of internal agents, often independently and often without IT involvement. Most organizations have no reliable way to know which tools are in use, which company data those tools can access, what is being shared with external AI services, or whether any of it is being measured.

The AI control plane is the architecture that solves these problems and puts enterprise AI governance on a single footing. It covers four capabilities (connect, distribute, secure, and observe) across every AI agent and every system those agents reach.

The rest of this guide walks through why this layer is now necessary, how it differs from the AI governance tools enterprises already have in place, and what it looks like in practice.

Two narrower guides test specific parts of this architecture. AARM defines a vendor-neutral bar for runtime interception and policy decisions, while the agent observability platform comparison separates engineering traces from the identity-bound audit record a control plane produces.

Companies want to adopt AI because it makes teams faster and produces better work. The budget is there, the appetite is there, and AI agents like Cursor, Claude Code, Copilot, and ChatGPT have already found their way into daily workflows across engineering, sales, finance, and ops.

The problem is execution. Rolling AI out consistently across an organization is hard. Most companies that try run into the same set of gaps:

Locking everything down kills adoption, and leaving everything open invites accumulating incidents.

An audit typically turns up dozens of AI tools in use across the organization, many never approved and none centrally managed. This is the shadow AI problem, and it is already widespread: in the 2026 IBM Cost of a Data Breach Report, 43% of breached organizations traced incidents to shadow AI, up from 20% a year earlier. The cost of doing nothing is clear: data is leaked through consumer AI tools, prompt-injection attacks target agent systems, and enterprises face regulatory pressure from the EU AI Act.

We're rolling out AI faster than we can govern it. I don't think anybody in this industry isn't.

Several categories of AI governance platform have emerged to address parts of the enterprise AI governance problem, each covering one slice of it.

View the AI control plane reference architecture on Speakeasy.

LLM gateways and AI gateways (Portkey, LiteLLM) sit between applications and language models. They handle routing across providers, API key management, rate limiting, caching, and cost tracking. An AI gateway operates at the model call layer.

MCP gateways and agent gateways (Speakeasy, Runlayer, Mintive (formerly MintMCP)) sit in front of MCP servers and handle authentication, authorization, and inspection of tool calls. An agent gateway is the runtime chokepoint that agent traffic flows through. The Model Context Protocol (MCP) is the emerging standard for how AI agents connect to external tools and data.

Identity and access controls (Okta, Microsoft Entra ID, Google Workspace) provide SSO and enterprise identity infrastructure. Most enterprises already have these in place, though they were not designed for AI agents.

Observability tools (Langfuse, Arize, Galileo) provide visibility into what AI systems are doing. They record which tools are being called, and what outcomes they produce, by which users, and with what frequency.

Policy and threat detection tools (Lakera, Fiddler, Treza) inspect prompts, responses, and tool calls in real time, looking for PII leakage, prompt injection, data exfiltration, and policy violations. The primitive that makes this possible at the AI agent itself is the agent hook, a lifecycle handler that fires on every prompt and tool call.

View the AI control plane vendor landscape on Speakeasy.

Each category solves a real problem. However, none of them covers the full picture because each sees a different part of the same interaction:

No single tool has a correlated view across all these layers. An incident that starts with a user authenticating through SSO, routes through an LLM gateway, triggers an MCP tool call, and leaks data in the response would leave traces in five different systems with no common thread connecting them.

Most enterprises already have vendors shipping AI governance features, and each occupies one layer. Hyperscaler controls (AWS Bedrock, Azure AI Foundry, Google Vertex AI) apply only to traffic routed through their platform. Enterprise platform vendors (ServiceNow, CrowdStrike, Salesforce, and the major GRC suites) own the policy and risk layer but sit above the traffic, though Salesforce's MuleSoft Agent Fabric and ServiceNow's AI Control Tower are now adding in-path enforcement. Point tools such as LLM gateways and MCP security tools are inside the traffic but see only their own layer.

AI governance requires owning the path between layers, and closing that gap means rebuilding significant parts of each incumbent architecture. The vendor-by-vendor analysis is in Can incumbent AI governance vendors close the gap?

The term "control plane" originated in networking and is now standard in systems like Kubernetes and service meshes, which have control planes for the same reason: they name the part of the system that governs the rest of the system. Some analysts and vendors call the same layer an agent control plane when they want to stress the agents it governs, but the architecture is the same. The four agent types define the working arrangements that layer must cover.

The AI control plane has two jobs. The first is enablement: rolling out AI capabilities across the organization so every team can use them. The second is governance: making sure that rollout doesn't cause a security incident, a data leak, a compliance violation, or a regulatory problem.

These two jobs pull in opposite directions. Enablement wants more tools, more data connections, and broader access. Governance wants every interaction inspected, every permission scoped, and every action audited.

The instinctive organizational response is to assign enablement and governance to different teams. For example, IT or security could own governance, and a platform team or newly hired Chief AI Officer could own enablement.

This split has a consistent failure mode. The governance team, lacking visibility into what people actually need from AI, designs controls that block the most useful capabilities. The enablement team, under pressure to ship, routes around the controls or negotiates exceptions. The result is either a governance posture that exists on paper while employees use personal ChatGPT accounts for real work, or an adoption program that moves fast and creates incidents the security team discovers in postmortems.

This failure is structural. Governance and enablement look like opposing forces because organizations treat them as separate layers. When the same layer that routes AI traffic also enforces policy on it, governance and enablement are no longer in tension.

Consider the infrastructure analogy: Kubernetes does not have one component that allows workloads to run and another that ensures they run safely. The control plane does both, by design.

An AI control plane does four things: connect, distribute, secure, and observe. The linked diagram shows how the four functions fit together.

View the four functions of an AI control plane on Speakeasy.

It should bring every AI agent (Claude, ChatGPT, Cursor, Copilot, Codex, internal agents, and product agents) and every system that matters (SaaS tools, internal APIs, databases, and skills) onto a single plane, with per-team registries and SSO-integrated identity.

The result is that new AI capabilities reach the right teams in days instead of months, and IT stops chasing a moving target of ad-hoc integrations.

It should govern how AI tools and access roll out across the organization, enforcing who can use what and under what conditions by scoping access to teams or roles, implementing credential management that keeps API keys out of config files, using OAuth 2.1 where the protocols support it, and maintaining a full audit trail.

The shift that matters here is that policies become versioned, testable, executable rules applied automatically at the point of use. The control plane evaluates policy on every request, unlike a wiki document that relies on people remembering to follow it.

The policy on paper becomes the policy at runtime, and audits stop surfacing questions the CISO can't answer.

It should inspect every prompt, response, and tool call in real time, with active blocking of PII and data exfiltration, passive detection of prompt injection and shadow MCPs, and integration with existing SIEM and security tooling rather than replacing it.

Incidents become detectable in real time, and the response team works from the tooling they already know.

It should measure what's actually happening. By recording token use by team, agent, tool, and user, then tracking that use against organizational targets, the control plane produces data that proves whether the AI investment is working.

Leadership gets real numbers behind the AI investment rather than anecdotes, and can distinguish adoption from adoption theater.

Two terms show up constantly in vendor pitches and analyst reports: AI guardrails and AI risk management. The AI control plane is how an enterprise operationalizes both.

AI guardrails are the runtime controls that constrain what AI agents are allowed to say and do, blocking PII leakage, prompt injection, unsafe tool calls, and data exfiltration. In the control plane, this is the Secure function: real-time inspection of every prompt, response, and tool call, integrated into existing SIEM tooling rather than replacing it. The threat models these guardrails address are covered in detail in AI security.

AI risk management is the broader program of identifying, measuring, and mitigating the risks AI introduces, whether regulatory, security, operational, or reputational. The control plane is the execution layer for that program. Connect gives risk teams a real-time inventory of every AI agent and tool in use. Distribute turns policy on paper into rules enforced at runtime. Secure detects active threats. Observe produces the audit trail and metrics that risk and compliance teams need to evidence the program.

Guardrails and risk management are different layers of the same architecture, and the control plane is what lets one organization run both.

It is easy to conflate the two, but they sit at different levels. An AI governance framework describes what an organization should do; an AI control plane is what actually does it.

Most AI governance programs are built on published frameworks: NIST AI RMF, ISO/IEC 42001, the EU AI Act, and the internal policy decks that follow them. They tell an organization to identify AI use cases, classify risk, document controls, monitor outcomes, and review periodically. The framework is the program. For how the major frameworks compare and where each one stops, see AI security frameworks.

An AI control plane is the runtime that executes the program. The framework says "log every interaction with sensitive data"; the control plane is what intercepts the interaction and writes the log. The framework says "scope access by role"; the control plane is what evaluates the role on every request and decides what passes.

The two are complementary. A governance framework without a control plane is a document. A control plane without a framework is infrastructure with nothing telling it what to enforce. Tools that ship policy templates without runtime enforcement leave the gap between policy and behavior open, and the control plane is what closes it.

When the pieces are in place, the operational shape of the organization changes in a handful of specific ways.

New tools get added to the central registry once and are available to the teams entitled to use them. Teams stop standing up their own integrations in parallel, and new AI capabilities reach the right people in days rather than months.

An engineer using Cursor, a salesperson using ChatGPT Enterprise, and an analyst using an internal agent are all subject to the same permission model, because the permissions live in the control plane rather than in each agent.

Security has visibility into AI in the same way it has visibility into the network, the endpoints, and the cloud, enabling real-time incident detection and providing data for audits to draw from.

The AI control plane gives leadership actual numbers for tracking adoption by team, discerning which tools are producing outcomes and which are shelfware, and seeing where AI is moving the business. Leaders can base investment decisions on concrete metrics rather than anecdotes.

New AI initiatives ship on shared infrastructure instead of rebuilding the connection, identity, and governance layers every time. The second and tenth AI projects cost a fraction of the first.

The components for doing each of these things exist today. What the control plane provides is the architectural foundation that makes them work together.

Deploying a control plane has costs, and they shape whether the deployment succeeds.

Adding the control plane to every model call and tool call means adding a network hop. For interactive use cases, the overhead is typically small, but for agentic workflows that chain many calls in sequence, it compounds. The content inspection layer adds further delay if it runs synchronously.

The AI control plane category emerged in 2025. The vendors building in this space are early, and the architecture has not yet settled.

Mitigating lock-in means preferring open standards at the integration points. Use components that you can swap without rebuilding everything connected to them, such as OAuth 2.1 for identity, OpenTelemetry for observability, and MCP for tool connections.

Evaluating a control plane? Score any vendor, or your homegrown setup, against the full set of capabilities, deployment, and vendor criteria. Open the evaluation checklist.

Speakeasy is building the AI control plane. We started with the connection and identity layer, the first place companies typically get stuck when moving beyond ad-hoc AI adoption, and have been extending across the four functions since. The linked coverage map shows where we are today.

View the Speakeasy coverage map on Speakeasy.

The reference architecture in this guide reflects what a complete solution needs to look like. It is the architecture we are building toward. If you are an engineering or platform team figuring out how AI should flow through your organization, get in touch.

An AI control plane is the governing layer between every AI agent in an organization and every system they're allowed to reach. It unifies connection, identity, policy enforcement, and observability so that every prompt, response, and tool call flows through a single controlled path.

The four functions are Connect (bring every AI agent and system onto a single plane), Distribute (roll out access and enforce who can use what with executable policies and a full audit trail), Secure (inspect every prompt, response, and tool call in real time), and Observe (measure adoption, token use, and outcomes by team, tool, and user).

AI is being rolled out faster than IT can govern it, leaving organizations with no consistent way to enforce identity, scope permissions, inspect interactions, or measure adoption. The AI control plane resolves this tension by making governance a built-in property of the enablement layer itself.

AI governance is the program: the policies, frameworks such as NIST AI RMF and ISO/IEC 42001, and processes that define how AI should be used in an organization. An AI control plane is the runtime architecture that executes that program: it enforces governance policy at the point of use, produces the audit trail governance reviews depend on, and gives governance teams a real-time inventory of every AI agent and tool. The framework defines what should happen; the control plane is what makes it happen.

An AI gateway is a general term for a proxy layer between AI agents and the systems they reach, most commonly an LLM gateway that routes model calls or an MCP gateway that handles tool calls. An AI control plane is broader: it unifies LLM gateway, MCP gateway, identity, policy, and observability under a single architecture so the same identity, policy, and audit trail apply across model calls, tool calls, and the agents making them.

An LLM gateway sits between applications and language models, handling routing, API keys, and rate limits, but it only sees prompts and completions. An AI control plane is broader, covering connection, identity, policy enforcement, and observability across every AI agent and system, with the LLM gateway as just one component inside it.

An MCP gateway protects the tool-calling path by handling authentication, authorization, and inspection of tool calls in front of MCP servers. An AI control plane includes MCP gateway functionality as one component alongside identity, policy, observability, and the broader enablement story across all AI agents and systems.

MCP has become the emerging standard for how AI agents connect to tools and data. An AI control plane treats MCP servers as one of the surfaces it governs: provisioning them through per-team registries, authenticating clients through SSO-integrated identity, inspecting tool calls in real time, and producing a full audit trail. New MCP-based capabilities reach the right teams in days instead of months.

An AI control plane inspects every prompt, response, and tool call in real time, with active blocking of payloads matching known prompt injection patterns and passive detection of suspicious instructions embedded in tool outputs or retrieved documents. Because the control plane sees the full path from user identity through model call to tool call, it can correlate signals that would be invisible to any single gateway and route alerts into existing SIEM tooling.

An AI control plane gives risk teams the things they otherwise lack: a real-time inventory of every AI agent and tool in use, an audit trail of every prompt and tool call, executable policy enforced at the point of use rather than documented in a wiki, and integration with existing SIEM and GRC tooling. Together these turn AI risk management from a paper exercise into a measurable, continuously enforced control.

Three forces are converging: a top-down board mandate to deliver AI transformation, visible sprawl of unapproved AI tools across organizations, and concrete risk from data leaks, prompt injection, and regulatory pressure such as the EU AI Act.

Ownership typically falls to the executive accountable for AI delivery, most often a CTO, CIO, CISO, or newly appointed Chief AI Officer tasked with turning the board's AI mandate into an operating reality.

Most organizations should start by mapping the architecture, identifying which pieces they already have, and deciding which gaps to close with focused vendors versus building in-house. Committing to a foundation designed for the full lifecycle tends to be cheaper than assembling multiple tools that were not built to see each other.

Speakeasy is building the AI control plane, starting with the connection and identity layer and extending across all four functions: Connect, Distribute, Secure, and Observe.

An AI agent control plane is the same architecture applied to agents across all four working arrangements: the governing layer between every agent (Claude, ChatGPT, Cursor, Copilot, and internal or role-based agents) and the systems those agents can reach. It enforces identity, scoped policy, real-time inspection, and observability on each agent's prompts and tool calls, so a single misbehaving or compromised agent cannot exceed its authorized scope. The terms "AI agent control plane" and "control plane for AI agents" both refer to the AI control plane.

The data plane is where the work happens: the model inference, tool calls, and API requests an agent makes. The control plane is the decision layer above it, covering identity, policy enforcement, real-time inspection, and observability that determine what an agent is allowed to do before it acts and record what it did. The term comes from networking, where systems like Kubernetes and service meshes separate the control plane that governs the system from the data plane that carries the traffic.

A reference AI control plane sits between three layers. On the left are the callers (people and AI agents). In the middle is the control plane itself: identity and access, an LLM gateway lane, an MCP gateway lane, policy and threat enforcement, and observability and audit. On the right are the destinations (LLMs, SaaS apps, internal APIs, databases, and the data warehouse). Every prompt, response, and tool call flows through this single governed path, so the same identity, policy, and audit trail apply across both model calls and tool calls.

An orchestration platform such as LangChain, LlamaIndex, or CrewAI composes LLM calls and tools into workflows, governing how an agent does its work. An AI control plane sits above orchestration and governs whether the agent should be doing that work at all: who the user is, which tools and data are permitted, what is inspected in real time, and what is recorded for audit. Orchestration builds the workflow, and the control plane governs and audits it.

MLOps platforms such as SageMaker, Vertex AI, and MLflow manage the model lifecycle: training, versioning, deployment, and monitoring of models. An AI control plane governs runtime agent behavior, covering identity, policy, inspection, and observability across every agent and the systems it reaches, regardless of which model or provider is used. MLOps operates on the models themselves, while the control plane operates on the interactions agents have with the rest of the enterprise.

If AI agents in your organization take real actions (calling APIs, accessing company data, using credentials, or operating across teams), and especially if they do so without per-action human review, you need a control plane. Without one, there is no consistent way to know which tools are in use, scope permissions, inspect what is shared with external AI services, or prove what an agent did when security, audit, or a regulator asks.

An AI control tower or governance dashboard sits above the traffic: it displays inventories, risks, and status, and some are adding enforcement (ServiceNow's AI Control Tower can now revoke a rogue agent's permissions in real time). An AI control plane sits in the path of the traffic itself, enforcing policy on every prompt, response, and tool call at runtime. In practice, the control plane is what produces the real-time inventory and audit trail a control tower displays.

The landscape spans several layers: LLM gateways (Portkey, LiteLLM), MCP gateways and MCP security (Speakeasy, Runlayer, Mintive), identity and access (Okta, Microsoft Entra ID), observability (Langfuse, Arize, Galileo), and policy and threat detection (Lakera, Fiddler, Treza). Broad enterprise platforms are also extending into the category, including Salesforce's MuleSoft Agent Fabric and ServiceNow's AI Control Tower. No single category covers the full picture on its own. An AI control plane unifies these layers so identity, policy, and audit apply across both model calls and tool calls. Speakeasy is building the control plane starting from the connection and identity layer and extending across all four functions.

For interactive use, the overhead of the added network hop is typically small. It compounds in agentic workflows that chain many sequential calls, and synchronous content inspection adds delay on the critical path. Running inspection asynchronously reduces latency, at the cost of catching some traffic after the fact rather than blocking it in real time.

An agent framework helps you build an agent, but it does not govern it. Once agents take consequential, autonomous actions across the organization, enforcing identity and policy inside each agent's own code stops being defensible, because a bug or compromise bypasses it and leaves no tamper-evident record. A control plane moves that enforcement and audit outside the agent, so policy holds even when an individual agent does not.