How Do You Threat-Model an Application That Uses Autonomous AI Agents?

Posted on Oct 14 • Originally published at particle41.com

You spent years building a threat model you trust. You know your trust boundaries, you've mapped your data flows, and your team can recite STRIDE in their sleep. Then someone shipped an autonomous AI agent into the product, and quietly, every assumption underneath that model cracked.

Here's the problem in one sentence: traditional software does what you programmed it to do, but an autonomous agent does what it is convinced to do. Those are not the same risk. An agent that reads customer emails, queries your database, calls third-party APIs, and writes to your file system is not a function you can fully specify. It is a decision-maker operating on untrusted input, and the input arrives in the same channel as its instructions.

That last point is the whole game. For an LLM-driven agent, instructions and data are both just text. There is no hardware boundary, no type system, no parser separating "this is a command" from "this is a value." OWASP names this as the top risk for LLM applications: a prompt injection vulnerability occurs when user prompts alter the model's behavior in unintended ways, because the model cannot reliably tell trusted instructions from untrusted content. Which means any untrusted text your agent ingests is potentially a command. Re-draw your threat model around that reality, and most of the new risk becomes legible.

We've been here before. SQL injection happened because we concatenated untrusted input into a command string. Prompt injection happens for the exact same structural reason: untrusted text lands in the same context window as your system instructions, and the model can't reliably tell them apart.

The forms it takes ( OWASP distinguishes the first two ):

You cannot fully prevent prompt injection today. There is no parameterize() for natural language; OWASP itself notes it is "unclear if there are fool-proof methods of prevention" given the stochastic nature of the models. So the threat model must assume injection will sometimes succeed, and constrain what a compromised agent can actually do. That shifts the entire focus from "stop the bad input" to "limit the blast radius." Defense lives at the tool layer, not the prompt layer.

You don't need a new framework. STRIDE, the six-category model (Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, Elevation of privilege) developed by Praerit Garg and Loren Kohnfelder at Microsoft, adapts cleanly once you treat the agent, its tools, and its memory as first-class assets with their own trust boundaries.

Walk each agent through those six lenses. The gaps show up fast.

An agent on its own can only produce text. The danger begins when that text drives a tool: a database query, an HTTP request, a shell command, a payment. The tools are the actuators, and the actuators are where you enforce security.

This is where most agentic applications are dangerously over-provisioned. A team building a customer-support agent gives it read-write database access because scoping felt like premature optimization during the prototype. Now a successful prompt injection means an attacker can read or modify any record. Every tool you grant an agent is a capability an attacker inherits the moment they win control of the prompt.

Threat-model each tool as if it will eventually be invoked by an adversary:

Information disclosure deserves its own pass because the exit routes are not obvious. An agent doesn't need a database dump tool to leak data. It needs any channel that reaches the outside.

Map the egress paths. If your agent can fetch a URL, an injected instruction can encode stolen data into that URL's query string and send it to an attacker's server. If your agent renders markdown that the user's browser loads, an image tag pointing at attacker.com/log?data=... exfiltrates on render. If your agent can send email or post to a webhook, those are exfil channels too.

The controls are concrete. Allowlist outbound domains for any fetch or webhook tool. Strip or sandbox rendered links and images in agent output. Classify the data each tool can return, and never let a tool that reads sensitive data share a context with a tool that can transmit to untrusted destinations. The principle: an agent that can read secret data and reach the open internet in the same session is one injection away from a breach.

Agents accumulate state. Conversation history, scratchpad memory, and shared vector stores all create cross-contamination risk. If your agent stores user A's data in a memory store that user B's session can retrieve, you have a tenancy leak that no firewall will catch.

Treat agent memory as a data store with the same isolation requirements as your database. Scope retrieval to the authenticated user. Don't let one tenant's documents poison a shared embedding index that another tenant queries. And critically, the agent should act with the user's permissions, not its own elevated service identity. An agent answering for a basic-tier user should not be able to reach admin-tier data just because the underlying service account can. OWASP's mitigation is to execute extensions in the user's context, tracking user authorization so actions run on downstream systems with that specific user's minimum privileges.

Threat-modeling agents isn't academic. Run this pass on every agent in production:

At Particle41, when we build agentic systems we threat-model the agent layer as its own trust boundary from day one, scoping tools tightly and treating all model output as untrusted by default. It's the same discipline our senior engineers apply to any privileged service, just pointed at a new class of actor. The teams that get burned are the ones who shipped the agent first and asked these questions after. Ask them now, while the answers are still cheap.