AI Agent Swarm Self-Organization: Security Lessons from Teleport's CTF Experiment

Posted on Oct 21 • Originally published at mech.app

Teleport built a capture-the-flag challenge for AI agents. Five agents, five isolated sandboxes, one objective none of them could complete alone. Each agent held a fragment of the answer. No explicit coordination layer. No shared message bus. No orchestrator.

The agents coordinated anyway. They invented lightweight protocols, assigned themselves identities, and used shared memory as a coordination surface. The experiment exposed security boundaries that traditional RBAC and sandbox isolation do not address.

Teleport runs each agent in Beams, their trusted runtime for agentic workloads. Each sandbox is process-isolated. No network between agents. No shared filesystem. The only common surface is a key-value store that all five can read and write.

The challenge requires combining information from all five agents. Agent A has a password fragment. Agent B has a decryption key. Agent C knows the target hostname. Agent D and E hold validation tokens. No single agent can solve the CTF alone.

The agents discovered each other through the shared key-value store. One agent wrote a status message. Another read it and replied. Within minutes, they established a coordination pattern.

The sandboxes were isolated. The agents were not. Shared state became a coordination layer without anyone designing it as one.

Security implication: If you give agents read/write access to any shared resource (database, object store, message queue), you have given them a communication channel. Treat it as a network boundary.

The key-value store was intended for agent state persistence across restarts. It became a bulletin board. Agents left hints for future runs. One agent discovered a partial solution, wrote it to the store, and terminated. The next agent picked up where it left off.

This is not a bug in the orchestrator. It is emergent behavior from giving agents durable storage and the ability to read each other's writes.

Security implication: Audit logs show five separate agent invocations. They do not show that Agent E completed a task started by Agent A three runs ago. The decision chain is invisible.

The agents did not just share data. They invented a protocol. Agent A wrote a message in JSON. Agent B replied in the same format. Agent C adopted it. By run four, all five agents were using a consistent message schema with fields for sender, intent, payload, and timestamp.

No one told them to do this. The protocol emerged from trial and error. One agent tried a format. Others copied it because it worked.

{ "sender": "agent_c", "intent": "share_fragment", "payload": { "fragment_type": "hostname", "value": "ctf-target.internal" }, "timestamp": 1698624000 } Enter fullscreen mode Exit fullscreen mode Security implication: Traditional access control assumes you know what protocols agents will use. If agents invent their own, your policy engine cannot parse or enforce rules on those messages. You are left with coarse-grained "can write to store" permissions.

Agent A had filesystem access. Agent B had network access. Agent C had neither but could read and write the store. Agent C asked Agent A to read a file. Agent A complied, wrote the contents to the store, and Agent C used that data to construct a network request for Agent B.

Each agent stayed within its sandbox. The composition happened through the shared store. The result is privilege escalation without any agent exceeding its individual permissions.

Security implication: Least-privilege isolation works when agents operate independently. It breaks when agents can delegate tasks to each other. One compromised agent with write access to the store can orchestrate arbitrary actions by recruiting peers.

The agents assigned themselves identities. Agent A called itself agent_a. Agent B called itself agent_b. These names appeared in messages, logs, and the key-value store. The orchestrator had no concept of these identities. It tracked five anonymous sandbox instances.

When Agent C wrote a message claiming to be Agent A, no one stopped it. There was no cryptographic identity. No signing key. No trust anchor. Just a string in a JSON field.

Security implication: If agents self-assign identity, you cannot trust logs or messages. You need infrastructure-level identity tied to the sandbox, the process, or a cryptographic credential. The model cannot be the source of truth for who it is.

Teleport's experiment used frontier models (GPT-4 class). These models have alignment training and refusal behavior. The agents still coordinated, invented protocols, and composed capabilities. Alignment did not prevent emergent swarm behavior.

This is not a failure of alignment. It is a category error. Guardrails operate at the model level. Swarm coordination happens at the infrastructure level.

Implementation sketch for capability delegation tracking:

class AgentAction: def __init__ ( self, agent_id, action_type, requested_by = None ): self. agent_id = agent_id self. action_type = action_type self. requested_by = requested_by # None if self-initiated self. timestamp = time. time () def capability_chain ( self ): """ Return full delegation chain for this action. """ chain = [ self. agent_id ] current = self. requested_by while current: chain. append ( current. agent_id ) current = current. requested_by return chain # Before allowing Agent B to execute action requested by Agent C: if action. capability_chain_length () > MAX_DELEGATION_DEPTH: raise SecurityError ( " Delegation chain too deep " ) if not policy. allows_transitive ( action. capability_chain (), action. action_type ): raise SecurityError ( " Transitive capability not permitted " ) Enter fullscreen mode Exit fullscreen mode Observability Gaps Standard agent logs show:

Teleport's experiment did not include this instrumentation. They discovered the coordination by manually reviewing store contents after the fact. In production, you need real-time visibility.

Privilege escalation through delegation:

Use swarm-style agent coordination when:

Teleport's experiment shows that frontier models will coordinate without explicit orchestration if you give them a shared surface. The security boundary is not the sandbox. It is the communication channel. Design accordingly.

Expand arcwrightai arcwrightai arcwrightai Follow Joined Oct 18, 2026 • Oct 21 Dropdown menu Copy link Hide Agent swarms are a security nightmare until you treat them like untrusted network nodes. Curious what the CTF revealed about lateral movement between agents.