How to Shadow-Test an AI Agent Without Letting It Act Twice

Signup Write New Story How to Shadow-Test an AI Agent Without Letting It Act Twice by Raju Dandigam by Raju Dandigam | @ rajudandigam Staff Software Engineer at Navan Staff Software Engineer at Navan, Docker Captain, Cypress Ambassador, and creator of AgentInspect, writing about production AI agents, TypeS

Staff Software Engineer at Navan, Docker Captain, Cypress Ambassador, and creator of AgentInspect, writing about production AI agents, TypeS

You have a new prompt, model, or orchestration policy. Offline tests look good, but the change will see vocabulary, timing, and context combinations that your fixture set never captured.

So you shadow-test it: send production traffic to both the current agent and the candidate, compare their decisions, and keep the current agent in control.

Then the candidate calls send_email too.

Now one user request has produced two real messages. The “safe” experiment has become a duplicate-action incident.

This synthetic failure happens when shadowing is implemented as “run the whole application twice.” For agents that can change the world, shadow testing must duplicate observation and decision—not authority and effects.

An action-taking agent contains at least four separable stages:

input -> decision proposal -> policy/approval -> tool execution -> outcome

The shadow should stop after the proposal and policy simulation:

+--> primary decision --> real policy --> execute sanitized input/context -| +--> shadow decision --> dry policy --> record only

This boundary is useful beyond experiments. It makes approvals testable, prevents a model response from directly becoming an external write, and gives you structured evidence to compare.

Instead of letting the model own an executable tool object, normalize its choice into a proposal:

type ActionProposal = { action: string; schemaVersion: number; args: Record<string, unknown>; reasonCode: string; confidence?: { value: number; source: "model" | "calibrator"; }; }; type PolicyDecision = { result: "allow" | "deny" | "require_approval"; policyVersion: string; reasonCodes: string[]; };

Do not collect hidden chain-of-thought. A bounded reason code such as invoice_overdue or user_requested_change is reviewable. If you store a short explanation, treat it as untrusted, redact it, and never require private reasoning from a model.

The executor accepts only a proposal that has passed real policy:

async function executeAuthorized( proposal: ActionProposal, decision: PolicyDecision, context: ExecutionContext, ) { if (context.mode!== "primary") { throw new Error("Effects are disabled outside primary mode"); } if (decision.result!== "allow") { throw new Error(`Action is not authorized: ${decision.result}`); } return toolRegistry.execute(proposal, context); }

That mode check is defense in depth. The stronger control is that the shadow environment never receives write-capable credentials.

A dryRun: true flag passed into a real tool is fragile. One tool may ignore it, a refactor may omit it, or a newly registered tool may default to execution.

type ToolDescriptor = { name: string; inputSchema: object; sideEffect: "read" | "write"; }; const shadowTools = realTools.map(({ name, inputSchema, sideEffect }) => ({ name, inputSchema, sideEffect, } satisfies ToolDescriptor));

The candidate sees the same tool names and schemas needed to make a decision, but the objects have no execute method. If it proposes send_email, the harness records the arguments and predicted policy result.

At the infrastructure layer, use separate identities and network policy:

Application checks can fail. Missing credentials fail closed.

Copying an entire production prompt into an experiment can expose personal, contractual, or security-sensitive data. Define a context projection:

type ShadowContext = { requestClass: string; locale: string; accountTier: string; recentEventCodes: string[]; policyFacts: Record<string, boolean | number | string>; }; function toShadowContext(input: LiveContext): ShadowContext { return { requestClass: input.requestClass, locale: input.locale, accountTier: input.accountTier, recentEventCodes: input.events.map(event => event.code).slice(-10), policyFacts: selectApprovedPolicyFacts(input), }; }

Run this projection through privacy and threat review. The candidate often needs less context than the primary runtime accumulated.

Two good agents can explain the same decision differently. Text similarity therefore creates noisy promotion gates.

type DecisionDiff = { sameAction: boolean; changedArgumentPaths: string[]; primaryPolicy: PolicyDecision["result"]; shadowPolicy: PolicyDecision["result"]; riskClass: "equivalent" | "benign_change" | "review" | "unsafe"; };

The highest-signal mismatches are usually:

Google's Agents CLI evaluation guide reflects this broader view of agent quality: it separates tool-use quality, trajectory quality, task success, final-response quality, hallucination, grounding, and safety. A shadow score should likewise be a vector, not a single “similarity” percentage.

The shadow can run the current policy engine in a non-authoritative mode. That reveals whether the candidate's proposal would be allowed.

const simulated = await policy.evaluate({ proposal: shadowProposal, principal: redactedPrincipalFacts, mode: "simulate", policyVersion: productionPolicyVersion, });

Simulation must not mint approval tokens, reserve inventory, mutate rate-limit counters, or write audit decisions into the authoritative ledger. If the policy system cannot evaluate without side effects, build a pure policy core and keep recording outside it.

Also record the policy version. A candidate comparison is uninterpretable if primary and shadow were evaluated under different rules without saying so.

It is easy to shadow only the simple requests that survive a convenient filter. The resulting dashboard says the candidate is safe because risky traffic never entered the experiment.

type ShadowEligibility = { eligible: boolean; reason: | "selected" | "privacy_excluded" | "unsupported_tool" | "budget_excluded" | "control_group"; };

Report both the evaluated population and exclusions. Stratify results by action type, risk level, language, account segment, workflow length, and whether the primary required recovery. Otherwise a large aggregate can hide the exact slice that matters.

One primary output versus one candidate output does not tell you the candidate's variance. For a bounded sample, run multiple candidate trials with fixed configuration and preserve the random seed where the provider supports it.

Do not mistake a seed for full determinism. Provider updates, retrieval changes, timestamps, tool data, and concurrency can still change behavior.

action agreement distribution forbidden-action proposal rate approval-bypass proposal rate argument-diff severity p50/p95 decision latency token and cost distribution variance across repeated candidate runs coverage and exclusion rate

Never invent thresholds because they sound mature. Set them from product risk and baseline behavior, then document the decision.

Shadowing is one stage, not a release strategy by itself.

Promotion should consider downstream outcomes, not only proposal agreement. The current agent may be consistently wrong. A candidate that disagrees could be an improvement, so route meaningful disagreements to labeled review rather than automatically treating the primary as truth.

A reviewable record needs enough context to reproduce the comparison without becoming a privacy archive:

type ShadowRecord = { experimentId: string; requestFingerprint: string; cohort: string; primary: { proposal: ActionProposal; policy: PolicyDecision }; candidate: { proposal: ActionProposal; policy: PolicyDecision }; diff: DecisionDiff; primaryVersion: string; candidateVersion: string; policyVersion: string; evaluatedAt: string; };

Store raw prompts and outputs only when authorized and necessary. Prefer schema-approved fields, stable fingerprints, and short retention. Observability should reduce uncertainty without expanding the blast radius of sensitive data.

I would encode this as an automated contract:

For every shadow run: external_write_count == 0 authoritative_approval_count == 0 production_message_publish_count == 0

If the telemetry needed to verify those invariants is missing, the experiment should fail closed or pause. “We did not see a write” is weaker than “the shadow identity could not write.”

The point of shadow testing is to learn from realistic inputs before granting realistic authority. Duplicate the decision surface. Keep the effect surface singular.

Using Cypress Cloud CLI and Test Replay for AI-Assisted CI Debugging