Your AI Agent Charged the Customer Twice and Every Dashboard Stayed Green

An AI agent hit a timeout. It retried the payment API call. The first request had already succeeded.

Every monitoring dashboard stayed green.

This is a silent failure class that traditional SRE metrics miss completely. We ask "did the request return HTTP 200?" The question that matters for agents with write access is:

Did the agent execute the same irreversible action twice?

When an LLM agent operates under non-deterministic retries, status codes stop telling you the truth about what happened.

Normal retry logic is boring and predictable. Agents are not:

Retries stack. The HTTP client retries, the orchestration framework retries, and the model decides to "try again". Each layer assumes it is the only one. Retries get rephrased. After a timeout the agent may re-plan and re-issue the action with reordered or slightly different parameters, so no lower layer recognizes it as a duplicate. A timeout means "unknown", not "failed". The agent can't tell if the write landed, and it tends to guess "do it again." Three rules for critical write paths

Fingerprint the action before dispatching it:

key = hash(action_name + sorted_parameters + caller_scope)

If the payload matches, the agent cannot emit a duplicate mutation. Sort keys and normalize types so logically identical payloads hash the same. Include the caller scope (tenant, session, workflow run) so unrelated users don't collide, and an intent ID if the same action can legitimately repeat.

Block repeated signatures inside a bounded window (say, 300 seconds).

Don't swallow duplicates silently. That hides the bug behind them. Intercept the call, reject it with an explicit error, and increment a counter. "Sliding" means each blocked attempt refreshes the window, so an agent stuck in a loop stays blocked.

If DAR spikes, don't blame the agent. It is almost always compensating for unhandled timeouts, slow database locks or upstream lag. DAR tends to show you those problems before your customers do.

A minimal gate in Python python import hashlib import json import threading import time

class DuplicateActionError(Exception): """Raised when an identical action was already submitted inside the window."""

class DedupGate: def init (self, ttl_seconds: float = 300, clock=time.monotonic): self.ttl = ttl_seconds self._clock = clock self._seen: dict[str, float] = {} self._lock = threading.Lock() self.allowed = 0 self.blocked = 0

@staticmethod def action_key(action_name: str, params: dict, caller_scope: str) -> str: canonical = json.dumps( {"a": action_name, "p": params, "s": caller_scope}, sort_keys=True, separators=(",", ":"), default=str, ) return hashlib.sha256(canonical.encode()).hexdigest() def acquire(self, action_name: str, params: dict, caller_scope: str) -> str: key = self.action_key(action_name, params, caller_scope) now = self._clock() with self._lock: # evict expired entries self._seen = {k: t for k, t in self._seen.items() if now - t < self.ttl} if key in self._seen: self._seen[key] = now # sliding: refresh the window self.blocked += 1 raise DuplicateActionError( f"'{action_name}' already submitted within {self.ttl:.0f}s" ) self._seen[key] = now self.allowed += 1 return key def release(self, key: str) -> None: """Call only on a *confirmed* failure, so a legitimate retry can proceed.""" with self._lock: self._seen.pop(key, None) @property def dar(self) -> float: total = self.allowed + self.blocked return self.blocked / total if total else 0.0 Enter fullscreen mode Exit fullscreen mode Using it around a payment call:

python gate = DedupGate(ttl_seconds=300)

def charge(order_id: str, amount_cents: int, tenant: str): params = {"order_id": order_id, "amount_cents": amount_cents} key = gate.acquire("create_charge", params, caller_scope=tenant) try: # also pass key as the provider's idempotency key if it supports one return payments.create_charge(**params, idempotency_key=key) except PaymentDeclined: gate.release(key) # definitive failure: safe to retry raise # on a timeout: do NOT release. Reconcile by looking up the charge instead.

The agent gets a typed error it can reason about ("already submitted, check status") instead of looping.

Gotchas Ambiguous timeouts. Don't release the key on a timeout. Reconcile against the downstream system first. Release only on definitive failure. In-memory state doesn't scale out. This gate lives in one process. With multiple workers, back it with a shared atomic store, for example Redis SET key value NX EX 300, or a DynamoDB conditional write. Defense in depth. Forward the action key as the provider's idempotency key where available. The gate protects you from your agent, and the provider key protects you from everything else. Legitimate repeats. Two deliberate $10 transfers are not duplicates. Put an intent ID into the key. Where this lives

I integrated this deduplication gate into the open-source agentsre telemetry framework so DAR sits next to the other signals an agent emits: https://lnkd.in/e7Qpux2j

How is your team handling idempotency and duplicate execution as agents move from read-only assistants to production write systems? Enforced at the agent layer, the API layer, or both? I'd like to hear the war stories.

Expand DEV VERIFY 👇 DEV VERIFY 👇 DEV VERIFY 👇 Follow Joined Oct 17, 2026 • Oct 17 Dropdown menu Copy link Hide tr.ee/dev-to