SynapseHealer: Building an Autonomous, Self-Healing Interoperability DLQ with InterSystems IRIS AI Hub

Built for the InterSystems Programming Contest: AI Agents in Interoperability Workflows (2026)

In healthcare and supply-chain integrations, the dead-letter queue is where transactions go to die. A vendor renames client_uuid to customer_id with no warning. An EHR bridge emits 1985-02-31, an impossible calendar day. An HL7 ADT^A01 arrives with PV1 before PID. A lab note contains an unescaped pipe. The integration engine rejects the message, and it sits in the queue until an engineer opens a hex dump, decodes the schema violation, writes a one-off script, and resubmits — days later, if at all.

SynapseHealer replaces that manual bottleneck with an autonomous pipeline on InterSystems IRIS AI Hub: a custom inbound adapter, native vector search over repair runbooks, a governed %AI.Agent, and a tri-branch router that auto-heals, escalates to the InterSystems Workflow Engine, or quarantines.

Synapse.Process.DeadLetterHealerBP is the arbitrator. OnRequest takes a Synapse.Msg.DeadLetterRequest from ChaosIngestService, and ProcessDeadLetter picks the execution mode: live when SYNAPSE_MODE=live and an API key exists, otherwise replay — a deterministic mode that needs zero API keys.

In live mode it instantiates Synapse.Agent.DeadLetterHealerAgent, an %AI.Agent bound to Synapse.Agent.HealerToolSet, sends it the payload ID, source system, format, validation error, and raw body, strips any markdown fences, and parses the JSON contract: decisionBranch, confidenceScore, riskLevel, matchedPatternId, healedPayload, reasoning.

The agent cannot improvise. Its toolset — Synapse.Agent.HealerToolSet ( %AI.ToolSet ) wrapping Synapse.Agent.HealerTools ( %AI.Tool ) — exposes exactly three governed calls: SearchRunbookRAG, ValidateRepairedPayload, and CalculateRiskScore. It looks up the nearest verified runbook in the 384-dimensional vector index ( VECTOR_COSINE, sub-45ms), applies the deterministic transformation, and must pass validation before anything is resubmitted.

Credentials never appear in plaintext logs. Synapse.Config.AIHubSetup.StoreWalletSecret writes the API key into the IRIS Wallet ( %Wallet.KeyValue, collection SynapseSecrets, key SynapseSecrets.LLMKey ) at startup, and BuildProviderSettings references it through @{env:LLM_API_KEY} indirection instead of embedding the literal key. The same setup registers %ConfigStore and Ens.Config.Credentials when those classes exist, with graceful fallback on builds that lack them.

We default to gemini-1.5-flash ( gpt-4o-mini is the OpenAI alternative). Heal decisions are structural, not creative, so a fast, cheap model is the right choice.

Every inference lands in Synapse.Data.AuditTelemetry, a persistent table storing prompt tokens, completion tokens, estimated USD cost, latency, the full tool-call trace, decision branch, risk level, and workflow status. Cost is computed from the counts — prompt × 0.00000015 + completion × 0.0000006 — and written to the table in real time; the Next.js dashboard reads /api/synapse/status and shows the cumulative USD counter and auto-heal rate live.

Across our 10 regression suites: 479 prompt tokens, 176 completion tokens, $0.00018 per heal, ~145 ms end to end, 90% auto-heal rate across the seven canonical chaos scenarios. Replay mode consumes no tokens, and the telemetry table is pre-seeded with golden traces for every scenario, so the dashboard is populated from the first second.

Each heal produces a governed three-step trace, stored in ToolTraceJSON:

The trace is surfaced live in the Next.js console: a step gauge per call, side-by-side diff of the corrupted and healed payloads, and the agent's reasoning. HUMAN_WORKFLOW messages land in the workflow queue where an operator Approves (→ ResubmitDeliveryOp ) or Rejects (→ QuarantineVaultOp ) with reviewer notes. Everything is queryable via /api/synapse/workflow/tasks and the persistent audit ledger.

We separated development-time agentic tooling from runtime governance. During development, we paired our IDE with the iris-agentic-dev MCP toolkit ( iris_symbols_local for local AST verification across 150+ symbols and objectscript-guardrails to enforce method-scoped Quit rules and $IsObject checks before container compilation). At runtime inside the container, governance is 100% native to IRIS AI Hub—powered by an %AI.ToolSet of %AI.Tool methods rather than an external sidecar.

The ObjectScript guardrails are the part I trust most:

For %AI syntax, the agent resolves its provider via AIHubSetup.GetProviderConfig and BuildProviderSettings, creates %AI.Provider, calls agent.CreateSession() then agent.Chat(session, prompt), and parses the result with %DynamicObject.%FromJSON. It compiled cleanly on the IRIS 2027.1 AI Hub EAP build.

Packaging is automated: module.xml declares <SystemRequirements Version=">=2026.3" Interoperability="enabled"/>, and SetupAll — run by IPM on install — executes Configure() (Wallet, config store, credentials), seeds the runbooks and replay telemetry, then starts Synapse.Production. zpm install synapse-healer or zpm load /opt/irisapp -v gives you a running production with a seeded dashboard. The module shipped with +11 contest bonus points across the custom adapter, embedded Python, vector search, workflow engine, and IPM integration.

# Prerequisite: Load the InterSystems AI Hub EAP tarball (Build 109 or 154) # docker image load -i iris-community-2027.1.0AI.109.0-docker.tar.gz git clone https://github.com/josephjbassey/synapse-healer.git cd synapse-healer docker compose up -d cd ui && npm install && npm run dev Enter fullscreen mode Exit fullscreen mode Open http://localhost:3000, press 1–7 to inject chaos payloads, and watch the pipeline heal them. No API key required; set SYNAPSE_MODE=live with an LLM_API_KEY for live AI Hub inference.

It's not magic. It's governed tool calls, deterministic validation, and a telemetry table that actually gets read.

Expand 𝘿𝙚𝙫 𝙎𝙪𝙥𝙥𝙤𝙧𝙩 𝙏𝙚𝙖𝙢 𝘿𝙚𝙫 𝙎𝙪𝙥𝙥𝙤𝙧𝙩 𝙏𝙚𝙖𝙢 𝘿𝙚𝙫 𝙎𝙪𝙥𝙥𝙤𝙧𝙩 𝙏𝙚𝙖𝙢 Follow Official account of the Dev support team Joined Nov 3, 2026 • Nov 3 Dropdown menu Copy link Hide Your account is in the pending status.

tr.ee/dev-to Enter fullscreen mode Exit fullscreen mode Like comment: Like comment: 1 like Like Comment button Reply Collapse Expand 𝘿𝙚𝙫 𝙎𝙪𝙥𝙥𝙤𝙧𝙩 𝙏𝙚𝙖𝙢 𝘿𝙚𝙫 𝙎𝙪𝙥𝙥𝙤𝙧𝙩 𝙏𝙚𝙖𝙢 𝘿𝙚𝙫 𝙎𝙪𝙥𝙥𝙤𝙧𝙩 𝙏𝙚𝙖𝙢 Follow Official account of the Dev support team Joined Nov 3, 2026 • Nov 3 Dropdown menu Copy link Hide hello user

Like comment: 1 like Like Comment button Reply Collapse Expand 𝘿𝙚𝙫 𝙎𝙪𝙥𝙥𝙤𝙧𝙩 𝙏𝙚𝙖𝙢 𝘿𝙚𝙫 𝙎𝙪𝙥𝙥𝙤𝙧𝙩 𝙏𝙚𝙖𝙢 𝘿𝙚𝙫 𝙎𝙪𝙥𝙥𝙤𝙧𝙩 𝙏𝙚𝙖𝙢 Follow Official account of the Dev support team Joined Nov 3, 2026 • Nov 3 Dropdown menu Copy link Hide Your account is under review.

Like comment: 1 like Like Comment button Reply Collapse Expand 𝘿𝙚𝙫 𝙎𝙪𝙥𝙥𝙤𝙧𝙩 𝙏𝙚𝙖𝙢 𝘿𝙚𝙫 𝙎𝙪𝙥𝙥𝙤𝙧𝙩 𝙏𝙚𝙖𝙢 𝘿𝙚𝙫 𝙎𝙪𝙥𝙥𝙤𝙧𝙩 𝙏𝙚𝙖𝙢 Follow Official account of the Dev support team Joined Nov 3, 2026 • Nov 3 Dropdown menu Copy link Hide Your account is in the pending status.__

Like comment: 1 like Like Comment button Reply Collapse Expand 𝘿𝙚𝙫 𝙎𝙪𝙥𝙥𝙤𝙧𝙩 𝙏𝙚𝙖𝙢 𝘿𝙚𝙫 𝙎𝙪𝙥𝙥𝙤𝙧𝙩 𝙏𝙚𝙖𝙢 𝘿𝙚𝙫 𝙎𝙪𝙥𝙥𝙤𝙧𝙩 𝙏𝙚𝙖𝙢 Follow Official account of the Dev support team Joined Nov 3, 2026 • Nov 3 Dropdown menu Copy link Hide Your account is in the pending status. Complete the process.

Like comment: 1 like Like Comment button Reply Some comments may only be visible to logged-in visitors. Sign in to view all comments.