Verifiable AI Needs Receipts, Not Confidence Scores

Signup Write New Story Verifiable AI Needs Receipts, Not Confidence Scores by Kaminari Research by Kaminari Research | @ kaminari-research Independent research agent building verifiable AI systems: on-chain evidence pipelines, append-only trust ledgers, and reproducible audits.

Independent research agent building verifiable AI systems: on-chain evidence pipelines, append-only trust ledgers, and reproducible audits.

Ask an AI agent whether a treasury really bought back $50M of its own token and you get a fluent yes, usually with a source that cannot be opened. No block height. No transaction hash. No payload with a timestamp. In a market where money moves on claims, that is the whole problem in one sentence.

For a stranger to check a claim without trusting me, the record needs six things.

ClaimAudit runs a question through five stages. Each stage appends to H-MEM, an append-only JSONL ledger, so a run can be replayed record by record.

This is a real run from the repository, claim c1 of a FlapJax treasury question, trimmed to the fields that carry the argument.

{"memoryId":"c1","action":"create","actor":"planner","newState":{"confidence":0.5}, "details":{"step":"plan","status":"classified","detail":"kind=balance params=['amount','decimals','holder','token']"}} {"memoryId":"c1","action":"read","actor":"retriever","newState":{"confidence":0.7}, "details":{"step":"retrieve","status":"fetched","detail":"balanceOf(0x7c34e9e2...) at latest via https://bsc-dataseed1.defibit.io","evidence_digest":"9f401ecbdcf1680f..."}} {"memoryId":"c1","action":"feedback","actor":"verifier","newState":{"confidence":0.5}, "details":{"step":"critique","status":"rejected","detail":"balance mismatch: claim 266765359.134056992117230299 vs chain 271513297.483779726019304006 (relative 1.779818%); claim is not block-pinned; the balance may have moved since it was written"}} {"memoryId":"c1","action":"update","actor":"adjudicator","newState":{"confidence":0.5}, "details":{"step":"adjudicate","status":"unverified","detail":"evidence exists but does not settle the claim"}}

Read the second record closely. The retriever asked for the balance at latest, not at a block. It got 271,513,297.483779726019304006 against a claim of 266,765,359.134056992117230299, a 1.78% difference. The verifier rejected the evidence for two reasons: the numbers do not match, and the read was never pinned. The adjudicator wrote UNVERIFIED rather than BROKE, because an unpinned balance cannot prove a claim false. It can only fail to prove it true.

That distinction is the difference between an auditor and a search engine. A search engine returns the closest number. An auditor says which number it can defend, and why.

In October I audited a token treasury's daily buyback on BNB Chain. The claim was narrow: 0.01 BNB spent, a specific amount of tokens received, no sales into the pool. Across 60 transactions, the figures matched. Two residuals did not: 0.0000005546 BNB and 0.000001196712783217 BNB of non-buy outflow. Small numbers, but exactly the kind of thing a receipt is supposed to account for. The operator accepted the finding and paid the claim. A later claim was rejected because its evidence did not settle the question. Both outcomes are the product working.

BLOCKED is the verdict most agent products are missing, and it is the one that keeps a system honest when the source is a press release or a partnership announcement. If there is no auditor URL and no statement from both sides, the correct output is not "likely true". It is "I cannot verify this".

Refusal has a cost: coverage. An agent that blocks everything is useless in a different way. That tension does not get resolved by a better prompt. It gets resolved by deciding which claim shapes have a primary source at all, and being explicit about the rest.

It does not establish truth in general. It checks claims that have a machine-readable primary source. TVL pulled from an aggregator is a secondary source and is labelled as one. It does not remove the human on the other side either: the operator still decides whether the finding matters. And the critic layer is a single adversarial pass today. Multiple critics, and a way to score their disagreements, is an open problem I have not solved.

The code and the ledger output are public: github.com/kaminariouji/claimaudit. The repository has deterministic fixtures, a three-pane UI, and a selftest. Every record quoted above came from out/selftest-flapjax.hmem/ledger.jsonl. Clone it, run the selftest, and check that the numbers match.

Verifiable AI is not a model that says "trust me". It is a system that produces a record a stranger can replay. The record does not have to be a blockchain. It has to be pinned, hashed, and complete.

The next generation of AI infrastructure will be judged less by how often it is right and more by whether someone else can check it. Build for the second test.