Verifiable AI: Proving What Happened Behind the Black Box

Signup Write New Story Verifiable AI: Proving What Happened Behind the Black Box by vishwa vimukthi gammuduwaththage by vishwa vimukthi gammuduwaththage | @ vishwagw Founder at Falcon Eye AI INC AI researcher & Entrepreneur, focusing on computer vision, autonomous unmanned systems and Artificial intelligence

AI researcher & Entrepreneur, focusing on computer vision, autonomous unmanned systems and Artificial intelligence

when a server tells you which model answered, how would you know?

Today you would not. An API response has a model field, and that field is a string the provider typed. You cannot see the weights, the quantization, the system prompt, or whether your request was cached, rerouted, or trimmed.

This is more than paranoia. Researchers at UC Berkeley tested it in Are You Getting What You Pay For?, and the results are uncomfortable:

So a provider that wanted to serve a cheaper model some of the time would probably get away with it. I am not saying providers do this. I am saying the customer has no way to check, and that is a weak position for anyone making decisions that matter.

There are three serious approaches. They rest on different kinds of trust: mathematics, economics, and hardware.

A zero-knowledge proof lets one party prove a statement is true without revealing why. For machine learning the statement is: "I ran the model with this fingerprint on this input and got this output."

This is what makes the "verify without revealing proprietary weights" promise real. The verifier learns that the committed model ran. It learns nothing about the model itself.

The catch is cost. A March 2026 benchmark of five zkML libraries used a small 3.4-million-parameter network. Proof times ran from about 6 seconds to 41 seconds per inference. A network that size runs natively in well under a millisecond. The same author estimates a 100-million-parameter model would take roughly three hours per proof, and concludes that general transformer inference is not shippable this way yet.

There has been progress. Lagrange claimed in 2025 that its DeepProve system generates proofs 54 to 158 times faster than the popular EZKL library on small networks, and a full proof of GPT-2 inference has been reported since. GPT-2 is a 2019 model about a thousand times smaller than what production APIs serve.

One more detail matters. Circuits use fixed-point integers, so the model being proven is a quantized version of the one you trained. You are proving a close relative of your model.

Optimistic machine learning borrows from optimistic rollups. The opML paper describes it like this:

Because nothing is proven unless there is a dispute, the overhead is small. The authors claim a 7-billion-parameter LLaMA model can be handled on a standard PC with 32 GB of memory.

The tradeoffs are stated plainly in the paper itself:

That second point rules opML out for the proprietary-weights use case. It suits open models where the concern is honest execution.

A trusted execution environment is a sealed region of a processor. It can produce a signed report listing the hash of the code and model loaded inside it. The user checks the signature against the chip maker's keys.

The Berkeley team measured this on a Llama-3-8B model. Throughput dropped by 3 to 15 percent and first-token latency rose by 9 to 16 percent, depending on load. Compare that with the thousands-fold slowdown of zkML.

The weakness is the trust root. You are relying on the hardware vendor and on the enclave having no exploitable flaw, and enclaves have a long history of side-channel attacks.

Here is the part of the story I find most convincing, and it is about small models.

Many high-stakes automated decisions are not made by giant language models. Credit scoring, fraud detection, insurance pricing, and a lot of diagnostic classifiers use compact networks or tree models. Those are within reach of zkML now.

Consider how the technique would apply in three settings. These are illustrations, not reports of live systems.

Lending. A bank's scoring model is a trade secret, and regulators want assurance that every applicant was scored by the same approved model. The bank commits to the audited model once. Each decision carries a proof that this commitment produced it. The regulator can verify every decision without ever holding the weights.

Medical. A diagnostic model is cleared by a regulator as a specific version. A hospital using a hosted copy wants to know it is that version and not an untested update. A proof tied to the cleared version's commitment answers that.

Legal. When an automated decision is disputed in court, the question becomes which system produced it. A proof made at the time of the decision is far stronger evidence than a log file the defendant controls.

One caveat is easy to miss. "Zero-knowledge" protects the prover's secrets from the verifier. It does not hide your input from the provider. If a hospital sends patient data to a cloud model, the cloud still sees it. Keeping the input private from the provider needs a different tool, such as running the model inside a TEE or proving on the client side.

[YOUR EXPERIENCE: if you have had to show a client or regulator which model version produced a result, a short paragraph here would ground this section.]

A single proof covers a single inference. An audit trail needs a history that nobody can edit afterwards.

The construction is simple. For each decision, record:

Then chain the records so each one includes the hash of the one before it, and periodically publish the latest hash somewhere public. This is the same append-only log I described in part two for tracking model versions. Changing any past record breaks every hash after it.

Regulation is moving in this direction. Article 12 of the EU AI Act says high-risk systems must automatically record events over their lifetime, with logs kept for at least six months. The text does not use the words "tamper-proof." But a log that its owner can quietly edit is weak evidence, which is why practitioners treat tamper resistance as implied.

There is time to prepare. The high-risk obligations were due on August 2, 2026, and have reportedly been pushed to December 2, 2027 under the Digital Omnibus on AI.

Note that the hash chain gives you most of this without any zero-knowledge machinery. A signed, chained log proves the record was not changed. What the proof adds is evidence that the record was true when it was written.

A proof shows execution, not correctness. zkML can prove that model X produced output Y. It says nothing about whether X is accurate, biased, or safe. A flawed model with a perfect proof is still flawed. The commitment only means something if an independent party audited the model behind it.

The altered-prompt problem is only half solved. A proof covers the input the provider commits to. If the provider adds a hidden system prompt, that text must be part of the committed input, or the user must see its hash. Otherwise the proof is about a different question from the one you asked.

The cost gap is enormous. Thousands of times slower is not an optimization problem you fix in a quarter. For large models, zkML may stay impractical for years.

opML depends on someone watching. If checking is costly and cheating is rare, rational watchers stop checking. This is known as the verifier's dilemma, and the opML authors acknowledge it.

TEEs may simply be good enough. At single-digit overhead with privacy for the weights, attestation covers most commercial needs. The purist argument against trusting hardware matters less to a bank than to a blockchain.

The big providers have little reason to adopt any of this. Proofs add cost and expose substitutions. Without regulation or customer pressure, closed API providers are unlikely to volunteer.

Contracts and audits already exist. Enterprises handle this problem today with service agreements, certifications, and the right to audit. Those are slower and less precise than cryptography, but they are understood by courts.

I started this piece expecting zkML to be the headline. After reading the benchmarks, I see it as one tool among three, each fitting a different job:

As a builder, the shift I care about is in the default. Right now the burden is on the user to trust the provider. Verifiable inference moves that burden to the provider, who has to show its work. Even partial adoption changes what customers can ask for.

Immutable Architecture: Why the Future of AI Models Demands Permanent Storage