I Built an AI That Checks Whether Digital Products Deliver What They Promise

Posted on Oct 13 • Originally published at pkazantsev.com

Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle Submission 💸

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle.

A digital-product listing can promise “48 pages”, “A4 and US Letter editions”, and “debt repayment worksheets.” Then the customer opens the ZIP and finds something else.

For a small seller, that turns into refunds, support messages, rough reviews, and the awful feeling that you missed something obvious. PromiseDiff is a local-first pre-publish check for Etsy, Gumroad, template, and educational-PDF sellers.

Paste in the listing. Upload the actual PDF or ZIP. PromiseDiff shows what the files can back up, what they contradict, and what they simply cannot prove.

The rule I cared about most: missing text is not a failure. A measured mismatch is a contradiction; a claim the files cannot prove is just unverified.

Every claim gets one of four plain outcomes:

Every result keeps the method and the evidence that led to it.

For example, a page-count claim is deterministic:

Listing: “The main PDF contains exactly 48 pages.” File: budget_planner_A4.pdf → 48 pages Result: SUPPORTED — DETERMINISTIC Enter fullscreen mode Exit fullscreen mode “Includes debt repayment worksheets” is different. A PDF might call that page a “Loan Balance Reduction Tracker.” Here Gemma looks at the extracted text, but PromiseDiff only accepts the result after checking the suggested filename, page, and quote against the actual PDF.

The demo uses a real 48-page A4 Budget Planner PDF. Nothing is mocked.

I used ProgressCut to cut the screen recording down to a short walkthrough.

The main PDF contains exactly 48 pages. Includes both A4 and US Letter PDF editions. Contains monthly budget review worksheets. Includes savings goal trackers. Enter fullscreen mode Exit fullscreen mode The upload only has the A4 edition. PromiseDiff confirms the page count, flags the missing US Letter version, and lets the seller open the evidence or export the report as Markdown.

Some questions have a simple answer; others need interpretation.

Asking a language model to count pages is slower and worse than reading the PDF metadata. But keyword search alone will not reliably connect “debt repayment” and “loan balance reduction.” So each tool only gets the jobs it can support with evidence.

missing evidence!= contradiction Enter fullscreen mode Exit fullscreen mode

If the document is scanned, the claim scope is ambiguous, a citation points to a nonexistent page, or a quote does not occur in extracted text, the answer is UNVERIFIED.

It is a small React + FastAPI app. The verification-flow diagram above shows how the pieces connect.

Uploads are untrusted, so the backend puts tight boundaries around them: size, PDF count, total pages, extracted text, ZIP ratio, file types, and expanded ZIP size. It rejects traversal paths, duplicate entries, encrypted PDFs, nested archives, and corrupt files. The original upload is never saved.

The model work is deliberately kept small:

Many sellers are working on unpublished material: course notes, templates, client downloads, and draft listings. With the local setup, Ollama keeps the listing text and extracted PDF text on the seller’s machine.

Open-weight Gemma also makes this affordable for a small operation. A local run has no per-audit third-party API bill, although someone still pays for the hardware or hosting. Sellers can repeat the same check before every release without handing their draft material to another service.

I made ten independently labelled synthetic PDF cases and ran them through the same local API the UI uses. The runner saves the raw responses and timings in benchmark/results.json.

exact-label accuracy = correct labels / total cases = 9 / 10 = 90% Enter fullscreen mode Exit fullscreen mode

xychart-beta title "Exact classification by category (10 synthetic cases)" x-axis [Supported, Contradicted, Unverified, OutOfScope] y-axis "Correct cases" 0 --> 4 bar [3, 2, 3, 1] Metric Result Caveat Exact-label accuracy 90% (9/10) Tiny synthetic evaluation, not a generalization claim. Contradiction precision 100% (2/2) Both measured failures were page/dimension facts. Contradiction recall 100% (2/2) Same small set.

Supported recall 75% (3/4) One semantic paraphrase was conservatively left unverified. Median API latency 2.11 s Local Gemma on this machine. Mean API latency 3.50 s Prompt-injection case took 13.81 s.

The miss is the useful bit. The listing said debt repayment worksheet and the PDF said Loan Balance Reduction Tracker. The model did not have strong enough evidence to connect them, so PromiseDiff returned UNVERIFIED instead of pretending it knew.

That is a limitation to fix, not a result to hide.

The repository has the local setup, real example PDF, test commands, security notes, and benchmark runner.