Don't Let the LLM Decide: How I Built a Predictable AI Tutor

The easy way to build an AI tutor is to give the LLM everything. Pass it the question, the learner's attempts and the answer. Tell it: "Help the learner, but don't give the answer away too early."

That works in a demo. It fails in an experiment. And it fails quietly in production too.

In the first post of this series I explained the experiment. I built an SQL tutor in two versions, called arms. Arm A makes you try before it helps, then gives a hint before the answer. Arm B gives the answer whenever you ask, like most AI tools. Six people used one or the other, then took a test a week later with no tutor.

This post is about how I built it. The core rule fits on one line:

The LLM never decides whether to help, or how much. Plain code decides. The LLM only writes the hint's wording, after that decision is made.

Picture the prompt version. "If the learner seems stuck, give a hint. If they've really tried, give the answer."

Now try to answer one question: what did Arm A receive?

You can't. "Seems stuck" is a judgment the model makes fresh every time. Two learners in the same situation can get different help. The same learner can get different help on a retry. The thing you're trying to test becomes a moving target.

There's a second problem. With six people, there were only about 23 moments in the whole study where the tutor had to decide what help to give. You can't judge a model's decision-making from 23 examples. But you can fully test a plain function, and I did, with 123 unit tests.

So I split the system into three kinds of parts, a framing from my 100x Engineers AI course:

The model is one small component. It's not the product.

Here's the heart of policy.js, trimmed. No DOM, no network, no LLM. It takes the log for one question and returns an action.

const ACTION = { NONE: ' none ', // already solved REFUSE_GATE: ' refuse_gate ', // Arm A, no real attempt yet REFUSE_RETRY: ' refuse_retry ', // Arm A, hint given, keep trying HINT: ' hint ', // the LLM writes the text REVEAL: ' reveal ', // the reference query }; const DEFAULT_N = 2; // attempts with the hint before the answer function decide ( item ) { const N = item. N?? DEFAULT_N; const counted = countableAttempts ( item. attempts ); const helpServed = item. helpServed || []; if ( item. attempts. some ( a => a. outcome === ' correct ' )) return { action: ACTION.

NONE }; // Arm B: the entire difference between the two groups is this line. if ( item. arm === ' B ' ) return { action: ACTION. REVEAL }; // Arm A: the gate. No help of any kind until one attempt counts. if ( counted. length === 0 ) return { action: ACTION.

REFUSE_GATE }; if ( helpServed. some ( h => h. action === ACTION. REVEAL )) return { action: ACTION. REVEAL }; const hint = helpServed. find ( h => h. action === ACTION.

HINT ); if (! hint ) return { action: ACTION. HINT }; const sinceHint = counted. length - hint. afterCountedAttempts; return sinceHint >= N? { action: ACTION. REVEAL }: { action: ACTION.

REFUSE_RETRY }; } Enter fullscreen mode Exit fullscreen mode The only thing of interest here is the Arm B line. That one if is the whole difference between the two versions. Anyone can read it and know exactly what each group got.

Every decision is logged with counted and sinceHint. So for any Help press, I can look at the log and prove why the tutor responded the way it did.

N is fixed at 2 for everyone. If it changed per person, I'd be testing two things at once, and three people per group can't separate them.

Arm A's rule is "try first". I call it the gate: no help until you've made an attempt. So the code needs to decide what a real attempt is. This turned out to be the hardest part of the policy.

My first version said: an attempt counts if its result set is different from the last one. Simple, and wrong.

The bug: a query that doesn't run has no result set. So for a learner who couldn't get anything to execute, no attempt ever counted. The gate never opened. In my earlier test rounds, 101 of 122 attempts had a syntax error and didn't run. This wasn't an edge case. It was the main case.

And a query that fails to parse can still carry a full, wrong mental model:

SELECT customer, COUNT ( * ) FROM orders WHERE COUNT ( * ) > 3 Enter fullscreen mode Exit fullscreen mode That doesn't run. It's also a perfect example of the exact mistake I was teaching about (filtering groups with WHERE instead of HAVING ). It should count.

But asdf doesn't parse either. If junk counts, a learner can type three bits of junk and get a hint while thinking about nothing. Research on tutoring systems calls this "gaming the system", and learners find that shortcut fast.

So I added a substance bar. An attempt must contain one SQL keyword and one name from this question's schema:

function isSubstantive ( sql, identifiers ) { const low = String ( sql?? '' ). toLowerCase (); if (! low. trim ()) return false; if (! SQL_KEYWORDS. some ( k => low. includes ( k ))) return false; if (! identifiers. length ) return true; // fail open, see below return identifiers. some ( id => id. length >= 4? low. includes ( id ): new RegExp ( ` \\ b ${ id } \\ b` ). test ( low ) // so "id" doesn't match "video" ); } Enter fullscreen mode Exit fullscreen mode Matching is by substring on purpose. Real learners wrote GroupBy and havingcount>=3. That's mangled spelling, but a real mental model. A strict token match would throw those away.

I checked it against all 128 attempts from the screening round. 121 passed. The 7 rejected were 6 empty submissions and one bare SELECT. No real attempt was lost.

Notice the fail open line. If the schema list fails to load, the bar only checks for a keyword. A misconfigured page must not silently refuse every learner. That would turn Arm A into "no help at all" without anyone noticing.

Here's the uncomfortable part. To write a useful hint about a wrong query, the LLM has to see the correct one. I call that the reference query. So it's holding the answer while it writes. Sometimes it just prints it.

If that reaches the learner, Arm A quietly becomes Arm B on that question. The experiment stops testing anything, and nothing in the logs looks wrong.

So every hint passes through hint-guard.js before the learner sees it. It's deterministic. It makes no model calls. It looks at text and says yes or no.

function inspect ( hint, { referenceQuery, learnerQuery }) { // 1. A runnable query: SELECT plus a real table after FROM/JOIN. // 2. A clause that distinguishes the reference query, // unless the learner already wrote it themselves. // 3. Too much of the reference reproduced word for word. // 4. A walkthrough: four or more SQL keywords named. } Enter fullscreen mode Exit fullscreen mode If a hint is rejected, the system asks the model once more. If that fails too, it serves a hand-written fallback for that question. Every attempt and every rejection is logged.

Results: on 101 real hints, zero false alarms. Against 24 leaks I wrote specifically to beat it, it caught 22.

Each rule in the guard exists because an earlier, simpler rule broke.

"SELECT and FROM means a leak." My first check rejected any hint containing both words. But they're normal English. "You started with SELECT, but where does the data come FROM?" is a great hint, and it got blocked. The fix: a runnable query must name a real table. The reference query tells you which tables exist, so look for those.

The semicolon rule. I also rejected "select...;" on the idea that a semicolon ends a statement. English uses semicolons too. "Nothing is processed; you need to say where the data comes from" got blocked. I deleted that rule instead of patching it, because the table rule already covered real queries.

Quoting the learner back to themselves. The prompt asks the model to quote what the learner wrote. So a hint would say "your MAX(price) is fine". MAX(price) is part of the answer, so the guard blocked it. 7 of 8 rejections in one test run were this. But the learner already wrote it. It's in their editor. Now a clause only counts as a leak if the learner doesn't already have it.

The leak with no SQL in it. "The FROM keyword picks the products table, then GROUP BY groups the rows by category..." No clause appears verbatim, but you could type the answer straight from it. The signal is shape, not content. A real hint talks about one or two things. A dictation names everything.

I didn't pick the threshold by feel. I measured it. Across the 101 real hints: 66 named no SQL keyword, 29 named one, 5 named two, 1 named three, and none named four. The two prose leaks named six and seven. A limit of four separates them with room on both sides.

The silent fallback. The LLM sits behind a Supabase edge function, so the API key never reaches the browser. Supabase pauses free projects after about a week without traffic. When paused, nothing errors. Arm A just serves the hand-written backup hints. The app looks fine, but it's quietly running a different tutor. I found this once, fixed it, and added a script to check it before every session.

If you're putting an LLM into a product, ask one question about each decision: does this need judgment, or does it need to be the same every time?

In my tutor, that split is why I could say exactly what each group got. It's also why the experiment could fail honestly. When two of my checks failed, I knew the failure was about learners, not about a model behaving differently on a Tuesday.

Next in the series: the mistakes I made running my first controlled experiment as a developer, including the one a Demo Day judge caught.

Vikrant Negi. Solo. Demo Day track. Submission 7 Oct 2026.

An SQL tutor that makes you try before it helps, and a 6-person pilot testing whether that changes what you can do a week later, with the tutor gone.

For where things stand today, read STATUS.md. That is the only file that claims to be current.

Live app: https://tranquil-starlight-f0129e.netlify.app/

Same page, same 16 practice questions, same grader. One config flag is the only difference.

Arm A scored 36 of 36 person-questions (3 learners x 12 held-out questions). Arm …