I tried to teach a small AI model a calculus textbook. The first thing I had to fix was how I was keeping score.

Five weeks, one free GPU, and a lesson about measuring before celebrating.

One of my training runs made the model 19 points worse at calculus.

That gap is really what this article is about. For a while, my numbers told me one story and the truth was another. Here's the whole thing: what I built, where it went wrong, and what I'd do differently next time.

Why I tried this At Konhrad, we're betting that you can take a textbook in almost any field, whether calculus, a BIM handbook or a legal code, and turn it into a small model that actually knows the subject. I didn't want to pitch that idea to anyone until I'd tried it myself.

So I kept it simple. I picked Thomas' Calculus, just the single-variable chapters (1 to 9), and Qwen2.5-1.5B-Instruct, an open model small enough to train on a free Kaggle GPU. My one rule was not to fool myself. I broke it more than once.

The model learned to format its answers the way I wanted. It didn't learn more calculus. My first scorer couldn't tell the difference, and that cost me weeks. I picked a subject the model already knew pretty well, so there wasn't much left to teach it.

Getting the book out of a PDF I honestly thought this would be the easy bit. It took three tools.

Nougat kept breaking on software version conflicts. Docling turned about a quarter of the formulas into garbage. MinerU finally gave me clean maths, but only on CPU, and slowly: around 39 minutes for every 58 pages. It also had a hidden timeout at about an hour, so I ended up splitting the book into batches of under 60 pages. Thirteen batches later, I had chapters 1 to 9 as clean text. By my rough estimate, that's about 550,000 tokens.

Turning a textbook into training data Here's the thing about textbooks. They explain ideas. Models learn from worked examples, and I didn't have any.

My first try gave me 194 examples, and they were lopsided: 15 for one section, a single one for the Mean Value Theorem. So I tried something else. I had the model attempt each of 1,072 problems eight times, and I only kept the answers I could verify. This is a known technique, usually called rejection-sampling fine-tuning or STaR. The idea is that solutions written in the model's own words are easier for it to learn than someone else's, and keeping only the verified ones nudges it toward its best behaviour. It's also cheap, because the attempts cost nothing but GPU time. Across those attempts, the model got 7.1% of problems right every time, 68.7% some of the time, and 24.3% never. For that last group, I paid a stronger model to write the solutions. Total cost: $2.25.

Cleaning took longer than generating. The worst surprise was finding 414 training problems that were really test or validation questions with the numbers changed, plus 12 that were copied from the test set word for word. If I'd left them in, the model would have been studying its own exam.

In the end I had 1,414 verified solutions. 1,218 came from the model itself, and 196 from the stronger one.

Teaching by example The first method I tried was supervised fine-tuning. You show the model worked solutions and train it to copy the style and the reasoning. I used LoRA, which only adjusts about 2% of the model.

My first proper run went badly. I used a high learning rate and three passes over the data, and accuracy on held-out problems dropped from 64.2% to 45.4%. It fell nearly as much on sections it had never trained on, so it hadn't learned anything wrong about calculus in particular. It had just been knocked around everywhere.

The next run was much gentler: a learning rate five times lower, two passes, about 19 minutes of training. It scored 65.8% against 64.2% for the original model. When I compared the two on the same 240 problems, the new one got 25 right that the old one missed, and missed 21 that the old one got right. A net gain of four problems. That's noise.

By this point every run was a few hours on Kaggle, so I'd built crash recovery into the notebooks. I didn't want a three-hour run dying on a typo.

Teaching by reward Looking at how often the model could solve problems when it got several tries, I was convinced it had more in it. So I tried reinforcement learning (GRPO). Instead of showing the model answers, you reward it when it gets one right. I gave it a point for a verified correct answer, nothing for a wrong one, and a small penalty for not putting a final answer in a box.

On a single T4 this was painfully slow. Each step only covered two problems, eight attempts each, and took about a minute and a half. The short run took 3 hours. The bigger one, which I'd hoped would be the breakthrough, took 7.

Neither moved accuracy in any real way (+0.4 and −1.2 points). One thing did change, though. The share of answers written in a box jumped to 98%. The model had found the easiest part of the reward to earn, and earned it.

How I was keeping score was the real problem This is where most of my time went.

My first scorer only gave credit if the answer was written inside a final box, the way I'd asked. The original model does that about 71% of the time. After fine-tuning, the models did it 88 to 99% of the time. So every fine-tuned model was getting credit just for formatting better.

That one mistake hurt in both directions. It made my early runs look like double-digit wins. It made the original model look like it scored 46%, when it really scored 62 to 64%. And it hid real damage: that aggressive run looked like it had lost 3 points when it had actually lost 19.

Article content Qwen2.5-1.5B-Instruct. "In the box" means the final answer was written inside \boxed{}, which my first scorer required. So I rebuilt the evaluation from scratch.

I checked the answer key. I had a stronger model solve all 1,442 problems twice, which cost $0.54. 94.5% of the reference answers held up, and I threw out 10 test problems whose answers I didn't trust before scoring anything. I made the test big enough. My first test set had 117 problems, which is too small to prove even a 5-point gain. I took it to 240. I kept a control. I held six whole sections out of training, 112 problems in all. If the model really learned the book, it should do better on what it studied than on those. I scored strictly as well as fairly. About half the answers could only be judged by a second AI, so I also report a score that relies on code alone.

Once the scoring was fixed, the answer was clear. In no run did the model improve more on the sections it had studied than on the ones it never saw.

Article content Change in accuracy, percentage points. Only the aggressive SFT run is statistically significant, and it got worse. The choice I got wrong If I'm honest, the biggest problem was the brief I set myself. The original model already solved about 64% of the test problems, and 51% of the sections it had never seen. Single-variable calculus is everywhere online, so the model had almost certainly met it before. And a textbook is mostly explanation. There just aren't many facts in it the model didn't already have.

I'd picked a subject with very little left to teach, and a method that mostly rewarded tidiness. That's also the limit of training on a model's own answers: it can only reinforce what the model can already do some of the time. Only 196 of my 1,414 solutions, about 14%, came from the stronger model, so very little new knowledge went in.

What I'd tell anyone starting a fine-tune Sort out your scoring before you train anything. Test it on the untouched model first. If the scorer is wrong, every decision after it is wrong too.

Keep some material the model never sees. It's the only way I found to tell real learning from better formatting. Make sure there's something to learn.

If the base model already does well, find harder or less familiar material. Bring new information. Training a model on its own correct answers mostly teaches it format.

Knowledge has to come from somewhere new. Go gently. A high learning rate damaged everything, and a low one barely changed anything.

Try a few settings on validation data before you trust any of them. Leave reinforcement learning for last. It needs lots of problems per update and a reward that's hard to game.

Where I'm taking this The pipeline itself worked. Getting clean text out of a PDF, building verified training data and measuring honestly are now repeatable, and that's the part I'm keeping.

What I'm changing is the bet. Next time I'll start with material a small model genuinely fails, somewhere under 30%, ideally text it can't have seen online, like a company's internal handbook or a niche technical standard. I'll bring in knowledge the model doesn't have, through solutions written by a stronger model and continued training on far more source text. And I'll only reach for reinforcement learning once there's something real to reinforce.

Next time, I’ll measure first, and celebrate only if the numbers hold up.

Himansh Tekumudi Founder & Partner @ Konhrad