I'm finally fine-tuning my model, because its prompt hit 175,000 characters

Line 60 of my model's prompt says "Never output em-dashes, in any form." Line 976 says it again, in case it forgot. The agent it runs in says it a third time. It still slips them in, so that agent ends up ripping them out of every reply in code, a token at a time, as it streams.

That's when I admitted the prompt had gotten out of hand.

Every version of Onyx, the model behind my local AI agent Flash, has secretly been a prompt. Not fine-tuned weights. A Modelfile wrapped around an open-weights base, with the system prompt doing all the work. Every time Onyx did something dumb, I wrote a rule. Onyx 2.6's prompt is now 174,826 characters long. That's about 980 lines of rules the model has to keep in mind while it works, and every new chat pays for all of them before it writes its first word. With Flash's own prompt and tool list on top, a fresh chat starts at around 256,000 characters.

So I'm finally doing the thing I put off for eight versions: fine-tuning. Onyx 3 moves the prompt into the weights. The plan cuts the system prompt to 650 characters and a new chat to around 82,000.

The training pipeline is done and tested end to end. The real run hasn't happened yet. Here's how it works.

I don't have a dataset of "good Onyx answers", and writing thousands by hand wasn't happening. But I do have a model that already gives them: Onyx 2.6, with its giant prompt, on the biggest base it runs on.

So 2.6 is the teacher. It answers a few thousand requests the way it would inside Flash, tools and all. Then the smaller base gets fine-tuned to give those same answers under the 650-character prompt. What the long prompt taught becomes what the model does on its own.

I wrote 161 requests by hand across 22 categories: code, work, everyday life, knowledge and plain talk. A script has the teacher write more in the same style until there are a couple thousand, weighted toward code. Another 54 are held back for the benchmark and never trained on.

Each request gets answered twice, inside a fresh copy of a small fixture project (a little Python shop, a webpage, a folder of notes and receipts). The teacher really uses Flash's tools: it reads files, edits them and runs the tests. Since it's running real shell commands, that step runs inside a throwaway Docker container that holds nothing else.

Every conversation also starts Flash with an empty home folder. My memories, sparks, settings and email never reach a prompt, so they never reach the weights.

The teacher isn't perfect either, which is kind of the point of all those rules. So every answer has to pass two filters before it's trained on.

First, hard rule checks in code. If it breaks one, it's dropped:

Softer ones, like "great question!", "hope this helps" or emoji nobody asked for, only cost points.

Then a judge model scores what's left out of 10. Anything under 7 is out, and of the two answers to each request, the better one wins.

Since I was retraining anyway, I added two rules, put last in the teacher's prompt so they win.

It writes the way you do. Type "hey" and you get "hey..." back. Type "Hey" and you get "Hey,...". Half the training requests are lowercase and half aren't, so it learns both. A conversation that gets the casing wrong is never trained on.

And it never uses an em dash. Yes, that one again. The prompt asking twice didn't work, and Flash's filter only cleans up afterward. This time the rule goes where the habit lives. Any teacher answer with a single em dash in it, or a double hyphen pretending not to be one, is thrown out before training, so Onyx 3 never sees one written.

The detail I like most: each training example is a whole conversation, with Flash's prompt, the tool list, the request and every tool result. But only the model's own words carry labels. In one example I checked, that was 70 tokens out of 23,713. Everything else is context it reads but is never trained to produce. Otherwise it would spend the run learning to recite the tool list.

It's LoRA on the attention and feed-forward weights, rank 32, two epochs. The 12b trains in bf16 on one 80 GB GPU, or in 4-bit on a 24 GB card, and it runs on Colab. My rough guess is 5 to 15 hours for the 12b, but the first ten steps print the real rate.

Onyx and Flash are built for each other. Three quarters of the training conversations happen inside Flash, with its real prompt, its real tools and real projects, so Onyx learns to read a file before editing it, run the tests, and stop when the job's done. It goes the other way too. Onyx's prompt is full of Flash: how to use its tools, how to write a long file in pieces so a reply never gets cut off, and how to behave when it's running as a spark on its own. Flash is the agent Onyx was shaped in, and Onyx is the model Flash recommends.

You can run Flash on any Ollama model that calls tools, and Onyx answers fine in a plain ollama run. But Onyx works best with Flash, and Flash works best with Onyx. Onyx 3 makes that even more true, since the 175,000 characters of how to be a good Flash agent are now in the weights instead of in every chat.

The benchmark runs 2.6 and 3 side by side on the 54 held-out requests. For each model it reports the judge's score, the share of answers that break no hard rule, the prompt tokens a new chat starts with, seconds to first reply from cold, and tokens per second.

The bar for shipping: match 2.6 on the first two, beat it on the rest. If the small prompt can't match the big one on quality, Onyx 3 doesn't ship, however fast it is.

I ran the whole pipeline end to end on a CPU, with a stand-in teacher and a tiny model shaped like the real base: request generation with Flash's real tools in the sandbox, judging, building the dataset, training, merging, GGUF conversion, building it into Ollama, and the benchmark. Everything connects.

What that can't tell me is whether Onyx 3 is any good. That takes the real teacher, the real base and a rented GPU, and that's next. Onyx 3 is Gemma 4 fine-tuned, Apache 2.0, and it'll run anywhere Ollama does.

I'll post the benchmark either way, including if 3 loses to the 175,000-character prompt.

Have you distilled a long system prompt into weights? I'd like to know what went wrong for you before it goes wrong for me.