Bigger AI models aren't always better: how a 70B model beat a 280B one by reading more

For a few years, the way to build a better AI model was simple. Make it bigger.

Then in 2022 DeepMind trained a model called Chinchilla. It had 70 billion parameters. It beat their own older model, Gopher, which had 280 billion. A quarter of the size, better scores.

How? Chinchilla didn't have a smarter design. It just read a lot more.

I wanted to see that for myself, so I trained a pile of tiny models on my laptop. Here's what happened.

Training costs compute, and compute is roughly parameters times tokens. (Times about 6, if you want the exact napkin version.) That means you can spend a budget two ways: a big brain that studies a little, or a smaller brain that studies a lot.

In 2020 a paper from OpenAI (Kaplan et al.) found that a model's error drops smoothly and predictably as you add parameters, data, or compute. Great, you can plan with that. Their advice for a fixed budget was to go for a very large model, train it on a relatively modest amount of data, and stop before it fully converges.

Two years later, the Chinchilla paper (Hoffmann et al.) re-ran the question more carefully and came to a different answer: model size and training tokens should grow together. Double the model, double the data.

To prove it, they trained Chinchilla with about the same compute budget as Gopher, but spent it differently:

(MMLU is a big multiple-choice knowledge test. The paper's abstract says 67.5%, a table inside says 67.6%. Same story either way.) Chinchilla also beat GPT-3 (175B), Jurassic-1 (178B) and Megatron-Turing NLG (530B) on a wide range of tasks.

The conclusion in the paper: a lot of the big models of the time were undertrained. Too much brain, not enough studying.

The rule of thumb people took from it: roughly 20 tokens of training data per parameter. Chinchilla itself is 1.4T / 70B, which is 20.

That rule plus the napkin formula gives you a tiny calculator. If compute is 6 × N × D and D = 20 × N, then compute = 120 × N². Flip it around:

from math import sqrt def split_budget ( compute_flops, tokens_per_param = 20 ): n = sqrt ( compute_flops / ( 6 * tokens_per_param )) return n, tokens_per_param * n for c in [ 1e22, 5.76e23, 1e24 ]: n, d = split_budget ( c ) print ( f " { c:. 2 e } FLOPs -> { n / 1e9:. 1 f } B parameters, { d / 1e12:. 2 f } T tokens " ) Enter fullscreen mode Exit fullscreen mode 1.00e+22 FLOPs -> 9.1B parameters, 0.18T tokens 5.76e+23 FLOPs -> 69.3B parameters, 1.39T tokens 1.00e+24 FLOPs -> 91.3B parameters, 1.83T tokens Enter fullscreen mode Exit fullscreen mode The middle line is the compute number from the Chinchilla paper's table. Out comes 69B parameters and 1.4T tokens. That's Chinchilla, from one line of arithmetic.

Billions of parameters isn't something I can do. But the idea, "for a fixed budget there's a best model size", should show up at any scale. So I made a toy version.

35 training runs. Then I measured the loss on the text the models never saw (lower is better).

Each coloured line is one budget. The star is the winner.

The oversized models lost, badly. At the biggest budget, the 3.2-million-parameter model got a loss of 2.61. A model about 150 times smaller, trained on far more text, got 1.75. The big one only got to read about 155 thousand tokens. That's under 0.05 tokens per parameter, a huge brain with one skim of the book.

The winner got bigger as the budget grew. Best sizes were about 7.4k, 7.4k, 10.7k and 20.6k parameters, for budgets of 1e11, 3e11, 1e12 and 3e12.

The bottom of each curve is flat. At 1e12, models of 7.4k, 10.7k and 20.6k parameters scored 2.068, 2.039 and 2.047. You don't need the exact perfect size. You need to avoid being way too big.

That's the same shape the Chinchilla paper describes: for a fixed budget, too-big loses, and the best model grows as the budget grows.

This is a toy, so I want to be straight about it:

So treat this as "the pattern shows up even in a toy", not "I verified Chinchilla's numbers". For the actual numbers, read the paper.

If you want to play with it, the whole experiment is about 100 lines of PyTorch. Change the data, the budgets or the sizes. I'd love to hear if you see the same curves, or different ones.