What it actually costs to train an LLM: GPUs, days and dollars, worked out step by step

You've seen the headlines. "Training a big model costs tens of millions of dollars."

Okay, but why that number? Where does it come from? I wanted to count it myself, with a formula short enough to fit on a napkin, and then check the answer against something real.

Turns out you can get surprisingly close. And the places where you don't get close are the most interesting part.

training FLOPs ≈ 6 × parameters × tokens Enter fullscreen mode Exit fullscreen mode That's it. Model size times data size times 6.

Where does the 6 come from? The usual explanation: for each token, the model does about 2 operations per parameter going forward (making a prediction) and about 4 going backward (figuring out how to improve). 2 + 4 = 6. It's a rule of thumb, not exact, but it's the one everybody uses.

Quick sanity check, because I don't trust a formula until it matches something:

Now, how fast does a GPU chew through FLOPs?

An NVIDIA H100 (the chip these big runs use) has a peak of about 989 TFLOP/s in the number format used for training. That's 989 trillion operations per second. Sounds huge. It is.

But "peak" is the speed on the spec sheet, in a perfect world. Real training spends time moving data around, waiting on other GPUs, and so on. Meta said their best setup reached over 400 TFLOP/s per GPU, which is about 40% of peak. (Their paper puts utilisation at 38 to 43%.)

GPU-hours = FLOPs / (speed per GPU × 3600) Enter fullscreen mode Exit fullscreen mode

Meta published GPU-hours for their models. So I could check my napkin against the real thing.

(I used the nominal sizes, 8B, 70B and 405B. The real counts are a bit different.)

The 405B result is fun: Meta reported 30.8M GPU-hours, and my napkin at 400 TFLOP/s said 26.3M. Not bad.

But look at the 8B. Meta reported 1.3M hours, and my napkin said 0.5M. Off by 2.6x.

So I flipped the question around. If Meta's hours are right, what speed did the GPUs actually deliver? Dividing the work by the reported hours gives:

The bigger the model, the better the GPUs were used. My guess, and it is a guess, is that small models don't keep a huge GPU busy as efficiently, and that the 6ND rule skips some real costs (attention, restarts, evaluation runs). I haven't dug into Meta's logs, so take that as speculation. The lesson I'd actually take: the napkin is great for order of magnitude, and the weak spot is always GPU speed. Use a range, not one number.

Big caveat: Meta owns its GPUs, so this isn't what Meta spent. It's what you'd pay to rent that much compute, and rental prices vary a lot. I'm using $2 to $4 per H100-hour as an assumption, not a quote.

That's just the final training run, by the way. It doesn't include failed experiments, data work, salaries, or the cost of all the smaller models you train first to figure out what to build.

Here's the part people miss. GPU-hours is a total. How many GPUs you need depends on how long you're willing to wait:

GPUs needed = GPU-hours / hours you can wait Enter fullscreen mode Exit fullscreen mode The 8B run at 1.3M GPU-hours:

(Those cluster sizes are my examples, not what Meta used for that model.)

For the 405B: 30.84M GPU-hours spread over 16,384 GPUs is about 1,880 hours, roughly 78 days, if every GPU worked the whole time. Real runs don't, so treat that as a rough ballpark.

Here's the whole thing as a script. Change the numbers.

def train_cost ( params, tokens, tflops_per_gpu, dollars_per_gpu_hour, n_gpus = None ): flops = 6 * params * tokens gpu_hours = flops / ( tflops_per_gpu * 1e12 ) / 3600 out = { " FLOPs ": f " { flops:. 2 e } ", " GPU-hours ": f " { gpu_hours:,. 0 f } ", " cost ": f " $ { gpu_hours * dollars_per_gpu_hour:,.

0 f } " } if n_gpus: out [ " days " ] = f " { gpu_hours / n_gpus / 24:. 1 f } days on { n_gpus } GPUs " return out # a 1B model, 20 tokens per parameter (Chinchilla's rule of thumb), one H100 at 150 TFLOP/s print ( train_cost ( 1e9, 20e9, 150, 2 )) print ( train_cost ( 1e9, 20e9, 150, 4 )) Enter fullscreen mode Exit fullscreen mode { ' FLOPs ': ' 1.20e+20 ', ' GPU-hours ': ' 222 ', ' cost ': ' $444 ' } { ' FLOPs ': ' 1.20e+20 ', ' GPU-hours ': ' 222 ', ' cost ': ' $889 ' } Enter fullscreen mode Exit fullscreen mode So a small 1B model, on a sensible amount of data, is about $450 to $900 of rented GPU time. That's a "weekend project and a credit card" number, not a "nation-state" number.

It's the 8B going to 70B going to 405B that moves you from "hobby" to "company" because the work multiplies each time.

And for fun, I measured my own laptop CPU while running small experiments: about 30 GFLOP/s on a tiny model. Same 1B model:

1.2e20 FLOPs / 3e10 FLOP/s = 4e9 seconds ≈ 127 years Enter fullscreen mode Exit fullscreen mode So no, I won't be training that one on the couch.

I used nominal model sizes, rented-GPU prices I picked myself, and Meta's reported hours as ground truth. The explanation for why the 8B ran at a lower share of peak is my guess. And the 6ND rule is a simplification: real runs have extra costs it ignores.

If you want to check my arithmetic, the numbers in the table come straight from the two Llama model cards and the Llama 3 paper. Run the script with your own model size and see what a "small" model costs. I was surprised how small.