Why MuonGS Beats AdamW and Muon in LLM Pretraining
Standard AdamW and vanilla Muon waste millions of GPU hours during LLM pretraining because uncontrolled weight-norm growth throttles the effective learning rate. We cannot keep burning H100 clusters on sub-optimal trajectory dynamics when Gram-Schmidt projection gives us direct mathematical control
Posted on Oct 29 • Originally published at bohdanhordii.dev
Standard AdamW and vanilla Muon waste millions of GPU hours during LLM pretraining because uncontrolled weight-norm growth throttles the effective learning rate. We cannot keep burning H100 clusters on sub-optimal trajectory dynamics when Gram-Schmidt projection gives us direct mathematical control over weight exploration.
Here is how MuonGS fundamentally shifts pretraining economics:
The one real trade-off is architectural sensitivity: Gram-Schmidt projection assumes smooth scale-invariant layers. Non-normalized components like token embeddings and language model output heads still require segregated coordinate-wise schedulers or traditional AdamW branches.
The Question to You: What is your team using for foundation pretraining and long-horizon fine-tuning in 2026? Are you still defaulting to AdamW, or running Muon and SOAP in production? Drop your stack below.
Bohdan Hordii // Systems & Sovereign Infrastructure Architect | bohdanhordii.dev