You've built a GPT from scratch, trained it, and generated text with different sampling strategies. But here's the uncomfortable truth: a 10M-parameter model trained on a laptop will never write coherent essays or answer questions reliably. Something fundamental changes when you scale up — and it follows precise mathematical laws.
In 2020, researchers at OpenAI discovered that neural network performance follows power laws: loss decreases predictably as you increase parameters, data, and compute. Two years later, DeepMind's Chinchilla paper showed that most labs were training their models wrong — using too many parameters and not enough data. These discoveries reshaped the entire field.
📐
The key insight
Performance is predictable — if you know the laws
Before scaling laws, training large models was trial and error. You'd spend millions of dollars on a training run and hope for the best. Scaling laws changed that: they let you predict the final loss of a model before training it, based on its size, dataset, and compute budget. This turned AI research from alchemy into engineering.
The formula is simple: L(N) = (N_c / N)^alpha, where L is the loss, N is the number of parameters, and N_c and alpha are constants fitted from experiments. Let's compute it.
Kaplan et al. scaling laws
The 2020 OpenAI paper "Scaling Laws for Neural Language Models" by Kaplan et al. established three key relationships. Loss decreases as a power law with each of:
Parameters (N): L(N) = (N_c / N)^alpha_N — more parameters, lower loss. But with diminishing returns: each 10x increase in parameters gives a fixed reduction in loss.
Dataset size (D): L(D) = (D_c / D)^alpha_D — more training tokens, lower loss. Same diminishing returns pattern.
Compute (C): L(C) = (C_c / C)^alpha_C — more FLOPs, lower loss. Compute is the master variable since C is roughly proportional to 6 * N * D.
The critical finding: these are smooth, predictable curves on a log-log plot. No sudden jumps, no plateaus — just steady improvement along a straight line. You can fit the curve with a small model and extrapolate to predict performance at 1000x the scale.
Going from 1M to 10M parameters (10x) drops loss from 4.02 to 3.37 — a reduction of 0.65. Going from 10B to 100B parameters (also 10x) drops loss from 1.99 to 1.67 — a reduction of only 0.32. Same 10x investment, half the gain. That's the diminishing returns of power laws.
The scaling curve
On a log-log plot, the power law becomes a straight line. This is the signature of scaling laws — and it holds across orders of magnitude.
Chinchilla scaling laws
In 2022, Hoffmann et al. at DeepMind published "Training Compute-Optimal Large Language Models" — the Chinchilla paper. Their key finding overturned the conventional wisdom:
🐭
The Chinchilla finding
D ≈ 20N — you need 20 tokens per parameter
Before Chinchilla, the common approach was to make models as large as possible and train on "enough" data. GPT-3 had 175B parameters but was trained on only 300B tokens — a ratio of ~1.7 tokens per parameter. Chinchilla showed this was massively undertrained.
The compute-optimal ratio is approximately D ≈ 20N: for every parameter, you should train on 20 tokens. A 70B model should see 1.4 trillion tokens. DeepMind proved this by training Chinchilla (70B params, 1.4T tokens) and showing it outperformed the much larger Gopher (280B params, 300B tokens) — despite using the same compute budget.
chinchilla_optimal.pyRead-only — NumPy
Expected output
=== Chinchilla Optimal Training ===
Model Params Tokens FLOPs Tok/Param
------------------------------------------------------------
Small 1e+08 2e+09 1.2e+18 20
Medium 1e+09 2e+10 1.2e+20 20
Large 1e+10 2e+11 1.2e+22 20
XL 7e+10 1.4e+12 5.88e+23 20
XXL 1.75e+11 3.5e+12 3.675e+23 20
=== GPT-3 vs Chinchilla-Optimal ===
GPT-3 actual: 2e+11 params, 3e+11 tokens
GPT-3 optimal: 2e+11 params, 4e+12 tokens
GPT-3 was undertrained by 12x!
The Chinchilla lesson:
For a fixed compute budget, you should not maximize model size. Instead, split your budget roughly equally between model parameters and training data. A smaller model trained on more data will outperform a larger model trained on less data — at the same cost.
Emergent abilities
While loss follows smooth power laws, some capabilities appear to emerge suddenly at certain scales. These are called emergent abilities — tasks that models below a threshold size cannot do at all, but models above it can do reliably.
Arithmetic: Models below ~10B parameters fail at 3-digit addition. Models above ~100B can do it reliably.
Chain-of-thought reasoning: Small models ignore "Let's think step by step." Large models use it to solve multi-step problems they'd otherwise fail.
Word unscrambling: Randomly appears around 60B parameters.
Translation: Not explicitly trained for it, but emerges in large models trained on multilingual data.
Debate alert:
The concept of "emergent abilities" is controversial. A 2023 paper by Schaeffer et al. argued that emergence is partly an artifact of the metrics used — when you switch from accuracy (all-or-nothing) to log-likelihood (continuous), the sharp transitions become smooth curves. The underlying loss always improves smoothly; it's the task performance metric that can show sharp jumps.
In-context learning
Perhaps the most surprising emergent ability is in-context learning (ICL) — the ability to learn new tasks from a few examples provided in the prompt, without any gradient updates. This is what makes few-shot prompting work.
Zero-shot: just the task
->
One-shot: 1 example
->
Few-shot: 3-5 examples
->
Better accuracy!
In-context learning is remarkable because the model doesn't update its weights. It "learns" the pattern from the examples in the prompt and applies it to the test case — all within a single forward pass. This ability scales with model size: small models ignore the examples, while large models use them effectively.
few_shot_demo.pyRead-only — PyTorch (conceptual)
Expected output
=== In-Context Learning ===
Prompt (few-shot):
Classify the sentiment:
Text: "This movie was amazing!" -> Positive
Text: "Terrible waste of time." -> Negative
Text: "I loved every minute of it!" -> Positive
Text: "Boring and predictable." ->
Model completion: Negative
Key insight: NO weight updates occurred!
The model 'learned' from examples purely through attention.
=== Why Scale Enables ICL ===
125M (GPT-2 Small) Cannot do ICL reliably
1.3B (GPT-2 XL) Weak ICL, inconsistent
6.7B (GPT-J) Moderate ICL ability
175B (GPT-3) Strong ICL, few-shot works well
Why scale matters — the loss curve tells the story
Let's tie everything together. The loss curve is the single most important diagnostic in LLM training. Here's how to read it and what scaling laws tell us.
plot_scaling_curves.pyRead-only — NumPy
Expected output
=== Scaling Curve Data (for plotting) ===
log10(N) N Loss
----------------------------------
6 1e+06 4.0159
7 1e+07 3.3712
8 1e+08 2.8300
9 1e+09 2.3756
10 1e+10 1.9943
11 1e+11 1.6741
12 1e+12 1.4055
13 1e+13 1.1802
=== Marginal Returns (loss reduction per 10x scale) ===
1e+06 -> 1e+07: loss drops 0.6447 (from 4.0159 to 3.3712)
1e+07 -> 1e+08: loss drops 0.5413 (from 3.3712 to 2.8300)
1e+08 -> 1e+09: loss drops 0.4545 (from 2.8300 to 2.3756)
1e+09 -> 1e+10: loss drops 0.3813 (from 2.3756 to 1.9943)
1e+10 -> 1e+11: loss drops 0.3201 (from 1.9943 to 1.6741)
1e+11 -> 1e+12: loss drops 0.2687 (from 1.6741 to 1.4055)
1e+12 -> 1e+13: loss drops 0.2253 (from 1.4055 to 1.1802)
Notice: each 10x step gives LESS reduction. That's the power law.
The big picture:
Loss improves smoothly and predictably. Capabilities emerge in bursts. The gap between "smooth loss" and "sudden capabilities" is the central mystery of large language models. Understanding scaling laws lets you predict the cost of reaching a target loss — and that's how labs decide whether to train the next generation of models.
Why this matters for LLMs
The science behind billion-dollar training runs
Every major AI lab uses scaling laws to plan their training runs. Before spending $100M+ on compute, they train small models (1B-10B parameters), fit the scaling curves, and extrapolate to predict whether their target model size will reach the desired performance level. This is how GPT-4, Claude, and Gemini were planned.
The Chinchilla paper changed the industry overnight. After its publication, every lab pivoted from "make models bigger" to "train models longer on more data." LLaMA (Meta, 2023) was explicitly designed to be Chinchilla-optimal: 65B parameters trained on 1.4T tokens. The result outperformed GPT-3 (175B) at a fraction of the cost.
Using the Kaplan scaling law formula L(N) = (N_c / N)^alpha with N_c = 8.8e13 and alpha = 0.076, predict the loss for a model with 1.8 trillion parameters (rumored GPT-4 size). How does it compare to GPT-3's predicted loss? Is the improvement worth the 10x parameter increase?
Compute (8.8e13 / 1.8e12) ** 0.076 for GPT-4 and (8.8e13 / 1.75e11) ** 0.076 for GPT-3. The difference should be small — that's why labs invest in better data, not just more parameters.
Challenge 2: Chinchilla-optimal budget planning
You have a compute budget of 1e24 FLOPs. Using C ≈ 6ND and D = 20N, solve for the optimal model size N and dataset size D. How many parameters and tokens should you use? Compare this to naively using all compute on a single giant model with minimal data.
Substitute D = 20N into C = 6ND to get C = 120 * N^2. Then N = sqrt(C / 120). With C = 1e24, you get N ≈ 2.89e11 (289B params) and D ≈ 5.77e12 (5.77T tokens). A "naive" approach might use 1T params with only 167B tokens — far from optimal.
Quiz
Question 1 of 5
According to the Kaplan scaling laws, how does loss relate to the number of parameters?
Question 2 of 5
What is the Chinchilla compute-optimal ratio of training tokens to parameters?
Question 3 of 5
What are "emergent abilities" in LLMs?
Question 4 of 5
What makes in-context learning (few-shot prompting) remarkable?
Question 5 of 5
According to the Chinchilla scaling law, what was wrong with GPT-3's training setup?
0
/5
You understand the science behind scaling LLMs.
Next in Arc D
Next: Supervised Fine-Tuning — SFT ->
You've seen what makes raw LLMs powerful. Now learn how to make them useful — fine-tuning a pretrained model on instruction-following data to create a helpful assistant.