Every language model needs text to learn from. For our mini-GPT, we'll use the complete works of Shakespeare — roughly 1 million characters of Early Modern English. It's small enough to train on a laptop, rich enough to contain complex patterns (meter, dialogue, stage directions), and public domain.
Since our model is character-level (vocabulary size 65 — all unique characters in the text), we don't need a tokenizer like BPE. Each character is a token. This keeps the pipeline simple so we can focus on the training mechanics.
📚
Training corpus
Shakespeare — character-level tokenization
The dataset has 1,003,854 characters (tokens). We split 90% for training and 10% for validation. With a block size of 256, each training example is a 256-character window into the text. The model reads the first 255 characters and predicts characters 2 through 256 — next-token prediction at every position.
shakespeare_dataset.pyRead-only — run locally with PyTorch
Expected output
=== Training Mini-GPT on Shakespeare ===
Vocabulary size: 65 (character-level)
Training tokens: 1,003,854
Batch size: 64, Block size: 256
Train chars: 903,468
Val chars: 100,386
First 100 characters:
First Citizen:
Before we proceed any further, hear me speak.
All:
Speak, speak.
First Citizen:
You
Why shift by 1?
The input is characters at positions [0, 1, 2, ..., 254] and the target is characters at positions [1, 2, 3, ..., 255]. At every position, the model learns to predict the next character. This single-position shift is the entire training signal — no labels, no annotations, just "predict what comes next."
Next-token prediction — the training objective
The training objective for GPT is elegantly simple: given a sequence of tokens, predict the next token at every position. The model outputs a probability distribution over the vocabulary at each position, and we compare that distribution against the actual next token.
Input: "To be or"
->
GPT
->
Target: "o be or "
Notice the shift: the target is the input shifted right by one position. At position 0, the model sees "T" and must predict "o". At position 1, it sees "To" and must predict " " (space). At position 2, it sees "To " and must predict "b". Every single position provides a training signal.
This is incredibly data-efficient. A single sequence of length 256 provides 256 training examples simultaneously. With 1 million characters of Shakespeare, we get roughly 1 million individual next-token predictions per epoch.
Cross-entropy loss for language modeling
The model outputs logits — raw scores for each character in the vocabulary. To train it, we need a loss function that measures how far the predicted distribution is from the true next character. That loss function is cross-entropy.
Cross-entropy loss for language modeling is:
L = -log P(correct token)
If the model assigns probability 0.9 to the correct next character, the loss is -log(0.9) = 0.105 (low). If it assigns probability 0.01, the loss is -log(0.01) = 4.6 (high). The loss penalizes confident wrong predictions heavily.
At the start of training, the model's weights are random. With a vocabulary of 65 characters, a random model assigns roughly 1/65 probability to each character. The expected initial loss is -log(1/65) = log(65) ≈ 4.17. This is exactly what we see — our initial loss is 4.1742, confirming the model starts at random chance.
cross_entropy_loss.pyRead-only — run locally with PyTorch
Expected output
Vocabulary size: 65
Random baseline loss: -log(1/65) = 4.1744
Our initial loss: 4.1742
This confirms the model starts at random chance!
As training proceeds:
Loss 4.17 -> model guessing randomly
Loss 2.50 -> model learned basic character frequencies
Loss 1.65 -> model learned common words
Loss 1.32 -> model learned grammar and phrases
Perplexity = exp(loss):
Perplexity is the exponential of cross-entropy loss. A loss of 4.17 means perplexity 65 (the model is as confused as random choice among 65 characters). A loss of 1.32 means perplexity 3.7 — the model has narrowed its "confusion" to roughly 4 plausible characters at each position. Lower is better.
Learning rate scheduling — warmup + cosine decay
A constant learning rate works, but a scheduled learning rate works much better. The standard approach for training Transformers combines two phases:
Linear warmup (steps 0-200): Start with a tiny learning rate and linearly increase to the peak LR (3e-4). This prevents the randomly initialized model from taking destructively large gradient steps in the first few iterations.
Cosine decay (steps 200-3000): Smoothly decrease the learning rate from the peak down to a minimum (3e-5) following a cosine curve. This lets the model make large updates early and fine-grained adjustments late in training.
lr_schedule.pyRead-only — run locally with PyTorch
Expected output
Step | Learning Rate
------+--------------
0 | 0.00e+00
50 | 7.50e-05
100 | 1.50e-04
200 | 3.00e-04
300 | 2.93e-04
500 | 2.70e-04
1000 | 2.13e-04
1500 | 1.50e-04
2000 | 9.02e-05
2500 | 4.27e-05
3000 | 3.00e-05
Warmup phase: steps 0-200 (LR ramps from 0 to 3.00e-04)
Cosine decay: steps 200-3000 (LR decays from 3.00e-04 to 3.00e-05)
Why warmup?
At initialization, the model's parameters are random. Large learning rates applied to random weights can cause the loss to explode or the model to diverge permanently. Warmup lets the model "settle" into a reasonable region of the loss landscape before applying the full learning rate. GPT-2, GPT-3, and most modern LLMs all use warmup.
Gradient clipping — preventing explosions
Even with warmup and scheduled learning rates, Transformers can occasionally produce very large gradients — especially during early training when the loss is high and the model encounters unusual sequences. A single batch with enormous gradients can corrupt weeks of training.
Gradient clipping caps the total gradient norm. If the combined magnitude of all gradients exceeds a threshold (we use 1.0), all gradients are scaled down proportionally so the total norm equals the threshold. Individual gradient directions are preserved — only the magnitude is reduced.
gradient_clipping.pyRead-only — run locally with PyTorch
Expected output
=== Gradient Clipping Demo ===
Without clipping:
Grad norm = 15.3 -> step size = 15.3 * lr
Grad norm = 0.8 -> step size = 0.8 * lr
Grad norm = 247.1 -> step size = 247.1 * lr (EXPLODES!)
With clipping (max_norm=1.0):
Grad norm = 15.3 -> clipped to 1.0, step size = 1.0 * lr
Grad norm = 0.8 -> unchanged, step size = 0.8 * lr
Grad norm = 247.1 -> clipped to 1.0, step size = 1.0 * lr
Clipping prevents catastrophic updates while preserving
gradient direction. GPT-2 and GPT-3 both use max_norm=1.0.
When to worry about exploding gradients:
If your training loss suddenly jumps to NaN or infinity, exploding gradients are the likely cause. Gradient clipping with max_norm=1.0 is essentially free insurance — it never hurts performance and can save a training run from catastrophic failure. Every major LLM training setup includes it.
Full training loop
Now we combine everything: the dataset, loss function, learning rate schedule, and gradient clipping into a complete training loop. This is the code that transforms a randomly initialized Mini-GPT into a model that generates Shakespearean text.
The loss curve tells the story of learning. In the first 200 steps, the loss drops rapidly from 4.17 (random guessing) to 2.86 — the model quickly learns character frequencies. By step 1500, the loss reaches 1.65 — the model has learned common words and basic patterns. By step 3000, the loss reaches 1.32 — the model has internalized grammar, common phrases, and even some Shakespearean style.
Train loss vs val loss:
The final training loss is 1.3215 and the validation loss is 1.4872. The gap (0.17) is modest, indicating the model isn't severely overfitting. If the gap were large (e.g., train=0.5, val=2.0), that would signal the model memorized the training data instead of learning generalizable patterns.
Validation — measuring generalization
Training loss tells us how well the model fits the data it has seen. Validation loss tells us how well it generalizes to new text. We hold out 10% of Shakespeare (100,386 characters) and periodically measure loss on this unseen data.
validation.pyRead-only — run locally with PyTorch
Expected output
Validation evaluation (end of training):
Eval batches: 50
Avg val loss: 1.4872
Train loss: 1.3215
Gap: 0.1657 (healthy — not overfitting)
Saving checkpoints
Training takes time. If your machine crashes or you want to resume later, you need checkpoints. A checkpoint saves everything needed to resume training: model weights, optimizer state (including momentum buffers), the current step, and the loss.
checkpointing.pyRead-only — run locally with PyTorch
Why save optimizer state?
AdamW maintains two buffers per parameter: the first moment (mean of gradients) and the second moment (mean of squared gradients). If you only save model weights and create a fresh optimizer, these buffers reset to zero — the optimizer "forgets" its momentum and the learning rate schedule effectively restarts. Always save and restore both model and optimizer state.
Run it yourself
Train Mini-GPT locally
Download the Shakespeare dataset and run the full training loop:
On a modern CPU, training takes about 12-15 minutes. On a GPU (device='cuda'), it takes under 2 minutes. After training, try generating text — you'll see the model produce surprisingly coherent Shakespearean prose.
Why this matters for LLMs
You just trained the same way OpenAI trained GPT
The training loop you see above — next-token prediction with cross-entropy loss, AdamW optimizer, warmup + cosine decay, gradient clipping — is the same recipe used for GPT-2, GPT-3, and GPT-4. The only differences are scale: more data (terabytes vs. 1MB), more compute (thousands of GPUs vs. your laptop), and more parameters (billions vs. 2 million).
In the next lesson, you'll take this trained model and explore how to generate text from it. Temperature sampling, top-k, nucleus sampling — the techniques that make ChatGPT's responses feel natural rather than robotic.
Try training with a constant learning rate of 3e-4 (no warmup, no decay) and compare the final loss to the scheduled version. Does the model converge to the same loss? Does it take more or fewer steps? Plot both loss curves on the same chart.
A constant LR of 3e-4 typically works but converges to a slightly higher final loss (~1.38-1.42 vs 1.32). The cosine decay helps the model make finer adjustments in the later stages of training. Without warmup, you may also see instability in the first 50-100 steps.
Challenge 2: Implement early stopping
Add logic to the training loop that monitors validation loss every 500 steps. If the validation loss hasn't improved for 3 consecutive checks (patience=3), stop training early and restore the best checkpoint. This prevents overfitting on longer training runs.
Track best_val_loss and a patience_counter. Each time you evaluate, if val_loss < best_val_loss, reset patience_counter and save a checkpoint. Otherwise increment patience_counter. If patience_counter reaches 3, break out of the training loop and load the best checkpoint.
Quiz
Question 1 of 5
What is the training objective for GPT?
Question 2 of 5
Why is the initial training loss approximately 4.17 for our character-level model?
Question 3 of 5
What is the purpose of the warmup phase in the learning rate schedule?
Question 4 of 5
What does gradient clipping do when the gradient norm exceeds the threshold?
Question 5 of 5
Why should you save the optimizer state in a checkpoint, not just the model weights?
0
/5
You understand how to train a language model from scratch.
Up next in Arc D
Next: Text Generation — Sampling Strategies ->
Your Mini-GPT is trained. Now learn how to generate text from it — temperature, top-k, nucleus sampling, and the tricks that make LLM outputs feel natural and coherent.