In the previous lesson you built the full encoder-decoder Transformer — the original architecture from "Attention Is All You Need." It works beautifully for tasks like translation, where you have a source sequence (English) and a target sequence (French). The encoder reads the source; the decoder generates the target.
But language modeling is different. You don't have two sequences. You have one sequence, and your job is to predict the next token given everything before it. There's no "source" to encode — just a stream of text to continue.
🔑
The key insight
No encoder needed for autoregressive generation
When your only task is "predict the next word," the encoder is dead weight. You don't need cross-attention to read a source sequence — there is no source sequence. All you need is the decoder: a stack of masked self-attention blocks that process the text left-to-right and predict what comes next.
This is what OpenAI realized with GPT-1 (2018). By removing the encoder and training a decoder-only Transformer on raw text, they created a model that could generate coherent paragraphs, answer questions, and even translate — all from a single architecture trained on a single objective: predict the next token.
Encoder
X
Cross-Attention
X
Decoder Stack
->
GPT!
Full Transformer vs GPT — what gets removed:
The encoder stack (all N layers), the cross-attention sub-layer in each decoder block (which attended to encoder output), and the separate source embedding. What remains: token embedding, positional embedding, N decoder blocks (each with masked self-attention + FFN + LayerNorm), and the output projection.
Causal masking — no peeking at the future
The most important mechanism in GPT is the causal mask. In the encoder-decoder Transformer, the encoder can see all positions (no mask), while the decoder uses a causal mask to prevent future-peeking. In GPT, since there's only a decoder, every attention operation is causally masked.
The causal mask is a lower-triangular matrix. Position i can attend to positions 0, 1, ..., i but not to positions i+1, i+2, etc. This ensures the model can only use information from the past — exactly what we need for next-token prediction.
causal_mask.pyRead-only — run locally with PyTorch
Expected output
Causal mask (1=attend, 0=blocked):
tensor([[1, 0, 0, 0, 0],
[1, 1, 0, 0, 0],
[1, 1, 1, 0, 0],
[1, 1, 1, 1, 0],
[1, 1, 1, 1, 1]])
Token 0 can see: [0]
Token 1 can see: [0, 1]
Token 2 can see: [0, 1, 2]
Token 3 can see: [0, 1, 2, 3]
Token 4 can see: [0, 1, 2, 3, 4]
This is what makes GPT autoregressive —
each position only depends on the past.
Training vs inference:
During training, we process the entire sequence at once (all positions in parallel) and the mask prevents cheating. During inference, we generate one token at a time — position 5 doesn't exist yet, so the mask is natural. The mask lets us train in parallel but generate sequentially.
Autoregressive generation
GPT generates text by autoregressive sampling: predict the next token, append it to the input, and repeat. Each step uses all previous tokens as context.
This loop continues until the model produces a special end-of-sequence token or reaches a maximum length. The key insight: the same model and the same weights handle both "understanding" (via attention to context) and "generation" (via the output projection). There is no separate encoder for understanding — the decoder does everything.
Why this works so well:
By training on next-token prediction across billions of tokens, the model learns grammar, facts, reasoning patterns, and even code. The training objective is simple (predict the next word), but the emergent capabilities are extraordinary. This is the foundation of GPT, Claude, Gemini, and Llama.
GPT architecture diagram
Compare this to the full Transformer from the previous lesson. The encoder side is gone. Cross-attention is gone. What remains is clean and elegant — a single stack of identical blocks.
Compared to the full Transformer:
GPT has only 2 sub-layers per block (masked self-attention + FFN) instead of 3 (the cross-attention is gone). It also uses learned positional embeddings instead of sinusoidal, and applies a final LayerNorm after the last block before the output projection. Weight tying shares the token embedding matrix with the output projection.
CausalSelfAttention class
This is the core attention module for GPT. Unlike the full Transformer's MultiHeadAttention (which could do self-attention or cross-attention), this only does causal self-attention — Q, K, V all come from the same input, and the causal mask is always applied.
causal_self_attention.pyRead-only — run locally with PyTorch
(1) Combined QKV projection — instead of 3 separate nn.Linear layers, we use one and split the output. This is faster on GPUs. (2) The causal mask is registered as a buffer, not a parameter — it's part of the model state but doesn't receive gradients. (3) Separate dropout for attention weights and the residual path.
GPTBlock — one decoder layer
Each GPT block has exactly 2 sub-layers: masked self-attention and a feed-forward network. Both have residual connections and layer normalization. GPT-2 introduced pre-norm (applying LayerNorm before the sub-layer, not after) — this stabilizes training for deep models.
GPT-1 used post-norm (LayerNorm after the residual add), like the original Transformer. GPT-2 switched to pre-norm (LayerNorm before the sub-layer), which makes training much more stable for deep models. Our implementation follows GPT-2's pre-norm convention — it's now the standard.
The full MiniGPT model
Now we assemble everything into a complete GPT model. This is the architecture you'll train in the next lesson — a small but fully functional language model.
It runs the model, takes the logits at the last position, samples a token, appends it to the sequence, and repeats. The temperature parameter controls randomness: lower = more deterministic, higher = more creative. This exact loop is what ChatGPT, Claude, and every LLM uses at inference time.
Mini-GPT parameter count
Let's count every parameter in our mini-GPT to understand where the model's capacity lives.
=== Mini-GPT Parameter Count ===
Config: vocab=10000, d_model=128, heads=4, layers=4, max_seq=256
Token embedding: 1,280,000
Position embedding: 32,768
Per block: 197,760
Total blocks (4): 791,040
Final LayerNorm: 256
Total (no weight tying): 3,384,064
Total (with weight tying): 2,104,064
GPT-1, GPT-2, GPT-3 — same architecture, different scale
The remarkable thing about GPT is that GPT-1, GPT-2, GPT-3, and even GPT-4 use fundamentally the same decoder-only architecture. The differences are scale: more layers, wider dimensions, more data, and more compute.
GPT-1 (2018): Proved decoder-only pretraining works. 12 layers, 768 dimensions, trained on BookCorpus (~5GB). Fine-tuned per task. The key contribution was showing that unsupervised pretraining + supervised fine-tuning beats task-specific architectures.
GPT-2 (2019): Showed that scale unlocks zero-shot abilities. Switched to pre-norm (LayerNorm before sub-layers). Trained on WebText (40GB from Reddit links). The 1.5B parameter model could write coherent essays — OpenAI initially withheld it, worried about misuse.
GPT-3 (2020): Demonstrated emergent abilities at 175B parameters — few-shot learning, arithmetic, code generation — without any fine-tuning. Same architecture as GPT-2, just 100x larger. Trained on a 570GB mix of books, web pages, and Wikipedia.
GPT-4 (2023): Rumored ~1.8 trillion parameters (mixture of experts). Multimodal (text + images). Architecture details are not public, but the core is still a decoder-only Transformer.
Emergent abilities:
When GPT-3 scaled to 175B parameters, abilities appeared that weren't present in smaller models — multi-step reasoning, translation between languages it wasn't explicitly trained on, and even basic arithmetic. This phenomenon, called "emergence," suggests that scale itself can unlock new capabilities without architectural changes.
Our mini-GPT spec and config
For the rest of this track, we'll train and experiment with a mini-GPT that's small enough to train on a laptop but large enough to learn interesting patterns.
🔧
Our training spec
Mini-GPT Configuration
Parameter
Our Mini-GPT
GPT-2 Small
vocab_size
10,000
50,257
d_model
128
768
n_heads
4
12
n_layers
4
12
max_seq_len
256
1024
d_ff (= 4 * d_model)
512
3072
Parameters (weight-tied)
~2.1M
~124M
Training time (laptop CPU)
~5 minutes
Days (GPU needed)
With 2.1M parameters, our mini-GPT is about 60x smaller than GPT-2 Small. But it uses the exact same architecture — every component is identical, just with smaller dimensions. In the next lesson, you'll train it on real text and see it generate coherent sentences.
Run it yourself
Run MiniGPT locally
Copy the full MiniGPT class from above and run it on your machine:
pip install torch
python mini_gpt.py
Try modifying the config — change d_model to 256 or n_layers to 8 and watch how the parameter count scales. The relationship between dimension, layers, and total parameters is key to understanding why GPT-3 has 175 billion parameters.
Why this matters for LLMs
You just built the architecture behind every modern LLM
The MiniGPT class you see above is, structurally, the same model as ChatGPT, Claude, Gemini, and Llama. The differences are scale (more layers, wider dimensions, more data) and training methodology (RLHF, instruction tuning) — not architecture.
In the next lesson, you'll train this model on real text and watch it go from random noise to generating coherent English. That's the moment when everything clicks — you'll see the exact same learning curve that OpenAI observed when training GPT-1.
MiniGPT: 2.1M paramsGPT-2: 1.5B paramsGPT-3: 175B paramsSame architecture, different scale
Challenge
Put it into practice
Challenge 1: Scale the config
Modify the MiniGPT config to match GPT-2 Small (d_model=768, n_heads=12, n_layers=12, max_seq_len=1024, vocab=50257). Count the parameters — you should get approximately 124M. How does the ratio of embedding parameters to block parameters change compared to our mini-GPT?
With vocab=50257 and d_model=768, the token embedding alone is 38.6M parameters. The blocks contribute about 85M. In our mini-GPT, embedding is 1.28M out of 2.1M total (~61%). In GPT-2 Small, it's 38.6M out of 124M (~31%). As models get wider, the blocks dominate.
Challenge 2: Implement temperature sampling
The generate() method accepts a temperature parameter. Try calling it with temperature=0.5 (more focused), 1.0 (default), and 2.0 (more random). Print the top-5 probabilities at each temperature for the same input. What happens to the probability distribution as temperature changes?
Dividing logits by a low temperature (0.5) before softmax sharpens the distribution — the top token gets nearly all the probability. A high temperature (2.0) flattens the distribution — all tokens become more equally likely. At temperature=0, it's equivalent to always picking the highest-probability token (greedy decoding).
Quiz
Question 1 of 5
Why doesn't GPT need an encoder?
Question 2 of 5
What does the causal mask do in GPT's self-attention?
Question 3 of 5
In autoregressive generation, what happens after GPT predicts the next token?
Question 4 of 5
What key innovation did GPT-2 introduce compared to GPT-1?
Question 5 of 5
What is "weight tying" in our MiniGPT implementation?
0
/5
You understand the architecture behind every modern LLM.
Up next in Arc D
Next: Training Your Mini-GPT ->
You've built the architecture. Now train it on real text and watch it go from random noise to generating coherent English sentences. Loss curves, learning rate schedules, and the magic moment when language emerges.