Why decoder-only?

In the previous lesson you built the full encoder-decoder Transformer — the original architecture from "Attention Is All You Need." It works beautifully for tasks like translation, where you have a source sequence (English) and a target sequence (French). The encoder reads the source; the decoder generates the target.

But language modeling is different. You don't have two sequences. You have one sequence, and your job is to predict the next token given everything before it. There's no "source" to encode — just a stream of text to continue.

🔑
The key insight
No encoder needed for autoregressive generation

When your only task is "predict the next word," the encoder is dead weight. You don't need cross-attention to read a source sequence — there is no source sequence. All you need is the decoder: a stack of masked self-attention blocks that process the text left-to-right and predict what comes next.

This is what OpenAI realized with GPT-1 (2018). By removing the encoder and training a decoder-only Transformer on raw text, they created a model that could generate coherent paragraphs, answer questions, and even translate — all from a single architecture trained on a single objective: predict the next token.

Encoder
X
Cross-Attention
X
Decoder Stack
->
GPT!
Full Transformer vs GPT — what gets removed:
The encoder stack (all N layers), the cross-attention sub-layer in each decoder block (which attended to encoder output), and the separate source embedding. What remains: token embedding, positional embedding, N decoder blocks (each with masked self-attention + FFN + LayerNorm), and the output projection.

Causal masking — no peeking at the future

The most important mechanism in GPT is the causal mask. In the encoder-decoder Transformer, the encoder can see all positions (no mask), while the decoder uses a causal mask to prevent future-peeking. In GPT, since there's only a decoder, every attention operation is causally masked.

The causal mask is a lower-triangular matrix. Position i can attend to positions 0, 1, ..., i but not to positions i+1, i+2, etc. This ensures the model can only use information from the past — exactly what we need for next-token prediction.

causal_mask.py Read-only — run locally with PyTorch
Expected output
Causal mask (1=attend, 0=blocked): tensor([[1, 0, 0, 0, 0], [1, 1, 0, 0, 0], [1, 1, 1, 0, 0], [1, 1, 1, 1, 0], [1, 1, 1, 1, 1]]) Token 0 can see: [0] Token 1 can see: [0, 1] Token 2 can see: [0, 1, 2] Token 3 can see: [0, 1, 2, 3] Token 4 can see: [0, 1, 2, 3, 4] This is what makes GPT autoregressive — each position only depends on the past.
Training vs inference:
During training, we process the entire sequence at once (all positions in parallel) and the mask prevents cheating. During inference, we generate one token at a time — position 5 doesn't exist yet, so the mask is natural. The mask lets us train in parallel but generate sequentially.

Autoregressive generation

GPT generates text by autoregressive sampling: predict the next token, append it to the input, and repeat. Each step uses all previous tokens as context.

Autoregressive Generation — Predict, Append, Repeat Step 1: The GPT cat Step 2: The cat GPT sat Step 3: The cat sat GPT on Each step: feed all tokens so far -> predict next -> append -> repeat

This loop continues until the model produces a special end-of-sequence token or reaches a maximum length. The key insight: the same model and the same weights handle both "understanding" (via attention to context) and "generation" (via the output projection). There is no separate encoder for understanding — the decoder does everything.

Why this works so well:
By training on next-token prediction across billions of tokens, the model learns grammar, facts, reasoning patterns, and even code. The training objective is simple (predict the next word), but the emergent capabilities are extraordinary. This is the foundation of GPT, Claude, Gemini, and Llama.

GPT architecture diagram

Compare this to the full Transformer from the previous lesson. The encoder side is gone. Cross-attention is gone. What remains is clean and elegant — a single stack of identical blocks.

GPT — Decoder-Only Architecture No encoder. No cross-attention. Just the decoder stack. Input Tokens "The cat sat on" Token Embedding + Position Embedding N x Masked Multi-Head Self-Attention (Causal) CAUSAL MASK Add & Layer Norm Feed-Forward Network (FFN) Add & Layer Norm Final Layer Norm Linear + Softmax Next Token Probabilities P("the")=0.3, P("mat")=0.6, ... Weight tying (shared) = Add & Norm (residual) = Residual connection = Weight tying
Compared to the full Transformer:
GPT has only 2 sub-layers per block (masked self-attention + FFN) instead of 3 (the cross-attention is gone). It also uses learned positional embeddings instead of sinusoidal, and applies a final LayerNorm after the last block before the output projection. Weight tying shares the token embedding matrix with the output projection.

CausalSelfAttention class

This is the core attention module for GPT. Unlike the full Transformer's MultiHeadAttention (which could do self-attention or cross-attention), this only does causal self-attention — Q, K, V all come from the same input, and the causal mask is always applied.

causal_self_attention.py Read-only — run locally with PyTorch
Expected output
Input shape: torch.Size([2, 10, 128]) Output shape: torch.Size([2, 10, 128]) Parameters: 66,048
Key design choices:
(1) Combined QKV projection — instead of 3 separate nn.Linear layers, we use one and split the output. This is faster on GPUs. (2) The causal mask is registered as a buffer, not a parameter — it's part of the model state but doesn't receive gradients. (3) Separate dropout for attention weights and the residual path.

GPTBlock — one decoder layer

Each GPT block has exactly 2 sub-layers: masked self-attention and a feed-forward network. Both have residual connections and layer normalization. GPT-2 introduced pre-norm (applying LayerNorm before the sub-layer, not after) — this stabilizes training for deep models.

gpt_block.py Read-only — run locally with PyTorch
Expected output
Input shape: torch.Size([2, 10, 128]) Output shape: torch.Size([2, 10, 128]) Block params: 197,760 Sub-layer breakdown: LayerNorm x2: 512 Attention: 66,048 FFN: 131,200
GPT-1 vs GPT-2 normalization:
GPT-1 used post-norm (LayerNorm after the residual add), like the original Transformer. GPT-2 switched to pre-norm (LayerNorm before the sub-layer), which makes training much more stable for deep models. Our implementation follows GPT-2's pre-norm convention — it's now the standard.

The full MiniGPT model

Now we assemble everything into a complete GPT model. This is the architecture you'll train in the next lesson — a small but fully functional language model.

mini_gpt.py Read-only — run locally with PyTorch
Expected output
Input shape: torch.Size([2, 32]) Output shape: torch.Size([2, 32, 10000]) Prompt length: 3 Generated length: 8 Generated tokens: [1, 2, 3, 7841, 2093, 5512, 9147, 3380]
The generate() method is autoregressive:
It runs the model, takes the logits at the last position, samples a token, appends it to the sequence, and repeats. The temperature parameter controls randomness: lower = more deterministic, higher = more creative. This exact loop is what ChatGPT, Claude, and every LLM uses at inference time.

Mini-GPT parameter count

Let's count every parameter in our mini-GPT to understand where the model's capacity lives.

count_mini_gpt_params.py Read-only — parameter breakdown
Expected output
=== Mini-GPT Parameter Count === Config: vocab=10000, d_model=128, heads=4, layers=4, max_seq=256 Token embedding: 1,280,000 Position embedding: 32,768 Per block: 197,760 Total blocks (4): 791,040 Final LayerNorm: 256 Total (no weight tying): 3,384,064 Total (with weight tying): 2,104,064

GPT-1, GPT-2, GPT-3 — same architecture, different scale

The remarkable thing about GPT is that GPT-1, GPT-2, GPT-3, and even GPT-4 use fundamentally the same decoder-only architecture. The differences are scale: more layers, wider dimensions, more data, and more compute.

gpt_comparison.py Read-only — model comparison
Expected output
=== GPT Model Comparison === Model Params Layers d_model Heads Data GPT-1 117M 12 768 12 ~5GB books GPT-2 1.5B 48 1600 25 WebText 40GB GPT-3 175B 96 12288 96 ~570GB mixed GPT-4 ~1.8T* ~120 ~? ~? ~13T tokens

What changed between versions

  • GPT-1 (2018): Proved decoder-only pretraining works. 12 layers, 768 dimensions, trained on BookCorpus (~5GB). Fine-tuned per task. The key contribution was showing that unsupervised pretraining + supervised fine-tuning beats task-specific architectures.
  • GPT-2 (2019): Showed that scale unlocks zero-shot abilities. Switched to pre-norm (LayerNorm before sub-layers). Trained on WebText (40GB from Reddit links). The 1.5B parameter model could write coherent essays — OpenAI initially withheld it, worried about misuse.
  • GPT-3 (2020): Demonstrated emergent abilities at 175B parameters — few-shot learning, arithmetic, code generation — without any fine-tuning. Same architecture as GPT-2, just 100x larger. Trained on a 570GB mix of books, web pages, and Wikipedia.
  • GPT-4 (2023): Rumored ~1.8 trillion parameters (mixture of experts). Multimodal (text + images). Architecture details are not public, but the core is still a decoder-only Transformer.
Emergent abilities:
When GPT-3 scaled to 175B parameters, abilities appeared that weren't present in smaller models — multi-step reasoning, translation between languages it wasn't explicitly trained on, and even basic arithmetic. This phenomenon, called "emergence," suggests that scale itself can unlock new capabilities without architectural changes.

Our mini-GPT spec and config

For the rest of this track, we'll train and experiment with a mini-GPT that's small enough to train on a laptop but large enough to learn interesting patterns.

🔧
Our training spec
Mini-GPT Configuration
ParameterOur Mini-GPTGPT-2 Small
vocab_size10,00050,257
d_model128768
n_heads412
n_layers412
max_seq_len2561024
d_ff (= 4 * d_model)5123072
Parameters (weight-tied)~2.1M~124M
Training time (laptop CPU)~5 minutesDays (GPU needed)

With 2.1M parameters, our mini-GPT is about 60x smaller than GPT-2 Small. But it uses the exact same architecture — every component is identical, just with smaller dimensions. In the next lesson, you'll train it on real text and see it generate coherent sentences.

Run it yourself

Run MiniGPT locally

Copy the full MiniGPT class from above and run it on your machine:

pip install torch
python mini_gpt.py

Try modifying the config — change d_model to 256 or n_layers to 8 and watch how the parameter count scales. The relationship between dimension, layers, and total parameters is key to understanding why GPT-3 has 175 billion parameters.

Why this matters for LLMs

You just built the architecture behind every modern LLM

The MiniGPT class you see above is, structurally, the same model as ChatGPT, Claude, Gemini, and Llama. The differences are scale (more layers, wider dimensions, more data) and training methodology (RLHF, instruction tuning) — not architecture.

In the next lesson, you'll train this model on real text and watch it go from random noise to generating coherent English. That's the moment when everything clicks — you'll see the exact same learning curve that OpenAI observed when training GPT-1.

MiniGPT: 2.1M params GPT-2: 1.5B params GPT-3: 175B params Same architecture, different scale

Challenge

Put it into practice
Challenge 1: Scale the config

Modify the MiniGPT config to match GPT-2 Small (d_model=768, n_heads=12, n_layers=12, max_seq_len=1024, vocab=50257). Count the parameters — you should get approximately 124M. How does the ratio of embedding parameters to block parameters change compared to our mini-GPT?

With vocab=50257 and d_model=768, the token embedding alone is 38.6M parameters. The blocks contribute about 85M. In our mini-GPT, embedding is 1.28M out of 2.1M total (~61%). In GPT-2 Small, it's 38.6M out of 124M (~31%). As models get wider, the blocks dominate.
Challenge 2: Implement temperature sampling

The generate() method accepts a temperature parameter. Try calling it with temperature=0.5 (more focused), 1.0 (default), and 2.0 (more random). Print the top-5 probabilities at each temperature for the same input. What happens to the probability distribution as temperature changes?

Dividing logits by a low temperature (0.5) before softmax sharpens the distribution — the top token gets nearly all the probability. A high temperature (2.0) flattens the distribution — all tokens become more equally likely. At temperature=0, it's equivalent to always picking the highest-probability token (greedy decoding).

Quiz

Question 1 of 5
Why doesn't GPT need an encoder?
Question 2 of 5
What does the causal mask do in GPT's self-attention?
Question 3 of 5
In autoregressive generation, what happens after GPT predicts the next token?
Question 4 of 5
What key innovation did GPT-2 introduce compared to GPT-1?
Question 5 of 5
What is "weight tying" in our MiniGPT implementation?
0
/5
You understand the architecture behind every modern LLM.
Up next in Arc D

Next: Training Your Mini-GPT ->

You've built the architecture. Now train it on real text and watch it go from random noise to generating coherent English sentences. Loss curves, learning rate schedules, and the magic moment when language emerges.

Next: Training Your Mini-GPT ->
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You've built GPT from scratch.
Now build the career to match.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →