In Lesson 9 you built scaled dot-product attention: Q, K, V matrices, softmax over scores, weighted sum of values. It works. But single-head attention has a fundamental limitation: it produces one set of attention weights per position. That means each token can only "look at" the input in one way.
Consider the sentence: "The cat sat on the mat because it was tired." What does "it" refer to? To answer that, the model needs to simultaneously track:
Syntactic structure: "it" is a pronoun that should agree with a singular noun
Semantic meaning: "tired" is a property of living things, so "it" likely refers to "cat" not "mat"
Positional relationships: "it" is closer to "mat" but grammatically linked to "cat"
A single attention head can only compute one weighted combination. It might focus on syntax or semantics or position -- but not all three at once. Multi-head attention solves this by running multiple attention computations in parallel, each learning to focus on different aspects.
🧠
The key insight
Vaswani et al., 2017
"Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions." In plain English: each head learns a different "question" to ask about the input. Head 1 might learn to ask "what's the subject?", Head 2 might ask "what's the nearest verb?", Head 3 might ask "is there a negation nearby?"
The original transformer used 8 heads with d_model = 512. GPT-3 uses 96 heads with d_model = 12288. More heads means more ways to look at the input simultaneously.
How head splitting works
The trick behind multi-head attention is surprisingly simple: instead of one large attention computation, split Q, K, V into smaller chunks and run attention on each chunk independently.
The math
Suppose you have d_model = 64 (your embedding dimension) and h = 4 heads. Each head gets d_k = d_model / h = 16 dimensions. Here's what happens:
Project: Multiply input X by learned weight matrices W_Q, W_K, W_V (each of shape d_model x d_model) to get Q, K, V
Split: Reshape Q, K, V from (seq_len, 64) to (h, seq_len, 16) -- each head gets a 16-dimensional slice
Attend: Run scaled dot-product attention independently on each head
Concat: Concatenate the h outputs back to (seq_len, 64)
Project: Multiply by W_O (shape d_model x d_model) for the final output
Why split instead of running h separate full-size attentions?
Splitting keeps the total computation constant. One head on 64 dimensions costs 64 x 64 = 4096 operations per position. Four heads on 16 dimensions each cost 4 x (16 x 16) = 1024 operations. The split version is actually cheaper while giving the model more ways to attend.
Multi-head attention flow
Code: single-head vs multi-head
Let's compare single-head and multi-head attention side by side. Same total parameters, but multi-head produces richer attention patterns.
Input X shape: (4, 64)
(seq_len=4, d_model=64)
==================================================
SINGLE-HEAD ATTENTION
==================================================
Q shape: (4, 64)
Output shape: (4, 64)
Attention weights (single head):
Token 0 attends to: [0.25 0.242 0.265 0.243]
Token 1 attends to: [0.237 0.26 0.247 0.256]
→ Only ONE attention pattern for each token
==================================================
MULTI-HEAD ATTENTION (4 heads)
==================================================
After split: each head gets Q/K/V of shape (4, 16)
Head 1 — Token 0 attends to: [0.267 0.241 0.231 0.261]
Head 2 — Token 0 attends to: [0.216 0.281 0.264 0.239]
Head 3 — Token 0 attends to: [0.247 0.252 0.245 0.256]
Head 4 — Token 0 attends to: [0.238 0.274 0.252 0.236]
Concatenated shape: (4, 64)
Final output shape: (4, 64)
→ 4 DIFFERENT attention patterns for each token!
Each head can focus on a different relationship.
With random weights, the differences between heads are subtle. But after training, each head learns dramatically different patterns. In trained GPT-2 models, researchers found heads that specialize in tracking previous tokens, detecting syntax, following coreference chains, and more.
Key observation:
Each head produces a different attention distribution even with the same input. Head 1 might weight token 0 heavily while Head 2 focuses on token 1. After training, these differences become pronounced and semantically meaningful.
Different heads, different patterns
After training, each head in a real transformer learns to detect different linguistic relationships. Here's what researchers have found by visualizing attention patterns in trained models:
This specialization emerges automatically during training. Nobody tells Head 1 to track positions or Head 3 to resolve pronouns. The gradient descent process discovers that having diverse attention patterns across heads produces better predictions.
Code: full multi-head attention
Here's the complete multi-head attention implementation. This is the exact computation that runs inside every transformer layer.
Config: seq_len=5, d_model=64, h=4, d_k=16
Input shape: (5, 64)
Output shape: (5, 64)
→ Same shape! Multi-head attention is shape-preserving.
Head 1 attention (token 'sat' attends to):
The : 0.180 █████
cat : 0.243 ███████
sat : 0.168 █████
on : 0.192 █████
mat : 0.216 ██████
Head 2 attention (token 'sat' attends to):
The : 0.214 ██████
cat : 0.191 █████
sat : 0.215 ██████
on : 0.157 ████
mat : 0.224 ██████
Head 3 attention (token 'sat' attends to):
The : 0.218 ██████
cat : 0.205 ██████
sat : 0.176 █████
on : 0.177 █████
mat : 0.224 ██████
Head 4 attention (token 'sat' attends to):
The : 0.172 █████
cat : 0.216 ██████
sat : 0.218 ██████
on : 0.191 █████
mat : 0.203 ██████
With random weights the patterns are near-uniform, but each head already shows slight differences. After training on real text, these differences become dramatic: some heads become "sharp" (attending to 1-2 tokens) while others stay "broad" (attending to everything).
Parameter counting: same cost, more power
A common misconception is that multi-head attention uses more parameters than single-head. It doesn't. Let's prove it with code.
=======================================================
PARAMETER COUNTING: Single-Head vs Multi-Head
=======================================================
SINGLE-HEAD (d_model=512):
W_Q: 512 x 512 = 262,144 params
W_K: 512 x 512 = 262,144 params
W_V: 512 x 512 = 262,144 params
Total: 786,432 params
MULTI-HEAD (d_model=512, h=8, d_k=64):
W_Q: 512 x 512 = 262,144 params
W_K: 512 x 512 = 262,144 params
W_V: 512 x 512 = 262,144 params
W_O: 512 x 512 = 262,144 params
Total: 1,048,576 params
Difference: W_O adds 262,144 params (25.0% of multi-head total)
But the COMPUTATION per head is 8x cheaper (d_k=64 vs d_model=512)
=======================================================
COMPUTATION per position (matrix multiply cost):
=======================================================
Single-head attention cost per (q,k) pair: 512
Multi-head: 8 heads x 64 dims = 512
Same total cost! But multi-head gets 8 different views.
=======================================================
REAL TRANSFORMER MODELS:
=======================================================
Original Transformer : d= 512, h= 8, d_k= 64, attn params= 1,048,576
GPT-2 Small : d= 768, h=12, d_k= 64, attn params= 2,359,296
GPT-2 Large : d= 1280, h=20, d_k= 64, attn params= 6,553,600
GPT-3 : d=12288, h=96, d_k=128, attn params= 603,979,776
GPT-4 (rumored) : d= 8192, h=64, d_k=128, attn params= 268,435,456
The efficiency insight:
Multi-head attention adds only 25% more parameters (the W_O matrix) compared to single-head, but gains h independent attention patterns. The attention computation itself costs the same because each head operates on d_k = d_model/h dimensions. This is why every transformer uses multi-head attention — it's nearly free to get multiple views of the input.
Why this matters for building an LLM
Multi-head attention is the core of every layer
Every transformer layer contains a multi-head attention block. GPT-2 Small has 12 layers x 12 heads = 144 independent attention computations per forward pass. GPT-3 has 96 layers x 96 heads = 9,216 attention computations. Each one learns a different pattern. Some attend locally (nearby tokens), some globally (beginning of sequence), some to specific syntactic roles. The diversity of these patterns is what gives transformers their representational power.
When researchers prune attention heads (remove them from a trained model), they find that some heads are redundant but others are critical. In one study, removing just 2 heads from a 144-head model caused translation quality to drop by 40%. The model had learned to rely on those specific heads for tracking subject-verb agreement across long distances.
h heads = h different viewssplit + attend + concatsame params as single-headGPT-3: 96 heads per layer
Try It Yourself
InteractiveExplore multi-head attention
Run this code to see multi-head attention in action. Try changing the number of heads, the model dimension, or the input to see how attention patterns differ across heads.
try_multi_head.pyEditable — runs in browser via Pyodide
Click "Run Code" to see the output...
Challenge
Put it into practice
Challenge 1: Vary the head count
Modify the code to run multi-head attention with h = 1, 2, 4, 8, 16, 32 heads (keeping d_model = 64). For each, print d_k and the total parameter count. What happens when h = d_model (64 heads of 1 dimension each)?
When d_k = 1, each head can only compute a scalar attention score (dot product of two scalars). The attention becomes extremely limited. In practice, d_k below 32 starts degrading quality, which is why most models keep d_k = 64 or 128.
Challenge 2: Masked multi-head attention
Add a causal mask to the multi-head attention function. The mask should prevent each position from attending to future positions (set those scores to -infinity before softmax). This is how GPT prevents the model from "cheating" during training by looking ahead.
Create a mask with np.triu(np.ones((seq_len, seq_len)), k=1) (upper triangle). Before softmax, set scores[mask == 1] = -1e9. After softmax, those positions will have weight ~0. Each token can only attend to itself and earlier tokens.
Quiz
Question 1 of 5
Why does multi-head attention use multiple heads instead of a single large attention computation?
Question 2 of 5
If d_model = 512 and h = 8, what is the dimension d_k of each head's Q, K, V?
Question 3 of 5
What happens after all heads compute their individual attention outputs?
Question 4 of 5
How does the total attention computation cost of multi-head compare to single-head attention?
Question 5 of 5
In a trained transformer, what kind of specialization do individual attention heads develop?
0
/5
You understand the mechanism that gives transformers their power.
Next in Arc C
Next: Positional Encoding & Feed-Forward ->
Attention is order-agnostic -- it treats "the cat sat" the same as "sat cat the." Positional encoding fixes that. Then we add the feed-forward network that completes each transformer layer.