Why one head isn't enough

In Lesson 9 you built scaled dot-product attention: Q, K, V matrices, softmax over scores, weighted sum of values. It works. But single-head attention has a fundamental limitation: it produces one set of attention weights per position. That means each token can only "look at" the input in one way.

Consider the sentence: "The cat sat on the mat because it was tired." What does "it" refer to? To answer that, the model needs to simultaneously track:

  • Syntactic structure: "it" is a pronoun that should agree with a singular noun
  • Semantic meaning: "tired" is a property of living things, so "it" likely refers to "cat" not "mat"
  • Positional relationships: "it" is closer to "mat" but grammatically linked to "cat"

A single attention head can only compute one weighted combination. It might focus on syntax or semantics or position -- but not all three at once. Multi-head attention solves this by running multiple attention computations in parallel, each learning to focus on different aspects.

🧠
The key insight
Vaswani et al., 2017

"Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions." In plain English: each head learns a different "question" to ask about the input. Head 1 might learn to ask "what's the subject?", Head 2 might ask "what's the nearest verb?", Head 3 might ask "is there a negation nearby?"

The original transformer used 8 heads with d_model = 512. GPT-3 uses 96 heads with d_model = 12288. More heads means more ways to look at the input simultaneously.

How head splitting works

The trick behind multi-head attention is surprisingly simple: instead of one large attention computation, split Q, K, V into smaller chunks and run attention on each chunk independently.

The math

Suppose you have d_model = 64 (your embedding dimension) and h = 4 heads. Each head gets d_k = d_model / h = 16 dimensions. Here's what happens:

  1. Project: Multiply input X by learned weight matrices W_Q, W_K, W_V (each of shape d_model x d_model) to get Q, K, V
  2. Split: Reshape Q, K, V from (seq_len, 64) to (h, seq_len, 16) -- each head gets a 16-dimensional slice
  3. Attend: Run scaled dot-product attention independently on each head
  4. Concat: Concatenate the h outputs back to (seq_len, 64)
  5. Project: Multiply by W_O (shape d_model x d_model) for the final output
Why split instead of running h separate full-size attentions?
Splitting keeps the total computation constant. One head on 64 dimensions costs 64 x 64 = 4096 operations per position. Four heads on 16 dimensions each cost 4 x (16 x 16) = 1024 operations. The split version is actually cheaper while giving the model more ways to attend.

Multi-head attention flow

Input X (seq_len, 64) W_Q (64x64) W_K (64x64) W_V (64x64) SPLIT into 4 heads (each 16-dim) Head 1 Attention(Q1,K1,V1) Head 2 Attention(Q2,K2,V2) Head 3 Attention(Q3,K3,V3) Head 4 Attention(Q4,K4,V4) (seq, 16) (seq, 16) (seq, 16) (seq, 16) CONCAT along last dim Concat (seq_len, 64) W_O -> Output (seq, 64)

Code: single-head vs multi-head

Let's compare single-head and multi-head attention side by side. Same total parameters, but multi-head produces richer attention patterns.

single_vs_multi_head.py NumPy — comparing attention approaches
Output
Input X shape: (4, 64) (seq_len=4, d_model=64) ================================================== SINGLE-HEAD ATTENTION ================================================== Q shape: (4, 64) Output shape: (4, 64) Attention weights (single head): Token 0 attends to: [0.25 0.242 0.265 0.243] Token 1 attends to: [0.237 0.26 0.247 0.256] → Only ONE attention pattern for each token ================================================== MULTI-HEAD ATTENTION (4 heads) ================================================== After split: each head gets Q/K/V of shape (4, 16) Head 1 — Token 0 attends to: [0.267 0.241 0.231 0.261] Head 2 — Token 0 attends to: [0.216 0.281 0.264 0.239] Head 3 — Token 0 attends to: [0.247 0.252 0.245 0.256] Head 4 — Token 0 attends to: [0.238 0.274 0.252 0.236] Concatenated shape: (4, 64) Final output shape: (4, 64) → 4 DIFFERENT attention patterns for each token! Each head can focus on a different relationship.

With random weights, the differences between heads are subtle. But after training, each head learns dramatically different patterns. In trained GPT-2 models, researchers found heads that specialize in tracking previous tokens, detecting syntax, following coreference chains, and more.

Key observation:
Each head produces a different attention distribution even with the same input. Head 1 might weight token 0 heavily while Head 2 focuses on token 1. After training, these differences become pronounced and semantically meaningful.

Different heads, different patterns

After training, each head in a real transformer learns to detect different linguistic relationships. Here's what researchers have found by visualizing attention patterns in trained models:

Trained attention heads learn different roles HEAD 1: Position "previous token" The cat sat down 0.7 0.8 0.7 Diagonal pattern Each token looks back 1 HEAD 2: Syntax "subject-verb" The cat sat down 0.8 0.1 Verb finds subject "sat" attends to "cat" HEAD 3: Coreference "pronoun resolution" cat sat it was 0.9 0.1 Pronoun finds referent "it" attends to "cat" HEAD 4: Global "broad context" The cat sat down 0.25 0.25 0.25 0.25 Uniform attention Captures global context

This specialization emerges automatically during training. Nobody tells Head 1 to track positions or Head 3 to resolve pronouns. The gradient descent process discovers that having diverse attention patterns across heads produces better predictions.

Code: full multi-head attention

Here's the complete multi-head attention implementation. This is the exact computation that runs inside every transformer layer.

multi_head_attention.py NumPy — complete implementation
Output
Config: seq_len=5, d_model=64, h=4, d_k=16 Input shape: (5, 64) Output shape: (5, 64) → Same shape! Multi-head attention is shape-preserving. Head 1 attention (token 'sat' attends to): The : 0.180 █████ cat : 0.243 ███████ sat : 0.168 █████ on : 0.192 █████ mat : 0.216 ██████ Head 2 attention (token 'sat' attends to): The : 0.214 ██████ cat : 0.191 █████ sat : 0.215 ██████ on : 0.157 ████ mat : 0.224 ██████ Head 3 attention (token 'sat' attends to): The : 0.218 ██████ cat : 0.205 ██████ sat : 0.176 █████ on : 0.177 █████ mat : 0.224 ██████ Head 4 attention (token 'sat' attends to): The : 0.172 █████ cat : 0.216 ██████ sat : 0.218 ██████ on : 0.191 █████ mat : 0.203 ██████

With random weights the patterns are near-uniform, but each head already shows slight differences. After training on real text, these differences become dramatic: some heads become "sharp" (attending to 1-2 tokens) while others stay "broad" (attending to everything).

Parameter counting: same cost, more power

A common misconception is that multi-head attention uses more parameters than single-head. It doesn't. Let's prove it with code.

parameter_counting.py NumPy — proving parameter equivalence
Output
======================================================= PARAMETER COUNTING: Single-Head vs Multi-Head ======================================================= SINGLE-HEAD (d_model=512): W_Q: 512 x 512 = 262,144 params W_K: 512 x 512 = 262,144 params W_V: 512 x 512 = 262,144 params Total: 786,432 params MULTI-HEAD (d_model=512, h=8, d_k=64): W_Q: 512 x 512 = 262,144 params W_K: 512 x 512 = 262,144 params W_V: 512 x 512 = 262,144 params W_O: 512 x 512 = 262,144 params Total: 1,048,576 params Difference: W_O adds 262,144 params (25.0% of multi-head total) But the COMPUTATION per head is 8x cheaper (d_k=64 vs d_model=512) ======================================================= COMPUTATION per position (matrix multiply cost): ======================================================= Single-head attention cost per (q,k) pair: 512 Multi-head: 8 heads x 64 dims = 512 Same total cost! But multi-head gets 8 different views. ======================================================= REAL TRANSFORMER MODELS: ======================================================= Original Transformer : d= 512, h= 8, d_k= 64, attn params= 1,048,576 GPT-2 Small : d= 768, h=12, d_k= 64, attn params= 2,359,296 GPT-2 Large : d= 1280, h=20, d_k= 64, attn params= 6,553,600 GPT-3 : d=12288, h=96, d_k=128, attn params= 603,979,776 GPT-4 (rumored) : d= 8192, h=64, d_k=128, attn params= 268,435,456
The efficiency insight:
Multi-head attention adds only 25% more parameters (the W_O matrix) compared to single-head, but gains h independent attention patterns. The attention computation itself costs the same because each head operates on d_k = d_model/h dimensions. This is why every transformer uses multi-head attention — it's nearly free to get multiple views of the input.

Why this matters for building an LLM

Multi-head attention is the core of every layer

Every transformer layer contains a multi-head attention block. GPT-2 Small has 12 layers x 12 heads = 144 independent attention computations per forward pass. GPT-3 has 96 layers x 96 heads = 9,216 attention computations. Each one learns a different pattern. Some attend locally (nearby tokens), some globally (beginning of sequence), some to specific syntactic roles. The diversity of these patterns is what gives transformers their representational power.

When researchers prune attention heads (remove them from a trained model), they find that some heads are redundant but others are critical. In one study, removing just 2 heads from a 144-head model caused translation quality to drop by 40%. The model had learned to rely on those specific heads for tracking subject-verb agreement across long distances.

h heads = h different views split + attend + concat same params as single-head GPT-3: 96 heads per layer

Try It Yourself

Interactive Explore multi-head attention

Run this code to see multi-head attention in action. Try changing the number of heads, the model dimension, or the input to see how attention patterns differ across heads.

try_multi_head.py Editable — runs in browser via Pyodide
Click "Run Code" to see the output...

Challenge

Put it into practice
Challenge 1: Vary the head count

Modify the code to run multi-head attention with h = 1, 2, 4, 8, 16, 32 heads (keeping d_model = 64). For each, print d_k and the total parameter count. What happens when h = d_model (64 heads of 1 dimension each)?

When d_k = 1, each head can only compute a scalar attention score (dot product of two scalars). The attention becomes extremely limited. In practice, d_k below 32 starts degrading quality, which is why most models keep d_k = 64 or 128.
Challenge 2: Masked multi-head attention

Add a causal mask to the multi-head attention function. The mask should prevent each position from attending to future positions (set those scores to -infinity before softmax). This is how GPT prevents the model from "cheating" during training by looking ahead.

Create a mask with np.triu(np.ones((seq_len, seq_len)), k=1) (upper triangle). Before softmax, set scores[mask == 1] = -1e9. After softmax, those positions will have weight ~0. Each token can only attend to itself and earlier tokens.

Quiz

Question 1 of 5
Why does multi-head attention use multiple heads instead of a single large attention computation?
Question 2 of 5
If d_model = 512 and h = 8, what is the dimension d_k of each head's Q, K, V?
Question 3 of 5
What happens after all heads compute their individual attention outputs?
Question 4 of 5
How does the total attention computation cost of multi-head compare to single-head attention?
Question 5 of 5
In a trained transformer, what kind of specialization do individual attention heads develop?
0
/5
You understand the mechanism that gives transformers their power.
Next in Arc C

Next: Positional Encoding & Feed-Forward ->

Attention is order-agnostic -- it treats "the cat sat" the same as "sat cat the." Positional encoding fixes that. Then we add the feed-forward network that completes each transformer layer.

Next: Positional Encoding & Feed-Forward ->
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You've split attention into multiple heads.
Now build the full transformer layer.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →