Why transformers need position information
In Lesson 10, you built multi-head attention — the core mechanism that lets each token attend to every other token. But there is a problem hiding in plain sight: attention treats input as a set, not a sequence.
An RNN processes "the cat sat" one word at a time — it knows "cat" comes after "the" because it processed "the" first. A transformer, by contrast, processes all tokens simultaneously. Without any additional signal, "the cat sat" and "sat cat the" produce identical attention scores. The model literally cannot tell them apart.
The "Attention Is All You Need" paper solved this with a simple but elegant trick: add a position-dependent vector to each token embedding before feeding it into the transformer. The position vectors follow a specific mathematical pattern — sinusoidal functions at different frequencies — so the model can learn to reason about relative positions.
This is positional encoding: a way to stamp each token with "I am at position 0," "I am at position 1," and so on, using the language of sine and cosine waves.
Sinusoidal positional encoding
The original transformer uses a fixed formula to generate position vectors. For a token at position pos and dimension index i:
- Even dimensions (2i): PE(pos, 2i) = sin(pos / 100002i/d_model)
- Odd dimensions (2i+1): PE(pos, 2i+1) = cos(pos / 100002i/d_model)
Each dimension oscillates at a different frequency — low dimensions change rapidly across positions, high dimensions change slowly. This creates a unique "fingerprint" for each position that the model can learn to decode.
Notice how the left columns (low dimensions) change rapidly — they oscillate with high frequency. The right columns (high dimensions) barely change at all across 10 positions. This multi-frequency pattern is what makes each position uniquely identifiable.
Learned positional encoding
The original transformer used the fixed sinusoidal formula. But there is an alternative: let the model learn a position embedding for each position, just like it learns token embeddings.
The feed-forward network (FFN)
Each transformer layer has two sublayers: multi-head attention (Lesson 10), and a position-wise feed-forward network. The FFN is applied to each token independently — the same weights, but each token gets its own forward pass.
The FFN is two linear transformations with a ReLU activation in between:
FFN(x) = max(0, xW1 + b1)W2 + b2
d_ff is typically 4 times d_model. In GPT-2, d_model=768 and d_ff=3072. The expansion gives the network more capacity to learn complex per-token transformations, then the second linear layer compresses back. Think of it as: attention decides what information to gather, the FFN decides what to do with it.
Layer normalization from scratch
Training deep networks is unstable — activations can explode or vanish as they pass through many layers. Layer normalization fixes this by normalizing the activations at each layer to have zero mean and unit variance.
Unlike batch normalization (which normalizes across the batch), layer norm normalizes across the feature dimension for each individual token. This makes it work with variable-length sequences and small batch sizes.
Residual connections: the skip highway
The transformer is deep — GPT-2 has 12 layers, GPT-3 has 96. Without help, gradients would vanish long before they reach the bottom layers. Residual connections (also called skip connections) solve this by adding the input directly to the output of each sublayer:
output = LayerNorm(x + sublayer(x))
The + x means: even if the sublayer learns nothing useful, the signal passes through unchanged. Gradients can flow directly through the addition, bypassing the sublayer entirely during backpropagation.
How it all fits together in a real transformer
Every transformer layer applies two sublayers in sequence, each wrapped with a residual connection and layer norm. This is the complete flow for one layer:
1. x = LayerNorm(x + MultiHeadAttention(x))
2. x = LayerNorm(x + FFN(x))
Stack 12 of these (GPT-2) or 96 (GPT-3), and you have the transformer backbone. The residual connections ensure gradients flow through all layers. The layer norms keep activations stable. The attention layers mix information between tokens. The FFN layers process each token independently.
Try It Yourself
Run this code to see positional encoding, FFN, layer norm, and residual connections working together — the building blocks you will assemble into a full transformer block in the next lesson.
Challenge
The sinusoidal encoding has the property that PE(pos + k) can be expressed as a linear combination of PE(pos). Compute the dot product between PE(0) and PE(1), between PE(5) and PE(6), and between PE(10) and PE(11). Are they similar? This shows the encoding captures relative distance.
np.dot(pe[0], pe[1]) and compare with np.dot(pe[5], pe[6]). The dot products of adjacent positions should be similar regardless of absolute position — that is the relative position property.The original transformer uses "post-norm": LayerNorm(x + sublayer(x)). GPT-2 uses "pre-norm": x + sublayer(LayerNorm(x)). Implement pre-norm and compare the output norms. Which keeps values more stable?
normed = layer_norm(x, gamma, beta); output = x + ffn(normed, W1, b1, W2, b2). Notice that in pre-norm, the residual is added after LayerNorm, so the skip connection carries unnormalized values — this tends to be more stable for deep networks.Quiz
You've built the transformer's foundation.
Now assemble the full architecture.
Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Agentic Engineer →