Why transformers need position information

In Lesson 10, you built multi-head attention — the core mechanism that lets each token attend to every other token. But there is a problem hiding in plain sight: attention treats input as a set, not a sequence.

An RNN processes "the cat sat" one word at a time — it knows "cat" comes after "the" because it processed "the" first. A transformer, by contrast, processes all tokens simultaneously. Without any additional signal, "the cat sat" and "sat cat the" produce identical attention scores. The model literally cannot tell them apart.

🧪
The key insight (Vaswani et al., 2017)
Position is not built in — it must be injected

The "Attention Is All You Need" paper solved this with a simple but elegant trick: add a position-dependent vector to each token embedding before feeding it into the transformer. The position vectors follow a specific mathematical pattern — sinusoidal functions at different frequencies — so the model can learn to reason about relative positions.

This is positional encoding: a way to stamp each token with "I am at position 0," "I am at position 1," and so on, using the language of sine and cosine waves.

Sinusoidal positional encoding

The original transformer uses a fixed formula to generate position vectors. For a token at position pos and dimension index i:

  • Even dimensions (2i): PE(pos, 2i) = sin(pos / 100002i/d_model)
  • Odd dimensions (2i+1): PE(pos, 2i+1) = cos(pos / 100002i/d_model)

Each dimension oscillates at a different frequency — low dimensions change rapidly across positions, high dimensions change slowly. This creates a unique "fingerprint" for each position that the model can learn to decode.

Why sine and cosine?
Because for any fixed offset k, PE(pos + k) can be expressed as a linear function of PE(pos). This means the model can learn to compute relative positions ("this token is 3 positions after that one") using simple linear operations — exactly what attention heads are good at.
sinusoidal_encoding.py NumPy — build PE from the formula
Output
PE shape: (20, 16) Position 0: [ 0. 1. 0. 1. 0. 1. 0. 1. 0. 1. 0. 1. 0. 1. 0. 1. ] Position 1: [ 0.841 0.54 0.382 0.924 0.158 0.987 0.063 0.998 0.025 1. 0.01 1. 0.004 1. 0.002 1. ] Position 5: [ -0.959 0.284 0.945 0.327 0.717 0.697 0.309 0.951 0.126 0.992 0.05 0.999 0.02 1. 0.008 1. ] All positions unique? True Token embeddings shape: (5, 16) PE slice shape: (5, 16) After adding PE: (5, 16) Without PE, 'cat sat' == 'sat cat' to the model With PE, position 0 gets different signal than position 1
Sinusoidal PE Heatmap Rows = positions, Columns = dimensions. Color = encoding value. Position Dimension 0 1 2 3 4 5 6 7 0 1 2 3 5 9 +1.0 0.0 -1.0 Left dims change fast; right dims change slowly

Notice how the left columns (low dimensions) change rapidly — they oscillate with high frequency. The right columns (high dimensions) barely change at all across 10 positions. This multi-frequency pattern is what makes each position uniquely identifiable.

Learned positional encoding

The original transformer used the fixed sinusoidal formula. But there is an alternative: let the model learn a position embedding for each position, just like it learns token embeddings.

learned_encoding.py NumPy — trainable position embeddings
Output
Learned PE shape: (20, 16) After adding learned PE: (5, 16) --- Sinusoidal vs Learned --- Sinusoidal: fixed formula, works at any length, 0 params Learned: trainable, task-specific, used by GPT-2 & BERT Adds 320 params (20 x 16) Max seq length fixed at training time In practice: learned encoding performs slightly better
What about RoPE?
Modern LLMs like LLaMA use Rotary Position Embeddings (RoPE), which encode position by rotating the query and key vectors. RoPE combines the best of both worlds: it is a fixed formula (no learned parameters), but it naturally encodes relative position. We will revisit RoPE in a later lesson when we build a full GPT.

The feed-forward network (FFN)

Each transformer layer has two sublayers: multi-head attention (Lesson 10), and a position-wise feed-forward network. The FFN is applied to each token independently — the same weights, but each token gets its own forward pass.

The FFN is two linear transformations with a ReLU activation in between:

FFN(x) = max(0, xW1 + b1)W2 + b2

feed_forward.py NumPy — FFN with ReLU
Output
Input shape: (5, 16) After W1: (5, 64) (expanded to 64 dims) After ReLU: (5, 64) (154 of 320 values zeroed) After W2: (5, 16) (back to 16 dims) FFN output: (5, 16) Same shape as input? True Token 0 output: [-0.0157 0.1517 0.0195 -0.1331]... Token 1 output: [ 0.0685 -0.0192 0.0649 0.0165]... The FFN gives each token its own 'thinking step' Attention mixes tokens; FFN processes each one alone
Why expand to 4x?
The inner dimension d_ff is typically 4 times d_model. In GPT-2, d_model=768 and d_ff=3072. The expansion gives the network more capacity to learn complex per-token transformations, then the second linear layer compresses back. Think of it as: attention decides what information to gather, the FFN decides what to do with it.

Layer normalization from scratch

Training deep networks is unstable — activations can explode or vanish as they pass through many layers. Layer normalization fixes this by normalizing the activations at each layer to have zero mean and unit variance.

Unlike batch normalization (which normalizes across the batch), layer norm normalizes across the feature dimension for each individual token. This makes it work with variable-length sequences and small batch sizes.

layer_norm.py NumPy — LayerNorm implementation
Output
Before LN — Token 0 mean: 3.1627, std: 4.9553 Before LN — Token 1 mean: 3.7431, std: 5.6012 After LN — Token 0 mean: 0.000000, std: 1.0000 After LN — Token 1 mean: 0.000000, std: 1.0000 With gamma=2, beta=0.5 — mean: 0.5000, std: 2.0000 BatchNorm normalizes across the batch; LayerNorm across features. For variable-length text, LayerNorm works much better.

Residual connections: the skip highway

The transformer is deep — GPT-2 has 12 layers, GPT-3 has 96. Without help, gradients would vanish long before they reach the bottom layers. Residual connections (also called skip connections) solve this by adding the input directly to the output of each sublayer:

output = LayerNorm(x + sublayer(x))

The + x means: even if the sublayer learns nothing useful, the signal passes through unchanged. Gradients can flow directly through the addition, bypassing the sublayer entirely during backpropagation.

residual_connections.py NumPy — residual + LayerNorm
Output
Input x: (5, 16), norm = 0.5209 FFN(x): (5, 16), norm = 0.6381 LN(x + FFN(x)): (5, 16), norm = 4.9249 --- Why Residual Connections Matter --- Without: gradient must flow through FFN (may vanish) With: d(x + f(x))/dx = 1 + f'(x) Even if f'(x)->0, gradient is at least 1 This lets GPT-3 stack 96 layers without vanishing gradients
Residual Connection Flow (One Transformer Sublayer) x (input) Sublayer (Attention or FFN) sublayer(x) Skip connection + x + Layer Normalization output output = LN(x + sublayer(x))

How it all fits together in a real transformer

Every transformer layer applies two sublayers in sequence, each wrapped with a residual connection and layer norm. This is the complete flow for one layer:

1. x = LayerNorm(x + MultiHeadAttention(x))
2. x = LayerNorm(x + FFN(x))

Stack 12 of these (GPT-2) or 96 (GPT-3), and you have the transformer backbone. The residual connections ensure gradients flow through all layers. The layer norms keep activations stable. The attention layers mix information between tokens. The FFN layers process each token independently.

PE + Embedding Multi-Head Attention Residual + LayerNorm FFN Residual + LayerNorm

Try It Yourself

Interactive Build the full transformer sublayer pipeline

Run this code to see positional encoding, FFN, layer norm, and residual connections working together — the building blocks you will assemble into a full transformer block in the next lesson.

try_pe_ffn.py Editable — runs in browser via Pyodide
Click "Run Code" to see the output...

Challenge

Put it into practice
Challenge 1: Verify the relative position property

The sinusoidal encoding has the property that PE(pos + k) can be expressed as a linear combination of PE(pos). Compute the dot product between PE(0) and PE(1), between PE(5) and PE(6), and between PE(10) and PE(11). Are they similar? This shows the encoding captures relative distance.

Use np.dot(pe[0], pe[1]) and compare with np.dot(pe[5], pe[6]). The dot products of adjacent positions should be similar regardless of absolute position — that is the relative position property.
Challenge 2: Pre-norm vs post-norm

The original transformer uses "post-norm": LayerNorm(x + sublayer(x)). GPT-2 uses "pre-norm": x + sublayer(LayerNorm(x)). Implement pre-norm and compare the output norms. Which keeps values more stable?

For pre-norm: normed = layer_norm(x, gamma, beta); output = x + ffn(normed, W1, b1, W2, b2). Notice that in pre-norm, the residual is added after LayerNorm, so the skip connection carries unnormalized values — this tends to be more stable for deep networks.

Quiz

Question 1 of 5
Why do transformers need positional encoding?
Question 2 of 5
In sinusoidal positional encoding, what happens at even dimension indices (2i)?
Question 3 of 5
What is the structure of the position-wise feed-forward network in a transformer?
Question 4 of 5
How does a residual connection help with training deep transformers?
Question 5 of 5
What does layer normalization do in a transformer?
0
/5
You've mastered the building blocks of a transformer layer.
Next in Arc C

Next: Building the Transformer Block ->

Combine multi-head attention, positional encoding, FFN, LayerNorm, and residual connections into a complete, working transformer layer.

Next: Building the Transformer Block ->
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You've built the transformer's foundation.
Now assemble the full architecture.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →