Why attention solves the bottleneck

In Lesson 8, we saw the fatal flaw of seq2seq: the entire input gets crushed into a single fixed-size context vector. Whether the input is 3 tokens or 300 tokens, the decoder has the same small vector to work from. Long sentences lose information. Early tokens get overwritten.

The solution is simple and elegant: let the decoder look back at every encoder hidden state, not just the final one. At each decoding step, the decoder decides which encoder states are most relevant and creates a custom-weighted combination of them.

🧪
The breakthrough (2014-2015)
Bahdanau, Cho & Bengio

In 2014, Dzmitry Bahdanau published "Neural Machine Translation by Jointly Learning to Align and Translate." Instead of compressing the entire input into one vector, the decoder attends to all encoder hidden states at each step. When generating the French word "chats," the decoder pays high attention to the English word "cats" and low attention to "I" and "love."

This single idea — attention — is arguably the most important concept in modern AI. It led directly to the Transformer (2017), which led to BERT, GPT, and every LLM you use today.

Seq2Seq (Lesson 8): Decoder sees only one vector
Encoder states
->
Single context vector
->
Decoder
With Attention: Decoder sees ALL encoder states
Encoder states
->
Weighted combination (per step)
->
Decoder
The key insight:
Instead of one fixed summary, attention creates a different summary for each decoder step. When producing the first output token, the decoder might focus on encoder states 3 and 4. When producing the second output token, it might focus on encoder state 1. The "summary" adapts to what's needed right now.

Query, key, value: the intuition

Attention uses three concepts borrowed from information retrieval: queries, keys, and values. The analogy is a database lookup — but a "soft" one.

The dictionary analogy

Think of a Python dictionary. You have a key, you look it up, you get the exact matching value. That's a hard lookup — either the key matches exactly or it doesn't.

Attention is a soft lookup. Your query doesn't match any key exactly. Instead, it matches every key a little bit, and you get a weighted blend of all values. Keys that match the query well get high weight; keys that don't match get low weight.

Query x Key -> Weights -> Weighted Values QUERY "What am I looking for?" KEY 1 "cats" KEY 2 "love" KEY 3 "I" 0.85 0.10 0.05 softmax weights VAL 1 v_cats VAL 2 v_love VAL 3 v_I OUTPUT 0.85*v_cats + 0.10*v_love + ... Query asks "what am I looking for?" -- Keys answer "here's what I have" Dot product measures similarity. Softmax normalizes to weights. Values carry the information. Output = weighted sum of values (a "soft" database lookup)
  • Query (Q): "What am I looking for?" — the current decoder state, asking for relevant information.
  • Key (K): "Here's what I have" — each encoder state advertises its content. The dot product between query and key measures how relevant that encoder state is.
  • Value (V): "Here's the actual information" — the content to extract. In simple attention, keys and values are the same (encoder hidden states). In the Transformer, they're separate learned projections.
Why separate K and V?
Think of a library catalog. The key is the catalog entry (title, subject tags) — it helps you find the right book. The value is the book's actual content. You search by key, but you read the value. Separating them lets the model learn different representations for "findability" versus "information content."

Code: simple attention from scratch

Let's start with the simplest form of attention — what Bahdanau introduced. The decoder has a query (its current hidden state). The encoder provides keys and values (its hidden states at each position). We compute dot products, apply softmax, and take a weighted sum.

simple_attention.py NumPy — dot-product attention
Output
Encoder hidden states: h[0] 'I': [ 0.497 -0.139 0.648 1.523] h[1] 'love': [-0.234 -0.234 1.579 -0.7 ] h[2] 'cats': [ 0.313 -1.245 -0.606 -1.516] Decoder query: [-0.202 0.319 0.856 -0.076] Raw attention scores (Q . K): 'I': 0.3340 'love': 1.2666 'cats': -0.2478 Attention weights (after softmax): 'I': 0.2416 'love': 0.6147 <-- highest 'cats': 0.1437 Sum: 1.0000 (always 1.0) Context vector (weighted sum of values): [ 0.0208 -0.3707 1.0392 -0.0464] This context vector is SPECIFIC to this decoder step. A different query would produce different weights and a different context.

That's the entire attention mechanism in 10 lines. The decoder's query is compared to each encoder key via dot product. Softmax turns the raw scores into a probability distribution. The weighted sum of values produces a context vector tailored to this specific decoder step.

No more bottleneck!
Unlike seq2seq where the decoder gets one fixed context vector, with attention the decoder gets a fresh, custom context vector at every step. When producing "chats" (French for "cats"), the attention mechanism assigns high weight to the encoder state for "cats." When producing "j'aime" ("I love"), it shifts attention to "love." The decoder can access any part of the input at any time.

Scaled dot-product attention

The simple dot-product attention has a problem. When the dimension d_k is large (say, 512), the dot products become very large numbers. Large inputs to softmax push it into regions where gradients are extremely small — the model stops learning.

The scaling fix

The Transformer paper ("Attention Is All You Need," Vaswani et al. 2017) introduced a simple fix: divide the dot products by sqrt(d_k) before applying softmax.

Scaled Dot-Product Attention Formula:
Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) V

Why sqrt(d_k)? If Q and K have entries drawn from a distribution with mean 0 and variance 1, their dot product has mean 0 and variance d_k. Dividing by sqrt(d_k) brings the variance back to 1, keeping softmax inputs in the well-behaved range.

scaled_attention.py NumPy — full scaled dot-product attention
Output
Scaled Dot-Product Attention ================================================== Q shape: (3, 8) (3 output queries) K shape: (4, 8) (4 input keys) V shape: (4, 8) (4 input values) Output shape: (3, 8) (3 output vectors) Attention weights (which input does each output attend to?): I love cats J'aime 0.321 0.187 0.198 0.294 ** les 0.178 0.250 0.382 ** 0.190 chats 0.264 0.171 0.336 ** 0.229 ** = highest attention for that output token Why scaling matters (d_k=512): Dot product (d=512): -9.30 After scaling /sqrt(512): -0.41 Without scaling, softmax sees extreme values -> vanishing gradients
Random weights alert:
The attention weights above are from random Q, K, V — they don't show meaningful translation patterns. In a trained model, "chats" would strongly attend to "cats" (weight near 0.8+), and "J'aime" would attend to "I" and "love." The mechanism is correct; the patterns emerge through training.

Attention weights as a heatmap

Attention weights form a matrix: rows are output positions, columns are input positions. Each row sums to 1.0 (it's a probability distribution). Visualizing this matrix as a heatmap shows which inputs each output "attends to."

Attention Weight Heatmap (Trained Model) English -> French translation I love cats <eos> J'aime les chats 0.45 0.40 0.10 0.05 0.15 0.20 0.55 0.10 0.05 0.05 0.85 0.05 Darker = higher attention weight. Each row sums to 1.0 "chats" strongly attends to "cats" (0.85) -- the model learned alignment!

In a trained translation model, the heatmap reveals interpretable patterns. "chats" (French for "cats") pays 85% of its attention to "cats." "J'aime" (French for "I love") splits attention between "I" and "love." The model learned to align source and target tokens — without being explicitly told which words correspond.

Code: attention weights distribution

Let's see how scaling affects the attention weight distribution. Without scaling, large dimensions cause softmax to produce near-one-hot distributions (one weight close to 1, rest near 0). With scaling, the distribution stays smooth.

attention_weights_distribution.py NumPy — effect of scaling on attention
Output
How d_k affects attention weight distributions ======================================================= d_k = 4: Token Unscaled Scaled ────── ────────── ────────── The 0.1250 0.1688 cat 0.2345 0.2280 sat 0.0543 0.1196 on 0.0247 0.0921 mat 0.5615 0.3916 Max weight: 0.5615 0.3916 Entropy: 1.3421 1.4879 d_k = 64: Token Unscaled Scaled ────── ────────── ────────── The 0.0000 0.0913 cat 0.0000 0.0744 sat 0.0001 0.1277 on 0.9999 0.5470 mat 0.0000 0.1596 Max weight: 0.9999 0.5470 Entropy: 0.0017 1.3297 d_k = 512: Token Unscaled Scaled ────── ────────── ────────── The 0.0000 0.1299 cat 0.0000 0.2363 sat 1.0000 0.3710 on 0.0000 0.1434 mat 0.0000 0.1195 Max weight: 1.0000 0.3710 Entropy: 0.0000 1.5405 ======================================================= Without scaling: as d_k grows, weights become near-one-hot -> softmax saturates -> gradients vanish -> learning stops With scaling: weights stay smooth regardless of dimension -> healthy gradients -> model keeps learning

At d_k=512, unscaled attention gives weight 1.0000 to one token and 0.0000 to everything else. The model can only attend to one token — it loses the ability to blend information. Scaling restores a smooth distribution (max weight 0.37) where the model can attend to multiple tokens with different strengths.

Try It Yourself

Interactive Compute attention weights with NumPy

Run this code to compute scaled dot-product attention. Try changing the seed, dimensions, or sequence length to see how attention weights change.

try_attention.py Editable — runs in browser via Pyodide
Click "Run Code" to see the output...

From attention to transformers to GPT

Why this is the most important idea in modern AI

The attention mechanism you just implemented is the foundation of every modern LLM. Here's the direct lineage:

2014: Bahdanau attention fixes the seq2seq bottleneck. Machine translation improves dramatically.

2017: Vaswani et al. ask "what if we use only attention, with no RNNs at all?" The Transformer is born. Self-attention lets every token attend to every other token — in parallel.

2018-2024: BERT, GPT, GPT-2, GPT-3, GPT-4, Claude, Llama — all built on the same scaled dot-product attention you implemented above.

Bahdanau attention (2014) Transformer (2017) self-attention softmax(QK^T/sqrt(d))V GPT = stacked attention

In the next lesson, you'll learn about multi-head attention — running multiple attention operations in parallel, each focusing on different aspects of the input. One head might track syntax, another semantics, another positional relationships. Multi-head attention is what makes transformers so powerful.

The key formula to remember:
Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) V. This single line of math is the beating heart of every GPT, Claude, and Llama model. You've now implemented it from scratch.

Common mistakes

  • Forgetting to scale: Without dividing by sqrt(d_k), softmax saturates in high dimensions. The model attends to only one token and gradients vanish. Always scale.
  • Confusing keys and values: Keys determine where to attend (via dot product with query). Values determine what information to extract. In simple attention they can be the same, but in transformers they're separate learned projections.
  • Forgetting softmax normalization: Raw dot products aren't probabilities. Softmax ensures weights are non-negative and sum to 1.0 — making attention a valid weighted average.
  • Applying softmax along the wrong axis: Each query gets its own distribution over keys. Softmax must be applied along the key dimension (last axis), not across queries.
  • Thinking attention is only for translation: Attention was invented for machine translation, but it generalizes. Self-attention (used in GPT) lets tokens attend to other tokens within the same sequence. That's the Transformer's core innovation.

Challenge

Put it into practice
Challenge 1: Implement masked attention

In autoregressive models (like GPT), the model should only attend to previous tokens, not future ones. Modify scaled_dot_product_attention to accept a boolean mask and set masked positions to -inf before softmax (so they get weight 0).

Create a lower-triangular matrix with np.tril(np.ones((n, n))). Where the mask is 0, set scores to -1e9 (a large negative number). Softmax will turn these into ~0.
Challenge 2: Compare additive vs dot-product attention

Bahdanau's original attention used additive attention: score = v^T tanh(W_q q + W_k k) instead of dot product. Implement both and compare: which is faster? Which produces smoother weight distributions?

For additive attention, you need weight matrices W_q (d, d_attn), W_k (d, d_attn), and a vector v (d_attn,). The score is v @ np.tanh(W_q @ q + W_k @ k). Dot-product is faster because it's a single matrix multiply; additive requires more operations but can be more expressive for mismatched Q/K dimensions.

Key takeaways

  • Attention eliminates the bottleneck: Instead of one fixed context vector, the decoder gets a fresh weighted combination of encoder states at each step.
  • Query, key, value: The query asks "what am I looking for?" Keys answer "here's what I have." Values carry the actual information. Dot product measures query-key similarity.
  • Scaling is essential: Dividing by sqrt(d_k) prevents softmax from saturating in high dimensions, keeping gradients healthy.
  • Attention weights are interpretable: The weight matrix shows which inputs each output attends to — a "soft alignment" learned automatically.
  • This formula powers every LLM: softmax(QK^T / sqrt(d_k)) V is the core of GPT, Claude, Llama, and every transformer-based model.

Quiz

Question 1 of 5
In scaled dot-product attention, why do we divide by sqrt(d_k)?
Question 2 of 5
What role does the "query" play in the attention mechanism?
Question 3 of 5
What does the softmax function ensure about attention weights?
Question 4 of 5
How does attention solve the seq2seq bottleneck problem?
Question 5 of 5
In the formula Attention(Q,K,V) = softmax(QK^T / sqrt(d_k))V, what does the matrix QK^T represent?
0
/5
You understand the attention mechanism that powers every modern LLM.
Next in Arc C

Next: Multi-Head Attention ->

Run multiple attention operations in parallel — one head for syntax, another for semantics, another for position. The key to transformer power.

Next: Multi-Head Attention ->
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You've implemented attention from scratch.
Now build the full transformer.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →