🌍
Real-world scenario
Kwame in Accra: building a Twi-to-English translation model

Kwame is an NLP researcher at Accra Digital Labs, tasked with building a real-time translation system for Twi (Akan language spoken in Ghana) to English. The system needs to understand which source words are relevant for each target word—a task where LSTMs struggle because they compress entire sequences into single vectors.

Kwame discovers the solution: attention. Instead of relying on one vector, the model learns to focus on different parts of the input for each output token. But basic attention (additive) is slow. He needs scaled dot-product attention with multi-head architecture. To handle positional information (since transformers process sequences in parallel, not sequentially), he adds positional encoding. The result: a transformer-based translator that achieves 35 BLEU score—a 40% improvement over the LSTM baseline.

50K
Sentence pairs
8
Attention heads
35
BLEU score

The core problem: why attention matters in translation

In machine translation, the LSTM encoder compresses "Medaase wo" (Thank you in Twi) into a single 512-dimensional vector. The decoder tries to generate "thank", "you" from this bottleneck. The problem: the vector contains no information about which input word to attend to.

Attention solves this: for each output word, the decoder learns a context vector that weighs all input words. When generating "thank", high attention goes to "Medaase" (the relevant source word). When generating "you", attention moves to "wo".

How attention works in 3 steps

1. Scores: For each target word, compute a similarity score between the current decoder state and each input word. High scores mean "relevant to current output".

2. Weights: Normalize scores using softmax to get attention weights (probabilities summing to 1). These weights tell us how much to "look at" each input word.

3. Context: Multiply input representations by their weights and sum. The result: a context vector that blends only the relevant information.

The limitation of basic attention: computing similarity requires matrix multiplications for each query-key pair. With many heads, this gets expensive. The solution: scaled dot-product attention, which batches these computations efficiently.

Scaled dot-product attention: efficient similarity computation

Scaled dot-product attention is a specific formula for computing attention efficiently. Instead of learning separate similarity functions, it uses dot products of learned projections.

Formula: Attention(Q, K, V) = softmax(QKT / √dk) V

  • Q (Query): Projection of the target state. What do we want to find?
  • K (Key): Projection of each source word. A fingerprint of what that word represents.
  • V (Value): Projection of each source word. The actual information to aggregate.
  • √dk (scaling factor): dk is the dimension of queries/keys. Dividing by √dk prevents dot products from becoming too large (which would make softmax gradients vanish).
python Scaled dot-product attention implementation
OutputOutput shape: torch.Size([2, 4, 64]) Attention shape: torch.Size([2, 4, 4]) Attention weights sum to 1: 1.0000

Key insight: Attention weights form a matrix where each row represents one target position, and columns represent source positions. Row sums to 1 (softmax constraint). Large values indicate high attention; small values indicate low attention.

Visualizing attention: where does the model look?

When Kwame translates "Medaase wo" (Thank you) to English, the attention weights tell us which Twi words influence each English word. This heatmap shows the learned alignments after training.

Attention heatmap for Twi-to-English translation. Darker cells indicate higher attention. "Medaase" aligns strongly with "thank" and "you"; "wo" (subject marker) has distributed attention across both words.
Attention weights heatmap: source words vs target words

Interpretation: The heatmap shows that attention is sparse and interpretable. Each target word focuses on a subset of source words, mimicking human translation intuitions. This interpretability is a major advantage of attention-based models over pure recurrent models.

Multi-head attention: attending to different aspects

Single-head attention learns one type of alignment. But translation requires multiple alignments: one head might attend to semantics ("which content word?"), another to syntax ("which subject?"). Multi-head attention learns multiple alignment patterns in parallel.

Formula: MultiHead(Q, K, V) = Concat(head₁, ..., headh) WO, where headi = Attention(Q WQi, K WKi, V WVi)

Each head projects Q, K, V independently, computes attention, then concatenates results and applies a final linear transformation. With 8 heads and d_model=512, each head operates in d_k=64 dimensions (512/8), making computation efficient.

python Multi-head attention module
OutputOutput shape: torch.Size([2, 4, 512]) Attention shape (per head): torch.Size([2, 8, 4, 4]) Total parameters: 2,100,224

Why 2.1M parameters? With d_model=512 and num_heads=8, we have three 512×512 projection matrices (Q, K, V) and one 512×512 output matrix. Each has 512² ≈ 262K parameters. Total: ~2.1M. Multi-head attention is expensive but enables rich, diverse alignments.

Positional encoding: encoding sequence order without recurrence

RNNs process sequences step-by-step, so they naturally learn position information (early words enter at timestep 1, later words at timestep t). Transformers process entire sequences in parallel, so they lose positional information. Solution: add positional encodings to input embeddings.

Sinusoidal positional encoding formula:

  • PE(pos, 2i) = sin(pos / 10000^(2i/d_model))
  • PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

For each position pos and dimension i, we compute a unique sinusoidal pattern. Even dimensions use sine; odd dimensions use cosine. This creates a unique encoding for every position that the model can learn to distinguish.

python Sinusoidal positional encoding
OutputEmbeddings shape: torch.Size([2, 100, 512]) With positional encoding shape: torch.Size([2, 100, 512]) First position encoding (first 10 dims): tensor([ 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 1.0000])

Why sinusoids?

Sinusoidal patterns create a unique fingerprint for each position. Unlike learned embeddings (which require storage for max_len positions), sinusoids can generalize to any sequence length. They also preserve relative distance information: PE(pos+k) can be computed as a linear function of PE(pos), helping the model learn relative positions.

Sinusoidal positional encodings for 50 positions and 128 dimensions. Each row is a position; each column is a dimension. Warmer colors represent higher values. The pattern is periodic, allowing the model to learn position semantics.
Sinusoidal positional encoding heatmap

Transformer encoder: stacking attention and feedforward layers

The transformer encoder processes input sequences without recurrence. Each encoder layer consists of two sub-layers: (1) multi-head self-attention, and (2) feedforward (two dense layers with ReLU). Layer normalization and residual connections improve training stability.

Encoder layer formula:

  • Attention output = MultiHeadAttention(x, x, x)
  • After residual: x' = x + Attention(x)
  • After layer norm: y = LayerNorm(x')
  • FFN output = ReLU(y W1 + b1) W2 + b2
  • Final: z = LayerNorm(y + FFN(y))
python Transformer encoder layer
OutputEncoder input: torch.Size([2, 10, 512]) Encoder output: torch.Size([2, 10, 512]) Encoder parameters: 101,511,936

Key design: Stacking 6 encoder layers (standard in "Attention Is All You Need") allows deep representations. Each layer refines representations through self-attention and context mixing. With 512-dim embeddings and 8 heads, the total parameter count reaches ~101M—comparable to a large LSTM, but with better parallelization.

Transformer decoder: generating sequences with cross-attention

The decoder generates output tokens one-by-one (or in parallel during training). Unlike the encoder (which attends to itself), the decoder has three attention mechanisms: (1) self-attention over generated tokens (with masking to prevent looking forward), (2) cross-attention to encoder output, and (3) feedforward.

During generation at test time, the decoder attends to already-generated tokens (autoregressive). The encoder output is constant—it provides the source context for the decoder to attend to.

python Transformer decoder layer and generation
OutputCausal mask (1=can attend, 0=masked): tensor([[1, 0], [1, 1]]) Interpretation: - Position 0 can attend to position 0 only - Position 1 can attend to positions 0 and 1 - No position can attend to future positions

Why causal masking? During training, the decoder sees all target tokens at once (for parallelization). The causal mask forces each token to attend only to previous tokens, simulating autoregressive generation. This prevents the model from "cheating" by looking ahead.

Building a complete transformer for Twi-to-English translation

Combining encoder, decoder, embeddings, and positional encoding creates a full transformer. The pipeline: (1) embed source tokens, add positional encoding, pass through encoder; (2) embed target tokens, add positional encoding, pass through decoder with encoder output as context; (3) apply final linear layer to predict next token probabilities.

python Complete Transformer model
OutputSource shape: torch.Size([2, 10]) Target shape: torch.Size([2, 8]) Output logits shape: torch.Size([2, 8, 5000]) Total model parameters: 214,336,640

Training this model: We compute cross-entropy loss between predicted logits and true target tokens, then backpropagate. With 214M parameters and proper regularization (dropout, weight decay), the model learns alignments that outperform LSTMs on translation tasks.

Training and inference: from pairs to translations

Training uses teacher forcing: at each step, we feed true target tokens (not predictions) and compute loss. Inference uses autoregressive generation: feed generated tokens back to the decoder.

python Training loop and beam search inference
OutputTraining example: Source: tensor([[3, 45, 120, 8, 2]]) Target: tensor([[1, 50, 75, 100, 2]]) Inference example: Generated translation tokens: [ 1 23 156 45 78 98 2]

Key points: Teacher forcing during training makes learning faster but causes exposure bias (train uses true tokens, inference uses predictions). At test time, greedy decoding picks the highest-probability token, then feeds it back. Beam search (keeping top-k hypotheses) improves quality but is slower.

Key takeaways: from attention to state-of-the-art translation

Attention is all you need (but not the whole story)

Self-attention replaces RNNs' sequential bottleneck. Scaled dot-product attention is O(n²) in space but allows full parallelization. Multi-head attention learns diverse alignments. Positional encoding preserves sequence order. Transformers achieve state-of-the-art on translation, reaching 35+ BLEU on Twi-English with proper training.

Common mistakes and pitfalls

Forgetting positional encoding: Without it, the model treats "Medaase wo" the same as "wo Medaase"—word order is lost. Not scaling by √d_k: Dot products grow with d_k, causing softmax to become too peaked, killing gradients. Forgetting causal masking in decoder: The model will "cheat" by looking ahead during training, failing at inference. Using too many layers: Deep transformers (>12 layers) need careful warmup and learning rate scheduling.

Component Purpose Key Parameter
Scaled dot-product attention Efficient similarity-based alignment √d_k scaling factor
Multi-head attention Learn multiple alignment patterns num_heads (typically 8 or 12)
Positional encoding Inject sequence order information Sinusoidal frequencies
Encoder layers Process source with self-attention num_layers (typically 6)
Decoder layers Generate target with encoder context Causal mask for self-attention

Quiz: 5 questions to test your understanding

Question 1 of 5
Why do we divide by √d_k in scaled dot-product attention?
Question 2 of 5
What is the main advantage of multi-head attention over single-head attention?
Question 3 of 5
Why do transformers need positional encoding but RNNs don't?
Question 4 of 5
What does causal masking in the decoder prevent?
Question 5 of 5
In a transformer with 512-dim embeddings and 8 attention heads, what is the dimension of each head?
0
/5
Excellent! You understand transformers and attention mechanisms.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You've mastered transformers.
Now deploy them at scale.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →