Full Transformer Architecture

The original "Attention Is All You Need" architecture with all components labeled. The encoder processes the source sequence; the decoder generates the target sequence one token at a time.

The Transformer — Encoder-Decoder Architecture "Attention Is All You Need" (Vaswani et al., 2017) ENCODER Input Tokens "The cat sat on the mat" Input Embedding + Positional Encoding Nx Multi-Head Self-Attention Add & Layer Norm Feed-Forward Network (FFN) Add & Layer Norm Encoder Output K, V from encoder DECODER Output Tokens (shifted right) "<start> Le chat ..." Output Embedding + Positional Encoding Nx Masked Multi-Head Self-Attention CAUSAL Add & Layer Norm Multi-Head Cross-Attention Add & Layer Norm Feed-Forward Network (FFN) Add & Layer Norm Linear + Softmax Output Probabilities P("Le")=0.01, P("chat")=0.85, ... Add & Norm Residual Cross-attention path
How to read this diagram:
Data flows bottom-to-top. The encoder (left, green) processes the full input sequence with bidirectional self-attention. The decoder (right, purple) generates output one token at a time using causal (masked) self-attention, then cross-attends to the encoder output for context. Both sides repeat their blocks N times.

Key Formulas

The six equations that define the transformer. Memorize these and you can reconstruct the entire architecture from scratch.

Scaled Dot-Product Attention
Attention(Q, K, V) = softmax(Q Kᵀ / √d᮸) V
Q = queries, K = keys, V = values, d_k = key dimension. The scaling factor 1/√d_k prevents dot products from growing too large as dimensions increase, which would push softmax into regions with vanishing gradients.
Multi-Head Attention
MultiHead(Q, K, V) = Concat(head₁, ..., headₙ) Wᴼ
where headᵢ = Attention(Q WᵢẐ, K Wᵢᵊ, V Wᵢᵡ)
Each head learns a different attention pattern. W_i^Q, W_i^K, W_i^V are per-head projection matrices (d_model x d_k). W^O is the output projection (h*d_k x d_model). With h heads and d_k = d_model/h, the total compute is similar to single-head attention with full dimensionality.
Position-wise Feed-Forward Network
FFN(x) = max(0, x W₁ + b₁) W₂ + b₂
Two linear transformations with a ReLU activation in between. W_1 expands from d_model to d_ff (typically 4x), then W_2 projects back down. Applied to each position independently — same weights, different positions. GPT-2+ uses GELU instead of ReLU.
Layer Normalization
LayerNorm(x) = γ · (x - μ) / (σ + ε) + β
μ and σ are the mean and standard deviation computed across the feature dimension (d_model) for each token independently. γ (scale) and β (shift) are learned parameters with shape (d_model,). ε is a small constant (e.g., 1e-5) for numerical stability.
Positional Encoding (sinusoidal)
PE(pos, 2i) = sin(pos / 10000^(2i / d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i / d_model))
pos = token position in the sequence, i = dimension index. Even dimensions use sin, odd dimensions use cos. Each dimension oscillates at a different frequency, allowing the model to learn relative positions. GPT uses learned positional embeddings instead of these sinusoidal encodings.
When to use which positional encoding:
The original transformer used sinusoidal (fixed) encodings. GPT-1/2/3 use learned positional embeddings. Modern architectures like LLaMA and Gemini use Rotary Position Embeddings (RoPE), which encode relative positions directly into the attention computation.

Parameter Counting

Know exactly where every parameter lives. This table shows the parameter count for each component in a single transformer block, then the full model.

🔢
Per-component breakdown
Parameters in a single transformer block
ComponentParameter CountNotes
Embedding vocab_size x d_model One-time cost. Dominates small models.
Position Embedding max_seq_len x d_model Learned (GPT) or zero params (sinusoidal).
Attention Q, K, V 3 x (d_model x d_k) x n_heads = 3 x d_model x d_model (since d_k = d_model / n_heads).
Attention output proj d_model x d_model W^O concatenates heads back.
Attention biases 3 x d_model + d_model = 4 x d_model (for QKV + output).
FFN layer 1 d_model x d_ff + d_ff Expand. Typically d_ff = 4 x d_model.
FFN layer 2 d_ff x d_model + d_model Contract back to d_model.
LayerNorm (x2) 2 x (2 x d_model) = 4 x d_model. Two norms per block (gamma + beta each).
Total per block (with d_ff = 4 x d_model)
params_per_block = 12 x d_model² + 13 x d_model
Breakdown: Attention = 4 x d_model^2 + 4 x d_model. FFN = 8 x d_model^2 + 5 x d_model. LayerNorm = 4 x d_model. For d_model = 768 (GPT-2 Small): 12 x 768^2 + 13 x 768 = 7,087,872 + 9,984 = 7,097,856 per block.
Total model parameters
total = vocab x d_model + seq x d_model + n_layers x params_per_block + 2 x d_model
Embedding + position embedding + all blocks + final LayerNorm. If weight tying is used (GPT-2/3), the output projection shares weights with the embedding, saving vocab_size x d_model parameters.
Rule of thumb:
For large models, approximately 2/3 of per-block parameters are in the FFN and 1/3 are in attention. The embedding layer dominates for small models (large vocab, small d_model) but becomes a minor fraction for large models (GPT-3's embedding is 12288 x 50257 = 617M out of 175B total = 0.35%).

Tensor Shapes Cheatsheet

Track the exact shape of every tensor as data flows through the transformer. This is the single most useful debugging reference when implementing from scratch.

Step Tensor Shape
Input token IDs input_ids (batch, seq_len)
After embedding x (batch, seq_len, d_model)
After positional encoding x (batch, seq_len, d_model)
Q, K, V (before reshape) q, k, v (batch, seq_len, d_model)
Q, K, V (per head) q, k, v (batch, n_heads, seq_len, d_k)
Attention scores Q K^T / sqrt(d_k) (batch, n_heads, seq_len, seq_len)
Attention weights softmax(scores) (batch, n_heads, seq_len, seq_len)
Attention output (per head) weights @ V (batch, n_heads, seq_len, d_k)
Concat heads + W^O attn_out (batch, seq_len, d_model)
FFN intermediate hidden (batch, seq_len, d_ff)
FFN output ffn_out (batch, seq_len, d_model)
After all blocks x (batch, seq_len, d_model)
Output logits logits (batch, seq_len, vocab_size)
The most common shape bug:
Forgetting to transpose after the multi-head split. You need (batch, seq_len, n_heads, d_k) -> (batch, n_heads, seq_len, d_k) via .transpose(1, 2). Without this, the attention matrix multiplications produce the wrong shapes and PyTorch will either throw an error or — worse — silently compute garbage.
Cross-attention shapes:
In the decoder's cross-attention layer, Q comes from the decoder (batch, n_heads, tgt_len, d_k) but K and V come from the encoder (batch, n_heads, src_len, d_k). The attention scores are (batch, n_heads, tgt_len, src_len) — each decoder position attends to all encoder positions.

GPT vs BERT vs T5

Three ways to use the transformer architecture. Same building blocks, different configurations, different strengths.

GPT BERT T5
Architecture Decoder-only Encoder-only Encoder-decoder
Attention mask Causal (left-to-right) Bidirectional (sees all tokens) Bidirectional (encoder) + causal (decoder)
Training objective Next token prediction Masked language model (predict 15% masked tokens) Text-to-text (map input text to output text)
Strengths Text generation, chat, code Classification, NER, sentence embeddings Translation, summarization, Q&A
Key models GPT-4, Claude, LLaMA, Gemini BERT, RoBERTa, DeBERTa T5, FLAN-T5, mT5, UL2
Can generate text? Yes (autoregressive) Not natively (fill-mask only) Yes (seq2seq)
Cross-attention? No No Yes (decoder attends to encoder)
Dominant today? Yes — powers all major chatbots Still used for embeddings/classification Less common; used for structured generation

Why decoder-only won

GPT-style decoder-only models dominate because next-token prediction is a universal objective. By predicting the next word across trillions of tokens, these models learn grammar, facts, reasoning, and code — all from one simple training signal. BERT-style models need task-specific fine-tuning. T5-style models need input-output pairs. Decoder-only models just need raw text, and the internet has unlimited raw text.

GPT-4: decoder-only Claude: decoder-only LLaMA: decoder-only Gemini: decoder-only

Common Hyperparameters

Reference ranges for transformer hyperparameters, from tiny research models to production-scale LLMs.

Hyperparameter Symbol Typical Range Examples
Model dimension d_model 128 - 12288 GPT-2: 768, GPT-3: 12288, LLaMA-7B: 4096
Number of heads n_heads 4 - 96 GPT-2: 12, GPT-3: 96, LLaMA-7B: 32
Number of layers n_layers 4 - 120+ GPT-2: 12-48, GPT-3: 96, LLaMA-7B: 32
FFN dimension d_ff 4 x d_model GPT-2: 3072, GPT-3: 49152, LLaMA-7B: 11008
Head dimension d_k d_model / n_heads GPT-2: 64, GPT-3: 128, LLaMA-7B: 128
Dropout dropout 0.0 - 0.3 GPT-2: 0.1, GPT-3: 0.1, LLaMA: 0.0
Learning rate (peak) lr 1e-5 - 6e-4 GPT-3 175B: 6e-5, LLaMA-7B: 3e-4
Batch size (tokens) batch_tokens 256K - 4M GPT-3: 3.2M tokens, LLaMA: 4M tokens
Context length max_seq_len 512 - 1M+ GPT-2: 1024, GPT-4: 128K, Claude: 200K
Vocab size vocab_size 30K - 150K GPT-2: 50257, LLaMA: 32000, GPT-4: ~100K
Scaling law relationships:
Chinchilla scaling laws (Hoffmann et al., 2022) showed that for compute-optimal training, model size and training data should scale roughly equally. Doubling the model size? Double the training tokens too. This insight shifted the field from "bigger model" to "bigger model + more data" — which is why LLaMA-7B trained on 1T tokens outperforms GPT-3 (175B) trained on 300B tokens on many benchmarks.
The d_ff mystery:
The original transformer and GPT-2 use d_ff = 4 x d_model, but LLaMA uses d_ff = 2.67 x d_model (with SwiGLU activation, which introduces a third weight matrix in the FFN). The "4x" convention is a starting point, not a law — modern architectures tune this ratio based on compute budgets.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You've pinned the cheatsheet.
Now build the intuition behind it.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →