Full Transformer Architecture
The original "Attention Is All You Need" architecture with all components labeled. The encoder processes the source sequence; the decoder generates the target sequence one token at a time.
Key Formulas
The six equations that define the transformer. Memorize these and you can reconstruct the entire architecture from scratch.
Parameter Counting
Know exactly where every parameter lives. This table shows the parameter count for each component in a single transformer block, then the full model.
| Component | Parameter Count | Notes |
|---|---|---|
| Embedding | vocab_size x d_model | One-time cost. Dominates small models. |
| Position Embedding | max_seq_len x d_model | Learned (GPT) or zero params (sinusoidal). |
| Attention Q, K, V | 3 x (d_model x d_k) x n_heads | = 3 x d_model x d_model (since d_k = d_model / n_heads). |
| Attention output proj | d_model x d_model | W^O concatenates heads back. |
| Attention biases | 3 x d_model + d_model | = 4 x d_model (for QKV + output). |
| FFN layer 1 | d_model x d_ff + d_ff | Expand. Typically d_ff = 4 x d_model. |
| FFN layer 2 | d_ff x d_model + d_model | Contract back to d_model. |
| LayerNorm (x2) | 2 x (2 x d_model) | = 4 x d_model. Two norms per block (gamma + beta each). |
Tensor Shapes Cheatsheet
Track the exact shape of every tensor as data flows through the transformer. This is the single most useful debugging reference when implementing from scratch.
| Step | Tensor | Shape |
|---|---|---|
| Input token IDs | input_ids | (batch, seq_len) |
| After embedding | x | (batch, seq_len, d_model) |
| After positional encoding | x | (batch, seq_len, d_model) |
| Q, K, V (before reshape) | q, k, v | (batch, seq_len, d_model) |
| Q, K, V (per head) | q, k, v | (batch, n_heads, seq_len, d_k) |
| Attention scores | Q K^T / sqrt(d_k) | (batch, n_heads, seq_len, seq_len) |
| Attention weights | softmax(scores) | (batch, n_heads, seq_len, seq_len) |
| Attention output (per head) | weights @ V | (batch, n_heads, seq_len, d_k) |
| Concat heads + W^O | attn_out | (batch, seq_len, d_model) |
| FFN intermediate | hidden | (batch, seq_len, d_ff) |
| FFN output | ffn_out | (batch, seq_len, d_model) |
| After all blocks | x | (batch, seq_len, d_model) |
| Output logits | logits | (batch, seq_len, vocab_size) |
(batch, seq_len, n_heads, d_k) -> (batch, n_heads, seq_len, d_k) via .transpose(1, 2). Without this, the attention matrix multiplications produce the wrong shapes and PyTorch will either throw an error or — worse — silently compute garbage.
GPT vs BERT vs T5
Three ways to use the transformer architecture. Same building blocks, different configurations, different strengths.
| GPT | BERT | T5 | |
|---|---|---|---|
| Architecture | Decoder-only | Encoder-only | Encoder-decoder |
| Attention mask | Causal (left-to-right) | Bidirectional (sees all tokens) | Bidirectional (encoder) + causal (decoder) |
| Training objective | Next token prediction | Masked language model (predict 15% masked tokens) | Text-to-text (map input text to output text) |
| Strengths | Text generation, chat, code | Classification, NER, sentence embeddings | Translation, summarization, Q&A |
| Key models | GPT-4, Claude, LLaMA, Gemini | BERT, RoBERTa, DeBERTa | T5, FLAN-T5, mT5, UL2 |
| Can generate text? | Yes (autoregressive) | Not natively (fill-mask only) | Yes (seq2seq) |
| Cross-attention? | No | No | Yes (decoder attends to encoder) |
| Dominant today? | Yes — powers all major chatbots | Still used for embeddings/classification | Less common; used for structured generation |
Why decoder-only won
GPT-style decoder-only models dominate because next-token prediction is a universal objective. By predicting the next word across trillions of tokens, these models learn grammar, facts, reasoning, and code — all from one simple training signal. BERT-style models need task-specific fine-tuning. T5-style models need input-output pairs. Decoder-only models just need raw text, and the internet has unlimited raw text.
Common Hyperparameters
Reference ranges for transformer hyperparameters, from tiny research models to production-scale LLMs.
| Hyperparameter | Symbol | Typical Range | Examples |
|---|---|---|---|
| Model dimension | d_model |
128 - 12288 | GPT-2: 768, GPT-3: 12288, LLaMA-7B: 4096 |
| Number of heads | n_heads |
4 - 96 | GPT-2: 12, GPT-3: 96, LLaMA-7B: 32 |
| Number of layers | n_layers |
4 - 120+ | GPT-2: 12-48, GPT-3: 96, LLaMA-7B: 32 |
| FFN dimension | d_ff |
4 x d_model | GPT-2: 3072, GPT-3: 49152, LLaMA-7B: 11008 |
| Head dimension | d_k |
d_model / n_heads | GPT-2: 64, GPT-3: 128, LLaMA-7B: 128 |
| Dropout | dropout |
0.0 - 0.3 | GPT-2: 0.1, GPT-3: 0.1, LLaMA: 0.0 |
| Learning rate (peak) | lr |
1e-5 - 6e-4 | GPT-3 175B: 6e-5, LLaMA-7B: 3e-4 |
| Batch size (tokens) | batch_tokens |
256K - 4M | GPT-3: 3.2M tokens, LLaMA: 4M tokens |
| Context length | max_seq_len |
512 - 1M+ | GPT-2: 1024, GPT-4: 128K, Claude: 200K |
| Vocab size | vocab_size |
30K - 150K | GPT-2: 50257, LLaMA: 32000, GPT-4: ~100K |
You've pinned the cheatsheet.
Now build the intuition behind it.
Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Agentic Engineer →