The repeating unit of every LLM
In previous lessons, you built multi-head attention, positional encoding, and feed-forward networks as individual components. Now it is time to assemble them into the transformer block — the fundamental building block that gets stacked to create GPT, BERT, and every modern language model.
A transformer block combines four operations into one unit: multi-head attention, a residual connection (add), layer normalization (norm), and a feed-forward network. The "Attention Is All You Need" paper (Vaswani et al., 2017) defined two types of blocks: one for the encoder and one for the decoder.
GPT-3 has 96 transformer blocks stacked on top of each other. Every single one has the same architecture: attention, add & norm, FFN, add & norm. The only differences are the learned weights. This uniformity is what makes transformers so scalable — you can add more blocks without changing any code.
The encoder block
An encoder block has two sub-layers. Each sub-layer is wrapped with a residual connection and layer normalization:
- Multi-Head Self-Attention: Each token attends to every other token in the input. No masking — the encoder sees the full sequence.
- Feed-Forward Network (FFN): Two linear layers with a ReLU (or GELU) activation in between, applied independently to each position.
output = LayerNorm(x + SubLayer(x)).
The decoder block
A decoder block has three sub-layers — one more than the encoder. The extra sub-layer is cross-attention, which lets the decoder attend to the encoder's output:
- Masked Multi-Head Self-Attention: Each token attends only to previous tokens (and itself). A causal mask prevents the decoder from "seeing the future."
- Multi-Head Cross-Attention: Queries come from the decoder, but keys and values come from the encoder output. This is how the decoder reads the source sequence.
- Feed-Forward Network: Same structure as the encoder FFN.
The causal mask
In a decoder, token 3 must not attend to tokens 4, 5, 6, etc. — those are "the future" that the model is trying to predict. We enforce this with a causal mask: an upper-triangular matrix of negative infinities that gets added to the attention scores before softmax.
The mask is added to the raw attention scores before softmax. Since softmax(-inf) = 0, masked positions contribute zero attention weight. The model literally cannot use future information.
np.triu(np.ones((seq_len, seq_len)), k=1) * -1e9. The triu function creates an upper-triangular matrix with ones above the diagonal, and we multiply by negative infinity (approximated as -1e9).
Code: encoder and decoder blocks in NumPy
Let's assemble everything. We'll build an encoder block, a decoder block with causal masking and cross-attention, and stack multiple blocks together.
Notice how the output shape matches the input shape for both blocks. This is essential for stacking: the output of block 1 feeds directly into block 2, then block 3, and so on. Also notice how the output norm stays stable across 4 blocks — that is layer normalization and residual connections doing their job.
multi_head_attention(x, x, ...)). In cross-attention, Q comes from the decoder but K and V come from the encoder output (multi_head_attention(x, enc_output, ...)). This is the bridge between encoder and decoder.
Stacking blocks: from 1 to 96
The original transformer used N=6 blocks for both encoder and decoder. Modern models use far more:
- GPT-2 Small: 12 decoder blocks
- GPT-2 Large: 36 decoder blocks
- GPT-3: 96 decoder blocks
- BERT Base: 12 encoder blocks
- BERT Large: 24 encoder blocks
- LLaMA 70B: 80 decoder blocks
Every block has identical architecture. The only thing that changes is the learned weight values. More blocks mean the model can learn deeper, more abstract representations — each block refines the previous one's output.
Why this matters for building an LLM
The transformer block is the LEGO brick of modern AI. When someone says "GPT-3 has 175 billion parameters," those parameters live inside 96 identical blocks, each containing multi-head attention weights, FFN weights, and layer normalization parameters. Understanding this block means understanding the entire architecture — the rest is just stacking and scaling.
Try It Yourself
Run this code to build a complete encoder block, pass data through it, and verify that residual connections preserve the signal across multiple stacked blocks.
Challenge
Modify the encoder block to remove the residual connections (replace x + attn_out with just attn_out). Stack 6 blocks and compare the output norm to the version with residuals. What happens?
Add cross-attention to the decoder block in the Try It code. Create a second set of weights for cross-attention, generate some "encoder output" data, and verify the decoder block output shape matches the target sequence length (not the encoder length).
multi_head_attention(x, enc_output, cross_W_q, cross_W_k, cross_W_v, cross_W_o, n_heads) — no mask needed for cross-attention. The output shape is determined by the query (decoder) length, not the key/value (encoder) length.Quiz
You've built the transformer block.
Now build entire models from scratch.
Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Agentic Engineer →