The repeating unit of every LLM

In previous lessons, you built multi-head attention, positional encoding, and feed-forward networks as individual components. Now it is time to assemble them into the transformer block — the fundamental building block that gets stacked to create GPT, BERT, and every modern language model.

A transformer block combines four operations into one unit: multi-head attention, a residual connection (add), layer normalization (norm), and a feed-forward network. The "Attention Is All You Need" paper (Vaswani et al., 2017) defined two types of blocks: one for the encoder and one for the decoder.

🧱
The key insight
Blocks are identical — only the data changes

GPT-3 has 96 transformer blocks stacked on top of each other. Every single one has the same architecture: attention, add & norm, FFN, add & norm. The only differences are the learned weights. This uniformity is what makes transformers so scalable — you can add more blocks without changing any code.

The encoder block

An encoder block has two sub-layers. Each sub-layer is wrapped with a residual connection and layer normalization:

  1. Multi-Head Self-Attention: Each token attends to every other token in the input. No masking — the encoder sees the full sequence.
  2. Feed-Forward Network (FFN): Two linear layers with a ReLU (or GELU) activation in between, applied independently to each position.
ENCODER BLOCK Input Embeddings Multi-Head Self-Attention residual Add & Norm Feed-Forward Network (FFN) Add & Norm Output Attention / FFN sub-layer Add & Norm (residual + layernorm) Residual (skip) connection
Why residual connections?
Without skip connections, gradients vanish as you stack more blocks. The residual path lets gradients flow directly backward through the network, making it possible to train 96-block models like GPT-3. The formula is simple: output = LayerNorm(x + SubLayer(x)).

The decoder block

A decoder block has three sub-layers — one more than the encoder. The extra sub-layer is cross-attention, which lets the decoder attend to the encoder's output:

  1. Masked Multi-Head Self-Attention: Each token attends only to previous tokens (and itself). A causal mask prevents the decoder from "seeing the future."
  2. Multi-Head Cross-Attention: Queries come from the decoder, but keys and values come from the encoder output. This is how the decoder reads the source sequence.
  3. Feed-Forward Network: Same structure as the encoder FFN.
DECODER BLOCK Target Embeddings Masked Multi-Head Self-Attention Add & Norm Multi-Head Cross-Attention Encoder Out (K, V) Add & Norm Feed-Forward Network (FFN) Add & Norm Output Masked self-attention Cross-attention (Q from decoder, K/V from encoder) Feed-forward network
GPT uses only the decoder.
GPT-style models (GPT-2, GPT-3, GPT-4, LLaMA, Mistral) use only the decoder block — with masked self-attention but no cross-attention sub-layer. There is no encoder to attend to. BERT uses only the encoder block. T5 and the original transformer use both.

The causal mask

In a decoder, token 3 must not attend to tokens 4, 5, 6, etc. — those are "the future" that the model is trying to predict. We enforce this with a causal mask: an upper-triangular matrix of negative infinities that gets added to the attention scores before softmax.

CAUSAL MASK (4 tokens) t1 t2 t3 t4 t1 t2 t3 t4 0 -inf -inf -inf 0 0 -inf -inf 0 0 0 -inf 0 0 0 0 t1 sees only itself t2 sees t1, t2 t3 sees t1-t3 t4 sees all

The mask is added to the raw attention scores before softmax. Since softmax(-inf) = 0, masked positions contribute zero attention weight. The model literally cannot use future information.

One line of NumPy.
The causal mask is just np.triu(np.ones((seq_len, seq_len)), k=1) * -1e9. The triu function creates an upper-triangular matrix with ones above the diagonal, and we multiply by negative infinity (approximated as -1e9).

Code: encoder and decoder blocks in NumPy

Let's assemble everything. We'll build an encoder block, a decoder block with causal masking and cross-attention, and stack multiple blocks together.

transformer_block.py NumPy — full encoder and decoder blocks
Output
=== Transformer Block Configuration === d_model: 32 n_heads: 4 d_head: 8 d_ff: 128 === Encoder Block === Input shape: (5, 32) (5 tokens, d_model=32) Output shape: (5, 32) (same shape!) Output[0,:5]: [-0.267 -0.286 1.073 -0.757 0.696] === Causal Mask (4 tokens) === Shape: (4, 4) Mask: [[ 0.e+00 -1.e+09 -1.e+09 -1.e+09] [ 0.e+00 0.e+00 -1.e+09 -1.e+09] [ 0.e+00 0.e+00 0.e+00 -1.e+09] [ 0.e+00 0.e+00 0.e+00 0.e+00]] (0 = can attend, -1e9 = blocked) === Decoder Block === Target input: (4, 32) (4 tokens) Encoder out: (5, 32) (5 tokens) Output shape: (4, 32) Output[0,:5]: [-1.316 0.42 0.505 -0.019 -0.155] === Stacking N=4 Encoder Blocks === Block 1 output norm: 4.8740 Block 2 output norm: 4.8886 Block 3 output norm: 4.8892 Block 4 output norm: 4.8893 Final output shape: (5, 32) Residual connections keep gradients flowing through 4 blocks!

Notice how the output shape matches the input shape for both blocks. This is essential for stacking: the output of block 1 feeds directly into block 2, then block 3, and so on. Also notice how the output norm stays stable across 4 blocks — that is layer normalization and residual connections doing their job.

Self-attention vs cross-attention.
In self-attention, Q, K, and V all come from the same input (multi_head_attention(x, x, ...)). In cross-attention, Q comes from the decoder but K and V come from the encoder output (multi_head_attention(x, enc_output, ...)). This is the bridge between encoder and decoder.

Stacking blocks: from 1 to 96

The original transformer used N=6 blocks for both encoder and decoder. Modern models use far more:

  • GPT-2 Small: 12 decoder blocks
  • GPT-2 Large: 36 decoder blocks
  • GPT-3: 96 decoder blocks
  • BERT Base: 12 encoder blocks
  • BERT Large: 24 encoder blocks
  • LLaMA 70B: 80 decoder blocks

Every block has identical architecture. The only thing that changes is the learned weight values. More blocks mean the model can learn deeper, more abstract representations — each block refines the previous one's output.

Why this matters for building an LLM

The transformer block is the LEGO brick of modern AI. When someone says "GPT-3 has 175 billion parameters," those parameters live inside 96 identical blocks, each containing multi-head attention weights, FFN weights, and layer normalization parameters. Understanding this block means understanding the entire architecture — the rest is just stacking and scaling.

GPT = N decoder blocks BERT = N encoder blocks T5 = N encoder + N decoder scale = more blocks + wider

Try It Yourself

Interactive Build and test an encoder block

Run this code to build a complete encoder block, pass data through it, and verify that residual connections preserve the signal across multiple stacked blocks.

try_transformer_block.py Editable — runs in browser via Pyodide
Click "Run Code" to see the output...

Challenge

Put it into practice
Challenge 1: Compare with and without residuals

Modify the encoder block to remove the residual connections (replace x + attn_out with just attn_out). Stack 6 blocks and compare the output norm to the version with residuals. What happens?

Without residuals, the norm should shrink toward zero (vanishing signal) or explode. The residual path acts as an "information highway" that preserves the original signal while the sub-layers add refinements.
Challenge 2: Build a full decoder block

Add cross-attention to the decoder block in the Try It code. Create a second set of weights for cross-attention, generate some "encoder output" data, and verify the decoder block output shape matches the target sequence length (not the encoder length).

For cross-attention, call multi_head_attention(x, enc_output, cross_W_q, cross_W_k, cross_W_v, cross_W_o, n_heads) — no mask needed for cross-attention. The output shape is determined by the query (decoder) length, not the key/value (encoder) length.

Quiz

Question 1 of 5
What are the two sub-layers inside an encoder transformer block?
Question 2 of 5
What does the causal mask prevent in a decoder block?
Question 3 of 5
In the decoder's cross-attention sub-layer, where do the queries, keys, and values come from?
Question 4 of 5
Why are residual (skip) connections essential when stacking many transformer blocks?
Question 5 of 5
GPT-style models (GPT-2, GPT-3, LLaMA) use which type of transformer block?
0
/5
You can build the fundamental block of every modern LLM.
Next in Arc C

Next: The Full Transformer ->

Combine encoder and decoder stacks, add the final linear + softmax layer, and build a complete transformer model end to end.

Next: The Full Transformer ->
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You've built the transformer block.
Now build entire models from scratch.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →