The architecture that changed everything

In previous lessons, you built every individual component: embeddings, positional encoding, multi-head attention, feed-forward networks, layer normalization, and the transformer block. Now we assemble them into the full Transformer — the original encoder-decoder model from the 2017 paper "Attention Is All You Need" by Vaswani et al.

This is the most important architecture in modern deep learning. It replaced RNNs, eliminated the seq2seq bottleneck, and became the foundation for GPT, BERT, T5, and every large language model you use today.

🏗
The original paper
"Attention Is All You Need" (Vaswani et al., 2017)

Eight Google researchers proposed a model that used only attention — no recurrence, no convolutions. It trained faster, parallelized better, and outperformed every existing model on machine translation. The architecture had two halves: an encoder that reads the input, and a decoder that generates the output. Both are stacks of identical blocks, connected by cross-attention.

The base model had d_model=512, N=6 layers, h=8 heads, and about 65 million parameters. Let's build it.

The full architecture

This is the iconic diagram from the original paper, recreated below. Study it carefully — every box corresponds to code you've already seen or will see in this lesson.

The Transformer — Full Architecture "Attention Is All You Need" (Vaswani et al., 2017) ENCODER Input Embedding + Positional Encoding N x Multi-Head Self-Attention Add & Layer Norm Feed-Forward Network (FFN) Add & Layer Norm Encoder Output Inputs "the cat sat" DECODER Output Embedding + Positional Encoding N x Masked Multi-Head Self-Attention Add & Layer Norm Multi-Head Cross- Attention (Enc->Dec) K, V from encoder Add & Layer Norm Feed-Forward Network (FFN) Add & Layer Norm Linear + Softmax Output Probabilities Outputs (shifted right) "<SOS> le chat" = Add & Norm (residual) = Cross-attention link = Residual connection
Key observation:
The encoder processes all input tokens in parallel (no recurrence!). The decoder also processes in parallel during training (using masked attention), but generates one token at a time during inference. The cross-attention layer is the bridge — it lets the decoder attend to the encoder's output at every layer.

The encoder stack

The encoder is a stack of N identical layers (N=6 in the original paper). Each layer has two sub-layers:

  1. Multi-head self-attention: Each token attends to every other token in the input. No masking — the encoder can see the full input at once. This is what eliminates the RNN's sequential bottleneck.
  2. Position-wise feed-forward network (FFN): Two linear layers with a ReLU activation in between. Applied independently to each position. Inner dimension d_ff = 2048 (4x the model dimension).

Both sub-layers have a residual connection and layer normalization: output = LayerNorm(x + SubLayer(x)). This keeps gradients flowing through deep stacks.

Input Tokens
->
Embed + PosEnc
->
Layer 1
->
Layer 2
->
...
->
Layer 6
->
Encoder Output

The encoder output is a tensor of shape (batch, seq_len, d_model). Unlike seq2seq, every position's representation is preserved — the decoder can attend to any of them through cross-attention. No bottleneck.

The decoder stack

The decoder is also N=6 identical layers, but each layer has three sub-layers:

  1. Masked multi-head self-attention: The decoder attends to its own previous outputs, but positions can only attend to earlier positions (not future ones). This is enforced by the causal mask — a triangular matrix of -inf values that prevents the model from cheating during training.
  2. Multi-head cross-attention: Queries come from the decoder, but Keys and Values come from the encoder output. This is how the decoder "reads" the source sentence. It's the replacement for the seq2seq bottleneck.
  3. Position-wise feed-forward network: Same structure as the encoder's FFN.
Why mask the decoder?
During training, we feed the entire target sequence in at once (for parallelism). But position 3 shouldn't be able to see position 4 — that would be giving away the answer. The causal mask ensures each position can only attend to positions 0, 1, ..., up to itself. During inference, this happens naturally since we generate one token at a time.

Cross-attention: the encoder-decoder bridge

In cross-attention, the decoder generates the Query vectors (Q), while the encoder output provides the Key (K) and Value (V) vectors. This means:

  • The decoder asks: "Which parts of the input should I focus on for this output position?"
  • The encoder answers: "Here are my representations — attend to whichever ones you need."
  • Every decoder layer gets fresh access to the encoder output. Information flows freely.

Output projection + softmax

After the decoder's final layer, each position has a d_model-dimensional vector. To produce a probability distribution over the vocabulary, we apply:

  1. Linear projection: A weight matrix of shape (d_model, vocab_size) projects from 512 dimensions to vocabulary size (e.g., 37,000 BPE tokens).
  2. Softmax: Converts raw logits into probabilities that sum to 1. The token with the highest probability is the prediction.
Weight tying:
In many implementations (including the original paper), the output projection matrix is the transpose of the embedding matrix. This means the embedding and output layers share the same weights — halving the parameter count of those layers and improving generalization.

NumPy: conceptual forward pass

This simplified NumPy code shows the full forward pass flow. It's not trainable — the goal is to trace exactly how data moves through each component.

transformer_forward_numpy.py Read-only — conceptual flow
Expected output
=== ENCODER === Input shape after embed+pos: (5, 512) Encoder output shape: (5, 512) === DECODER === Input shape after embed+pos: (4, 512) Causal mask: [[1 0 0 0] [1 1 0 0] [1 1 1 0] [1 1 1 1]] === OUTPUT === Logits shape: (4, 1000) Predictions: [742 891 233 107] (Random weights -> random predictions, but the FLOW is correct!)
Trace the data flow:
Encoder: embed -> [self-attn -> add&norm -> ffn -> add&norm] x 6 -> encoder_output. Decoder: embed -> [masked-self-attn -> add&norm -> cross-attn(encoder_output) -> add&norm -> ffn -> add&norm] x 6 -> linear -> softmax -> predictions. The cross-attention call on line 99 is the critical bridge between encoder and decoder.

PyTorch: complete Transformer

Here's the full Transformer as a proper PyTorch nn.Module. This is production-level code — the same structure used in real implementations.

transformer_pytorch.py Read-only — run locally with PyTorch
Expected output
Output shape: torch.Size([2, 8, 10000]) Total parameters: 64,736,256

Training with teacher forcing

During training, the decoder receives the ground-truth target sequence (shifted right by one position) as input. At each position, the model predicts the next token, and we compute cross-entropy loss against the actual next token.

Teacher Forcing: Decoder Input vs Target Decoder input: <SOS> le chat est Target (loss): le chat est <EOS> Each position predicts the NEXT token. Loss = cross-entropy at every position. The causal mask ensures position i cannot see positions i+1, i+2, ...

This is called teacher forcing because we force the model to use the correct previous tokens (not its own predictions). Combined with the causal mask, the entire target sequence can be processed in one parallel forward pass — no sequential generation needed during training.

Parameter counting

Let's count the parameters for the base Transformer (d_model=512, N=6, h=8, d_ff=2048, vocab=37000):

count_params.py Read-only — parameter breakdown
Expected output
=== Parameter Count: Base Transformer === Multi-head attention (per layer): 1,050,624 Feed-forward network (per layer): 2,098,176 Layer norm (per sub-layer): 1,024 Encoder layer total: 3,150,848 Decoder layer total: 4,202,496 Encoder stack (6 layers): 18,905,088 Decoder stack (6 layers): 25,214,976 Source embedding: 18,944,000 Target embedding: 18,944,000 Output projection (if separate): 18,981,000 ================================================== Total (no weight tying): 81,989,064 Total (with weight tying): 82,008,064 The original paper's base model: ~65M parameters The 'big' model (d=1024, N=6, h=16): ~213M parameters
Where do the parameters live?
The embeddings account for roughly half the total. The feed-forward networks are the largest per-layer component (2M each vs 1M for attention). This ratio — FFN being 2x the attention parameters — holds across most transformer architectures, including GPT.

Run it yourself

Run the full Transformer locally

The full Transformer is too complex to run in the browser. Here's how to run it on your machine:

pip install torch numpy
python transformer_pytorch.py

You can also try training it on a simple task like number reversal or copy:

# Quick training loop (add to transformer_pytorch.py)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)
criterion = nn.CrossEntropyLoss(ignore_index=0)  # ignore padding

for epoch in range(100):
    logits = model(src, tgt[:, :-1])  # predict next tokens
    loss = criterion(
        logits.reshape(-1, tgt_vocab_size),
        tgt[:, 1:].reshape(-1)  # shifted targets
    )
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    if epoch % 10 == 0:
        print(f"Epoch {epoch}: loss = {loss.item():.4f}")

On a laptop CPU, training a small Transformer (d_model=64, N=2) on the copy task takes about 30 seconds. The full d_model=512, N=6 model needs a GPU for any real task.

Why this matters for LLMs

From Transformer to GPT: one step away

The full encoder-decoder Transformer you just built is used in models like T5 (Google), BART (Meta), and mBART (multilingual translation). But the biggest revolution came from using only the decoder — that's GPT.

In the next lesson, you'll strip away the encoder, remove cross-attention, and build the decoder-only architecture that powers ChatGPT, Claude, and Gemini. The leap from this lesson to GPT is surprisingly small — most of the code you've already written carries over directly.

Encoder-Decoder -> T5, BART Decoder-Only -> GPT, Claude Encoder-Only -> BERT d_model=512 -> d_model=12288 (GPT-3)

Challenge

Put it into practice
Challenge 1: Train on number reversal

Using the PyTorch Transformer class above, train on the number reversal task from Lesson 8 (seq2seq). Use d_model=64, N=2, h=4 for a small model that trains fast on CPU. Compare accuracy against the RNN seq2seq model — the Transformer should converge much faster and handle longer sequences better.

Use a vocabulary of 12 tokens (0-9, SOS, EOS). Generate pairs like ("1234", "4321"). Tokenize each character separately. Train with CrossEntropyLoss and teacher forcing for 500 epochs on 1000 pairs. Test on sequences of length 8-10 to see the advantage over seq2seq.
Challenge 2: Parameter counting for GPT-2

Modify the parameter counting script for GPT-2 Small: decoder-only, d_model=768, N=12, h=12, d_ff=3072, vocab=50257. How many parameters does it have? How does that compare to the original Transformer?

GPT-2 is decoder-only, so there's no encoder stack and no cross-attention layers. Each decoder layer has: 1 self-attention + 1 FFN + 2 layer norms. Plus one embedding layer (weight-tied with output). You should get approximately 124M parameters — matching OpenAI's reported number.

Quiz

Question 1 of 5
In the original Transformer, where do the Keys and Values come from in the decoder's cross-attention layer?
Question 2 of 5
Why does the decoder use a causal (triangular) mask during training?
Question 3 of 5
How many sub-layers does each decoder layer have, compared to each encoder layer?
Question 4 of 5
In the base Transformer (d_model=512, d_ff=2048), which component has more parameters per layer?
Question 5 of 5
What is "weight tying" in the Transformer output layer?
0
/5
You understand the architecture that launched modern AI.
Arc D begins!

Next: GPT Architecture — Decoder Only ->

Strip away the encoder. Remove cross-attention. Keep just the decoder stack — and you get GPT, the architecture behind ChatGPT, Claude, and every modern LLM. The leap is smaller than you think.

Next: GPT Architecture — Decoder Only ->
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You've built the Transformer.
Now build the career to match.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →