The architecture that changed everything
In previous lessons, you built every individual component: embeddings, positional encoding, multi-head attention, feed-forward networks, layer normalization, and the transformer block. Now we assemble them into the full Transformer — the original encoder-decoder model from the 2017 paper "Attention Is All You Need" by Vaswani et al.
This is the most important architecture in modern deep learning. It replaced RNNs, eliminated the seq2seq bottleneck, and became the foundation for GPT, BERT, T5, and every large language model you use today.
Eight Google researchers proposed a model that used only attention — no recurrence, no convolutions. It trained faster, parallelized better, and outperformed every existing model on machine translation. The architecture had two halves: an encoder that reads the input, and a decoder that generates the output. Both are stacks of identical blocks, connected by cross-attention.
The base model had d_model=512, N=6 layers, h=8 heads, and about 65 million parameters. Let's build it.
The full architecture
This is the iconic diagram from the original paper, recreated below. Study it carefully — every box corresponds to code you've already seen or will see in this lesson.
The encoder stack
The encoder is a stack of N identical layers (N=6 in the original paper). Each layer has two sub-layers:
- Multi-head self-attention: Each token attends to every other token in the input. No masking — the encoder can see the full input at once. This is what eliminates the RNN's sequential bottleneck.
- Position-wise feed-forward network (FFN): Two linear layers with a ReLU activation in between. Applied independently to each position. Inner dimension d_ff = 2048 (4x the model dimension).
Both sub-layers have a residual connection and layer normalization: output = LayerNorm(x + SubLayer(x)). This keeps gradients flowing through deep stacks.
The encoder output is a tensor of shape (batch, seq_len, d_model). Unlike seq2seq, every position's representation is preserved — the decoder can attend to any of them through cross-attention. No bottleneck.
The decoder stack
The decoder is also N=6 identical layers, but each layer has three sub-layers:
- Masked multi-head self-attention: The decoder attends to its own previous outputs, but positions can only attend to earlier positions (not future ones). This is enforced by the causal mask — a triangular matrix of -inf values that prevents the model from cheating during training.
- Multi-head cross-attention: Queries come from the decoder, but Keys and Values come from the encoder output. This is how the decoder "reads" the source sentence. It's the replacement for the seq2seq bottleneck.
- Position-wise feed-forward network: Same structure as the encoder's FFN.
Cross-attention: the encoder-decoder bridge
In cross-attention, the decoder generates the Query vectors (Q), while the encoder output provides the Key (K) and Value (V) vectors. This means:
- The decoder asks: "Which parts of the input should I focus on for this output position?"
- The encoder answers: "Here are my representations — attend to whichever ones you need."
- Every decoder layer gets fresh access to the encoder output. Information flows freely.
Output projection + softmax
After the decoder's final layer, each position has a d_model-dimensional vector. To produce a probability distribution over the vocabulary, we apply:
- Linear projection: A weight matrix of shape (d_model, vocab_size) projects from 512 dimensions to vocabulary size (e.g., 37,000 BPE tokens).
- Softmax: Converts raw logits into probabilities that sum to 1. The token with the highest probability is the prediction.
NumPy: conceptual forward pass
This simplified NumPy code shows the full forward pass flow. It's not trainable — the goal is to trace exactly how data moves through each component.
PyTorch: complete Transformer
Here's the full Transformer as a proper PyTorch nn.Module. This is production-level code — the same structure used in real implementations.
Training with teacher forcing
During training, the decoder receives the ground-truth target sequence (shifted right by one position) as input. At each position, the model predicts the next token, and we compute cross-entropy loss against the actual next token.
This is called teacher forcing because we force the model to use the correct previous tokens (not its own predictions). Combined with the causal mask, the entire target sequence can be processed in one parallel forward pass — no sequential generation needed during training.
Parameter counting
Let's count the parameters for the base Transformer (d_model=512, N=6, h=8, d_ff=2048, vocab=37000):
Run it yourself
Run the full Transformer locally
The full Transformer is too complex to run in the browser. Here's how to run it on your machine:
pip install torch numpy python transformer_pytorch.py
You can also try training it on a simple task like number reversal or copy:
# Quick training loop (add to transformer_pytorch.py)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)
criterion = nn.CrossEntropyLoss(ignore_index=0) # ignore padding
for epoch in range(100):
logits = model(src, tgt[:, :-1]) # predict next tokens
loss = criterion(
logits.reshape(-1, tgt_vocab_size),
tgt[:, 1:].reshape(-1) # shifted targets
)
optimizer.zero_grad()
loss.backward()
optimizer.step()
if epoch % 10 == 0:
print(f"Epoch {epoch}: loss = {loss.item():.4f}")
On a laptop CPU, training a small Transformer (d_model=64, N=2) on the copy task takes about 30 seconds. The full d_model=512, N=6 model needs a GPU for any real task.
Why this matters for LLMs
From Transformer to GPT: one step away
The full encoder-decoder Transformer you just built is used in models like T5 (Google), BART (Meta), and mBART (multilingual translation). But the biggest revolution came from using only the decoder — that's GPT.
In the next lesson, you'll strip away the encoder, remove cross-attention, and build the decoder-only architecture that powers ChatGPT, Claude, and Gemini. The leap from this lesson to GPT is surprisingly small — most of the code you've already written carries over directly.
Challenge
Using the PyTorch Transformer class above, train on the number reversal task from Lesson 8 (seq2seq). Use d_model=64, N=2, h=4 for a small model that trains fast on CPU. Compare accuracy against the RNN seq2seq model — the Transformer should converge much faster and handle longer sequences better.
Modify the parameter counting script for GPT-2 Small: decoder-only, d_model=768, N=12, h=12, d_ff=3072, vocab=50257. How many parameters does it have? How does that compare to the original Transformer?
Quiz
You've built the Transformer.
Now build the career to match.
Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Agentic Engineer →