Why backpropagation matters
In the previous lesson, you built a single neuron. It could learn a straight line. But a single neuron cannot learn XOR, cannot learn language, cannot learn anything interesting. You need layers. And layers need a way to figure out which weight caused the error.
That is backpropagation: the algorithm that flows the error backward through the network, telling each weight exactly how much it contributed to the mistake. Without it, deep learning does not exist. Every transformer, every GPT, every LLM you will build in this track relies on backprop to learn.
Imagine a factory with 100 workers on an assembly line. The final product is defective. Who is responsible? You need to trace the defect backward through each step. Backpropagation does the same thing: it traces the loss backward through each layer, assigning "blame" to each weight proportional to its contribution to the error.
The chain rule: calculus in 5 minutes
Backpropagation is just the chain rule applied repeatedly. If you have a composite function f(g(x)), the derivative of f with respect to x is:
df/dx = df/dg * dg/dx
That is the entire secret. You break a complex computation into small steps, compute the local derivative at each step, and multiply them together. Let's see it in code.
Computational graphs: visualizing the chain rule
A computational graph makes backpropagation mechanical. Every operation is a node. Forward pass flows left to right (computing values). Backward pass flows right to left (computing gradients). At each node, you multiply the incoming gradient by the local derivative.
Reading the backward pass: the loss L = 49 has gradient 1 with respect to itself. The square node's local derivative is 2 * (input) = 2 * 7 = 14. The add node passes the gradient through unchanged. The multiply node routes: dL/dx = 14 * w = 42, dL/dw = 14 * x = 28. Every gradient is just local derivative times incoming gradient.
Loss functions: measuring how wrong you are
Before you can backpropagate, you need a number that says "how wrong is the network?" That number is the loss. Two loss functions dominate deep learning:
Mean Squared Error (MSE)
MSE = mean((predicted - actual)²). Used for regression. Penalizes large errors quadratically — an error of 2 costs 4x more than an error of 1. Simple, differentiable, and the default choice when your output is continuous.
Cross-Entropy Loss
CE = -mean(actual * log(predicted)). Used for classification. Punishes confident wrong answers severely: if the network predicts 0.99 for the wrong class, cross-entropy explodes. This is what GPT and every language model uses — predicting the next token is a classification problem.
The XOR problem: why you need layers
XOR (exclusive or) is the classic problem that broke single-layer networks. The inputs are (0,0), (0,1), (1,0), (1,1). The outputs are 0, 1, 1, 0. A single neuron draws one straight line — it cannot separate XOR's outputs because they are not linearly separable.
The hidden layer transforms the input space. Each hidden neuron draws its own line. The output neuron combines them. Two lines can carve out the XOR region. This is why depth matters — and why backpropagation is essential: we need to train those hidden neurons too.
Full backpropagation: XOR from scratch
Here is the complete algorithm. A 2-layer network with 4 hidden neurons, sigmoid activations, MSE loss, and manual gradient computation. No PyTorch, no autograd — just NumPy and the chain rule.
The forward pass
For each sample: multiply inputs by weights, add bias, apply sigmoid. Repeat for the next layer. Compute loss at the end.
The backward pass
Start from the loss. Compute dL/d(output). Multiply by sigmoid derivative to get the output layer's delta. Propagate backward: multiply by the transposed weights to get the hidden layer's delta. Update all weights using these deltas.
Line-by-line breakdown
- Forward pass:
hidden_z = X @ w_hidden + b_hidden— matrix multiply inputs (4x2) by weights (2x4), giving 4 samples x 4 hidden neurons. Apply sigmoid to squash to [0,1]. - Output layer: Same thing — hidden activations (4x4) times output weights (4x1) gives 4 predictions. Sigmoid again.
- Loss: MSE between predictions and targets. This single number is what we minimize.
- Output delta:
d_output = (output_a - y) * sigmoid_deriv(output_a)— the error signal at the output, scaled by how much the sigmoid can change. - Hidden delta:
d_hidden = (d_output @ w_output.T) * sigmoid_deriv(hidden_a)— propagate the error backward through the output weights, then scale by the hidden sigmoid derivative. - Weight updates:
w -= lr * (activations.T @ delta) / N— gradient descent. The gradient is the product of the upstream activations and the downstream delta.
Understanding gradient flow
The backward pass follows a strict pattern. At each layer, two things happen:
- Propagate: The error from the layer above is multiplied by the transposed weight matrix. This distributes the blame to each neuron in the current layer, proportional to how much it contributed.
- Gate: The result is multiplied element-wise by the activation function's derivative. If a neuron was saturated (sigmoid output near 0 or 1), its derivative is near zero — it receives almost no gradient. This is the vanishing gradient problem.
Why this matters for LLMs
The vanishing gradient problem is why sigmoid and tanh fell out of favor for deep networks. Modern transformers use ReLU variants and layer normalization to keep gradients flowing through 100+ layers. When you build your transformer in Arc C, you will see exactly how attention + layer norm solve this. But the underlying algorithm is still backpropagation — the same chain rule you just implemented.
Modify the learning rate and see how it affects training. Try lr = 0.1 (too slow), lr = 1.0 (just right), and lr = 5.0 (too fast — does it diverge?). Click Run to execute in your browser with Pyodide.
After computing the analytical gradients for w_output, verify them numerically. Perturb each weight by h = 1e-5, recompute the loss, and check that (loss_plus - loss_minus) / (2*h) matches your analytical gradient within 1e-4.
Hint: Loop over each element of w_output. Save the original value. Set w_output[i,j] += h, run the full forward pass, record loss_plus. Set w_output[i,j] -= 2*h, run forward, record loss_minus. Restore the original. The numerical gradient is (loss_plus - loss_minus) / (2*h). Compare to your analytical gradient matrix at [i,j].
Replace sigmoid with relu(x) = max(0, x) and its derivative relu_deriv(x) = (x > 0).astype(float). Keep sigmoid only on the output layer. Does training converge faster? What happens if you use ReLU on the output layer too?
Hint: Define relu = lambda x: np.maximum(0, x) and relu_deriv = lambda x: (x > 0).astype(float). Use ReLU for the hidden layer only. For the hidden delta, use relu_deriv(hidden_z) not relu_deriv(hidden_a) — ReLU's derivative depends on the pre-activation value. Using ReLU on the output squashes nothing, so predictions can exceed [0,1] — keep sigmoid there.
Test your knowledge
You've mastered backpropagation.
Now build the full transformer.
Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Agentic Engineer →