Why backpropagation matters

In the previous lesson, you built a single neuron. It could learn a straight line. But a single neuron cannot learn XOR, cannot learn language, cannot learn anything interesting. You need layers. And layers need a way to figure out which weight caused the error.

That is backpropagation: the algorithm that flows the error backward through the network, telling each weight exactly how much it contributed to the mistake. Without it, deep learning does not exist. Every transformer, every GPT, every LLM you will build in this track relies on backprop to learn.

🔗
The core idea
Credit assignment: who caused the error?

Imagine a factory with 100 workers on an assembly line. The final product is defective. Who is responsible? You need to trace the defect backward through each step. Backpropagation does the same thing: it traces the loss backward through each layer, assigning "blame" to each weight proportional to its contribution to the error.

Lesson roadmap:
Chain rule (the math tool) → Loss functions (how we measure error) → Computational graphs (the bookkeeping) → Full backprop on XOR (the algorithm in action).

The chain rule: calculus in 5 minutes

Backpropagation is just the chain rule applied repeatedly. If you have a composite function f(g(x)), the derivative of f with respect to x is:

df/dx = df/dg * dg/dx

That is the entire secret. You break a complex computation into small steps, compute the local derivative at each step, and multiply them together. Let's see it in code.

chain_rule.py Chain rule with a simple composite function
Output
x = 2.0 g(x) = 3x + 1 = 7.0 f(g) = g^2 = 49.0 df/dg = 2g = 14.0 dg/dx = 3.0 df/dx = df/dg * dg/dx = 42.0 Numerical gradient check: 42.0000 Match: True
Key insight:
The numerical gradient check (perturb x by a tiny amount, measure how f changes) is your debugging tool. If the analytical gradient and numerical gradient disagree, your math is wrong. Always verify.

Computational graphs: visualizing the chain rule

A computational graph makes backpropagation mechanical. Every operation is a node. Forward pass flows left to right (computing values). Backward pass flows right to left (computing gradients). At each node, you multiply the incoming gradient by the local derivative.

Computational Graph: f(x,w,b) = (x*w + b)² x = 2 w = 3 x * w = 6 b = 1 + = 7 ()² = 49 L = 49 dL/dL = 1 2*7 = 14 14*1 = 14 dL/db = 14 dL/dx = 14*w = 42 dL/dw = 14*x = 28 Forward Backward
Forward pass (blue) computes the output. Backward pass (red dashed) propagates gradients using the chain rule at each node.

Reading the backward pass: the loss L = 49 has gradient 1 with respect to itself. The square node's local derivative is 2 * (input) = 2 * 7 = 14. The add node passes the gradient through unchanged. The multiply node routes: dL/dx = 14 * w = 42, dL/dw = 14 * x = 28. Every gradient is just local derivative times incoming gradient.

Loss functions: measuring how wrong you are

Before you can backpropagate, you need a number that says "how wrong is the network?" That number is the loss. Two loss functions dominate deep learning:

Mean Squared Error (MSE)

MSE = mean((predicted - actual)²). Used for regression. Penalizes large errors quadratically — an error of 2 costs 4x more than an error of 1. Simple, differentiable, and the default choice when your output is continuous.

Cross-Entropy Loss

CE = -mean(actual * log(predicted)). Used for classification. Punishes confident wrong answers severely: if the network predicts 0.99 for the wrong class, cross-entropy explodes. This is what GPT and every language model uses — predicting the next token is a classification problem.

loss_functions.py MSE vs Cross-Entropy comparison
Output
=== MSE Loss === Predictions: [2.5 0.8 3.1] Actual: [3. 1. 3. ] MSE Loss: 0.1000 Gradient: [-0.33333333 -0.13333333 0.06666667] === Cross-Entropy Loss === Predictions: [0.9 0.2 0.8] Actual: [1. 0. 1.] CE Loss: 0.1438 Confident wrong (pred=0.99, actual=0): CE = 4.6052 Confident right (pred=0.99, actual=1): CE = 0.0101
Why cross-entropy for LLMs:
A language model predicts the next token from a vocabulary of 50,000+ words. Each prediction is a classification. Cross-entropy loss punishes the model harshly when it is confident about the wrong token — exactly the behavior you want during training.

The XOR problem: why you need layers

XOR (exclusive or) is the classic problem that broke single-layer networks. The inputs are (0,0), (0,1), (1,0), (1,1). The outputs are 0, 1, 1, 0. A single neuron draws one straight line — it cannot separate XOR's outputs because they are not linearly separable.

Why XOR Needs Two Layers Single Neuron (fails) (0,0)=0 (1,0)=1 (0,1)=1 (1,1)=0 No single line works! 2-Layer Network (solves it) (0,0)=0 (1,0)=1 (0,1)=1 (1,1)=0 Two lines = solved! Output 0 Output 1
A single neuron draws one line. XOR requires two decision boundaries — only possible with a hidden layer.

The hidden layer transforms the input space. Each hidden neuron draws its own line. The output neuron combines them. Two lines can carve out the XOR region. This is why depth matters — and why backpropagation is essential: we need to train those hidden neurons too.

Full backpropagation: XOR from scratch

Here is the complete algorithm. A 2-layer network with 4 hidden neurons, sigmoid activations, MSE loss, and manual gradient computation. No PyTorch, no autograd — just NumPy and the chain rule.

The forward pass

For each sample: multiply inputs by weights, add bias, apply sigmoid. Repeat for the next layer. Compute loss at the end.

The backward pass

Start from the loss. Compute dL/d(output). Multiply by sigmoid derivative to get the output layer's delta. Propagate backward: multiply by the transposed weights to get the hidden layer's delta. Update all weights using these deltas.

xor_backprop.py Full XOR solver with manual backpropagation
Output
Epoch 0: loss = 0.255675 Epoch 2000: loss = 0.202843 Epoch 4000: loss = 0.005199 Epoch 6000: loss = 0.001742 Epoch 8000: loss = 0.001005 Epoch 10000: loss = 0.000696 Final predictions: [0 0] → 0.0298 (target: 0) [0 1] → 0.9748 (target: 1) [1 0] → 0.9748 (target: 1) [1 1] → 0.0250 (target: 0)

Line-by-line breakdown

  1. Forward pass: hidden_z = X @ w_hidden + b_hidden — matrix multiply inputs (4x2) by weights (2x4), giving 4 samples x 4 hidden neurons. Apply sigmoid to squash to [0,1].
  2. Output layer: Same thing — hidden activations (4x4) times output weights (4x1) gives 4 predictions. Sigmoid again.
  3. Loss: MSE between predictions and targets. This single number is what we minimize.
  4. Output delta: d_output = (output_a - y) * sigmoid_deriv(output_a) — the error signal at the output, scaled by how much the sigmoid can change.
  5. Hidden delta: d_hidden = (d_output @ w_output.T) * sigmoid_deriv(hidden_a) — propagate the error backward through the output weights, then scale by the hidden sigmoid derivative.
  6. Weight updates: w -= lr * (activations.T @ delta) / N — gradient descent. The gradient is the product of the upstream activations and the downstream delta.
Why divide by 4?
We have 4 training samples. Dividing by N=4 gives us the average gradient across all samples. This keeps the learning rate stable regardless of batch size — the same principle used in every modern framework.

Understanding gradient flow

The backward pass follows a strict pattern. At each layer, two things happen:

  1. Propagate: The error from the layer above is multiplied by the transposed weight matrix. This distributes the blame to each neuron in the current layer, proportional to how much it contributed.
  2. Gate: The result is multiplied element-wise by the activation function's derivative. If a neuron was saturated (sigmoid output near 0 or 1), its derivative is near zero — it receives almost no gradient. This is the vanishing gradient problem.

Why this matters for LLMs

The vanishing gradient problem is why sigmoid and tanh fell out of favor for deep networks. Modern transformers use ReLU variants and layer normalization to keep gradients flowing through 100+ layers. When you build your transformer in Arc C, you will see exactly how attention + layer norm solve this. But the underlying algorithm is still backpropagation — the same chain rule you just implemented.

Vanishing Gradients ReLU Layer Norm Transformers
Try It Yourself: Experiment with learning rates

Modify the learning rate and see how it affects training. Try lr = 0.1 (too slow), lr = 1.0 (just right), and lr = 5.0 (too fast — does it diverge?). Click Run to execute in your browser with Pyodide.

Mini Challenge: Deepen your understanding
Challenge 1: Add a numerical gradient check

After computing the analytical gradients for w_output, verify them numerically. Perturb each weight by h = 1e-5, recompute the loss, and check that (loss_plus - loss_minus) / (2*h) matches your analytical gradient within 1e-4.

Hint: Loop over each element of w_output. Save the original value. Set w_output[i,j] += h, run the full forward pass, record loss_plus. Set w_output[i,j] -= 2*h, run forward, record loss_minus. Restore the original. The numerical gradient is (loss_plus - loss_minus) / (2*h). Compare to your analytical gradient matrix at [i,j].

Challenge 2: Replace sigmoid with ReLU

Replace sigmoid with relu(x) = max(0, x) and its derivative relu_deriv(x) = (x > 0).astype(float). Keep sigmoid only on the output layer. Does training converge faster? What happens if you use ReLU on the output layer too?

Hint: Define relu = lambda x: np.maximum(0, x) and relu_deriv = lambda x: (x > 0).astype(float). Use ReLU for the hidden layer only. For the hidden delta, use relu_deriv(hidden_z) not relu_deriv(hidden_a) — ReLU's derivative depends on the pre-activation value. Using ReLU on the output squashes nothing, so predictions can exceed [0,1] — keep sigmoid there.

Test your knowledge

Question 1 of 5
What does the chain rule allow you to compute?
Question 2 of 5
Why can't a single neuron solve XOR?
Question 3 of 5
What is the key difference between MSE and cross-entropy loss?
Question 4 of 5
In the backward pass, what does multiplying by sigmoid_deriv accomplish?
Question 5 of 5
Why do we divide the weight gradient by the number of samples (N)?
0
/5
Excellent! You understand backpropagation and gradient flow.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You've mastered backpropagation.
Now build the full transformer.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →