Why build a full neural network?
In Lessons 1 and 2, you built a single neuron and learned backpropagation. That was a chain of one-dimensional operations. Real networks have layers: stacks of neurons where each layer's output feeds the next layer's input. Every modern LLM, from GPT to Llama, is just a very deep stack of layers.
Building a full network in NumPy teaches you what PyTorch, TensorFlow, and JAX do behind the scenes. When you write nn.Linear(16, 8) in PyTorch, it creates a weight matrix and a bias vector. When you call loss.backward(), it runs the exact chain rule computation you'll implement here. No mystery.
By the end of this lesson, you'll have a NeuralNetwork class that takes any list of layer sizes (e.g., [2, 16, 8, 1]), initializes weights using He initialization, runs a forward pass through ReLU hidden layers and a sigmoid output, computes gradients via backpropagation, and trains with gradient descent. You'll classify the moons dataset at 97.5% accuracy with 193 total parameters.
Layers and architecture
A neural network layer is a matrix multiply followed by an activation function. If your input has 2 features and a layer has 16 neurons, the weight matrix is 2 x 16, the bias is 1 x 16, and the output is m x 16 (where m is the number of samples).
Stack multiple layers and you get a deep network. Our architecture: [2, 16, 8, 1] means input layer (2 features), first hidden layer (16 neurons), second hidden layer (8 neurons), and output layer (1 neuron).
Weight initialization: Xavier vs He
If you initialize all weights to zero, every neuron computes the same thing. If you initialize too large, activations explode. Too small, gradients vanish. Initialization is one of the most important choices in deep learning.
Xavier initialization (Glorot)
Scale weights by sqrt(1 / fan_in) where fan_in is the number of inputs. Works well with sigmoid and tanh activations. Named after Xavier Glorot, who proved in 2010 that this keeps variance stable through layers.
He initialization
Scale weights by sqrt(2 / fan_in). Designed for ReLU activations. The factor of 2 compensates for ReLU killing half the values (all negatives become zero). Named after Kaiming He, who showed this in 2015.
np.random.randn(...) * np.sqrt(2.0 / fan_in).
The full NeuralNetwork class
Here's the complete implementation. The constructor takes a list of layer sizes and creates weight matrices and bias vectors. Forward propagation chains matrix multiplies through ReLU activations (with sigmoid on the output). Backpropagation computes gradients layer by layer, updating weights with gradient descent.
Loss drops from 0.42 to 0.01 over 1000 epochs. The network learns a non-linear decision boundary that separates the two moons. 97.5% accuracy means only 5 out of 200 points are misclassified.
Training loss curve
Here's the loss at each checkpoint. The steepest drop happens in the first 200 epochs. After epoch 400, the network is fine-tuning its decision boundary.
Learning rate: too high, too low, just right
The learning rate controls how big a step you take during gradient descent. It's the single most important hyperparameter in training.
- Too high (lr=5.0): Loss oscillates wildly or diverges to NaN. The network overshoots the minimum on every step.
- Too low (lr=0.001): Loss decreases, but painfully slowly. After 1000 epochs, the network hasn't converged.
- Just right (lr=0.5): Steep initial drop, smooth convergence. Our network reaches 97.5% accuracy in 1000 epochs.
Diverges
Too slow
97.5% acc
Mini-batch gradient descent
Our code above uses full-batch gradient descent: every epoch computes the gradient over all 200 samples. This works for small datasets, but with millions of samples (like LLM training data), computing the gradient over the entire dataset for each step is too slow.
Three flavors of gradient descent
- Batch (full): Use all samples per step. Stable gradients, slow on large data.
- Stochastic (SGD): Use 1 sample per step. Noisy gradients, but fast updates.
- Mini-batch: Use a small batch (32, 64, 128 samples) per step. Best of both worlds. This is what every LLM uses.
Mini-batch adds noise to the gradient, which actually helps the network escape shallow local minima. It's one of the reasons SGD works so well in practice despite being "noisy."
Overfitting vs underfitting
Two failure modes of neural networks:
A network with [2, 1] (no hidden layers) can only learn a straight line. The moons dataset is curved, so the best a linear model can do is ~85% accuracy. Fix: add more layers or more neurons.
A network with [2, 256, 256, 256, 1] has tens of thousands of parameters for 200 data points. It memorizes every point, including noise. Training accuracy: 100%. Test accuracy: could be 80%. Fix: reduce model size, add dropout, use early stopping, get more data.
How this connects to building an LLM
Every component of GPT is built from these same primitives. The transformer's feed-forward network? Two linear layers with a ReLU in between, exactly like our hidden layers. Attention heads? Matrix multiplies with learned weights. The training loop? Forward pass, compute loss, backward pass, update weights. The only difference is scale: GPT-3 has 96 layers and 175 billion parameters. You just built the same pattern with 3 layers and 193 parameters. In the next lesson, you'll translate this NumPy code to PyTorch, where nn.Linear replaces your manual weight matrices and loss.backward() replaces your manual chain rule.
Try it yourself
Experiment with the neural network below. Try changing the architecture, learning rate, or number of epochs. The code runs in your browser using Pyodide (Python in WebAssembly).
Interactive: Build and train your own network
Challenge 1: Underfitting
Change the architecture to [2, 1] (no hidden layers). What accuracy do you get? Why can't a single linear layer classify the moons?
Challenge 2: Deeper network
Try [2, 32, 32, 16, 1]. Does more depth help? Count the parameters.
Challenge 3: Learning rate sweep
Train the same [2, 16, 8, 1] network with lr=0.001, 0.1, 0.5, and 5.0. Which converges fastest? Which diverges?
Hint 1: [2, 1] gives ~50-65% accuracy because the decision boundary is linear, but moons are curved.
Hint 2: [2, 32, 32, 16, 1] has 32*2+32 + 32*32+32 + 32*16+16 + 16*1+1 = 1,713 parameters. More parameters than data points, so watch for overfitting.
Hint 3: lr=5.0 will likely produce NaN loss. lr=0.001 will still be above 0.1 loss at epoch 1000.
Test your knowledge
You've built a neural network from scratch.
Now build the models that power the industry.
Join the Best AI Program: The AI-Era Agentic Engineer โ 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Agentic Engineer โ