Why build a full neural network?

In Lessons 1 and 2, you built a single neuron and learned backpropagation. That was a chain of one-dimensional operations. Real networks have layers: stacks of neurons where each layer's output feeds the next layer's input. Every modern LLM, from GPT to Llama, is just a very deep stack of layers.

Building a full network in NumPy teaches you what PyTorch, TensorFlow, and JAX do behind the scenes. When you write nn.Linear(16, 8) in PyTorch, it creates a weight matrix and a bias vector. When you call loss.backward(), it runs the exact chain rule computation you'll implement here. No mystery.

๐Ÿงช
What you'll build
A NeuralNetwork class with forward, backward, and train

By the end of this lesson, you'll have a NeuralNetwork class that takes any list of layer sizes (e.g., [2, 16, 8, 1]), initializes weights using He initialization, runs a forward pass through ReLU hidden layers and a sigmoid output, computes gradients via backpropagation, and trains with gradient descent. You'll classify the moons dataset at 97.5% accuracy with 193 total parameters.

Layers and architecture

A neural network layer is a matrix multiply followed by an activation function. If your input has 2 features and a layer has 16 neurons, the weight matrix is 2 x 16, the bias is 1 x 16, and the output is m x 16 (where m is the number of samples).

Stack multiple layers and you get a deep network. Our architecture: [2, 16, 8, 1] means input layer (2 features), first hidden layer (16 neurons), second hidden layer (8 neurons), and output layer (1 neuron).

Input 2 neurons Hidden 1 16 neurons Hidden 2 8 neurons Output 1 neuron x1 x2 ... ... y W: 2x16 W: 16x8 W: 8x1
Parameter count:
Layer 1: 2x16 + 16 = 48. Layer 2: 16x8 + 8 = 136. Layer 3: 8x1 + 1 = 9. Total: 193 parameters. GPT-3 has 175 billion. Same idea, different scale.

Weight initialization: Xavier vs He

If you initialize all weights to zero, every neuron computes the same thing. If you initialize too large, activations explode. Too small, gradients vanish. Initialization is one of the most important choices in deep learning.

Xavier initialization (Glorot)

Scale weights by sqrt(1 / fan_in) where fan_in is the number of inputs. Works well with sigmoid and tanh activations. Named after Xavier Glorot, who proved in 2010 that this keeps variance stable through layers.

He initialization

Scale weights by sqrt(2 / fan_in). Designed for ReLU activations. The factor of 2 compensates for ReLU killing half the values (all negatives become zero). Named after Kaiming He, who showed this in 2015.

initialization.py Xavier vs He comparison
Output
Xavier โ€” mean: -0.000879, std: 0.0884 expected std: 0.0884 He โ€” mean: -0.001243, std: 0.1250 expected std: 0.1250 Too big โ€” mean: -0.009931, std: 1.0009 Activations will explode!
Rule of thumb:
Use He initialization with ReLU. Use Xavier with sigmoid or tanh. Our network uses ReLU hidden layers, so we use He: np.random.randn(...) * np.sqrt(2.0 / fan_in).

The full NeuralNetwork class

Here's the complete implementation. The constructor takes a list of layer sizes and creates weight matrices and bias vectors. Forward propagation chains matrix multiplies through ReLU activations (with sigmoid on the output). Backpropagation computes gradients layer by layer, updating weights with gradient descent.

neural_network.py Complete implementation
Output
Epoch 0: loss = 0.424579 Epoch 200: loss = 0.045763 Epoch 400: loss = 0.015617 Epoch 600: loss = 0.013096 Epoch 800: loss = 0.012687 Epoch 1000: loss = 0.012204 Accuracy: 97.5% Architecture: [2, 16, 8, 1] Total parameters: 193

Loss drops from 0.42 to 0.01 over 1000 epochs. The network learns a non-linear decision boundary that separates the two moons. 97.5% accuracy means only 5 out of 200 points are misclassified.

Training loss curve

Here's the loss at each checkpoint. The steepest drop happens in the first 200 epochs. After epoch 400, the network is fine-tuning its decision boundary.

0.45 0.30 0.20 0.10 0.00 Loss (MSE) 0 200 400 600 800 1000 Epoch 0.4246 0.0458 0.0156 0.0127 0.0122

Learning rate: too high, too low, just right

The learning rate controls how big a step you take during gradient descent. It's the single most important hyperparameter in training.

  • Too high (lr=5.0): Loss oscillates wildly or diverges to NaN. The network overshoots the minimum on every step.
  • Too low (lr=0.001): Loss decreases, but painfully slowly. After 1000 epochs, the network hasn't converged.
  • Just right (lr=0.5): Steep initial drop, smooth convergence. Our network reaches 97.5% accuracy in 1000 epochs.
lr=5.0
Diverges
lr=0.001
Too slow
lr=0.5
97.5% acc
Practical advice:
Start with lr=0.01 and try 0.1, 0.5, 1.0. If loss goes to NaN, halve the learning rate. If loss barely moves, double it. This manual search is what researchers did before learning rate schedulers.

Mini-batch gradient descent

Our code above uses full-batch gradient descent: every epoch computes the gradient over all 200 samples. This works for small datasets, but with millions of samples (like LLM training data), computing the gradient over the entire dataset for each step is too slow.

Three flavors of gradient descent

  • Batch (full): Use all samples per step. Stable gradients, slow on large data.
  • Stochastic (SGD): Use 1 sample per step. Noisy gradients, but fast updates.
  • Mini-batch: Use a small batch (32, 64, 128 samples) per step. Best of both worlds. This is what every LLM uses.
Shuffle data
โ†’
Split into batches
โ†’
Forward + backward per batch
โ†’
Update weights

Mini-batch adds noise to the gradient, which actually helps the network escape shallow local minima. It's one of the reasons SGD works so well in practice despite being "noisy."

Overfitting vs underfitting

Two failure modes of neural networks:

๐Ÿ“‰
Underfitting
The model is too simple to capture the pattern

A network with [2, 1] (no hidden layers) can only learn a straight line. The moons dataset is curved, so the best a linear model can do is ~85% accuracy. Fix: add more layers or more neurons.

๐Ÿ“ˆ
Overfitting
The model memorizes noise instead of learning the pattern

A network with [2, 256, 256, 256, 1] has tens of thousands of parameters for 200 data points. It memorizes every point, including noise. Training accuracy: 100%. Test accuracy: could be 80%. Fix: reduce model size, add dropout, use early stopping, get more data.

Our [2, 16, 8, 1] is the sweet spot:
193 parameters for 200 samples. Enough capacity to learn the curved boundary, not enough to memorize noise. The 97.5% accuracy confirms the model generalizes well.

How this connects to building an LLM

Every component of GPT is built from these same primitives. The transformer's feed-forward network? Two linear layers with a ReLU in between, exactly like our hidden layers. Attention heads? Matrix multiplies with learned weights. The training loop? Forward pass, compute loss, backward pass, update weights. The only difference is scale: GPT-3 has 96 layers and 175 billion parameters. You just built the same pattern with 3 layers and 193 parameters. In the next lesson, you'll translate this NumPy code to PyTorch, where nn.Linear replaces your manual weight matrices and loss.backward() replaces your manual chain rule.

Transformer FFN Weight Matrices Loss Functions PyTorch Autograd

Try it yourself

Experiment with the neural network below. Try changing the architecture, learning rate, or number of epochs. The code runs in your browser using Pyodide (Python in WebAssembly).

Interactive: Build and train your own network

Click "Run Code" to execute. Try changing [2, 16, 8, 1] to [2, 32, 16, 1] or lr=0.5 to lr=0.01.
Mini Challenge: Experiment with architectures

Challenge 1: Underfitting

Change the architecture to [2, 1] (no hidden layers). What accuracy do you get? Why can't a single linear layer classify the moons?

Challenge 2: Deeper network

Try [2, 32, 32, 16, 1]. Does more depth help? Count the parameters.

Challenge 3: Learning rate sweep

Train the same [2, 16, 8, 1] network with lr=0.001, 0.1, 0.5, and 5.0. Which converges fastest? Which diverges?

Hint 1: [2, 1] gives ~50-65% accuracy because the decision boundary is linear, but moons are curved.

Hint 2: [2, 32, 32, 16, 1] has 32*2+32 + 32*32+32 + 32*16+16 + 16*1+1 = 1,713 parameters. More parameters than data points, so watch for overfitting.

Hint 3: lr=5.0 will likely produce NaN loss. lr=0.001 will still be above 0.1 loss at epoch 1000.

Test your knowledge

Question 1 of 5
Why does He initialization use sqrt(2/fan_in) instead of sqrt(1/fan_in)?
Question 2 of 5
What happens when the learning rate is too high?
Question 3 of 5
In a [2, 16, 8, 1] network, what is the shape of the weight matrix between the first hidden layer (16 neurons) and the second hidden layer (8 neurons)?
Question 4 of 5
Why does mini-batch gradient descent work better than full-batch on large datasets?
Question 5 of 5
A model has 100% training accuracy but 75% test accuracy. What is the most likely problem?
0
/5
Great work! You understand neural network architecture and training.
Africa & India's Best AI Academy
๐Ÿค– AI-first ๐ŸŽ“ Placement Assistance

You've built a neural network from scratch.
Now build the models that power the industry.

Join the Best AI Program: The AI-Era Agentic Engineer โ€” 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer โ†’