⚡
Real-world scenario
Aisha at M-Kopa Solar (Nairobi): scaling from perceptron to a real neural network

Last lesson, Aisha built a single perceptron—a linear decision boundary. But medical charge prediction isn't linear. A single neuron can't capture the complex relationships: how age, BMI, smoker status, and region combine to determine charges. This lesson, she builds her first true neural network: 3 hidden layers (64, 32, 1 neurons) that learn non-linear transformations of her features. Her R² jumps from 0.941 to 0.9575. This is the leap from rules to learning.

By the end, Aisha can predict charges for any customer profile and explain how the network learned to spot complex patterns in the data. She's no longer writing rules; the network writes them.

2,561
Total parameters
3
Hidden layers
0.9575
Final R² Score
📊 Dataset: Insurance charges — 1,338 rows × 7 columns (age, BMI, smoker, region, children, sex → charges). First introduced in Lesson 01.

Architecture of FirstNet — the network design

How the layers connect

Input layer (6 neurons): age, bmi, smoker, region, children, sex — raw features from insurance data.

Hidden layer 1 (64 neurons): each neuron is a weighted combination of all 6 inputs, passed through ReLU activation. It learns initial charge patterns: "if age is high and smoker is yes, charge goes up." 6×64 = 384 weights + 64 biases = 448 parameters.

Hidden layer 2 (32 neurons): receives 64 outputs from layer 1, learns to combine those patterns. 64×32 = 2,048 weights + 32 biases = 2,080 parameters. Dimension reduction (64→32) is deliberate—learns a compressed representation.

Output layer (1 neuron): final prediction of charges. 32×1 = 32 weights + 1 bias = 33 parameters. No activation function (we want unbounded regression, not 0-1 range).

Total: 448 + 2,080 + 33 = 2,561 parameters. Each one learned during training.

FirstNet architecture: 6 inputs → 64 hidden neurons → 32 hidden neurons → 1 output. ReLU activations between layers.
Neural network architecture diagram showing layer structure and connections

Why this architecture?

Layer 1 (64 units): Learns basic feature combinations. One neuron might learn "smoker_yes + high_age → high_charge". Another learns "high_bmi → moderate_charge". 64 neurons capture many such patterns in parallel.

Layer 2 (32 units): Combines patterns from layer 1. Learns higher-level abstractions: "smoker + age both high → very high charge". The dimension reduction (64→32) is a bottleneck—forces the network to compress information, learning only the most important patterns.

No BatchNorm or Dropout yet: For this lesson, we keep it simple. ReLU activation is fast and avoids vanishing gradients (where gradients approach zero). No output activation because regression targets are unbounded (charges can be $5k–$50k).

Depth > Width: 3 layers with 64→32→1 neurons work better than a single wide layer (e.g., 6→128→1). Depth allows learning hierarchical, multi-step transformations.

Step 1: Define the network architecture

We inherit from nn.Module and define layers in __init__, then the forward pass in forward().

firstnet.py Define a 3-layer neural network with 2,561 parameters
Output
Total parameters: 2561 Model architecture: FirstNet( (fc1): Linear(in_features=6, out_features=64, bias=True) (fc2): Linear(in_features=64, out_features=32, bias=True) (fc3): Linear(in_features=32, out_features=1, bias=True) )

Step 2: Define loss function and optimizer

MSE (Mean Squared Error) measures prediction error. Adam optimizer updates weights using adaptive learning rates—smart, fast convergence.

optimizer.py Set up MSE loss and Adam optimizer
Output
Loss function: MSELoss() Optimizer: Adam (lr=0.01) Learning rate: 0.01

Step 3: Prepare data with DataLoader

Batching speeds up training and stabilizes gradients. 32 samples per batch is standard for insurance data (1,338 samples = ~42 batches per epoch).

dataloader.py Create DataLoader for batching
Output
Total training samples: 1070 Batch size: 32 Batches per epoch: 34 First batch shape: X: torch.Size([32, 6]), y: torch.Size([32, 1])

Step 4: Training loop — learn weights over 100 epochs

For each epoch: (1) iterate through batches, (2) forward pass, (3) compute loss, (4) backward pass (compute gradients), (5) optimizer.step() (update weights). Also compute validation loss.

train.py Train FirstNet for 100 epochs with validation
Output
Epoch 25: Train 0.0338, Val 0.0409 Epoch 50: Train 0.0314, Val 0.0415 Epoch 75: Train 0.0299, Val 0.0414 Epoch 100: Train 0.0285, Val 0.0413

Training dynamics — what's happening under the hood?

Training loss decreases steadily (network learning patterns). Validation loss improves until epoch 25, then plateaus—an early stopping signal.

Training and validation loss over 100 epochs. Train loss keeps improving; validation plateaus around epoch 25.
Line chart showing training and validation loss curves
Validation loss stops improving after epoch 25. This is an early stopping signal—the model has learned the generalizable patterns; further training just fits noise. In production, you'd stop at epoch 30 and get slightly better validation performance (smaller generalization gap).

Step 5: Unscale predictions for business metrics

The model learned on scaled targets (mean=0, std=1). To report in dollars, we reverse the scaling.

unscale.py Unscale predictions and compute MAE and R²
Output
MAE: $1,789.17 → Average prediction is off by $1,789.17 R²: 0.9575 → Model explains 95.75% of variance

Step 6: Make a prediction — Arjun's profile

Arjun is 35 years old, BMI 22.5, non-smoker, 0 children. What will his annual charges be?

predict.py Predict charges for a new customer
Output
Arjun's profile: Age: 35, BMI: 22.5, Smoker: No, Region: 1, Children: 0 Predicted annual charges: $8,456.32

Actual vs predicted — how good are our forecasts?

Points close to the dashed line (perfect prediction) = good. Scattered widely = high error. Our R² = 0.9575 means 95.75% of variance is captured.

Predicted vs actual charges on test set. Points near the diagonal = accurate predictions. R² = 0.9575.
Scatter plot of predicted vs actual charges showing strong correlation

How AI systems use this

FirstNet is a foundation. This architecture—input → hidden layers → output—is used everywhere: image classification (conv layers instead of linear), NLP (transformers instead of ReLU), recommendation systems. The only difference is the activation function and layer type. The principle is always the same: learn hierarchical, non-linear transformations of data. Aisha's 3-layer network predicts charges; Google's BERT uses 24 layers to predict the next word. Same idea, different scale.

Multi-layer perceptron Forward pass Backpropagation Optimization Generalization

Also used in...

Domain Spotlight

🏥
Healthcare: Patient risk scoring
Age, vitals, lab results → 3-layer network predicts readmission risk. R² = 0.87, MAE = 0.12 (12% error in probability).
💰
Fintech: Loan default prediction
Income, credit score, employment → network predicts default probability. Banks use this to approve/reject loans.
⚡
Energy: Demand forecasting
Temperature, time-of-day, weather → grid demand prediction. M-Kopa Solar uses this to manage battery inventory.
🎯 Mini Challenge: Add BatchNorm and Dropout

Task: Modify FirstNet to add BatchNorm after layer 1 and Dropout (p=0.2) after layer 2. Why would these help? Train for 50 epochs and compare validation loss to the original.

Hint 1: Import: from torch.nn import BatchNorm1d, Dropout

Hint 2: In __init__: self.bn1 = BatchNorm1d(64) and self.drop = Dropout(p=0.2)

Hint 3: In forward(): x = self.bn1(torch.relu(self.fc1(x))) and x = self.drop(torch.relu(self.fc2(x)))

Hint 4: Remember to call model.train() during training and model.eval() during validation.

Expected: Validation loss should be slightly lower due to better regularization and stabilized gradients.

Interview questions: test your understanding

2,561 total. Breakdown: layer 1 has 6×64 + 64 = 448 parameters, layer 2 has 64×32 + 32 = 2,080 parameters, layer 3 has 32×1 + 1 = 33 parameters. 448 + 2,080 + 33 = 2,561.
Initial feature transformations. Each of the 64 neurons learns a different weighted combination of the 6 inputs. One might learn "if age is high AND smoker is yes, output high". Another learns "if BMI is high, output moderate". These 64 patterns are building blocks for higher layers.
Scaled targets (mean=0, std=1) make gradient updates stable. Raw targets range from $1k–$60k, so unscaled gradients become very large or very small, causing training instability (exploding/vanishing gradients). Scaling puts the target on the same numeric scale as the network's outputs.
Early stopping opportunity. After epoch 25, the model isn't learning generalizable patterns; it's fitting training noise (overfitting). The best test performance is around epoch 30. Continuing to epoch 100 increases validation loss slightly (worse test performance). Early stopping would save compute and improve real-world performance.
BatchNorm normalizes layer inputs (mean=0, std=1), stabilizing training and reducing internal covariate shift. Dropout randomly zeros neurons during training (not at inference), preventing co-adaptation (neurons learning to rely on each other instead of generalizing). Both improve generalization and allow higher learning rates without divergence.

Quiz: 5 questions to test your knowledge

Question 1 of 5
FirstNet has how many hidden neurons total (across both hidden layers)?
Question 2 of 5
What was the training loss at epoch 100?
Question 3 of 5
What was the final MAE on the test set?
Question 4 of 5
Which activation function is used between hidden layers?
Question 5 of 5
What was the final R² on the test set?
0
/5
Excellent work! You understand neural networks and training dynamics.

Cheatsheet: PyTorch neural networks at a glance

Component Syntax Purpose
Define layer nn.Linear(in, out) Fully connected layer with weights and biases
ReLU activation torch.relu(x) Non-linearity: max(0, x) — no vanishing gradients
Loss function nn.MSELoss() Mean squared error for regression targets
Optimizer torch.optim.Adam(model.parameters(), lr=0.01) Adaptive learning rate, combines momentum + RMSprop
DataLoader DataLoader(dataset, batch_size=32) Batching for efficient mini-batch gradient descent
Train mode model.train() Enable dropout and batchnorm updates
Eval mode model.eval() Disable dropout, use running stats for batchnorm
Backward pass loss.backward() Compute gradients via backpropagation
Update weights optimizer.step() Apply gradient updates to weights

Common mistakes: what breaks and how to fix it

Mistake 1: Forgetting model.eval() before validation
Dropout and BatchNorm behave differently in train vs eval mode. Skipping model.eval() means dropout is still active (zeros neurons), and batchnorm uses batch statistics instead of running statistics. Validation metrics will be artificially low.
Mistake 2: Not scaling the target variable
Raw targets ($1k–$60k) cause instability. Scaled targets (mean=0, std=1) stabilize gradient flow. Without scaling, loss.backward() produces very large or very small gradients, causing divergence or slow convergence.
Mistake 3: Overfitting — ignoring when validation loss plateaus
Training loss keeps dropping, but validation loss increases. Model is memorizing training noise. Fix: implement early stopping (stop at epoch 25–30), add dropout, or reduce model capacity.
Mistake 4: Using the wrong loss function
MSELoss is for regression (unbounded targets). For binary classification, use BCELoss or BCEWithLogitsLoss. For multi-class, use CrossEntropyLoss. Wrong loss = wrong gradients = poor learning.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You've built your first neural network.
Now train it on real-world production data.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →