🧬 From biological neuron to artificial neuron

Your brain contains roughly 86 billion neurons. Each one receives signals from other neurons through branch-like structures called dendrites, processes those signals in the cell body, and if the combined signal is strong enough, fires an output through the axon.

An artificial neuron does the same thing — in math:

Dendrites become inputs (x1, x2, ...). Synaptic strengths become weights (w1, w2, ...). The cell body computes a weighted sum + bias. The axon firing becomes the activation function deciding the output.

The formula for a single neuron is deceptively simple:

z = w1*x1 + w2*x2 + ... + wn*xn + b   then   output = activation(z)

That's it. Every neural network — from a 2-input AND gate to GPT-4 with 1.8 trillion parameters — is built from variations of this equation.

x1 input x2 input w1 w2 Σ + b sum bias (b) z activation f(z) out 0 or 1
A single artificial neuron: inputs multiplied by weights, summed with a bias, then passed through an activation function.

⚖️ The weighted sum

The weighted sum is the core computation. Each input gets multiplied by its weight (how important that input is), and the results are summed together with a bias term.

Think of it like a voting system. Each input is a voter, each weight is how influential that voter is, and the bias is the default disposition before any votes come in.

🔌
Analogy
The AND gate as a neuron

An AND gate outputs 1 only when both inputs are 1. Can a single neuron learn this? Yes! With the right weights and bias:

  • w1 = 0.5, w2 = 0.5, bias = -0.7
  • When both inputs are 1: z = 0.5 + 0.5 - 0.7 = 0.3 (positive, fires)
  • When only one input is 1: z = 0.5 - 0.7 = -0.2 (negative, doesn't fire)
  • When both are 0: z = -0.7 (negative, doesn't fire)
Python and_gate_neuron.py
Output
Input: [0 0] → z=-0.70 → sigmoid=0.3318 → predict: 0 (actual: 0) Input: [0 1] → z=-0.20 → sigmoid=0.4502 → predict: 0 (actual: 0) Input: [1 0] → z=-0.20 → sigmoid=0.4502 → predict: 0 (actual: 0) Input: [1 1] → z=0.30 → sigmoid=0.5744 → predict: 1 (actual: 1)

The neuron correctly implements AND. With weights = [0.5, 0.5] and bias = -0.7, only when both inputs are 1 does the weighted sum exceed the threshold.

OR gate — just change the bias

An OR gate fires when at least one input is 1. We keep the same weights but make the bias less negative:

Python or_gate_neuron.py
Output
OR gate: Input: [0 0] → predict: 0 Input: [0 1] → predict: 1 Input: [1 0] → predict: 1 Input: [1 1] → predict: 1

Key insight: The same neuron structure can learn AND or OR — the only difference is the weights and bias. Learning = finding the right weights. That's what training a neural network means.

⚡ Activation functions: sigmoid, ReLU, tanh

The activation function decides whether (and how strongly) the neuron fires. Without it, a neuron is just a linear equation — and stacking linear equations gives you... another linear equation. Activation functions introduce non-linearity, which is what lets neural networks learn complex patterns.

Sigmoid: the classic

Squashes any input to a value between 0 and 1. Historically used everywhere, but has a vanishing gradient problem — more on that in the backpropagation lesson.

sigmoid(z) = 1 / (1 + e^(-z))    Output range: (0, 1)

ReLU: the modern workhorse

Returns 0 for negative inputs, and the input itself for positive inputs. Simple, fast, and works remarkably well. Used in most modern networks.

ReLU(z) = max(0, z)    Output range: [0, infinity)

Tanh: the centered version

Like sigmoid but centered at zero — outputs range from -1 to 1. Often used in RNNs and when you need negative outputs.

tanh(z) = (e^z - e^(-z)) / (e^z + e^(-z))    Output range: (-1, 1)

Python activation_comparison.py
Output
z | sigmoid | ReLU | tanh -----+---------+--------+------- -5 | 0.0067 | 0.00 | -0.9999 -2 | 0.1192 | 0.00 | -0.9640 -1 | 0.2689 | 0.00 | -0.7616 0 | 0.5000 | 0.00 | 0.0000 1 | 0.7311 | 1.00 | 0.7616 2 | 0.8808 | 2.00 | 0.9640 5 | 0.9933 | 5.00 | 0.9999
Sigmoid z 0 1 ReLU z Tanh z -1 1
Sigmoid squashes to (0,1). ReLU clips negatives to 0. Tanh centers the output at zero (-1,1).
💡 Which activation to use?

Hidden layers: ReLU (or variants like LeakyReLU, GELU) for most cases. Binary classification output: Sigmoid. Multi-class output: Softmax. RNNs: Tanh. When in doubt, start with ReLU.

🎯 A single neuron as a classifier

A single neuron with sigmoid activation is a binary classifier — it draws a straight line (or hyperplane in higher dimensions) that separates two classes. This is exactly what logistic regression does. In fact, logistic regression is a single neuron.

📐
Geometric View
The decision boundary

The equation w1*x1 + w2*x2 + b = 0 defines a line. Everything on one side gets classified as 1, everything on the other as 0. The weights determine the angle of the line, and the bias shifts it.

This is why a single neuron can learn AND and OR (linearly separable) but cannot learn XOR — XOR requires a curved boundary that no single line can draw. You need multiple neurons (a hidden layer) for that.

🔄 The forward pass

The forward pass is the process of feeding inputs through the neuron (or network) to produce an output. It's "forward" because data flows in one direction: input → computation → output.

For a single neuron, the forward pass is three steps:

  1. Weighted sum: z = dot(inputs, weights) + bias
  2. Activation: output = activation(z)
  3. Prediction: Compare output to threshold (e.g., 0.5)

Later, when we stack neurons into layers and layers into networks, the forward pass chains these computations — the output of one layer becomes the input to the next. That's the "deep" in deep learning.

📉 A neuron that learns: gradient descent

So far we've hand-picked weights. But the power of neural networks is that they learn the right weights from data. The algorithm is gradient descent: measure how wrong you are (loss), compute which direction to adjust weights (gradient), and take a small step in that direction.

Python neuron_gradient_descent.py
Output
Epoch 0 | loss=0.6931 | w=0.075 | b=-0.050 Epoch 50 | loss=0.2085 | w=2.551 | b=-8.583 Epoch 100 | loss=0.1061 | w=3.870 | b=-13.185 Epoch 150 | loss=0.0710 | w=4.768 | b=-16.337 Final predictions: x=1 → 0.0001 → class 0 (actual: 0) x=2 → 0.0044 → class 0 (actual: 0) x=3 → 0.1227 → class 0 (actual: 0) x=4 → 0.8239 → class 1 (actual: 1) x=5 → 0.9905 → class 1 (actual: 1)

The neuron started with random weights and learned to separate the two classes. The loss dropped from 0.69 (random guessing) to 0.07. This is the same principle behind training GPT — just scaled to billions of parameters.

What just happened: The neuron learned that w ≈ 4.8 and b ≈ -16.3 create a decision boundary near x = 3.4. Points below get classified 0, points above get classified 1. We didn't hand-code this — the gradient descent algorithm found it.

🐍 Try It Yourself

Experiment with different weights and biases. Can you make the neuron act as an AND gate? An OR gate? What happens if you set both weights to 1.0 and bias to -1.5?

🐍 Try It Yourself — Neuron Playground
Output
Click ▶ Run to execute...
Challenges

Challenge 1: Build a NAND gate

A NAND gate outputs 1 for everything except when both inputs are 1. Find weights and bias that make a single neuron implement NAND. (Hint: negate the AND gate weights.)

Hint: If AND uses w=[0.5, 0.5] and b=-0.7, try flipping the signs: w=[-0.5, -0.5] and b=0.7. When both inputs are 1, z = -0.5 - 0.5 + 0.7 = -0.3, so sigmoid gives less than 0.5. When both are 0, z = 0.7, so sigmoid gives more than 0.5.

Challenge 2: Why can't a single neuron learn XOR?

XOR outputs 1 when inputs differ: (0,1)→1, (1,0)→1, (0,0)→0, (1,1)→0. Try to find weights and bias that work. Explain why it's impossible with a single neuron.

Hint: A single neuron draws a straight line as its decision boundary. Plot the XOR truth table on a 2D grid — the 1s are at (0,1) and (1,0), the 0s at (0,0) and (1,1). No single straight line can separate them. You need at least 2 neurons in a hidden layer, which is exactly what the next lesson covers.

Test your knowledge

Question 1 of 5
What does the bias term (b) do in a neuron?
Question 2 of 5
Which activation function outputs values in the range (0, 1)?
Question 3 of 5
Why are activation functions necessary in neural networks?
Question 4 of 5
A single neuron cannot learn XOR because:
Question 5 of 5
In gradient descent, what does the learning rate control?
0
/5
Great job!

⚠️ Common mistakes

  • Forgetting the bias: Without a bias term, the decision boundary is forced to pass through the origin. This severely limits what the neuron can learn.
  • Confusing weights with inputs: Inputs are the data. Weights are the learnable parameters. The neuron learns by adjusting weights, not by changing the data.
  • Using sigmoid everywhere: Sigmoid causes vanishing gradients in deep networks. Use ReLU for hidden layers and sigmoid only for binary classification outputs.
  • Learning rate too high: If your loss jumps around or diverges, the learning rate is too large. Start with 0.01 or 0.001 and adjust.
  • Expecting a single neuron to solve everything: A single neuron can only learn linearly separable problems. Real-world problems need layers of neurons.

🔑 Key takeaways

  • ✅ A neuron computes: output = activation(dot(inputs, weights) + bias). That's the entire forward pass.
  • ✅ Activation functions (sigmoid, ReLU, tanh) introduce non-linearity — without them, any network collapses to a single linear function.
  • ✅ A single neuron is a linear classifier. It can learn AND and OR gates, but not XOR. For complex problems, you need layers.
  • ✅ Gradient descent adjusts weights to minimize loss. Learning rate controls step size. This is how neural networks "learn."
  • ✅ Everything in deep learning — from a single neuron to GPT — follows this pattern: forward pass, compute loss, compute gradients, update weights.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You just built a neuron from scratch.
Imagine building an entire LLM.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →