Why sequential data needs a new architecture

In the previous lessons, every input was independent: a fixed-size vector goes in, a prediction comes out. But language is inherently sequential. The meaning of "bank" depends on whether the previous words were "river" or "savings." A feedforward network sees each input in isolation — it has no memory of what came before.

🧪
The problem
Predicting the next character

Given the string "hell", what comes next? Probably "o". Given "hello worl", probably "d". To make these predictions, the network needs to remember what it has seen so far. A feedforward network can't do this — it processes each character independently. An RNN carries a hidden state that accumulates information from every previous time step.

Sequential data is everywhere beyond language: stock prices over time, audio waveforms, sensor readings, DNA sequences. Any task where order matters needs an architecture that respects order. The RNN was the first neural architecture designed for this — and understanding it is essential before you can appreciate why the transformer replaced it.

The vanilla RNN: one equation, repeated

An RNN processes a sequence one element at a time. At each time step t, it takes two inputs: the current input xt and the previous hidden state ht-1. It produces a new hidden state ht that encodes everything the network has seen so far.

Vanilla RNN equations
ht = tanh(Wxh · xt + Whh · ht-1 + bh)
yt = Why · ht + by

That's it. The same two weight matrices (Wxh and Whh) are applied at every time step — the RNN "unrolls" by repeating itself. The hidden state ht is the network's memory: a compressed summary of all inputs up to time t.

RNN unrolled through "hello"

Here is the same RNN cell processing the word "hello" character by character. Notice how the hidden state passes from one step to the next, accumulating context:

h₀=0 t=1 t=2 t=3 t=4 t=5 RNN RNN RNN RNN RNN h₁ h₂ h₃ h₄ 'h' 'e' 'l' 'l' 'o' P('e') P('l') P('l') P('o') P(' ') Input Hidden state Prediction

The key insight: the same weight matrices Wxh, Whh, and Why are shared across all time steps. The RNN is not five different networks — it is one network applied five times, with the hidden state threading through as memory.

Character-level RNN in NumPy

Let's build a character-level RNN from scratch. The task: given a character, predict the next character. We train on a simple text corpus and then generate new text by sampling from the model's predictions.

char_rnn_numpy.py NumPy — runs in Pyodide
Output
Vocabulary (13 chars): [' ', 'a', 'd', 'e', 'g', 'h', 'i', 'l', 'n', 'o', 'r', 't', 'w'] Epoch 0: loss = 2.5497 Epoch 100: loss = 1.5732 Epoch 200: loss = 1.7687 Epoch 300: loss = 1.8347 Epoch 400: loss = 1.8650 Epoch 500: loss = 1.8807 Generated text (from "h"): helo thelo ain ain again ain ai

The model learned the structure of its training data. "helo thelo ain" resembles "hello there" — not perfect, but remarkable given that it learned character by character from just 35 characters of text. Notice the loss plateaus and even rises slightly after epoch 100 — this is a symptom of the vanishing gradient problem, which we address next.

Why character-level?
Character-level models are simple to understand: the vocabulary is just letters and spaces. Real LLMs use subword tokenization (BPE) instead — you'll build that in future lessons. But the RNN architecture is identical at any granularity.

The vanishing gradient problem

The vanilla RNN has a fatal flaw. During backpropagation through time (BPTT), gradients must flow backward through every time step. At each step, they're multiplied by the weight matrix Whh and the derivative of tanh. After many steps, these repeated multiplications either shrink the gradient to near-zero (vanishing) or explode it to infinity (exploding).

Why gradients vanish

The derivative of tanh is always between 0 and 1. Multiply a number by 0.2 fifty times and you get 0.250 — effectively zero. This means the RNN cannot learn dependencies longer than roughly 10-20 time steps. If the context needed to predict the next word was 50 characters ago, the gradient from that error signal has vanished by the time it reaches the relevant time step.

Gradient magnitude during backprop (vanishing) t=10 1.00 t=8 0.81 t=6 0.53 t=4 0.28 t=3 0.10 t=2 0.04 t=1 ~0 Loss signal

The gradient at t=1 is effectively zero — the network can't learn from early context. This is why our character-level RNN's loss plateaued: it could only learn short-range patterns like "he" -> "l" but not long-range dependencies like remembering to close a word it started 20 characters ago.

Exploding gradients
are the opposite problem: gradient values grow exponentially. The fix is simple — gradient clipping (cap gradients at a max value, like we did with np.clip(grad, -5, 5)). Vanishing gradients are harder because you can't force a near-zero signal to become large without changing the architecture.

LSTM: Long Short-Term Memory

The LSTM (Hochreiter & Schmidhuber, 1997) solves vanishing gradients with a clever trick: it adds a cell state Ct that flows through time with minimal interference. Instead of one tanh squashing everything, the LSTM uses three gates to control what information flows in, what gets forgotten, and what gets output.

The three gates

LSTM gate equations
Forget gate (what to erase from memory):
ft = sigmoid(Wf · [ht-1, xt] + bf)

Input gate (what new info to store):
it = sigmoid(Wi · [ht-1, xt] + bi)
C̃t = tanh(WC · [ht-1, xt] + bC)

Cell state update:
Ct = ft * Ct-1 + it * C̃t

Output gate (what to expose as hidden state):
ot = sigmoid(Wo · [ht-1, xt] + bo)
ht = ot * tanh(Ct)

LSTM cell diagram

Cell State C (information highway) Ct-1 Ct Forget Gate f = sigmoid() x Input Gate i = sigmoid() C̃ = tanh() candidate x + Output Gate o = sigmoid() tanh(Ct) x ht [ht-1, xt] Forget: erase old info Input: write new info Output: read from cell Green line: cell state highway

The cell state (green line at top) is the key innovation. It flows through time with only element-wise operations (multiply and add) — no matrix multiplications, no tanh squashing. This means gradients can flow backward through the cell state unimpeded for hundreds of time steps. The gates learn when to write, erase, and read from this long-term memory.

Why sigmoid gates?
Sigmoid outputs values between 0 and 1, which act as soft switches. A forget gate value of 0.9 means "keep 90% of the old memory." A value of 0.1 means "erase 90%." The network learns these gate values during training — it discovers which information to remember and which to discard.

LSTM in code: NumPy and PyTorch

Here's a single LSTM step in NumPy, showing exactly how the gates work. Then the PyTorch version, which wraps everything into nn.LSTM.

NumPy LSTM step (conceptual)

lstm_step_numpy.py Read-only — conceptual
Output
Input shape: (1, 13) Hidden state shape: (1, 64) Cell state shape: (1, 64) Parameters: 4 * 77 * 64 = 19712

Notice: an LSTM has 4x the parameters of a vanilla RNN (four weight matrices instead of one). Each gate is essentially its own neural network layer.

PyTorch nn.LSTM

lstm_pytorch.py Read-only — PyTorch required
Output
Total parameters: 34,573 Embedding: 832 LSTM: 33,024 Output FC: 845 Input shape: torch.Size([1, 5]) Output shape: torch.Size([1, 5, 13]) Hidden shape: torch.Size([1, 1, 64]) Cell shape: torch.Size([1, 1, 64])

PyTorch's nn.LSTM handles all four gates internally. You pass in a sequence and get back the output at every time step plus the final hidden and cell states. The API is simple; the gate math inside is exactly what we wrote by hand.

GRU: the simplified LSTM

The Gated Recurrent Unit (Cho et al., 2014) simplifies the LSTM by merging the forget and input gates into a single update gate and combining the cell state and hidden state. Fewer parameters, often comparable performance.

Feature LSTM GRU
Gates 3 (forget, input, output) 2 (reset, update)
States ht + Ct (separate) ht only
Parameters 4 * (n + m) * n 3 * (n + m) * n
Speed Slower (more computation) ~25% faster
Long sequences Often slightly better Competitive
PyTorch nn.LSTM nn.GRU
GRU equations
Update gate: zt = sigmoid(Wz · [ht-1, xt])
Reset gate: rt = sigmoid(Wr · [ht-1, xt])
Candidate: h̃t = tanh(W · [rt * ht-1, xt])
Output: ht = (1 - zt) * ht-1 + zt * h̃t

The update gate zt acts like a combined forget+input gate: when zt is close to 0, the old hidden state passes through unchanged (remembering); when close to 1, the new candidate replaces it (updating). In practice, GRU and LSTM perform similarly on most tasks — GRU is a good default when you want fewer parameters.

Bidirectional RNNs

A standard RNN reads left to right. But in many tasks, future context matters too. "The animal didn't cross the street because it was too [wide/tired]" — to fill in the blank, you need both what came before and what comes after.

A bidirectional RNN runs two separate RNNs: one left-to-right and one right-to-left. Their hidden states are concatenated at each time step, giving the model access to both past and future context.

Forward RNN →
h1, h2, ... hT
+
← Backward RNN
h'T, h'T-1, ... h'1
=
[ht; h't]
Full context at each step
bidirectional_pytorch.py Read-only — PyTorch required
Output
Output shape: torch.Size([1, 10, 128]) Hidden shape: torch.Size([2, 1, 64])
Bidirectional RNNs can't generate text.
They need the full sequence upfront (to read it backward), so they're useless for next-token prediction. They're used for classification, tagging, and encoding (like BERT's encoder). For language generation, you need unidirectional models — which is what GPT does.

Try It Yourself

Character-Level RNN — train and generate text

This runs the NumPy RNN in your browser with Pyodide. Train on the sample text and watch it learn to generate similar character patterns. Try changing the text or hidden_size.

Output
Click "Run Code" to train the character-level RNN...

Why this matters for building an LLM

RNNs and LSTMs were the dominant language model architecture from 2013-2017. Google Translate, OpenAI's early text generators, and most chatbots used stacked LSTMs. But RNNs process tokens one at a time — they can't be parallelized on GPUs. This sequential bottleneck is what motivated the creation of the transformer (next arc). Understanding why RNNs fail at scale is essential to appreciating what transformers fix.

Sequential processing Hidden state = memory LSTM gates = learned forgetting Can't parallelize Replaced by attention

Next lesson: Seq2Seq & The Bottleneck — you'll see how two RNNs (encoder + decoder) can translate languages, and why compressing an entire sentence into a single vector creates the bottleneck that attention was invented to solve.

Challenge

Put it into practice
Challenge 1: Longer training text

Replace the short training text with a longer string — try a paragraph from a book or song lyrics. Increase hidden_size to 64 and train for 1000 epochs. Does the generated text capture more structure? What happens if you lower the learning rate to 0.01?

With more text, the vocabulary grows (more unique characters). You may need to increase the training epochs proportionally. Lower learning rate gives smoother convergence but slower training. Try generating from different seed characters.
Challenge 2: Implement LSTM training

Take the LSTM step function from the conceptual code section and build a full training loop. Train it on the same "hello world" text and compare the loss curve to the vanilla RNN. Does the LSTM converge faster or achieve lower loss?

The tricky part is backpropagation through the gates. For a first attempt, use a simplified gradient (backprop through only the last time step, like the vanilla RNN code does). For full BPTT through an LSTM, you'll need to store all gate values at each time step and reverse-propagate through them.

Quiz

Question 1 of 5
What is the hidden state in an RNN?
Question 2 of 5
Why do vanilla RNNs struggle with long sequences?
Question 3 of 5
Which LSTM gate decides what old information to discard from the cell state?
Question 4 of 5
How does a GRU differ from an LSTM?
Question 5 of 5
Why can't bidirectional RNNs be used for text generation?
0
/5
Well done! You understand how RNNs process sequences and why LSTMs were a breakthrough.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You've trained a neural network to generate text.
Now build the architecture that powers ChatGPT.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →