Why sequential data needs a new architecture
In the previous lessons, every input was independent: a fixed-size vector goes in, a prediction comes out. But language is inherently sequential. The meaning of "bank" depends on whether the previous words were "river" or "savings." A feedforward network sees each input in isolation — it has no memory of what came before.
Given the string "hell", what comes next? Probably "o". Given "hello worl", probably "d". To make these predictions, the network needs to remember what it has seen so far. A feedforward network can't do this — it processes each character independently. An RNN carries a hidden state that accumulates information from every previous time step.
Sequential data is everywhere beyond language: stock prices over time, audio waveforms, sensor readings, DNA sequences. Any task where order matters needs an architecture that respects order. The RNN was the first neural architecture designed for this — and understanding it is essential before you can appreciate why the transformer replaced it.
The vanilla RNN: one equation, repeated
An RNN processes a sequence one element at a time. At each time step t, it takes two inputs: the current input xt and the previous hidden state ht-1. It produces a new hidden state ht that encodes everything the network has seen so far.
yt = Why · ht + by
That's it. The same two weight matrices (Wxh and Whh) are applied at every time step — the RNN "unrolls" by repeating itself. The hidden state ht is the network's memory: a compressed summary of all inputs up to time t.
RNN unrolled through "hello"
Here is the same RNN cell processing the word "hello" character by character. Notice how the hidden state passes from one step to the next, accumulating context:
The key insight: the same weight matrices Wxh, Whh, and Why are shared across all time steps. The RNN is not five different networks — it is one network applied five times, with the hidden state threading through as memory.
Character-level RNN in NumPy
Let's build a character-level RNN from scratch. The task: given a character, predict the next character. We train on a simple text corpus and then generate new text by sampling from the model's predictions.
The model learned the structure of its training data. "helo thelo ain" resembles "hello there" — not perfect, but remarkable given that it learned character by character from just 35 characters of text. Notice the loss plateaus and even rises slightly after epoch 100 — this is a symptom of the vanishing gradient problem, which we address next.
The vanishing gradient problem
The vanilla RNN has a fatal flaw. During backpropagation through time (BPTT), gradients must flow backward through every time step. At each step, they're multiplied by the weight matrix Whh and the derivative of tanh. After many steps, these repeated multiplications either shrink the gradient to near-zero (vanishing) or explode it to infinity (exploding).
Why gradients vanish
The derivative of tanh is always between 0 and 1. Multiply a number by 0.2 fifty times and you get 0.250 — effectively zero. This means the RNN cannot learn dependencies longer than roughly 10-20 time steps. If the context needed to predict the next word was 50 characters ago, the gradient from that error signal has vanished by the time it reaches the relevant time step.
The gradient at t=1 is effectively zero — the network can't learn from early context. This is why our character-level RNN's loss plateaued: it could only learn short-range patterns like "he" -> "l" but not long-range dependencies like remembering to close a word it started 20 characters ago.
np.clip(grad, -5, 5)). Vanishing gradients are harder because you can't force a near-zero signal to become large without changing the architecture.
LSTM: Long Short-Term Memory
The LSTM (Hochreiter & Schmidhuber, 1997) solves vanishing gradients with a clever trick: it adds a cell state Ct that flows through time with minimal interference. Instead of one tanh squashing everything, the LSTM uses three gates to control what information flows in, what gets forgotten, and what gets output.
The three gates
ft = sigmoid(Wf · [ht-1, xt] + bf)
Input gate (what new info to store):
it = sigmoid(Wi · [ht-1, xt] + bi)
C̃t = tanh(WC · [ht-1, xt] + bC)
Cell state update:
Ct = ft * Ct-1 + it * C̃t
Output gate (what to expose as hidden state):
ot = sigmoid(Wo · [ht-1, xt] + bo)
ht = ot * tanh(Ct)
LSTM cell diagram
The cell state (green line at top) is the key innovation. It flows through time with only element-wise operations (multiply and add) — no matrix multiplications, no tanh squashing. This means gradients can flow backward through the cell state unimpeded for hundreds of time steps. The gates learn when to write, erase, and read from this long-term memory.
LSTM in code: NumPy and PyTorch
Here's a single LSTM step in NumPy, showing exactly how the gates work. Then the PyTorch version, which wraps everything into nn.LSTM.
NumPy LSTM step (conceptual)
Notice: an LSTM has 4x the parameters of a vanilla RNN (four weight matrices instead of one). Each gate is essentially its own neural network layer.
PyTorch nn.LSTM
PyTorch's nn.LSTM handles all four gates internally. You pass in a sequence and get back the output at every time step plus the final hidden and cell states. The API is simple; the gate math inside is exactly what we wrote by hand.
GRU: the simplified LSTM
The Gated Recurrent Unit (Cho et al., 2014) simplifies the LSTM by merging the forget and input gates into a single update gate and combining the cell state and hidden state. Fewer parameters, often comparable performance.
| Feature | LSTM | GRU |
|---|---|---|
| Gates | 3 (forget, input, output) | 2 (reset, update) |
| States | ht + Ct (separate) | ht only |
| Parameters | 4 * (n + m) * n | 3 * (n + m) * n |
| Speed | Slower (more computation) | ~25% faster |
| Long sequences | Often slightly better | Competitive |
| PyTorch | nn.LSTM | nn.GRU |
Reset gate: rt = sigmoid(Wr · [ht-1, xt])
Candidate: h̃t = tanh(W · [rt * ht-1, xt])
Output: ht = (1 - zt) * ht-1 + zt * h̃t
The update gate zt acts like a combined forget+input gate: when zt is close to 0, the old hidden state passes through unchanged (remembering); when close to 1, the new candidate replaces it (updating). In practice, GRU and LSTM perform similarly on most tasks — GRU is a good default when you want fewer parameters.
Bidirectional RNNs
A standard RNN reads left to right. But in many tasks, future context matters too. "The animal didn't cross the street because it was too [wide/tired]" — to fill in the blank, you need both what came before and what comes after.
A bidirectional RNN runs two separate RNNs: one left-to-right and one right-to-left. Their hidden states are concatenated at each time step, giving the model access to both past and future context.
h1, h2, ... hT
h'T, h'T-1, ... h'1
Full context at each step
Try It Yourself
Character-Level RNN — train and generate text
This runs the NumPy RNN in your browser with Pyodide. Train on the sample text and watch it learn to generate similar character patterns. Try changing the text or hidden_size.
Why this matters for building an LLM
RNNs and LSTMs were the dominant language model architecture from 2013-2017. Google Translate, OpenAI's early text generators, and most chatbots used stacked LSTMs. But RNNs process tokens one at a time — they can't be parallelized on GPUs. This sequential bottleneck is what motivated the creation of the transformer (next arc). Understanding why RNNs fail at scale is essential to appreciating what transformers fix.
Next lesson: Seq2Seq & The Bottleneck — you'll see how two RNNs (encoder + decoder) can translate languages, and why compressing an entire sentence into a single vector creates the bottleneck that attention was invented to solve.
Challenge
Replace the short training text with a longer string — try a paragraph from a book or song lyrics. Increase hidden_size to 64 and train for 1000 epochs. Does the generated text capture more structure? What happens if you lower the learning rate to 0.01?
Take the LSTM step function from the conceptual code section and build a full training loop. Train it on the same "hello world" text and compare the loss curve to the vanilla RNN. Does the LSTM converge faster or achieve lower loss?
Quiz
You've trained a neural network to generate text.
Now build the architecture that powers ChatGPT.
Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Agentic Engineer →