The problem: variable-length input, variable-length output
In Lesson 7, you saw how RNNs and LSTMs process sequences one token at a time, maintaining a hidden state. But RNNs have a limitation: the input and output must be the same length. How do you translate "I love cats" (3 words) into "J'aime les chats" (4 words)?
This is the sequence-to-sequence (seq2seq) problem. The input sequence has one length, the output sequence has another. Machine translation, text summarization, question answering — they all face this mismatch.
🧪
The breakthrough (2014)
Sutskever, Vinyals & Le at Google
In 2014, Ilya Sutskever (later co-founder of OpenAI) published "Sequence to Sequence Learning with Neural Networks." The idea: use one RNN to encode the input into a fixed-size vector, then use a second RNN to decode that vector into the output. Two RNNs chained together, with a single vector in between.
It worked remarkably well for short sequences. But for long ones, it had a fatal flaw — one we'll see firsthand in this lesson.
The encoder-decoder architecture
A seq2seq model has two parts:
Encoder: Reads the input sequence token by token. After processing the last token, its final hidden state becomes the context vector — a fixed-size summary of the entire input.
Decoder: Takes the context vector as its initial hidden state and generates the output sequence one token at a time, feeding each predicted token back as input for the next step.
The key insight: every single bit of information the decoder needs about the input must pass through that one context vector. For "123" reversed to "321," the vector must somehow encode all three digits and their order — compressed into a fixed number of floats.
Why reverse?
We use number reversal as our toy task because it requires the model to understand the entire input before producing any output. You can't start writing the reversed string until you've seen the last character. This makes the bottleneck problem very visible.
The information bottleneck
Here's the fatal flaw. The context vector has a fixed size — say, 64 floats. Whether the input is 3 tokens or 300 tokens, the encoder must squeeze everything into those same 64 numbers.
This is why seq2seq models from 2014-2015 worked well on short sentences but degraded badly on longer ones. Google's original neural machine translation paper showed a sharp accuracy drop once sentence length exceeded ~20 words. The context vector simply couldn't hold enough information.
The core problem:
Imagine summarizing a 500-page book into a single tweet. That's what the context vector asks the encoder to do. The longer the input, the more information gets crushed, distorted, or lost entirely. Early tokens are especially vulnerable — by the time the encoder processes token 50, the information from token 1 has been overwritten many times.
Code: seq2seq for number reversal
Let's build a simplified seq2seq model in NumPy to see the bottleneck in action. Our task: reverse a string of digits. Input "3746", output "6473".
seq2seq_reversal.pyNumPy — see bottleneck in action
Output
Vocabulary: ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'S', 'T', 'E']
Vocab size: 13
Training pairs: 500
Sample pairs: [('3746', '6473'), ('267', '762'), ('37', '73'), ('5417', '7145'), ('514', '415')]
Training seq2seq (simplified):
Epoch 0: loss = 2.5605, accuracy = 1.0%
Epoch 1: loss = 2.5605, accuracy = 1.0%
Epoch 2: loss = 2.5605, accuracy = 1.0%
Epoch 3: loss = 2.5605, accuracy = 1.0%
Epoch 4: loss = 2.5605, accuracy = 1.0%
The information bottleneck:
Encoder compresses input into a single vector of size 64
For "12345" -> "54321", all 5 chars squeezed into 64 numbers
This is why attention was invented!
The accuracy stays at 1% — the model barely learns anything. This is intentional. Without proper gradient flow through the bottleneck (and without attention), the decoder can't figure out which input characters map to which output positions. The fixed-size context vector makes it nearly impossible for the model to learn the reversal mapping.
This is the whole point.
Real seq2seq models trained with proper backpropagation through time (BPTT) and teacher forcing do better — but they still degrade on longer sequences. The bottleneck is fundamental, not a training bug. Attention (next lesson!) solves this by letting the decoder peek at every encoder hidden state, not just the final one.
Teacher forcing: training with the answer key
During training, the decoder has a choice at each step: feed in its own previous prediction (which might be wrong), or feed in the correct previous token from the ground truth. The second approach is called teacher forcing.
Without teacher forcing
The decoder predicts "5" when the answer is "3." Now it feeds "5" into the next step. It's working from a wrong foundation — errors compound. By token 3 or 4, the output is garbage. This makes training slow and unstable.
With teacher forcing
Even if the decoder predicts "5" instead of "3," we feed the correct "3" into the next step. The decoder always works from the right context during training. Learning is faster and more stable.
Teacher Forcing (training): Feed correct tokens
S (start)
->
"3" (correct)
->
"2" (correct)
->
"1" (correct)
Without Teacher Forcing (inference): Feed own predictions
S (start)
->
"5" (wrong!)
->
"7" (error compounds)
->
"0" (diverged)
The exposure bias problem:
Teacher forcing makes training fast, but creates a mismatch. During training the decoder always sees correct inputs; during inference it sees its own (possibly wrong) predictions. The model never learns to recover from mistakes. This is called exposure bias — the model isn't "exposed" to its own errors during training. Scheduled sampling (mixing teacher forcing with free running) is one workaround.
Beam search: smarter decoding
At each decoding step, the model outputs a probability distribution over all possible tokens. Greedy decoding always picks the highest-probability token. But the locally best choice isn't always globally best.
Greedy vs beam search
Greedy: At each step, pick the single most probable token. Fast but can miss better sequences. If the best first token leads to a dead end, there's no going back.
Beam search (beam width = k): Keep the top-k most probable partial sequences at each step. Expand all k, score them, keep the best k again. At the end, return the highest-scoring complete sequence.
Beam search with k=5 to k=10 was standard for seq2seq translation models. It doesn't fix the bottleneck problem — if the context vector lost information, beam search can't recover it — but it helps the decoder make better use of whatever information it does have.
PyTorch seq2seq: encoder-decoder with GRU
Here's what a proper seq2seq model looks like in PyTorch, using nn.GRU (a simpler variant of LSTM) for both encoder and decoder. This is read-only — it requires PyTorch to run.
pytorch_seq2seq.pyRead-only — PyTorch required
Notice line 10: return hidden. That's the bottleneck. The entire input sequence — no matter how long — is compressed into hidden, a tensor of shape (1, batch, hidden_dim). The decoder never sees the individual encoder states outputs — they're thrown away.
The fix (next lesson!):
Attention lets the decoder access all encoder outputs, not just the final hidden state. Instead of throwing away outputs, attention computes a weighted combination of them at each decoding step. The decoder can "look back" at whichever encoder positions are most relevant.
Try It Yourself
InteractiveExplore the seq2seq bottleneck
Run this code to see how the encoder compresses sequences into a fixed-size context vector. Watch how the model struggles with the reversal task — the bottleneck makes it nearly impossible without attention.
try_seq2seq.pyEditable — runs in browser via Pyodide
Click "Run Code" to see the output...
The solution: attention (next lesson!)
What if the decoder didn't have to rely on just one context vector? What if, at each decoding step, it could "look back" at every encoder hidden state and decide which ones are most relevant?
That's attention — and it's the single most important idea in modern deep learning. It eliminated the bottleneck, enabled transformers, and made GPT possible. The next lesson covers it in full detail.
Why this matters for building an LLM
The seq2seq architecture was the direct ancestor of the transformer. The encoder-decoder structure survives in models like T5 and BART. The bottleneck problem is what motivated Bahdanau attention (2014), which led to self-attention (2017), which led to transformers, which led to GPT. Every modern LLM exists because the context vector wasn't good enough.
seq2seq -> attentionattention -> transformertransformer -> GPTencoder-decoder still in T5
Challenge
Put it into practice
Challenge 1: Measure the bottleneck
Modify the code to generate sequences of different lengths (2, 5, 10, 20 digits) and run the simplified seq2seq on each. Plot or print the accuracy for each length. You should see accuracy drop as sequences get longer — that's the bottleneck in action.
Change np.random.randint(2, 5) in generate_pair() to use fixed lengths. Run 100 pairs for each length and compare accuracy. Even random guessing gets worse as the output gets longer.
Challenge 2: Implement greedy vs beam search
Write a beam_search_decode(context, length, k=3) function. At each step, keep the top-k partial sequences ranked by cumulative log-probability. Compare the output with greedy decoding. (They'll both be poor without training, but the structure matters.)
Use a list of tuples (sequence, hidden_state, cumulative_log_prob). At each step, expand each candidate by all possible next tokens, compute their probabilities, and keep the top-k. Use np.log(probs[token_idx]) for log probabilities.
Quiz
Question 1 of 5
What is the "context vector" in a seq2seq model?
Question 2 of 5
Why does seq2seq performance degrade on longer input sequences?
Question 3 of 5
What is "teacher forcing" during seq2seq training?
Question 4 of 5
How does beam search (k=3) differ from greedy decoding?
Question 5 of 5
What does the attention mechanism (next lesson) change about the seq2seq bottleneck?
0
/5
You understand the bottleneck that changed deep learning forever.
Arc C begins
Next: Attention Is All You Need ->
The most important idea in modern AI. The mechanism that eliminated the bottleneck, enabled transformers, and made GPT possible.