Why machines need numbers
In the previous four lessons you built neurons, backpropagation, neural networks, and learned PyTorch. Every one of those systems operated on numbers — arrays of floats flowing through matrix multiplications. But language models process text. How does "The cat sat on the mat" become something a neural network can compute with?
A neural network needs a fixed-size numerical input for every piece of text. The process of converting raw text into a sequence of integers is called tokenization. The mapping from text chunks to integers is the vocabulary. Every LLM — GPT, LLaMA, Claude — starts here.
There are three main approaches to tokenization, each with different trade-offs. We'll explore all three, then build the one that modern LLMs actually use: Byte Pair Encoding (BPE).
Character-level tokenization
The simplest approach: treat every character as a token. The vocabulary is just the set of unique characters in your text — typically 50-100 entries (letters, digits, punctuation).
Word-level tokenization
The opposite extreme: treat every whole word as a token. Split on whitespace and punctuation, assign each unique word an integer.
Three approaches compared
The same sentence tokenized at character, word, and subword levels:
Subword tokenization splits common words into themselves ("the", "and") and rare words into meaningful pieces ("un" + "believ" + "able"). This gives you a compact vocabulary (~30K-50K tokens) with short sequences and zero out-of-vocabulary problems. Every modern LLM uses some form of subword tokenization.
Byte Pair Encoding (BPE)
BPE is the algorithm behind GPT's tokenizer. The idea is elegantly simple: start with individual characters, then repeatedly merge the most frequent pair of adjacent tokens into a new token. After enough merges, common words become single tokens while rare words stay split into meaningful subpieces.
How BPE works step by step
- Initialize: Split every word into characters, with a special end-of-word marker
</w> - Count pairs: Find the most frequent pair of adjacent symbols across all words
- Merge: Replace every occurrence of that pair with a new merged symbol
- Repeat: Go back to step 2 until you've done enough merges
Building a BPE tokenizer from scratch
Here is a complete BPE implementation in pure Python. The corpus is "low lower newest widest low lower lower" — small enough to trace every merge by hand, large enough to see real patterns emerge.
What happened in those 8 merges?
- Merges 1-2: "l" and "o" appear together 5 times (in "low" x2 + "lower" x3), so they merge first. Then "lo"+"w" merges into "low".
- Merges 3-5: "lower" appears 3 times, so BPE merges "low"+"e" → "lowe", then "lowe"+"r" → "lower", then "lower"+"</w>" into a complete word token.
- Merge 6: "low" + "</w>" appears 2 times (the standalone "low"), so it becomes a complete word token too.
- Merges 7-8: "est" appears in both "newest" and "widest" (freq 2), so "e"+"s" and then "es"+"t" merge — BPE discovers the shared suffix.
Special tokens
Real tokenizers add special tokens that carry structural meaning beyond the text itself. These tokens tell the model where sentences start, where they end, and how to handle edge cases.
| Token | Purpose | Used By |
|---|---|---|
| [PAD] | Pads short sequences to equal length in a batch | BERT, GPT |
| [UNK] | Represents unknown or out-of-vocabulary tokens | BERT, older models |
| [CLS] | Classification token — placed at start for sentence-level tasks | BERT |
| [SEP] | Separator between two sentences in a pair | BERT |
| [BOS] | Beginning of sequence — signals where text starts | GPT, LLaMA |
| [EOS] | End of sequence — tells the model to stop generating | GPT, LLaMA |
For example, BERT tokenizes a sentence pair like this:
While GPT-style models use:
Why this matters for building an LLM
Tokenization is the very first step in an LLM's pipeline. Before attention, before embeddings, before any neural network computation — the raw text must become a sequence of integers. The quality of your tokenizer directly affects model performance:
Next lesson: you'll learn how those token IDs get converted into dense vectors — word embeddings — the numerical representations that capture meaning.
Try It Yourself
Experiment with BPE. Change the corpus, adjust the number of merges, and watch how the algorithm discovers subword patterns. Try adding words with shared prefixes or suffixes.
Challenge
Extend the BPE tokenizer with an encode(text) function that takes a string and returns a list of token IDs, and a decode(ids) function that converts IDs back to text. You'll need to apply the learned merges in order and build a token-to-ID mapping.
Run BPE on a corpus that mixes English and another language (e.g., "jambo habari hello how are you jambo hello jambo"). Does BPE discover language-specific subwords? How does vocabulary size compare to an English-only corpus of the same length?
Quiz
[EOS] special token in GPT-style models?You've built a tokenizer from scratch.
Now build the full LLM.
Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Agentic Engineer →