Why machines need numbers

In the previous four lessons you built neurons, backpropagation, neural networks, and learned PyTorch. Every one of those systems operated on numbers — arrays of floats flowing through matrix multiplications. But language models process text. How does "The cat sat on the mat" become something a neural network can compute with?

🔤
The problem
Text in, numbers out

A neural network needs a fixed-size numerical input for every piece of text. The process of converting raw text into a sequence of integers is called tokenization. The mapping from text chunks to integers is the vocabulary. Every LLM — GPT, LLaMA, Claude — starts here.

There are three main approaches to tokenization, each with different trade-offs. We'll explore all three, then build the one that modern LLMs actually use: Byte Pair Encoding (BPE).

Character-level tokenization

The simplest approach: treat every character as a token. The vocabulary is just the set of unique characters in your text — typically 50-100 entries (letters, digits, punctuation).

char_tokenizer.py Read-only
Output
Vocabulary (8 tokens): ' ' → 0 'd' → 1 'e' → 2 'h' → 3 'l' → 4 'o' → 5 'r' → 6 'w' → 7 "hello world" → [3, 2, 4, 4, 5, 0, 7, 5, 6, 4, 1] [3, 2, 4, 4, 5, 0, 7, 5, 6, 4, 1] → "hello world"
Pros:
Tiny vocabulary (no out-of-vocabulary words ever). Cons: Sequences become extremely long. "hello world" is 11 tokens. A 1000-word article would be ~5000 tokens. Transformers have a fixed context window — character-level wastes it on individual letters instead of meaningful chunks.

Word-level tokenization

The opposite extreme: treat every whole word as a token. Split on whitespace and punctuation, assign each unique word an integer.

word_tokenizer.py Read-only
Output
Vocabulary (5 tokens): "cat" → 0 "mat" → 1 "on" → 2 "sat" → 3 "the" → 4 "the cat sat on the mat" → [4, 0, 3, 2, 4, 1] Trying to tokenize: "the dog sat on the rug" "the" → 4 "dog" → ??? (OUT OF VOCABULARY) "sat" → 3 "on" → 2 "the" → 4 "rug" → ??? (OUT OF VOCABULARY)
The OOV problem:
Any word not in the training vocabulary is unknown. English has ~170,000 words in current use, plus names, technical terms, typos, and other languages. A word-level tokenizer either needs an impossibly large vocabulary or constantly hits unknown tokens. Neither option is acceptable for an LLM.

Three approaches compared

The same sentence tokenized at character, word, and subword levels:

Tokenizing: "lower newest" CHAR-LEVEL (12 tokens, vocab ~50) l o w e r _ n e w e s t WORD-LEVEL (2 tokens, vocab ~170K+) lower newest SUBWORD (BPE) (4 tokens, vocab ~30K-50K) low er new est Subword finds the sweet spot: short sequences + small vocabulary + no OOV

Subword tokenization splits common words into themselves ("the", "and") and rare words into meaningful pieces ("un" + "believ" + "able"). This gives you a compact vocabulary (~30K-50K tokens) with short sequences and zero out-of-vocabulary problems. Every modern LLM uses some form of subword tokenization.

Byte Pair Encoding (BPE)

BPE is the algorithm behind GPT's tokenizer. The idea is elegantly simple: start with individual characters, then repeatedly merge the most frequent pair of adjacent tokens into a new token. After enough merges, common words become single tokens while rare words stay split into meaningful subpieces.

How BPE works step by step

  1. Initialize: Split every word into characters, with a special end-of-word marker </w>
  2. Count pairs: Find the most frequent pair of adjacent symbols across all words
  3. Merge: Replace every occurrence of that pair with a new merged symbol
  4. Repeat: Go back to step 2 until you've done enough merges
BPE merging "lower" step by step START l o w e r merge l+o lo w e r MERGE 2 lo+w → low e r MERGE 3 low+e → lowe r MERGE 4 lowe+r → lower "lower" appears 3x in corpus — BPE merges it into a single token

Building a BPE tokenizer from scratch

Here is a complete BPE implementation in pure Python. The corpus is "low lower newest widest low lower lower" — small enough to trace every merge by hand, large enough to see real patterns emerge.

bpe_tokenizer.py Read-only
Output
Initial vocabulary: l o w : 2 l o w e r : 3 n e w e s t : 1 w i d e s t : 1 Running 8 BPE merges: Merge 1: l + o → lo (freq: 5) Merge 2: lo + w → low (freq: 5) Merge 3: low + e → lowe (freq: 3) Merge 4: lowe + r → lower (freq: 3) Merge 5: lower + → lower (freq: 3) Merge 6: low + → low (freq: 2) Merge 7: e + s → es (freq: 2) Merge 8: es + t → est (freq: 2) Final vocabulary: low: 2 lower: 3 n e w est : 1 w i d est : 1

What happened in those 8 merges?

  1. Merges 1-2: "l" and "o" appear together 5 times (in "low" x2 + "lower" x3), so they merge first. Then "lo"+"w" merges into "low".
  2. Merges 3-5: "lower" appears 3 times, so BPE merges "low"+"e" → "lowe", then "lowe"+"r" → "lower", then "lower"+"</w>" into a complete word token.
  3. Merge 6: "low" + "</w>" appears 2 times (the standalone "low"), so it becomes a complete word token too.
  4. Merges 7-8: "est" appears in both "newest" and "widest" (freq 2), so "e"+"s" and then "es"+"t" merge — BPE discovers the shared suffix.
Key insight:
BPE discovered that "low", "lower", and "est" are meaningful subword units — without any linguistic knowledge. It found these patterns purely from frequency statistics. GPT-2's tokenizer runs 50,000 merges on billions of words to build its vocabulary.

Special tokens

Real tokenizers add special tokens that carry structural meaning beyond the text itself. These tokens tell the model where sentences start, where they end, and how to handle edge cases.

Token Purpose Used By
[PAD] Pads short sequences to equal length in a batch BERT, GPT
[UNK] Represents unknown or out-of-vocabulary tokens BERT, older models
[CLS] Classification token — placed at start for sentence-level tasks BERT
[SEP] Separator between two sentences in a pair BERT
[BOS] Beginning of sequence — signals where text starts GPT, LLaMA
[EOS] End of sequence — tells the model to stop generating GPT, LLaMA

For example, BERT tokenizes a sentence pair like this:

[CLS]
→
the cat sat
→
[SEP]
→
on the mat
→
[SEP]

While GPT-style models use:

[BOS]
→
the cat sat on the mat
→
[EOS]
Why BPE almost eliminates [UNK]:
With subword tokenization, even a completely novel word like "ChatGPTification" gets split into known subpieces: "Chat" + "G" + "PT" + "ification". The model can still process it because it has seen those subwords in other contexts. This is why modern LLMs rarely produce [UNK] tokens.

Why this matters for building an LLM

Tokenization is the very first step in an LLM's pipeline. Before attention, before embeddings, before any neural network computation — the raw text must become a sequence of integers. The quality of your tokenizer directly affects model performance:

GPT-2: 50,257 BPE tokens GPT-4: ~100K tokens LLaMA: 32,000 tokens BOS/EOS for generation Vocab size = embedding dim

Next lesson: you'll learn how those token IDs get converted into dense vectors — word embeddings — the numerical representations that capture meaning.

Try It Yourself

Experiment with BPE. Change the corpus, adjust the number of merges, and watch how the algorithm discovers subword patterns. Try adding words with shared prefixes or suffixes.

🐍 Try It Yourself — BPE Tokenizer
Output
Click ▶ Run to execute...

Challenge

Put it into practice
Challenge 1: Add encode and decode functions

Extend the BPE tokenizer with an encode(text) function that takes a string and returns a list of token IDs, and a decode(ids) function that converts IDs back to text. You'll need to apply the learned merges in order and build a token-to-ID mapping.

Store your merge rules as an ordered list. For encoding, start by splitting the input word into characters + </w>, then apply each merge rule in order. For the ID mapping, collect all unique tokens from the final vocabulary and assign sequential integers.
Challenge 2: Multilingual BPE

Run BPE on a corpus that mixes English and another language (e.g., "jambo habari hello how are you jambo hello jambo"). Does BPE discover language-specific subwords? How does vocabulary size compare to an English-only corpus of the same length?

BPE doesn't know what language it's processing — it just finds frequent byte pairs. You'll likely see that shared characters across languages merge first, while language-specific patterns take more merges to emerge. This is why multilingual models like mBERT need larger vocabularies.

Quiz

Question 1 of 5
What is the main disadvantage of character-level tokenization?
Question 2 of 5
In BPE, what determines which pair of tokens gets merged next?
Question 3 of 5
What is the purpose of the [EOS] special token in GPT-style models?
Question 4 of 5
Why does subword tokenization (like BPE) almost eliminate the out-of-vocabulary problem?
Question 5 of 5
GPT-2 uses a BPE vocabulary of approximately 50,000 tokens. What would happen if you doubled the number of BPE merges?
0
/5
Great work! You understand how LLMs convert text into tokens.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You've built a tokenizer from scratch.
Now build the full LLM.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →