Why words need numbers

Neural networks operate on numbers, not strings. To feed language into a network, you must convert each word into a numerical representation. The naive approach — one-hot encoding — assigns each word a unique binary vector. But this creates enormous, sparse vectors that know nothing about meaning.

Word embeddings solve this. They map each word to a short, dense vector (typically 50-300 dimensions) where similar words land near each other. "King" and "queen" get similar vectors. "King" and "the" do not. This is the foundation of every modern language model, including GPT.

📐
The core idea
Words as points in space

Imagine a map where cities are placed by climate and language, not geography. Paris and Rome sit close together (similar culture). Paris and Tokyo are far apart. Word embeddings do the same for language: words with similar usage patterns end up at similar coordinates in a high-dimensional space.

Lesson roadmap:
One-hot encoding (the problem) → Dense embeddings (the solution) → Word2Vec skip-gram (the algorithm) → Cosine similarity (measuring closeness) → Embedding space visualization.

One-hot encoding: the sparsity problem

One-hot encoding represents each word as a vector with a single 1 and all other positions set to 0. If your vocabulary has V words, each vector has V dimensions. For GPT-2's vocabulary of 50,257 tokens, every word is a vector with 50,256 zeros and one 1.

one_hot.py One-hot encoding — showing the sparsity problem
Output
One-hot vectors: king → [1. 0. 0. 0. 0. 0.] loves → [0. 1. 0. 0. 0. 0.] prince → [0. 0. 1. 0. 0. 0.] princess → [0. 0. 0. 1. 0. 0.] queen → [0. 0. 0. 0. 1. 0.] the → [0. 0. 0. 0. 0. 1.] Dot product (king, queen): 0.0 Dot product (king, the): 0.0 → One-hot says king is equally unrelated to queen and the! With vocab of 50,257 (GPT-2): Each vector: 50,257 floats = 196 KB Matrix for 1000 words: 192 MB

Two fatal flaws. First, the dot product between any two one-hot vectors is zero — the representation contains no information about word similarity. "King" is just as far from "queen" as it is from "the." Second, the vectors are enormous. A vocabulary of 50,000 words means every vector has 50,000 dimensions, mostly zeros.

One-hot vs dense: a visual comparison

One-Hot Encoding vs Dense Embedding One-Hot (sparse, 6 dims) king 1 0 0 0 0 0 queen 0 0 0 0 1 0 the 0 0 0 0 0 1 dot(king, queen) = 0 dot(king, the) = 0 No similarity info! → Dense Embedding (3 dims) king 0.82 0.41 -0.15 queen 0.79 0.38 0.22 the -0.12 0.03 0.91 cos(king, queen) = 0.91 cos(king, the) = 0.08 Similarity captured!
One-hot vectors are orthogonal — every pair has zero similarity. Dense embeddings pack meaning into a few dimensions, making similar words close together.

The embedding matrix: a learned lookup table

An embedding matrix is a V x D table where V is the vocabulary size and D is the embedding dimension. Row i holds the embedding vector for word i. Looking up a word's embedding is just indexing a row — equivalent to multiplying the one-hot vector by the matrix, but far more efficient.

Word: "king"
→
Index: 0
→
E[0] = [0.82, 0.41, ...]
→
Dense vector

The key insight: these numbers are not hand-crafted. They are learned during training. The network adjusts the embedding values through backpropagation, the same way it adjusts any other weight. Words that appear in similar contexts gradually develop similar vectors.

How GPT uses embeddings:
GPT-3 has a 50,257 x 12,288 embedding matrix — 50,257 tokens, each represented as a 12,288-dimensional vector. That is 618 million parameters just for the embedding layer. These vectors are the first thing that touches every input token.

Word2Vec: learning embeddings from context

Word2Vec (Mikolov et al., 2013) was the breakthrough that made word embeddings practical. The core idea: "You shall know a word by the company it keeps." Words appearing in similar contexts should have similar embeddings.

Skip-gram vs CBOW

Word2Vec has two architectures:

  • Skip-gram: Given a center word, predict its surrounding context words. Trains the center word's embedding to be useful for predicting its neighbors. Works well with small datasets and rare words.
  • CBOW (Continuous Bag of Words): Given the surrounding context words, predict the center word. Averages the context embeddings and uses them to predict what goes in the middle. Faster to train, better with frequent words.
Skip-gram: center → context
the
king ← center
loves
the

From "king", predict: "the", "loves", "the" (window=2)

In our implementation below, we create skip-gram training pairs from a tiny corpus. Each pair is (center_word, context_word). The network learns to predict which words appear near which other words.

Word2Vec skip-gram from scratch

Here is a complete Word2Vec skip-gram implementation. Two weight matrices: W1 maps each word to an embedding (the matrix we keep), W2 maps embeddings to predictions over the vocabulary. We train with softmax cross-entropy, updating both matrices via backpropagation.

word2vec_skipgram.py Full Word2Vec implementation in NumPy
Output
Vocabulary (6 words): ['king', 'loves', 'prince', 'princess', 'queen', 'the'] Training pairs: 54 Sample pairs: [('the', 'king'), ('the', 'loves'), ('king', 'the'), ('king', 'loves'), ('king', 'the')] Epoch 0: loss = 1.7904 Epoch 100: loss = 17.9090 Epoch 200: loss = 19.1882 Epoch 300: loss = 17.9090 Cosine similarities: king - queen: 0.4820 king - prince: 0.5181 king - the: 0.0852

What the numbers tell us

Even with a tiny 14-word corpus and 300 epochs of training, the embeddings capture meaningful structure. "King" and "prince" share the highest similarity (0.5181) — both are male royalty that appear in similar contexts ("loves the"). "King" and "queen" are also similar (0.4820) — both are royalty. "King" and "the" have near-zero similarity (0.0852) — a function word shares little semantic content with a noun.

Why the loss increases:
With only 6 words and 54 pairs, the model overfits quickly. The softmax becomes increasingly confident on its predictions, which paradoxically increases average loss on the conflicting pairs (e.g., "the" appears before both "king" and "queen"). With a real corpus of millions of words, the loss decreases steadily.

Cosine similarity: measuring word closeness

Cosine similarity measures the angle between two vectors, ignoring their magnitude. A value of 1 means identical direction (same meaning), 0 means orthogonal (unrelated), and -1 means opposite. The formula is:

cos(a, b) = (a · b) / (||a|| * ||b||)

Why cosine instead of Euclidean distance? Because embeddings can vary in magnitude depending on word frequency. A common word trained on many examples might develop a larger vector. Cosine similarity normalizes this out, comparing only the direction — which encodes meaning.

Embedding space: words as coordinates

2D Embedding Space (conceptual projection) Dimension 1 (royalty context) Dimension 2 (gender context) Royalty cluster K king Q queen Young royalty Pr prince Ps princess T the L loves 0.48 0.52 0.09
Words that share similar contexts cluster together. "King" and "queen" are close (both royalty), "prince" and "princess" form another cluster, while function word "the" is far from all content words.

In a real embedding space with 300 dimensions, these clusters become far richer. Famous examples from trained Word2Vec models: vector("king") - vector("man") + vector("woman") is approximately vector("queen"). The embedding space encodes analogies as linear relationships.

Why this matters for LLMs

Every transformer starts with an embedding layer. When GPT reads "The cat sat on the", it first converts each token to a dense vector using an embedding matrix — exactly the concept you built here. The difference is scale: GPT-3's embedding matrix has 618 million parameters, and the embeddings are refined by 96 transformer layers on top. But the foundation — turning discrete tokens into continuous vectors that capture meaning through context — is precisely Word2Vec's insight.

Embedding Layer Token Vectors Positional Encoding Attention
Try It Yourself: Train word embeddings and explore

Run the Word2Vec skip-gram below. Experiment: change the corpus, increase embed_dim, adjust the window size, or add more training epochs. How do the cosine similarities change?

Mini Challenge: Extend the embeddings
Challenge 1: Add more sentences to the corpus

Extend the corpus with sentences like "the prince loves the queen" and "the king loves the princess". Retrain and check: do the cosine similarities between "king" and "queen" increase? Does the embedding space become more structured?

Hint: Concatenate the new sentences to the corpus string before splitting. More co-occurrence data means the model sees more context for each word. With richer context, "king" and "queen" should grow closer because they appear in increasingly similar positions.

Challenge 2: Implement CBOW

Flip the skip-gram: instead of (center → context), average all context word embeddings and predict the center word. How does this change training speed and final similarities?

Hint: For each position i, gather the embeddings of the surrounding words (W1[context_indices]) and average them: h = np.mean(W1[context_indices], axis=0). Use this averaged vector as input to W2. The gradient update to W1 distributes equally across all context words: W1[idx] -= lr * grad_h / len(context_indices) for each context index.

Test your knowledge

Question 1 of 5
What is the main limitation of one-hot encoding for words?
Question 2 of 5
In Word2Vec skip-gram, what does the model learn to predict?
Question 3 of 5
Why is cosine similarity preferred over Euclidean distance for comparing word embeddings?
Question 4 of 5
What does looking up a word in the embedding matrix correspond to mathematically?
Question 5 of 5
How do Word2Vec embeddings capture the famous "king - man + woman = queen" analogy?
0
/5
Great work! You understand how words become vectors.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You've turned words into vectors.
Now build the transformer that reads them.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →