Why words need numbers
Neural networks operate on numbers, not strings. To feed language into a network, you must convert each word into a numerical representation. The naive approach — one-hot encoding — assigns each word a unique binary vector. But this creates enormous, sparse vectors that know nothing about meaning.
Word embeddings solve this. They map each word to a short, dense vector (typically 50-300 dimensions) where similar words land near each other. "King" and "queen" get similar vectors. "King" and "the" do not. This is the foundation of every modern language model, including GPT.
Imagine a map where cities are placed by climate and language, not geography. Paris and Rome sit close together (similar culture). Paris and Tokyo are far apart. Word embeddings do the same for language: words with similar usage patterns end up at similar coordinates in a high-dimensional space.
One-hot encoding: the sparsity problem
One-hot encoding represents each word as a vector with a single 1 and all other positions set to 0. If your vocabulary has V words, each vector has V dimensions. For GPT-2's vocabulary of 50,257 tokens, every word is a vector with 50,256 zeros and one 1.
Two fatal flaws. First, the dot product between any two one-hot vectors is zero — the representation contains no information about word similarity. "King" is just as far from "queen" as it is from "the." Second, the vectors are enormous. A vocabulary of 50,000 words means every vector has 50,000 dimensions, mostly zeros.
One-hot vs dense: a visual comparison
The embedding matrix: a learned lookup table
An embedding matrix is a V x D table where V is the vocabulary size and D is the embedding dimension. Row i holds the embedding vector for word i. Looking up a word's embedding is just indexing a row — equivalent to multiplying the one-hot vector by the matrix, but far more efficient.
The key insight: these numbers are not hand-crafted. They are learned during training. The network adjusts the embedding values through backpropagation, the same way it adjusts any other weight. Words that appear in similar contexts gradually develop similar vectors.
Word2Vec: learning embeddings from context
Word2Vec (Mikolov et al., 2013) was the breakthrough that made word embeddings practical. The core idea: "You shall know a word by the company it keeps." Words appearing in similar contexts should have similar embeddings.
Skip-gram vs CBOW
Word2Vec has two architectures:
- Skip-gram: Given a center word, predict its surrounding context words. Trains the center word's embedding to be useful for predicting its neighbors. Works well with small datasets and rare words.
- CBOW (Continuous Bag of Words): Given the surrounding context words, predict the center word. Averages the context embeddings and uses them to predict what goes in the middle. Faster to train, better with frequent words.
From "king", predict: "the", "loves", "the" (window=2)
In our implementation below, we create skip-gram training pairs from a tiny corpus. Each pair is (center_word, context_word). The network learns to predict which words appear near which other words.
Word2Vec skip-gram from scratch
Here is a complete Word2Vec skip-gram implementation. Two weight matrices: W1 maps each word to an embedding (the matrix we keep), W2 maps embeddings to predictions over the vocabulary. We train with softmax cross-entropy, updating both matrices via backpropagation.
What the numbers tell us
Even with a tiny 14-word corpus and 300 epochs of training, the embeddings capture meaningful structure. "King" and "prince" share the highest similarity (0.5181) — both are male royalty that appear in similar contexts ("loves the"). "King" and "queen" are also similar (0.4820) — both are royalty. "King" and "the" have near-zero similarity (0.0852) — a function word shares little semantic content with a noun.
Cosine similarity: measuring word closeness
Cosine similarity measures the angle between two vectors, ignoring their magnitude. A value of 1 means identical direction (same meaning), 0 means orthogonal (unrelated), and -1 means opposite. The formula is:
cos(a, b) = (a · b) / (||a|| * ||b||)
Why cosine instead of Euclidean distance? Because embeddings can vary in magnitude depending on word frequency. A common word trained on many examples might develop a larger vector. Cosine similarity normalizes this out, comparing only the direction — which encodes meaning.
Embedding space: words as coordinates
In a real embedding space with 300 dimensions, these clusters become far richer. Famous examples from trained Word2Vec models: vector("king") - vector("man") + vector("woman") is approximately vector("queen"). The embedding space encodes analogies as linear relationships.
Why this matters for LLMs
Every transformer starts with an embedding layer. When GPT reads "The cat sat on the", it first converts each token to a dense vector using an embedding matrix — exactly the concept you built here. The difference is scale: GPT-3's embedding matrix has 618 million parameters, and the embeddings are refined by 96 transformer layers on top. But the foundation — turning discrete tokens into continuous vectors that capture meaning through context — is precisely Word2Vec's insight.
Run the Word2Vec skip-gram below. Experiment: change the corpus, increase embed_dim, adjust the window size, or add more training epochs. How do the cosine similarities change?
Extend the corpus with sentences like "the prince loves the queen" and "the king loves the princess". Retrain and check: do the cosine similarities between "king" and "queen" increase? Does the embedding space become more structured?
Hint: Concatenate the new sentences to the corpus string before splitting. More co-occurrence data means the model sees more context for each word. With richer context, "king" and "queen" should grow closer because they appear in increasingly similar positions.
Flip the skip-gram: instead of (center → context), average all context word embeddings and predict the center word. How does this change training speed and final similarities?
Hint: For each position i, gather the embeddings of the surrounding words (W1[context_indices]) and average them: h = np.mean(W1[context_indices], axis=0). Use this averaged vector as input to W2. The gradient update to W1 distributes equally across all context words: W1[idx] -= lr * grad_h / len(context_indices) for each context index.
Test your knowledge
You've turned words into vectors.
Now build the transformer that reads them.
Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Agentic Engineer →