Master dense vector representations from one-hot to Word2Vec, GloVe, and FastText. Build embeddings for low-resource languages like Twi.
Word2Vec • GloVe • FastText
Semantic Similarity
Ghanaian Languages
22 min readIntermediate
🌍
Meet Kwame
NLP Researcher Building Tools for Ghanaian Languages
Kwame is building NLP tools for Twi (Akan language, spoken by ~9 million people in Ghana). He has a corpus of 2.3M Twi words but no pretrained embeddings. Using one-hot encoding produces sparse, high-dimensional vectors (2.3M dimensions) that don't capture semantic relationships. He needs dense embeddings that capture meaning: "ɔbaabuo" (woman) and "ɔbarima" (man) should be close; "ɔbaabuo" - "ɔbarima" + "ɔwa" (girl) should approach "ɔwa" (girl). Word embeddings will make this happen.
2.3M
Twi word tokens
50-300
Embedding dimensions
99.99%
Sparsity reduced
🎯 Why Embeddings? The One-Hot Problem
One-hot encoding is sparse and context-agnostic. A vocabulary of 2.3M words → 2.3M-dimensional vectors with a single 1 and 2.3M-1 zeros. This wastes memory and loses semantic meaning:
One-Hot Limitations
Sparse: Most dimensions are 0; wastes memory and computation.
No semantics: Distance between any two words is constant (Euclidean distance = √2). "ɔbaabuo" (woman) is as far from "ɔbarima" (man) as from a random word.
No shared learning: Each word is orthogonal; the model can't transfer patterns from one word to similar words.
Doesn't scale: A 2.3M-dimensional embedding layer has billions of parameters. RNNs and Transformers would crawl.
Dense Embeddings (Word2Vec, GloVe, FastText)
Dense: 50–300 dimensions capture all meaning. ~99.99% reduction in dimensionality.
Semantic: Similar words (woman/man, king/queen) cluster together. Arithmetic relationships emerge: king − man + woman ≈ queen.
Shared learning: Related words share embedding space structure. "ɔbaabuo" and "ɔbaa" (woman/female) learn together.
Efficient: 300D × 2.3M = 690M parameters (manageable). RNNs process each time step independently.
🔤 Word2Vec: CBOW & Skip-Gram
Word2Vec learns embeddings by predicting context from words (CBOW) or words from context (Skip-gram). The intuition: if two words appear in similar contexts, their embeddings should be similar.
CBOW: Context → Word
Given surrounding words, predict the center word. Example: "Kwame na _________ aba." (Kwame and ___ came.) Context: [Kwame, na, aba] → Predict: "Ama". The embedding of "Ama" learns to be close to the averaged embeddings of context words.
Skip-gram: Word → Context
Given a word, predict surrounding words. Example: "ɔbaabuo" → Predict: [ɔbarima, nti, ɛ]. Skip-gram is more flexible and handles rare words better. Kwame prefers Skip-gram for Twi.
GloVe combines global co-occurrence statistics with the local context windows of Word2Vec. It factors a word co-occurrence matrix, learning embeddings that minimize the difference between dot products and log co-occurrence counts.
GloVe vs Word2Vec
Word2Vec: Predicts context locally (window-based). Fast, good on small corpora.
GloVe: Uses global statistics (entire co-occurrence matrix). Better on large corpora (100M+ tokens); slower training.
Hybrid approach: GloVe captures global word relationships; Word2Vec captures syntactic subtleties. For Kwame's 2.3M Twi tokens, Word2Vec is sufficient.
GloVe embedding vectors (pretrained, for reference)
GloVe results
Loaded 400000 GloVe vectors (100 dimensions)
GloVe similarity(woman, man): 0.8639
king - man + woman ≈ queen (similarity: 0.8292)
⚡ FastText: Subword Embeddings
FastText extends Word2Vec by learning embeddings for character n-grams, not just whole words. This is powerful for morphologically rich languages (like Twi) and handles out-of-vocabulary (OOV) words gracefully.
Why FastText for Twi?
Morphology: Twi words inflect heavily. "ɔbaabuo" (woman), "ɔbaabuo+no" (the woman), "ɔbaabuo+a" (the women) share stems. FastText captures the "ɔbaabuo" structure.
OOV handling: If a test word isn't in training, FastText can still embed it (sum of subword n-grams). Word2Vec returns a zero vector.
Rare words: Rare inflections get reasonable embeddings from shared subwords.
FastText for Twi (morphologically rich language)
FastText subword embeddings
ɔbaabuo embedding: [ 0.044 -0.022 0.108 -0.035 0.012]
OOV embedding for 'ɔbaabuo+i':
[0.039 -0.018 0.102 -0.031 0.009]
Shape: (100,)
Not from exact word, but reconstructed from 3/4/5-grams
Similarity(ɔbaabuo, ɔbaabuo+no): 0.7834
Words similar to 'ɔbaabuo':
ɔbaabuo+no: 0.7834
Akosua: 0.5912
Ama: 0.5604
ɔbaa: 0.5381
nnipa: 0.4834
🔥 PyTorch nn.Embedding in RNNs
PyTorch's nn.Embedding layer turns word indices into dense vectors. It's a lookup table initialized randomly or with pretrained embeddings. During training, embeddings update via backprop.
nn.Embedding for Twi text classification (sentiment analysis)
Instead of training embeddings from scratch, Kwame can load pretrained vectors (e.g., Word2Vec, GloVe, FastText) and use them to initialize nn.Embedding.
Load pretrained Word2Vec into PyTorch nn.Embedding
One of the most striking properties of word embeddings: vector arithmetic captures semantic relationships. King − Man + Woman ≈ Queen. In Twi: "ɔbaabuo" − "ɔbarima" + ... (analogies in morphology and semantics).
2D visualization of word embeddings (t-SNE projection of 100D space)
The t-SNE projection reveals clear semantic structure: gender pairs (woman/man, girl/boy) cluster together; numbers form a line; verbs and nouns separate spatially. Embedding arithmetic works because embeddings encode linear relationships.
Task: Kwame's initial Word2Vec embeddings (trained on 2.3M Twi tokens) achieve 0.61 average similarity on gender analogies (ɔbaabuo-ɔbarima vs ɔbaabuo_nta-ɔbarima_nta). Train a FastText model on the same corpus and compare. Does FastText improve analogy performance? Which morphological patterns does FastText capture better?
Hint: FastText's subword n-grams may help rare inflections (ɔbaabuo+a, ɔbaabuo+no) form better analogies because they share morpheme structure.
💼 Interview Questions
A word embedding is a dense vector (e.g., 100 dimensions) that represents a word's meaning in continuous space. It's better than one-hot encoding because: (1) dense vectors are memory-efficient (~99% reduction for 2.3M vocab), (2) semantically similar words have nearby embeddings (distance captures meaning), (3) embeddings support arithmetic (king − man + woman ≈ queen), and (4) they enable transfer learning—embeddings trained on one task can be reused for others. One-hot encoding is sparse (2.3M zeros), loses all semantic info, and doesn't scale.
CBOW (Continuous Bag of Words) predicts the center word from context: given [Kwame, na, ____, aba], predict the missing word. Skip-gram does the opposite: given a word, predict surrounding words. Skip-gram is more flexible, works better on rare words, and is generally preferred. For Kwame's Twi corpus, Skip-gram captures long-range semantic relationships better because it has 2× as many training examples (each word can be a target for multiple context windows).
FastText learns embeddings for subword n-grams (3-grams, 4-grams, 5-grams), not just whole words. Twi words inflect heavily (ɔbaabuo, ɔbaabuo+no, ɔbaabuo+a share the stem "ɔbaabuo"). FastText captures the shared structure—related inflections have similar embeddings because they share n-grams. Word2Vec treats each form independently, missing that connection. Additionally, FastText can embed OOV (out-of-vocabulary) words by reconstructing them from their n-grams. A rare inflection like "ɔbaabuo+i" can still be embedded even if never seen in training.
Embeddings encode semantic relationships as linear directions in space. The vector (king − man) captures the "gender offset" (from male to female). Adding that offset to "woman" (a female figure) shifts it toward another female figure with similar status—"queen." This works because embeddings naturally learn that king and queen differ mainly in gender, and man and woman differ in the same gender axis. The arithmetic (king − man + woman) transfers the gender relationship. For Twi: "ɔbaabuo" − "ɔbarima" = gender offset; adding it to "ɔbarima_nta" (boy) should approach "ɔbaabuo_nta" (girl). This magic emerges from training—the model doesn't explicitly learn "gender axis," but embeddings naturally organize themselves that way.
(1) Check your window size—if too small (e.g., window=2), context may miss long-range morphological relationships. Increase to window=5 or window=8. (2) Check min_count—if "ɔbaabuo_nta" is rare (< 10 occurrences), it may not have enough context windows to learn a good embedding. Lower min_count to 2 or 1. (3) Increase epochs and vector_size—50 dims may be too small; try 100 or 150. (4) Use FastText instead of Word2Vec—subword n-grams will capture the shared "ɔbaabuo" stem. (5) Inspect your tokenizer—ensure "ɔbaabuo" and "ɔbaabuo_nta" are separate tokens, not merged or split incorrectly. (6) Check corpus size—2.3M tokens is small; similarity may be noisy. If you have more data, train on it. (7) Consider domain balance—if "ɔbaabuo_nta" appears mostly in certain texts (poetry, narrative), they may have different contexts than "ɔbaabuo" (news), reducing similarity.
✓ Quick Quiz
Question 1 / 5
One-hot encoding of a 2.3M-word Twi vocabulary produces vectors of what dimensionality?
Question 2 / 5
In Word2Vec Skip-gram, what is the model trying to predict?
Question 3 / 5
Why is FastText better than Word2Vec for Twi, a morphologically rich language?
Question 4 / 5
In PyTorch's nn.Embedding, what does the weight matrix represent?
Question 5 / 5
The embedding arithmetic ɔbaabuo − ɔbarima + ɔbarima_nta ≈ ɔbaabuo_nta assumes what about the embedding space?
0
📋 Quick Cheatsheet
Concept
Code / Usage
Notes
Train Word2Vec Skip-gram
Word2Vec(sentences, vector_size=100, sg=1)
sg=1 for Skip-gram; sg=0 for CBOW. Fast, good on small corpora.
Train FastText
FastText(sentences, vector_size=100, sg=1)
Subword n-grams; handles OOV & morphology. Better for Twi.
Embedding similarity
model.wv.similarity('ɔbaabuo', 'ɔbarima')
Cosine similarity in [0, 1]. High = semantically close.
Find similar words
model.wv.most_similar('ɔbaabuo', topn=5)
Returns top-5 nearest neighbors in embedding space.
Initialize from Word2Vec, GloVe, or FastText matrix. Can freeze (requires_grad=False).
Freeze embeddings (no updates)
embedding_layer.weight.requires_grad = False
Useful if pretrained embeddings are already high-quality; saves memory.
⚠️ Common Mistakes
Using one-hot encoding for large vocabularies
One-hot produces 2.3M-dimensional vectors for a Twi corpus; RNNs crawl, memory explodes. Always use dense embeddings (Word2Vec, GloVe, FastText, or learned nn.Embedding). Reduces to ~100-300 dims with better semantics.
Choosing Word2Vec over FastText for morphologically rich languages
Twi inflects heavily (ɔbaabuo, ɔbaabuo+no, ɔbaabuo+a). Word2Vec treats each form independently. FastText's n-gram subwords capture shared morpheme structure, yielding better OOV handling and analogy performance. Use FastText for morphologically rich languages.
Too-small window size in Word2Vec
Window=2 captures only immediate neighbors; semantic relationships need wider context. Use window=5 to window=10. For Twi, wider windows help capture discourse patterns.
Setting min_count too high
min_count=10 discards rare inflections (e.g., ɔbaabuo_nta if it appears 5 times). For low-resource languages, lower min_count to 2–3 to retain morphological variants.
Not freezing pretrained embeddings when fine-tuning
Pretrained embeddings (trained on 100M tokens) can overfit if updated on a small downstream task (10K tweets). Freeze embedding_layer.weight (requires_grad=False) to preserve knowledge, or use very low learning rates (1e-4).
✦ Key Takeaways
Dense word embeddings (100–300 dims) replace sparse one-hot encoding (millions of dims) and capture semantic meaning: similar words cluster together, embedding arithmetic encodes relationships.
Word2Vec learns embeddings via CBOW (context → word) or Skip-gram (word → context). Skip-gram is generally better; fast training, good on small corpora.
GloVe uses global co-occurrence statistics for richer representations on large corpora (100M+ tokens). Slower training but often better quality.
FastText extends Word2Vec with subword n-grams, perfect for morphologically rich languages like Twi. Handles OOV words gracefully; "ɔbaabuo+i" (unseen inflection) can still be embedded.
PyTorch's nn.Embedding is a learnable lookup table. Initialize with pretrained vectors or train end-to-end. Can freeze embeddings to preserve pretrained knowledge.
Embedding arithmetic (king − man + woman ≈ queen) works because embeddings encode semantic relationships as linear directions. Kwame can solve Twi morphological analogies this way.
Kwame's Twi embeddings (trained on 2.3M tokens via Word2Vec/FastText) enable downstream NLP tasks: sentiment analysis, machine translation, named entity recognition—all without manually labeled embeddings.