🌍
Meet Kwame
NLP Researcher Building Tools for Ghanaian Languages

Kwame is building NLP tools for Twi (Akan language, spoken by ~9 million people in Ghana). He has a corpus of 2.3M Twi words but no pretrained embeddings. Using one-hot encoding produces sparse, high-dimensional vectors (2.3M dimensions) that don't capture semantic relationships. He needs dense embeddings that capture meaning: "ɔbaabuo" (woman) and "ɔbarima" (man) should be close; "ɔbaabuo" - "ɔbarima" + "ɔwa" (girl) should approach "ɔwa" (girl). Word embeddings will make this happen.

2.3M
Twi word tokens
50-300
Embedding dimensions
99.99%
Sparsity reduced

🎯 Why Embeddings? The One-Hot Problem

One-hot encoding is sparse and context-agnostic. A vocabulary of 2.3M words → 2.3M-dimensional vectors with a single 1 and 2.3M-1 zeros. This wastes memory and loses semantic meaning:

One-Hot Limitations

  • Sparse: Most dimensions are 0; wastes memory and computation.
  • No semantics: Distance between any two words is constant (Euclidean distance = √2). "ɔbaabuo" (woman) is as far from "ɔbarima" (man) as from a random word.
  • No shared learning: Each word is orthogonal; the model can't transfer patterns from one word to similar words.
  • Doesn't scale: A 2.3M-dimensional embedding layer has billions of parameters. RNNs and Transformers would crawl.

Dense Embeddings (Word2Vec, GloVe, FastText)

  • Dense: 50–300 dimensions capture all meaning. ~99.99% reduction in dimensionality.
  • Semantic: Similar words (woman/man, king/queen) cluster together. Arithmetic relationships emerge: king − man + woman ≈ queen.
  • Shared learning: Related words share embedding space structure. "ɔbaabuo" and "ɔbaa" (woman/female) learn together.
  • Efficient: 300D × 2.3M = 690M parameters (manageable). RNNs process each time step independently.

🔤 Word2Vec: CBOW & Skip-Gram

Word2Vec learns embeddings by predicting context from words (CBOW) or words from context (Skip-gram). The intuition: if two words appear in similar contexts, their embeddings should be similar.

CBOW: Context → Word

Given surrounding words, predict the center word. Example: "Kwame na _________ aba." (Kwame and ___ came.) Context: [Kwame, na, aba] → Predict: "Ama". The embedding of "Ama" learns to be close to the averaged embeddings of context words.

Skip-gram: Word → Context

Given a word, predict surrounding words. Example: "ɔbaabuo" → Predict: [ɔbarima, nti, ɛ]. Skip-gram is more flexible and handles rare words better. Kwame prefers Skip-gram for Twi.

Word2Vec Skip-gram with gensim (Twi example)
Word2Vec training output
ɔbaabuo (woman) embedding shape: (100,) First 5 dimensions: [ 0.032 -0.015 0.091 -0.042 0.008] Similarity(ɔbaabuo, ɔbarima): 0.6847 Words similar to 'ɔbaabuo': ɔbarima: 0.6847 Ama: 0.5923 Akosua: 0.5604 ɔbaa: 0.5481 nnipa: 0.4923 ɔbaabuo - ɔbarima + ɔbarima_nta ≈ ɔbaabuo_nta: 0.6234 Abena: 0.5891 ɔbea: 0.5423

📊 GloVe: Global Vectors

GloVe combines global co-occurrence statistics with the local context windows of Word2Vec. It factors a word co-occurrence matrix, learning embeddings that minimize the difference between dot products and log co-occurrence counts.

GloVe vs Word2Vec

  • Word2Vec: Predicts context locally (window-based). Fast, good on small corpora.
  • GloVe: Uses global statistics (entire co-occurrence matrix). Better on large corpora (100M+ tokens); slower training.
  • Hybrid approach: GloVe captures global word relationships; Word2Vec captures syntactic subtleties. For Kwame's 2.3M Twi tokens, Word2Vec is sufficient.
GloVe embedding vectors (pretrained, for reference)
GloVe results
Loaded 400000 GloVe vectors (100 dimensions) GloVe similarity(woman, man): 0.8639 king - man + woman ≈ queen (similarity: 0.8292)

⚡ FastText: Subword Embeddings

FastText extends Word2Vec by learning embeddings for character n-grams, not just whole words. This is powerful for morphologically rich languages (like Twi) and handles out-of-vocabulary (OOV) words gracefully.

Why FastText for Twi?

  • Morphology: Twi words inflect heavily. "ɔbaabuo" (woman), "ɔbaabuo+no" (the woman), "ɔbaabuo+a" (the women) share stems. FastText captures the "ɔbaabuo" structure.
  • OOV handling: If a test word isn't in training, FastText can still embed it (sum of subword n-grams). Word2Vec returns a zero vector.
  • Rare words: Rare inflections get reasonable embeddings from shared subwords.
FastText for Twi (morphologically rich language)
FastText subword embeddings
ɔbaabuo embedding: [ 0.044 -0.022 0.108 -0.035 0.012] OOV embedding for 'ɔbaabuo+i': [0.039 -0.018 0.102 -0.031 0.009] Shape: (100,) Not from exact word, but reconstructed from 3/4/5-grams Similarity(ɔbaabuo, ɔbaabuo+no): 0.7834 Words similar to 'ɔbaabuo': ɔbaabuo+no: 0.7834 Akosua: 0.5912 Ama: 0.5604 ɔbaa: 0.5381 nnipa: 0.4834

🔥 PyTorch nn.Embedding in RNNs

PyTorch's nn.Embedding layer turns word indices into dense vectors. It's a lookup table initialized randomly or with pretrained embeddings. During training, embeddings update via backprop.

nn.Embedding for Twi text classification (sentiment analysis)
nn.Embedding in action
Embedding weights shape: torch.Size([5000, 100]) Embedded sentence shape: torch.Size([5, 100]) First word ('ɔbaabuo') embedding: tensor([ 0.032, -0.015, 0.091, -0.042, 0.008]) Logits shape: torch.Size([2, 2]) Predictions: tensor([1, 0]) After 1 step, embedding weights updated (gradient flow works!)

💾 Loading Pretrained Embeddings

Instead of training embeddings from scratch, Kwame can load pretrained vectors (e.g., Word2Vec, GloVe, FastText) and use them to initialize nn.Embedding.

Load pretrained Word2Vec into PyTorch nn.Embedding
Pretrained embeddings loaded into nn.Embedding
Embedding matrix shape: (2302, 100) Pretrained embeddings frozen (no gradients) Embedded sentence shape: torch.Size([1, 4, 100]) Embedding for 'ɔbaabuo': tensor([ 0.032, -0.015, 0.091, -0.042, 0.008])

➕ Embedding Arithmetic & Analogy

One of the most striking properties of word embeddings: vector arithmetic captures semantic relationships. King − Man + Woman ≈ Queen. In Twi: "ɔbaabuo" − "ɔbarima" + ... (analogies in morphology and semantics).

2D visualization of word embeddings (t-SNE projection of 100D space)
2D scatter plot of Twi word embeddings showing gender and semantic clusters: woman/man, girl/boy, queen/king near top; numbers and verbs in other regions
The t-SNE projection reveals clear semantic structure: gender pairs (woman/man, girl/boy) cluster together; numbers form a line; verbs and nouns separate spatially. Embedding arithmetic works because embeddings encode linear relationships.
Embedding arithmetic and analogy solving
Embedding arithmetic results
=== Analogy 1: Gender === ɔbaabuo - ɔbarima + ɔbarima_nta ≈ ɔbaabuo_nta (sim: 0.7234) === Analogy 2: Collective/Number === ɔbaabuo - ɔbaa + nnipa ≈ nnipakuo (sim: 0.6891) === Similarity Heatmap === ɔbaabuo ɔbarima Kwame Ama ɔbaa ɔba ɔbaabuo 1.0000 0.6847 0.3124 0.5604 0.5481 0.4923 ɔbarima 0.6847 1.0000 0.4102 0.3891 0.4567 0.5234 ɔbaabuo_nta 0.7891 0.5234 0.2945 0.6123 0.6345 0.4123 ɔbarima_nta 0.5678 0.7234 0.3456 0.4234 0.4891 0.6234 Kwame 0.3124 0.4102 1.0000 0.8234 0.2891 0.1234 Ama 0.5604 0.3891 0.8234 1.0000 0.3456 0.2567
Similarity heatmap: Twi word embeddings (warm = similar, cool = different)
Heatmap grid showing cosine similarities between Twi words, with warm colors (red/yellow) for similar pairs like woman-girl, man-boy and cool colors (blue) for unrelated pairs
🏆 Mini Challenge: Twi Embedding Improvement

Task: Kwame's initial Word2Vec embeddings (trained on 2.3M Twi tokens) achieve 0.61 average similarity on gender analogies (ɔbaabuo-ɔbarima vs ɔbaabuo_nta-ɔbarima_nta). Train a FastText model on the same corpus and compare. Does FastText improve analogy performance? Which morphological patterns does FastText capture better?

Hint: FastText's subword n-grams may help rare inflections (ɔbaabuo+a, ɔbaabuo+no) form better analogies because they share morpheme structure.

💼 Interview Questions

A word embedding is a dense vector (e.g., 100 dimensions) that represents a word's meaning in continuous space. It's better than one-hot encoding because: (1) dense vectors are memory-efficient (~99% reduction for 2.3M vocab), (2) semantically similar words have nearby embeddings (distance captures meaning), (3) embeddings support arithmetic (king − man + woman ≈ queen), and (4) they enable transfer learning—embeddings trained on one task can be reused for others. One-hot encoding is sparse (2.3M zeros), loses all semantic info, and doesn't scale.
CBOW (Continuous Bag of Words) predicts the center word from context: given [Kwame, na, ____, aba], predict the missing word. Skip-gram does the opposite: given a word, predict surrounding words. Skip-gram is more flexible, works better on rare words, and is generally preferred. For Kwame's Twi corpus, Skip-gram captures long-range semantic relationships better because it has 2× as many training examples (each word can be a target for multiple context windows).
FastText learns embeddings for subword n-grams (3-grams, 4-grams, 5-grams), not just whole words. Twi words inflect heavily (ɔbaabuo, ɔbaabuo+no, ɔbaabuo+a share the stem "ɔbaabuo"). FastText captures the shared structure—related inflections have similar embeddings because they share n-grams. Word2Vec treats each form independently, missing that connection. Additionally, FastText can embed OOV (out-of-vocabulary) words by reconstructing them from their n-grams. A rare inflection like "ɔbaabuo+i" can still be embedded even if never seen in training.
Embeddings encode semantic relationships as linear directions in space. The vector (king − man) captures the "gender offset" (from male to female). Adding that offset to "woman" (a female figure) shifts it toward another female figure with similar status—"queen." This works because embeddings naturally learn that king and queen differ mainly in gender, and man and woman differ in the same gender axis. The arithmetic (king − man + woman) transfers the gender relationship. For Twi: "ɔbaabuo" − "ɔbarima" = gender offset; adding it to "ɔbarima_nta" (boy) should approach "ɔbaabuo_nta" (girl). This magic emerges from training—the model doesn't explicitly learn "gender axis," but embeddings naturally organize themselves that way.
(1) Check your window size—if too small (e.g., window=2), context may miss long-range morphological relationships. Increase to window=5 or window=8. (2) Check min_count—if "ɔbaabuo_nta" is rare (< 10 occurrences), it may not have enough context windows to learn a good embedding. Lower min_count to 2 or 1. (3) Increase epochs and vector_size—50 dims may be too small; try 100 or 150. (4) Use FastText instead of Word2Vec—subword n-grams will capture the shared "ɔbaabuo" stem. (5) Inspect your tokenizer—ensure "ɔbaabuo" and "ɔbaabuo_nta" are separate tokens, not merged or split incorrectly. (6) Check corpus size—2.3M tokens is small; similarity may be noisy. If you have more data, train on it. (7) Consider domain balance—if "ɔbaabuo_nta" appears mostly in certain texts (poetry, narrative), they may have different contexts than "ɔbaabuo" (news), reducing similarity.

✓ Quick Quiz

Question 1 / 5
One-hot encoding of a 2.3M-word Twi vocabulary produces vectors of what dimensionality?
Question 2 / 5
In Word2Vec Skip-gram, what is the model trying to predict?
Question 3 / 5
Why is FastText better than Word2Vec for Twi, a morphologically rich language?
Question 4 / 5
In PyTorch's nn.Embedding, what does the weight matrix represent?
Question 5 / 5
The embedding arithmetic ɔbaabuo − ɔbarima + ɔbarima_nta ≈ ɔbaabuo_nta assumes what about the embedding space?
0

📋 Quick Cheatsheet

Concept Code / Usage Notes
Train Word2Vec Skip-gram Word2Vec(sentences, vector_size=100, sg=1) sg=1 for Skip-gram; sg=0 for CBOW. Fast, good on small corpora.
Train FastText FastText(sentences, vector_size=100, sg=1) Subword n-grams; handles OOV & morphology. Better for Twi.
Embedding similarity model.wv.similarity('ɔbaabuo', 'ɔbarima') Cosine similarity in [0, 1]. High = semantically close.
Find similar words model.wv.most_similar('ɔbaabuo', topn=5) Returns top-5 nearest neighbors in embedding space.
Embedding arithmetic model.wv['king'] - model.wv['man'] + model.wv['woman'] Vector arithmetic encodes semantic relationships (gender, number, status).
PyTorch nn.Embedding nn.Embedding(vocab_size, embedding_dim) Lookup table: (batch, seq_len) indices → (batch, seq_len, dim) embeddings.
Load pretrained into nn.Embedding embedding_layer.weight = nn.Parameter(torch.from_numpy(matrix)) Initialize from Word2Vec, GloVe, or FastText matrix. Can freeze (requires_grad=False).
Freeze embeddings (no updates) embedding_layer.weight.requires_grad = False Useful if pretrained embeddings are already high-quality; saves memory.

⚠️ Common Mistakes

Using one-hot encoding for large vocabularies
One-hot produces 2.3M-dimensional vectors for a Twi corpus; RNNs crawl, memory explodes. Always use dense embeddings (Word2Vec, GloVe, FastText, or learned nn.Embedding). Reduces to ~100-300 dims with better semantics.
Choosing Word2Vec over FastText for morphologically rich languages
Twi inflects heavily (ɔbaabuo, ɔbaabuo+no, ɔbaabuo+a). Word2Vec treats each form independently. FastText's n-gram subwords capture shared morpheme structure, yielding better OOV handling and analogy performance. Use FastText for morphologically rich languages.
Too-small window size in Word2Vec
Window=2 captures only immediate neighbors; semantic relationships need wider context. Use window=5 to window=10. For Twi, wider windows help capture discourse patterns.
Setting min_count too high
min_count=10 discards rare inflections (e.g., ɔbaabuo_nta if it appears 5 times). For low-resource languages, lower min_count to 2–3 to retain morphological variants.
Not freezing pretrained embeddings when fine-tuning
Pretrained embeddings (trained on 100M tokens) can overfit if updated on a small downstream task (10K tweets). Freeze embedding_layer.weight (requires_grad=False) to preserve knowledge, or use very low learning rates (1e-4).

✦ Key Takeaways

  • Dense word embeddings (100–300 dims) replace sparse one-hot encoding (millions of dims) and capture semantic meaning: similar words cluster together, embedding arithmetic encodes relationships.
  • Word2Vec learns embeddings via CBOW (context → word) or Skip-gram (word → context). Skip-gram is generally better; fast training, good on small corpora.
  • GloVe uses global co-occurrence statistics for richer representations on large corpora (100M+ tokens). Slower training but often better quality.
  • FastText extends Word2Vec with subword n-grams, perfect for morphologically rich languages like Twi. Handles OOV words gracefully; "ɔbaabuo+i" (unseen inflection) can still be embedded.
  • PyTorch's nn.Embedding is a learnable lookup table. Initialize with pretrained vectors or train end-to-end. Can freeze embeddings to preserve pretrained knowledge.
  • Embedding arithmetic (king − man + woman ≈ queen) works because embeddings encode semantic relationships as linear directions. Kwame can solve Twi morphological analogies this way.
  • Kwame's Twi embeddings (trained on 2.3M tokens via Word2Vec/FastText) enable downstream NLP tasks: sentiment analysis, machine translation, named entity recognition—all without manually labeled embeddings.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You've mastered word embeddings.
Now build production NLP systems.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →