🌍
Meet Kwame
NLP Researcher Building Tools for Ghanaian Languages

Kwame is building sentiment analysis for Twi social media posts (Akan language, 9M speakers in Ghana). Training a transformer from scratch requires 100M+ text examples and weeks of compute. But multilingual BERT—pretrained on 104 languages including Twi—is available. He'll fine-tune it on his 8,000 labeled Twi posts to detect sentiment (positive/negative/neutral), achieving 91% accuracy in 2 hours.

110M
BERT parameters
2 hours
Fine-tune time
91%
Sentiment accuracy

🏗️ BERT Architecture: What Is BERT?

BERT (Bidirectional Encoder Representations from Transformers) is a 12-layer (base) or 24-layer (large) transformer encoder pretrained on 3.3B words from Wikipedia and BookCorpus. Unlike LSTMs or unidirectional GPT, BERT reads left-to-right AND right-to-left, capturing rich context.

BERT Building Blocks

  • 12 transformer encoder layers: Each layer: multi-head attention (12 heads) + feed-forward. No decoder, no autoregressive generation.
  • 768 hidden dimensions: Each token → 768D vector. Kwame's "olúwa" (lord) gets a rich contextual representation.
  • Bidirectional: When encoding "Kwame na Ama ba fie" (Kwame and Ama came home), "Ama" sees left context (Kwame, na) AND right context (ba, fie).
  • [CLS] token: Special token at sequence start. Its final hidden state = sentence-level representation for classification.
  • [SEP] token: Separates two sentences (for tasks like sentence-pair classification).
BERT Base: 12 layers × 768 dims × 12 attention heads
BERT model sizes and architecture comparison
BERT-base: 110M params. BERT-large: 340M params. Larger models capture more knowledge but require more compute and memory. DistilBERT (Lesson 23 challenge) reduces BERT-base to 66M params (40% smaller) with 97% performance retention.

🎯 Pretraining Objectives: MLM & NSP

BERT learns two objectives on unlabeled text, enabling it to understand language deeply before any fine-tuning:

1. Masked Language Modeling (MLM)

Mask 15% of tokens randomly; predict them from context. Example: "Kwame na [MASK] ba fie." → Predict "Ama". The model learns that "Ama" is likely given male context (Kwame), conjunction (na), and result (ba fie). During pretraining, BERT sees millions of masked positions, learning syntax, semantics, and domain knowledge.

2. Next Sentence Prediction (NSP)

Given two sentences, predict if they're consecutive (IsNext) or random (NotNext). Example: Pair "Kwame wɔ fie" (Kwame is home) with either "Ama ba" (Ama came) [IsNext] or "Bɛɛf wo sɛ..." (There is beef...) [NotNext]. The model learns discourse coherence and relationships.

MLM & NSP loss (conceptual)

🔤 Tokenizers: WordPiece & BPE

BERT converts text to tokens (subword units) before encoding. Two main strategies:

WordPiece (BERT)

Starts with 30k vocabulary. Merges frequent subwords. "unbelievable" → ["un", "##believe", "##able"]. Prefixes are whole tokens; continuations start with "##". Perfect for Twi: "ɔbaabuo" (woman) → ["ɔbaabuo"] or ["ɔba", "##abuo"] depending on frequency.

BPE (RoBERTa, GPT)

Byte Pair Encoding merges pairs bottom-up. Starts with 50k merges. "unbelievable" → ["unbelievable"] (already high-frequency). More flexible for rare words and morphologically rich languages.

Tokenization examples

🚀 HuggingFace Pipeline API: The Easy Way

HuggingFace's pipeline API abstracts tokenization, model loading, and inference. One line to classify Twi sentiment:

HuggingFace pipeline for sentiment

Why pipeline? No manual tokenization, no batch handling, no GPU config. HuggingFace handles it. Perfect for prototyping and inference.

🎓 Fine-tuning BERT for Classification

Kwame has 8,000 labeled Twi tweets (positive/negative/neutral). He'll fine-tune multilingual BERT (pretrained on Twi!) on this task. Steps:

Fine-tuning Workflow

  • 1. Load pretrained BERT: bert-base-multilingual-cased (weights from 104-language pretraining).
  • 2. Add task-specific head: Linear layer: [CLS] representation (768D) → 3 classes (positive/negative/neutral).
  • 3. Train on labeled data: 8,000 Twi tweets with cross-entropy loss.
  • 4. Evaluate on test set: Accuracy, F1-score, confusion matrix.
Fine-tuning accuracy improves with more labeled data; DistilBERT trains 2.5x faster
BERT fine-tuning performance vs training time
BERT-base: 91% accuracy after 2 epochs on 8K Twi tweets (2 hours). DistilBERT: 89% after 3 epochs (30 min). RoBERTa: 92% after 2 epochs (2.5 hours). Trade accuracy for speed with DistilBERT or accept marginal gains with RoBERTa.
Fine-tuning BERT for Twi sentiment classification

⚖️ BERT Variants: Size vs. Accuracy Trade-offs

BERT-base is the default, but variants exist for different needs:

Model Parameters Training Time Accuracy Use Case
bert-base-multilingual-cased 110M 2.0h 91.2% Standard; good accuracy
distilbert-base-multilingual-cased 66M (-40%) 0.8h (-60%) 89.1% Production inference; mobile
xlm-roberta-base 125M 2.5h 92.4% Better multilingual
albert-base-v2 12M (-89%) 1.2h 87.8% Tiny model; fast inference

DistilBERT: Distilled from BERT using knowledge distillation. Keeps 97% of BERT's knowledge in 66M parameters.

RoBERTa: Improved BERT pretraining (more data, longer training). Better for downstream tasks.

ALBERT: Parameter sharing across layers reduces model size to 12M. Trade-off: slightly lower accuracy but incredible speed.

🎯 Inference & Production Deployment

Once fine-tuned, Kwame deploys his Twi sentiment classifier to a REST API:

FastAPI server for Twi sentiment inference

Optimization tips:

  • Use DistilBERT for real-time inference (30ms vs. 80ms for BERT-base).
  • Batch requests: Process 32 texts together, not one-by-one.
  • Quantize to INT8: Reduce model size 4x (110M → 27.5M) with minimal accuracy loss.
  • Cache tokenizer outputs for repeated predictions.
Question 1 of 5
In BERT pretraining, what is the purpose of the Masked Language Model (MLM) objective?
Question 2 of 5
WordPiece tokenization represents "unbelievable" as ["un", "##believe", "##able"]. What does "##" indicate?
Question 3 of 5
When fine-tuning BERT for sentiment classification, what is the role of the [CLS] token's final hidden state?
Question 4 of 5
DistilBERT achieves 40% parameter reduction (110M → 66M) while retaining 97% performance. Which technique enables this?
Question 5 of 5
Kwame's classifier uses bert-base-multilingual-cased. Why is multilingual BERT suitable for Twi?

💼 5 Interview Questions & Answers

BERT (MLM): Bidirectional. Masks 15% of tokens and predicts them from left and right context. Example: "[MASK] na Ama ba fie" → predicts "Kwame" using "na", "Ama", "ba", "fie". Sees future tokens.

GPT (Causal): Unidirectional (left-to-right). Predicts next token only from past tokens. Example: "Kwame na" → predict "Ama". Never sees future. GPT is generative (write essays); BERT is understanding-focused (classification, QA).

The [SEP] token (ID 102) separates two input sequences. Used in pair classification tasks like: "Kwame na Ama ba fie [SEP] Yoo a, na ebi den aba." (two sentences) → [CLS] output predicts entailment, contradiction, or neutral. Position: Appended at the end of second sentence.

1. Use multilingual pretrained model: bert-base-multilingual-cased is pretrained on both English and Twi. Cross-lingual transfer helps.

2. Augment training data: Mix English sentiment + Twi sentiment. Back-translate Twi ↔ English.

3. Fine-tune on domain: Continue pretraining on unlabeled Twi social media (MLM objective) before classification fine-tuning.

4. Reduce learning rate: Use 1e-5 (lower than 2e-5 default) to make cautious updates.

Speed: 2.5x faster inference (30ms vs. 80ms). If you need sub-50ms latency, DistilBERT is essential.

Memory: 40% smaller (66M vs. 110M params). Fits on phones and edge devices.

Cost: Fewer GPU hours = lower cloud costs. Distil ~$0.30/1M requests; BERT ~$0.75/1M.

Accuracy trade-off: 89% (Distil) vs. 91% (BERT). Many production systems tolerate this 2% difference.

1. Learning rate too high (0.01): Catastrophic forgetting. Fix: Use 2e-5.

2. No validation set: Overfit to train. Fix: Always use 80/20 train/test.

3. Max sequence length too short (64): Truncate important context. Fix: Use 128 or 256.

4. Batch size too large (64) on limited GPU: OOM error. Fix: Start with 16.

5. Train too long (5+ epochs): Validation accuracy plateaus or drops. Fix: Monitor eval loss; stop at 2-3 epochs.

🎯 Challenge: Build a DistilBERT Sentiment Classifier

Your Task
Build a sentiment classifier for Twi using DistilBERT (not BERT-base).
  • Download the Twi sentiment dataset: 8,000 tweets (positive/negative/neutral).
  • Fine-tune distilbert-base-multilingual-cased for 2 epochs with batch size 32 and learning rate 2e-5.
  • Evaluate on test set (20% of data). Report accuracy, F1-score, and confusion matrix.
  • Time the training and inference on a single GPU. Compare speed vs. BERT-base.
  • Optional: Deploy on a Hugging Face Space or Gradio app for public inference.
Bonus: Fine-tune RoBERTa-base (xlm-roberta-base) on the same data. Compare accuracy, speed, and parameter count. Which model would you recommend for production?

📋 BERT & HuggingFace Cheatsheet

Concept Definition / Code
[CLS] token Special token at sequence start. Its final hidden state = sentence-level representation. ID: 101.
[SEP] token Separates two input sequences. ID: 102.
MLM Masked Language Model. Mask 15% of tokens; predict from context. Enables bidirectional learning.
NSP Next Sentence Prediction. Classify if two sentences are consecutive (IsNext / NotNext).
WordPiece Tokenization with subword units. Prefix whole tokens; continuations start with ##. vocab = 30k.
BPE Byte Pair Encoding. Merges frequent pairs. Used by RoBERTa, GPT. vocab >= 50k.
from_pretrained() Load pretrained model: AutoModel.from_pretrained("bert-base-multilingual-cased")
pipeline() Easy inference: pipeline("sentiment-analysis", model="...")
Fine-tune LR Use 2e-5 or 5e-5 (much lower than training from scratch). Preserves pretrained knowledge.
DistilBERT 66M params (-40%), 2.5x faster, 97% BERT performance. Use for production inference.
RoBERTa Improved BERT pretraining (more data, longer). 125M params. 1-2% better accuracy.
ALBERT 12M params (-89%), parameter sharing. Ultra-fast but 3-5% lower accuracy.

⚠️ Common Mistakes & Fixes

Mistake #1: Learning rate too high
Using lr=0.01 (training from scratch rate) causes catastrophic forgetting. Pretrained weights diverge, losing knowledge. Fix: Use lr=2e-5 for fine-tuning. BERT is already smart; small updates suffice.
Mistake #2: No max_length truncation
If a Twi document is 1,000 tokens, BERT chokes (trained on 512 max). Fix: Tokenize with truncation=True, max_length=512. Longer texts are rare; losing tail tokens is acceptable.
Mistake #3: Forgetting to add [CLS] and [SEP]
Manually tokenizing "Kwame na Ama" without special tokens loses positional encoding. Fix: Always use tokenizer.encode(text, add_special_tokens=True) or let HuggingFace trainer handle it.
Mistake #4: Overfitting with too many epochs
Training 8,000-tweet dataset for 10 epochs causes validation accuracy to plateau or drop. Fix: Monitor validation loss; stop at 2-3 epochs. Use early stopping to auto-stop when eval loss worsens.
Mistake #5: Not using gradient accumulation with large batch sizes
Wanting batch_size=64 but GPU has 8GB RAM → OOM. Fix: Use gradient accumulation: effective batch 64 = 4 steps of 16. Or reduce to batch 16-32.

🎓 Key Takeaways

  • BERT is bidirectional: Learns from left and right context via MLM pretraining. Perfect for understanding-based tasks (classification, NER, QA).
  • Pretraining saves compute: Fine-tuning on 8,000 Twi tweets takes 2 hours; training from scratch needs 100M+ examples. Leverage existing models.
  • Multilingual BERT supports low-resource languages: Twi wasn't in BERT's training, but cross-lingual knowledge transfers. Multilingual models enable NLP for 100+ languages.
  • Tokenizers matter: WordPiece vs. BPE affects how subwords are split. Choose a tokenizer trained on your language/domain if possible.
  • HuggingFace pipeline abstracts complexity: One line to classify. Production-grade inference with batching, GPU support, and auto-quantization.
  • Fine-tuning hyperparameters are tuned: lr=2e-5, batch=16, epochs=2-3 work across tasks. Not magic; empirically validated.
  • DistilBERT is your speed secret: 40% smaller, 2.5x faster, 97% accuracy. Use for real-time inference; BERT-base for best quality if latency allows.
  • Evaluate thoughtfully: Accuracy alone hides imbalance. Use F1-score, confusion matrix, and per-class metrics.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Ready to Master Deep Learning & Transformers?

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →