🎬
Real-world scenario
Kwame at Accra Social Media Analytics: build sentiment analyzer for Ghanaian social feedback

Kwame is an NLP researcher at a startup in Accra building a social media analytics platform for African businesses. His task: classify customer sentiment from product reviews and social comments in real time. He has 50,000 labeled movie reviews (25K positive, 25K negative) from a public NLP dataset. He needs to build and compare multiple models: start with LSTM (fast, interpretable), add attention mechanism (BiLSTM+Attn), then fine-tune transformer models (DistilBERT for mobile, BERT-base for accuracy). His goal: understand the accuracy-latency tradeoff and choose the best model for production deployment to 100K+ daily users. Final comparison: LSTM 81.2% vs BiLSTM+Attn 84.5% vs DistilBERT 86.8% vs BERT-base 87.1%.

50K
Movie reviews
4 models
To compare
87.1%
Best accuracy

📝 Meet the data: Movie Reviews

We're building a sentiment classifier on 50,000 movie reviews — 25,000 positive (rating 7+) and 25,000 negative (rating 4-). The reviews are raw text (English), ranging from 10 to 2000 words, with real user language (spelling errors, slang, emojis).

Dataset stats: 25K positive reviews (avg 150 words), 25K negative reviews (avg 148 words), vocab size 88K unique tokens after cleaning. This is a balanced binary classification problem.

What we're building: a production-grade sentiment analyzer

The full pipeline:

  1. Load movie reviews (50K) and split into train/val/test (35K/7.5K/7.5K)
  2. Preprocess: clean text, tokenize, build vocabulary, create word embeddings
  3. LSTM baseline: single-layer LSTM, Adam optimizer, train to 81.2% accuracy
  4. BiLSTM+Attention: bidirectional LSTM with attention mechanism, 84.5% accuracy
  5. DistilBERT fine-tune: lightweight transformer, 86.8% accuracy, fast inference
  6. BERT-base fine-tune: full BERT model, 87.1% accuracy, best performance
  7. Head-to-head comparison: accuracy, speed, memory, confusion matrices, error analysis
  8. Deployment: save best model, test on production data, monitor for drift

Expected results: LSTM 81.2%, BiLSTM+Attn 84.5%, DistilBERT 86.8%, BERT-base 87.1%. Trade-off: BERT slower but most accurate; DistilBERT balances speed and accuracy; LSTM fastest but least accurate.

Step 1: Load & preprocess movie reviews

Start by loading the dataset and cleaning text. Remove HTML, convert to lowercase, tokenize, handle special characters.

1_load_preprocess.py Load reviews and build vocabulary
Output
Loaded 50000 reviews Positive: 25000, Negative: 25000 Sample cleaned review: this movie is fantastic the acting is superb and the plot keeps you engaged throughout the... Vocabulary size: 50000 Top 10 words: ['the', 'a', 'and', 'is', 'to', 'of', 'in', 'it', 'that', 'i']

The dataset is perfectly balanced. Word frequency follows Zipfian distribution (the, a, and dominate). After cleaning, we have 50K vocabulary tokens to work with.

Step 2: Build LSTM baseline (81.2% accuracy)

Start with a simple LSTM: embed words, pass through LSTM, pool final hidden state, classify with linear layer. Fast training, interpretable.

2_lstm_baseline.py Single-layer LSTM sentiment classifier
Output
Parameters: 6,412,801 Batch loss: 0.6892

The LSTM baseline has ~6.4M parameters. After 5 epochs on 35K training reviews, it reaches 81.2% test accuracy. Fast training (2 hours GPU), but limited by single-direction context.

Step 3: BiLSTM with attention (84.5% accuracy)

Improve the LSTM by using bidirectional layers (see both directions of context) and attention mechanism (focus on important words). Let the model learn which words matter for sentiment.

3_bilstm_attention.py BiLSTM with self-attention layer
Output
Parameters: 6,821,249 Test Accuracy: 84.5% Top attended words: ['excellent', 'terrible', 'amazing', 'awful', 'masterpiece'] Bottom attended words: ['the', 'a', 'and', 'is']

BiLSTM+Attn adds 3.3% absolute accuracy over LSTM baseline. Bidirectional context + attention helps the model focus on opinion words and ignore articles/prepositions. Still only ~7 hours training.

Step 4: DistilBERT fine-tuning (86.8% accuracy)

Switch to transformers: DistilBERT is a lightweight BERT (40% smaller, 60% faster). Pre-trained on Wikipedia + BookCorpus, we fine-tune on our reviews for 2-3 epochs.

4_distilbert_finetune.py Fine-tune DistilBERT for sentiment
Output
DistilBERT parameters: 66,362,880 Epoch 1 loss: 0.3256 Epoch 2 loss: 0.1842 Test Accuracy: 86.8% Speed (GPU): 50ms/review (1000 reviews/min) Speed (CPU): 200ms/review (300 reviews/min)

DistilBERT reaches 86.8% accuracy in just 2 epochs (4-5 hours GPU). The massive jump from LSTM (81.2%) shows transformer pre-training captures deep semantic understanding. Trade-off: 66M parameters (vs 6.4M for LSTM) but still deployable on edge devices.

Step 5: BERT-base fine-tuning (87.1% accuracy)

Use full BERT-base for best accuracy. Same fine-tuning procedure, but with more parameters and capacity. Only slightly slower than DistilBERT, but +0.3% accuracy gains matter in production.

5_bert_finetune.py Fine-tune BERT-base (best accuracy)
Output
BERT-base parameters: 109,482,240 Epoch 1 loss: 0.3102 Epoch 2 loss: 0.1725 Epoch 3 loss: 0.1201 Test Accuracy: 87.1% Speed (GPU): 70ms/review (857 reviews/min) Speed (CPU): 350ms/review (286 reviews/min)

BERT-base is the accuracy champion at 87.1%. Takes 6 hours to fine-tune (3 epochs), slightly slower inference than DistilBERT, but 0.3% absolute gain in accuracy. For production: use BERT if accuracy is critical (e.g., content moderation); use DistilBERT if speed matters (e.g., real-time web API).

Step 6: Head-to-head comparison

Compare all four models on the same test set. Create confusion matrices, analyze error patterns, measure latency, memory footprint.

Model Accuracy Parameters Train Time Inference (GPU) Inference (CPU)
LSTM Baseline 81.2% 6.4M 2 hours 5ms 50ms
BiLSTM+Attn 84.5% 6.8M 7 hours 12ms 120ms
DistilBERT 86.8% 66M 4 hours 50ms 200ms
BERT-base 87.1% 110M 6 hours 70ms 350ms
Key insight: DistilBERT wins the accuracy-speed-parameters tradeoff. For Kwame's Accra platform (100K daily users, limited server), DistilBERT's 86.8% accuracy + 50ms latency + 66M parameters is ideal. BERT-base for premium tier; LSTM for edge devices.

Error Analysis: What the models get wrong

Common mistakes across models:

Sarcasm & irony
"This movie is SO good!" — all models struggle with inverted sentiment. LSTM: -15% accuracy on sarcasm. BERT: -8% (pre-training helps).
Mixed sentiment
"Great plot but terrible acting" — review is truly neutral. LSTM predicts 55% confident positive; BERT predicts 52%. Models default to dominant sentiment word.
Negation scope
"Not a bad movie" — negative word + negation = positive. LSTM treats "bad" as negative (loses -3% accuracy). BERT handles better (-1%).

Step 7: Deploy to production

Save best model (DistilBERT), wrap in API, monitor for sentiment drift.

7_deploy.py Save & wrap model for API
Output
Models saved to disk. sentiment-model-distilbert/ config.json (741 bytes) pytorch_model.bin (267 MB) tokenizer.json (711 KB) special_tokens_map.json (112 bytes) API running on port 8000 Test inference: 51ms {'sentiment': 'POSITIVE', 'confidence': 0.9987}

DistilBERT model deployed as REST API. Daily logs track sentiment distribution (should be ~50% positive if data unchanged). If positive rate jumps to 70%, alerts Kwame to investigate (potential bias in new data or user behavior shift).

AI in Practice: Why BERT Beats LSTM

LSTM processes text left-to-right: "great [hidden state] movie" never sees "great" when processing "movie". BERT uses masked language modeling: tokens see all other tokens at once (with attention). This bidirectional context + 12 transformer layers + pre-training on 3.3 billion words → BERT understands "great movie" (6% accuracy gain) and "not bad" (understands negation). Lesson: if you have labeled data + compute, fine-tune BERT. If data is scarce, LSTM + hand-crafted features often wins. Kwame's 50K reviews + GPU = BERT's territory.

transformers attention mechanisms transfer learning sentiment analysis
Challenge: Adapt for Multilingual Sentiment

Kwame's next project: classify sentiment from reviews in English, Twi (Ghana's local language), and Pidgin. Movie reviews come in mixed-language comments.

Task: How would you adapt the DistilBERT approach for 3 languages? (Hint: HuggingFace offers XLM-RoBERTa, mBERT, or XLMR pre-trained on 100+ languages.)
Use xlm-roberta-base from HuggingFace instead of DistilBERT. It's pre-trained on 2.5TB text in 100+ languages. Fine-tune on 10K reviews each (30K total) in your 3 languages. Evaluation: test on held-out reviews from each language separately and on code-switched (Twi+English) comments. You'll find cross-lingual transfer helps: a model trained on 30K English reviews can classify Twi comments at 75% accuracy (even without Twi in training, due to shared semantic space learned during pre-training).

Interview prep: questions you'll face

Answer: BERT uses bidirectional transformers. Each token attends to all other tokens simultaneously, capturing long-range dependencies. LSTM is sequential: left-to-right or right-to-left, missing full context until the end. Example: "not bad" — LSTM processes "not" without seeing "bad"; BERT sees both atoms at layer 1. Plus, BERT is pre-trained on 3.3B words, learning general English semantics; LSTM starts from scratch. Fine-tuning BERT leverages transfer learning. Together: 6% gain.
Answer: DistilBERT. For production at 10M scale, latency + cost dominate. DistilBERT: 50ms/review, 66M params, 86.8% accuracy. BERT-base: 70ms, 110M params, 87.1% accuracy. 10M * (70-50)ms = 200K compute hours difference per batch prediction. Cost: $500K+ per year on inference. The 0.3% accuracy gap (maybe 30K false positives) is worth $500K savings. Deploy DistilBERT, monitor false positives, A/B test BERT on 1% traffic for validation.
Answer: Sarcasm is hard because it requires pragmatic understanding. Three approaches: (1) Collect sarcasm-labeled data (500-1K examples). Fine-tune DistilBERT on mixed data (general reviews + sarcasm). Helps ~10%. (2) Use context: Kwame's platform logs user rating + comment. If rating=2 but text="fantastic movie", flag as likely sarcasm. (3) Ensemble: BERT for accuracy + simple keyword rules for known sarcasm markers (ALL CAPS, extra punctuation!!!). In practice: label 1K sarcasm examples, add to training (0.1% of 50K), re-train DistilBERT. Sarcasm accuracy improves from 65% to 75%.
Answer: (1) Collect mistakes: Log all false positives daily (e.g., predicted positive, user feedback negative). 100 mistakes/day. (2) Analysis: Every 2 weeks, label the top 200 mistakes, add to training set. Re-train DistilBERT. (3) Evaluation: A/B test new model on 5% traffic. If accuracy improves, roll out to 100%. (4) Drift detection: Monitor sentiment distribution. If positive rate shifts >5%, retrain. (5) Seasonal updates: Movie reviews vary by genre/season. Quarterly re-training on recent data. Expected improvement: +0.5% accuracy every quarter from data diversity and handling edge cases.

Full pipeline: end-to-end notebook

Complete code combining all steps (load, preprocess, train 4 models, compare, deploy):

full_sentiment_pipeline.py Complete end-to-end sentiment analyzer (250 lines)

Quiz: test your understanding

Question 1 of 5
Why does BERT outperform LSTM on sentiment analysis?
Question 2 of 5
For a production API with 100K daily requests, which model is best?
Question 3 of 5
How many movie reviews were used in this project?
Question 4 of 5
What technique does the BiLSTM+Attention model use to improve over baseline LSTM?
Question 5 of 5
What's the biggest weakness of all four models?
0
/5
Excellent! You've mastered sentiment analysis with transformers.

Cheatsheet: model comparison & deployment

Model Accuracy Best For Code Pattern
LSTM Baseline 81.2% Edge devices, fast training nn.LSTM(embed_dim, hidden)
BiLSTM+Attn 84.5% Mobile, interpretable nn.LSTM(..., bidirectional=True)
DistilBERT 86.8% Production API (100K+ users) DistilBertForSequenceClassification.from_pretrained(...)
BERT-base 87.1% High-accuracy systems BertForSequenceClassification.from_pretrained(...)

Quick deployment: Save DistilBERT model with model.save_pretrained('path'). Load in production: pipeline('sentiment-analysis', model='path', device=0). Monitor with logs.

Common mistakes: what breaks and how to fix it

Mistake 1: Not padding sequences to same length
LSTM expects fixed-size input. Use nn.utils.rnn.pad_sequence() or truncate to max_length=256. Batch size also breaks if sequences vary wildly. Solution: pad to max in each batch, use pack_padded_sequence to ignore padding in LSTM.
Mistake 2: Using wrong tokenizer for BERT
BERT and DistilBERT have different tokenizers. If you use BertTokenizer with DistilBertModel, vocab mismatch causes [UNK] explosion. Always match: DistilBertTokenizer + DistilBertForSequenceClassification.
Mistake 3: Not calling model.eval() before inference
Training uses dropout and batch norm; inference uses different behavior. Always call model.eval() before testing. Forgetting this inflates test accuracy by 3-5%.
Mistake 4: Over-fitting to small validation set
With 35K training reviews, use stratified k-fold (5 folds, 7K each) instead of single 7.5K val set. Single val set can be unlucky; a bad random split inflates reported accuracy. K-fold gives robust estimate.
Mistake 5: Training BERT with same learning rate as LSTM
BERT fine-tuning uses lr=2e-5 (very small). Using 1e-3 (LSTM default) destroys pre-trained weights. Always use AdamW + warmup schedule for transformers.

Key takeaways

Kwame's journey from LSTM baseline to BERT production model captures the evolution of NLP in practice.

  • Baseline → complexity: Start with simple LSTM (fast feedback), then add attention, then transformers. Each step costs more (training time, compute) but gains accuracy. Don't start with BERT if you haven't validated the problem with LSTM.
  • Pre-training is magic: BERT's 6% accuracy gain comes entirely from seeing 3.3B words before your task. Transfer learning is one of deep learning's superpowers. Always look for a pre-trained model in your domain.
  • Transformers democratize NLP: BERT fine-tuning requires only 2-3 epochs (4-6 GPU hours). A decade ago, this required months of expert feature engineering. Today, you can build state-of-the-art sentiment analysis in a weekend.
  • Production = trade-off: DistilBERT at 86.8% beats BERT at 87.1% in practice (speed, cost, memory). Know your constraints: 100K users → DistilBERT. 1K users → BERT. 1M users → LSTM or DistilBERT on CPU.
  • Error analysis beats accuracy metrics: 87.1% average hides sarcasm (65% accuracy). Understand failure modes. Use confidence scores: flag low-confidence predictions for human review.
  • Continuous improvement: Ship DistilBERT, log mistakes daily, retrain monthly on hard examples. Expect +0.5% improvement per quarter from active learning.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You've built a production sentiment analyzer with transformers.
Now deploy it to millions.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →