Build sentiment classifiers on 50K movie reviews. Kwame in Accra compares LSTM baseline (81.2%) vs BiLSTM+Attn (84.5%) vs DistilBERT (86.8%) vs BERT-base (87.1%). Full end-to-end pipeline: tokenization, embedding, training, evaluation, error analysis, deployment. NLP capstone.
📝 Text Classification🧠 LSTM & Transformers🔄 BERT Fine-tuning📊 Head-to-head Comparison⏱️ ~35 min
Lesson 24 of Deep Learning Track (Arc E — NLP & Transformers)
🎬
Real-world scenario
Kwame at Accra Social Media Analytics: build sentiment analyzer for Ghanaian social feedback
Kwame is an NLP researcher at a startup in Accra building a social media analytics platform for African businesses. His task: classify customer sentiment from product reviews and social comments in real time. He has 50,000 labeled movie reviews (25K positive, 25K negative) from a public NLP dataset. He needs to build and compare multiple models: start with LSTM (fast, interpretable), add attention mechanism (BiLSTM+Attn), then fine-tune transformer models (DistilBERT for mobile, BERT-base for accuracy). His goal: understand the accuracy-latency tradeoff and choose the best model for production deployment to 100K+ daily users. Final comparison: LSTM 81.2% vs BiLSTM+Attn 84.5% vs DistilBERT 86.8% vs BERT-base 87.1%.
50K
Movie reviews
4 models
To compare
87.1%
Best accuracy
📝 Meet the data: Movie Reviews
We're building a sentiment classifier on 50,000 movie reviews — 25,000 positive (rating 7+) and 25,000 negative (rating 4-). The reviews are raw text (English), ranging from 10 to 2000 words, with real user language (spelling errors, slang, emojis).
Dataset stats: 25K positive reviews (avg 150 words), 25K negative reviews (avg 148 words), vocab size 88K unique tokens after cleaning. This is a balanced binary classification problem.
What we're building: a production-grade sentiment analyzer
The full pipeline:
Load movie reviews (50K) and split into train/val/test (35K/7.5K/7.5K)
Preprocess: clean text, tokenize, build vocabulary, create word embeddings
LSTM baseline: single-layer LSTM, Adam optimizer, train to 81.2% accuracy
BiLSTM+Attention: bidirectional LSTM with attention mechanism, 84.5% accuracy
DistilBERT fine-tune: lightweight transformer, 86.8% accuracy, fast inference
BERT-base fine-tune: full BERT model, 87.1% accuracy, best performance
Deployment: save best model, test on production data, monitor for drift
Expected results: LSTM 81.2%, BiLSTM+Attn 84.5%, DistilBERT 86.8%, BERT-base 87.1%. Trade-off: BERT slower but most accurate; DistilBERT balances speed and accuracy; LSTM fastest but least accurate.
Step 1: Load & preprocess movie reviews
Start by loading the dataset and cleaning text. Remove HTML, convert to lowercase, tokenize, handle special characters.
1_load_preprocess.pyLoad reviews and build vocabulary
Output
Loaded 50000 reviews
Positive: 25000, Negative: 25000
Sample cleaned review:
this movie is fantastic the acting is superb and the plot keeps you engaged throughout the...
Vocabulary size: 50000
Top 10 words: ['the', 'a', 'and', 'is', 'to', 'of', 'in', 'it', 'that', 'i']
The dataset is perfectly balanced. Word frequency follows Zipfian distribution (the, a, and dominate). After cleaning, we have 50K vocabulary tokens to work with.
Step 2: Build LSTM baseline (81.2% accuracy)
Start with a simple LSTM: embed words, pass through LSTM, pool final hidden state, classify with linear layer. Fast training, interpretable.
The LSTM baseline has ~6.4M parameters. After 5 epochs on 35K training reviews, it reaches 81.2% test accuracy. Fast training (2 hours GPU), but limited by single-direction context.
Step 3: BiLSTM with attention (84.5% accuracy)
Improve the LSTM by using bidirectional layers (see both directions of context) and attention mechanism (focus on important words). Let the model learn which words matter for sentiment.
3_bilstm_attention.pyBiLSTM with self-attention layer
Output
Parameters: 6,821,249
Test Accuracy: 84.5%
Top attended words: ['excellent', 'terrible', 'amazing', 'awful', 'masterpiece']
Bottom attended words: ['the', 'a', 'and', 'is']
BiLSTM+Attn adds 3.3% absolute accuracy over LSTM baseline. Bidirectional context + attention helps the model focus on opinion words and ignore articles/prepositions. Still only ~7 hours training.
Step 4: DistilBERT fine-tuning (86.8% accuracy)
Switch to transformers: DistilBERT is a lightweight BERT (40% smaller, 60% faster). Pre-trained on Wikipedia + BookCorpus, we fine-tune on our reviews for 2-3 epochs.
4_distilbert_finetune.pyFine-tune DistilBERT for sentiment
DistilBERT reaches 86.8% accuracy in just 2 epochs (4-5 hours GPU). The massive jump from LSTM (81.2%) shows transformer pre-training captures deep semantic understanding. Trade-off: 66M parameters (vs 6.4M for LSTM) but still deployable on edge devices.
Step 5: BERT-base fine-tuning (87.1% accuracy)
Use full BERT-base for best accuracy. Same fine-tuning procedure, but with more parameters and capacity. Only slightly slower than DistilBERT, but +0.3% accuracy gains matter in production.
BERT-base is the accuracy champion at 87.1%. Takes 6 hours to fine-tune (3 epochs), slightly slower inference than DistilBERT, but 0.3% absolute gain in accuracy. For production: use BERT if accuracy is critical (e.g., content moderation); use DistilBERT if speed matters (e.g., real-time web API).
Step 6: Head-to-head comparison
Compare all four models on the same test set. Create confusion matrices, analyze error patterns, measure latency, memory footprint.
Model
Accuracy
Parameters
Train Time
Inference (GPU)
Inference (CPU)
LSTM Baseline
81.2%
6.4M
2 hours
5ms
50ms
BiLSTM+Attn
84.5%
6.8M
7 hours
12ms
120ms
DistilBERT
86.8%
66M
4 hours
50ms
200ms
BERT-base
87.1%
110M
6 hours
70ms
350ms
Key insight: DistilBERT wins the accuracy-speed-parameters tradeoff. For Kwame's Accra platform (100K daily users, limited server), DistilBERT's 86.8% accuracy + 50ms latency + 66M parameters is ideal. BERT-base for premium tier; LSTM for edge devices.
Error Analysis: What the models get wrong
Common mistakes across models:
Sarcasm & irony
"This movie is SO good!" — all models struggle with inverted sentiment. LSTM: -15% accuracy on sarcasm. BERT: -8% (pre-training helps).
Mixed sentiment
"Great plot but terrible acting" — review is truly neutral. LSTM predicts 55% confident positive; BERT predicts 52%. Models default to dominant sentiment word.
Negation scope
"Not a bad movie" — negative word + negation = positive. LSTM treats "bad" as negative (loses -3% accuracy). BERT handles better (-1%).
Step 7: Deploy to production
Save best model (DistilBERT), wrap in API, monitor for sentiment drift.
7_deploy.pySave & wrap model for API
# Save DistilBERT model
model.save_pretrained('sentiment-model-distilbert')
tokenizer.save_pretrained('sentiment-model-distilbert')
# Load in production
from transformers import pipeline
sentiment_pipeline = pipeline(
'sentiment-analysis',
model='sentiment-model-distilbert',
tokenizer='sentiment-model-distilbert',
device=0 # GPU 0
)
# API inference
def classify_review(text):
result = sentiment_pipeline(text[:512]) # Truncate to max length
label = result[0]['label'] # 'POSITIVE' or 'NEGATIVE'
score = result[0]['score'] # Confidence (0-1)
return {'sentiment': label, 'confidence': score}
# Example
review = "This movie exceeded my expectations. The cinematography was stunning!"
print(classify_review(review))
# Output: {'sentiment': 'POSITIVE', 'confidence': 0.9987}
# Monitor drift: log predictions daily
import json
from datetime import datetime
log_file = 'sentiment_predictions.jsonl'
def log_prediction(text, prediction):
entry = {
'timestamp': datetime.now().isoformat(),
'text': text[:100],
'sentiment': prediction['sentiment'],
'confidence': prediction['confidence']
}
with open(log_file, 'a') as f:
f.write(json.dumps(entry) + '\n')
Output
Models saved to disk.
sentiment-model-distilbert/
config.json (741 bytes)
pytorch_model.bin (267 MB)
tokenizer.json (711 KB)
special_tokens_map.json (112 bytes)
API running on port 8000
Test inference: 51ms
{'sentiment': 'POSITIVE', 'confidence': 0.9987}
DistilBERT model deployed as REST API. Daily logs track sentiment distribution (should be ~50% positive if data unchanged). If positive rate jumps to 70%, alerts Kwame to investigate (potential bias in new data or user behavior shift).
AI in Practice: Why BERT Beats LSTM
LSTM processes text left-to-right: "great [hidden state] movie" never sees "great" when processing "movie". BERT uses masked language modeling: tokens see all other tokens at once (with attention). This bidirectional context + 12 transformer layers + pre-training on 3.3 billion words → BERT understands "great movie" (6% accuracy gain) and "not bad" (understands negation). Lesson: if you have labeled data + compute, fine-tune BERT. If data is scarce, LSTM + hand-crafted features often wins. Kwame's 50K reviews + GPU = BERT's territory.
Kwame's next project: classify sentiment from reviews in English, Twi (Ghana's local language), and Pidgin. Movie reviews come in mixed-language comments.
Task: How would you adapt the DistilBERT approach for 3 languages? (Hint: HuggingFace offers XLM-RoBERTa, mBERT, or XLMR pre-trained on 100+ languages.)
Use xlm-roberta-base from HuggingFace instead of DistilBERT. It's pre-trained on 2.5TB text in 100+ languages. Fine-tune on 10K reviews each (30K total) in your 3 languages. Evaluation: test on held-out reviews from each language separately and on code-switched (Twi+English) comments. You'll find cross-lingual transfer helps: a model trained on 30K English reviews can classify Twi comments at 75% accuracy (even without Twi in training, due to shared semantic space learned during pre-training).
Interview prep: questions you'll face
Answer: BERT uses bidirectional transformers. Each token attends to all other tokens simultaneously, capturing long-range dependencies. LSTM is sequential: left-to-right or right-to-left, missing full context until the end. Example: "not bad" — LSTM processes "not" without seeing "bad"; BERT sees both atoms at layer 1. Plus, BERT is pre-trained on 3.3B words, learning general English semantics; LSTM starts from scratch. Fine-tuning BERT leverages transfer learning. Together: 6% gain.
Answer: DistilBERT. For production at 10M scale, latency + cost dominate. DistilBERT: 50ms/review, 66M params, 86.8% accuracy. BERT-base: 70ms, 110M params, 87.1% accuracy. 10M * (70-50)ms = 200K compute hours difference per batch prediction. Cost: $500K+ per year on inference. The 0.3% accuracy gap (maybe 30K false positives) is worth $500K savings. Deploy DistilBERT, monitor false positives, A/B test BERT on 1% traffic for validation.
Answer: Sarcasm is hard because it requires pragmatic understanding. Three approaches: (1) Collect sarcasm-labeled data (500-1K examples). Fine-tune DistilBERT on mixed data (general reviews + sarcasm). Helps ~10%. (2) Use context: Kwame's platform logs user rating + comment. If rating=2 but text="fantastic movie", flag as likely sarcasm. (3) Ensemble: BERT for accuracy + simple keyword rules for known sarcasm markers (ALL CAPS, extra punctuation!!!). In practice: label 1K sarcasm examples, add to training (0.1% of 50K), re-train DistilBERT. Sarcasm accuracy improves from 65% to 75%.
Answer: (1) Collect mistakes: Log all false positives daily (e.g., predicted positive, user feedback negative). 100 mistakes/day. (2) Analysis: Every 2 weeks, label the top 200 mistakes, add to training set. Re-train DistilBERT. (3) Evaluation: A/B test new model on 5% traffic. If accuracy improves, roll out to 100%. (4) Drift detection: Monitor sentiment distribution. If positive rate shifts >5%, retrain. (5) Seasonal updates: Movie reviews vary by genre/season. Quarterly re-training on recent data. Expected improvement: +0.5% accuracy every quarter from data diversity and handling edge cases.
Quick deployment: Save DistilBERT model with model.save_pretrained('path'). Load in production: pipeline('sentiment-analysis', model='path', device=0). Monitor with logs.
Common mistakes: what breaks and how to fix it
Mistake 1: Not padding sequences to same length
LSTM expects fixed-size input. Use nn.utils.rnn.pad_sequence() or truncate to max_length=256. Batch size also breaks if sequences vary wildly. Solution: pad to max in each batch, use pack_padded_sequence to ignore padding in LSTM.
Mistake 2: Using wrong tokenizer for BERT
BERT and DistilBERT have different tokenizers. If you use BertTokenizer with DistilBertModel, vocab mismatch causes [UNK] explosion. Always match: DistilBertTokenizer + DistilBertForSequenceClassification.
Mistake 3: Not calling model.eval() before inference
Training uses dropout and batch norm; inference uses different behavior. Always call model.eval() before testing. Forgetting this inflates test accuracy by 3-5%.
Mistake 4: Over-fitting to small validation set
With 35K training reviews, use stratified k-fold (5 folds, 7K each) instead of single 7.5K val set. Single val set can be unlucky; a bad random split inflates reported accuracy. K-fold gives robust estimate.
Mistake 5: Training BERT with same learning rate as LSTM
BERT fine-tuning uses lr=2e-5 (very small). Using 1e-3 (LSTM default) destroys pre-trained weights. Always use AdamW + warmup schedule for transformers.
Key takeaways
Kwame's journey from LSTM baseline to BERT production model captures the evolution of NLP in practice.
Baseline → complexity: Start with simple LSTM (fast feedback), then add attention, then transformers. Each step costs more (training time, compute) but gains accuracy. Don't start with BERT if you haven't validated the problem with LSTM.
Pre-training is magic: BERT's 6% accuracy gain comes entirely from seeing 3.3B words before your task. Transfer learning is one of deep learning's superpowers. Always look for a pre-trained model in your domain.
Transformers democratize NLP: BERT fine-tuning requires only 2-3 epochs (4-6 GPU hours). A decade ago, this required months of expert feature engineering. Today, you can build state-of-the-art sentiment analysis in a weekend.
Production = trade-off: DistilBERT at 86.8% beats BERT at 87.1% in practice (speed, cost, memory). Know your constraints: 100K users → DistilBERT. 1K users → BERT. 1M users → LSTM or DistilBERT on CPU.
Error analysis beats accuracy metrics: 87.1% average hides sarcasm (65% accuracy). Understand failure modes. Use confidence scores: flag low-confidence predictions for human review.
Continuous improvement: Ship DistilBERT, log mistakes daily, retrain monthly on hard examples. Expect +0.5% improvement per quarter from active learning.