⚠️
Real-world crisis
TelcoMax Nairobi: 5,000 customers, 3,858 stay vs 1,142 churn. One model predicts everyone stays.

Kofi is a data scientist at TelcoMax, a telecom company in Nairobi. His churn dataset has a severe imbalance: 77.16% of customers don't churn, only 22.84% do. When he trains a logistic regression baseline, it's mathematically lazy—it predicts "no churn" for almost everyone. Result: F1=0.009, catching only 1 true churner out of 228!

Accuracy looks good (77%), but F1 is useless. To save revenue, Kofi needs every churner caught. This lesson teaches three battle-tested fixes: class weights, SMOTE oversampling, and threshold tuning. By the end, his F1 jumps to 0.443, catching 80% of churners.

3,858 vs 1,142
Class distribution
3.38:1
Imbalance ratio
0.009→0.443
F1 improvement

What is class imbalance and why it breaks models

Most real-world datasets are skewed. Fraud happens in 0.1% of transactions. Rare diseases affect 5% of patients. Churn is usually 15–30%. When one class dominates, a "dumb" model that predicts the majority class looks accurate but is useless.

The imbalance trap: A model that predicts "no churn" for everyone achieves 77% accuracy on TelcoMax data. But it catches ZERO actual churners. Accuracy is a lie; you need precision, recall, and F1 score.
Why accuracy fails: With 3.38:1 imbalance, 77% accuracy is free. You need metrics that penalize missing the minority class: recall (% of true churners caught), precision (% of predicted churners actually correct), and F1 (harmonic mean).

Confusion matrix: the truth behind accuracy

Let's trace the baseline model's failure on test data (1,000 samples, 228 churners):

baseline_model.py Naive logistic regression on imbalanced data
Output
Baseline Confusion Matrix: TN=770, FP=2, FN=227, TP=1 Baseline F1 Score: 0.0087 Recall (% of churners caught): 0.0044

Crisis: 227 of 228 churners are missed! The model's 77% accuracy is misleading. Here's what we fix with the three solutions below.

Solution 1: Class Weights — penalize the majority

Tell the model: "Wrong predictions on churners are worse than on non-churners." Use class_weight='balanced' to weight minority samples higher.

class_weights.py Logistic regression with balanced class weights
Output
Class Weights Confusion Matrix: TN=441, FP=331, FN=69, TP=159 F1 Score: 0.4429 Recall: 0.6974 AUC: 0.6695

Major improvement! F1 jumps from 0.009 to 0.443. We catch 159 of 228 churners (69.7% recall). Trade-off: 331 false positives. Good first step.

Solution 2: SMOTE — oversample the minority class

SMOTE (Synthetic Minority Oversampling Technique) generates synthetic churner samples by interpolating between real churners. This balances the dataset: 3,858 → 3,086, 1,142 → 3,086.

smote_example.py SMOTE oversampling + baseline logistic regression
Output
Before SMOTE: {0: 3086, 1: 914} After SMOTE: {0: 3086, 1: 3086} SMOTE Confusion Matrix: TN=444, FP=328, FN=71, TP=157 F1 Score: 0.4404 Recall: 0.6885 AUC: 0.6680

SMOTE delivers F1=0.440, nearly identical to class weights. Both catch 69–70% of churners. SMOTE is best when you have plenty of train data and want controlled synthetic samples.

Solution 3: Threshold Tuning — optimize for recall

By default, sklearn predicts "churn" if P(churn) > 0.5. Lower the threshold to 0.415 to catch 80% of churners—trading precision for recall.

threshold_tuning.py Adjust decision threshold to maximize recall
Output
Threshold=0.415 Confusion Matrix: TN=305, FP=467, FN=46, TP=182 Precision: 0.2804 Recall: 0.7982 F1 Score: 0.4151

Perfect! At threshold 0.415, we catch 182 of 228 churners (79.8% recall). Cost: 467 false alarms. TelcoMax can now reach out to high-risk customers before they leave.

When to use each method

All three solutions work, but when should you pick each?

Method F1 Score When to use Pros Cons
Class Weights 0.4429 Default choice. Fast, no extra data. Simple, fast, works with most models. Less control over distribution.
SMOTE 0.4404 Lots of train data. Want synthetic samples. Balanced training set, reduces overfitting risk. Creates artificial samples, may leak test info.
Threshold Tuning 0.4151 Care about specific recall target (80%+). No model retraining. Direct control of recall. Low precision. Many false positives.

Pro tip: Combine methods. Use class_weight + threshold tuning for maximum F1 and control.

Precision-Recall curve: see the trade-off

As you lower the threshold, recall increases but precision drops. Here's the curve from threshold 0.1 to 1.0:

Precision vs Recall at different thresholds (Test set, 228 churners)
Precision-recall curve showing threshold trade-offs
At threshold 0.415: precision 0.28, recall 0.80. Sweet spot for catching churners despite false alarms.

Also used in...

Domain Spotlight

🚨
Fraud Detection
99% legitimate, 1% fraud. Class weights or SMOTE catch fraudsters without drowning in false alarms.
🏥
Medical Diagnosis
Rare disease (5% prevalence). Threshold tuning at high recall saves lives—false positives trigger follow-up tests.
🏭
Manufacturing Defects
98% good parts, 2% defective. SMOTE or class weights catch defects before they reach customers.

How AI systems use this

Every real-world AI system faces imbalance: spam is 99% of emails, cancer is rare, fraud rarer. Production models use class weights, SMOTE, or threshold tuning—sometimes all three. LLMs also face class imbalance when training on rare but critical concepts (safety, rare languages). The fix is the same: weight the rare cases higher.

Class Imbalance Metrics Beyond Accuracy Production ML Cost-Sensitive Learning
Challenge: Balance medical data

Task: You have 1,000 patient records: 50 with rare disease, 950 healthy. Train a logistic regression with class_weight='balanced'. Report F1, recall, and precision on test set (20%).

Hint 1: Split with stratify=y to preserve 5% disease rate.

Hint 2: Use LogisticRegression(class_weight='balanced').

Hint 3: Calculate f1_score(y_test, y_pred), recall_score(), precision_score().

Interview questions: test your understanding

Class imbalance means one class dominates. Example: 95% healthy, 5% disease. A "predict healthy" model gets 95% accuracy but catches zero sick people. Accuracy is useless; use F1, recall, precision instead.
With 77% churn vs 23% no-churn, predicting "no churn" for everyone gives 77% accuracy. But you catch zero churners, so F1=0. Accuracy ignores false negatives; F1, recall, precision penalize them.
SMOTE creates synthetic minority samples by interpolating between real ones, balancing the training set so the model doesn't ignore rare cases.
Use class_weight='balanced' for speed and simplicity. Use SMOTE if you have lots of train data and want full balance. Class weights are usually faster and work fine.
Lower the decision threshold below 0.5 (e.g., to 0.415) to increase recall. This catches more churners but generates false alarms. If cost is low (e.g., one email), the trade-off is worth it.

Quiz: test your knowledge

Question 1 of 5
On TelcoMax data (77% no-churn, 23% churn), a baseline model predicts "no churn" for everyone. What is its F1 score?
Question 2 of 5
With class_weight='balanced', what is the F1 score on TelcoMax test data?
Question 3 of 5
SMOTE resamples the training set. What happens to the test set?
Question 4 of 5
At threshold 0.415, TelcoMax catches 182 of 228 churners. What is the recall?
Question 5 of 5
You lower the decision threshold from 0.5 to 0.3. What happens to recall?
0
/5
Excellent! You master imbalanced data.

Cheatsheet: imbalanced data at a glance

Problem / Solution Code Result
Class Weights (balanced) LogisticRegression(class_weight='balanced') F1=0.4429, fast
SMOTE oversampling SMOTE().fit_resample(X_train, y_train) F1=0.4404, full balance
Threshold tuning (80% recall) y_pred = (y_proba >= 0.415).astype(int) Recall=0.798, F1=0.4151
Check imbalance ratio y.value_counts() / len(y) See % of each class
Metrics (not accuracy!) f1_score, recall_score, precision_score Penalize false negatives

Common mistakes: what breaks and how to fix it

Mistake 1: Trusting accuracy on imbalanced data
A "predict majority" model gets 95% accuracy but F1=0.01. Fix: Always check F1, recall, precision, and AUC alongside accuracy.
Mistake 2: Applying SMOTE before train/test split
This leaks test information into synthetic training samples. Fix: Split first, apply SMOTE only to training data.
Mistake 3: Ignoring false positives in threshold tuning
Lowering threshold to 0.1 catches every churner but floods your team with false alarms. Fix: Balance recall vs cost of false positives.
Mistake 4: Not stratifying when splitting imbalanced data
Random split might leave all rare cases in test. Fix: Use stratify=y to preserve class distribution.

Key takeaways

Imbalanced data breaks accuracy. Use F1, recall, precision, AUC instead. Three fixes work:

  • Class Weights (balanced): Easiest. One line of code. F1 = 0.443.
  • SMOTE: Synthetic oversampling. Best with lots of train data. F1 = 0.440.
  • Threshold Tuning: No retraining. Control recall directly. Threshold = 0.415 → 80% recall.
  • Combine methods: Class weights + threshold tuning maximizes F1 and gives you control.
  • Always stratify splits: Preserve class balance in train and test.
  • Check your metrics: Precision, recall, F1, AUC, confusion matrix. Accuracy is a lie on imbalanced data.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Master machine learning the right way

Join the Best AI Program: The AI-Era Analyst — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Analyst →