💳
Real-world scenario
NovaPay (Nairobi fintech): 10,000 transactions, 5.21% fraud, ₹50K per miss

Kamau works at NovaPay, a fintech company in Nairobi. The company processes 10,000 daily transactions, but only 5.21% are fraudulent. That's a brutal 18.19:1 imbalance. Most classifiers will cheat: predict "not fraud" for everything and get 94.79% accuracy—while missing all the fraud.

Worse: every missed fraud costs NovaPay ₹50,000 in liability. But false alerts annoy customers. Kamau must balance precision and recall, build features that expose fraud patterns, and tune the decision threshold to catch 90% of fraud without drowning in false alarms.

10,000
Transactions
5.21%
Fraud Rate
18.19:1
Class Imbalance

Step 1: Load and diagnose imbalance

We load 10,000 transactions and spot the fraud distribution. Imbalance is the core challenge—only 521 frauds vs 9,479 legitimate.

load_eda.py Check imbalance ratio and fraud patterns
Output
Shape: (10000, 8) Fraud distribution: 0 9479 1 521 dtype: int64 Fraud rate: 0.0521 Imbalance ratio: 18.19:1 Fraud by channel: online 0.0655 offline 0.0321

Online fraud is 6.55% vs offline 3.21%. This guides feature engineering.

Step 2: Engineer 3 predictive features

Raw fields have weak signals. We engineer is_night, amount_log, and high_amount to expose fraud patterns.

feature_engineering.py Engineer features from raw data
Output
Engineered shape: (10000, 11)

Amount is the strongest feature (importance 0.3199). Log-amount (0.2422) is second.

Step 3: Baseline—Logistic Regression

LogisticRegression without balancing catches only 16 of 104 frauds. It avoids false positives at the cost of missing real fraud.

baseline_lr.py Train baseline logistic regression
Output
Baseline LR: F1=0.2645, AUC=0.7853 TP=16, FP=1, TN=1895, FN=88

Only 16 frauds caught. We need stronger models and threshold tuning.

Step 4: Random Forest with balanced weights

RandomForest with class_weight='balanced' improves but still misses fraud. F1=0.2276, AUC=0.7530.

balanced_rf.py Random Forest with balanced classes
Output
RF Balanced: F1=0.2276, AUC=0.7530 TP=14, FP=5, TN=1891, FN=90

Step 5: Gradient Boosting—best AUC (0.7735)

GBM detects fraud patterns better. AUC=0.7735, F1=0.3008, catching 20 of 104 frauds at threshold 0.5.

gbm_model.py Gradient Boosting Classifier
Output
GBM: F1=0.3008, AUC=0.7735 TP=20, FP=9, TN=1887, FN=84 amount 0.3199 amount_log 0.2422 num_transactions_24h 0.1608 distance_from_home 0.1160 customer_age 0.0822

Step 6: Threshold tuning for 90% recall

We need to catch 90% of fraud. Lowering threshold to 0.020 flags 94 of 104 frauds, but with 1264 false alerts.

threshold_tuning.py Optimize threshold for recall=0.90
Output
Threshold=0.0200 TP=94, FP=1264, TN=632, FN=10 Recall=0.9038, Precision=0.0692

Step 7: Cost-benefit—why 90% recall makes sense

At threshold 0.020: 10 missed frauds × ₹50K = ₹5L loss. But 1264 false alerts annoy customers. It's a business tradeoff.

Cost of 1 missed fraud: ₹50,000
At recall=90%: 10 missed × ₹50K = ₹5,00,000 loss
False alerts: 1264 (customer friction)
At recall=85%: 16 missed × ₹50K = ₹8,00,000 loss (worse)
Optimal zone: 85-95% recall balances liability vs user experience

Business stakeholders must decide: is ₹5L loss acceptable to avoid annoying 1264 users? Usually yes for fintech.

Also used in...

Domain Spotlight

🏦
Banking: AML & Know Your Customer
Detect suspicious deposits, threshold tuning for regulatory thresholds, cost-benefit on compliance fines vs customer friction.
🛒
E-commerce: Return fraud detection
Catch customers abusing return policies. Imbalance: 2% of returns are fraudulent. Tune recall to protect margins.
🏥
Insurance: Claims fraud
Detect fraudulent medical claims. Imbalanced: 1-5% fraud. Threshold tuning balances payout accuracy vs claim processing delays.

How AI systems handle imbalance

LLMs train on imbalanced data too—rare languages, edge cases, uncommon tokens. They use techniques like focal loss, oversampling, and threshold tuning just like fraud detectors. Even GPT uses cost-aware loss functions to prioritize rare patterns.

Imbalanced Data Focal Loss Threshold Optimization Cost-Sensitive Learning
🎯 Mini Challenge: Optimize for precision instead

Task: Create a second threshold that maximizes precision (minimize false alerts). What recall do you get? Why might NovaPay use this for low-value transactions?

Hint: Use f1_scores = 2 * precisions * recalls / (precisions + recalls) to maximize F1, then pick a higher threshold. Higher threshold = fewer alerts, higher precision.

Interview questions: test your understanding

A model that predicts "not fraud" for everything gets 94.79% accuracy (9479/10000 correct), but catches zero frauds. Accuracy ignores class imbalance. Recall and F1 matter more.
Recall = of all real frauds, how many did we catch? Precision = of all flagged frauds, how many were real? Recall 0.90 = catch 90% of fraud. Precision 0.07 = 7% of alerts are true fraud.
The model learns to output probabilities (0-1). Threshold is a business decision, not a model parameter. Change thresholds without retraining to adapt to new cost structures, regulatory changes, or user tolerance.
GBM iteratively corrects misclassified examples, focusing on hard cases. For imbalanced fraud detection, this sequential correction beats RF's averaging. GBM achieved AUC 0.7735 vs RF's 0.7530.
Threshold 0.01 catches even more fraud (99%+ recall) but flags almost every transaction. You get extreme false positives (customer support overload). There's no perfect threshold—it's a precision-recall tradeoff set by business priorities.

Quiz: 5 questions to test your knowledge

Question 1 of 5
What is the imbalance ratio of NovaPay's fraud data?
Question 2 of 5
If a model predicts "not fraud" for every transaction, what's its accuracy on this dataset?
Question 3 of 5
Which model achieved the highest AUC?
Question 4 of 5
At threshold 0.020, how many frauds does GBM catch?
Question 5 of 5
What was the most important feature for GBM?
0
/5
Well done! You understand imbalanced classification.

Cheatsheet: Fraud detection at a glance

Step Method Key Metric
Diagnose imbalance y.value_counts() Ratio & fraud %
Engineer features Time-of-day, log-amount, high-value flags Feature importance
Baseline model LogisticRegression (check accuracy trap) Recall, not accuracy
Balanced models GradientBoosting best for imbalance AUC-ROC > 0.75
Threshold tune precision_recall_curve() Recall = target (90%)
Cost-benefit Cost per miss × FN + cost per alert × FP Optimal threshold

Common mistakes: what breaks and how to fix it

Mistake 1: Optimizing for accuracy on imbalanced data
A model that predicts "not fraud" gets 94.79% accuracy but catches zero frauds. Fix: Use recall, F1, or AUC instead. Accuracy is misleading.
Mistake 2: Using threshold 0.5 for imbalanced data
Default threshold 0.5 is arbitrary. For fraud, you likely need threshold 0.01-0.05 to catch enough fraud. Fix: Use precision-recall curves to find your target recall.
Mistake 3: Not tuning threshold after training
Threshold is a business decision, not a model parameter. Regulatory changes or cost structures require new thresholds. Fix: train once, tune thresholds on validation set, deploy with new threshold.
Mistake 4: Ignoring false positives
90% recall with 0.07 precision = 1264 false alerts. Customers get annoyed. Fix: Run cost-benefit analysis with business stakeholders to find acceptable precision-recall tradeoff.

Key takeaways

  • Imbalance is the main challenge. With 18.19:1 ratio, accuracy is a trap. Recall and F1 matter.
  • Feature engineering is critical. Amount, log-amount, and transaction patterns expose fraud better than raw fields.
  • Gradient Boosting beats simpler models. GBM achieved AUC 0.7735 vs LogisticRegression 0.7853 vs RandomForest 0.7530.
  • Threshold tuning is post-training. Adjust threshold to hit target recall (90%) without retraining.
  • Cost-benefit drives decisions. 10 missed frauds cost ₹5L. 1264 false alerts annoy users. Balance defines the threshold.
  • Real-world fintech requires speed + accuracy. Batch detection for historical data, real-time models for live transactions.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Master fraud detection and AI systems

Join the Best AI Program: The AI-Era Analyst — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Analyst →