Build a fraud detection system for NovaPay (Nairobi fintech) with 10,000 transactions. Master imbalanced data, feature engineering, and threshold tuning to catch 90% of fraud.
NovaPay (Nairobi fintech): 10,000 transactions, 5.21% fraud, ₹50K per miss
Kamau works at NovaPay, a fintech company in Nairobi. The company processes 10,000 daily transactions, but only 5.21% are fraudulent. That's a brutal 18.19:1 imbalance. Most classifiers will cheat: predict "not fraud" for everything and get 94.79% accuracy—while missing all the fraud.
Worse: every missed fraud costs NovaPay ₹50,000 in liability. But false alerts annoy customers. Kamau must balance precision and recall, build features that expose fraud patterns, and tune the decision threshold to catch 90% of fraud without drowning in false alarms.
10,000
Transactions
5.21%
Fraud Rate
18.19:1
Class Imbalance
Step 1: Load and diagnose imbalance
We load 10,000 transactions and spot the fraud distribution. Imbalance is the core challenge—only 521 frauds vs 9,479 legitimate.
load_eda.pyCheck imbalance ratio and fraud patterns
At threshold 0.020: 10 missed frauds × ₹50K = ₹5L loss. But 1264 false alerts annoy customers. It's a business tradeoff.
Cost of 1 missed fraud: ₹50,000 At recall=90%: 10 missed × ₹50K = ₹5,00,000 loss False alerts: 1264 (customer friction) At recall=85%: 16 missed × ₹50K = ₹8,00,000 loss (worse) Optimal zone: 85-95% recall balances liability vs user experience
Business stakeholders must decide: is ₹5L loss acceptable to avoid annoying 1264 users? Usually yes for fintech.
Also used in...
Domain Spotlight
🏦
Banking: AML & Know Your Customer
Detect suspicious deposits, threshold tuning for regulatory thresholds, cost-benefit on compliance fines vs customer friction.
🛒
E-commerce: Return fraud detection
Catch customers abusing return policies. Imbalance: 2% of returns are fraudulent. Tune recall to protect margins.
🏥
Insurance: Claims fraud
Detect fraudulent medical claims. Imbalanced: 1-5% fraud. Threshold tuning balances payout accuracy vs claim processing delays.
How AI systems handle imbalance
LLMs train on imbalanced data too—rare languages, edge cases, uncommon tokens. They use techniques like focal loss, oversampling, and threshold tuning just like fraud detectors. Even GPT uses cost-aware loss functions to prioritize rare patterns.
Task: Create a second threshold that maximizes precision (minimize false alerts). What recall do you get? Why might NovaPay use this for low-value transactions?
Hint: Use f1_scores = 2 * precisions * recalls / (precisions + recalls) to maximize F1, then pick a higher threshold. Higher threshold = fewer alerts, higher precision.
Interview questions: test your understanding
A model that predicts "not fraud" for everything gets 94.79% accuracy (9479/10000 correct), but catches zero frauds. Accuracy ignores class imbalance. Recall and F1 matter more.
Recall = of all real frauds, how many did we catch? Precision = of all flagged frauds, how many were real? Recall 0.90 = catch 90% of fraud. Precision 0.07 = 7% of alerts are true fraud.
The model learns to output probabilities (0-1). Threshold is a business decision, not a model parameter. Change thresholds without retraining to adapt to new cost structures, regulatory changes, or user tolerance.
GBM iteratively corrects misclassified examples, focusing on hard cases. For imbalanced fraud detection, this sequential correction beats RF's averaging. GBM achieved AUC 0.7735 vs RF's 0.7530.
Threshold 0.01 catches even more fraud (99%+ recall) but flags almost every transaction. You get extreme false positives (customer support overload). There's no perfect threshold—it's a precision-recall tradeoff set by business priorities.
Quiz: 5 questions to test your knowledge
Question 1 of 5
What is the imbalance ratio of NovaPay's fraud data?
Question 2 of 5
If a model predicts "not fraud" for every transaction, what's its accuracy on this dataset?
Question 3 of 5
Which model achieved the highest AUC?
Question 4 of 5
At threshold 0.020, how many frauds does GBM catch?
Question 5 of 5
What was the most important feature for GBM?
0
/5
Well done! You understand imbalanced classification.
Cheatsheet: Fraud detection at a glance
Step
Method
Key Metric
Diagnose imbalance
y.value_counts()
Ratio & fraud %
Engineer features
Time-of-day, log-amount, high-value flags
Feature importance
Baseline model
LogisticRegression (check accuracy trap)
Recall, not accuracy
Balanced models
GradientBoosting best for imbalance
AUC-ROC > 0.75
Threshold tune
precision_recall_curve()
Recall = target (90%)
Cost-benefit
Cost per miss × FN + cost per alert × FP
Optimal threshold
Common mistakes: what breaks and how to fix it
Mistake 1: Optimizing for accuracy on imbalanced data
A model that predicts "not fraud" gets 94.79% accuracy but catches zero frauds. Fix: Use recall, F1, or AUC instead. Accuracy is misleading.
Mistake 2: Using threshold 0.5 for imbalanced data
Default threshold 0.5 is arbitrary. For fraud, you likely need threshold 0.01-0.05 to catch enough fraud. Fix: Use precision-recall curves to find your target recall.
Mistake 3: Not tuning threshold after training
Threshold is a business decision, not a model parameter. Regulatory changes or cost structures require new thresholds. Fix: train once, tune thresholds on validation set, deploy with new threshold.
Mistake 4: Ignoring false positives
90% recall with 0.07 precision = 1264 false alerts. Customers get annoyed. Fix: Run cost-benefit analysis with business stakeholders to find acceptable precision-recall tradeoff.
Key takeaways
Imbalance is the main challenge. With 18.19:1 ratio, accuracy is a trap. Recall and F1 matter.
Feature engineering is critical. Amount, log-amount, and transaction patterns expose fraud better than raw fields.
Gradient Boosting beats simpler models. GBM achieved AUC 0.7735 vs LogisticRegression 0.7853 vs RandomForest 0.7530.
Threshold tuning is post-training. Adjust threshold to hit target recall (90%) without retraining.