🏦
Real-world scenario
NeoBank CEO: "Which model should we deploy for credit scoring?"

You're a data scientist at NeoBank, a digital bank in Mumbai. The risk committee has trained 5 different classification models on 10,000 loan applicants. Each model works—but which one should the bank deploy?

Logistic Regression is fast and interpretable. Random Forest and XGBoost are "fancier"—do they earn their complexity? Decision Trees are transparent to regulators. SVM captures non-linear patterns. Today, you'll build all five, compare them side-by-side, and recommend one to the board. The answer isn't always "the most accurate."

10,000
Applicants in dataset
40.5%
Default rate
5
Models to compare

The business brief: which model should NeoBank deploy?

Before we build, let's set the decision criteria. The CEO asks three questions:

1. How accurate is each model? — Accuracy, precision, recall, F1.
2. How well does it rank applicants? — AUC (does it assign higher default probability to actual defaulters?).
3. Can we explain it? — Logistic Regression coefficients vs XGBoost's black box.
4. Is it fast enough? — Tree-based models are slower than logistic regression at inference.
5. Does it handle production updates? — How often do we retrain? What if defaults shift?

Phase 1: Exploratory data analysis

First, we explore the NeoBank credit dataset. 10,000 applicants, 11 columns. Let's spot the patterns that models will learn.

eda.py Load and explore NeoBank data
Output
Shape: (10000, 11) Data types: credit_score int64 age int64 employment_years int64 loan_amount float64 income float64 debt_to_income float64 education object home_ownership object default int64 dtype: int64 Default distribution: Rate: 0.405 (40.5%) 4052 defaults, 5948 non-defaults

Key EDA findings

Before modeling, we identify what features matter:

eda_correlations.py Feature importance by correlation
Output
Correlation with default: income: 0.007 debt_to_income: 0.101 loan_amount: 0.103 employment_years: -0.133 age: -0.287 credit_score: -0.441 ← strongest Default by education: High School: 45.1% Bachelor: 39.0% Master: 38.9% PhD: 35.8% Default by home_ownership: Rent: 44.0% Mortgage: 39.8% Own: 35.8%
Insight: Credit score dominates. Each std-dev increase in credit score cuts default risk by 44%. Education and home ownership matter but are secondary. This explains why Logistic Regression performs well—the relationship is largely linear.

EDA visualizations

Figure 1: Feature correlation with default (strongest predictors first)
Feature correlation with default: credit_score has -0.441 correlation, age -0.287, employment_years -0.133, loan_amount 0.103, debt_to_income 0.101, income 0.007
Figure 2: Default rate by education and home ownership
Default rate by education: High School 45.1%, Bachelor 39.0%, Master 38.9%, PhD 35.8%. Default rate by home ownership: Rent 44.0%, Mortgage 39.8%, Own 35.8%

Phase 2: Preprocessing and feature engineering

We prepare data for modeling: encode categoricals, scale for distance-based models, split with stratification.

preprocess.py Encode, scale, and split
Output
Train shape: (8000, 8), Test shape: (2000, 8) Train default rate: 0.405 Test default rate: 0.404

Phase 3: Train all 5 models

Now we train each model on the same train/test split. This ensures fair comparison.

Model 1: Logistic Regression

model_lr.py Logistic Regression
Output
Logistic Regression: Accuracy: 0.7370 Precision: 0.6868 Recall: 0.6444 F1: 0.6650 AUC: 0.8029 ← WINNER

Model 2: Decision Tree

model_dt.py Decision Tree (max_depth=5)
Output
Decision Tree (max_depth=5): Accuracy: 0.7120 Precision: 0.6581 Recall: 0.6012 F1: 0.6284 AUC: 0.7729

Model 3: Random Forest

model_rf.py Random Forest (100 trees)
Output
Random Forest (100 trees): Accuracy: 0.7290 Precision: 0.6791 Recall: 0.6272 F1: 0.6521 AUC: 0.7879

Model 4: XGBoost

model_xgb.py XGBoost (100 rounds, lr=0.1)
Output
XGBoost (100 rounds): Accuracy: 0.7395 Precision: 0.6934 Recall: 0.6395 F1: 0.6654 AUC: 0.7988 ← close second

Model 5: Support Vector Machine

model_svm.py SVM with RBF kernel
Output
SVM (RBF kernel): Accuracy: 0.7360 Precision: 0.6937 Recall: 0.6235 F1: 0.6567 AUC: 0.7841

Phase 4: Model comparison

Side-by-side metrics for all 5 models. Higher is better for all metrics.

Model Accuracy Precision Recall F1 Score AUC
Logistic Regression 0.7370 0.6868 0.6444 0.6650 0.8029
Decision Tree (d=5) 0.7120 0.6581 0.6012 0.6284 0.7729
Random Forest 0.7290 0.6791 0.6272 0.6521 0.7879
XGBoost 0.7395 0.6934 0.6395 0.6654 0.7988
SVM (RBF) 0.7360 0.6937 0.6235 0.6567 0.7841
The plot twist: XGBoost has highest accuracy (73.95%), but Logistic Regression wins on AUC (0.8029 vs 0.7988). For credit scoring, AUC matters more—it measures ranking ability. The bank can choose a threshold based on risk appetite, and higher AUC means better discrimination across all thresholds.

Phase 5: ROC curve comparison

The ROC curve shows true positive rate (recall) vs false positive rate. Higher and to the left = better ranking. AUC is the area under the curve.

Figure 3: ROC curves for all 5 models (higher is better)
ROC curves for 5 credit scoring models: Logistic Regression AUC 0.8029, XGBoost AUC 0.7988, Random Forest AUC 0.7879, SVM AUC 0.7841, Decision Tree AUC 0.7729
Reading the ROC: Logistic Regression (purple) and XGBoost (blue) curves hug the top-left corner, meaning they catch more true defaulters while flagging fewer false positives. The gap between them is tiny—both are excellent at ranking. Decision Tree (gray) lags behind because a single tree has higher variance. The diagonal line (random guessing) shows AUC=0.5.

Phase 6: Business recommendation for the NeoBank CEO

Now we translate metrics into a business decision. Which model should NeoBank deploy?

The case for Logistic Regression

Recommendation: Deploy Logistic Regression

  • Highest AUC (0.8029): Best at ranking applicants by default risk. The bank can set any threshold and trust the ranking.
  • Interpretability: Coefficients show exactly which features matter. Credit score has coefficient -1.11, meaning each std-dev increase cuts default log-odds by 1.11. Regulators love this.
  • Fast inference: Predicting 10,000 applicants takes milliseconds. XGBoost is slower.
  • Stable over time: Linear models are robust to small data shifts. XGBoost can overfit if defaults change.
  • Easier maintenance: No hyperparameter tuning needed. No tree pruning. Just retrain on new data.

Why not XGBoost?

XGBoost has higher accuracy (73.95% vs 73.70%) but lower AUC (0.7988 vs 0.8029). Why does this happen? Because the data is mostly linear. Credit score dominates (-0.441 correlation). XGBoost's complexity doesn't help—it's overfitting the noise.

Why not Decision Trees?

Single trees overfit (AUC=0.7729). Although they're interpretable, Random Forest (AUC=0.7879) is a better ensemble. But LR still wins.

The golden rule: Pick the simplest model that solves the problem. If Logistic Regression works, use it. Only escalate to forests or boosting if simpler models fail.

What's next: improving the model

After deployment, the real work begins. Here are ways to improve:

  • Threshold tuning: The default threshold 0.5 might not be optimal. If false positives (denying good customers) cost more than false negatives, lower the threshold to 0.3. Recompute precision and recall.
  • Cost-sensitive learning: Assign weights to classes. FN (approving a defaulter) costs ₹100,000; FP (denying a good customer) costs ₹1,000. Train with these costs.
  • Feature engineering: Create interaction features (credit_score × loan_amount). Test polynomial features. Use domain expertise.
  • Class rebalancing: 40.5% default rate is imbalanced. Try SMOTE or class weights.
  • Retraining schedule: Defaults drift over time (economy, interest rates). Retrain quarterly with new data.
  • A/B testing: Deploy LR as the champion. Test XGBoost or RF as challenger models. Measure AUC and business metrics (approval rate, default rate).

How real banks do this

At HDFC, ICICI, SouthBank India: model comparison is quarterly. They maintain champion/challenger frameworks where the production model (champion) is tested against new candidates (challengers). If a challenger has significantly better AUC (usually >1% improvement) and doesn't break regulators' interpretability rules, they promote it. This cycle repeats for years. The production model in 2024 might not be the same as 2020—it evolves with data.

Model Selection Champion/Challenger A/B Testing Regulatory Compliance
🎯 Mini Challenge: Retrain with a different threshold

Task: Using Logistic Regression predictions, compute precision and recall at thresholds 0.3, 0.5, and 0.7. At which threshold does recall reach 0.70 (catch 70% of defaulters)? What's the precision at that threshold?

Hint: Use y_pred_proba_lr > threshold to create binary predictions at custom thresholds.

Hint: Recall = TP / (TP + FN). Higher threshold → lower recall, higher precision.

Hint: Loop through thresholds [0.3, 0.5, 0.7] and compute metrics for each.

Interview questions: test your understanding

Different models have different trade-offs. Logistic Regression is interpretable but can't capture non-linearity. XGBoost captures non-linearity but is a black box. Comparing ensures you pick the best fit for both the data AND the business constraints (speed, interpretability, regulatory requirements).
Accuracy doesn't distinguish between types of errors. With 40% default rate, a model that always predicts "default" gets 40% accuracy—useless. AUC measures ranking ability: does the model assign higher probability to actual defaulters? AUC=0.8 means 80% chance a random defaulter gets higher score than a non-defaulter. This is what matters for thresholding.
The relationship between features and default is mostly linear (credit_score dominates, r=-0.441). XGBoost's complexity doesn't help capture non-linearity that doesn't exist. Instead, XGBoost overfits the noise, slightly hurting ranking ability. This is a reminder: simpler models often win on real-world data.
With Logistic Regression: "Your credit score is 520. Our model learned that a 520 score means 68% default probability (our threshold is 50%). We therefore predicted default and rejected you. To improve, raise your credit score or reduce your debt-to-income ratio." With XGBoost: "Our ensemble model predicts 68% default probability. Here are the features that contributed most, but we can't explain the exact reasoning." Regulators prefer the first answer.
No. The model was trained on 40% base rate. In a 5% market: (1) Logistic Regression's coefficients remain valid—the relative importances don't change. But (2) the threshold 0.5 is wrong. With 5% base rate, you'd need to lower the threshold to 0.1 or lower to catch the same number of defaulters. (3) Precision and recall metrics will shift dramatically. At minimum: recompute metrics on new market data, adjust threshold, and run a validation test. Ideally: retrain on local data.

Quiz: 5 questions to test your knowledge

Question 1 of 5
Which model had the highest AUC?
Question 2 of 5
What is the correlation between credit_score and default?
Question 3 of 5
Why does XGBoost (highest accuracy) lose to Logistic Regression (highest AUC)?
Question 4 of 5
What is the default rate for applicants with high school education?
Question 5 of 5
For credit scoring, why might you prefer Logistic Regression over a more complex model?
0
/5
Excellent! You understand model selection.

Cheatsheet: model comparison at a glance

Model Best For Pros Cons
Logistic Regression Linear relationships, interpretability Fast, interpretable, calibrated probabilities, high AUC on linear data Can't capture non-linearity
Decision Tree Explainability to stakeholders Visual rules, no scaling needed, handles non-linearity Overfits easily, high variance
Random Forest Robust predictions, feature importance Low variance, OOB validation, non-linear Less interpretable, slower than LR
XGBoost Maximum performance (when data is non-linear) Handles missing values, regularized, state-of-the-art Black box, slower, overfits if not tuned
SVM Small datasets, non-linear boundaries Kernel trick, memory efficient Slow on large data, no native probabilities, no feature importance

Common mistakes in model comparison

Mistake 1: Picking the model with highest accuracy
Accuracy is misleading with imbalanced data. Use AUC or F1. A model that always predicts "no default" gets 60% accuracy but AUC=0.5 (useless).
Mistake 2: Not considering deployment constraints
XGBoost is harder to deploy and explain than logistic regression. Regulatory approval takes weeks. Simplicity matters.
Mistake 3: Comparing models trained on different data
Always use the same train/test split for fair comparison. Otherwise, randomness clouds the results.
Mistake 4: Ignoring the business cost matrix
FP (denying a good customer) and FN (approving a defaulter) have different costs. Pick the model that minimizes expected cost, not accuracy.

Key takeaways

Model comparison is about trade-offs, not absolutes.

  • Use AUC, not accuracy. AUC measures ranking ability. Accuracy is misleading with imbalanced data.
  • Simpler models often win. If logistic regression has high AUC, use it. Don't escalate to forests or boosting without reason.
  • Interpretability matters for regulated industries. Banks need to explain credit decisions. XGBoost is a black box.
  • Speed matters in production. Logistic Regression predicts 10,000 applicants in milliseconds. XGBoost takes seconds.
  • Different models excel at different things. LR: interpretability + calibrated probs. RF: non-linear + feature importance. XGB: max performance. SVM: high-dimensional data.
  • Always compare on the same train/test split. Use stratified splitting to preserve class balance.
  • Maintain champion/challenger frameworks. Deploy the champion. Test challengers quarterly. Promote if significantly better.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Ready to deploy ML models?

Join the Best AI Program: The AI-Era Analyst — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Analyst →