NeoBank (Mumbai): Single tree overfits. Can 100 trees vote better?
Last lesson, your decision tree achieved 71.2% test accuracy but 72.2% trainingโoverfitting. A single tree's path can be noisy. What if you trained 100 trees on random subsets and let them vote? That's Random Forest. Each tree is simple but captures different patterns. Together, they reduce variance and give robust predictions for credit default.
100
Trees voting
0.7139
OOB accuracy
0.4316
credit_score importance
Bagging: bootstrap aggregating
Random Forest uses Bagging: train each tree on a random sample (with replacement), then vote.
Steps:
1. Bootstrap sample ~63% of data (with replacement)
2. Train tree on that sample with random feature subsets
3. Repeat 100 times (n_estimators)
4. Predict: all trees vote โ majority wins
Why? Each tree overfits differently. When voting, noise cancels out. You get variance reduction.
Out-of-Bag (OOB) score: Each tree trains on ~63% of data. The ~37% not used validates without touching test set. Average across trees = OOB score.
Why? Free cross-validation. OOB=0.7139 vs test=0.7290 shows good generalization.
Feature importance: forest vs single tree
Forest redistributes importance. Single tree: credit_score=79%. Forest: 43%. Why? Single tree greedily picks credit_score. Forest's random feature subsets let other features compete.
Random Forest feature importance โ credit_score still dominates, but importance is more distributed than a single tree.
credit_score accounts for 43.2% of all splits across 100 trees. But notice how other features share the remaining 57%โmuch more balanced than a single tree would be. This is bagging's power: it democratizes feature contributions.
More trees = better? n_estimators experiment
Test 10, 50, 100, 200, 500 trees. Watch diminishing returns:
More trees improve OOB score, but returns diminish after ~100 trees.
200 trees is a safe default. Adding more barely helps but increases training time.
Also used in...
Domain Spotlight
๐ณ
Banking: Credit risk
Train forest on applicant features. OOB validates without test leakage.
๐ฅ
Healthcare: Patient outcomes
Predict readmission from clinical features. Feature importance guides interventions.
How AI systems use this
Ensemble methods power recommendation systems, fraud detection, and ranking. Your smartphone's predictive text uses ensembles. Random Forests form the backbone of recommendation engines at Netflix and Spotify.
Set max_features='sqrt' vs max_features=None. How do feature importance and OOB score change?
With max_features='sqrt', each split considers only sqrt(n_features) features, increasing variance across trees and distributing importance more evenly.
Interview questions
Bootstrap Aggregating: train each tree on a random sample (with replacement), then average their predictions via majority vote.
Out-of-Bag score. Each tree is trained on ~63% of data. The remaining ~37% is used as a free validation set. OOB approximates cross-validation accuracy.
A single tree greedily picks credit_score at the root every time. In a forest, random feature subsets force other features (loan_amount, income, employment_years) to appear at the root of some trees, distributing importance.
Diminishing returns. After enough trees, the ensemble's variance is already minimized. More trees increase computation without improving predictions.
AUC is more important for credit scoring because it measures ranking ability across all thresholds. The bank can then choose a threshold based on business costs. Higher AUC (logistic: 0.80 vs RF: 0.79) means better overall discrimination.
Quiz: 5 questions
Q1 of 5
Bagging stands for?
Q2 of 5
OOB score uses what data?
Q3 of 5
credit_score importance in forest vs single tree?
Q4 of 5
500 trees vs 100 trees accuracy?
Q5 of 5
How does the forest make predictions?
0
/5
Great! You understand bagging.
Cheatsheet
Concept
Code
Notes
Train forest
RandomForestClassifier(n_estimators=100)
Default is good
OOB validation
oob_score=True
Free cross-validation
Feature importance
rf.feature_importances_
More stable than single tree
Random features
max_features='sqrt'
Increase diversity
Common mistakes
Too few trees (10, 20)
Results are noisy. Use 100+ for stability.
Unlimited depth in forest
Even in forest, overfits. Set max_depth.
Ignoring OOB
OOB is free validation. Always check it.
Treating RF feature importance as definitive
It's biased toward high-cardinality features. Use multiple methods to confirm.
Key takeaways
Bagging reduces variance: Random Forest trains many trees on random data subsets, then votes. Noise cancels out.
OOB score is free validation: ~37% of data per tree is never seen. Use it instead of keeping a separate validation set.
Feature importance is redistributed: Single trees are greedy. Forests give other features a chance to shine (credit_score: 79% โ 43%).
Diminishing returns after 100 trees: More trees add compute cost without meaningful accuracy gain. 100 is usually the sweet spot.
Still set max_depth: Even with ensemble averaging, unlimited depth overfits. Limit to 10-15 for stability.