🏦
Real-world scenario
NeoBank (Mumbai): Explain every loan decision to regulators

NeoBank's risk team in Mumbai uses machine learning to decide which loan applicants will default. But here's the problem: when a regulator asks "Why did you deny this person's ₹50,000 loan?", a logistic regression coefficient doesn't cut it. They need to hear: "This person's credit score is 520, which is below our threshold of 580. Applicants in that range default 65% of the time."

That's where decision trees shine. They make plain-language rules. A tree reads like a flowchart: "If credit_score ≤ 580 AND loan_amount > ₹80,000, then high default risk." Every decision is explainable, auditable, and defensible to regulators.

79%
Credit score importance
Depth 5
Sweet spot (not overfitted)
0.712
Test accuracy at depth 5

Why decision trees? Three reasons

Logistic regression and neural networks are black boxes. Decision trees are transparent.

1. Interpretable. Every path from root to leaf is a human-readable rule. Show it to a regulator, a customer, a business analyst. No coefficients, no math.
2. No scaling needed. Trees split on thresholds, not distances. ₹10,000 and ₹1 million are equally readable. StandardScaler? Not required.
3. Handles mixed data. Numeric, categorical, missing values — trees handle all of it. No LabelEncoder, no one-hot encoding fuss.

The cost: trees tend to overfit. A deep tree memorizes the training data instead of learning patterns. We'll tackle that with pruning.

How splitting works: the tree algorithm

A decision tree grows by asking a series of yes/no questions. At each node, it picks a feature and a threshold, splitting the data to maximize purity (homogeneity of class labels).

Start at the root. All 10,000 applicants. Mix of defaulters (40.5%) and non-defaulters (59.5%). The algorithm asks: "Which single feature and threshold best separates defaulters from non-defaulters?" Answer: credit_score ≤ 580 splits the data the cleanest. Left branch gets high-risk applicants, right gets low-risk.

Then repeat. In the left branch (credit_score ≤ 580), ask again: what's the next best split? Maybe employment_years ≤ 2? Each split reduces impurity (the mixture of classes) until nodes become pure or we hit depth limits.

Gini impurity: measuring node purity

How does the tree know which split is "best"? It measures impurity with a metric called Gini impurity.

gini_formula.py Gini = 1 - sum(p_i²) for each class
Output
Root Gini: 0.4795 Pure node Gini: 0.0000 Balanced node Gini: 0.5000 (most impure)

Gini ranges from 0 (pure: all one class) to 0.5 for binary (impure: 50/50 split). The tree tries to find splits that reduce Gini the most.

Entropy: an alternative to Gini

Entropy is another purity metric. Scikit-learn supports both.

entropy_formula.py Entropy = -sum(p_i * log2(p_i))
Output
Root Entropy: 0.9677 Pure node Entropy: 0.0000 Balanced node Entropy: 1.0000

Both Gini and Entropy work well. Gini is faster (no logarithms). In practice, they give nearly identical results. We'll use Gini (the default) unless specified otherwise.

Training a decision tree with scikit-learn

Let's build a tree on NeoBank's credit data. Start simple: no depth limit. Watch it overfit.

train_tree.py Train a decision tree on credit default data
Output
Unlimited depth: Train accuracy: 1.0000 Test accuracy: 0.6175 Tree depth: 45 Tree leaves: 2468

Perfect training accuracy (1.0) but test accuracy drops to 0.62. The tree has 45 levels and 2,468 leaf nodes — it's memorizing. We need to prune.

Visualizing the tree (pruned to depth 3)

A shallow decision tree (max_depth=3). Each node shows the split condition, Gini impurity, and class distribution.
Decision tree visualization with depth 3
Read left = "True" (condition met), right = "False". Blue nodes lean toward Safe, orange toward Default. The root split on credit_score ≤ 532.5 is the most important decision.

Feature importance: which features matter most?

After training, the tree tells us which features it relied on. Credit score dominates.

feature_importance.py Extract feature importance from the tree
Output
feature importance credit_score 0.7901 loan_amount 0.0720 employment_years 0.0699 debt_to_income 0.0416 income 0.0138 education 0.0055 (other features) 0.0071 Top feature: credit_score (79.0%)

Credit score captures 79% of the tree's decision-making. That makes sense: it's the strongest predictor of default. Loan amount and employment years contribute next.

Feature importance from the decision tree — credit_score dominates, followed by loan_amount.
Feature importance bar chart for decision tree

Overfitting with depth: the critical experiment

Shallow trees underfit (miss patterns). Deep trees overfit (memorize noise). Let's find the sweet spot.

depth_experiment.py Test tree accuracy at different depths
Output
Depth 1: train=0.6985, test=0.6835, gap=0.0150 Depth 3: train=0.7047, test=0.7080, gap=-0.0033 Depth 5: train=0.7220, test=0.7120, gap=0.0100 ← sweet spot Depth 7: train=0.7544, test=0.6895, gap=0.0649 Depth 10: train=0.8329, test=0.6615, gap=0.1714 Depth None: train=1.0000, test=0.6175, gap=0.3825
At depth 5, the test accuracy peaks at 0.712 and the gap between train and test is minimal. Beyond depth 5, the tree starts fitting noise, and test accuracy plummets. Depth 5 is our sweet spot for NeoBank's data.

Notice the pattern:

  • Depth 3: Underfitting. Train and test are close but both are low (0.704 and 0.708).
  • Depth 5: Just right. Test peaks at 0.712. Train and test gap is small (0.01).
  • Depth 7+: Overfitting. Train rises (0.754, 0.833) but test drops (0.690, 0.662). The gap widens.
  • Depth None: Severe overfitting. Train = 1.0 (memorized), test = 0.618 (worse than depth 3).
Train vs test accuracy across depths 1–24. The gap between lines is the overfitting signal.
Train vs test accuracy across tree depths showing overfitting

Pruning: the levers to prevent overfitting

Decision trees have three main pruning levers in scikit-learn. Use them together.

Parameter What it does Example Effect max_depth Limits tree height max_depth=5 Stops growth at 5 levels (our sweet spot for NeoBank) min_samples_split Min samples in node to split min_samples_split=20 Won't split if node has < 20 samples (avoid tiny branches) min_samples_leaf Min samples in leaf node min_samples_leaf=10 Ensures leaves have >= 10 samples (smoother predictions)
prune_tree.py Pruned tree with reasonable stopping criteria
Output
Pruned Decision Tree (depth=5, min_split=20, min_leaf=10): Accuracy: 0.7120 F1 score: 0.6284 AUC: 0.7729 Tree depth: 5 Tree leaves: 32

With pruning: depth 5, 32 leaves. Clean, interpretable, and performant on test data (0.712 accuracy). This is the tree we'd show to a regulator.

Understanding tree structure: from root to leaf

A decision tree's power is that you can trace any prediction. For an applicant with credit_score=520 and loan_amount=₹90,000, the tree would navigate:

Root: "Is credit_score ≤ 580?" → Yes, go left (high-risk)
Left child: "Is employment_years ≤ 2?" → Yes, go left (very high-risk)
Left leaf: "Predict: Default" (65% defaulters in this leaf)

Result: This applicant is flagged as likely to default because they have low credit score (520) AND short employment (1 year). You can show this path to the regulator.

Scikit-learn has a function to visualize trees:

visualize_tree.py Print tree structure with sklearn.tree.plot_tree
Output
Tree saved to tree_visualization.png

For a depth=3 tree, the visualization is compact. At each node, you see: the split condition (feature and threshold), the Gini impurity, the sample count, and the class distribution.

Also used in...

Domain Spotlight

🏦
Banking: Loan decisions must be auditable
Regulators (RBI, SEBI in India; CBK in Kenya) require explainability. Decision trees let banks justify each denial in plain language.
🏥
Healthcare: Clinical decision support
Doctors trust decision trees over neural networks. "If blood_glucose > 200 AND BMI > 30, risk=high" is actionable medical guidance.
📦
E-commerce: Product routing rules
Trees learn rules like: "If order_size > ₹5000, use premium_logistics" and "If customer_age < 25, show_fashion_deals". Easy to deploy and modify.

How AI systems use this

Decision trees are the building blocks of modern machine learning. Random Forest (Lesson 12) is 100 trees voting together. XGBoost (Lesson 13) trains trees sequentially, each correcting the previous one's errors. Gradient Boosting, LightGBM, and CatBoost all build on this foundation. Every tree-based model you'll encounter relies on the splitting, Gini, and pruning concepts from this lesson.

Ensemble Methods Interpretability Feature Selection Gradient Boosting
Challenge: Find the optimal depth for your dataset

Task: Load credit_default.csv. Train decision trees with depths 1, 3, 5, 7, 10. For each, compute test accuracy and the train-test gap. Which depth minimizes overfitting while maximizing test accuracy? Print the results in a table.

Hint 1: Use a loop: for depth in [1, 3, 5, 7, 10]:

Hint 2: For each depth, create a DecisionTreeClassifier and fit on train data

Hint 3: Use accuracy_score(y_test, dt.predict(X_test)) to get test accuracy

Hint 4: Gap = train_accuracy - test_accuracy. A small gap (< 0.05) is good; a large gap (> 0.15) signals overfitting.

Interview questions: test your understanding

Trees are interpretable—you can show regulators exactly why a decision was made. They also handle non-linear relationships and don't need feature scaling. Logistic regression is simpler and faster, but a tree's "If A and B, then C" rule is more human-friendly.
How mixed a node is. Gini = 0 means all samples are one class (pure). Gini = 0.5 means a 50/50 split for binary classification (most impure). The tree tries to find splits that reduce Gini the most.
The tree memorized every training sample, creating a leaf for each unique combination of features. It learned noise, not patterns. Test accuracy is worse than reasonable depths because the model is overfitting severely. This is why we prune with max_depth, min_samples_split, and min_samples_leaf.
Not necessarily. A single tree picks features greedily—credit_score is so strong that it dominates splits. But in a Random Forest, other features get chances to split when credit_score isn't available. Importance can shift with ensemble methods or different random states.
Follow the root-to-leaf path for that applicant's features. E.g., "This applicant has credit_score=520 (≤580, go left), employment_years=1 (≤2, go left), debt_to_income=0.45 (≤0.50, go left) → leaf predicts Default. The model flagged them because of low credit and short employment history, which are associated with 65% default rates in our training data."

Quiz: 5 questions to test your knowledge

Question 1 of 5
What does Gini impurity = 0 mean?
Question 2 of 5
Which feature splits at the root of NeoBank's tree?
Question 3 of 5
Depth=None (unlimited): train accuracy=1.0, test accuracy=0.62. What happened?
Question 4 of 5
Do decision trees need feature scaling (StandardScaler)?
Question 5 of 5
What is the optimal depth for NeoBank's credit scoring tree?
0
/5
Excellent! You understand decision trees.

Cheatsheet: decision trees at a glance

Concept What it does Code
Gini impurity Measures node purity criterion='gini'
Entropy Alternative purity measure criterion='entropy'
max_depth Limits tree height max_depth=5
min_samples_split Min samples to allow split min_samples_split=20
min_samples_leaf Min samples per leaf min_samples_leaf=10
feature_importances_ Which features matter most dt.feature_importances_
get_depth() Check final tree height dt.get_depth()

Common mistakes: what breaks and how to fix it

Mistake 1: Not limiting depth — trees grow unlimited
Without max_depth, trees memorize training data (train=1.0, test=0.62). Fix: set max_depth=5 and monitor the train-test gap. Stop when gap exceeds 0.05.
Mistake 2: Using Gini vs Entropy debate as excuse for slow tuning
Both give nearly identical results (0.7729 vs 0.7725 AUC). Pick one and move on. Don't waste time comparing them — spend time on pruning parameters.
Mistake 3: Trusting single-tree feature importance as ground truth
Trees are greedy — credit_score dominates because it's strong, not because others are weak. In Random Forest, importance is more distributed. Use multiple methods (permutation importance, SHAP) to confirm.
Mistake 4: Ignoring class imbalance in the tree
With 40% default, 60% non-default, trees can be biased. Fix: set class_weight='balanced' to penalize minority class errors more heavily.

Key takeaways

Decision trees are your window into machine learning. Everything that follows—Random Forests, XGBoost, Gradient Boosting—is built on these foundations.

  • Interpretability matters. A tree's rule "If credit_score ≤ 580 AND loan_amount > ₹80,000, then default" is worth its weight in regulatory compliance.
  • Gini impurity guides splits. Lower Gini = purer node. The tree recursively finds splits that minimize Gini the most.
  • Depth controls overfitting. Deep trees memorize. Shallow trees miss patterns. Find the sweet spot with train-test comparison.
  • Prune aggressively. Use max_depth, min_samples_split, and min_samples_leaf together. A smaller tree is almost always better than a deep one.
  • Feature importance is a hint, not gospel. One tree is biased. Use ensembles to confirm which features truly matter.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Ready to build explainable AI?

Join the Best AI Program: The AI-Era Analyst — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Analyst →