🏥
Real-world scenario
HealthPulse (Hyderabad): Breast cancer classifier needs tuning

HealthPulse is a diagnostic lab in Hyderabad. Their Random Forest cancer classifier works—F1=0.963. But in healthcare, every 0.1% matters. They can't just use default parameters. Two doctors ask: what settings optimize F1 the most? GridSearch will test 36 combinations. RandomizedSearch will sample 30. Learning curves will show if the model overfits.

By the end, HealthPulse finds: GridSearch yields F1=0.9685 with max_depth=7 and 50 trees. RandomizedSearch finds F1=0.9667 with max_depth=14 and 253 trees. Learning curves reveal slight overfitting (train~0.995, val~0.960)—but acceptable for deployment.

36
Grid combinations
30
Random samples
569
Patient records

What are hyperparameters vs parameters?

Confusion here costs time. Know the difference.

Parameters: Learned by the model during training. Example: regression coefficients (β₀, β₁) in y = β₀ + β₁x. You don't set these—sklearn learns them.

Hyperparameters: You set before training. The knobs you turn. Example: RandomForest's max_depth (tree height), n_estimators (tree count), min_samples_split (rows before split).

In sklearn, hyperparameters are arguments to the model constructor: RandomForestClassifier(max_depth=7, n_estimators=50, ...). Parameters live inside the trained model object .coef_, .feature_importances_, etc.

GridSearchCV vs RandomizedSearchCV: when to use what

Both search for the best hyperparameters. The strategy differs.

Aspect GridSearchCV RandomizedSearchCV
Strategy Tests every combination Samples random combinations
Time complexity Slow (3 params × 3 values = 27 combos) Fast (sample 10 of those 27)
Best for Small search space (< 100 combos) Large space (100+ combos)
Risk Thorough but expensive Might miss the global best, but typically finds good local best fast
Example max_depth: [3,7,11], n_est: [10,50,100] max_depth: randint(1,30), n_est: randint(10,300)

GridSearchCV: exhaustive grid search

HealthPulse starts with GridSearch. They define a grid of hyperparameters to test.

grid_search.py GridSearchCV on HealthPulse data
Output
Best F1: 0.9685 Best params: {'max_depth': 7, 'min_samples_split': 5, 'n_estimators': 50} Test F1: 0.9583
Grid found max_depth=7 and n_estimators=50 as the sweet spot. Deeper trees and more trees didn't help—overfitting risk rose.

RandomizedSearchCV: fast random sampling

HealthPulse's data scientist wants to explore a wider space faster. Enter RandomizedSearchCV.

random_search.py RandomizedSearchCV on HealthPulse data
Output
Best F1: 0.9667 Best params: {'max_depth': 14, 'min_samples_leaf': 5, 'min_samples_split': 2, 'n_estimators': 253} Test F1: 0.9655
RandomSearch found a different optimum: deeper trees (max_depth=14) and many more trees (253). F1 is slightly lower (0.9667 vs 0.9685), but exploration was 5x faster.

Learning curves: diagnose overfit and underfit

Good scores don't guarantee good generalization. Plot learning curves to see train vs validation performance.

learning_curves.py Detect overfitting with learning curves
Output
Training set sizes: [ 40 80 120 160 200 240 280 320 360 400] Train F1 mean: 0.9951 Val F1 mean: 0.9598 Gap (train - val): 0.0353 ⚠️ Slight overfitting detected, but acceptable in healthcare.
Learning curve for GBM: train vs validation F1 as training set size increases
Learning curve showing train F1 (rising to ~0.995) and validation F1 (rising to ~0.96) with increasing training set size

The gap of 0.0353 is small—train and validation curves are close. In healthcare, a 96% validation F1 is strong, and the slight gap is acceptable. If the gap were > 0.1, we'd reduce max_depth or add regularization.

Validation curves: tune one parameter at a time

Sometimes you want to understand one parameter's effect. Validation curves plot train and test F1 as you vary a single hyperparameter.

validation_curve.py See how max_depth affects performance
Output
Optimal max_depth: 7 Train F1 at optimal: 0.9912 Val F1 at optimal: 0.9651
Validation curve for Random Forest: train vs validation F1 as max_depth varies from 1 to 20
Validation curve showing train F1 (high, near 0.99) and validation F1 (peaking around depth 7 at 0.965) across different max_depth values

This confirms max_depth=7 is the best choice. Beyond depth 7, validation F1 plateaus or drops—overfitting.

Interpreting GridSearch results: the heatmap

GridSearchCV stores all results in .cv_results_. Let's visualize them.

visualize_grid.py Create a heatmap of grid results
Output
param_n_estimators 50 100 150 param_max_depth 3 0.9425 0.9442 0.9451 7 0.9685 0.9612 0.9598 11 0.9584 0.9571 0.9567
The heatmap reveals: max_depth=7 with n_estimators=50 gives F1=0.9685 (top-left corner). Deeper or more trees reduce F1—classic overfitting behavior.

Why cv=5 inside GridSearchCV matters

GridSearchCV uses cross-validation to evaluate each hyperparameter combo. This prevents lucky splits and gives robust estimates.

Process:
1. For each param combo (e.g., max_depth=7, n_est=50)
2. Run 5-fold cross-validation on train set
3. Record mean F1 across 5 folds
4. After all combos tested, pick the best
5. Refit on full train set with best params
6. Evaluate on held-out test set

This two-stage approach prevents data leakage. The test set never influences hyperparameter selection.

Also used in...

Domain spotlight

🏦
Banking: Credit risk models
Tune gradient boosting max_depth, learning_rate to balance fraud detection (precision) and loan approval rate (recall).
📱
Telecom: Churn prediction
GridSearch over RandomForest params to minimize customer loss (FN cost >> FP cost in retention campaigns).
🛒
E-commerce: Recommendation systems
Tune matrix factorization rank, regularization to maximize relevance (NDCG) while keeping compute low.

How AI systems use this

Large language models (GPT, Claude) use hyperparameter tuning at scale. OpenAI tunes learning_rate, batch_size, and weight_decay on billions of tokens. Every model you train benefits from the same principles: exhaustive or smart search over the parameter space, cross-validation to avoid lucky splits, and learning curves to spot overfitting.

Model Selection Cross-validation Regularization Scaling Laws
🎯 Mini challenge: tune a decision tree

Task: Load a dataset (breast_cancer or iris). Use GridSearchCV with DecisionTreeClassifier to find the best max_depth and min_samples_split. Plot a learning curve. Print the best params and test accuracy.

Hint 1: Use DecisionTreeClassifier(). Import from sklearn.tree.

Hint 2: param_grid = {'max_depth': [3,5,7,10], 'min_samples_split': [2,5,10]}

Hint 3: GridSearchCV(model, param_grid, cv=5, scoring='accuracy')

Hint 4: learning_curve() with cv=5 will give train and val curves.

Interview questions: test your understanding

Parameters are learned by the model (e.g., decision tree split thresholds). Hyperparameters are set by you before training (e.g., max_depth, n_estimators). You control hyperparameters; the model learns parameters.
It tests every combination of hyperparameters in a grid. For each combo, it runs k-fold cross-validation, records the mean score, then returns the best combo. It's exhaustive but slow for large grids.
When your hyperparameter space is large (> 100 combos). Randomized samples n_iter combos randomly, which is much faster. You sacrifice some exploration for speed, but usually find good solutions.
The model overfits. It memorizes train data (99% F1) but fails to generalize (91% F1). Fix: reduce model complexity (lower max_depth), add regularization, or collect more train data.
Never. That's data leakage. If you tune on test data, the test set influences hyperparameter selection, making test scores optimistically biased. Always: (1) split train/test first, (2) use GridSearchCV on train only (with internal CV), (3) evaluate final model on untouched test set.

Quiz: 5 questions to test your knowledge

Question 1 of 5
You set max_depth=10 in RandomForestClassifier(). Is max_depth a parameter or hyperparameter?
Question 2 of 5
GridSearchCV tests 3 x 3 x 3 = 27 hyperparameter combos with cv=5. How many times is the model trained?
Question 3 of 5
Your learning curve shows train F1=0.99 and validation F1=0.97. What's the diagnosis?
Question 4 of 5
GridSearch found best_score_=0.9685 during CV. You then evaluate on test set and get 0.9583. Why is test < CV?
Question 5 of 5
You have 100 hyperparameter combos to test. Grid would take 2 hours. RandomizedSearch with n_iter=30 takes 20 minutes. Which should you use?
0
/5
Excellent! You understand hyperparameter tuning.

Cheatsheet: hyperparameter tuning at a glance

Task Code When to use
Grid search (exhaustive) GridSearchCV(model, param_grid, cv=5) Small grid (< 100 combos)
Random search (fast) RandomizedSearchCV(model, param_dist, n_iter=30, cv=5) Large space (100+ combos)
Learning curves learning_curve(model, X, y, cv=5) Detect overfitting
Validation curves validation_curve(model, X, y, param_name='max_depth', param_range=[...]) Tune single param
Extract results grid_search.cv_results_ Analyze all tested combos
Best params grid_search.best_params_ View winning combo
Best model grid_search.best_estimator_ Retrained on all train data

Common mistakes: what breaks and how to fix it

Mistake 1: Tuning on test data
Data leakage. Test performance becomes optimistically biased. Fix: always split train/test first, tune on train only with internal CV, evaluate on held-out test.
Mistake 2: Ignoring learning curves
You find high CV score but don't check if the model overfits. Fix: always plot learning curves after tuning. Gap < 0.05 is safe, > 0.1 needs regularization.
Mistake 3: Using default cv=5 when cv=10 is better
5-fold CV is fast but noisy. With 10-fold or stratified 5-fold, estimates are more stable. Fix: use cv=5 for speed in exploration, cv=10 for final validation.
Mistake 4: Scaling inside GridSearchCV
If you scale before splitting, train and test statistics are mixed. Use Pipeline(scaler, model) inside GridSearchCV to avoid leakage.

Key takeaways

Hyperparameter tuning separates good models from great ones.

  • Know your parameters vs hyperparameters: Parameters are learned. Hyperparameters are knobs you turn.
  • GridSearchCV for small spaces, RandomizedSearchCV for large ones. Balance thoroughness with speed.
  • Always use cross-validation inside search. cv=5 is standard. Prevents lucky splits.
  • Plot learning curves after tuning. Train-val gap > 0.05 signals overfitting. Reduce complexity or regularize.
  • Never tune on test data. Split first, tune on train with internal CV, evaluate on real test set.
  • CV score ≠ test score. Expect test F1 to be 1-3% lower. That's normal.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Ready to build production models?

Join the Best AI Program: The AI-Era Analyst — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Analyst →