HealthPulse is a diagnostic lab in Hyderabad. Their Random Forest cancer classifier works—F1=0.963. But in healthcare, every 0.1% matters. They can't just use default parameters. Two doctors ask: what settings optimize F1 the most? GridSearch will test 36 combinations. RandomizedSearch will sample 30. Learning curves will show if the model overfits.
By the end, HealthPulse finds: GridSearch yields F1=0.9685 with max_depth=7 and 50 trees. RandomizedSearch finds F1=0.9667 with max_depth=14 and 253 trees. Learning curves reveal slight overfitting (train~0.995, val~0.960)—but acceptable for deployment.
What are hyperparameters vs parameters?
Confusion here costs time. Know the difference.
Hyperparameters: You set before training. The knobs you turn. Example: RandomForest's max_depth (tree height), n_estimators (tree count), min_samples_split (rows before split).
In sklearn, hyperparameters are arguments to the model constructor: RandomForestClassifier(max_depth=7, n_estimators=50, ...). Parameters live inside the trained model object .coef_, .feature_importances_, etc.
GridSearchCV vs RandomizedSearchCV: when to use what
Both search for the best hyperparameters. The strategy differs.
| Aspect | GridSearchCV | RandomizedSearchCV |
|---|---|---|
| Strategy | Tests every combination | Samples random combinations |
| Time complexity | Slow (3 params × 3 values = 27 combos) | Fast (sample 10 of those 27) |
| Best for | Small search space (< 100 combos) | Large space (100+ combos) |
| Risk | Thorough but expensive | Might miss the global best, but typically finds good local best fast |
| Example | max_depth: [3,7,11], n_est: [10,50,100] | max_depth: randint(1,30), n_est: randint(10,300) |
GridSearchCV: exhaustive grid search
HealthPulse starts with GridSearch. They define a grid of hyperparameters to test.
RandomizedSearchCV: fast random sampling
HealthPulse's data scientist wants to explore a wider space faster. Enter RandomizedSearchCV.
Learning curves: diagnose overfit and underfit
Good scores don't guarantee good generalization. Plot learning curves to see train vs validation performance.
The gap of 0.0353 is small—train and validation curves are close. In healthcare, a 96% validation F1 is strong, and the slight gap is acceptable. If the gap were > 0.1, we'd reduce max_depth or add regularization.
Validation curves: tune one parameter at a time
Sometimes you want to understand one parameter's effect. Validation curves plot train and test F1 as you vary a single hyperparameter.
This confirms max_depth=7 is the best choice. Beyond depth 7, validation F1 plateaus or drops—overfitting.
Interpreting GridSearch results: the heatmap
GridSearchCV stores all results in .cv_results_. Let's visualize them.
Why cv=5 inside GridSearchCV matters
GridSearchCV uses cross-validation to evaluate each hyperparameter combo. This prevents lucky splits and gives robust estimates.
1. For each param combo (e.g., max_depth=7, n_est=50)
2. Run 5-fold cross-validation on train set
3. Record mean F1 across 5 folds
4. After all combos tested, pick the best
5. Refit on full train set with best params
6. Evaluate on held-out test set
This two-stage approach prevents data leakage. The test set never influences hyperparameter selection.
Also used in...
Domain spotlight
How AI systems use this
Large language models (GPT, Claude) use hyperparameter tuning at scale. OpenAI tunes learning_rate, batch_size, and weight_decay on billions of tokens. Every model you train benefits from the same principles: exhaustive or smart search over the parameter space, cross-validation to avoid lucky splits, and learning curves to spot overfitting.
Task: Load a dataset (breast_cancer or iris). Use GridSearchCV with DecisionTreeClassifier to find the best max_depth and min_samples_split. Plot a learning curve. Print the best params and test accuracy.
Hint 1: Use DecisionTreeClassifier(). Import from sklearn.tree.
Hint 2: param_grid = {'max_depth': [3,5,7,10], 'min_samples_split': [2,5,10]}
Hint 3: GridSearchCV(model, param_grid, cv=5, scoring='accuracy')
Hint 4: learning_curve() with cv=5 will give train and val curves.
Interview questions: test your understanding
Quiz: 5 questions to test your knowledge
Cheatsheet: hyperparameter tuning at a glance
| Task | Code | When to use |
|---|---|---|
| Grid search (exhaustive) | GridSearchCV(model, param_grid, cv=5) |
Small grid (< 100 combos) |
| Random search (fast) | RandomizedSearchCV(model, param_dist, n_iter=30, cv=5) |
Large space (100+ combos) |
| Learning curves | learning_curve(model, X, y, cv=5) |
Detect overfitting |
| Validation curves | validation_curve(model, X, y, param_name='max_depth', param_range=[...]) |
Tune single param |
| Extract results | grid_search.cv_results_ |
Analyze all tested combos |
| Best params | grid_search.best_params_ |
View winning combo |
| Best model | grid_search.best_estimator_ |
Retrained on all train data |
Common mistakes: what breaks and how to fix it
Key takeaways
Hyperparameter tuning separates good models from great ones.
- Know your parameters vs hyperparameters: Parameters are learned. Hyperparameters are knobs you turn.
- GridSearchCV for small spaces, RandomizedSearchCV for large ones. Balance thoroughness with speed.
- Always use cross-validation inside search. cv=5 is standard. Prevents lucky splits.
- Plot learning curves after tuning. Train-val gap > 0.05 signals overfitting. Reduce complexity or regularize.
- Never tune on test data. Split first, tune on train with internal CV, evaluate on real test set.
- CV score ≠ test score. Expect test F1 to be 1-3% lower. That's normal.
Ready to build production models?
Join the Best AI Program: The AI-Era Analyst — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Analyst →