Master k-fold, stratified, leave-one-out, and nested CV. Stop relying on lucky train/test splits. Real HealthPulse cancer diagnosis data.
🎯 Model Evaluation📊 scikit-learn🏥 Healthcare Data⚡ Intermediate📖 ~22 min
Lesson 17 of Data Science Track
🏥
Real-world scenario
HealthPulse (Hyderabad): One lucky split might hide the truth
HealthPulse is a diagnostic lab in Hyderabad building a breast cancer classifier. A logistic regression model scores 98.2% accuracy on one random 80/20 split. Perfect! the team says. But what if that split was lucky? What if another split gives 94%? Which is the true performance?
Cross-validation solves this. Instead of trusting one split, it tests on 5 (or 10) different splits and reports the average. HealthPulse's logistic regression: 98.02% ± 1.28%. That ± is real—it's the variance across folds. Random Forest without tuning? 96.34% ± 1.84%. More variable, lower average. LR wins.
5-fold
Standard choice
0.9802 ± 0.0128
LR true performance
569
Cancer samples
Why single split is risky
A train/test split divides your data once, randomly. On that one split, your model's test score could be a fluke. Maybe your model got lucky. Maybe the test set was easier. You'll never know.
Cross-validation repeats the split-and-test 5, 10, or 20 times, each time with a different random split. You get 5 or 10 test scores. Their mean and std tell you the true expected performance, with confidence bounds.
Single split: One accuracy number. Could be lucky or unlucky. High variance. Unreliable. K-fold CV: k accuracy numbers. Average them. Much more reliable. You see the variance (std). You know the model's true performance range.
How k-fold cross-validation works
k-fold splits your data into k equal groups (folds). Then, for each fold:
Use that fold as test data.
Use the other k-1 folds as train data.
Train the model and record test accuracy.
After k iterations, you have k accuracy scores. Their mean ± std is your cross-validated estimate.
5-fold CV: each fold rotates as test set
HealthPulse's 5-fold split: [0.967, 0.978, 0.967, 1.000, 0.989]. Mean = 0.9802, Std = 0.0128. This is logistic regression's true performance, not luck.
5-fold CV with logistic regression
Let's train a logistic regression model on HealthPulse cancer data and run 5-fold cross-validation:
If your target is imbalanced (e.g., 10% cancer, 90% healthy), a random split might put all cancers in train and none in test. Stratified CV preserves the class distribution in each fold.
cv_stratified.pyStratified 5-fold CV with F1 score
Output
Positive rate: 62.74%
Stratified 5-fold F1 scores: [0.984 0.984 0.982 0.987 0.980]
Mean F1: 0.9825 ± 0.0026
Compare models using cross-validation
The real power of CV: compare 3 models fairly. Which is best? Not on one split, but across many.
cv_compare.pyCompare Logistic Regression, Random Forest, Gradient Boosting
Leave-one-out (LOO) CV is k-fold where k = n (number of samples). For 569 samples, you train 569 times, each time leaving out exactly 1 sample as test. Unbiased, but slow and high variance.
cv_loocv.pyLeave-one-out cross-validation
Output
LOOCV accuracy: 0.9825
Standard error: 0.0042
Time: 45.2s
Vs 5-fold: faster but unbiased estimate of true error
Nested CV for hyperparameter tuning
When you use GridSearchCV inside cross-validation, you need nested CV: outer loop for honest test scores, inner loop for tuning. Without this, you overfit to the test data.
cv_nested.pyNested CV with GridSearchCV
Output
Best C: 1, Test F1: 0.9843
Best C: 1, Test F1: 0.9856
Best C: 1, Test F1: 0.9865
Best C: 1, Test F1: 0.9847
Best C: 1, Test F1: 0.9853
Nested CV F1: 0.9853 ± 0.0008
Cross-validation methods at a glance
Method
Folds
Use case
Pros
Cons
5-fold
5
Default, balanced
Fast, reliable
Arbitrary choice of k
10-fold
10
More stable mean
Smoother estimate
Slower, higher std
Stratified k-fold
k
Imbalanced classes
Preserves class ratio
Only for classification
Leave-one-out
n
Small datasets (< 100)
Unbiased, high fidelity
Very slow, high variance
Nested CV
outer + inner
Tuning + evaluation
Honest hyperparameter scores
Expensive, nested loops
Also used in...
Domain Spotlight
🏥
Healthcare: Diagnostic models
Stratified CV ensures minority diseases appear in all folds. Nested CV tunes thresholds for sensitivity/specificity tradeoffs.
💳
Finance: Fraud detection
Imbalanced data (0.1% fraud). Stratified CV with nested tuning to catch rare frauds without overfitting.
📱
Telecom: Churn prediction
Large dataset (100k+ rows). 5-fold CV to benchmark LR, RF, XGBoost. 10-fold for stability.
How AI systems use this
Every serious ML system uses cross-validation to avoid overfitting. GPT researchers use stratified CV on text classification tasks. Self-driving cars use nested CV to tune perception models. Nested CV is standard in medical AI where overfitting can harm patients. Single splits are only for final test-set evaluation, never for model selection.
Model EvaluationHyperparameter TuningAvoiding OverfittingStatistical Rigor
Challenge: k-fold CV on real data
Task: Load a dataset (e.g., iris or wine). Train 3 models (LR, RF, SVM) using 5-fold stratified CV. Print F1 scores for each. Which model has the lowest std (most stable)?
Hint 1: from sklearn.datasets import load_iris
Hint 2: cross_val_score(..., cv=StratifiedKFold(5, shuffle=True), scoring='f1_weighted')
Hint 3: Compare std: print(f"{name}: {scores.std():.4f}")
Interview questions: test your understanding
Splits data into 5 equal folds. Trains 5 times: each fold becomes test once, the other 4 are train. You get 5 test scores. The mean ± std is your estimate of true performance.
Ensures each fold has the same class distribution as the full dataset. If your target is 30% positive, each fold is 30% positive. Prevents unlucky splits.
Neither universally. 5-fold is fast, has lower per-fold variance. 10-fold is smoother but slower and has higher std because each fold is smaller. Use 5-fold for imbalanced data or fast feedback. Use 10-fold for production models where stability matters.
Only on tiny datasets (< 100 samples) where you can afford n training runs. Even then, LOOCV has high variance and is slow. 5-fold is usually better unless you have <50 samples.
Normal GridSearchCV on the full dataset (then test on holdout) leaks test info into tuning. Nested CV: outer CV estimates true performance, inner CV (within each outer fold) tunes hyperparams. Only the inner best params matter; you report outer fold scores.
Quiz: 5 questions to test your knowledge
Question 1 of 5
Your 5-fold CV gives F1 scores: [0.95, 0.96, 0.94, 0.97, 0.96]. What is the mean?
Question 2 of 5
You use StratifiedKFold on a balanced dataset (50% positive). What happens?
Question 3 of 5
Leave-one-out CV on 569 samples means how many training runs?
Question 4 of 5
In nested CV, which loop is for honest model evaluation?
Question 5 of 5
Model A: F1=0.95±0.12. Model B: F1=0.94±0.02. Which is more stable?
0
/5
Excellent! You understand cross-validation.
Common mistakes: what breaks and how to fix it
Mistake 1: Using random KFold on imbalanced data
A fold might have 0% of the rare class. Fix: always use StratifiedKFold for classification.
Mistake 2: Using GridSearchCV without nested CV
Your final CV scores are optimistic—they saw the test data during tuning. Fix: wrap GridSearchCV inside an outer CV loop.
Mistake 3: Scaling before CV
The scaler learns statistics from all folds, leaking test info. Fix: scale inside the CV loop (use Pipeline or fit on train only in each fold).
Mistake 4: Comparing single split to CV scores
One fold gave 99%, CV gave 95%. They're different estimates; CV is more honest. Don't panic.
Key takeaways
Cross-validation is the backbone of honest model evaluation.
Always use CV, not single split. One split is risky; k-fold is standard.
Use StratifiedKFold for classification. Preserves class balance in each fold.
5-fold is the default. Fast and reliable. 10-fold for production models.
Leave-one-out only for tiny datasets. Otherwise, it's slow and high-variance.
Nested CV when tuning hyperparameters. Outer CV for honest scores, inner CV for tuning.
Report mean ± std. The std tells you if your model is stable across data variations.