Dr. Priya works at HealthPulse, a diagnostic lab in Hyderabad. She's built a breast cancer classifier on 569 patient samples—30 features (tumor size, concavity, symmetry, etc.). The model reports 98.25% accuracy. Sounds perfect, right?
But then she digs deeper. Out of 72 actual malignant tumors, the model found 71 (Recall = 0.9861). It missed 1. That one missed diagnosis—a false negative (FN=1)—could mean a patient leaves thinking she's cancer-free when she isn't. Meanwhile, there's 1 false positive (FP=1): a benign tumor flagged as malignant, causing unnecessary stress and follow-up tests. Which error is costlier? When accuracy alone misleads, Priya needs the metrics dashboard: precision, recall, F1, AUC.
The confusion matrix: your scorecard
All metrics come from 4 numbers in the confusion matrix. Let's decode HealthPulse's results:
Key takeaway: Accuracy = (TP + TN) / Total = (71 + 41) / 114 = 0.9825 (98.25%). High, but hides the 1 FN that could kill.
Five metrics, five angles on the same data
All computed from the same 4 numbers above. Each answers a different business question.
Precision: Of predicted positives, how many were truly positive? TP / (TP + FP) — "Are my alarms trustworthy?"
Recall: Of actual positives, how many did I find? TP / (TP + FN) — "Did I catch all the cancers?"
F1: Harmonic mean of precision and recall. 2 * (Prec * Rec) / (Prec + Rec) — Balance both.
AUC: Area under ROC curve. Probability the model ranks a random positive higher than a random negative.
Computing metrics with scikit-learn
Load the data, train a model, get all five metrics in 10 lines of code:
What each metric tells us
Accuracy = 0.9825 (98.25%)
Of 114 test samples, 112 were predicted correctly. Looks great—but in healthcare, this hides danger. If the dataset has 98% benign and 2% malignant, a naive model that predicts "all benign" would get 98% accuracy while catching zero cancers. Accuracy fails on imbalanced classes.
Precision = 0.9861 (98.61%)
Of 72 predictions of "cancer" (TP + FP), 71 were truly cancer. Only 1 false alarm. Great for reducing unnecessary tests. But it ignores FN—missed diagnoses.
Recall = 0.9861 (98.61%)
Of 72 actual cancers, the model found 71. We missed 1. In screening, recall is critical: we want to catch as many true positives as possible. A recall of 0.95 means 1 in 20 real cancers go undetected—unacceptable in medical settings.
F1 = 0.9861 (98.61%)
The harmonic mean of precision and recall. When both are high and balanced, F1 is high. Useful when you care equally about false positives and false negatives. In this case, precision = recall = 0.9861, so F1 = 0.9861.
AUC = 0.9954 (99.54%)
The model's ability to separate cancers from benign samples across all possible thresholds. AUC = 1.0 means perfect ranking (all malignant samples ranked higher than all benign). AUC = 0.5 means random. 0.9954 is excellent: almost no overlap between the distributions.
ROC curve: threshold tradeoff
Behind every prediction is a probability. By default, we classify as cancer if P(cancer) >= 0.5. But we can tune the threshold.
As threshold decreases (predict cancer more easily), TPR (recall) increases but so does FPR. You choose the point that suits your tolerance for false positives vs. false negatives.
Threshold tuning: control precision-recall tradeoff
At threshold = 0.5, we get the confusion matrix above. But what if we lower the threshold to catch more cancers (higher recall)?
Business insight: At threshold 0.547, we catch 95.83% of true cancers (higher recall, fewer FN). Precision drops to 98.63%, meaning a few more false positives. For HealthPulse, missing a cancer (FN) is catastrophic; a follow-up test for a false positive is manageable. So lowering the threshold is justified.
Also used in...
Domain Spotlight
How AI systems use this
Every classifier—from logistic regression to transformer-based medical AI—must choose a threshold and optimize a metric. ChatGPT optimizes for relevance (precision-like). Spam filters optimize for recall (catch most spam). Self-driving cars optimize for recall on collision detection (miss zero obstacles). Metrics shape behavior. Without understanding them, you deploy biased or dangerous models.
Task: You're HealthPulse. FN cost = ₹10,00,000 (₹10L—life, lawsuit, reputation). FP cost = ₹10,000 (follow-up test). Use sklearn to find the threshold that minimizes total cost: (FN_count * 10L) + (FP_count * 10K). Report optimal threshold and resulting precision/recall.
Hint 1: Get probabilities with y_proba = model.predict_proba(X_test)[:, 1]
Hint 2: Loop through thresholds 0.1 to 0.9 in 0.01 steps
Hint 3: For each threshold, create y_pred = (y_proba >= threshold).astype(int)
Hint 4: Count FP and FN: FP = sum((y_pred==1) & (y_test==0)), FN = sum((y_pred==0) & (y_test==1))
Hint 5: Track the threshold with minimum cost
Interview questions: test your understanding
Quiz: 5 questions to test your knowledge
Cheatsheet: metrics at a glance
| Metric | Formula | When to use |
|---|---|---|
| Accuracy | (TP + TN) / Total |
Balanced classes only |
| Precision | TP / (TP + FP) |
FP cost is high (e.g., spam labels) |
| Recall | TP / (TP + FN) |
FN cost is high (e.g., cancer screening) |
| F1 | 2*(Prec*Rec)/(Prec+Rec) |
Balance precision and recall |
| AUC | Area under ROC curve |
Threshold-independent ranking quality |
Common mistakes: what breaks and how to fix it
Key takeaways
Metrics are not neutral. They encode business decisions. Choose wisely.
- Never trust accuracy on imbalanced data. A 99% accurate model might have recall=0.
- Precision & Recall have a tradeoff. Lower threshold → higher recall, lower precision. Tune to your cost.
- AUC is threshold-independent. One number to assess ranking quality across all thresholds.
- Confusion matrix is ground truth. Always compute TP, FP, FN, TN. All metrics come from these 4 numbers.
- F1 balances precision and recall. Use when both matter equally. Use precision/recall separately when costs differ.
- Threshold tuning is cost-driven. With FN cost = 100x FP cost, lower the threshold and accept more false alarms.
Deploy models that matter?
Join the Best AI Program: The AI-Era Analyst — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Analyst →