🏥
Real-world scenario
HealthPulse (Hyderabad): 569 breast cancer samples, 98.25% accuracy, but which metric should we optimize?

Dr. Priya works at HealthPulse, a diagnostic lab in Hyderabad. She's built a breast cancer classifier on 569 patient samples—30 features (tumor size, concavity, symmetry, etc.). The model reports 98.25% accuracy. Sounds perfect, right?

But then she digs deeper. Out of 72 actual malignant tumors, the model found 71 (Recall = 0.9861). It missed 1. That one missed diagnosis—a false negative (FN=1)—could mean a patient leaves thinking she's cancer-free when she isn't. Meanwhile, there's 1 false positive (FP=1): a benign tumor flagged as malignant, causing unnecessary stress and follow-up tests. Which error is costlier? When accuracy alone misleads, Priya needs the metrics dashboard: precision, recall, F1, AUC.

569
Samples (BC Wisconsin)
0.9954
AUC (excellent)
1 FN, 1 FP
Misclassifications

The confusion matrix: your scorecard

All metrics come from 4 numbers in the confusion matrix. Let's decode HealthPulse's results:

Predicted → Predicted Negative Predicted Positive
Actual Negative TN = 41
Correct negatives
FP = 1
False alarm
Actual Positive FN = 1
Missed cancer
TP = 71
Correctly caught
Confusion Matrix Heatmap: HealthPulse test results
Confusion matrix heatmap showing true positives, true negatives, false positives, and false negatives
Total samples: 42 benign (TN+FP), 72 malignant (FN+TP). In healthcare, FN cost >>> FP cost. Missing cancer is fatal; false alarms are manageable.

Key takeaway: Accuracy = (TP + TN) / Total = (71 + 41) / 114 = 0.9825 (98.25%). High, but hides the 1 FN that could kill.

Five metrics, five angles on the same data

All computed from the same 4 numbers above. Each answers a different business question.

Accuracy: Of all predictions, how many were correct? (TP + TN) / Total
Precision: Of predicted positives, how many were truly positive? TP / (TP + FP) — "Are my alarms trustworthy?"
Recall: Of actual positives, how many did I find? TP / (TP + FN) — "Did I catch all the cancers?"
F1: Harmonic mean of precision and recall. 2 * (Prec * Rec) / (Prec + Rec) — Balance both.
AUC: Area under ROC curve. Probability the model ranks a random positive higher than a random negative.

Computing metrics with scikit-learn

Load the data, train a model, get all five metrics in 10 lines of code:

compute_metrics.py Calculate all error metrics
Output
Accuracy: 0.9825 Precision: 0.9861 Recall: 0.9861 F1 Score: 0.9861 AUC: 0.9954 Confusion Matrix: [[41 1] [ 1 71]]

What each metric tells us

Accuracy = 0.9825 (98.25%)

Of 114 test samples, 112 were predicted correctly. Looks great—but in healthcare, this hides danger. If the dataset has 98% benign and 2% malignant, a naive model that predicts "all benign" would get 98% accuracy while catching zero cancers. Accuracy fails on imbalanced classes.

Precision = 0.9861 (98.61%)

Of 72 predictions of "cancer" (TP + FP), 71 were truly cancer. Only 1 false alarm. Great for reducing unnecessary tests. But it ignores FN—missed diagnoses.

Recall = 0.9861 (98.61%)

Of 72 actual cancers, the model found 71. We missed 1. In screening, recall is critical: we want to catch as many true positives as possible. A recall of 0.95 means 1 in 20 real cancers go undetected—unacceptable in medical settings.

F1 = 0.9861 (98.61%)

The harmonic mean of precision and recall. When both are high and balanced, F1 is high. Useful when you care equally about false positives and false negatives. In this case, precision = recall = 0.9861, so F1 = 0.9861.

AUC = 0.9954 (99.54%)

The model's ability to separate cancers from benign samples across all possible thresholds. AUC = 1.0 means perfect ranking (all malignant samples ranked higher than all benign). AUC = 0.5 means random. 0.9954 is excellent: almost no overlap between the distributions.

ROC curve: threshold tradeoff

Behind every prediction is a probability. By default, we classify as cancer if P(cancer) >= 0.5. But we can tune the threshold.

ROC Curve: HealthPulse model on test set (AUC = 0.9954)
ROC and Precision-Recall curves for HealthPulse breast cancer classification model
The purple area is AUC. Larger area = better model. Diagonal line = random classifier (AUC=0.5). Our model hugs the top-left: excellent separation.

As threshold decreases (predict cancer more easily), TPR (recall) increases but so does FPR. You choose the point that suits your tolerance for false positives vs. false negatives.

Threshold tuning: control precision-recall tradeoff

At threshold = 0.5, we get the confusion matrix above. But what if we lower the threshold to catch more cancers (higher recall)?

tune_threshold.py Find threshold for 95% recall
Output
To achieve 95% recall: Threshold: 0.547 Precision: 0.9863 Recall: 0.9583 Interpretation: Lower threshold to 0.547 → catch more cancers (95%+) But precision drops slightly: trade fewer FNs for more FPs
Precision, Recall, and F1 vs. Classification Threshold
Line chart showing precision, recall, and F1 score across different classification thresholds

Business insight: At threshold 0.547, we catch 95.83% of true cancers (higher recall, fewer FN). Precision drops to 98.63%, meaning a few more false positives. For HealthPulse, missing a cancer (FN) is catastrophic; a follow-up test for a false positive is manageable. So lowering the threshold is justified.

Also used in...

Domain Spotlight

🏦
Banking: Fraud detection
FP cost = ₹500 (block legitimate transaction). FN cost = ₹50,000 (miss fraud). Optimize for high recall, accept lower precision. Threshold tuning essential.
🎯
E-commerce: Ad targeting
FP cost = ₹5 (show ad to uninterested user). FN cost = ₹100 (miss interested buyer). Optimize for precision, filter spam. Recall can be lower.
🚔
Security: Intrusion detection
FP cost = investigate false alert. FN cost = breach. High recall (catch attacks), accept alerts. Tune threshold to max recall while keeping FP manageable.

How AI systems use this

Every classifier—from logistic regression to transformer-based medical AI—must choose a threshold and optimize a metric. ChatGPT optimizes for relevance (precision-like). Spam filters optimize for recall (catch most spam). Self-driving cars optimize for recall on collision detection (miss zero obstacles). Metrics shape behavior. Without understanding them, you deploy biased or dangerous models.

Classification Evaluation Threshold Optimization Cost-Benefit Analysis Model Deployment
🎯 Mini Challenge: Cost-driven threshold

Task: You're HealthPulse. FN cost = ₹10,00,000 (₹10L—life, lawsuit, reputation). FP cost = ₹10,000 (follow-up test). Use sklearn to find the threshold that minimizes total cost: (FN_count * 10L) + (FP_count * 10K). Report optimal threshold and resulting precision/recall.

Hint 1: Get probabilities with y_proba = model.predict_proba(X_test)[:, 1]

Hint 2: Loop through thresholds 0.1 to 0.9 in 0.01 steps

Hint 3: For each threshold, create y_pred = (y_proba >= threshold).astype(int)

Hint 4: Count FP and FN: FP = sum((y_pred==1) & (y_test==0)), FN = sum((y_pred==0) & (y_test==1))

Hint 5: Track the threshold with minimum cost

Interview questions: test your understanding

Precision: Of predicted positives, how many are true? (TP / (TP+FP)). Answer: "Are my alarms correct?" Recall: Of actual positives, how many did I find? (TP / (TP+FN)). Answer: "Did I catch all of them?" Precision is about alarm quality; recall is about coverage.
AUC = Area Under the ROC Curve. It measures the probability that the model ranks a random positive sample higher than a random negative sample. AUC=1.0 means perfect ranking; AUC=0.5 means random. It's threshold-independent—one number that captures overall discriminative ability.
Accuracy is misleading on imbalanced classes. If 99% of samples are negative, a model that predicts all negatives gets 99% accuracy while catching zero positives (recall=0). Use precision, recall, F1, or AUC instead. Accuracy hides the cost of errors.
Depends on the cost of errors. Medical diagnosis: optimize recall (find all cancers, false positives are follow-up tests). Spam filtering: optimize precision (don't block legit email, miss some spam). Ad targeting: optimize precision (don't waste ad spend). Fraud: optimize recall (catch fraud, investigate some false alarms). Tune the threshold to your cost structure.
Loop through possible thresholds. For each, compute FP and FN. Calculate total cost = (FP * 10K) + (FN * 10L). Choose the threshold minimizing total cost. Since FN cost is 100x higher, the optimal threshold will be lower than 0.5 (predict positive more often to catch more true positives, accepting more false positives). This is cost-sensitive threshold optimization.

Quiz: 5 questions to test your knowledge

Question 1 of 5
HealthPulse's model has TP=71, FP=1, FN=1, TN=41. What is the precision?
Question 2 of 5
What does the 1 FN (false negative) in HealthPulse's results mean?
Question 3 of 5
On imbalanced data (99% negative class), why is accuracy misleading?
Question 4 of 5
What is AUC = 0.9954 telling us?
Question 5 of 5
To minimize healthcare cost (FN = ₹10L, FP = ₹10K), should you lower or raise the threshold from 0.5?
0
/5
Great job! You understand error metrics.

Cheatsheet: metrics at a glance

Metric Formula When to use
Accuracy (TP + TN) / Total Balanced classes only
Precision TP / (TP + FP) FP cost is high (e.g., spam labels)
Recall TP / (TP + FN) FN cost is high (e.g., cancer screening)
F1 2*(Prec*Rec)/(Prec+Rec) Balance precision and recall
AUC Area under ROC curve Threshold-independent ranking quality

Common mistakes: what breaks and how to fix it

Mistake 1: Trusting accuracy on imbalanced data
If 98% of samples are negative, accuracy can be high (98%) even if recall is 0. Fix: always check precision, recall, F1, or AUC. Never rely on accuracy alone.
Mistake 2: Optimizing the wrong metric
Cancer screening: optimize recall (catch all cancers). Spam filtering: optimize precision (don't block legit email). You must understand the business cost first. Fix: talk to stakeholders about FP and FN costs.
Mistake 3: Ignoring the confusion matrix
A single metric (accuracy, AUC) can hide problems. The 1 FN in HealthPulse's matrix is invisible in a single number. Fix: always compute and interpret the confusion matrix.
Mistake 4: Tuning threshold on test set
This is data leakage! You're overfitting the threshold. Fix: use a separate validation set to find the threshold, test set only for final evaluation.

Key takeaways

Metrics are not neutral. They encode business decisions. Choose wisely.

  • Never trust accuracy on imbalanced data. A 99% accurate model might have recall=0.
  • Precision & Recall have a tradeoff. Lower threshold → higher recall, lower precision. Tune to your cost.
  • AUC is threshold-independent. One number to assess ranking quality across all thresholds.
  • Confusion matrix is ground truth. Always compute TP, FP, FN, TN. All metrics come from these 4 numbers.
  • F1 balances precision and recall. Use when both matter equally. Use precision/recall separately when costs differ.
  • Threshold tuning is cost-driven. With FN cost = 100x FP cost, lower the threshold and accept more false alarms.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Deploy models that matter?

Join the Best AI Program: The AI-Era Analyst — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Analyst →