πŸ’Š
Your context
Esther β€” NGO M&E analyst in Nakuru

You're analyzing 120 clinic visits from a rural health NGO to write a donor report. Your dataset loads fine: patient_id, age, systolic BP, glucose, visit duration. But something's off. One patient has BP = 250 mmHg (physically impossible). Another visit lasted 480 minutes (8 hours?). A weight column shows 8 kg for a 35-year-old adult. Are these data-entry errors? Equipment failures? Real medical emergencies? Your report must distinguish signal from noise β€” donors want accuracy, clinicians want actionable alerts. A boxplot reveals outliers in 30 seconds. Deeper investigation tells you which ones matter.

πŸ“₯
clinic_visits_extended.csv
120 rows Γ— 7 columns. Includes 5–8 intentional outliers.

What Is an Outlier?

An outlier is a data point that differs dramatically from the rest. It stands alone in the distribution's tail. But "different" is not synonymous with "wrong" β€” every outlier demands investigation.

Systolic BP distribution (clinic patients). Most cluster 110–140 mmHg. Three points sit far above.
Systolic BP distribution with visible outliers
What this shows: A boxplot compresses your distribution into 5 numbers: min, Q1, median, Q3, max. Points beyond the whiskers (typically 1.5 Γ— IQR from Q1/Q3) are flagged as outliers. The three red dotsβ€”systolic BP of 238, 242, 250 mmHgβ€”are physically implausible for human patients. Investigation needed: Are these data-entry errors? Equipment malfunction? Real hypertensive crisis from another clinic branch?

The 1.5 Γ— IQR Rule

The Interquartile Range (IQR) is Q3 βˆ’ Q1, the spread of the middle 50% of data. Outliers are defined as points below Q1 βˆ’ 1.5Γ—IQR or above Q3 + 1.5Γ—IQR.

Python: IQR outlier detection

For your clinic data: Q1 = 118, Q3 = 131, IQR = 13. Bounds: 118 βˆ’ 19.5 = 98.5 (lower), 131 + 19.5 = 150.5 (upper). Any systolic BP > 150.5 or < 98.5 is flagged.

IQR rule applied to 120 clinic visits. Table shows the 4 flagged outliers by systolic BP.
Patient ID Systolic BP Age Notes
P015 142 36 High but real (emergency visit)
P046 141 35 High but real (hypertensive)
P068 142 35 High but real (emergency visit)
P080 144 36 Slight elevation, real patient
P092 145 35 Systolic 250 is impossible error
Interpretation: The IQR rule flagged 5 rows. Four have systolic BP of 141–145, which is high but plausible for hypertensive crises at an NGO clinic. One (P092, systolic 250) is physically impossible β€” likely a data-entry error (keyboard double-press: 150 β†’ 250?). The rule caught the error but also flagged real emergencies. You must investigate each outlier, not blindly drop.

Z-Scores: Another Lens

A Z-score measures how many standard deviations a point is from the mean. Z > 3 or Z < -3 are often flagged as outliers. Z-scores are useful for symmetric distributions but miss outliers in skewed data.

Python: Z-score outlier detection

For your glucose data: mean β‰ˆ 98, std β‰ˆ 9. A Z-score of 3 flags glucose > 125 or < 71. Compare IQR vs Z-score results: they often differ, especially with skewed data. IQR is more robust to extreme outliers; Z-scores assume normality.

IQR Rule

Pros: Robust to extremes. Works on skewed data.

Cons: Flags fewer outliers in heavy-tailed distributions.

Z-Score

Pros: Sensitive to extreme values. Good for symmetric data.

Cons: Assumes normality. Misses outliers in skewed data.

Lesson: Different methods flag different points. Neither is "correct" β€” they reveal different perspectives on your data. Use both, then domain knowledge to decide.

Domain-Aware Bounds: The Clinician's View

Statistical rules are useful, but domain expertise trumps math. In healthcare, systolic BP < 60 or > 200 mmHg is physically implausible for a living, conscious patient. A weight of 8 kg for a 35-year-old is impossible. These hard rules catch errors that statistics miss.

Python: Domain-aware outlier flagging

In your clinic data:

  • Patient P092: glucose = 250 mmHg (no β€” glucose is mg/dL, not mmHg; data mislabeled)
  • Patient P015: visit_duration = 480 min (8 hours for a routine clinic visit β€” data-entry error, likely 48 min)
  • Patient P101: weight = 8 kg (impossible for age 22 β€” should be ~60 kg)
Why domain bounds matter: A clinician sees "250 mmHg" and knows instantly: "Impossible. Fix it." A statistician using IQR might think "Extreme but within range." Domain knowledge is your strongest defense against errors.

Handling Outliers: Winsorize, Drop, or Cap?

Once you've identified outliers, you have three choices:

1 Drop the row

Use when: The outlier is a clear error and losing that row won't bias your analysis.

Remove the entire row. Simplest, but loses data.

df_clean = df[~((df['systolic_bp'] < 60) | (df['systolic_bp'] > 200))]

2 Winsorize (cap)

Use when: The outlier is real but extreme; you want to keep the row but reduce its influence.

Replace outliers with the boundary value (1st/99th percentile). Keeps all rows, softens extremes.

from scipy.stats import mstats df['systolic_bp'] = mstats.winsorize(df['systolic_bp'], limits=[0.01, 0.01])

3 Flag & keep

Use when: The outlier is signal, not noise (e.g., hypertensive crisis, rare allergy).

Add a boolean column. Keep the value, let your model learn it's unusual.

df['is_high_bp'] = df['systolic_bp'] > 150

For clinic data: Drop rows with impossible values (P092 glucose error). Flag real emergencies (P015, P046 high BP). Winsorize extreme visit durations if you suspect recording errors but want to preserve the session.

Example decision: Systolic BP 250? Drop it (error). Systolic BP 145? Flag it (real hypertensive crisis). Visit duration 480 min? Winsorize to 300 min or drop (likely keyboard entry error, e.g., "48" β†’ "480").

Robust Statistics: Median & MAD

When outliers are present, the mean and standard deviation are unreliable. The median and Median Absolute Deviation (MAD) are robust alternatives.

Effect of one extreme outlier (blood glucose = 400) on mean vs median.
Metric With Outlier Without Outlier Change
Mean glucose 102 mg/dL 98 mg/dL +4% bias
Median glucose 98 mg/dL 98 mg/dL 0% (stable)
Std Dev 18 mg/dL 9 mg/dL +100% inflation
MAD 8 mg/dL 8 mg/dL 0% (stable)
Python: Robust statistics
Takeaway: One outlier inflated the mean by 4% and the std by 100%. The median and MAD didn't budge. When outliers exist, report medians, not means. Use MAD instead of std for confidence intervals.
0/3
Quiz Progress
Q1: The 1.5 Γ— IQR rule flags points beyond Q1 βˆ’ 1.5Γ—IQR and Q3 + 1.5Γ—IQR. What does this rule rely on?
The assumption that all data is normally distributed
The spread of the middle 50% of data (IQR) and a fixed multiplier
The mean and standard deviation of the data
Domain expertise about what values are medically plausible
Q2: Your data has a skewed distribution (long right tail). Z-score flags fewer outliers than IQR. Why?
Z-scores are more conservative by design
Z-scores assume symmetry. The right tail of skewed data increases the mean and std, so Z-scores become less extreme
Skewed data makes Z-scores invalid; you should use IQR always
IQR is always more sensitive to outliers than Z-scores
Q3: A patient's systolic BP = 250 mmHg. Your IQR rule (bounds 98.5 to 150.5) flags it as an outlier. A clinician says "Impossible β€” human BP can't exceed 200." What should you do?
Trust the IQR rule; it's statistical and objective
Trust the clinician; domain expertise beats statistics. This is a data-entry error (e.g., 150 β†’ 250).
Winsorize it to 150.5 so the model doesn't break
Keep it as-is; the model can learn to ignore it

Common Pitfalls

1. Blindly dropping all outliers

Not all outliers are errors. A hypertensive crisis (BP 145) is real and important for clinical care. Dropping it loses signal.

2. Trusting Z-scores on skewed data

In income data, a few high earners skew the distribution. Z-scores become unreliable. Use IQR or domain bounds instead.

3. Using the same outlier rule on all columns

Systolic BP and visit duration have different scales and distributions. A 1.5Γ—IQR threshold may be too strict for one, too loose for another.

4. Not documenting your decision

If you drop 5 rows, note why: "P092 had physically impossible glucose; P101 weight = 8kg for age 22." Stakeholders need transparency.

5. Winsorizing without understanding the impact

Capping outliers at 99th percentile softens their influence but may hide real extremes from your analysis. Check your data before and after.

6. Ignoring the outlier's context

A visit lasting 8 hours is an outlier β€” but was the patient in a hypertensive crisis needing extended monitoring? Context changes how you handle it.

Outlier Detection Cheatsheet

Method Formula / Code Best for Weakness
IQR Rule |q1 βˆ’ 1.5Γ—IQR|, |q3 + 1.5Γ—IQR| Any distribution, especially skewed May flag too many in heavy-tailed data
Z-Score |x βˆ’ mean| / std > 3 Symmetric, normal-like data Fails on skewed data; sensitive to extremes
Domain Bounds x < min_bound or x > max_bound Medical, financial, physical constraints Requires domain knowledge; can be subjective
Modified Z-Score (MAD) |x βˆ’ median| / MAD > 3.5 Robust detection with outliers present Less familiar; fewer built-in functions

Outlier patterns are universal in health data

Whether you're analyzing clinic data in Nakuru (Kenya), Mumbai (India), Manila (Philippines), or Manchester (UK), the same errors appear: keyboard doubles (150 β†’ 250), truncated durations (48 β†’ 480), weight entry in pounds instead of kg. The IQR rule, Z-score, and domain bounds work worldwide. A 250 mmHg BP is impossible everywhere. A baby weighing 8 kg is impossible everywhere. Good clinical reasoning transcends geography.

Africa & India's Best AI Academy
πŸ€– AI-first πŸŽ“ Placement Assistance

Ready to build dashboards your stakeholders trust?

Join the Best AI Program: The AI-Era Analyst β€” 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Analyst β†’