💳
Your context
Arjun — SouthBank India analyst

You're reviewing 2,000 loan applications to profile default risk. Your DataFrame loads fine, but df.info() reveals a problem: pan_number has 1,734 non-null entries out of 2,000. That's 266 blanks. Are they random? All from one branch? How do they affect your model? A heatmap tells you in 2 seconds. A careless df.dropna() could lose 40% of your data.

The Missingness Heatmap — See the Problem at a Glance

Every row is a loan application, every column is a field. Purple cells = data present, white cells = null. Patterns jump out immediately.

Loan application dataset: 2,000 rows × 11 columns. Each cell colored by presence.
Heatmap of loan applications showing missing cells as lighter bands; co_applicant and pan_number columns show the most nulls.
What this shows: Missingness is not random. The pan_number column has a vertical stripe of nulls (rows from Kochi), indicating a branch-level data collection issue. The co_applicant column is sparse throughout—many single-applicant loans. This is information, not garbage. Your strategy changes per column.

How Pandas Represents Missing Data

Different source formats encode nulls differently. Pandas normalizes them all to NaN (numeric) or None (object).

Source Value Pandas reads as dtype impact Example
Empty cell (blank) NaN float64 or object df.loc[0, 'income'] is NaN
"NaN" (string) NaN float64 CSV: "col1,col2\n1,NaN"
"NA" (string) NaN float64 R export, Excel legacy
"None" (string) NaN object (stays string!) Careless JSON export
"-" or "–" Stays as string object Manual data entry convention
0 (numeric zero) 0 (NOT null!) int64 or float64 Dangerous! Silently treats 0 as data.
"" (empty string) Stays as "" object Must manually fix with replace(r'^$', np.nan)
Why this matters: 0 is NOT missing. If your CSV has credit_score column with a 0, pandas treats it as data (valid score). But if it has an empty cell, pandas reads NaN. When loading, always check df.dtypes and df.isnull().sum() first.

Finding Missing Data — 4 Essential Commands

Before you decide what to do, you need to see the problem.

1. Count nulls per column: isnull().sum()

Tell you how many blanks in each field:

📄
Sample data file to follow along
Save next to your .py so pd.read_csv('loan_applications.csv') works.
Find nulls by column
Column Null Count
co_applicant 22
monthly_income 15
pan_number 12
credit_score 10
aadhaar_linked 8
app_id 0
branch 0
What this shows: Out of 80 applications, co_applicant has 22 nulls (28%). That's expected—most loans are single-applicant. But monthly_income has 15 blanks (19%), which is a problem. Your next step: is this random, or clustered by branch/employment type?

2. Nulls as percentage: Calculate the burden

Raw counts don't show severity. A 1-null column in a 100-row dataset is fine. A 1-null column in a 10-row dataset is disaster.

Calculate null percentages
Null percentage by column (SouthBank data)
Null percentages by column with co_applicant at 47%
What this shows: co_applicant at 27.5% and monthly_income at 18.75% are red flags in isolation. But context matters. co_applicant is sparse because many loans are single-applicant—that's a business reality, not a data quality issue. monthly_income being missing for 19% of rows might indicate non-salaried applicants (self-employed, students) who estimate irregular income, or a data collection gap in one branch.

3. Find rows with ANY missing: dropna() preview

How many rows would you lose if you dropped all rows with any null?

Preview the cost of dropna()
The danger: Using df.dropna() blindly would keep only 23 rows (29% of data). Your model would be trained on less than a third. This is why you cannot drop all rows with any null. You must decide column-by-column.

4. Pattern detection: Null by group

Is missingness random, or does it cluster by branch, employment type, etc.?

Are nulls clustered by branch?
Actionable insight: 61.5% of Kochi applications lack PAN. This is a systematic issue (branch policy), not random noise. Your strategy: investigate Kochi's process, or flag that branch separately in your model.

Drop, Fill, or Flag — When to Use Each

Three strategies. Choose based on missingness pattern, data type, and domain knowledge.

1 DROP (remove rows)

When: <2% of rows affected AND missingness is random

Use this for critical fields where missing = unusable. Only safe if you're losing a tiny fraction of rows.

df_clean = df.dropna(subset=['pan_number'])

✓ Simple, keeps only complete records

✗ Wastes rows; biases toward completeness

2 FILL (impute values)

When: Missingness is systematic AND a sensible default exists

Numeric: median/mean by group. Categorical: 'unknown'. Time-series: forward-fill.

# Numeric: median by group
df['monthly_income'].fillna(
  df.groupby('employment_type')
    ['monthly_income'].transform('median')
)

# Categorical
df['co_applicant'].fillna('solo')

✓ Keeps all rows; domain-aware

✗ Introduces artificial patterns if done wrong

3 FLAG (create boolean)

When: Missingness itself is a signal (first-time borrowers, no credit history, etc.)

Create a new column: is_missing_credit_score = True/False. Keep the row.

df['has_credit_score'] = df['credit_score'].notna()
# Now train on both credit_score AND has_credit_score
# Model learns: "missing = likely first-timer"

✓ Preserves signal; keeps all rows

✗ Adds complexity; only works if missingness is meaningful

Before-and-after example: monthly_income

Our 80 applications have 15 missing monthly_income values (19%). These are scattered across employment types. Let's compare strategies:

Strategy 1: Drop (loses 19% of rows)
app_id employment_type monthly_income
APP0001 salaried ₹45,000
APP0003 self-employed ₹72,500
APP0007 salaried ₹58,200
... rows with nulls removed ...
APP0078 student ₹15,000
Result: 65 rows remain (lost 15)
Strategy 2: Fill with median by employment_type
app_id employment_type monthly_income filled?
APP0001 salaried ₹45,000 —
APP0002 salaried ₹52,000 ✓ median
APP0003 self-employed ₹72,500 —
APP0010 student ₹18,500 ✓ median
... 80 rows total, 15 filled with group median ...
Result: 80 rows remain; income filled by employment type
Strategy 3: Flag the missing (create signal column)
app_id monthly_income income_reported
APP0001 ₹45,000 True
APP0002 NaN False
APP0003 ₹72,500 True
APP0010 NaN False
Model now sees: "missing income = likely self-employed or student"
Result: 80 rows remain; missingness encoded as feature

The Imputation Trap — Avoid Artificial Peaks

Filling all nulls with the global median creates a false clustering in your distribution. This biases any downstream model.

Wrong way: Global median

If monthly_income has median ₹62,000, filling all 15 nulls with ₹62,000 creates an artificial spike:

Monthly income distribution: naive fill with global median
Income histogram with artificial spike from naive median fill
The problem: A pile of ₹62,000 incomes that weren't real creates a fake peak. Any clustering algorithm, statistical test, or model that relies on income distribution is now biased. You've created false information.

Right way: Fill by group, then add random jitter (optional)

Fill with the median of the group (salaried, self-employed, etc.), not the global median. Even better, add a small random perturbation so filled values don't create a spike.

Fill missing income by employment type, then add noise
Monthly income distribution: filled by group + noise (realistic)
Income histogram with realistic group-based imputation
Why this works: Filling by employment_type ensures self-employed applicants get realistic income medians. Adding ±10% noise spreads the imputed values so they don't form a visible spike. The distribution stays smooth and realistic, matching the natural variation in your data.

Document Your Decisions — Data Quality Note

Always tell your stakeholders what you did with missing data. Here's a template:

Data Quality Note — SouthBank Loan Applications

Dataset overview: 80 applications, 11 fields.
Missing data:
• Total: 67 rows (84%) contain at least one null.
• co_applicant: 22 nulls (28%) — expected for single-applicant loans; dropped from modeling.
• monthly_income: 15 nulls (19%) — filled with median by employment_type (salaried: ₹58K, self-employed: ₹72K, student: ₹18K).
• pan_number: 12 nulls (15%) — 10 from Kochi branch (policy issue). Dropped rows with null PAN per requirement.
• credit_score: 10 nulls (12%) — flagged with boolean `has_credit_score`; kept for segmentation (likely first-time borrowers).
• aadhaar_linked: 8 nulls (10%) — filled with 'not_verified' (conservative assumption).

Final row count: 68 rows (15% dropped due to PAN requirement).
Assumptions: Filled monthly_income preserves employment-type distribution. Credit score absence treated as data point, not deletion.

Cheatsheet: Null Handling Commands

Command What it does Example
df.isnull() Returns boolean DataFrame: True where null, False where data df.isnull().sum()
df.notna() Inverse of isnull(); True where data exists df[df['income'].notna()]
df.isnull().sum() Count nulls per column df.isnull().sum().sort_values(ascending=False)
df.isnull().any(axis=1) True for rows with any null df[df.isnull().any(axis=1)] # rows with nulls
df.dropna() Remove all rows with any null df.dropna() # usually too aggressive
df.dropna(subset=['col']) Remove rows null in specific column(s) df.dropna(subset=['pan_number'])
df['col'].fillna(value) Fill nulls with scalar value df['category'].fillna('unknown')
df['col'].fillna(df['col'].median()) Fill nulls with median df['income'].fillna(df['income'].median())
df.groupby('x')['col'].transform(lambda x: x.fillna(x.median())) Fill by group (e.g., median by department) df.groupby('dept')['salary'].transform(lambda x: x.fillna(x.median()))
df['col'].fillna(method='ffill') Forward fill (time-series): use previous value df['price'].fillna(method='ffill')
df['col'].fillna(method='bfill') Backward fill (time-series): use next value df['price'].fillna(method='bfill')
df['was_null'] = df['col'].isnull() Create boolean flag for nulls df['no_credit_score'] = df['credit'].isnull()
df.interpolate() Linear interpolation between values (time-series) df['temperature'].interpolate(method='linear')
df.dropna(thresh=n) Drop rows with fewer than n non-null values df.dropna(thresh=8) # keep rows with ≥8 non-nulls

Mini Challenge

Profile Missing Data in Your Own Dataset

Difficulty: Easy (5 min)

Task:

  1. Load loan_applications.csv (link above).
  2. Run df.isnull().sum().sort_values(ascending=False) — which 3 columns have the most nulls?
  3. Calculate null % for each column: (df.isnull().sum() / len(df) * 100).round(1).
  4. Check how many rows would be lost with df.dropna().
  5. Write a 4-line data quality note documenting your findings.

Expected output:

co_applicant 27.5%
monthly_income 18.75%
pan_number 15.0%
...
Rows with nulls: 57 / 80 (71.3%)
Safe drop threshold: ~2 rows

Common Pitfalls — Avoid These

1. Treating NaN as 0

Missing doesn't mean zero. A null credit_score is not a bad score of 0—it's unknown. Using 0 in statistical formulas biases medians and means downward.

2. Dropping all rows with any null

df.dropna() keeps only complete records. In most real datasets, this loses >50% of rows. Be surgical: drop only critical columns.

3. Mean/median imputation without grouping

Filling all income nulls with ₹62K (global median) creates artificial clustering. Always fill by group (by employment_type, branch, etc.).

4. Not documenting what you dropped

Your stakeholder asks: "Why did we lose 15 applications?" If you didn't document your null-handling decisions, you can't answer. Write it down.

5. Comparing with == to NaN

df['income'] == np.nan returns False always. Use df['income'].isnull() instead.

Interview Questions

What's the difference between dropna() and fillna()? When would you use each? easy +

Answer:

dropna() removes rows (or columns) with nulls, reducing your dataset size. Use it when:

  • Missing data is <2% of rows (safe to lose)
  • The column is critical and has no sensible default

fillna() replaces nulls with a value, keeping all rows. Use it when:

  • A reasonable default exists (median for numeric, 'unknown' for categorical)
  • You want to preserve row count for regression/classification

Example: In a loan dataset, drop rows missing required identity (PAN), but fill missing income with group median.

You have 100,000 rows. One column is 85% null. What do you do? easy +

Answer: 85% null is too high for any strategy:

  • Drop the column. You'd be filling 85,000 values with guesses. The noise would overwhelm any signal.
  • Check if the column was optional. If yes, remove it entirely from your model.
  • If no, escalate: is there a data collection issue? Was this column always null?

Rule of thumb: If >50% null, strongly consider dropping. If 20–50%, investigate the pattern before deciding.

Your model's feature importance shows the 'is_null_flag' is highly predictive. Why might that be? Is it a problem? medium +

Answer: It could be a feature or a problem, depending on context:

Positive scenario: Missing credit_score is a genuine signal. Applicants without a credit history (students, first-time borrowers) are a distinct segment. The model learned: "no credit score = higher risk." This is valuable.

Negative scenario: Missing data is due to a data collection error, and the model is exploiting that error. Example: all rows missing income are from one date range when the system was down. The model learns the date pattern, not the actual business logic.

How to tell: Investigate the nulls. Are they systematic (by group, time, branch) or random? If systematic and meaningful, keep the flag. If due to error, fix the data collection.

You fill missing income with median by employment type. Your colleague says "Why not just use mean?" How do you respond? medium +

Answer: Median is more robust to outliers:

  • Mean: A few high-earners (outliers) pull the average up. Fill all missing with the skewed mean? Unrealistic.
  • Median: The middle value, immune to extremes. More representative of a typical applicant.

Example: 10 salaries: ₹40K, ₹45K, ₹48K, ₹50K, ₹52K, ₹55K, ₹58K, ₹62K, ₹68K, ₹2M (CEO error?)

  • Mean = ₹249K (wildly inflated by the CEO)
  • Median = ₹53.5K (realistic for most employees)

Use median unless you have a reason to believe the distribution is symmetric (rarely the case in real data).

You train a model on imputed data, then evaluate it on a test set with the original nulls still present. Will your model perform well in production? hard +

Answer: No. You've created a train-test mismatch.

The problem:

  • Train set: No nulls (all filled with medians). The model learns on smooth, imputed data.
  • Test set: Still has nulls. The model sees data it's never encountered.
  • Production: Also has nulls. The model will fail or make poor predictions.

The fix: Apply the SAME imputation pipeline to train, test, AND production:

1. Fit imputation on train data (e.g., calculate median)
2. Apply to train, test, and production
3. Never fit imputation on test/production

This is why scikit-learn's Pipeline is valuable—it bundles imputation and modeling so they're applied consistently.

0/5
Quiz Progress
Q1: What does df.isnull().sum() return?
A boolean DataFrame showing True/False for each cell
A Series with null count per column
The total count of nulls in the entire DataFrame
A cleaned DataFrame with no nulls
Q2: You have 1,000 rows. One column is 23% null (230 rows). Should you drop all 230 rows?
Yes, drop them immediately to clean the data
No, investigate the pattern first. Maybe fill by group or flag the nulls.
Yes, but keep rows where multiple columns are null
Drop just that column, not the rows
Q3: You fill all missing income values with the global median (₹55K). What's the risk?
No risk; the median is always safe
You create an artificial spike at ₹55K in the distribution, biasing downstream models
You lose the imputed rows when you run dropna() again
The median becomes invalid for future analysis
Q4: Which code correctly fills missing income with the median by employment type?
df['income'].fillna(df['income'].median())
df['income'] = df.groupby('employment_type')['income'].transform(lambda x: x.fillna(x.median()))
df.groupby('employment_type')['income'].fillna(df['income'].median())
df['income'].fillna(method='median')
Q5: You notice credit_score is missing for 15% of applicants, all of whom are students. What's the best strategy?
Drop all 15%; students don't matter
Create a boolean flag 'has_credit_score' and keep the rows; missing = first-timer signal
Fill with the median credit score for all applicants
Drop just the credit_score column entirely

Also Used In

Telecom (CDR analysis)

Missing call details (dropped calls, no network) are signal, not garbage. Flag them rather than drop.

E-commerce (checkout funnels)

Missing payment_method = abandoned checkout. Treat as a behavioral segment, not a data error.

Public health (clinic data)

Missing immunization_date may indicate a genuine gap in coverage, not a recording error. Document and report it.

Why This Matters for AI/ML

Training data quality is bottleneck #1. Large language models and classification systems fail silently on biased, incomplete data. When you fill missing values with global medians, you're teaching the model a false pattern. When you drop 70% of rows, you're training on a non-representative sample.

Feature engineering teams spend 40% of time on missing data handling. The pandas techniques here—groupby imputation, flagging nulls—are the foundation of production ML pipelines. Learn them now; you'll use them constantly.

Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Ready to clean your first dataset?

Join the Best AI Program: The AI-Era Analyst — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Analyst →