📞
Real-world scenario
Wanjiku at TelcoMax (Nairobi): exploring churn before modeling

Wanjiku is a data scientist at TelcoMax, a telecom company in Nairobi. The CEO asked: "We lose 23% of customers every quarter. Can data science help?" Before she trains any machine learning model, she must understand the data statistically. What do customer ages look like? Which features predict churn? Are the differences significant, or just noise?

She loads 5,000 customer records and starts asking questions. Are tenure and total charges correlated? Do month-to-month contracts churn more than yearly contracts? She uses pandas describe(), correlation matrices, and group analysis to turn raw data into business insights.

5,000
Customers
22.8%
Churn rate
0.771
Max correlation

Step 1: Explore the data with pandas

The first step in any analysis is to load the data and peek at its shape, types, and content. Here's how Wanjiku starts:

explore.pyPython
Output: df.head() shows
age tenure_months monthly_charges total_charges contract churn
0 45 12 65.5 786 Month-to-month 1
1 52 61 89.2 5446 One year 0
2 38 8 55.1 441 Month-to-month 1
3 69 71 110.0 7810 Two year 0
4 41 19 72.5 1378 Month-to-month 1

Step 2: Understand distributions with describe()

The describe() method gives you mean, std, min, max, and quartiles in one table. This is where most insights start:

describe.pyPython
Output
age tenure_months monthly_charges
count 5000.000 5000.000 5000.000
mean 46.281 36.076 70.224
std 16.471 20.652 28.745
min 18.000 1.000 20.010
25% 32.000 18.000 45.648
50% 46.000 36.000 70.525
75% 61.000 54.000 94.728
max 74.000 71.000 119.990
What this tells us: Average customer age is 46.3 years, with tenure centered at 36 months. Monthly charges average 70.22. The standard deviations (16.5, 20.7, 28.7) show good spread—no features are heavily concentrated.

Value counts: categorical distributions

For categorical columns like contract type and churn, use value_counts() to see the distribution:

counts.pyPython
Output
Value Count
contract
Month-to-month 2553
One year 1483
Two year 964
churn
No churn (0) 3858
Churn (1) 1142

Step 3: Correlations—which features link to churn?

Correlation measures linear relationships between numeric columns, from -1 (perfect negative) to +1 (perfect positive). For predicting churn, strong correlations are gold:

corr.pyPython
Output
tenure_months monthly_charges total_charges
tenure_months 1.000 0.028 0.771
monthly_charges 0.028 1.000 0.539
total_charges 0.771 0.539 1.000
Key finding: Tenure and total charges have a strong positive correlation (0.771)—longer customers have paid more. Monthly charges and total charges are moderate (0.539). For a churn model, tenure_months will be a strong predictor.

Step 4: Group comparisons—where's the real insight?

Raw statistics hide group differences. Wanjiku compares churners vs. non-churners, and contract types:

groups.pyPython
Output: Tenure & charges by churn
churn tenure_months (avg) monthly_charges (avg)
0 (No churn) 37.3 69.87
1 (Churn) 31.8 71.42
Churn rate by contract type
contract Churn count Total Churn rate
Month-to-month 842 2553 0.329
One year 162 1483 0.109
Two year 138 964 0.146
The insight that changes everything: Churners have shorter tenure (31.8 vs 37.3 months). But the contract column is explosive: month-to-month customers churn at 32.9%, while yearly contracts churn at only 10.9%. A single feature can drive your model.

Step 5: Hypothesis testing—is the difference real?

Differences in group means could be luck or real. Hypothesis testing uses a p-value to decide. A p-value below 0.05 means "this difference is statistically significant (only 5% chance it's random)":

ttest.pyPython
Output
t-statistic: -8.456 p-value: 2.34e-16
Conclusion: With p < 0.001, the tenure difference is not random. Churners truly have shorter relationships with TelcoMax. This is a real signal for the model.

Step 6: Skewness—is the distribution balanced?

Skewness measures whether a distribution is symmetric (skew ≈ 0) or lopsided. Right-skew means a tail on the right; left-skew means a tail on the left:

skew.pyPython
Output
age 0.008 tenure_months 0.008 monthly_charges -0.010
Interpretation: Skewness values near 0 mean symmetric distributions. Our features are well-balanced—no heavy tails. This is good: no need for special handling (like log transforms).

Also used in...

Domain Spotlight

🏦
Banking: default prediction
Loan defaults are rare (5% churn). Group by income, collateral, loan term to find risk factors.
🛒
E-commerce: customer retention
Correlation between order frequency, cart value, and repeat purchases. Skewness in purchase amounts.
🏥
Healthcare: readmission risk
Hospital stays show right-skewed length. Correlate age, comorbidities with readmission.

How AI systems use this

Feature selection in machine learning starts with correlation analysis. A feature with 0.77 correlation to the target is a strong predictor. Models like logistic regression weight such features heavily. Before any deep learning model trains, engineers describe() the dataset to catch distribution shifts and data quality issues. Every production ML system checks p-values to ensure model improvements are statistically real, not noise.

Feature Selection Data Quality Statistical Testing Bias Detection
Mini Challenge: Find the strongest churn predictor

Task: Load telco_churn.csv. Compute the correlation between churn and all numeric columns (age, tenure_months, monthly_charges, total_charges). Which feature correlates strongest with churn? (Hint: you'll need to convert churn to numeric first: df['churn'].astype(int))

Hint: Use df[['age', 'tenure_months', 'monthly_charges', 'total_charges', 'churn']].corr() after converting churn to int. Look at the 'churn' row.

Interview questions: test your understanding

Count, mean, standard deviation, min, 25th/50th/75th percentiles, and max. A 7-number summary that captures the distribution.
No linear relationship between two variables. They move independently of each other.
When the data is skewed or has outliers. The median is robust—a million-unit outlier doesn't shift it like it would the mean.
There's a 3% chance the observed difference happened by random chance alone. Since 3% < 5%, the difference is statistically significant.
Probably not. High correlation (multicollinearity) means the features carry redundant information. Drop one and retrain. Your model will be simpler and more interpretable.

Quiz: 5 questions to test your knowledge

Question 1 of 5
Correlation 0.771 between tenure and total charges means?
Question 2 of 5
Which contract type has the highest churn rate?
Question 3 of 5
Skewness near 0 means the distribution is?
Question 4 of 5
Median tenure in the dataset is?
Question 5 of 5
A p-value < 0.05 means?
0
/5
Great job! You understand statistical analysis for ML.

Cheatsheet: statistics methods at a glance

Stat Pandas method When to use
Mean df['col'].mean() Symmetric data
Median df['col'].median() Skewed data or outliers
Std deviation df['col'].std() Measure spread around mean
Correlation df.corr() Linear relationships
Value counts df['col'].value_counts() Categorical distribution
Summary df.describe() Quick overview of numerics
Skewness df['col'].skew() Check distribution symmetry
Group mean df.groupby('x')['y'].mean() Compare groups

Common mistakes: what trips people up

Mistake 1: Confusing correlation with causation
Month-to-month contracts have short tenure AND high churn. Correlation = 0.5. But does contract type CAUSE churn, or do already-churning customers choose M2M? Experiment to prove causation.
Mistake 2: Using mean on skewed data
If salary data has a few executives earning 10x the median, mean is inflated. Use median instead: df['salary'].median()
Mistake 3: Ignoring missing values in correlation
Pandas drops NaN silently when computing correlations. A column with 20% missing becomes a feature on only 80% of data. Check df.isnull().sum() first.
Mistake 4: Looking only at overall stats, not per-group
Overall churn = 22.8% hides that M2M is 32.9%. Always groupby categorical features to find the real signal.

Key takeaways

  • Start every analysis with df.describe() and df.info()—understand your data before modeling.
  • Correlation 0.77 is strong and signals a powerful predictor; 0 means no linear link; -0.5 means negative.
  • Group comparisons often reveal more than raw statistics. Month-to-month churn (32.9%) vs yearly (10.9%) is the real insight.
  • p-values < 0.05 mean statistically significant. The tenure difference between churners and non-churners is real, not random.
  • Skewness near 0 = symmetric, good for most models. Right-skew requires attention (log transforms, robust scalers).
  • Feature selection in ML begins with correlation analysis and group comparisons, not guessing.