Wanjiku is a data scientist at TelcoMax, a telecom company in Nairobi. The CEO asked: "We lose 23% of customers every quarter. Can data science help?" Before she trains any machine learning model, she must understand the data statistically. What do customer ages look like? Which features predict churn? Are the differences significant, or just noise?
She loads 5,000 customer records and starts asking questions. Are tenure and total charges correlated? Do month-to-month contracts churn more than yearly contracts? She uses pandas describe(), correlation matrices, and group analysis to turn raw data into business insights.
Step 1: Explore the data with pandas
The first step in any analysis is to load the data and peek at its shape, types, and content. Here's how Wanjiku starts:
| age | tenure_months | monthly_charges | total_charges | contract | churn | |
|---|---|---|---|---|---|---|
| 0 | 45 | 12 | 65.5 | 786 | Month-to-month | 1 |
| 1 | 52 | 61 | 89.2 | 5446 | One year | 0 |
| 2 | 38 | 8 | 55.1 | 441 | Month-to-month | 1 |
| 3 | 69 | 71 | 110.0 | 7810 | Two year | 0 |
| 4 | 41 | 19 | 72.5 | 1378 | Month-to-month | 1 |
Step 2: Understand distributions with describe()
The describe() method gives you mean, std, min, max, and quartiles in one table. This is where most insights start:
| age | tenure_months | monthly_charges | |
|---|---|---|---|
| count | 5000.000 | 5000.000 | 5000.000 |
| mean | 46.281 | 36.076 | 70.224 |
| std | 16.471 | 20.652 | 28.745 |
| min | 18.000 | 1.000 | 20.010 |
| 25% | 32.000 | 18.000 | 45.648 |
| 50% | 46.000 | 36.000 | 70.525 |
| 75% | 61.000 | 54.000 | 94.728 |
| max | 74.000 | 71.000 | 119.990 |
Value counts: categorical distributions
For categorical columns like contract type and churn, use value_counts() to see the distribution:
| Value | Count |
|---|---|
| contract | |
| Month-to-month | 2553 |
| One year | 1483 |
| Two year | 964 |
| churn | |
| No churn (0) | 3858 |
| Churn (1) | 1142 |
Step 3: Correlations—which features link to churn?
Correlation measures linear relationships between numeric columns, from -1 (perfect negative) to +1 (perfect positive). For predicting churn, strong correlations are gold:
| tenure_months | monthly_charges | total_charges | |
|---|---|---|---|
| tenure_months | 1.000 | 0.028 | 0.771 |
| monthly_charges | 0.028 | 1.000 | 0.539 |
| total_charges | 0.771 | 0.539 | 1.000 |
Step 4: Group comparisons—where's the real insight?
Raw statistics hide group differences. Wanjiku compares churners vs. non-churners, and contract types:
| churn | tenure_months (avg) | monthly_charges (avg) |
|---|---|---|
| 0 (No churn) | 37.3 | 69.87 |
| 1 (Churn) | 31.8 | 71.42 |
| contract | Churn count | Total | Churn rate |
|---|---|---|---|
| Month-to-month | 842 | 2553 | 0.329 |
| One year | 162 | 1483 | 0.109 |
| Two year | 138 | 964 | 0.146 |
Step 5: Hypothesis testing—is the difference real?
Differences in group means could be luck or real. Hypothesis testing uses a p-value to decide. A p-value below 0.05 means "this difference is statistically significant (only 5% chance it's random)":
Step 6: Skewness—is the distribution balanced?
Skewness measures whether a distribution is symmetric (skew ≈ 0) or lopsided. Right-skew means a tail on the right; left-skew means a tail on the left:
Also used in...
Domain Spotlight
How AI systems use this
Feature selection in machine learning starts with correlation analysis. A feature with 0.77 correlation to the target is a strong predictor. Models like logistic regression weight such features heavily. Before any deep learning model trains, engineers describe() the dataset to catch distribution shifts and data quality issues. Every production ML system checks p-values to ensure model improvements are statistically real, not noise.
Task: Load telco_churn.csv. Compute the correlation between churn and all numeric columns (age, tenure_months, monthly_charges, total_charges). Which feature correlates strongest with churn? (Hint: you'll need to convert churn to numeric first: df['churn'].astype(int))
Hint: Use df[['age', 'tenure_months', 'monthly_charges', 'total_charges', 'churn']].corr() after converting churn to int. Look at the 'churn' row.
Interview questions: test your understanding
Quiz: 5 questions to test your knowledge
Cheatsheet: statistics methods at a glance
| Stat | Pandas method | When to use |
|---|---|---|
| Mean | df['col'].mean() |
Symmetric data |
| Median | df['col'].median() |
Skewed data or outliers |
| Std deviation | df['col'].std() |
Measure spread around mean |
| Correlation | df.corr() |
Linear relationships |
| Value counts | df['col'].value_counts() |
Categorical distribution |
| Summary | df.describe() |
Quick overview of numerics |
| Skewness | df['col'].skew() |
Check distribution symmetry |
| Group mean | df.groupby('x')['y'].mean() |
Compare groups |
Common mistakes: what trips people up
df['salary'].median()df.isnull().sum() first.Key takeaways
- Start every analysis with
df.describe()anddf.info()—understand your data before modeling. - Correlation 0.77 is strong and signals a powerful predictor; 0 means no linear link; -0.5 means negative.
- Group comparisons often reveal more than raw statistics. Month-to-month churn (32.9%) vs yearly (10.9%) is the real insight.
- p-values < 0.05 mean statistically significant. The tenure difference between churners and non-churners is real, not random.
- Skewness near 0 = symmetric, good for most models. Right-skew requires attention (log transforms, robust scalers).
- Feature selection in ML begins with correlation analysis and group comparisons, not guessing.