💹
Your context
Rebecca — Payments processor analyst

You're analyzing merchant health KPIs from a global payments processor. You've just calculated the correlation between support ticket volume and merchant churn rate: r = 0.82 (strong positive). Your instinct: "More support tickets cause churn!" But is that right? Maybe merchants struggling technically contact support more before leaving. Or maybe both are signals of a deeper merchant quality issue. The correlation is real, but the causation story needs evidence. This lesson teaches you how to spot the difference—and avoid billion-rupee mistakes in product decisions.

What Correlation Is (and Isn't)

Correlation quantifies how two variables move together. When one goes up, does the other tend to go up too? Down? Not at all?

Average Ticket Size vs Gross Merchandise Value (₹)
Scatter plot of ticket size vs GMV showing weak positive correlation
What this shows: The upward slope tells us: as average ticket size grows, GMV tends to grow too. Two variables moving in sync. Correlation captures this pattern in a single number: r.

The word "correlation" does three things

1. Describes direction: positive (both go up), negative (one up, one down), or zero (no pattern).
2. Describes strength: how tightly the points cluster around an imaginary line.
3. Does NOT say one causes the other. That is the trap.

The -1 to +1 Scale: Understanding r

Correlation is always between -1 and +1. Here's what the landmarks mean:

Four correlation scenarios (2×2 grid)
The Correlation Scale (-1 to +1) r = +0.95 (Strong Positive) r = +0.3 (Weak Positive) r ≈ 0 (No Correlation) r = -0.8 (Strong Negative) +1 = perfect upward line 0 = no pattern -1 = perfect downward line Most real data: -0.5 to +0.5
What this shows: r = +0.95: tight cluster around upward line. r = +0.3: loose cluster, hard to see pattern. r ≈ 0: cloud with no slope. r = -0.8: tight cluster downward. The farther from 0, the stronger the relationship.

The `.corr()` Matrix: See All Relationships at Once

In pandas, one line shows you how every numeric column correlates with every other:

Output: A table where each cell is an r value. Here's the merchant data:

Correlation matrix heatmap: purple=positive, white=neutral, amber=negative
Correlation matrix heatmap showing relationships between merchant KPIs
What to read: Look for the strongest r values (darkest purple or warmest amber). refund_rate ↔ chargeback_rate: r = 0.66 (both are quality signals). support_tickets ↔ churned: r = 0.31 (related, but not perfectly predictive). monthly_txn_count ↔ gmv_inr: r = 0.21 (weak—because GMV depends on ticket size too).
📥 Download sample data: merchant_kpis.csv (180 merchants, 9 columns)

Pearson vs Spearman: When to Use Each

There are two main correlation formulas. They give different answers when the relationship is non-linear.

Pearson (default in pandas)

Assumes a linear relationship. If your data has a curve or U-shape, Pearson can give misleading r values. Example: support ticket count vs. merchant quality follows a U (bad merchants and top performers both contact support often).

Spearman

Compares ranks, not raw values. A merchant with 5 support tickets ranks higher than one with 2. It doesn't care about the exact distance. Perfect for spotting monotonic (always increasing or always decreasing) relationships.

U-shaped data: Pearson misses it, Spearman catches it
U-Shaped Relationship Pearson r 0.08 "Almost no relationship" (but there IS a pattern!) Spearman r 0.72 "Strong relationship"
When to use each: For linear data (most fintech KPIs), Pearson is fine. For rank-based or curved relationships, use Spearman. Easy rule: calculate both, compare. If they differ wildly, your relationship is non-linear—Spearman is more honest.

The Causation Trap: Three Ways Correlation Lies

A strong correlation does NOT mean one variable causes the other. Here are the three reasons why:

🔗 Confounding Variable

Both variables are caused by something else. Example: Ice cream sales ↔ drowning deaths (r ≈ 0.95). Does ice cream cause drowning? No—summer causes both. Smart merchants have high GMV AND low churn (both caused by good operational practices), not one causing the other.

↔️ Reverse Causation

You've got the arrow backwards. Do support tickets cause churn? Or do struggling merchants contact support more often before leaving? Rebecca's data shows r = 0.31, but the direction is ambiguous from correlation alone.

🎲 Pure Coincidence

Two unrelated things drift together randomly. Test 100 random variables against each other; some will correlate by chance (p-hacking). Real example: Pirates 🏴‍☠️ vs. Global warming, r = -0.97 over 300 years. Same ship? No—pure statistical noise.

⚠️ Causation requires: Correlation + mechanism + controlled experiments. Correlation alone tells you nothing. A product team that cuts support tickets to reduce churn (based on r = 0.31 alone) will make merchants angrier and churn will climb.

When Correlation IS Useful (Despite Its Limits)

Correlation is not causation, but it's not useless either. Use it for these three things:

✓ Feature Selection

When building a model, drop features that are highly correlated with each other (r > 0.9). They're redundant and inflate variance. Keep the one that makes business sense.

✓ Quick Sanity Check

Before you model or experiment, check: does my hypothesis show *any* correlation? If GMV and churn are uncorrelated (r ≈ 0), your theory is probably wrong. Correlation is a necessary, not sufficient, condition.

✓ Anomaly Detection

Track correlation over time. If support_tickets ↔ churned suddenly breaks from r = 0.31 to r = -0.05, something changed in your product or merchant base. Worth investigating.

Quick Decision Flowchart

Should I use this correlation?
Correlation Decision Flow Found r = 0.7+? Do you have a testable causal mechanism? YES Run experiment or A/B test NO Don't assume causation Use for: feature selection, redundancy checks, sanity tests

Test Your Understanding

Three questions to lock in what you've learned.

Question 1 of 3: What does r = -0.85 mean?

Select the best interpretation.

Question 2 of 3: When do you prefer Spearman over Pearson?

Select all that apply.

Question 3 of 3: Two variables have r = 0.92. What can you conclude?

Choose the most accurate statement.

Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Learn Regression (Next Step)

Join the Best AI Program: The AI-Era Analyst — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Analyst →