🏥
Real-world scenario
Arjun (SafeGuard Insurance, Bangalore): age+BMI alone give R²=0.23. Can polynomial features help?

Arjun is an actuary at SafeGuard Insurance in Bangalore. His linear regression on age and BMI explains only 23% of variance in medical charges. He wonders: should he try polynomial features? Degree 2? Degree 5? If he adds too many terms, will the model overfit on training data and fail on new customers?

The answer teaches a critical lesson: more features don't always mean better predictions. Adding smoker status (a single binary column) jumped R² from 0.23 to 0.96. But squaring age and BMI barely helped. Arjun learns that feature selection beats feature engineering—and how to spot overfitting before it breaks the model.

1,338
Policies
5 → 20
Features (deg 2 → deg 5)
0.24 → 0.96
R² (age+bmi → all features)

Can we do better than R²=0.23?

Linear regression on age and BMI leaves a lot of variance unexplained. Many relationships in the real world aren't perfectly linear. Maybe age and charges have a curve—charges rise slowly in youth, then accelerate. Let's try polynomial features.

The idea: instead of just age and BMI, create new features like age², BMI², and age×BMI (interactions). More features might capture non-linear patterns. But we'll discover a trap: more features can lead to overfitting, where the model memorizes training noise and fails on new data.

What are polynomial features?

Polynomial features expand your original features into higher-degree terms. Starting with two features (age, BMI), here's what each degree creates:

Degree 1 (baseline): age, bmi → 2 features
Degree 2: age, bmi, age², bmi², age×bmi → 5 features
Degree 3: plus age³, bmi³, age²×bmi, age×bmi² → 9 features
Degree 5: every combination up to fifth power → 20 features

Notice: the number of features grows fast. With just 2 input features, degree 5 creates 20 features. With 50 input features, degree 3 creates over 22,000 features. This is why sklearn's PolynomialFeatures is so dangerous—it's easy to create more features than rows, guaranteeing overfitting.

The experiment: degrees 1 through 5

Arjun tests polynomial degrees 1–5 on age and BMI (he excludes smoker for now to see the pure effect of polynomial expansion). Here are the tested results:

polynomial_experiment.py Compare train vs test R² across degrees
Output
Degree 1: 2 features | Train R²=0.1900 | Test R²=0.2308 Degree 2: 5 features | Train R²=0.1933 | Test R²=0.2357 Degree 3: 9 features | Train R²=0.1958 | Test R²=0.2313 Degree 4: 14 features | Train R²=0.1987 | Test R²=0.2243 Degree 5: 20 features | Train R²=0.2081 | Test R²=0.2177 Best test R²: Degree 2 with 5 features → R²=0.2357
Key insight: Train R² keeps climbing (0.19 → 0.21) while test R² peaks at degree 2 and drops. This is overfitting in action.

Visualizing overfitting: train vs test divergence

Look at what's happening:

Train R² vs Test R² across polynomial degrees (age + BMI). Degree 5 overfits — test R² drops.
Train vs Test R² showing overfitting at degree 5
Polynomial fits on age vs charges: linear (degree 1), quadratic (degree 2), and overfitting (degree 5).
Three polynomial curve fits showing underfitting to overfitting

See the pattern? The purple line (train R²) steadily climbs—the model fits training data better with each degree. But the orange line (test R²) peaks at degree 2 and falls. The gap between them is overfitting: degree 5 has memorized training noise that doesn't generalize.

The bias-variance tradeoff: too simple vs too complex

This is a core ML principle. Every model faces a tradeoff:

Underfitting (high bias): Model too simple, misses real patterns. Both train and test R² are low (0.19 for degree 1).
Sweet spot (degree 2): Model captures signal, ignores noise. Test R²=0.24 is best we achieve on new data.
Overfitting (high variance): Model too complex, memorizes noise. Train R² keeps improving but test R² drops (degree 5).

The goal is the sweet spot—just complex enough to learn real patterns, but not so complex that it memorizes training quirks.

The real lesson: feature selection beats feature engineering

Here's the stunning result: with age and BMI alone, the best we can achieve is R²=0.24, even with degree 5 polynomial expansion. But when Arjun adds one more feature—smoker status (binary: yes/no)—watch what happens:

add_smoker_feature.py Compare: polynomial vs single good feature
Output
Degree 5 polynomial (age + BMI only): R² = 0.2177 Linear model (age + BMI + smoker): R² = 0.7392 Improvement from adding smoker: 52.1 percentage points

Wow. Adding a single binary feature is 52 percentage points better than expanding into 20 polynomial terms. This teaches the hardest lesson in machine learning: a good feature is worth more than a fancy algorithm. And when you include all features (age, BMI, smoker, children, sex, region), you hit R²=0.96 with just degree-1 (linear) terms.

Using PolynomialFeatures in sklearn

Here's how to safely use polynomial features without overfitting:

safe_polynomial_usage.py Fit on train, transform test (no leakage)
Output
Train R²: 0.1933 Test R²: 0.2357 Features created: 5 Feature names: ['age' 'bmi' 'age^2' 'bmi^2' 'age bmi']

Key rules: (1) Always split before any feature engineering. (2) Fit transformers on train only. (3) Transform test with the fitted transformer. This prevents data leakage and gives honest estimates of test performance.

Multiple regression: more features, but only good ones

Multiple regression adds more original features (not transformations) to your model. With all SafeGuard features:

Feature Type Example Values age Numeric 18, 35, 64 bmi Numeric 15.9, 28.5, 42.1 children Numeric 0, 1, 2, 3, 4, 5 smoker Binary (yes/no) yes (1), no (0) sex Categorical (M/F) male, female region Categorical (4 regions) northeast, northwest, southeast, southwest

When you include all these features with proper encoding (smoker as 0/1, sex as 0/1, region as one-hot), linear regression (degree 1) achieves R²=0.96. No polynomial tricks needed.

Adding the right features matters more than polynomial degree. Smoker alone jumps R² from 0.23 to 0.96.
Bar chart comparing R² scores as features are added progressively
The lesson: before adding polynomial terms, check if you have the right features. One binary variable (smoker) was worth more than all polynomial degrees combined.

Also used in...

Domain Spotlight

🏥
Healthcare: Non-linear age-risk curves
Charge risk may grow quadratically with age (age²). But adding condition type (heart disease, diabetes) often beats polynomial expansion.
📊
Finance: Diminishing returns in investment
Returns on investment may follow quadratic or cubic curves. A single feature—market sector—often explains more variance than polynomial terms.
🌾
Agriculture: Fertilizer yield response
Crop yield vs fertilizer amount is non-linear (quadratic). But rainfall data usually adds more predictive power than adding fertilizer².

How AI systems use this

GPT and modern neural networks don't use polynomial features—they're too explicit. Instead, neural networks learn non-linear relationships automatically through activation functions (ReLU, sigmoid, tanh). But the principle is the same: overfitting is the enemy. That's why neural networks use dropout (randomly turning off neurons), batch normalization, and regularization (L1/L2 penalties). You're learning the same bias-variance tradeoff that powers every AI system.

Overfitting Feature Engineering Model Complexity Generalization
🎯 Mini Challenge: polynomial + smoker interaction

Task: Train a degree-2 polynomial model on age and BMI with smoker status included. (Hint: add smoker as a numeric feature before PolynomialFeatures.) Does the test R² improve vs degree-2 without smoker? By how much? Why is the interaction term age×smoker meaningful for insurance?

Hint 1: Create a dataframe with columns: age, bmi, smoker (as 0/1)

Hint 2: Use PolynomialFeatures(degree=2).fit_transform() on all three columns

Hint 3: The interaction term age×smoker appears in the polynomial expansion. It captures: "smokers' charges accelerate with age differently than non-smokers'"

Hint 4: Compare test R² with the earlier degree-2 (age+bmi only) result: 0.2357

Interview questions: test your understanding

Creates 5 features: the original two (age, bmi), their squares (age², bmi²), and their interaction (age×bmi). It adds all terms up to degree 2.
The model memorizes training data patterns (including noise) and fails on new data.
Borderline. The gap is small, but with 20 features for 1338 rows, we're at risk (feature-to-row ratio too high). The real sign is that degree 2 (only 5 features) gets better test R² (0.24) with fewer parameters—a more parsimonious model wins.
Because the true relationship is mostly linear at these scales, and the dominant predictor (smoker status) is binary—polynomial terms on binary variables don't add information. Arjun chose the wrong features to expand.
Degree 3 on 50 features creates roughly 22,100 features—way more than rows. The model will overfit catastrophically. You need regularization (Ridge/Lasso, lesson 8) or feature selection first. Never expand features without considering the feature-to-row ratio.

Quiz: 5 questions to test your knowledge

Question 1 of 5
PolynomialFeatures(degree=2) on 2 input features creates how many output features?
Question 2 of 5
In our experiment, which degree gave the best test R²?
Question 3 of 5
Train R² keeps improving but test R² drops = ?
Question 4 of 5
Arjun's R² jumped from 0.23 to 0.74 by adding which feature?
Question 5 of 5
The key lesson: feature _____ beats feature _____.
0
/5
Excellent! You understand polynomial regression and overfitting.

Cheatsheet: polynomial & multiple regression at a glance

Concept Code / Formula Key Idea
PolynomialFeatures PolynomialFeatures(degree=2) Adds x², x₁x₂ terms; grows fast with degree
Fit on train only poly.fit_transform(X_train) Prevents data leakage
Transform test poly.transform(X_test) Use fitted transformer, no new fit
Feature count formula (n+d)! / (n!d!) n=inputs, d=degree; grows exponentially
Overfitting sign train R² >> test R² Model memorized training noise
Sweet spot Best test R², not best train R² Optimal complexity for new data
Feature selection Add informative features, not expansions Good feature > many polynomial terms

Common mistakes: what breaks and how to fix it

Mistake 1: Using too high a polynomial degree without checking test score
Easy to create 1000s of features. Always compare test R² across degrees and pick the one with best test performance, not highest train R². Fix: make a table of degree vs test R², plot it, choose the peak.
Mistake 2: Forgetting to fit_transform on train, transform-only on test
If you fit the polynomial transformer on test data, you leak information and inflate test scores. Fix: poly.fit_transform(X_train), then poly.transform(X_test). Same transformer, two different method calls.
Mistake 3: Not scaling before polynomial expansion
Large features get huge when squared (age²=4096, charges²=billions). This breaks numerical stability. Fix: StandardScaler first, then PolynomialFeatures.
Mistake 4: Confusing "more features" with "better model"
Our best test R² (0.24) came from 5 features (degree 2), not 20 (degree 5). Arjun's real breakthrough was adding smoker (1 feature), not expanding polynomials. Fix: always validate on test data and choose the simpler model if test R² is equal.

Key takeaways

Polynomial features seem powerful, but they're a double-edged sword.

  • Polynomial expansion grows exponentially. Two features, degree 3 → 9 features. Fifty features, degree 3 → 22,100 features. Watch your feature-to-row ratio.
  • Overfitting is your enemy. When train R² keeps improving but test R² drops, the model is memorizing noise. Compare test R² across degrees, not train R².
  • The bias-variance tradeoff is real. Simple models underfit (high bias). Complex models overfit (high variance). The sweet spot is the test score peak.
  • Feature selection beats feature engineering. A single good feature (smoker) added 52 points to R². Twenty polynomial terms added 0.01 points. Choose your features wisely.
  • Always split before preprocessing. Fit transformers on train only, transform test with the fitted transformer. No leakage.
  • Most real problems don't need polynomials. A linear model with the right features (lesson 6 achieved R²=0.96) beats polynomial tricks every time.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Ready to master regression at scale?

Join the Best AI Program: The AI-Era Analyst — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Analyst →