Wanjiku works at TelcoMax, a telecom company in Nairobi. She has a dataset of 5,000 customers with 150 missing values scattered across three columns. Seven columns contain text (contract type, internet service, device brand) that sklearn's models can't read. And the numeric columns are on wildly different scales: age (18–74), monthly charges (20–120), tenure in months (1–71).
Before she can train a churn prediction model, all this must be cleaned. Without preprocessing, the model will crash or learn noise instead of signal. But with the right pipeline—fill missing, encode categories, scale numbers, split smartly—she's ready.
The preprocessing pipeline: 5 steps to model-ready data
All preprocessing follows the same checklist. We'll show every step with code and output.
2. Fill missing — median for numeric, mode for categorical
3. Encode categoricals — LabelEncoder for binary, get_dummies for multi-class
4. Scale numeric features — StandardScaler to mean=0, std=1
5. Train/test split — stratify to preserve class balance
Step 1: Spot and handle missing values
Missing data breaks sklearn models. First, we diagnose: which columns have gaps, and how many?
Good news: only 150 missing values out of 65,000 (0.23%). We'll fill them with smart strategies rather than drop rows.
Step 2: Fill missing values
For numeric columns like total_charges, we use the median (robust to outliers). For categorical columns like online_security, we use the mode (most common value).
Step 3: Encode categorical features
sklearn needs numbers, not text. We use two strategies: LabelEncoder for binary categories (e.g., Male/Female → 0/1), and one-hot encoding for multi-class (e.g., contract type → three new columns).
Notice: drop_first=True prevents multicollinearity. With 3 contract types, we keep 2 columns (the third is implied).
Step 4: Scale numeric features
Models like logistic regression, SVM, and neural networks expect scaled data. We use StandardScaler: transforms each feature to mean=0, std=1.
Step 5: Split into train and test with stratification
Always split before scaling (fit scaler on train only to avoid data leakage). Use stratify=y to preserve class balance.
Perfect! Both train and test have ~22.8% churn. Stratification worked—the split mirrors the original distribution.
The complete preprocessing pipeline
Here's the full script from raw CSV to train-ready arrays:
Also used in...
Domain Spotlight
How AI systems use this
Every machine learning model—from scikit-learn to TensorFlow to transformers—starts with clean, scaled data. GPT was trained on deduplicated, tokenized text (a form of preprocessing). Your churn model needs the same care. Garbage in, garbage out. Bad preprocessing creates bad predictions, no matter how fancy the algorithm.
Task: Load a CSV with insurance data. It has missing values in 'bmi'. Fill with the median. One-hot encode 'region'. Scale 'age' and 'bmi'. Split 80/20 with stratification on 'smoker'. Print final train/test shapes and smoker rates.
Hint 1: Use df['bmi'].fillna(df['bmi'].median(), inplace=True)
Hint 2: One-hot: pd.get_dummies(df['region'], prefix='region', drop_first=True)
Hint 3: Scale only numeric: scaler.fit_transform(X_train[['age', 'bmi']])
Hint 4: Stratify: train_test_split(..., stratify=y) where y='smoker'
Interview questions: test your understanding
Quiz: 5 questions to test your knowledge
Cheatsheet: preprocessing at a glance
| Step | Method | When to Use |
|---|---|---|
| Check missing | df.isnull().sum() |
Always first |
| Fill numeric | df['col'].fillna(df['col'].median()) |
Few missing, no pattern |
| Fill categorical | df['col'].fillna(df['col'].mode()[0]) |
Few missing, most-frequent strategy |
| Label encode binary | LabelEncoder().fit_transform(df['col']) |
Binary categories (Male/Female) |
| One-hot multi-class | pd.get_dummies(df['col'], drop_first=True) |
Multi-class categories (3+ groups) |
| Scale numeric | StandardScaler().fit_transform(X) |
Before KNN, SVM, neural nets |
| Train/test split | train_test_split(stratify=y) |
Always, before scaling |
Common mistakes: what breaks and how to fix it
drop_first=True.stratify=y to preserve original distribution.Key takeaways
Preprocessing is invisible but critical. A 1-hour preprocessing session saves a week of debugging bad models.
- Always check missing first: Use
df.isnull().sum(). Decide: drop or fill. - Fill numeric with median, categorical with mode. Both are robust to outliers and rare values.
- Encode binary with LabelEncoder (0/1), multi-class with one-hot (drop_first=True).
- Scale numeric features to mean=0, std=1 after split. Fit on train, transform test.
- Always stratify when splitting. Preserve class balance in train and test.
- Garbage in, garbage out. Clean data makes any model better.
Ready to build ML models?
Join the Best AI Program: The AI-Era Analyst — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Analyst →