📞
Real-world scenario
Wanjiku (TelcoMax Nairobi): 5,000 customers, messy data, no model yet

Wanjiku works at TelcoMax, a telecom company in Nairobi. She has a dataset of 5,000 customers with 150 missing values scattered across three columns. Seven columns contain text (contract type, internet service, device brand) that sklearn's models can't read. And the numeric columns are on wildly different scales: age (18–74), monthly charges (20–120), tenure in months (1–71).

Before she can train a churn prediction model, all this must be cleaned. Without preprocessing, the model will crash or learn noise instead of signal. But with the right pipeline—fill missing, encode categories, scale numbers, split smartly—she's ready.

150
Missing values
7
Categorical columns
4
Numerics to scale

The preprocessing pipeline: 5 steps to model-ready data

All preprocessing follows the same checklist. We'll show every step with code and output.

1. Check missing values — locate gaps, decide: drop or fill
2. Fill missing — median for numeric, mode for categorical
3. Encode categoricals — LabelEncoder for binary, get_dummies for multi-class
4. Scale numeric features — StandardScaler to mean=0, std=1
5. Train/test split — stratify to preserve class balance

Step 1: Spot and handle missing values

Missing data breaks sklearn models. First, we diagnose: which columns have gaps, and how many?

check_missing.py Inventory missing data
Output
Shape: (5000, 13) Missing values: total_charges 50 online_security 50 tech_support 50 dtype: int64

Good news: only 150 missing values out of 65,000 (0.23%). We'll fill them with smart strategies rather than drop rows.

Step 2: Fill missing values

For numeric columns like total_charges, we use the median (robust to outliers). For categorical columns like online_security, we use the mode (most common value).

fill_missing.py Fill with median and mode
Output
Missing after fill: 0 Updated shape: (5000, 13) Dtypes: object 7 int64 4 float64 2 dtype: int64

Step 3: Encode categorical features

sklearn needs numbers, not text. We use two strategies: LabelEncoder for binary categories (e.g., Male/Female → 0/1), and one-hot encoding for multi-class (e.g., contract type → three new columns).

encode.py Label encode binary, one-hot multi-class
Output
Gender encoding: {'Female': 0, 'Male': 1} One-hot contract shape: (5000, 2) Columns: ['contract_One year', 'contract_Two year']

Notice: drop_first=True prevents multicollinearity. With 3 contract types, we keep 2 columns (the third is implied).

Step 4: Scale numeric features

Models like logistic regression, SVM, and neural networks expect scaled data. We use StandardScaler: transforms each feature to mean=0, std=1.

scale.py StandardScaler: mean 0, std 1
Output
Before scaling: Age: mean=46.28, std=16.47 Charges: mean=70.22, std=28.75 After scaling: Age: mean=0.000000, std=1.000000 Charges: mean=-0.000001, std=1.000001

Step 5: Split into train and test with stratification

Always split before scaling (fit scaler on train only to avoid data leakage). Use stratify=y to preserve class balance.

split.py 80/20 split with stratification
Output
Train shape: (4000, 4) Test shape: (1000, 4) Train churn rate: 0.229 Test churn rate: 0.228 Train: {0: 3086, 1: 914} Test: {0: 772, 1: 228}

Perfect! Both train and test have ~22.8% churn. Stratification worked—the split mirrors the original distribution.

The complete preprocessing pipeline

Here's the full script from raw CSV to train-ready arrays:

pipeline.py Full preprocessing end-to-end
Output
Ready for modeling: train (4000, 4), test (1000, 4)

Also used in...

Domain Spotlight

💳
Banking: Fraud detection models
Fill missing transaction IDs, encode transaction types, scale transaction amounts. Stratify by fraud/non-fraud for balanced splits.
🏥
Healthcare: Patient risk scoring
Fill missing blood pressure readings with median, encode diagnosis codes, scale vitals. Stratify by disease severity.
🛒
E-commerce: Purchase prediction
Fill missing product ratings, one-hot encode category, scale price and clicks. Stratify by purchase/no-purchase.

How AI systems use this

Every machine learning model—from scikit-learn to TensorFlow to transformers—starts with clean, scaled data. GPT was trained on deduplicated, tokenized text (a form of preprocessing). Your churn model needs the same care. Garbage in, garbage out. Bad preprocessing creates bad predictions, no matter how fancy the algorithm.

Data Pipelines Feature Engineering Model Training Data Leakage Prevention
🎯 Mini Challenge: Preprocess insurance data

Task: Load a CSV with insurance data. It has missing values in 'bmi'. Fill with the median. One-hot encode 'region'. Scale 'age' and 'bmi'. Split 80/20 with stratification on 'smoker'. Print final train/test shapes and smoker rates.

Hint 1: Use df['bmi'].fillna(df['bmi'].median(), inplace=True)

Hint 2: One-hot: pd.get_dummies(df['region'], prefix='region', drop_first=True)

Hint 3: Scale only numeric: scaler.fit_transform(X_train[['age', 'bmi']])

Hint 4: Stratify: train_test_split(..., stratify=y) where y='smoker'

Interview questions: test your understanding

sklearn models expect numeric arrays (floats, ints), not text. They can't learn from strings like "Male" or "Fiber optic". You must encode them to numbers first (0/1 or 0/1/2).
Transforms each feature to mean=0 and standard deviation=1. This helps distance-based models (KNN, SVM, neural nets) learn equally from all features, regardless of their original scale.
Without stratification, a random split might leave all rare classes in test (or train). Stratify preserves the original class distribution, giving honest estimates of model performance on both train and test.
Always split first, then scale. Fit the scaler on train data only. Transform test using the same scaler. Scaling before split leaks information from test into the model, inflating performance.
Drop the column if it's low-importance or redundant. Drop rows only if < 5% missing and missing is random. With 50%, you lose half your data either way. Better: impute using other features (KNN imputation) if missingness is not random, or drop if the column is weak anyway.

Quiz: 5 questions to test your knowledge

Question 1 of 5
You find 50 missing values in the 'total_charges' column. How do you fill them?
Question 2 of 5
You have a categorical column 'internet_service' with 3 categories. When you use one-hot encoding with drop_first=True, how many new columns are created?
Question 3 of 5
After StandardScaler, what are the mean and standard deviation of each feature?
Question 4 of 5
You scale your data, then split into train/test. Your friend scales after splitting. Who did it correctly?
Question 5 of 5
Your dataset has 22.8% churn. After 80/20 split with stratify=y, what churn rate do you expect in test?
0
/5
Excellent! You understand data preprocessing.

Cheatsheet: preprocessing at a glance

Step Method When to Use
Check missing df.isnull().sum() Always first
Fill numeric df['col'].fillna(df['col'].median()) Few missing, no pattern
Fill categorical df['col'].fillna(df['col'].mode()[0]) Few missing, most-frequent strategy
Label encode binary LabelEncoder().fit_transform(df['col']) Binary categories (Male/Female)
One-hot multi-class pd.get_dummies(df['col'], drop_first=True) Multi-class categories (3+ groups)
Scale numeric StandardScaler().fit_transform(X) Before KNN, SVM, neural nets
Train/test split train_test_split(stratify=y) Always, before scaling

Common mistakes: what breaks and how to fix it

Mistake 1: Scaling before train/test split
Data leakage! The scaler learned statistics from test data, making the model look better than it is. Fix: split first, fit scaler on train only, transform test with the same scaler.
Mistake 2: One-hot encoding high-cardinality columns
A column with 1000 unique values becomes 1000 new columns, causing memory bloat. Fix: group rare values into "Other" or use target encoding.
Mistake 3: Forgetting drop_first in get_dummies
Without it, n categories create n columns, causing multicollinearity. Fix: always use drop_first=True.
Mistake 4: Not checking class balance after split
Random split might create imbalanced test set. Fix: use stratify=y to preserve original distribution.

Key takeaways

Preprocessing is invisible but critical. A 1-hour preprocessing session saves a week of debugging bad models.

  • Always check missing first: Use df.isnull().sum(). Decide: drop or fill.
  • Fill numeric with median, categorical with mode. Both are robust to outliers and rare values.
  • Encode binary with LabelEncoder (0/1), multi-class with one-hot (drop_first=True).
  • Scale numeric features to mean=0, std=1 after split. Fit on train, transform test.
  • Always stratify when splitting. Preserve class balance in train and test.
  • Garbage in, garbage out. Clean data makes any model better.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Ready to build ML models?

Join the Best AI Program: The AI-Era Analyst — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Analyst →