📱
Real-world scenario
Amara (TelcoMax Nairobi): 5,000 customers, 19 features, can't visualize or model efficiently

Amara is a data scientist at TelcoMax, a telecom in Nairobi. She has a preprocessed dataset of 5,000 customers with 19 features (after one-hot encoding: service type, contract length, internet options, device categories). She wants to:

  • Visualize customer groups in 2D to understand churn patterns. But 19D plots are impossible.
  • Speed up model training. Fewer features = faster fitting and less overfitting risk.
  • Reduce noise. Some features are correlated; PCA isolates signal.

PCA solves this: compress 19 features into uncorrelated principal components. Keep the top 2 for plotting (captures 18.25% of variance), or the top 15 for modeling (captures 90%).

5,000
Rows · Customers
19
Encoded features
2 / 15
PCA components

What is PCA? Finding directions of maximum variance

Principal Component Analysis (PCA) answers: "In which directions does my data vary the most?" Instead of keeping all original features, PCA creates new features (principal components) that are uncorrelated and ordered by how much variance they explain.

Core idea: Rotate your feature space to align with the directions where data spreads out most. The first principal component (PC1) points in the direction of maximum variance. PC2 points perpendicular, capturing the second-most variance. And so on.

Result: You can keep the first k components (e.g., 2 or 15) and drop the rest without losing much information.

Why is this useful? Fewer features mean faster training, less overfitting, easier visualization, and cleaner signal (by discarding noise-heavy components).

Eigenvalues and eigenvectors: the math intuition

PCA relies on two concepts from linear algebra:

  • Eigenvector: A direction (vector) in your feature space. For PCA, each eigenvector of the covariance matrix is a principal component direction.
  • Eigenvalue: A number telling you how much variance is explained along that eigenvector direction. Larger eigenvalue = more variance = more important principal component.

When you compute PCA on your data, scikit-learn's PCA().fit(X) finds the covariance matrix, computes its eigenvalues (and eigenvectors), and sorts them by magnitude (largest first). The eigenvalues directly tell you the explained variance per component.

Example: If your first 5 eigenvalues are [102, 80, 79, 77, 76] (in arbitrary units), then PC1 explains 102/(102+80+79+77+76) ≈ 10.23% of total variance, PC2 explains 8.02%, and so on. This is the "explained variance ratio."

Explained variance ratio: which components to keep?

Each component explains a percentage of total variance. The "explained variance ratio" is individual component variance divided by total. The "cumulative explained variance" sums these up.

explained_variance.py Calculate explained variance per component
Output
Shape: (5000, 19) Explained variance ratio (first 5): PC1: 0.1023 PC2: 0.0802 PC3: 0.0792 PC4: 0.0772 PC5: 0.0759 Cumulative explained variance (first 5): PC1-PC1: 0.1023 PC1-PC2: 0.1825 PC1-PC3: 0.2617 PC1-PC4: 0.3389 PC1-PC5: 0.4149 Components for 90% variance: 15 Actual cumulative at PC15: 0.9001

Key insights: PC1 alone captures 10.23%. The first 2 components together capture 18.25%—enough to visualize 5,000 customers in 2D. For modeling, we need 15 components to retain 90% of variance.

Scree plot: visualizing variance drop-off

A scree plot shows the explained variance ratio for each component. You're looking for the "elbow"—where variance drops sharply. Keep components before the elbow.

Scree Plot: Explained Variance per Component (first 10)
Scree plot showing explained variance ratio for first 10 components
Notice the steep drop after PC1, then a gentler decline. For visualization, 2 components are often enough (the "knee"). For modeling, we look further until cumulative variance reaches a threshold (90%, 95%, etc.).

Cumulative variance: deciding how many components

The cumulative variance plot shows total variance retained by the first k components. Use this to decide: Do I want 90%? 95%? The steeper the initial climb, the better—fewer components needed.

Cumulative Explained Variance: First 15 Components (reach 90% at PC15)
Cumulative explained variance reaching 90% at PC15
The curve flattens as you add more components—diminishing returns. At PC15, you've captured ~90% of variance. Beyond that, each extra component adds little value.

PCA for visualization: compress to 2D

Humans can't see in 19 dimensions. PCA(n_components=2) gives you a 2D view of your data.

pca_2d.py Compress to 2D for plotting
Output
Original shape: (5000, 19) After 2D PCA: (5000, 2) Explained variance (2 components): 0.1825

You've compressed 19 features to 2, retaining 18.25% of variance. The plot lets you spot clusters, outliers, or class separation. For churn prediction, you might see churned customers clustering away from retained ones.

2D PCA projection of 5,000 telecom customers (colored by churn status)
2D scatter plot of PCA-compressed customer data

PCA for modeling: reduce features, keep 90%

For training a classifier, you typically want more variance retained (90%+) than for visualization.

pca_model.py Reduce to 15 components for 90% variance
Output
Original: (5000, 19) After PCA: (5000, 15) Cumulative variance: 0.9001 Logistic Regression (15 PCA components) Train accuracy: 0.8124 Test accuracy: 0.8057 Reduction: 19 → 15 features (21% fewer)

The model achieves 80.57% accuracy on test data with fewer features. PCA speeds up training and often improves generalization by removing noise-heavy low-variance components.

PCA limitations: it's linear, interpretability is hard

PCA is powerful but has important limits:

  • Linear only: PCA finds linear combinations of original features. If your data has nonlinear structure (e.g., a spiral, nested clusters), PCA won't capture it well. Use t-SNE, UMAP, or autoencoders for nonlinear data.
  • Interpretation: Each PC is a mix of all original features. You can't easily say "PC1 = customer age + tenure." You need to check the component loadings (eigenvector values).
  • Assumes normalization: Outliers or large-scale variables dominate PCA. Always standardize your data before PCA.
  • Variance ≠ Usefulness: High-variance features might not be predictive for your target. PCA ignores the target, so an uninformative but high-variance feature gets a high PC ranking.

For supervised learning (classification, regression), consider supervised alternatives like Linear Discriminant Analysis (LDA) or PLS regression.

Understanding component loadings: which features matter?

To interpret what each PC represents, examine the loadings (coefficients). For example, pca.components_[0] shows how much each original feature contributes to PC1.

loadings.py Inspect component loadings
Output
PC1 (top 5 positive contributors): contract_one_year 0.0342 internet_dsl 0.0310 device_x700 0.0298 online_security_yes 0.0287 phone_service_yes 0.0271 PC1 (top 5 negative contributors): contract_month_month -0.0265 internet_fiber -0.0251 device_laptop -0.0248 tech_support_yes -0.0238 online_backup_no -0.0225 PC1 explains 0.1023 (10.23%) of variance Interpretation: PC1 is a weighted mix of all features, weighted by these loadings.

Notice: Customers with one-year contracts, DSL internet, and security add-ons push PC1 high. Those with month-to-month, fiber, and fewer add-ons push it low. PC1 roughly separates long-term, committed customers from short-term, minimal-service ones.

Also used in...

Domain Spotlight

🎬
NLP & Text: Document visualization
TF-IDF word frequencies (1000+ features) → PCA(2) for 2D scatter of documents. Visualize topic clusters or class separation.
🖼️
Computer Vision: Image compression
Image pixel data (thousands of dimensions) → PCA captures "eigenfaces" or "eigenvectors." Fewer components = smaller file size.
🧬
Genomics: Gene expression analysis
Gene expression across 20,000 genes → PCA(2-3) to visualize cell types or disease groups. Identify key genes driving variance.

How AI systems use this

PCA is foundational to machine learning pipelines. Before training deep neural networks, researchers often apply PCA to compress input features, speeding up backpropagation. Autoencoders are a nonlinear extension of PCA, learning hierarchical feature compression. Even large language models use dimensionality reduction internally—transformers project embeddings to lower dimensions for efficiency and to reduce overfitting on noisy words.

Feature Compression Noise Reduction Model Acceleration Autoencoder Inspiration
🎯 Mini Challenge: PCA on wine dataset

Task: Load the sklearn wine dataset (178 samples, 13 features). Fit PCA. Print the explained variance ratio for the first 3 components. Then fit PCA(n_components=2), transform the data, and plot the 2D scatter (color by wine class). How much variance do the first 2 PCs capture?

Hint 1: from sklearn.datasets import load_wine; wine = load_wine(); X = wine.data

Hint 2: pca = PCA(); pca.fit(X); print(pca.explained_variance_ratio_[:3])

Hint 3: pca2 = PCA(n_components=2); X_pca = pca2.fit_transform(X)

Hint 4: Plot: plt.scatter(X_pca[:, 0], X_pca[:, 1], c=wine.target)

Interview questions: test your understanding

PCA finds uncorrelated directions (principal components) along which your data varies the most, allowing you to compress features with minimal information loss.
The percentage of total data variance captured by a single principal component. If PC1 has an explained variance ratio of 0.1023, it accounts for 10.23% of all variation in your data.
When your data has strong nonlinear structure (PCA is linear), when you need to interpret features (PCA components are hard to interpret), or when high variance is not correlated with your target label. Use supervised methods (LDA, PLS) or nonlinear methods (t-SNE, UMAP) in those cases.
Yes! For visualization, 18.25% is often enough to see clusters or outliers. For modeling, you'd want more (e.g., 15 components for 90%). The use case determines the threshold. Visualization ≠ modeling.
Much more! PCA shines when features are correlated because it can compress them into fewer uncorrelated PCs. Uncorrelated features offer little compression benefit—each already captures unique information. Correlation = PCA opportunity.

Quiz: 5 questions to test your knowledge

Question 1 of 5
What is the primary goal of PCA?
Question 2 of 5
If your first 2 PCs capture 18.25% of variance, and you're doing visualization, is this acceptable?
Question 3 of 5
In PCA, what does an eigenvalue represent?
Question 4 of 5
Your data has 19 correlated features. To retain 90% variance, you need 15 PCs. What does this tell you?
Question 5 of 5
What is a key limitation of PCA?
0
/5
Excellent! You understand PCA and dimensionality reduction.

Cheatsheet: PCA at a glance

Task Code Output/Use
Fit PCA pca = PCA(); pca.fit(X) Learns covariance structure
Transform data X_pca = pca.transform(X) Projects X to PC space
Fit & transform X_pca = pca.fit_transform(X) Combined in one step
Get variance (ratio) pca.explained_variance_ratio_ [0.1023, 0.0802, ...] per component
Cumulative variance np.cumsum(pca.explained_variance_ratio_) [0.1023, 0.1825, ...] running total
2D visualization PCA(n_components=2).fit_transform(X) (N, 2) array for scatter plot
Specify variance % PCA(n_components=0.90) Auto-selects k to reach 90%
Get loadings pca.components_[0] Feature contributions to PC1

Common mistakes: what breaks and how to fix it

Mistake 1: Skipping standardization before PCA
PCA is sensitive to scale. A feature with range 0–1000 dominates one with range 0–1. Fix: use StandardScaler to mean=0, std=1 before PCA.
Mistake 2: Using 90% variance threshold for visualization
Overkill. 2D visualization can be informative with 15–30% variance. Fix: for visualization, optimize for human interpretability, not variance %. For modeling, aim for 85–95%.
Mistake 3: Fitting PCA on entire dataset, then train/test split
Data leakage! PCA learns from test data. Fix: split first, fit PCA on train only, transform test with the same PCA.
Mistake 4: Assuming high variance = predictive
A high-variance component might be noise. PCA ignores your target. Fix: for supervised learning, use LDA or check correlation between PCs and target.

Key takeaways

PCA is a foundational technique for compressing data, visualizing high-dimensional patterns, and speeding up models.

  • PCA finds uncorrelated directions (PCs) ranked by variance. The first PC captures the most variance, the second captures the most residual, and so on.
  • Explained variance ratio tells you what % each PC explains. Use cumulative variance to decide: 2 components for visualization (18%), 15 for modeling (90%).
  • Scree plots and cumulative variance plots help you choose k. Look for the elbow or a threshold (90%, 95%).
  • Always standardize before PCA. Scale-dependent variables skew the analysis.
  • PCA is linear—good for correlated features, not nonlinear patterns. For nonlinear data, try t-SNE, UMAP, or autoencoders.
  • Split data before fitting PCA to avoid leakage. Fit on train, transform test with the same PCA.
  • High variance ≠ predictive. For supervised tasks, use LDA or inspect component–target correlation.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Ready to reduce dimensions and accelerate ML?

Join the Best AI Program: The AI-Era Analyst — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Analyst →