Use this reference to compare ML algorithms, understand when to use each, and quickly look up hyperparameters. Bookmark it.

Supervised Learning: Regression

Use when your target variable is continuous (price, temperature, sales volume).

Algorithm Type Pros Cons When to Use Key Hyperparams
Linear Regression Parametric Fast, interpretable, baseline Assumes linear relationship, sensitive to outliers First regression attempt, baseline model fit_intercept, normalize
Polynomial Regression Parametric Captures non-linear patterns, still interpretable Overfitting risk with high degrees, slower Curved relationships visible in data degree (2-4 typical)
Ridge Regression Parametric Prevents overfitting via regularization, handles multicollinearity Adds bias, slower than OLS, lambda tuning needed Correlated features, moderate overfitting alpha (0.01โ€“10)
Lasso Regression Parametric Feature selection (zero coefficients), interpretable Unstable with correlated features, slower convergence Feature selection needed, sparse models alpha (0.001โ€“1)
ElasticNet Parametric Combines Ridge + Lasso, stable with correlations Two hyperparams to tune (alpha, l1_ratio) Highly correlated features + feature selection alpha, l1_ratio (0-1)

Supervised Learning: Classification

Use when your target variable is categorical (yes/no, cat/dog/bird, churn/retain).

Algorithm Type Pros Cons When to Use Key Hyperparams
Logistic Regression Parametric Fast, interpretable, probability outputs, handles binary & multi-class Assumes linear decision boundary, not for complex patterns Binary/multi-class baseline, interpretability needed C (1/lambda, 0.01โ€“1)
Decision Tree Non-parametric Highly interpretable, handles non-linear, no scaling needed Overfitting (deep trees), unstable with small data changes Interpretability critical, small datasets max_depth, min_samples_leaf, min_samples_split
Random Forest Ensemble High accuracy, handles non-linear, feature importance, robust Less interpretable, slower prediction, memory-heavy Accuracy > interpretability, medium-large datasets n_estimators, max_depth, min_samples_leaf
Gradient Boosting / XGBoost Ensemble Highest accuracy on structured data, feature interaction, fast Slow to train, complex hyperparameter tuning, overfitting risk Competition/production, need best accuracy n_estimators, learning_rate, max_depth, reg_lambda
Support Vector Machine (SVM) Parametric Powerful on high-dimensional data, flexible with kernels Slow on large datasets (>100k), needs scaling, hyperparameter tuning Small-medium datasets, high dimensions C, kernel, gamma (RBF)

Unsupervised Learning: Clustering & Dimensionality Reduction

Use when you have no target variable. Clustering finds groups; dimensionality reduction reduces features.

Algorithm Type Pros Cons When to Use Key Hyperparams
K-Means Centroid-based clustering Fast, simple, works on large datasets, scalable Must specify k, sensitive to outliers, spherical clusters only Customer segmentation, quick clustering, moderate k known n_clusters (k), init, n_init, random_state
Hierarchical Clustering (Agglomerative) Hierarchical clustering Dendrograms show structure, flexible linkage, no k needed upfront Slow on large datasets (O(nยฒ)), can't undo merges Dendrogram visualization, small-medium datasets, unknown k linkage (ward, complete, average), distance_threshold
DBSCAN Density-based clustering Finds arbitrary-shape clusters, auto-detects outliers, no k needed Sensitive to eps/min_samples, slow on high dimensions, varying densities Unknown cluster count, non-spherical clusters, outlier detection eps, min_samples
PCA (Principal Component Analysis) Dimensionality reduction Reduces features while preserving variance, visualization (2D/3D), speeds training Loses interpretability, assumes linear relationships Visualization, feature reduction for slow models, curse of dimensionality n_components

Decision Flowchart: Which Algorithm?

Start at the top and follow the questions down.

Q1: Do you have a target variable (labeled data)?

YES โ†’ Go to Q2. NO โ†’ Go to Unsupervised (Clustering/Reduction)

Q2: Is your target continuous or categorical?

CONTINUOUS (price, temp) โ†’ Regression. CATEGORICAL (yes/no, class) โ†’ Go to Q3

Q3 (Regression): Any linear relationship visible?

YES, simple โ†’ Linear Regression. MAYBE, curved โ†’ Polynomial. MANY CORRELATED FEATURES โ†’ Ridge/Lasso/ElasticNet. Need Feature Selection โ†’ Lasso/ElasticNet.

Q3 (Classification): Binary or multi-class?

BINARY โ†’ Go to Q4. MULTI-CLASS (3+ classes) โ†’ Random Forest or XGBoost

Q4: Interpretability or accuracy?

INTERPRETABILITY (explain decisions) โ†’ Logistic Regression or Decision Tree. ACCURACY (best score) โ†’ XGBoost or Random Forest.

Q5 (Decision Tree path): Small dataset or large?

SMALL (<5k) โ†’ Single Decision Tree. LARGE (5k+) โ†’ Random Forest. HUGE (100k+) โ†’ XGBoost.

Q5 (Unsupervised): Group or reduce?

GROUP (find clusters) โ†’ K-Means (if k known) or DBSCAN (if not). REDUCE (fewer features) โ†’ PCA. HIERARCHICAL (dendrogram) โ†’ Agglomerative Clustering.

RULES OF THUMB:

โ€ข Binary classification โ†’ Try Logistic Regression first, then XGBoost if accuracy needed.
โ€ข Multiclass โ†’ Random Forest or XGBoost.
โ€ข Need interpretability โ†’ Decision Tree or Logistic Regression.
โ€ข High accuracy required โ†’ XGBoost (structured data) or SVM (small-medium data).
โ€ข No target variable โ†’ K-Means, DBSCAN, or PCA.

Complexity vs Interpretability Matrix

Top-left = simple & interpretable. Bottom-right = complex & black-box.

โฌ…๏ธ SIMPLE | INTERPRETABLE โžก๏ธ
Logistic Regression
Binary/multi-class. Clear coefficients show feature impact.
โฌ‡๏ธ COMPLEX | BLACK-BOX โฌ†๏ธ
Linear Regression
Simple parametric. Interpretable coefficients.
โ†“ INCREASING COMPLEXITY โ†“
Decision Tree
Visual splits, interpretable paths. Overfits easily.
โ†“ INCREASING COMPLEXITY โ†“
K-Means, PCA
Moderate complexity. Simple unsupervised methods.
โ†“ INCREASING COMPLEXITY โ†“
Random Forest
Ensemble of trees. Less interpretable, high accuracy.
โ†“ INCREASING COMPLEXITY โ†“
SVM (non-linear)
Kernel magic, high accuracy. Hard to interpret.
โ†“ INCREASING COMPLEXITY โ†“
XGBoost / Gradient Boosting
State-of-art accuracy. Very hard to interpret.
HIERARCHICAL CLUSTERING, DBSCAN
DBSCAN
Density-based, finds arbitrary shapes. Moderate complexity.

Feature Scaling: Which Algorithms Need It?

Some algorithms are distance-based and need scaled features (mean=0, std=1). Others are tree-based and don't care.

MUST SCALE
KNN, SVM, Logistic Regression, Linear Regression, K-Means, PCA, Neural Networks
DON'T NEED SCALING
Decision Tree, Random Forest, XGBoost, Gradient Boosting, DBSCAN (with proper distance metric)
Why? Tree-based models split on feature values, not distances. Distance-based models treat all dimensions equally, so unscaled features with large ranges dominate. Always scale before KNN, SVM, neural nets, or dimensionality reduction.

Quick Reference: Strengths & Weaknesses

Algorithm Best For Avoid When Speed
Linear Regression Baseline, interpretability Non-linear patterns โšก Very fast
Logistic Regression Binary classification, prob outputs Non-linear boundaries โšก Very fast
Decision Tree Interpretability, small data Large data (overfits), deep trees โšก Fast
Random Forest High accuracy, balanced, robust Real-time prediction, interpretability needed ๐ŸŸก Moderate
XGBoost Best accuracy on structured data Real-time, no time for tuning ๐ŸŸก Moderate (train), โšก Fast (predict)
SVM Small-medium datasets, high dims 100k+ samples, real-time, non-structured ๐ŸŸก Moderate to slow
K-Means Quick segmentation, large data Unknown k, non-spherical clusters โšก Fast
DBSCAN Unknown cluster count, outliers Very large data, high dimensions ๐ŸŸก Moderate
PCA Visualization, feature reduction Non-linear relationships โšก Fast

Key Insights & Tips

1. Start Simple, Then Ensemble
Always start with a baseline: Logistic Regression for classification, Linear Regression for regression. If accuracy is poor, escalate to ensembles (Random Forest, XGBoost).
2. Interpretability vs Accuracy Trade-off
Need to explain decisions? Use Logistic Regression or Decision Trees. Need maximum accuracy? Use XGBoost or Random Forest.
3. Always Scale (or Tell Me Not To)
If using KNN, SVM, Logistic Regression, or PCA, ALWAYS scale. Tree-based models don't care. When in doubt, scale.
4. Hyperparameter Tuning is 80% of the Work
The algorithm matters; hyperparameters matter more. GridSearchCV or RandomizedSearchCV can find optimal parameters faster than guessing.
5. Ensemble Methods Usually Win
Random Forest and XGBoost combine multiple weak learners. On most datasets, they outperform single algorithms. But beware overfitting with deep trees.
6. For Competition/Production: XGBoost
Highest accuracy on structured data. Kaggle winners use it. But requires careful hyperparameter tuning and time.
7. For Unsupervised: Start with K-Means or PCA
K-Means if you know cluster count. PCA if you want visualization or feature reduction. DBSCAN if clusters have arbitrary shapes.
Africa & India's Best AI Academy
๐Ÿค– AI-first ๐ŸŽ“ Placement Assistance

Master ML algorithms hands-on

Join the Best AI Program: The AI-Era Analyst โ€” 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Analyst โ†’