Use this reference to compare ML algorithms, understand when to use each, and quickly look up hyperparameters. Bookmark it.
Supervised Learning: Regression
Use when your target variable is continuous (price, temperature, sales volume).
| Algorithm | Type | Pros | Cons | When to Use | Key Hyperparams |
|---|---|---|---|---|---|
| Linear Regression | Parametric | Fast, interpretable, baseline | Assumes linear relationship, sensitive to outliers | First regression attempt, baseline model | fit_intercept, normalize |
| Polynomial Regression | Parametric | Captures non-linear patterns, still interpretable | Overfitting risk with high degrees, slower | Curved relationships visible in data | degree (2-4 typical) |
| Ridge Regression | Parametric | Prevents overfitting via regularization, handles multicollinearity | Adds bias, slower than OLS, lambda tuning needed | Correlated features, moderate overfitting | alpha (0.01โ10) |
| Lasso Regression | Parametric | Feature selection (zero coefficients), interpretable | Unstable with correlated features, slower convergence | Feature selection needed, sparse models | alpha (0.001โ1) |
| ElasticNet | Parametric | Combines Ridge + Lasso, stable with correlations | Two hyperparams to tune (alpha, l1_ratio) | Highly correlated features + feature selection | alpha, l1_ratio (0-1) |
Supervised Learning: Classification
Use when your target variable is categorical (yes/no, cat/dog/bird, churn/retain).
| Algorithm | Type | Pros | Cons | When to Use | Key Hyperparams |
|---|---|---|---|---|---|
| Logistic Regression | Parametric | Fast, interpretable, probability outputs, handles binary & multi-class | Assumes linear decision boundary, not for complex patterns | Binary/multi-class baseline, interpretability needed | C (1/lambda, 0.01โ1) |
| Decision Tree | Non-parametric | Highly interpretable, handles non-linear, no scaling needed | Overfitting (deep trees), unstable with small data changes | Interpretability critical, small datasets | max_depth, min_samples_leaf, min_samples_split |
| Random Forest | Ensemble | High accuracy, handles non-linear, feature importance, robust | Less interpretable, slower prediction, memory-heavy | Accuracy > interpretability, medium-large datasets | n_estimators, max_depth, min_samples_leaf |
| Gradient Boosting / XGBoost | Ensemble | Highest accuracy on structured data, feature interaction, fast | Slow to train, complex hyperparameter tuning, overfitting risk | Competition/production, need best accuracy | n_estimators, learning_rate, max_depth, reg_lambda |
| Support Vector Machine (SVM) | Parametric | Powerful on high-dimensional data, flexible with kernels | Slow on large datasets (>100k), needs scaling, hyperparameter tuning | Small-medium datasets, high dimensions | C, kernel, gamma (RBF) |
Unsupervised Learning: Clustering & Dimensionality Reduction
Use when you have no target variable. Clustering finds groups; dimensionality reduction reduces features.
| Algorithm | Type | Pros | Cons | When to Use | Key Hyperparams |
|---|---|---|---|---|---|
| K-Means | Centroid-based clustering | Fast, simple, works on large datasets, scalable | Must specify k, sensitive to outliers, spherical clusters only | Customer segmentation, quick clustering, moderate k known | n_clusters (k), init, n_init, random_state |
| Hierarchical Clustering (Agglomerative) | Hierarchical clustering | Dendrograms show structure, flexible linkage, no k needed upfront | Slow on large datasets (O(nยฒ)), can't undo merges | Dendrogram visualization, small-medium datasets, unknown k | linkage (ward, complete, average), distance_threshold |
| DBSCAN | Density-based clustering | Finds arbitrary-shape clusters, auto-detects outliers, no k needed | Sensitive to eps/min_samples, slow on high dimensions, varying densities | Unknown cluster count, non-spherical clusters, outlier detection | eps, min_samples |
| PCA (Principal Component Analysis) | Dimensionality reduction | Reduces features while preserving variance, visualization (2D/3D), speeds training | Loses interpretability, assumes linear relationships | Visualization, feature reduction for slow models, curse of dimensionality | n_components |
Decision Flowchart: Which Algorithm?
Start at the top and follow the questions down.
YES โ Go to Q2. NO โ Go to Unsupervised (Clustering/Reduction)
CONTINUOUS (price, temp) โ Regression. CATEGORICAL (yes/no, class) โ Go to Q3
YES, simple โ Linear Regression. MAYBE, curved โ Polynomial. MANY CORRELATED FEATURES โ Ridge/Lasso/ElasticNet. Need Feature Selection โ Lasso/ElasticNet.
BINARY โ Go to Q4. MULTI-CLASS (3+ classes) โ Random Forest or XGBoost
INTERPRETABILITY (explain decisions) โ Logistic Regression or Decision Tree. ACCURACY (best score) โ XGBoost or Random Forest.
SMALL (<5k) โ Single Decision Tree. LARGE (5k+) โ Random Forest. HUGE (100k+) โ XGBoost.
GROUP (find clusters) โ K-Means (if k known) or DBSCAN (if not). REDUCE (fewer features) โ PCA. HIERARCHICAL (dendrogram) โ Agglomerative Clustering.
โข Binary classification โ Try Logistic Regression first, then XGBoost if accuracy needed.
โข Multiclass โ Random Forest or XGBoost.
โข Need interpretability โ Decision Tree or Logistic Regression.
โข High accuracy required โ XGBoost (structured data) or SVM (small-medium data).
โข No target variable โ K-Means, DBSCAN, or PCA.
Complexity vs Interpretability Matrix
Top-left = simple & interpretable. Bottom-right = complex & black-box.
Feature Scaling: Which Algorithms Need It?
Some algorithms are distance-based and need scaled features (mean=0, std=1). Others are tree-based and don't care.
Quick Reference: Strengths & Weaknesses
| Algorithm | Best For | Avoid When | Speed |
|---|---|---|---|
| Linear Regression | Baseline, interpretability | Non-linear patterns | โก Very fast |
| Logistic Regression | Binary classification, prob outputs | Non-linear boundaries | โก Very fast |
| Decision Tree | Interpretability, small data | Large data (overfits), deep trees | โก Fast |
| Random Forest | High accuracy, balanced, robust | Real-time prediction, interpretability needed | ๐ก Moderate |
| XGBoost | Best accuracy on structured data | Real-time, no time for tuning | ๐ก Moderate (train), โก Fast (predict) |
| SVM | Small-medium datasets, high dims | 100k+ samples, real-time, non-structured | ๐ก Moderate to slow |
| K-Means | Quick segmentation, large data | Unknown k, non-spherical clusters | โก Fast |
| DBSCAN | Unknown cluster count, outliers | Very large data, high dimensions | ๐ก Moderate |
| PCA | Visualization, feature reduction | Non-linear relationships | โก Fast |
Key Insights & Tips
Always start with a baseline: Logistic Regression for classification, Linear Regression for regression. If accuracy is poor, escalate to ensembles (Random Forest, XGBoost).
Need to explain decisions? Use Logistic Regression or Decision Trees. Need maximum accuracy? Use XGBoost or Random Forest.
If using KNN, SVM, Logistic Regression, or PCA, ALWAYS scale. Tree-based models don't care. When in doubt, scale.
The algorithm matters; hyperparameters matter more. GridSearchCV or RandomizedSearchCV can find optimal parameters faster than guessing.
Random Forest and XGBoost combine multiple weak learners. On most datasets, they outperform single algorithms. But beware overfitting with deep trees.
Highest accuracy on structured data. Kaggle winners use it. But requires careful hyperparameter tuning and time.
K-Means if you know cluster count. PCA if you want visualization or feature reduction. DBSCAN if clusters have arbitrary shapes.
Master ML algorithms hands-on
Join the Best AI Program: The AI-Era Analyst โ 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Analyst โ