💳
Real-world scenario
NeoBank Mumbai: find the widest margin between defaulters and safe loans

NeoBank's risk team trained logistic regression and random forests. Both scored AUC=0.80. But they wonder: can we find a boundary that maximizes the gap between the two classes? A boundary with a wide margin tends to generalize better.

That's where Support Vector Machines come in. SVM finds the hyperplane (decision boundary) that leaves the maximum margin—the widest empty space—between the closest defaulters and non-defaulters. These closest points are called support vectors.

10,000
Applicants
0.7360
RBF Accuracy
0.7841
RBF AUC

What is SVM? Finding the widest margin

In binary classification, you need a line (or hyperplane in high dimensions) that separates two classes. Infinite lines can do this. But which one is best?

Support Vector Machines pick the line with the widest gap—the largest margin—between the two classes. The intuition: a boundary with room to spare will handle new, slightly different data better than a tight boundary that just barely separates.

Margin: The perpendicular distance from the decision boundary to the nearest data point.
Support vectors: The data points that define the margin (the closest ones to the boundary).
Goal: Maximize the margin while keeping misclassifications low.

Support vectors: the decision makers

Most data points don't matter. Only the points closest to the boundary—the support vectors—influence the decision boundary. If you move a point far from the boundary, it won't change SVM's decision. But move a support vector, and the entire boundary shifts.

This is why SVM is efficient: once trained, you only need to store and evaluate the support vectors, not the entire dataset.

identify_support_vectors.py Count and visualize support vectors
Output
Total training samples: 4000 Support vectors: 1847 Percentage: 46.2% First 10 support vector indices: [ 0 2 5 7 10 12 14 17 19 21]
SVM decision boundary with RBF kernel on credit_score vs income. Orange circles mark support vectors.
2D scatter plot showing decision boundary with support vectors highlighted as orange circles
Support vectors are the data points closest to the boundary — they're the only ones that matter for the decision.

The kernel trick: non-linear boundaries

What if the data isn't linearly separable? In 2D, a single line can't separate the classes. But if we could add a third dimension, the points might separate with a plane.

The kernel trick does this implicitly. It transforms data to a higher-dimensional space without actually computing the transformation. Three kernels are common:

Kernel Boundary Type When to Use Code
Linear Straight line / flat hyperplane Data is (mostly) linearly separable kernel='linear'
RBF (Radial Basis Function) Curved, non-linear boundary Non-linear relationships; default choice kernel='rbf'
Polynomial Polynomial curves (degree 2, 3, ...) Non-linear but want interpretability kernel='poly', degree=3

The kernel trick computes similarity between points instead of coordinates. For RBF: K(x, y) = exp(-gamma * ||x - y||²). Two points far apart have low similarity; nearby points have high similarity.

Linear vs Polynomial vs RBF kernel boundaries on the same 2D credit data.
Three subplots showing different decision boundaries: linear (straight line), polynomial (curved), and RBF (complex curved)

Training SVM with scikit-learn

Always scale features first. SVM computes distances; unscaled features distort the boundary. Use StandardScaler.

train_svm_basic.py RBF kernel SVM in 5 lines
Output
Accuracy: 0.7360 F1 score: 0.6567 AUC: 0.7841

The C parameter: margin vs. misclassification trade-off

C controls the penalty for misclassifying a point. Low C allows more misclassifications for a wider margin (underfitting). High C penalizes misclassifications heavily, creating a narrow, complex boundary (overfitting).

tune_c_parameter.py Experiment with C values
Output
C= 0.1 | Train: 0.7243 | Test: 0.7255 C= 1.0 | Train: 0.7355 | Test: 0.7360 C= 10.0 | Train: 0.7408 | Test: 0.7355 C= 100.0 | Train: 0.7502 | Test: 0.7295

Sweet spot: C=1.0 or C=10.0. Beyond C=100, test accuracy drops—overfitting. With C=0.1, the model underfits slightly.

Comparing kernels: linear vs. RBF vs. polynomial

Different kernels create different decision boundaries. Let's train all three on the same data.

compare_kernels.py Test linear, RBF, and polynomial kernels
Output
Kernel Comparison: ------------------------------------------------------------ Kernel Accuracy F1 AUC ------------------------------------------------------------ linear 0.7345 0.6720 0.7821 rbf 0.7360 0.6567 0.7841 poly 0.7170 0.6462 0.7655 ------------------------------------------------------------

RBF wins on AUC. Linear is close and faster. Polynomial underperforms. For this dataset, the relationship is nearly linear, so RBF's complexity doesn't add much.

Why SVM requires scaling

SVM computes distances between points. Without scaling, features with large ranges dominate. Example: if credit_score ranges 300-850 and income_ratio ranges 0-1, the distance is almost entirely determined by credit_score.

scaling_impact.py SVM accuracy with and without scaling
Output
Accuracy without scaling: 0.5847 Accuracy with scaling: 0.7360 Difference: 0.1513

Huge difference! Scaling boosts accuracy by 15 points. Always scale before SVM.

How SVM fits into modern AI

SVMs dominated machine learning before deep learning. They won competitions, powered spam filters, and detected faces in images. Today, deep neural networks outperform SVMs on images and text. But on tabular data (credit scoring, insurance, biology), SVMs remain competitive and interpretable. The kernel trick—implicitly mapping to higher dimensions—influenced how neural networks handle non-linearity.

Kernel Methods Margin Maximization Support Vectors Historical Deep Learning Context

Also used in...

Domain Spotlight

💳
Fraud detection: Non-linear fraud patterns
RBF kernel captures complex fraud boundaries. Support vectors are the suspicious transactions that matter most.
🧬
Bioinformatics: Gene expression classification
High-dimensional gene data (10,000+ features). SVM scales well. Kernel trick handles non-linear gene interactions.
📧
Text classification: Spam filters, sentiment
SVM with linear kernel fast on sparse text vectors. RBF kernel captures semantic nuances.
Challenge: Find the optimal C value

Task: Train SVM with RBF kernel using C values [0.01, 0.1, 1, 10, 100]. For each, print train accuracy, test accuracy, and number of support vectors. Which C minimizes test accuracy loss while keeping support vectors low?

Hint 1: Loop over c_values and train inside: for c in [0.01, 0.1, 1, 10, 100]:

Hint 2: Number of support vectors: len(svm.support_)

Hint 3: Test accuracy: svm.score(X_test_scaled, y_test)

Hint 4: Look for the "elbow point" where test accuracy is good but support vectors plateau.

Interview questions: test your understanding

The perpendicular distance from the decision boundary to the nearest data points (support vectors). SVM maximizes this margin because a boundary with more room to spare generalizes better to new data.
SVM computes distances between points. Unscaled features with large ranges dominate the distance calculation, biasing the model. Scaling ensures all features contribute equally.
It implicitly maps data to a higher-dimensional space where non-linearly separable data becomes linearly separable. The mapping is done via a kernel function (like RBF) without actually computing the transformation, saving memory and time.
C=0.1 allows more misclassifications for a wider margin (underfitting risk, simpler boundary). C=10 penalizes misclassifications heavily, creating a narrow, complex boundary (overfitting risk). C=1.0 is a balanced default.
Logistic regression is simpler, faster, and interpretable. Use it when performance is similar. Use SVM if the data has non-linear patterns that LR misses, or if you're in high-dimensional space. Benchmarking both is good practice.

Quiz: 5 questions to test your knowledge

Question 1 of 5
What does SVM maximize?
Question 2 of 5
What are support vectors?
Question 3 of 5
The RBF kernel creates which type of boundary?
Question 4 of 5
SVM accuracy dropped from 0.74 to 0.58 after removing StandardScaler. Why?
Question 5 of 5
With C=100, test accuracy is 0.7295 vs C=1.0 giving 0.7360. Why?
0
/5
Excellent! You understand Support Vector Machines.

Cheatsheet: SVM at a glance

Concept Code Notes
Train SVM SVC(kernel='rbf') Always scale first with StandardScaler
Linear kernel kernel='linear' Fast, interpretable, for linear data
RBF kernel kernel='rbf' Default, non-linear boundaries, slower
Polynomial kernel kernel='poly', degree=3 Polynomial boundaries, tune degree
C parameter C=1.0 Low C = wide margin; high C = narrow margin
Support vectors svm.support_ Indices of training points defining boundary
Probabilities probability=True, then predict_proba() SVM uses Platt scaling to calibrate probs

Common mistakes: what breaks and how to fix it

Mistake 1: Not scaling features
SVM is distance-based. Unscaled features distort the boundary. Always use StandardScaler().fit_transform(X_train) before fitting.
Mistake 2: Using SVM on very large datasets
SVM scales O(n²) to O(n³) in memory. For >100K rows, use linear SVM or switch to logistic regression or neural networks. XGBoost is faster on large tabular data.
Mistake 3: Never tuning the C parameter
Default C=1.0 isn't always optimal. Grid search over C=[0.01, 0.1, 1, 10, 100] with cross-validation to find the sweet spot.
Mistake 4: Expecting feature importance from SVM
Non-linear SVMs don't provide feature importance directly (unlike decision trees). Use model-agnostic methods like SHAP if you need explanations.

Key takeaways

  • SVM finds the widest margin between classes. Only support vectors (closest points) define the boundary.
  • Always scale features first. SVM computes distances; unscaled features dominate.
  • Kernel trick enables non-linear boundaries: RBF for non-linear data, linear for fast inference on linearly separable data.
  • C parameter controls the margin-misclassification trade-off. Low C = wider margin, high C = narrow, complex boundary.
  • SVM on tabular data is competitive, but slower on large datasets than logistic regression or boosting.
  • When AUC is tied with simpler models, choose simplicity: logistic regression is faster and more interpretable.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Ready to deploy classification models?

Join the Best AI Program: The AI-Era Analyst — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Analyst →