The widest road between two classes
Two sensors on a bottling line, a scatter of crates, one question: where do you draw the line? A support vector machine answers that differently from every other classifier in this section. It does not fit a probability curve or split one feature at a time. It looks for the boundary that sits as far as possible from the closest point of either class, and it decides where that boundary goes using only those closest points.
Everything else in your training set is ignored. Shift a comfortably classified crate by half a bar of pressure and the boundary does not move at all. The points sitting on the edge of the gap are the support vectors, and the width of the gap is the margin. That is the whole idea: maximise the margin, and let a handful of borderline cases define the model.
This gives the method an unusual property. A model trained on 630 crates might be defined by 118 of them. The rest contribute nothing to the fitted boundary, which is why an SVM often generalises well on modest datasets and why its memory footprint depends on the number of support vectors rather than the number of rows.
Start with data. A bottling plant audits crates of 24 and fails them when the seal pressure drifts in either direction while cap torque is low, so the acceptable region is a band rather than a half-plane.
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(65)
n = 900
seal_pressure = rng.normal(4.20, 0.45, n).round(3) # bar
cap_torque = rng.normal(1.55, 0.30, n).round(3) # newton metres
risk = 2.9 * (seal_pressure - 4.20) ** 2 - 1.8 * (cap_torque - 1.55) - 0.33
fails = (risk + rng.normal(0, 0.18, n) > 0).astype(int)
crates = pd.DataFrame({"seal_pressure": seal_pressure,
"cap_torque": cap_torque,
"failed_audit": fails})
print(crates.shape)
print(crates["failed_audit"].value_counts().to_dict())
# (900, 3)
# {1: 537, 0: 363}
X = crates[["seal_pressure", "cap_torque"]].to_numpy()
y = crates["failed_audit"].to_numpy()
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3,
random_state=0, stratify=y)
What a support vector machine actually optimises
The hard version of the problem assumes the classes can be separated perfectly, which real data never allows. The version you actually fit is the soft margin: it permits points to sit inside the margin, or on the wrong side of the boundary, and charges a penalty for each violation. The objective balances two things that pull against each other. Make the margin wide, and keep the total penalty small.
C sets the exchange rate. A small C means violations are cheap, so the model buys a wide margin and tolerates mistakes. A large C makes violations expensive, so the boundary contorts to get training points right. That is the same tension as regularisation strength in how logistic regression fits a decision boundary, except that the penalty here counts margin violations rather than the size of the coefficients.
One consequence catches people out. The margin is a distance, measured in whatever units your features happen to use. A feature measured in grams contributes a thousand times more to that distance than the same quantity in kilograms, so an unscaled SVM silently ranks your features by unit of measurement. Standardising first is not optional, which is why feature scaling and normalisation matters more for this algorithm than for most.
from sklearn.svm import SVC
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
lin = make_pipeline(StandardScaler(), SVC(kernel="linear", C=1.0, random_state=0))
lin.fit(X_tr, y_tr)
print("linear kernel test accuracy:", round(lin.score(X_te, y_te), 3))
print("support vectors:", lin[-1].n_support_.tolist())
# linear kernel test accuracy: 0.752
# support vectors: [169, 169]
raw = SVC(kernel="rbf", C=1.0, random_state=0).fit(X_tr, y_tr)
scaled = make_pipeline(StandardScaler(),
SVC(kernel="rbf", C=1.0, random_state=0)).fit(X_tr, y_tr)
print("rbf unscaled:", round(raw.score(X_te, y_te), 3),
"| rbf scaled:", round(scaled.score(X_te, y_te), 3))
# rbf unscaled: 0.911 | rbf scaled: 0.941
A straight line manages 0.752 on a problem whose true boundary is a parabola, and 338 of the 630 training crates end up as support vectors. That ratio is the diagnostic: when more than half your training data is propping up the boundary, the kernel is wrong for the shape of the data.
Kernels, and the grid you have to search
A kernel is a similarity function between two rows. Swap the dot product in the SVM objective for a different similarity measure and the boundary becomes curved in the original feature space, without you ever constructing the extra features. The radial basis function kernel scores two points by how close they are, with gamma controlling how fast that score decays with distance. Low gamma means every point influences a wide region and the boundary comes out smooth; high gamma means influence is local and the boundary can wrap tightly around individual points.
C and gamma interact, so tuning one at a time will mislead you. Search them together.
rows = []
for C in [0.01, 0.1, 1, 10, 100]:
for g in ["scale", 0.1, 1.0, 10.0]:
m = make_pipeline(StandardScaler(),
SVC(kernel="rbf", C=C, gamma=g, random_state=0)).fit(X_tr, y_tr)
rows.append({"C": C, "gamma": g,
"train": round(m.score(X_tr, y_tr), 3),
"test": round(m.score(X_te, y_te), 3),
"n_sv": int(m[-1].n_support_.sum())})
print(pd.DataFrame(rows).to_string(index=False))
# C gamma train test n_sv
# 0.01 scale 0.770 0.770 510
# 0.01 0.1 0.597 0.596 509
# 0.10 scale 0.925 0.926 276
# 0.10 10.0 0.898 0.893 491
# 1.00 scale 0.922 0.941 154
# 1.00 10.0 0.946 0.948 299
# 10.00 scale 0.933 0.948 118
# 10.00 10.0 0.956 0.948 247
# 100.00 scale 0.933 0.944 103
# 100.00 10.0 0.965 0.907 221
Three patterns in that grid are worth more than the best cell. Support vector count falls as C rises, from 510 at C=0.01 down to 103 at C=100, because an expensive penalty forces the boundary to commit. Accuracy plateaus around 0.948 across several very different settings, so the choice between them is noise rather than signal. And the bottom-right corner is the classic failure: C=100 with gamma=10 reaches 0.965 on training data and drops to 0.907 on test, the widest train-test gap in the table.
The default gamma="scale" divides by the number of features times their variance. It is a sensible starting point and, on this problem, matches or beats every hand-picked value at C of 1 or above.
When an SVM is the wrong choice
Kernel SVMs scale badly. Training cost sits between quadratic and cubic in the number of rows, so the method that shines on 900 crates becomes impractical somewhere in the hundreds of thousands. Worse, when the true boundary is close to linear, the kernel buys you nothing while charging the full price.
import time
from sklearn.svm import LinearSVC
rng2 = np.random.default_rng(650)
Xw = rng2.normal(size=(8000, 120))
w = rng2.normal(size=120) * (rng2.random(120) < 0.25)
yw = ((Xw @ w + rng2.normal(0, 2.5, 8000)) > 0).astype(int)
Xw_tr, Xw_te, yw_tr, yw_te = train_test_split(Xw, yw, test_size=0.3, random_state=0)
for name, model in [
("LinearSVC", make_pipeline(StandardScaler(),
LinearSVC(C=1.0, dual="auto", random_state=0))),
("SVC rbf", make_pipeline(StandardScaler(),
SVC(kernel="rbf", C=1.0, random_state=0)))]:
t0 = time.perf_counter()
model.fit(Xw_tr, yw_tr)
print(f"{name:10s} test acc {model.score(Xw_te, yw_te):.3f} "
f"fit {time.perf_counter() - t0:.2f}s")
# LinearSVC test acc 0.857 fit 0.05s
# SVC rbf test acc 0.855 fit 2.46s
Identical accuracy to within a thousandth, and roughly fifty times the fitting time in that run. Wall-clock timings move around on a shared machine, so treat the ratio as an order of magnitude rather than a measurement, and note that it grows with row count. On wide data with a linear signal, LinearSVC is the right tool and the kernel is a tax. The other two reasons to walk away: an SVM gives you a signed distance rather than a probability, and probability=True fits a separate calibration model by cross-validation, which is slow and often poorly calibrated on small data. If you need honest probabilities, logistic regression is the better starting point. If you need to explain the model to an auditor, how decision trees split the data will be read and understood where a kernel boundary will not. And on low-dimensional problems with plenty of rows, how k-nearest neighbours classifies a new point often matches an RBF SVM for a fraction of the tuning effort.
The scikit-learn user guide on support vector machines sets out the full kernel list and the complexity figures.
Frequently Asked Questions
What is the difference between SVC and LinearSVC?
SVC solves the kernel formulation and supports linear, poly, rbf and sigmoid kernels. LinearSVC solves the linear case directly with a different solver that scales far better in rows and features. For a linear boundary on a large dataset, use LinearSVC. Their results differ slightly because they penalise the intercept differently.
Do support vector machines need feature scaling?
Yes, always. The margin is a geometric distance, so a feature with a large numeric range dominates it regardless of how informative that feature is. On the bottling data an unscaled RBF model scored 0.911 against 0.941 for the same model inside a StandardScaler pipeline.
How many support vectors is too many?
There is no fixed threshold, but if a majority of your training rows end up as support vectors, the model is struggling. At C=0.01 the bottling model kept 510 of 630 rows and scored 0.770. At C=10 it kept 118 and scored 0.948.
Key Takeaways
- Tune
Candgammatogether on a grid, because their effects interact and a one-at-a-time search will settle on the wrong pair. - Always wrap an SVM in a
StandardScalerpipeline, since the margin is measured in your features’ own units. - Treat the support vector count as a diagnostic: a high fraction of training rows retained means the kernel or
Cdoes not suit the data. - Reach for
LinearSVCwhen the signal is linear and the data is wide, because the RBF kernel cost one to two orders of magnitude more fitting time here for no accuracy gain. - Pick a different algorithm when you need calibrated probabilities or an explainable boundary, as the SVM gives you neither for free.