Four classes, and three ways to get there

Most of the algorithms in this section were designed for a yes-or-no answer. Multi-class classification is what you do when the label has three or more values that are mutually exclusive: one of four material streams, one of seven fault codes, one of twelve product categories. There are three strategies, and scikit-learn picks one for you unless you say otherwise.

One-vs-rest fits one binary model per class, each trained to separate its own class from everything else, then takes the class whose model is most confident. One-vs-one fits a model for every pair of classes and holds a vote. A handful of algorithms skip the wrapping entirely and optimise all classes at once, which is what LogisticRegression does with its softmax objective and what a decision tree does naturally by putting a class label in each leaf.

The practical consequences differ more than the descriptions suggest. One-vs-rest fits k models but each sees the whole dataset. One-vs-one fits k * (k - 1) / 2 models, more of them, but each sees only the rows belonging to two classes. For four classes that is six small models against four large ones, and the six are often faster.

Here is the data: a materials recovery facility sorting a conveyor with a near-infrared head.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(66)

centres = {
    "PET":       (0.62, 0.31, 0.18, 32.0),
    "HDPE":      (0.58, 0.35, 0.21, 38.0),
    "paper":     (0.30, 0.64, 0.57, 12.0),
    "aluminium": (0.44, 0.58, 0.49, 16.0),
}
counts = {"PET": 1300, "HDPE": 1000, "paper": 1500, "aluminium": 200}
spread = np.array([0.075, 0.080, 0.075, 7.0])

frames = []
for label, (b1, b2, b3, mass) in centres.items():
    block = rng.normal(np.array([b1, b2, b3, mass]), spread, size=(counts[label], 4))
    frames.append(pd.DataFrame(block, columns=["band_940", "band_1200",
                                               "band_1650", "mass_g"])
                  .assign(material=label))

items = pd.concat(frames, ignore_index=True).sample(frac=1, random_state=0)
items = items.reset_index(drop=True)
print(items.shape)
print(items["material"].value_counts().to_dict())
# (4000, 5)
# {'paper': 1500, 'PET': 1300, 'HDPE': 1000, 'aluminium': 200}

X = items[["band_940", "band_1200", "band_1650", "mass_g"]].to_numpy()
y = items["material"].to_numpy()
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          random_state=0, stratify=y)

PET and HDPE overlap heavily, which is true of the real problem too. Aluminium is both rare and sitting between the plastics and the paper cluster.

What changes in the fitted object

Pass a y with four distinct values and the estimator reshapes itself. Reading those shapes is the fastest way to tell which strategy you got.

from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

soft = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000, random_state=0))
soft.fit(X_tr, y_tr)
print("classes_:", soft[-1].classes_.tolist())
print("coef_ shape:", soft[-1].coef_.shape)
print("decision_function shape:", soft.decision_function(X_te[:5]).shape)
print("one row of predict_proba:", soft.predict_proba(X_te[:1]).round(3).tolist())
print("accuracy:", round(soft.score(X_te, y_te), 3))
# classes_: ['HDPE', 'PET', 'aluminium', 'paper']
# coef_ shape: (4, 4)
# decision_function shape: (5, 4)
# one row of predict_proba: [[0.123, 0.876, 0.001, 0.0]]
# accuracy: 0.806

Four classes by four features gives a coef_ of shape (4, 4), one coefficient vector per class. classes_ is sorted, and it is sorted as strings, so aluminium lands third and the column order of predict_proba follows that list rather than the order you wrote the dictionary in. Index those columns by looking up the label in classes_, never by a position you assumed.

predict_proba rows sum to one across classes, because softmax normalises over the set. That is a real difference from a one-vs-rest wrapper, where each binary model produces its own independent probability and the row does not sum to anything in particular. If you are thresholding probabilities rather than taking the argmax, that distinction decides whether your threshold means what you think it means.

Choosing between one-vs-rest and one-vs-one

When the base estimator has no native multi-class support, you wrap it. SVC wraps itself with one-vs-one by default; LinearSVC uses one-vs-rest. You can force either with an explicit wrapper, which also lets you count what was fitted.

from sklearn.multiclass import OneVsRestClassifier, OneVsOneClassifier
from sklearn.svm import SVC
import time

base = SVC(kernel="rbf", C=10, random_state=0)
for name, wrapper in [("one-vs-rest", OneVsRestClassifier(base)),
                      ("one-vs-one", OneVsOneClassifier(base))]:
    t0 = time.perf_counter()
    m = make_pipeline(StandardScaler(), wrapper).fit(X_tr, y_tr)
    print(f"{name:12s} binary models: {len(m[-1].estimators_):2d}   "
          f"accuracy: {m.score(X_te, y_te):.3f}   "
          f"fit {time.perf_counter() - t0:.2f}s")
# one-vs-rest  binary models:  4   accuracy: 0.810   fit 0.27s
# one-vs-one   binary models:  6   accuracy: 0.808   fit 0.14s

Six models, and consistently the faster of the two across repeated runs. Each one-vs-one model trains on the rows of two classes only, and since the SVM’s cost grows faster than linearly in rows, cutting the training set beats cutting the model count. The accuracy difference, 0.810 against 0.808, is two items out of a thousand and should not be read as a result.

My default is to leave the strategy alone unless profiling says otherwise. The exception is one-vs-rest on a lopsided label. Each of its binary models faces a split more extreme than the original problem, because every class that is not the target gets folded into the negative side: the aluminium model here trains on 5% positives against 95% everything-else. One-vs-one never does that, since each model sees exactly two classes.

Two pointers rather than detours. How logistic regression produces probabilities is the binary case this chapter generalises, how a support vector machine finds its boundary explains why the pairwise split is cheap for SVC, and the logistic regression chapter in the Predictive Modelling course covers the statistical reading of multinomial coefficients that this chapter deliberately skips.

Why plain accuracy lies in multi-class classification

A single accuracy number across four classes is an average weighted by how common each class is. The rare class can fail completely and barely move it.

from sklearn.metrics import classification_report, f1_score
pred = soft.predict(X_te)
print(classification_report(y_te, pred, digits=3))
#               precision    recall  f1-score   support
#         HDPE      0.675     0.648     0.661       250
#          PET      0.737     0.760     0.748       325
#    aluminium      0.844     0.540     0.659        50
#        paper      0.941     0.987     0.964       375
#     accuracy                          0.806      1000
print("macro F1   :", round(f1_score(y_te, pred, average="macro"), 3))
print("weighted F1:", round(f1_score(y_te, pred, average="weighted"), 3))
# macro F1   : 0.758
# weighted F1: 0.803

Accuracy says 0.806, which sounds like a working sorter. The per-class rows say the plant is dropping 46% of its aluminium, the one stream with real scrap value. Precision on aluminium is high at 0.844, so when the model does call aluminium it is usually right; it simply refuses to call it often enough.

Macro F1 averages the per-class scores with equal weight and lands at 0.758. Weighted F1 weights by class frequency and lands at 0.803, close to accuracy. The gap between those two numbers, 0.045 here, is a direct measure of how much your headline figure depends on the easy majority classes. Report macro when every class matters operationally. Report weighted when you genuinely care about overall throughput. Reporting only accuracy on an uneven label is how a failing rare class ships to production.

Paper is nearly perfect at 0.964 F1 because its reflectance signature sits far from the others. Almost all remaining error is PET against HDPE, two plastics whose near-infrared bands genuinely overlap. No amount of model selection fixes that; it needs a fourth sensor band. Knowing which pair of classes the errors live in is worth more than any aggregate, and scikit-learn’s multiclass user guide lists which estimators handle this natively and which get wrapped.

Frequently Asked Questions

Is one-vs-rest or one-vs-one better?

Neither wins on accuracy often enough to matter. Pick on cost: one-vs-one fits more models but each on a smaller slice, which made it the faster of the two on this four-class SVM despite fitting six models against four. One-vs-rest keeps one interpretable score per class, which is handier when you want to threshold rather than argmax.

Can I use accuracy for multi-class problems?

You can, but it hides uneven per-class performance. The sorter above was 80.6% accurate while recovering only 54% of aluminium. Look at macro F1 alongside accuracy; a wide gap between macro and weighted averages tells you the headline number is carried by your common classes.

Do I need to one-hot encode the target for multi-class?

No. Pass the label column as it is, whether strings or integers, and scikit-learn handles the encoding internally through classes_. One-hot targets are a neural network convention, and LabelBinarizer only becomes necessary when you are computing certain metrics by hand.

Key Takeaways

  • Check coef_ and classes_ after fitting, because the column order of predict_proba follows the sorted classes_ list and not the order in your data.
  • Leave scikit-learn’s default strategy alone unless profiling tells you otherwise, since the accuracy difference between one-vs-rest and one-vs-one is usually noise.
  • Expect one-vs-one to be faster than its larger model count suggests, because each pairwise model trains on only two classes’ rows.
  • Report macro F1 beside accuracy, and treat a large gap between macro and weighted averages as a warning that a rare class is failing.
  • Read the per-class table before touching the model, because knowing that your errors concentrate in one class pair usually points at a data problem rather than an algorithm problem.