Start from the shape of the data

Choosing the right classifier has no general answer, and anyone who tells you one algorithm wins is describing their last project rather than yours. What does have an answer is narrower and more useful: given the shape of this dataset and these constraints, which two or three candidates are worth fitting? Everything after that is a bake-off you run in about twenty lines.

The properties that actually decide it are the geometry of the boundary, the number of rows against the number of features, whether the features interact, how much you care about calibrated probabilities, and what you can afford at prediction time. Not the algorithm’s reputation.

Here is the bake-off. Three synthetic datasets with deliberately different structure, five classifiers from this section, one split each.

import time
import numpy as np
import pandas as pd
from sklearn.base import clone
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC

def wide_linear(rng, n=3000, d=200):
    X = rng.normal(size=(n, d))
    w = rng.normal(size=d) * (rng.random(d) < 0.10)
    return X, ((X @ w + rng.normal(0, 2.0, n)) > 0).astype(int)

def curved_band(rng, n=1500):
    X = np.column_stack([rng.normal(4.20, 0.45, n), rng.normal(1.55, 0.30, n)])
    risk = 2.9 * (X[:, 0] - 4.20) ** 2 - 1.8 * (X[:, 1] - 1.55) - 0.33
    return X, (risk + rng.normal(0, 0.18, n) > 0).astype(int)

def step_interactions(rng, n=4000, d=8):
    X = rng.uniform(0, 1, size=(n, d))
    hit = ((X[:, 0] > 0.6) & (X[:, 1] < 0.4)) | ((X[:, 2] > 0.7) & (X[:, 3] > 0.5))
    return X, (hit ^ (rng.random(n) < 0.08)).astype(int)

MODELS = {
    "logistic regression": make_pipeline(StandardScaler(),
                                         LogisticRegression(max_iter=2000, random_state=0)),
    "gaussian naive bayes": GaussianNB(),
    "k-nearest neighbours": make_pipeline(StandardScaler(),
                                          KNeighborsClassifier(n_neighbors=15)),
    "decision tree": DecisionTreeClassifier(max_depth=6, random_state=0),
    "svm, rbf kernel": make_pipeline(StandardScaler(),
                                     SVC(kernel="rbf", C=10, random_state=0)),
}
DATASETS = {"wide and linear": wide_linear,
            "curved boundary": curved_band,
            "step interactions": step_interactions}

rows = []
for dname, builder in DATASETS.items():
    X, y = builder(np.random.default_rng(68))
    X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3,
                                              random_state=0, stratify=y)
    for mname, model in MODELS.items():
        t0 = time.perf_counter()
        fitted = clone(model).fit(X_tr, y_tr)
        rows.append({"dataset": dname, "model": mname,
                     "accuracy": round(fitted.score(X_te, y_te), 3),
                     "fit_ms": round((time.perf_counter() - t0) * 1000, 1)})

table = pd.DataFrame(rows)
print(table.pivot(index="model", columns="dataset", values="accuracy").to_string())
print(table.pivot(index="model", columns="dataset", values="fit_ms").to_string())
# dataset               curved boundary  step interactions  wide and linear
# model
# decision tree                   0.907              0.894            0.657
# gaussian naive bayes            0.920              0.782            0.780
# k-nearest neighbours            0.929              0.824            0.609
# logistic regression             0.744              0.772            0.846
# svm, rbf kernel                 0.944              0.844            0.830

Three datasets, three different winners. The RBF support vector machine takes the curved two-feature problem at 0.944, where logistic regression manages 0.744 because no straight line fits a parabolic band. The depth-6 decision tree takes the interaction problem at 0.894, because threshold-and-threshold rules are exactly what a tree represents. And on the wide linear problem, logistic regression wins at 0.846 while the tree collapses to 0.657 and nearest neighbours to 0.609, both ruined by 200 dimensions in which distance stops being informative.

No ranking survives across the three columns. That is the honest state of model selection, and it is why a benchmark on your data beats any amount of reading about which algorithm is strongest.

What the timings add

The second pivot prints fitting time, and those milliseconds depend on the machine, so read the ratios rather than the numbers. The ordering was stable across runs here: the RBF SVM was the slowest fit on all three datasets, tens of times slower than logistic regression on the 200-feature problem, while Gaussian naive Bayes fitted in single-digit milliseconds on all three.

Fitting time matters less than people assume, because you fit once and predict millions of times. Prediction cost is the one that bites. A tree answers with a handful of comparisons. Nearest neighbours has to search the training set for every single query, so its cost at serving time grows with the data you trained on. A kernel SVM evaluates the kernel against every support vector. If you are serving at 2,000 requests a second behind a 40 millisecond budget, that difference decides the architecture before accuracy gets a vote.

Sample size changes the answer

Run the same five models on growing slices of one dataset and the winner moves.

rng = np.random.default_rng(680)
X_big, y_big = step_interactions(rng, n=12000)
X_pool, X_hold, y_pool, y_hold = train_test_split(X_big, y_big, test_size=4000,
                                                  random_state=0, stratify=y_big)
sweep = []
for n in [150, 400, 1200, 8000]:
    row = {"n_train": n}
    for mname, model in MODELS.items():
        row[mname] = round(clone(model).fit(X_pool[:n], y_pool[:n]).score(X_hold, y_hold), 3)
    sweep.append(row)
sweep = pd.DataFrame(sweep).set_index("n_train")
print(sweep.to_string())
print("best at each size:", sweep.idxmax(axis=1).to_dict())
#          logistic regression  gaussian naive bayes  k-nearest neighbours  decision tree  svm, rbf kernel
# n_train
# 150                    0.750                 0.757                 0.727          0.721            0.734
# 400                    0.763                 0.763                 0.770          0.806            0.763
# 1200                   0.770                 0.776                 0.813          0.883            0.825
# 8000                   0.771                 0.778                 0.854          0.909            0.875
# best at each size: {150: 'gaussian naive bayes', 400: 'decision tree', 1200: 'decision tree', 8000: 'decision tree'}

At 150 training rows the crudest model wins. Gaussian naive Bayes assumes the features are independent given the class, which is flatly false for this data, and it still beats the tree at 0.757 against 0.721 because its wrong assumption is cheap and the tree cannot learn two interactions from 150 examples. By 400 rows the tree has overtaken everything, and the gap widens from there: 0.806, then 0.883, then 0.909, while logistic regression and naive Bayes flatten out around 0.77 no matter how much more data you give them.

That flattening is the bias ceiling. A model whose form cannot represent the true boundary stops improving, and extra rows are wasted on it. The flexible models keep climbing. So the right question is not which classifier is best but which is best at the sample size you have, and the answer at 150 rows is genuinely different from the answer at 8,000.

Pick a set of candidates, fit them all under cross-validation rather than one split, and compare with the metric that matches your decision. One split on 4,000 held-out rows is adequate for a demonstration and not adequate for a real choice; a few points of difference on a single split is frequently noise.

A checklist for choosing the right classifier

If this is true Start with Rather than
Many features, few rows, roughly linear signal Logistic regression or LinearSVC Trees or nearest neighbours
Few features, many rows, curved boundary RBF SVM or nearest neighbours Logistic regression
Threshold-style interactions between features Decision tree Naive Bayes
Tiny training set, need a baseline today Gaussian naive Bayes Anything tuned
You must explain each decision Decision tree or logistic regression Kernel SVM
Probabilities feed a cost calculation Logistic regression Naive Bayes or SVM
Strict prediction latency Logistic regression or a shallow tree Nearest neighbours

Two things the table leaves out. Fit a trivial baseline first, DummyClassifier(strategy="most_frequent"), because on an uneven label it sets the floor that everything else has to clear, as handling imbalanced classes shows in detail. And scale your features before you compare, since how k-nearest neighbours classifies a new point and how a support vector machine finds its boundary both depend on distance while a decision tree and a naive Bayes classifier do not. An unscaled bake-off compares the wrong thing.

Section 8 adds ensembles, which change these answers again, and gradient-boosted trees in particular tend to win on the kind of tabular data most jobs involve. The scikit-learn estimator selection map is a reasonable first filter when you want a shortlist rather than a bake-off.

Frequently Asked Questions

Which classifier is best for tabular data?

For tabular data with a few thousand rows or more, a gradient-boosted tree is the usual strongest single model, and section 8 covers that family. Among the algorithms in this section, a depth-limited decision tree held up best on interaction-heavy data, winning at every sample size above 400 rows in the sweep above.

How many models should I try?

Three or four, chosen for the shape of your data rather than at random, then tuned. Fitting fifteen untuned models tells you less than fitting three tuned ones, and it invites you to pick a winner from differences that are really variance between splits.

Does naive Bayes have any place with modern libraries?

Yes, as a first baseline and on genuinely small training sets. It was the cheapest model to fit in every column of the timing table and beat every other model here at 150 training rows. Its independence assumption is nearly always wrong, which is exactly why it needs so little data to fit.

Key Takeaways

  • Benchmark on your own data, because three datasets here produced three different winners and no ranking held across all of them.
  • Match the model to the boundary: curved and low-dimensional favours kernels and neighbours, wide and linear favours a linear model, threshold interactions favour trees.
  • Check the sample size before you choose, since naive Bayes won at 150 rows and lost badly at 8,000 on the same problem.
  • Weigh prediction latency, not fitting time, because nearest neighbours and kernel SVMs carry their training data into every query.
  • Fit a DummyClassifier baseline and scale your features first, or your comparison measures preprocessing rather than algorithms.