Types of Machine Learning Systems

Three questions, not one label

People usually classify a machine learning system with a single word — “it’s a supervised model” — and stop there. That word answers only one of three questions, and the other two determine more about how the system behaves in production.

Every system can be placed on three independent axes:

1. How much supervision does it get during training?
2. Does it learn from all the data at once, or continuously?
3. Does it generalise by comparing to remembered examples, or by building a model?

The axes are independent. You can have a supervised, online, model-based system, or a supervised, batch, instance-based one. Naming all three tells you what the system needs, how it fails, and what it will cost to run.

Axis 1: how much supervision

This is the axis you already half know, and each of these has its own chapter, so one line each is enough here.

Supervised learning trains on examples where the correct answer is attached — past deliveries with the actual minutes, subject lines with a spam label. Both examples in machine learning vs traditional programming were supervised.

Unsupervised learning trains on data with no answers attached and looks for structure — natural groupings of customers, unusual transactions, a compressed representation of high-dimensional data.

Semi-supervised learning sits between them, using a small labelled set alongside a large unlabelled one, which matches the common real situation where data is cheap and labels are expensive.

Reinforcement learning has no fixed dataset at all. An agent takes actions in an environment and receives rewards or penalties, learning a policy by trial over many episodes.

The practical question this axis answers is: what do I have to collect before I can start? Supervised methods are the most capable and the most demanding, because someone has to produce the labels, and that is usually the real budget line in a project.

Axis 2: batch or online

Batch learning trains on the whole dataset in one go, offline. The fitted model is then deployed and does not change. When the world moves, you retrain from scratch on fresh data and deploy a new version. Every .fit() call so far in this course has been batch learning.

It is simple, reproducible, and easy to review — you can point at exactly which model version made which decision. The costs are that retraining on a very large dataset is expensive in time and compute, and that the model is stale between releases.

Online learning feeds data in sequentially, in small batches, updating the parameters as it goes. Nothing is discarded and refitted; the model keeps moving.

In scikit-learn this is partial_fit() rather than fit():

import numpy as np
from sklearn.linear_model import SGDRegressor

# flat size in m², monthly rent in thousands
size = np.array([[38.], [45.], [52.], [60.], [68.], [75.], [83.], [90.]])
rent = np.array([14., 16., 19., 22., 24., 27., 31., 33.])

s = (size - size.mean()) / size.std()
query = (np.array([[64.]]) - size.mean()) / size.std()

sgd = SGDRegressor(learning_rate="constant", eta0=0.05, random_state=0)

for epoch in range(1, 5):
    for c in range(0, 8, 2):
        sgd.partial_fit(s[c:c + 2], rent[c:c + 2])   # two rows at a time
    print(epoch, sgd.predict(query).round(2))

# 1 [ 8.1 ]
# 2 [13.39]
# 3 [16.84]
# 4 [19.09]

The estimate climbs towards the right answer without the full dataset ever being present in one call. That is the point: online learning suits data arriving as a stream, datasets too large to hold in memory, and settings where patterns shift faster than a retraining cycle — pricing, ad ranking, fraud.

Its failure mode is severe and worth stating plainly. Because the model absorbs whatever arrives, a burst of bad or adversarial data degrades it within minutes, and there is no previous version sitting in memory to fall back on. Anyone running online learning needs input monitoring and the ability to roll back to a saved snapshot. The learning_rate setting controls how fast old information is forgotten, which is exactly the tradeoff between adapting quickly and being easy to corrupt.

Axis 3: instance-based or model-based

The last axis is about what happens when a genuinely new input arrives.

Instance-based systems memorise the training examples and answer by similarity. Nothing is condensed. k-nearest neighbours finds the closest stored rows and averages their answers.

Model-based systems compress the training data into parameters — the coefficients from how machines learn from data — and then discard the rows entirely. Prediction is arithmetic on those parameters.

The difference is invisible on ordinary inputs and dramatic at the edges:

import numpy as np
from sklearn.neighbors import KNeighborsRegressor
from sklearn.linear_model import LinearRegression

size = np.array([[38.], [45.], [52.], [60.], [68.], [75.], [83.], [90.]])
rent = np.array([14., 16., 19., 22., 24., 27., 31., 33.])

knn = KNeighborsRegressor(n_neighbors=2).fit(size, rent)
lin = LinearRegression().fit(size, rent)

# A 64 m² flat — comfortably inside the training range
print(knn.predict([[64.]]).round(2), lin.predict([[64.]]).round(2))
# [23.] [23.3]

# A 140 m² flat — far outside anything seen
print(knn.predict([[140.]]).round(2), lin.predict([[140.]]).round(2))
# [32.] [51.48]

Inside the range they agree. Outside it they diverge completely. The k-nearest neighbours model returns 32, because the two largest flats it remembers rent for 31 and 33 and it cannot conceive of anything beyond them. The linear model extrapolates its learned rate of roughly 0.37 per m² and says 51.5.

Neither is automatically right. If rent per square metre flattens for large flats, the instance-based answer is closer. If it continues, the model-based one is. What matters is knowing which behaviour you have bought: instance-based methods are conservative and refuse to extrapolate, model-based methods extrapolate confidently and can be confidently wrong.

There is an operational difference too. Instance-based methods train instantly and predict slowly, because prediction means searching the stored data, and the whole training set has to ship with the model. Model-based methods train slowly and predict in microseconds from a handful of numbers.

Reading a real system through all three axes

Read the middle row. Supervised tells you that you need labelled clicks. Online tells you that you need streaming infrastructure and a rollback plan. Model-based tells you predictions will be fast and cheap, and that it will extrapolate into regions it has never seen. Three words, and you already know most of what the project requires.

The axes move over a project’s life

A system’s position is not fixed. Almost every project should start batch, supervised and model-based, because that combination is the easiest to build, the easiest to evaluate and the easiest to explain to whoever signs off on it. You fit on a snapshot, you measure, you deploy a single artefact.

Systems migrate later, and always for a specific reason. A team moves to online learning when they can show the model going stale between releases — not because streaming sounds modern. A team moves to an instance-based method when extrapolation is actively harmful, or when new categories appear constantly and refitting cannot keep up. A team drops to semi-supervised when the labelling bill stops being payable.

So when you inherit a system, ask why it sits where it does. An online model with no monitoring is usually a team that wanted freshness and did not price the failure mode. Naming the axes is a design tool, but it is a review tool too.

Get into the habit of naming all three before you choose an estimator. The choice of algorithm follows from the answers rather than the other way round.

Key Takeaways

• Classify every system on all three axes — supervision, batch versus online, instance versus model-based — because each one determines a different part of your infrastructure and cost.
• Treat labelling as the real constraint on supervised projects, and check whether a semi-supervised or unsupervised framing gets you far enough.
• Add input monitoring and a rollback snapshot before deploying online learning, since bad data corrupts a continuously updating model within minutes.
• Expect instance-based models to refuse to extrapolate and model-based ones to extrapolate confidently, and pick the behaviour that matches how your quantity actually behaves at the extremes.
• Name the three axes before choosing an estimator, because the algorithm should follow from the requirements rather than the reverse.