Nobody wrote the answers down
A hardware chain has four years of till receipts. Nobody ever recorded which customers are “trade” and which are “weekend DIY”, because nobody was asked to. The categories are not missing from the database — they never existed. Yet anyone who has worked a Saturday shift will tell you the two groups behave nothing alike.
Unsupervised learning is what you use when the data has no labels and you want the structure that is already sitting in it. There is no target column, no right answer per row, and therefore no performance measure of the kind every machine learning problem normally needs. The algorithm looks at how rows relate to each other and reports what it finds.
That last sentence contains the whole difficulty. With the labelled data supervised learning requires, a bad model announces itself with a bad score. Here, a bad model returns a tidy, confident answer that is completely wrong, and nothing in the output tells you. Evaluating the result is the hard part of the job, not fitting it.
What unsupervised learning actually finds
Four broad jobs cover almost everything you will meet.
Clustering groups rows that resemble each other. Customer segments, document topics, sensor readings that behave alike. KMeans, DBSCAN, agglomerative clustering.
Dimensionality reduction compresses many columns into a few that keep most of the variation. Two hundred survey questions become five underlying attitudes. PCA is the workhorse; t-SNE and UMAP are for visualisation rather than downstream modelling.
Anomaly detection learns what normal looks like and flags departures from it. Useful precisely when you cannot label the rare thing, because you have seen only a handful of examples. IsolationForest, OneClassSVM.
Association rule mining finds items that co-occur. The supermarket basket question — people who buy this also buy that.
Dimensionality reduction is the one beginners underrate. It rarely produces the deliverable itself, but it makes everything downstream faster and less prone to overfitting, and it is often the honest fix when you have four hundred columns and eight hundred rows.
Clustering a customer base
Here is the hardware chain’s problem, with two behavioural columns.
import numpy as np
import pandas as pd
from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import silhouette_score
rng = np.random.default_rng(3)
baskets = pd.DataFrame({
"visits_per_year": np.concatenate([
rng.normal(4, 1.2, 120), rng.normal(26, 4, 120), rng.normal(11, 2.5, 120)]),
"avg_basket_gbp": np.concatenate([
rng.normal(240, 40, 120), rng.normal(38, 9, 120), rng.normal(95, 18, 120)]),
})
X = StandardScaler().fit_transform(baskets) # scaling is not optional here
km = KMeans(n_clusters=3, n_init=10, random_state=3).fit(X)
baskets["segment"] = km.labels_
print(baskets.groupby("segment").mean().round(1))
# visits_per_year avg_basket_gbp
# segment
# 0 26.2 39.2
# 1 3.9 244.9
# 2 11.2 96.2Segment 0 comes in weekly and spends about £39 — the trade counter. Segment 1 appears four times a year and spends £245 — the kitchen-refit customer. Segment 2 sits between them. Nobody defined those groups; the algorithm recovered them from spending behaviour alone.
StandardScaler matters more than it looks. K-means measures straight-line distance between rows, so a column running to 300 dominates a column running to 30 purely because of its units. Skip the scaling and you will cluster on whichever variable happens to be measured in larger numbers.
K-means always returns k clusters
Now the uncomfortable part. Feed it pure noise and it still answers:
noise = rng.normal(0, 1, (360, 2)) # no structure whatsoever
km_noise = KMeans(n_clusters=3, n_init=10, random_state=3).fit(noise)
print(round(silhouette_score(noise, km_noise.labels_), 3)) # 0.349Three neat segments, carved out of random numbers. The algorithm cannot decline. Ask for three clusters and you get three clusters, whether or not three groups exist, and the output is a clean table you could put in a deck on Monday morning.
Judging a result with no right answer
Since there is no accuracy score, you need substitutes. Use all three of these, because each catches something the others miss.
An internal metric. The silhouette score measures how much tighter each row sits with its own cluster than with the nearest neighbouring one, on a scale from -1 to 1. Sweep across candidate values of k and look for a peak:
for k in [2, 3, 4, 5]:
labels = KMeans(n_clusters=k, n_init=10, random_state=3).fit_predict(X)
print(k, round(silhouette_score(X, labels), 3))
# 2 0.618
# 3 0.708 <- genuine structure
# 4 0.623
# 5 0.548The real data peaks at 0.708 and the noise scored 0.349, so the metric does discriminate. But 0.349 is not zero, which is exactly the warning: a mediocre score does not prove there is nothing there, and a decent score does not prove there is.
Stability. Refit on a random 80% of rows, several times, with different seeds. Segments that survive resampling are probably real. Segments that dissolve were artefacts of one particular fit.
A domain check. Show the profile table to the person who runs the trade counter. If the segments match categories the business already recognises under different names, you have found something. If nobody can describe what segment 2 means, you have found an arithmetic result rather than a customer group.
The third check is the one people skip and the one that matters most. I would rather ship four clusters a category manager can name than six with a marginally better silhouette score.
Where it fails, and what to reach for instead
The characteristic failure is not a wrong number. It is a plausible answer to a question nobody should have asked. Clustering will always partition your data, and human beings are extremely willing to invent a story for each partition after the fact.
Three situations where you should not be here at all. If you already know the categories and have examples of each, label a sample and classify — you will get a better result and be able to prove it. If the structure you want is defined by an outcome (customers likely to churn), that outcome is a label, so this is a supervised problem wearing a disguise — and the supervision axis that classifies machine learning systems will settle that question in a sentence. If your columns are mostly categorical with high cardinality, distance-based clustering degrades badly, and you are better off with association rules or a purpose-built method.
Unsupervised methods earn their place when the categories genuinely do not exist yet, when labelling is impossible at any price, or as a preprocessing step that feeds something else. Used that way they are indispensable. Used as a substitute for deciding what you want to know, they generate confident nonsense.
Both estimators above are documented in the scikit-learn guides to clustering methods and silhouette_score.
Frequently Asked Questions
How do you know if unsupervised learning worked?
You cannot score it directly, so triangulate. Check an internal metric such as silhouette, refit on random subsets to see which groups survive, and show the profiles to someone with domain knowledge. Agreement across all three is evidence. Any one of them alone is easy to fool.
Is clustering the same as unsupervised learning?
Clustering is the best-known member of the family, not the whole of it. Dimensionality reduction, anomaly detection and association rule mining are also unsupervised, since none uses labelled outcomes. Clustering dominates introductions because its output is easy to picture, which slightly distorts people’s sense of the field.
When should you use unsupervised learning instead of supervised?
Use it when no labels exist and creating them is impossible or unaffordable, or when you want to discover categories rather than sort rows into ones you already defined. If you know the outcome you care about and can label even a few thousand rows, a supervised model will beat it and you can prove the difference.
Key Takeaways
- Scale your features before any distance-based method, because k-means will otherwise cluster on whichever column happens to use the largest units.
- Treat every clustering result as a hypothesis, since the algorithm partitions random noise just as confidently as it partitions real structure.
- Validate with an internal metric, a stability check across resampled fits, and a domain expert who can name the groups — no single check is sufficient.
- Reframe the problem as supervised whenever the structure you want is defined by an outcome, because a churn segment is a label in disguise.
- Use dimensionality reduction as a preprocessing step when columns outnumber rows, rather than treating it as a deliverable in itself.