Reinforcement Learning in Brief

Learning from consequences, not answers

Reinforcement learning trains a decision-maker by letting it act and observing what happens. There is no dataset of correct answers. An agent picks an action, the environment responds with a reward or a penalty, and over many repetitions the agent works out which actions pay.

The difference from the labelled examples supervised learning consumes is sharper than it first appears. A supervised model is told “the answer for this row was 7”. An agent is told “that action earned you 0.3” — not what it should have done, only how the thing it did turned out. It must work out the better option by trying alternatives, and every trial costs something.

This chapter is deliberately short. Reinforcement learning is a large field with its own textbooks, and you will not use it in most data roles. What you do need is enough to recognise the shape of an RL problem, follow a conversation about it, and — more usefully — recognise when someone is proposing it for a problem that does not need it.

The five pieces of the loop

Every reinforcement learning setup decomposes into the same parts. A warehouse robot deciding which aisle to enter next makes them concrete.

The agent is the thing making decisions. The robot’s controller.

The environment is everything else, including the parts the agent cannot see. The warehouse, the stock, the humans wandering through it.

The state is what the agent knows right now. Current position, battery level, the pick list outstanding.

The action is what it can do from here. Move north, move east, lift, wait.

The reward is a number arriving after each action. Plus ten for a completed pick, minus one per second elapsed, minus five hundred for touching a person.

What the agent learns is a policy: a mapping from state to action, chosen to maximise total reward over time rather than the reward from the next step alone. That “over time” clause is what makes the problem hard, and it is why the optimisation techniques from how machines learn from data do not transfer directly.

Notice that the reward function is written by you. It is not discovered in the data. Get it wrong and the agent will do exactly what you asked instead of what you meant — a robot penalised per second will learn to take corners fast enough to knock stock off shelves, because nothing in the reward mentioned the stock.

Explore or exploit, in twenty lines

The core tension is easiest to see in the simplest possible RL problem, the multi-armed bandit: several options, unknown payoffs, and a limited number of tries. A coffee chain testing three promotional banners has exactly this problem.

Every time you show a banner you learn something about it, and you give up the chance to show the one you currently believe is best. Explore too little and you lock onto a mediocre option early. Explore too much and you spend the whole campaign on banners you already know are worse.

import numpy as np

rng = np.random.default_rng(42)
true_rates = np.array([0.03, 0.055, 0.04])   # real click-through, unknown to agent

counts = np.zeros(3)     # times each banner was shown
values = np.zeros(3)     # running estimate of each banner's rate
epsilon = 0.1            # 10% of the time, try something at random

for impression in range(5000):
    if rng.random() < epsilon:
        arm = rng.integers(3)              # explore
    else:
        arm = int(np.argmax(values))       # exploit current best

    reward = 1.0 if rng.random() < true_rates[arm] else 0.0
    counts[arm] += 1
    values[arm] += (reward - values[arm]) / counts[arm]   # running mean

print(counts.astype(int))   # [ 342 3880  778]
print(values.round(4))      # [0.0322 0.0544 0.036 ]

The agent concentrated 78% of impressions on banner 1 and estimated its true 0.055 rate as 0.0544. Nobody told it which banner was best. It found out by paying for the information, and the 342 impressions spent on banner 0 are the price of knowing that banner 0 was worse.

epsilon controls that trade directly. Set it to zero and the agent commits to whatever looked best after a handful of trials, which is often wrong. Set it to 0.5 and half your budget goes to deliberately suboptimal choices forever. In production you usually decay it — explore heavily at first, then settle.

Why the full problem is much harder

The bandit above has one state and immediate rewards. Real reinforcement learning has neither, and both gaps are severe.

Credit assignment. A chess agent loses on move 60. Which move was the mistake? The reward arrives long after the decision that caused it, and the agent must apportion blame across a long chain of actions, most of which were fine. This is the central technical problem of the field.

Sample hunger. Systems that learn to play Atari or Go consume millions of episodes. That is affordable in a simulator and impossible against reality, which is why serious RL almost always means building a simulator first — and a simulator accurate enough to train against is frequently harder to build than the original problem.

Non-stationarity. The agent’s own behaviour changes the environment. Its early data describes a world its later self no longer inhabits.

Reward hacking. Agents find loopholes with unnerving reliability. A boat-race agent famously learned to spin in circles collecting respawning bonus targets rather than finish the race. The reward said points; nobody wrote “and also finish”.

When reinforcement learning is the wrong tool

Almost always, in a normal data job. The honest test has three parts, and you need all three: the decision must repeat many thousands of times, the outcome must feed back quickly enough to learn from, and you must be able to run it either in a cheap simulator or on live traffic you are willing to sacrifice.

Miss any one and pick something else. If you have a fixed historical dataset and a known outcome, that is a supervised problem — the framing in the supervision axis of machine learning systems will tell you which family you are in faster than any algorithm comparison. If you want to find structure with no outcome in mind, unsupervised methods apply.

The narrow slice where RL genuinely earns its keep: content and ad ranking under live traffic, dynamic pricing, robotics with a good simulator, recommendation where today’s suggestion changes tomorrow’s behaviour, and reinforcement learning from human feedback for language models. Bandits specifically are worth knowing because they are cheap, they run on real traffic, and they beat a fixed A/B test whenever you care about revenue during the experiment rather than only after it.

For the full treatment, Sutton and Barto’s Reinforcement Learning: An Introduction is the standard reference and is free from the authors.

Frequently Asked Questions

Is reinforcement learning supervised or unsupervised?

Neither. It is a third category. Supervised learning gets the correct answer for each row; unsupervised learning gets no feedback at all. Reinforcement learning gets evaluative feedback — a score telling it how good its choice was, without revealing what the better choice would have been.

Do you need a simulator for reinforcement learning?

For most non-trivial problems, yes. Agents need enormous numbers of trials, and running those against reality is slow, expensive or dangerous. Bandit problems are the exception: they have few states and quick feedback, so they run successfully on live traffic. That is why they see far more commercial use.

Is reinforcement learning used in real products?

Yes, in a narrower set than the publicity suggests. Recommendation and ad ranking, dynamic pricing, data-centre cooling, robotic control, and the human feedback stage used to align large language models. Outside those areas most production machine learning remains supervised, and that is unlikely to change soon.

Key Takeaways

• Test any proposed reinforcement learning project against three requirements — many repetitions, fast feedback, and a simulator or expendable live traffic — and reframe it as supervised learning if it misses even one.
• Write the reward function with as much care as the algorithm, since the agent will optimise precisely what you wrote rather than what you intended.
• Tune the exploration rate deliberately and decay it over time, because a fixed zero locks in early mistakes and a fixed high value wastes budget forever.
• Reach for a multi-armed bandit rather than a fixed A/B test when you care about performance during the experiment, not just the verdict at the end.
• Expect credit assignment and sample efficiency to dominate the engineering effort on any RL system with delayed rewards.