A concept trilogy · how machines learn

Two Rivers

One stream learns by being shown the answer. The other learns by being told the score. They rise from the same spring in the 1940s, run side by side for fifty years, and meet in the systems we use now.

Made by [email protected]

41 videos · Complete · overture, three volumes, finale and hidden track

Overture

The Spine

The overture is the whole map in one song. Each volume then opens with its own explainer track that teaches its river end to end, so the tracks after it can go deep on one person or one idea.

Two Rivers

Overture · the spine · Gospel

In the roomGives you the whole map in one song: two ways of learning rise from the same 1940s spring, run apart for fifty years, and meet in the systems we use now.

Volume I

The Network

River One · deep learning and neural networks

River One learns by being shown. You give it a picture and the right label, it guesses, you measure how wrong the guess was, and every adjustable number in the system moves a small step in the direction that would have made it less wrong. Repeat a few billion times. That is the whole river, and the volume follows it from a 1943 drawing of a neuron to a model that predicts the next word of the internet.

The teaching spine is the first seven chapters of 3Blue1Brown’s neural-network series: what a network is, gradient descent, backpropagation in words and in calculus, then the Transformer, attention, and where facts live inside the layers. Welch Labs supplies the second voice for the hardware and history, the perceptron in particular. Talking Nets supplies the method for the people: let each one tell it in their own voice, verse by verse.

I added one track the videos skip. 3Blue1Brown teaches with squared error; every modern language model is trained on cross-entropy, so it gets its own song.

I.0

Downhill

Volume explainer · UK rap

In the roomThe full River One idea in four minutes: layers of weighted sums, a score for wrongness, a step downhill, blame sent backwards, then scale.

I.1

A Logical Calculus

Song · Chamber folk

In the roomThe first artificial neuron: inputs add up, and the unit fires if the total crosses a threshold. Wire enough of them together and you can compute any logical statement.

I.2

Summer at Dartmouth

Song · Pop-punk

In the roomWhere the field got its name and its first split: most of the room bet on rules and symbols, and a minority bet on learning machines.

I.3

Walk, Talk, See, Write

Song · Brass soul

In the roomThe perceptron and its learning rule: when it gets one wrong, nudge the weights toward the right answer. It can draw one straight line through the data, and some problems need two.

I.4

Thirteen Thousand Knobs

Song · Electro

In the roomWhat a network is and how gradient descent trains it, using 3Blue1Brown’s digit reader: a 28 × 28 image in, ten guesses out, about thirteen thousand weights and biases in between.

I.5

Surprise

Song · Alt-R&B

In the roomCross-entropy loss: the model spreads its belief across every possible answer, and the penalty is the negative log of the probability it gave the right one. Being confidently wrong costs far more than being unsure.

I.6

Blame Flows Backwards

Song · Boom-bap

In the roomBackpropagation: the chain rule run in reverse through the network, handing every weight its exact share of the error, so a network with hidden layers can be trained.

I.7

Tahoe, 2012

Song · Soca-calypso

In the roomWhy deep learning won when it did: a big labelled dataset, graphics chips doing the arithmetic, and a network deep enough to use both. Then the money arrives.

I.8

Query, Key, Value

Song · Synth-pop

In the roomSelf-attention: every word asks a question (query), advertises what it holds (key) and carries content to share (value). Matching questions to adverts decides how much of each word’s value flows into every other, all at once.

I.9

Next Word

Song · volume closer · Art pop

In the roomPretraining: predict the next word of a vast amount of text, measured by cross-entropy, at enormous scale. Facts end up stored in the layers. The result knows a great deal and has no manners, which is the problem Volume Three solves.

Volume II

The Reward

River Two · reinforcement learning

River Two learns by being told the score. Nobody gives it the right answer. It acts, the world responds, now and then a reward arrives, and it has to work out which of its many earlier moves earned it. That is the credit assignment problem, and the volume follows the answers to it from cats in boxes to a cell in a monkey’s brain that turns out to be running the same equation as a computer program.

The teaching spine is David Silver’s 2015 UCL lectures, with Sutton and Barto’s textbook behind them. Silver builds the subject in a strict order: what a decision problem is, Bellman, then learning from experience, then the actor and the critic. The songs keep that order so each one stands on the last.

The early history is tangled with River One, and that tangle is useful. Shannon and Minsky appear in both volumes because the 1950s machines learned from reward and were built from artificial neurons. The track on that decade shows the rivers before they separate.

II.0

Try, Get Told

Volume explainer · Amapiano

In the roomThe whole of River Two: an agent, a world, actions, rewards. A policy says what to do; a value says how good things look from here. Bellman links today’s value to tomorrow’s, and learning means correcting your guesses as the surprises come in.

II.1

Chickens in the Cellar

Song · Sea shanty

In the roomThe law of effect: an action followed by satisfaction gets stamped in, and one followed by discomfort gets stamped out. Learning happens through trial, error and reward rather than insight.

II.2

Theseus

Song · Indie-electronic

In the roomThe 1950s, where the two rivers share a source: machines built from artificial neurons that learned from reward, and a mouse that learned a maze.

II.3

The Principle of Optimality

Song · Bluegrass

In the roomThe Bellman equation: the value of where you are equals the reward you get now plus the discounted value of where you land. Any piece of a best route is itself a best route, so a huge planning problem becomes a one-step consistency check repeated everywhere.

II.4

A Guess From a Guess

Song · Americana duet

In the roomTemporal-difference learning, the bootstrapping at the heart of RL: update today’s prediction toward tomorrow’s prediction, without waiting for the final result.

II.5

Juice

Song · Melodic house

In the roomThe discovery that dopamine neurons signal prediction error: they fire for an unexpected reward, shift their firing to the cue that predicts it, and dip when an expected reward fails to arrive. That is the TD error from II.4, found in a brain.

II.6

Actor and Critic

Song · Liquid drum and bass

In the roomActor-critic: one part chooses actions, another predicts how well things are going and reports the surprise. The actor learns from the critic’s surprise rather than raw reward, and the critic learns from what happens next. Neither starts competent, and they improve each other.

II.7

Q

Song · Northern soul

In the roomQ-learning: learn the worth of each move in each place, and nudge it towards the reward plus the best move from where you land. It learns the best way even while exploring.

II.8

Make It More Likely

Song · UK garage

In the roomPolicy gradients: nudge the odds of everything you did by how much better or worse than usual it went. The engine under RLHF, PPO and GRPO.

II.9

The Restaurant Problem

Interlude · Bossa nova

In the roomExplore versus exploit: keep going to your favourite restaurant or try the new one. A learner that only exploits never finds better options.

Volume III

The Confluence

Deep reinforcement learning, then post-training and the Chinese labs

The rivers meet when a neural network becomes the function inside a reinforcement learner. River One supplies the network that can read pixels or text; River Two supplies the way to learn from a score. DeepMind joins them on Atari in 2013 and on Go in 2016. Then the same pairing gets turned on language models, and the subject becomes post-training: teaching a model that knows everything how to behave.

The teaching spine is Nathan Lambert’s RLHF book, which covers far more than human feedback: preference data, reward models, the leash that keeps a tuned model close to where it started, verifiable rewards, and the reasoning models. From there the volume follows the work that has moved fastest since 2025, much of it published by Chinese labs, and ends with the race.

Three tracks here are about economics and people: the doing-more-with-less engineering, the labs themselves, and the race between them and the US frontier labs. The figures in those tracks are current to mid-2026 and are the ones to refresh before you play them to anyone.

III.0

Where the Rivers Meet

Volume explainer · Gospel

In the roomDeep RL and post-training in one go: networks as the policy and value; pretraining gives knowledge, post-training gives behaviour; the post-training ladder of supervised fine-tuning, preference learning, verifiable rewards and distillation; then efficiency.

III.1

Breakout

Song · Indie dance

In the roomDQN: a convolutional network learns action values from pixels and the score alone, across dozens of Atari games, once two stabilisers are added.

III.2

Move 37

Song · K-pop

In the roomAlphaGo: a policy network proposes moves, a value network judges positions, and tree search plays out the most promising lines. Then AlphaGo Zero drops the human games and learns by playing itself.

III.3

The Leash

Song · Modern soul

In the roomRLHF: humans compare pairs of answers, a reward model learns to predict their preference, and RL pushes the language model toward higher reward while a KL penalty, the leash, keeps it close to the original so it doesn’t game the reward model.

III.4

Show Your Work

Song · Math rock

In the roomVerifiable rewards and reasoning: when the answer can be checked by a program, the checker becomes the reward. Models trained this way learn to think longer before answering. Preference learning also gets simpler with DPO, which skips the separate reward model.

III.5

Aha

Song · Mandopop R&B

In the roomGRPO and DeepSeek-R1: sample a group of answers to the same question, score each one, and reward answers for beating the group average. The group replaces the critic, which saves memory and compute.

III.6

Many Teachers

Song · Afrobeat

In the roomDistillation from first principles to MOPD. A teacher’s full spread of probabilities carries more than its top answer. Off-policy distillation copies the teacher’s finished answers. On-policy distillation lets the student write and has the teacher grade every word. Multi-teacher on-policy distillation uses a separate specialist teacher per skill, each trained by RL, so one student inherits all of them.

III.7

Shorter Numbers

Song · Garage rock

In the roomHow the Chinese labs do more with less. Mixture of experts wakes only a few specialists per word. Compressed and sparse attention stops the model holding every earlier word in memory at full size. Lower-precision numbers halve and halve again the memory and arithmetic. Multi-token prediction drafts several words at once. The arrangement is the argument: two instruments.

III.8

The Labs

Posse cut · Posse cut

In the roomWho the Chinese labs are, how they’re funded, and what their culture shows: open weights, detailed papers, young engineering-led teams, and constraint as a design principle.

III.9

More With Less

Song · Liquid drum and bass

In the roomThe race: US frontier labs spending hundreds of billions on compute, Chinese labs releasing open models months behind at a fraction of the price, and who captures the value along the chain of chips, memory, power and applications.

Finale

The Sea

The rivers meet at Manaus the way the real ones do: side by side for miles, black water beside gold, before they blend. The finale ends at Devprayag, where two rivers meet and only then take the name everyone knows.

The Sea

Finale · the wrap-up · Gospel

In the roomTies all three volumes to one idea: every system in the trilogy learns by correcting an error. River One’s error is the loss; River Two’s is the TD error; post-training mixes them. Then it looks forward to models that keep learning after release.

Jürgen’s Footnotes

Hidden track · Weimar cabaret

In the roomThe running joke paid off with affection. Jürgen Schmidhuber reads his priority claims; the video scores each one. Some are right, and the field owes a real debt to his lab for LSTM.