Practice Makes It Smaller
Small steps downhill toward fewer mistakes. Gradient descent, plus a beautiful animation to watch.
A quick note on how we teach here. You won't find equations or numbers in this series, and that is on purpose. The goal is understanding, not arithmetic, so every idea is explained in plain words and simple pictures. When you want the math later, the rest of Internals Decoded is ready for you.
Welcome back. Last time we gave the machine a way to measure its own mistakes. After each guess, it works out how far that guess sits from the right answer, counting how sure it was, and that measure is called the loss. But owning a measuring stick does not make you a better cook. You have to use it to change something. Now we turn that measurement into actual improvement. How does the machine really get better at a task?
The answer is surprisingly steady. To improve, the machine changes itself a tiny bit at a time. Each change moves in the direction that makes its loss a little smaller. No grand overhaul. No sudden flash of understanding. Just a long, careful walk downhill toward doing better.
Finding the bottom of a valley in thick fog
Imagine you are standing somewhere in a wide valley, surrounded by thick fog. You cannot see the bottom. You cannot see the shape of the whole land. But you can feel it. Right where you stand, the ground slopes a certain way. If you place one foot carefully, you can tell which direction goes down. So you take a small step in that downhill direction. Then you stop, feel the slope again, and take another small step. Do this over and over, and eventually you reach a low point, even though you never saw a map. It may not be the very lowest spot in the whole land, but in practice it is usually low enough to work well.
This is how the machine learns. Picture every possible setting of its knobs as a different spot in this land, and its loss as the height of the ground at that spot. A large loss is high up on the slope. A small loss is down near the valley floor. The machine does not know the whole landscape. But after it tries a batch of examples, it can feel which way its knobs should move to make the loss smaller. It takes a small nudge in that direction. Then it repeats the process, again and again.
The power of small steps
Why not take one giant leap straight to the bottom? In the fog, a big leap might send you tumbling the wrong way, or overshoot the low point and leave you climbing the other side. Small steps are safer. Even if the slope is tricky, each small step inches the machine closer to doing a little better. It never has to be perfect in one go.
Steps that are too small have a cost too: the walk takes forever. So the people training the machine choose a setting that controls how big each step is, and they call it the learning rate. Set it too big and the machine keeps jumping past the low point. Set it too small and training crawls.
This is what training really looks like. The machine sees some examples, makes its guesses, and measures its loss. Then it uses that measurement to figure out which tiny twists to its knobs will shrink the loss. It makes those twists, and the next time it sees similar examples, it is a little less wrong. Then it does it all over again with more examples. A long, patient sequence. Nothing dramatic. But with enough steps, the mistakes can shrink from huge to small.
How it knows which way is downhill
This is the clever bit. You might think the machine would have to test every possible nudge, one at a time, for every single knob inside it, and check whether the loss went up or down. That would take forever. The machine has a smarter way. Its loss comes from a fixed recipe that compares its guess with the right answer, and that same recipe can also tell it the slope. For every knob, it can work out which way to turn it to make the loss shrink, and how strongly that knob matters right now. No test steps required.
Names for the walk
People call this downhill method gradient descent. The word "gradient" just means slope. And "descent" means walking down. So gradient descent is exactly the foggy valley walk we described: follow the slope downward, step by patient step. The part of the system that runs this walk is called the optimizer. You might smile when you hear that. It is a grand name for a very simple habit. Keep stepping in the direction that makes things a little better.
There is one more name you will hear right beside gradient descent. To find the slope for every knob at once, the machine passes its miss backward through itself, from the answer back toward the input, so that each knob finds out how much it helped cause the miss and which way it should turn. People call this backward pass backpropagation. You saw a glimpse of it in Learning From Examples, when the machine worked backward through its own settings. Backpropagation finds the slope. Gradient descent takes the step.
Watch the walk
Some ideas ask to be watched. The YouTube channel 3Blue1Brown animates this walk beautifully in a video called Gradient descent, how neural networks learn. You can see the little steps march downhill on a real landscape. Fair warning: the video shows some symbols and numbers, and it talks about neurons and layers, which you will meet in a few parts from now. Let the symbols wash over you and watch the ball roll downhill. That picture is the whole idea.
Where this is heading
Learning is not a single leap. It is a long walk of countless tiny adjustments, each one a step downhill toward a smaller loss. Backpropagation works out which way is down for every knob, and gradient descent takes the step. But what exactly are those knobs the machine keeps nudging? They have a name, and there are far more of them than you might guess.
Test yourself
A friend says training is wasteful: why not change all the knobs in one giant jump, straight to the answer? Using the foggy valley, explain why the machine takes small steps instead, and what goes wrong if the steps are too small.
Answer: In the fog, the machine only knows the slope right where it stands. It cannot see the whole land, so it cannot know where the bottom is. A giant jump could land somewhere worse, or sail right past the low point. So it takes a small step downhill, feels the slope again from its new spot, and steps again. Backpropagation tells it which way is downhill for every knob, and gradient descent takes the step. But steps that are too small make the walk painfully slow, so the people training the machine pick a step size, called the learning rate, that is big enough to make progress and small enough not to overshoot.
Next: The Knobs Inside. It opens the lid on the knobs the machine keeps nudging, called parameters, and shows how many of them the biggest machines have.
