How Likely, Really

A quick note on how we teach here. You won't find equations or numbers in this series, and that is on purpose. The goal is understanding, not arithmetic, so every idea is explained in plain words and simple pictures. When you want the math later, the rest of Internals Decoded is ready for you.

Welcome back. In the last piece, we watched a machine guess what word might come next in a sentence. It read all the words before, gave every word it knows a score, and picked a likely one. Today we look closely at that word "likely". A guess is never just one single answer. The machine always has a sense of how expected each possibility is.

When you hear someone say "it is likely to rain," you do not take it as a guarantee. You just feel that rain is more expected than sunshine. Likelihood is how expected something is. The less likely something is, the more it would surprise you. Things you half expected feel ordinary. Rare things feel surprising. That everyday feeling is what a machine leans on when it makes a careful guess.

A feeling you already have

Think about the weather. You look out the window. The sky is gray, the air feels damp, and a cool breeze is blowing. You might say "rain looks likely." You are not making a promise. You are not doing any arithmetic. You are remembering all the other days that felt like this one. On most of those days, rain showed up. So your mind expects rain, even though you know it might hold off.

Your mind builds this sense without ever counting. It soaks up experience and lets the past color what you expect next.

How a machine feels about words

Now let us return to the machine that is guessing the next word. It does not pull a single word out of thin air. It holds a whole cloud of possibilities, and each one carries a different weight of expectation.

Imagine you are reading a story and you come to the sentence "She poured the tea into the ..." Your mind immediately leans toward a small set of words. "Cup" feels ordinary. "Mug" feels ordinary too. "Saucer" feels a little less expected, but still possible. "Garden" feels surprising. "Elephant" feels completely out of place. You did not do any math. You just felt that some words fit the scene and others would make you blink.

A machine does something very similar. After reading the words that came before, it has a sense of how expected each option is. Picture its leanings as a row of bars, one for each word: tall bars for "cup" and "mug", a shorter one for "saucer", and bars for "garden" and "elephant" so short you can barely see them. Even the silliest option keeps a tiny leaning. And the machine works out these leanings fresh for every new word, from everything written so far.

Giving the feeling a name

That sense of how expected something is has a name. People call it probability. Do not let the word scare you. Probability is a way of putting a size on that leaning. People usually write it as a percentage, but you do not need the figures here. What matters is that every option gets a share, the likely ones get big shares, and all the shares together make up the whole.

When the machine guesses the next word, it holds a probability for each possible word. In our tea sentence, "cup" and "mug" hold big shares, "saucer" a smaller one, and "elephant" a tiny sliver. The idea itself is something you have understood since you were a child. You just called it "likely" or "unlikely". The machine calls it probability.

Drawing the next word like a raffle

So which word does the machine write? You might expect it to take the biggest share every time. Usually it does something more interesting. It draws the next word like a raffle. Likely words hold many tickets, less likely words hold a few, and the silliest words hold so few that they practically never win. Some systems go a step further and take the silliest words out of the raffle altogether. Then one ticket is drawn. Most of the time a likely word wins, but now and then a less likely one does. This drawing is called sampling.

Why not just take the top word every time? Because writing built only from the most expected words tends to come out flat, and it can even get stuck repeating itself. A little chance keeps the writing natural.

Sampling is also the main reason the same question can get a different answer each time you ask. The machine's leanings are the same each time, but the raffle is drawn fresh, and one different word early on can send the rest of the reply down a different path.

A dial for the raffle

There is one more idea that sits on top of the raffle. People can change how the tickets are shared out. Turn a dial one way and the favorite words get nearly all the tickets, so the machine plays it safe. Turn it the other way and the tickets spread more evenly, so long shots win more often. This dial is called temperature. In most chat apps you never see it, because the app sets it for you, and some of the newest models keep it fixed. The piece Careful or Adventurous, later in this series, gives temperature a full picture of its own.

Where this is heading

Likelihood is how expected something is, and the less likely something is, the more it would surprise you. A machine holds a probability, a share, for every word that could come next, and it draws its next word like a raffle, which we call sampling. Temperature is a dial that changes how the tickets are shared out.

But we have left one enormous question unanswered. We have said the machine "learns" patterns and "guesses well", but we have not said how it ever got good at this. How does a pile of switches come to feel that "cup" fits better than "elephant"? That is the real beginning of machine learning, and it starts with a choice between two ways of making a machine smart.

Test yourself

You ask a chatbot the same question twice, word for word, and get two answers that start differently. A friend says the chatbot must have changed its mind. Using the raffle picture, explain what really happened.

Answer: Nothing changed its mind. Both times the machine read the same question and came up with the same leanings: the same share of tickets for every word that could come first. But each word is drawn like a raffle ticket, and the draw is made fresh every time. Usually a likely word wins, but sometimes a different likely word wins instead. Once the first words differ, every word after them is guessed from a different beginning, so the two replies drift apart. This drawing is called sampling, and it is the main reason the same question can get a different answer.

Next: Two Ways to Make a Machine Smart. It compares writing every rule by hand with letting a machine learn from examples, which is what machine learning means.