Careful or Adventurous

A quick note on how we teach here. You won't find equations or numbers in this series, and that is on purpose. The goal is understanding, not arithmetic, so every idea is explained in plain words and simple pictures. When you want the math later, the rest of Internals Decoded is ready for you.

Welcome back. Last time we took a second look at the map of meaning, where every token has a place and those places are knobs set by training. Now we have one more idea to explore before all the pieces of the groundwork are in place. It is about a choice the model makes every time it writes the next token, and how we can nudge that choice.

The heart of it is this. A model can be careful and pick the most expected next piece, or it can be a little adventurous and sometimes pick a less expected one. That single decision shapes whether its writing feels steady and plain or surprising and creative.

A raffle with a dial

You met the raffle in How Likely, Really. After reading everything written so far, the model has a leaning for every token it knows, a row of bars, tall for the likely ones and tiny for the unlikely ones. It does not simply take the tallest bar. It draws the next token like a raffle: likely tokens hold many tickets, unlikely ones hold a few. Most of the time a likely token wins, but now and then a long shot does.

Now picture a dial on that raffle. The dial does not change the leanings. It changes how the tickets are shared out. Turn it toward careful, and the favorites get nearly all the tickets. Turn it toward adventurous, and the tickets spread out more evenly, so the long shots win more often. That earlier piece only sketched this dial. Here it is in action.

The tea sentence, with the dial turned

Take the sentence from that same piece: "She poured the tea into the ..." The model's bars are tall for "cup" and "mug", shorter for "saucer", and tiny slivers for "garden" and "elephant". Now imagine asking it to finish the sentence again and again, with the dial in different places.

Turned down, toward careful, the favorites hold nearly every ticket. You get "cup" nearly every time, with the odd "mug". The writing is reliable, but ask for a whole story this way and it can come out flat, or even start repeating itself.

At its standard setting, the tickets simply follow the model's leanings. Mostly "cup" and "mug", now and then "saucer", and almost never anything strange.

Turned up, toward adventurous, the tickets spread out. "Saucer" turns up often, and once in a while you get "She poured the tea into the garden." That might be the start of a delightful story, or just an odd one. Turn the dial right up and even "elephant" wins now and then, and the sentences that follow start to wander off track.

The model does not change

The important thing is that the model itself does not change when you turn the dial. Same knobs, set the same way. Its leanings, the bars, are exactly what they were. The only thing that changes is how willing it is to step off the beaten path, because only the sharing of the raffle tickets has changed.

When to turn it which way

People keep the dial low when the most expected answer is usually the right one: facts, summaries, code, filling in a form. They turn it up for brainstorming, stories and poems, where surprise is the point. Google's guide to setting parameter values for its Gemini models says much the same: lower temperatures suit prompts that need a less open-ended or creative response, and higher ones can lead to more diverse or creative results. Turned up too far, a model starts saying things that are odd or simply untrue, because it keeps picking tokens that its own leanings said were unlikely.

You may never see this dial yourself. In most chat apps, the app sets it for you. Tools built for developers often let you turn it. And some of the newest models keep it fixed. At the time of writing, Anthropic's guide to its Messages API says its latest Claude models do not accept a different temperature at all, and suggests guiding them with the words of your request instead.

What people call this dial

When people who work with language models talk about this idea, they call the dial temperature. You first met the word in How Likely, Really, and now it has a clear picture to live in. A low temperature means the model is careful. It hugs the most likely tokens tightly. A high temperature means the model is more adventurous. It spreads its chances more evenly across many possible tokens, even the long shots.

So when you hear someone say they turned the temperature down, they mean they asked the model to play it safe. When they turned it up, they asked the model to surprise them.

Where this is heading

The temperature dial is the last new idea in this series. One final piece remains. It walks back over everything you have learned, gathers the words that are yours now, and points you to the next series, AI, From Zero, with an honest word about what is new there.

Test yourself

You want an AI to fill in a tax form accurately, and later to brainstorm silly names for a cat. Which way would you turn the temperature dial for each, and why?

Answer: For the tax form, turn it down. There, the most expected answer is usually the right one, and a long shot could put something wrong in a box that matters. A low temperature gives the favorite tokens nearly all the raffle tickets, so the model sticks to its most likely choices. For the cat names, turn it up. Surprise is the whole point, and the long shots are where the funny names live. Either way, the model itself is the same: same knobs, set the same way, same leanings. Only the sharing of the tickets changes. And if your app keeps the dial out of reach, you can still ask in plain words for careful work or for playful ideas.

Next: You're Ready. What Comes Next. It looks back over the whole series, gathers the words that are yours now, and points you to where to go next.

Sources

1. Google's guide to setting parameter values for its Gemini models
2. Anthropic's guide to its Messages API