Chain of Thought: When Thinking Out Loud Helps

Chain-of-thought prompting asks a language model to write out intermediate reasoning steps before its final answer. Each step it writes becomes text it reads before writing the next, so one hard problem turns into a series of smaller ones. On math, logic and planning with several steps, the gains can be large. On simple or intuitive tasks, the extra tokens cost time and money and can even make the answer worse.

There is a newer half to this story. Today the main way to get step-by-step reasoning is often not a prompt at all: reasoning models think internally before they answer, and you turn that thinking up or down with a setting. This article covers both, because the old technique explains the new one, and because it still matters for models that do not think on their own.

In the paper that named the technique, Google's 540 billion parameter PaLM model solved 17.9% of GSM8K (a set of grade school math word problems) with standard prompting, and 56.9% when the prompt included eight worked examples with their reasoning (Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, Wei et al.). The opposite can happen too. On an artificial grammar task, a kind of pattern learning where people also do worse when they deliberate, GPT-4o fell from 87.5% correct when answering directly to 64.4% with chain of thought (Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse, Liu et al. 2024). Chain of thought is not a universal upgrade. It helps when the problem breaks into steps.

In Few-Shot: Examples Beat Instructions we gave our customer support reply assistant a few examples, and it started handling more varied emails. But complex tickets still broke down: "A customer says their device failed three weeks out of warranty, but they bought it with a premium credit card that extends coverage. Should we approve a replacement?" That needs not just pattern matching but a deliberate sequence of checks. This article picks up there.

What is chain-of-thought prompting, and how does it differ from standard prompting?

Chain-of-thought prompting puts worked examples in the context that show not just a final answer but the reasoning that leads to it. The model then writes its own reasoning before answering, and conditions its final answer on that text (Wei et al.). A standard prompt, by contrast, asks for the answer directly.

For our assistant, a standard prompt might be:

Customer email: "My laptop broke 3 weeks after warranty. I bought it with my credit card that extends coverage. What can you do?"
Respond helpfully.

A chain-of-thought version would instead show an exemplar like:

Customer email: "..."
Let's think step by step:
1. Check warranty status: standard warranty expired 3 weeks ago.
2. Check extended coverage: credit card extends by 1 year, so still covered.
3. Determine action: replacement is covered under extended warranty.
4. Draft reply: apologetic, confirm coverage, ask for card details.
Response: [drafted reply]

After one or two such exemplars, the model produces a similar chain before answering. That reasoning text becomes part of the model's own context for the final reply, just as if an agent had jotted down their checks before typing the email.

You do not always need exemplars. In 2022, Kojima et al. found that simply adding "Let's think step by step" before the answer raised the accuracy of OpenAI's text-davinci-002 from 17.7% to 78.7% on the MultiArith math set and from 10.4% to 40.7% on GSM8K (Large Language Models are Zero-Shot Reasoners). This is called zero-shot chain of thought.

Reasoning models changed the rules

Today's frontier models can do this reasoning on their own. OpenAI's reasoning models, Anthropic's Claude models with thinking, and Google's Gemini models all generate reasoning tokens internally before they write the reply. You do not prompt for this. You switch it on, or tune how much of it you want, with a setting:

• OpenAI: the reasoning.effort parameter, with values from none and minimal up to high, xhigh and max. Not every model supports every value, and the default depends on the model (medium for GPT-5.5) (OpenAI's reasoning guide).
• Anthropic: adaptive thinking, where Claude decides when and how much to think, tuned with the effort parameter (low, medium, high, xhigh or max). On the newest models, such as Claude Opus 5.5, thinking is always on and effort is the only control (Anthropic's guide to thinking).
• Google: Gemini models think by default, and a thinking level (minimal, low, medium or high, depending on the model) sets how much (Google's Gemini thinking guide).

The advice changed with them. OpenAI's guide for its reasoning models says to avoid chain-of-thought prompts, since "prompting them to 'think step by step' or 'explain your reasoning' is unnecessary" (OpenAI's reasoning best practices). Anthropic says to "prefer general instructions over prescriptive steps", because "a prompt like 'think thoroughly' often produces better reasoning than a hand-written step-by-step plan", and calls manual chain of thought a fallback for when thinking is off. On its newest models, it warns, a prompt that asks the model to write out its reasoning, for example in <thinking> tags, "may be declined" (Anthropic's prompting best practices). Such a request can come back as a refusal, while asking for a short explanation of the answer or the evidence behind it is fine (Anthropic's guide to refusals).

So with a reasoning model, give it the goal, the facts and what a good answer looks like, set the effort, and let it plan. For our warranty ticket, that means giving it the warranty and card policies and asking for a decision with a short explanation that cites them, rather than scripting the four checks. If you want to see how it got there, ask for a summary of its reasoning: the providers do not hand out the raw reasoning tokens, but OpenAI, Anthropic and Google all return summaries when you ask (in Anthropic's case through the thinking display setting).

Thinking is not free. Reasoning tokens are billed as output tokens by all three providers, they take up room in the context window, and they add waiting time. OpenAI suggests reserving at least 25,000 tokens for reasoning and output when you start experimenting. The old rule, skip step-by-step reasoning on easy tasks, now often means a low effort setting.

Manual chain of thought still has three jobs. It helps models that do not think on their own, such as many smaller open models and fast modes with thinking off. It makes reasoning part of the visible answer, on models that allow it, when someone needs to review it. And it is the basis of self-consistency, below.

Why does step-by-step reasoning improve hard tasks?

One explanation is that models learned from text in which answers to hard questions usually come with explanations, so a prompt that invites the explanation first looks more like what the model saw in training. That is a hypothesis, not a settled finding.

A more concrete reason is that writing steps lets the model solve a problem a piece at a time. Abbe et al. describe some problems as having high "globality": the answer depends on many parts of the input at once, and such problems are hard for a transformer to learn as one direct mapping (How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpad). Intermediate steps, the "scratchpad" that Nye et al. showed helps models do multi step computation such as long addition (Show Your Work: Scratchpads for Intermediate Computation with Language Models), turn the problem into a series of smaller ones, each depending on a few facts. For our warranty ticket, checking the warranty date is one small step; checking the credit card policy is another; combining them into a decision is a third. The model takes them one at a time rather than trying to swallow everything in one gulp.

Self-consistency builds on this. You sample several independent reasoning chains with some randomness, then keep the answer that appears most often. Valid chains tend to reach the same answer while mistakes scatter, so the vote filters out many errors. On GSM8K it added 17.9 percentage points for PaLM 540B, from 56.5% to 74.4% (Self-Consistency Improves Chain of Thought Reasoning in Language Models, Wang et al.). Most APIs already sample with some randomness by default. Anthropic's newest models do not let you change the temperature at all (only the default of 1.0 is accepted), so you simply call several times (Anthropic's Messages API reference).

How does chain-of-thought work inside a language model?

Every token the model generates is appended to the sequence and read on the next step through attention. When a chain of thought is present, the model's own reasoning tokens become extra context that guides later predictions. The final answer can draw on every step before it, so the model in effect reads its own notes before answering.

Without chain of thought, the model maps the input straight to the output:

With chain of thought, the flow becomes:

Each step's output becomes input for the next. There is no separate thinking module in a standard model; the reasoning is ordinary text that the model then conditions on. Programmers know a human version of this as rubber duck debugging, popularized by The Pragmatic Programmer: explaining code line by line, even to a rubber duck, often reveals the bug. Writing the steps down gives the model the same chance to catch a gap before it commits to an answer.

When does thinking out loud hurt performance?

Not every task benefits. A review of more than 100 papers, plus new tests of 14 models on 20 datasets, found that chain of thought helps mainly on math and logic, with much smaller gains on other tasks. On the MMLU knowledge test, answering directly was almost as accurate unless the question or answer involved an equals sign (To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning, Sprague et al.).

It can also do harm. Liu et al. picked tasks from psychology where deliberating makes people worse, such as learning hidden patterns or recognizing a face after describing it in words, and found that chain of thought cut model accuracy on several of them. On the artificial grammar task, OpenAI's o1-preview, which reasons before answering, scored 36.3 percentage points below GPT-4o answering directly on the same problems. The authors note that models and people do not think in the same way, but the tasks where thinking hurts people turned out to be a good place to look.

The same applies to everyday tasks. If you ask a model to "write a friendly, empathetic reply to this customer" and the ticket is straightforward, a chain-of-thought prompt like "First list the customer's emotions, then choose a tone, then draft" can produce stilted, overthought output. For the support assistant, a simple "I'm sorry, your tracking shows the package is delayed" needs no chain. But a request that requires computing a prorated refund under three overlapping policies does.

The engineering rule: use chain of thought, or a higher effort setting on a reasoning model, when the task has several interacting logical steps. Skip it, or use a low effort setting, when the answer is a direct lookup or a matter of tone.

Quick Reference: Chain-of-thought at a glance

Frequently Asked Questions

Q: Should I write "think step by step" for a reasoning model? No. OpenAI says it is unnecessary for its reasoning models, and Anthropic's newest models may decline a prompt that asks them to write out their reasoning. Raise the effort setting instead, and ask for a short explanation of the answer if you need one.

Q: How many chain-of-thought exemplars do I need in a prompt? For a model without built-in thinking, two or three well chosen exemplars that cover the key reasoning patterns are usually enough. More can make the model copy irrelevant details instead of the general "reason, then answer" format.

Q: Can I just add "Let's think step by step" without exemplars? Yes. This zero-shot chain of thought comes from Kojima et al. (2022) and often works on problems of moderate difficulty. It is less reliable than full exemplars for highly structured reasoning, but it is a quick first attempt on a model that does not think on its own.

Q: Does self-consistency always improve results? No. If the model's mistake is systematic, every sample can reach the same wrong answer, and voting will not help. It works best when errors are random and varied, which tends to be true for harder reasoning. It also multiplies your cost by the number of samples.

Q: Is chain-of-thought safe to use when the intermediate reasoning might contain sensitive information? Take care. Visible reasoning text, or reasoning summaries, may surface personal information or internal assumptions that are not in the final answer, which creates privacy risk. Filter or leave out intermediate traces in production logs.

Q: Can I use chain-of-thought with a small model? In 2022, it helped only very large models: Wei et al. saw gains at around 100 billion parameters, while smaller models wrote fluent but illogical chains. Today many small models are trained to reason; DeepSeek, for example, released reasoning models from 1.5 to 70 billion parameters distilled from its R1 model (DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning). Test on your model.

Test yourself

Your team built a customer service bot that answers "Where is my package?" with a tracking lookup and a friendly message. Describe a request to this bot where adding chain of thought would hurt rather than help, explain why, and say what setting you would change instead if the bot runs on a reasoning model.

Answer: Take a plain "Where is my package?" with a valid tracking number. The answer is one lookup plus a short status message, such as "Your package is out for delivery today." Adding a chain-of-thought exemplar like "First check the tracking API, then check delivery zones, then compose the message" makes the model write a reasoning chain before every reply. That adds waiting time and several times more output tokens for no gain, because the reply is one step. Worse, a scripted chain can invite the model to invent a step, such as a delivery zone check that is not in the API response, and put a made-up detail in the reply. The task is structurally simple, so explicit steps add noise, not structure. On a reasoning model, do not add a step-by-step prompt at all; set the effort low (or turn thinking off where the model allows it) for these lookups, and keep higher effort for tickets that really involve several policies.

Next: Structured Outputs: JSON You Can Actually Parse. It shows how to make a model's answer match a schema your code can parse, and what can still go wrong.

If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.

Sources

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Wei et al. 2022)
• Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse (Liu et al. 2024)
• Large Language Models are Zero-Shot Reasoners (Kojima et al. 2022)
• Reasoning models (OpenAI, read October 2026)
• Reasoning best practices (OpenAI, read October 2026)
• Thinking (Anthropic, read October 2026)
• Prompting best practices (Anthropic, read October 2026)
• Refusals and fallback (Anthropic, read October 2026)
• Gemini thinking (Google, read October 2026)
• How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpad (Abbe et al. 2024)
• Show Your Work: Scratchpads for Intermediate Computation with Language Models (Nye et al. 2021)
• Self-Consistency Improves Chain of Thought Reasoning in Language Models (Wang et al. 2022)
• Create a Message (Anthropic API reference, read October 2026)
• The Pragmatic Programmer, 20th Anniversary Edition (Thomas and Hunt)
• To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning (Sprague et al. 2024)
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (DeepSeek-AI 2025)