Chain of Thought: When Thinking Out Loud Helps

Chain-of-thought prompting tells a language model to generate explicit intermediate reasoning steps before the final answer. Internally, this aligns inference with the training data’s step-by-step explanations, enables self-consistency over multiple sampled chains, and reduces the “globality” of a problem by breaking a hard direct mapping into a sequence of easier local predictions. The result is large accuracy gains on math, logic, and multi-step planning tasks, but for simple, intuitive tasks the extra tokens are a dead weight that can even introduce errors.

The surprise: forcing a model to think out loud can double its accuracy on hard problems, yet for insight-like problems that rely on holistic pattern recognition the same technique can lower performance by 70 percentage points. The same pattern shows up in human studies. Chain-of-thought is not a universal upgrade; it is a specific reconfiguration of the computational graph that helps only when the problem genuinely decomposes.

In the previous episode we added few-shot examples to our customer-support reply assistant, and it started handling more varied queries. But complex tickets still broke down: “A customer says their device failed three weeks out of warranty, but they bought it with a premium credit card that extends coverage. Should we approve a replacement?” That kind of reasoning needs not just pattern matching but a deliberate sequence of checks. This article picks up exactly there.

What is chain-of-thought prompting, and how does it differ from standard prompting?

Chain-of-thought prompting inserts worked examples into the context that show not just a final answer but a detailed, step-by-step reasoning trace. The model then generates its own reasoning text before answering. This is different from a “zero-shot” prompt that asks for a direct answer: the context shapes the probability distribution so that the most likely continuation is a reasoning chain, and the model conditions its final answer on that self-generated intermediate text Wei et al., 2022.

For our assistant, a standard prompt might be:

Customer email: "My laptop broke 3 weeks after warranty. I bought it with my credit card that extends coverage. What can you do?"
Respond helpfully.

A chain-of-thought version would instead show an exemplar like:

Customer email: "..."
Let's think step by step:
1. Check warranty status: standard warranty expired 3 weeks ago.
2. Check extended coverage: credit card extends by 1 year, so still covered.
3. Determine action: replacement is covered under extended warranty.
4. Draft reply: apologetic, confirm coverage, ask for card details.
Response: [drafted reply]

After one or two such exemplars, the model will produce a similar chain before answering. That reasoning text becomes part of the model’s own context for generating the final reply, just as if an assistant had jotted down their thought process before typing the email.

Why does step-by-step reasoning improve hard tasks?

Large language models are trained on trillions of tokens where high-quality answers to complex questions almost always come with explanations. When we ask for a bare answer, we create a small distribution shift: the model must jump from problem to solution without the explanatory scaffolding it saw during training. Chain-of-thought prompting removes that mismatch by making the inference-time context look like the training-time contexts that included reasoning Wei et al., 2022.

Mechanically, this matters because many reasoning problems have a high “globality degree”: the answer depends on many interacting parts of the input in a non-local way. Learning a direct mapping from all those parts to the answer is hard. By generating intermediate steps, each of which depends on only a small subset of the information, the model converts one high-globality problem into a sequence of low-globality problems, each of which is easier to predict accurately Nye et al., 2021. For our warranty ticket, checking the warranty date is a local operation on a small piece of text; checking the credit card policy is another; combining them into a decision is a third. The model composes them one at a time rather than trying to swallow everything in one gulp.

Self-consistency builds on this by sampling multiple independent reasoning chains (using stochastic decoding) and then selecting the answer that appears most often. This exploits the fact that the model distributes probability mass across different plausible chains; valid reasoning paths tend to arrive at the same answer, while errors differ, so majority voting filters out many mistakes Wang et al., 2023. In practice, this can add 10-20 percentage points of accuracy on benchmarks like GSM8K math problems, often enough to match or beat specially fine-tuned models.

How does chain-of-thought work inside a language model?

Every token the model generates is appended to the prompt and fed back into the transformer through self-attention on the next step. When a chain-of-thought is present, the model’s own reasoning tokens serve as additional conditioned context that guides later predictions. The final answer token attends to all the preceding reasoning steps, so the model effectively “reads its own notes” before answering.

This can be visualized as a data flow that externalizes latent structure. Without CoT, the model maps input directly to output:

With chain-of-thought, the flow becomes:

Each step’s output becomes input for the next, much like a human jotting down subproblems and then synthesizing. Self-consistency expands this into multiple parallel reasoning traces, then picks the most common conclusion.

The process works because the model’s pretraining includes vast amounts of human-written chains. The tokens “Let’s think step by step” trigger a high-probability region of the distribution that corresponds to structured explanation, not because the model has an internal “thinking” module but because the statistical pattern is strong. In effect, CoT prompting is a form of in-context meta-learning: the model infers from the exemplars that the task format is “reason then answer,” and it dynamically adapts its decoding to match that format.

What does cognitive science tell us about verbalizing thoughts?

Human brains show a strikingly parallel benefit from externalizing intermediate states. According to Baddeley’s working memory model, we have a phonological loop that maintains verbal information through rehearsal, and a visuo-spatial sketchpad for images Baddeley, 2012. When a person speaks their thoughts aloud, the loop is extended into the environment: the spoken words are heard and re-encoded, creating an external auditory buffer that supplements internal memory. This offloads cognitive load and frees executive resources for higher-level reasoning.

Speaking also forces a serialization of fuzzy mental representations into explicit symbolic form. This often reveals inconsistencies or gaps, the same principle behind rubber duck debugging, where explaining code line by line uncovers bugs that silent review misses [Hunt & Thomas, 1999 / pragmatic programmer concept]. In learning studies, students who self-explain while solving problems construct deeper, more generalizable knowledge than those who study passively. The act of explanation triggers new inferences, not just retrieval of old ones Chi et al., 1989.

These effects map cleanly onto LLM behavior. A model’s reasoning tokens, like spoken thoughts, become a persistent external trace that conditions subsequent predictions. The forced serialization into discrete tokens encourages the model to make commitments that expose inconsistencies, exactly what makes the final answer more accurate when the problem is decomposable. And when we sample multiple reasoning chains, we mimic the human strategy of “thinking from different angles” before settling on a conclusion.

When does thinking out loud hurt performance?

Not all problems benefit from verbalization. In a classic line of work on insight problem solving, researchers found that asking participants to speak their thoughts while solving insight puzzles (like the “nine-dot” problem) cut solution rates dramatically, from 57% in silent conditions to 13% when verbalizing [Ball & Stevens, 2005 / verbal overshadowing in insight tasks]. The explanation is that insight often depends on unconscious, non-verbalizable restructuring of the problem representation. Forcing speech shifts attention toward the verbalizable surface features and away from the subtle transformations needed for the “aha” moment.

The same principle applies to language models. Tasks that rely on holistic pattern recognition, stylistic matching, or massive parallel constraint satisfaction, the “feel” of a correct answer, do not benefit from an explicit reasoning chain. If you ask a model to “write a friendly, empathetic reply to this customer” and the ticket is straightforward, a chain-of-thought prompt like “First list the customer’s emotions, then choose a tone, then draft” will often produce stilted, overthought output. The extra tokens act as noise, pulling the model away from the direct stylistic trajectory it would naturally follow.

The underlying cause is that explicit reasoning forces the model to commit to a particular decomposition of the problem. If the decomposition is misaligned with the task’s actual structure, the chain becomes a constraint that limits performance rather than a scaffold Nye et al., 2021. For the support assistant, a simple “I’m sorry, your tracking shows the package is delayed” needs no chain. But a request that requires computing a prorated refund based on three overlapping policies absolutely does.

The engineering rule: use chain-of-thought when the task demands multiple interacting logical steps. Skip it when the answer is essentially a direct pattern completion.

Quick Reference: Chain-of-thought at a glance

Frequently Asked Questions

Q: How many chain-of-thought exemplars do I need in a prompt? Two or three well-chosen exemplars that cover the key reasoning patterns are usually enough. More can cause the model to fixate on irrelevant details rather than abstracting the general “reason then answer” format.

Q: Can I just add “Let’s think step by step” without exemplars? Yes, this so-called “zero-shot CoT” often works for moderate difficulty tasks. It’s less reliable than full exemplars for highly structured reasoning but can be a quick first attempt.

Q: Does self-consistency always improve results? No. If the model’s error causes all samples to converge on the same wrong answer, voting won’t help. It works best when errors are stochastic and diverse, which tends to be true for harder reasoning.

Q: Is chain-of-thought safe to use when the intermediate reasoning might contain sensitive information? Caution is warranted. The reasoning text may surface PII (personally identifiable information) or internal assumptions that aren’t in the final output, creating privacy risk. You may need to filter or omit intermediate traces in production logs.

Q: Can I use chain-of-thought with a small model? Small models (under ~10 billion parameters) often fail to produce meaningful reasoning chains because the probability distribution is not shaped by enough training data with step-by-step patterns. CoT is most effective with larger models.

Test yourself

Your team built a customer-service bot that answers tracking questions with a direct look-up and a friendly message. You want it to also handle “Was my package shipped with carbon-neutral delivery?” which requires checking both a carrier API (application programming interface) and a company policy database, then composing a plain-English answer. You’re considering a chain-of-thought prompt. Describe a single real-world scenario where adding CoT would hurt the bot’s performance instead of helping, and explain why it fails in that case.

Answer: Suppose a customer asks “Where is my package?” and the tracking number is valid. The direct answer is a simple lookup plus a pre-templated status message (e.g., “Your package is out for delivery today.”). Adding a CoT exemplar like “Let’s think step by step: first, check the tracking API; then check internal delivery zones; then compose message” forces the model to generate a verbose reasoning chain before the answer. This increases latency by several seconds and token costs by 2-3× for no gain, because the reply can be produced in one shot with high confidence. Worse, the forced explicit reasoning might cause the model to hallucinate details (like a phantom delivery zone check) that don’t exist in the API response, introducing factual errors into the final message. The problem is structurally simple: a single API call with a deterministic response mapping. Explicit decomposition adds noise rather than structure.

If you want this kind of breakdown every week, how real systems and techniques actually work under the hood, from prompt engineering to model internals, subscribe to Internals Decoded at internalsdecoded.com.

Sources

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Wei et al., 2022)
• Self-Consistency Improves Chain of Thought Reasoning in Language Models (Wang et al., 2023)
• Show Your Work: Scratchpads for Intermediate Computation with Language Models (Nye et al., 2021)
• Working Memory (Baddeley, 2012)
• Self-explanations: How students study and use examples in learning to solve problems (Chi et al., 1989)
• The Pragmatic Programmer (Hunt & Thomas, 1999), rubber duck debugging concept