Why Your Prompts Fail (and the Anatomy That Works)

Most prompts fail not because you forgot a magic phrase but because the model had to guess. A short, vague prompt leaves it to invent the persona, the length, the policy and the format. A long, unstructured one buries the rule that mattered in the place where models use information least reliably. The fix is a structured skeleton (role, task, constraints and format) with the critical rules placed where the model reads them best, plus the habit of checking that a change actually worked.

This series is about techniques that measurably change what a model writes, so we will keep asking one question: how would you know? Throughout, we follow one example, a customer support assistant for a fictional company, Acme Corp, that drafts replies to customer emails.

What does the model actually see?

You do not need the internals to write good prompts, but four facts explain most failures.

First, your text becomes tokens, chunks of words, as you saw in Tokens: The Currency of AI. A small edit can change how the text splits.

Second, everything becomes one sequence. The system message, the conversation history, any documents and your latest message are joined into a single run of tokens, and the model reads all of it before it writes.

Third, the model does not use every position equally well. Attention, which you met in Attention, Gently: How Models Decide What Matters, lets each token draw on the ones before it, but in long inputs the details at the start and the end tend to be used more reliably than the details in the middle.

Fourth, the model writes one token at a time, each chosen in light of everything before it, so an early wrong turn shapes the rest of the reply.

That is why placement and structure are not cosmetic. The rest of this article shows what goes wrong, starting with where you put things.

Why does placement of instructions matter so much?

In long inputs, models tend to use information at the start and the end more reliably than information in the middle. The clearest measurement comes from a 2023 study, Lost in the Middle: How Language Models Use Long Contexts (Liu et al.). The researchers gave models a question and 20 retrieved documents, only one of which held the answer, and moved that document from first place to last. GPT-3.5-Turbo answered 75.8% of the questions correctly when the answer was in the first document, 53.8% when it was in the tenth, and 63.2% when it was in the last. In the middle, it did worse than the 56.1% it scored with no documents at all.

Two cautions keep this honest. The study measured question answering over documents, not instruction following, and it tested 2023 models; how deep the dip is varies a lot from model to model and task to task. And the cause is not settled. The authors found a similar, smaller dip in a model that encodes position differently (MPT, which uses a scheme called ALiBi), so it is probably a mix of how position is encoded, the fact that each token can only look back at earlier ones, and patterns in the training data. The practical rule holds anyway: do not bury a rule that matters in the middle, and test your own model at the lengths you actually use.

Now picture our support assistant with a long system prompt that sets the brand voice, the reply guidelines, the escalation rules, and a rule that every reply must include the refund policy link. If that rule sits in the middle of a 2,000 token system prompt, with the customer's email at the end, the assistant may answer without the link. Two changes usually help: state the rule near the start, and repeat it in one short sentence right before the model writes. OpenAI's prompting guide for GPT-4.1 recommends this for long prompts: place your instructions "at both the beginning and end of the provided context", which it found worked better than either one alone (GPT-4.1 Prompting Guide). Anthropic gives similar advice for long documents: put them at the top and the question at the end, which it says can improve response quality "by up to 30 percent in tests" (Anthropic's prompting best practices).

Whether the move helped is something you can check rather than hope. Run the same 20 test emails before and after the change, and count the replies that include the link. Part 6, Evaluating Prompts: Beyond 'Looks Good to Me', builds a small harness for exactly this kind of count.

Instruction drift is a related effect in long conversations. A rule you gave in an early message slides toward the middle as the chat grows, and even the system prompt, which stays at the very start, competes with more and more text. Repeating the key rules near the end of the context is the usual remedy; part 5 comes back to it for agents.

What are the most common failure modes in prompts?

A 2025 survey sorts prompt failures into six groups: specification and intent, input and content, structure and formatting, context and memory, performance and efficiency, and maintainability and engineering (A Taxonomy of Prompt Defects in LLM Systems, Tian et al.). Naming the group turns "the model acted weird" into a defect you can find and fix. Here is how each one shows up in our assistant.

A specification defect: the prompt says "reply helpfully" but never says what helpful means in that channel. Asked to answer through a text message channel that caps replies at 160 characters, the assistant writes a warm 500 word reply. The intent was right; the constraint was missing.

An input defect: the documents in the prompt conflict. A retrieval step (retrieval augmented generation, or RAG) feeds in an outdated refund policy alongside the question, and the assistant cites the old policy with confidence, because the outdated text was right there in its prompt and models trust the context they are given.

A structure defect: no clear separators. With instructions, examples and the customer's email run together in one block, the assistant answers an old message from the history instead of the new one. Marking each part, for example with XML style tags such as <customer_email>, lets the model tell them apart.

A context defect: the app trims the conversation to fit and drops the early rule "always verify account identity". The assistant carries on without it, and nobody notices until a security review.

A performance defect: the prompt loads 10,000 tokens of examples on every call. It works, but the latency and cost make it hard to scale.

A maintainability defect: the prompt is a string buried in backend code with no version and no tests. An edit that fixes one case breaks ten others, and customers find the regression first.

The group tells you where to look. Specification and structure defects are fixed in the prompt text. Context and performance defects are fixed in the code that assembles the prompt. Maintainability defects are fixed in how you version and test it, which is what part 6 is about.

How do tokenization and truncation cause hidden failures?

Tokenization is the first quiet source of surprises. The same words can split differently with a capital letter, a leading space or a punctuation mark. With the o200k_base encoding that OpenAI's open source tiktoken library uses for GPT-4o and later models, "customer support" is two tokens ("customer" and " support"), while "CUSTOMER SUPPORT" is three ("CUSTOM", "ER" and " SUPPORT"). Text you put in capitals for emphasis is a different input to the model, and token counts, which set both your cost and how much fits, depend on details like these.

Truncation is the second. Here the raw APIs are strict: send a request larger than the model's context window and it fails. OpenAI's Responses API returns a 400 error unless you opt in to truncation: "auto", which drops items from the beginning of the conversation to make the input fit (OpenAI's reference for creating a response). Anthropic's API returns a 400 error saying the prompt is too long (Anthropic's guide to context windows).

Silent loss happens one layer up, in your own code or framework. Chat apps and agent frameworks often trim or summarize old turns to stay under the limit, and if they trim from the start, your earliest instructions go first. The reply then comes back normally, with no error, and simply breaks a rule the model never saw.

Catching this takes logging. Both APIs report the tokens each request used in a usage field. Log that number for every call, alert when it nears the limit, and record what your trimming step removed. On the output side, the stop reason tells you when a reply hit its token cap (length on OpenAI's Chat Completions, max_tokens on Anthropic's API), which usually means a cut off answer. Without these checks, you end up debugging a missing rule as if it were a reasoning failure.

How does the instruction hierarchy affect what I can control?

Your prompt is not the only thing shaping the reply. Models are trained to weigh instructions by where they come from. OpenAI's Model Spec calls this the chain of command: OpenAI's own rules come first, then the developer's instructions, then the user's, while tool outputs, quoted text and attached files carry no authority by default, so any instructions inside them are treated as information. On Anthropic's API, you set your instructions through the top level system parameter.

What you can and cannot see matters here. Consumer chat apps such as ChatGPT and Claude.ai add system prompts of their own. On the raw API, what the provider adds is mechanical: Anthropic, for example, says that when you pass tools, the API "automatically includes a special system prompt for the model that enables tool use" (Anthropic's tool use overview). Refusals on the API come from the model's trained policies, not from a hidden list of rules. So if a customer asks about the right medicine dose for a child, the assistant may add a safety caveat or decline to advise, and nothing in your prompt explains why: that behavior is trained in. Work with it rather than against it, for example by telling the assistant to point such questions to a pharmacist or doctor.

System prompts carry real weight in another way too. A 2025 study gave six commercial models the same demographic details either in the system prompt or in the user message, across 50 groups, and found significant differences in how the models represented those groups and how they made allocation decisions (Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs), Neumann et al.). Put in the system prompt only what you mean to apply to every conversation.

The hierarchy also explains prompt injection. Text from a customer or a retrieved web page can pose as instructions, and because everything ends up in the same token sequence, no prompt can rule this out completely. Prompt Injection: The Attack Every AI App Inherits covers the attack and the defenses. The short version for now: treat the model's output as untrusted before your code acts on it, the risk OWASP lists as improper output handling.

What does a reliable prompt structure look like?

A reliable prompt has four explicit parts. The role says who the model is and which standing policies it follows. The task says exactly what to produce, from which input. The constraints set the tone, the length, and what the model may and may not do. The format sets the shape of the output. Put the role and standing policies at the start, the input and task after them, and a one sentence reminder of the most important rule right before the model writes.

Here is the naive prompt our assistant started with:

You are a support assistant. Answer customer questions.

It fails because the model has free rein to invent the persona, the length and the policy. The structured version:

• Role: "You are a customer support agent for Acme Corp. You follow these policies: never promise refunds over $50 without manager approval, always include the help center link, and remain empathetic."
• Task: "Given the customer email below, draft a reply that addresses their issue, citing the relevant policy and escalating if needed."
• Constraints: "Keep replies under 150 words. Do not mention the competitor Globex. If the issue is a billing dispute, start with an apology and state the escalation timeline."
• Format: "Reply in plain text, with a subject line on the first line and the body below it. Start the body with Hello and the customer's name."

In an API call, the role and standing policies go in the system message: the system parameter on Anthropic's API, or the developer message (or the instructions parameter) on OpenAI's. The customer's email, wrapped in tags so the model can tell it apart from your instructions, goes in the user message, followed by the task, constraints and format, and then one short reminder of the billing dispute rule. This placement uses the strong positions described above: the standing policies sit at the very start, and the rule that matters most for this ticket sits at the very end, right before the reply begins.

SYSTEM
You are a customer support agent for Acme Corp. You follow these policies:
never promise refunds over $50 without manager approval, always include the
help center link, and remain empathetic.

USER
<customer_email>
{the customer's email}
</customer_email>

Draft a reply to the email above that addresses the issue, citing the
relevant policy and escalating if needed. Keep it under 150 words. Do not
mention the competitor Globex. Reply in plain text: a subject line first,
then the body, starting with "Hello" and the customer's name.

If this is a billing dispute, start with an apology and state the
escalation timeline.

The skeleton does not make a model perfect. It removes the guessing behind most failures, and it gives you named parts you can test one at a time. The rest of this series fills in those parts: examples that pin down tone and edge cases, reasoning for problems with several steps, schemas for output your code can parse, system prompts for agents, evaluations that tell you whether a change helped, and prompts assembled from live data at runtime.

Quick Reference

Frequently Asked Questions

Q: Should I put the most important instruction at the start or the end? Both, in a long prompt. State it near the start and repeat it in one short sentence just before the model writes. OpenAI's GPT-4.1 guide found that both beats either one alone, and that if you state it only once, above the context works better than below.

Q: Do I need to understand transformers to write good prompts? No. Knowing that text becomes tokens, that the model reads one long sequence, and that the middle of a long input is used least reliably covers most of what matters for writing prompts. The AI From Zero series explains the internals if you want them.

Q: Will the API tell me if my prompt was cut off? The raw APIs from OpenAI and Anthropic refuse an over-long request with an error rather than cutting it. Loss happens when your framework or app trims the history to fit, or when you opt in to automatic truncation. Log the token count you send and what your trimming dropped.

Q: Why does the model refuse things my prompt allows? On the API, refusals come from the model's trained policies, not from hidden text in your prompt. Consumer chat apps add their own system prompts on top. You cannot switch trained policies off, so design around them, for example by telling the assistant where to send the questions it should not answer.

Test yourself

Your support assistant's 2,500 token system prompt includes, in its middle, the rule "Never promise a refund over $50 without manager approval." Over a week, a handful of replies promise larger refunds anyway. What two changes do you make to the prompt, what do you add outside the prompt in case it is still not enough, and how do you check that the changes worked?

Answer: First, move the rule out of the middle: state it near the start of the system prompt with the other standing policies, and repeat it in one short sentence right before the model writes, after the customer's email. Second, make the rule a clear constraint with a clear alternative, for example "If a refund over $50 seems fair, say a manager will review it; never promise it yourself." Because a model can still slip, add a check outside the prompt: have your code flag any draft that mentions a refund over $50 for a person to review before it is sent. Then measure. Collect 20 or so test emails that tempt the assistant toward big refunds, run them before and after the change, and count the replies that promise too much. Also log the input token count on each request, to rule out a trimming step that drops the rule.

Next: Few-Shot: Examples Beat Instructions. It shows how a few well chosen examples pin down the tone, length and edge cases that instructions leave vague.

If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.

Sources

• Lost in the Middle: How Language Models Use Long Contexts (Liu et al. 2023)
• GPT-4.1 Prompting Guide (OpenAI Cookbook)
• Prompting best practices (Anthropic, read October 2026)
• A Taxonomy of Prompt Defects in LLM Systems (Tian et al. 2025)
• tiktoken (OpenAI)
• Create a response (OpenAI API reference, read October 2026)
• Context windows (Anthropic, read October 2026)
• OpenAI Model Spec (version of 18 August 2026)
• Tool use overview (Anthropic, read October 2026)
• Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs) (Neumann et al. 2025)
• LLM05:2025 Improper Output Handling (OWASP)