Structured Outputs: JSON You Can Actually Parse

When your customer support reply assistant extracts a customer's name, order number and issue category from a chat transcript, your code needs that data in a shape it can read. Structured outputs give you that shape by turning a JSON (JavaScript Object Notation) Schema into a grammar and using it to block invalid tokens while the model writes. With strict structured outputs, every completed reply that is not a refusal matches your schema.

That last sentence has two exceptions in it, and they matter. A reply can stop early when it hits its token limit, leaving cut off JSON, and the model can refuse for safety reasons instead of filling in your schema. A reply can also be valid and wrong: the right shape with the wrong values. So the most reliable systems combine two things: constraints that stop malformed output from being generated at all, and a small check in your code for the cases the constraints cannot cover.

How does constrained decoding actually work under the hood?

A language model writes one token at a time. At each step it scores every token in its vocabulary and picks one. Normally any token is allowed. Constrained decoding adds a rule set derived from your JSON Schema that says which sequences are legal. At each step, the system blocks every token that would break the rules and rescales the remaining probabilities so they add up to 1 again. The model never steps outside the allowed language.

The rule set is a grammar. A context-free grammar (CFG) is a set of rules for which text sequences are legal, the same kind of rules that define a programming language's syntax. Open source tools such as Outlines showed how to do this efficiently by building an index over the model's vocabulary, so that checking which tokens are allowed adds little overhead (Efficient Guided Generation for Large Language Models, Willard and Louf). The providers do the same inside their APIs: Anthropic, for example, describes its structured outputs as "constrained sampling with compiled grammar artifacts" (Anthropic's structured outputs guide). A schema like:

{
  "type": "object",
  "properties": {
    "name": { "type": "string" },
    "order_id": { "type": "integer" },
    "category": { "enum": ["billing", "shipping", "returns"] }
  },
  "required": ["name", "order_id", "category"],
  "additionalProperties": false
}

becomes a grammar in which an extraction object must contain exactly those three keys with those types, and the category must be one of three string values. The grammar tracks where it is: inside an object, expecting a property name; inside a value, limited to a type; inside an enum, allowing only the listed options. At any point, the grammar's state says exactly which tokens keep the partial output on a path that can still finish as a valid document.

So the model cannot omit a required field, cannot add extra properties when additionalProperties is false, cannot put a string where an integer belongs, and cannot pick a category outside the enum. These become token level impossibilities.

Why is JSON mode different from full structured outputs?

JSON mode is the older, looser feature. On OpenAI's Responses API you turn it on with text: {"format": {"type": "json_object"}} (on Chat Completions, response_format: {"type": "json_object"}). It constrains the model to valid JSON, but any valid JSON: you can parse the result, yet the object can have any keys, types and nesting. The model might call the field orderId instead of order_id, or nest the answer inside another object. JSON mode also needs an instruction somewhere in the prompt to produce JSON (OpenAI's structured outputs guide), and OpenAI's API reference marks it as not recommended for GPT-4o and newer models (OpenAI's reference for creating a response).

Structured outputs with a json_schema format and strict: true replace the generic JSON grammar with one derived from your schema. The model cannot produce anything that fails your schema at the structural level: required fields, types and enum membership are all enforced during generation. OpenAI's guide puts the promise plainly: the model "will always generate responses that adhere to your supplied JSON Schema", within the subset of JSON Schema it supports.

For our support assistant, the difference matters. With JSON mode, the assistant might extract the name and order number correctly but put the category in a field called issue_type. Your downstream code breaks, and you are back to writing defensive parsers. With structured outputs, the category field will exist and will hold one of billing, shipping or returns, because those are the only values the grammar allows there. (One small caveat from Anthropic's guide: it does not guarantee the capitalization of enum values, so compare them case insensitively.)

What does a minimal working call look like?

Here is the extraction with each provider's Python SDK. You describe the schema once as a Pydantic model, and the SDK turns it into JSON Schema, sends it, and parses the reply back into the model.

from typing import Literal

from pydantic import BaseModel, ValidationError

class TicketExtraction(BaseModel):
    name: str
    order_id: int
    category: Literal["billing", "shipping", "returns"]

transcript = "Hi, this is Dana Lee. Order 48213 still hasn't arrived."

With OpenAI's Responses API:

from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY from the environment

try:
    response = client.responses.parse(
        model="gpt-6-astra",
        input=[
            {"role": "developer", "content": "Extract the customer's name, order number and issue category."},
            {"role": "user", "content": transcript},
        ],
        text_format=TicketExtraction,
    )
except ValidationError:
    ticket = None  # the reply was cut off at its token limit, so the JSON is incomplete
else:
    ticket = response.output_parsed  # None if the model refused

With Anthropic's Messages API:

from anthropic import Anthropic

client = Anthropic()  # reads ANTHROPIC_API_KEY from the environment

try:
    response = client.messages.parse(
        model="claude-opus-5-5",
        max_tokens=1024,
        system="Extract the customer's name, order number and issue category.",
        messages=[{"role": "user", "content": transcript}],
        output_format=TicketExtraction,
    )
    ticket = response.parsed_output
except ValidationError:
    ticket = None  # a refusal, or a reply cut off at max_tokens: no valid object

When ticket is None, retry or send the ticket to a person. Under the hood, the OpenAI SDK sends text: {"format": {"type": "json_schema", "name": ..., "schema": ..., "strict": true}}, and the Anthropic SDK sends output_config: {"format": {"type": "json_schema", "schema": ...}} (OpenAI's structured outputs guide, Anthropic's structured outputs guide). We checked both snippets with openai 3.35.1 and anthropic 1.15.0 against a local stand-in server that returned recorded responses (a normal reply, a refusal and a cut off reply), not against the live APIs.

How do function calling and structured outputs relate?

Function calling, also called tool calling, uses the same machinery, but only in strict mode. When you define a tool with a JSON Schema for its parameters and turn strict mode on, the provider constrains the arguments the model writes to that schema. On OpenAI that is strict: true on the function; its guide says strict mode makes calls "reliably adhere to the function schema, instead of being best effort", that Responses requests try to use strict mode by default when the schema allows it, and that Chat Completions requests are not strict unless you ask (OpenAI's function calling guide). On Anthropic it is strict tool use, also strict: true on the tool (Anthropic's structured outputs guide). Without strict mode, tool arguments are usually valid but not always.

Older code often defined a fake function just to get structured data back, a habit from before structured response formats existed. Now that OpenAI, Anthropic and Google all offer structured response formats, follow the line OpenAI's guide draws: use tools when the model should trigger something in your system, such as looking up an order, and use a structured response format when you want the model's answer itself in a fixed shape. Some of the newest Claude models (Claude Opus 5.5, Sonnet 5.5 and Fable 5.1, among others) also reject a request that forces a tool call, a tool_choice of any or a named tool, which rules out the old trick there (Anthropic's guide to thinking). Both features share the same constrained decoding underneath, so the choice is about what the output is for.

What happens when constraints clash with what the model wants to say?

Constrained decoding pushes the model into a narrower set of tokens. When the schema is strict or the model is small, this can hurt quality, because the tokens the model prefers keep getting blocked and it is pushed toward continuations that are valid but wrong. A 2026 paper calls this a "projection tax" (The Hidden Cost of Structured Generation in LLMs: Draft-Conditioned Constrained Decoding, Reddy et al.).

Its fix is to separate reasoning from formatting. Generate an unconstrained draft first, letting the model work freely, then run constrained decoding with the draft in its context, so it only has to map the content into the schema. Under a strict JSON constraint on GSM8K (a set of grade school math word problems), this raised the strict accuracy of a 1 billion parameter model (Llama 3.2 1B) from 15.2% to 39.0%.

For our assistant: if the extraction needs reasoning (this customer raised three issues; which one is primary?), do that reasoning in an unconstrained first pass, then constrain the formatting pass. Reasoning models, which you met in Chain of Thought: When Thinking Out Loud Helps, get much of this for free, because they think before the constrained answer. Anthropic's guide says its grammars apply only to the final output, not to thinking, "allowing Claude to think freely while still producing structured output in the final response". It also warns that a schema field asking for the model's step by step reasoning can trigger a refusal on its newest models, so ask for a short explanation field instead.

What guarantees do I actually get, and where do I still need validation?

Structured outputs guarantee structural correctness for a completed, non-refused reply: valid JSON, required keys present, types correct, enum values within the allowed set. They do not guarantee semantic correctness. The model can put a plausible but wrong name in the name field or assign billing to a shipping complaint. The grammar constrains form, not meaning.

Numeric bounds depend on the provider. In principle a grammar can express many of them (forbidding a leading minus sign is enough for "not negative"), and some providers enforce them: OpenAI's strict mode supports minimum, maximum and related keywords for numbers, and Google's Gemini supports minimum and maximum (Google's structured output guide). Anthropic's API does not support numerical constraints and returns an error if you send them; its SDK helpers strip them from the schema, write them into the field's description and check the reply on your side instead (Anthropic's structured outputs guide). Always check your provider's list of supported keywords.

Production systems layer checks on meaning over the structural guarantee. Guardrails AI runs validators on the output and lets you choose what happens when one fails, such as raising an error, fixing the value or asking the model again. Instructor builds on Pydantic models, and its README says "failed validations are automatically retried with the error message". These add latency but catch the errors a grammar cannot express.

For the support assistant, you would use structured outputs to guarantee that order_id is an integer and category is one of the three values. Then you would add a check that looks the order_id up in your database and flags cases where the extracted category contradicts the order's actual status. Structured outputs remove the parsing headache. Validation catches the business rule violations.

Anatomy of a reliable structured output pipeline

Here is what a production pipeline looks like for our support assistant, combining constraints at decoding time with checks afterwards:

The unconstrained pass does the reasoning. The constrained pass enforces the schema. The stop reason tells you whether the reply was refused or cut off. The validator checks business rules. If anything fails, the loop asks again, a limited number of times, before handing the ticket to a person. This treats the model as one component in a dependable data pipeline, not as a magic box you hope behaves.

Grammars cost a little up front. Anthropic notes extra latency the first time it sees a schema while the grammar compiles, and it caches compiled grammars for 24 hours from their last use. It also adds a short system prompt explaining the expected format, which you pay for as input tokens. After the first request, the masking adds little per token, and removing parse failures and retries usually makes the whole system cheaper and faster than best effort JSON with defensive code.

Quick Reference

Frequently Asked Questions

Q: Does structured outputs cost more in tokens or latency?

A little. Anthropic adds a short system prompt describing the format, which counts as input tokens, and the first request with a new schema waits while the grammar compiles; after that the compiled grammar is cached. The per token overhead is small, and the total cost is often lower because you stop paying for retries after parse failures.

Q: Can I use structured outputs with streaming?

Yes. OpenAI and Anthropic both support streaming with structured outputs. Each piece you receive is part of a document that will match your schema if it completes, so buffer the text and parse it at the end, or use a parser that can read partial JSON.

Q: What if my schema is too complex for the grammar compiler?

Providers support a subset of JSON Schema, and the subsets differ. OpenAI's strict mode supports anyOf (not at the root), definitions and recursive schemas, but not allOf, not or if/then/else. Anthropic supports anyOf, allOf with limits and $ref, but not recursive schemas or numerical constraints. Both reject a schema that uses an unsupported feature with an error, so you find out at once. For rules a schema cannot express, such as checks across fields, validate in code.

Q: How do I handle optional fields in my schema?

It depends on the provider. In OpenAI's strict mode, every field must be listed in required; you make a field optional by allowing null, for example "type": ["string", "null"], and the model then returns null when it has no value. Other providers may allow fields that are truly optional, so check their docs.

Q: What is the fallback if I cannot use provider-native structured outputs?

Use a library like Guardrails AI or Instructor. These parse the model's output (handling common formatting issues such as markdown code fences), validate it against your schema, and ask again with the error on failure. You get schema adherence at the cost of possibly several model calls, but it works with any provider, including local models.

Test yourself

Your customer support assistant extracts order_id (integer, required) and refund_amount (number, required, with minimum: 0). Your team moved it to a provider whose structured outputs do not support minimum, and a helper removed the keyword from the schema before sending it. On one call, the model returns {"order_id": 12345, "refund_amount": -50.00}. The JSON parses, both fields exist and the types are correct, but the refund is negative, and your downstream code processes it and charges the customer. What went wrong, and how do you fix it?

Answer: The bound never reached the grammar. The provider does not support minimum, so the schema it compiled said only "a number", and -50.00 is a perfectly valid number. Structured outputs did their job, structure; the business rule was never enforced. The fix has two parts. First, check your provider's supported keywords: where bounds are supported (OpenAI's strict mode, Gemini), keep them in the schema; where they are not, use an SDK helper that validates them on your side (Anthropic's does) or add the check yourself. Second, and either way, validate business rules in code before any money moves: refund_amount must be at least 0 and no more than the order total. If the check fails, retry with a clear error such as "refund_amount must not be negative", or send the ticket to a person. Structured outputs guarantee structure, not meaning.

Next: System Prompts for Agents: The Job Description. It gives the assistant tools, such as looking up orders and issuing refunds, and shows how to write the system prompt that tells it when, and when not, to use them.

If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.

Sources

• Structured model outputs (OpenAI, read October 2026)
• Create a response (OpenAI API reference, read October 2026)
• Function calling (OpenAI, read October 2026)
• Structured outputs (Anthropic, read October 2026)
• Thinking (Anthropic, read October 2026)
• Structured output (Google, read October 2026)
• Efficient Guided Generation for Large Language Models (Willard and Louf 2023)
• The Hidden Cost of Structured Generation in LLMs: Draft-Conditioned Constrained Decoding (Reddy et al. 2026)
• Guardrails AI (GitHub)
• Instructor (GitHub)