Structured Outputs: JSON You Can Actually Parse

When your customer-support reply assistant extracts a customer's name, order number, and issue category from a chat transcript, you need that data to be machine-readable every single time. Structured outputs make this possible by converting a JSON (JavaScript Object Notation) Schema into a context-free grammar and using it to mask invalid tokens during generation. The model literally cannot produce output that violates your schema.

Here is the counterintuitive part: the most reliable structured-output systems do not parse the model's output after the fact. They prevent bad output from ever being generated. By the time your code sees the response, it is already guaranteed to match the shape you asked for. No regex hacks, no defensive try/catch around JSON.parse, no retry loops. The constraint lives in the decoding loop itself.

How does constrained decoding actually work under the hood?

An autoregressive language model generates text one token at a time by sampling from a probability distribution over its vocabulary, given the context so far. Normally, every token is fair game. Constrained decoding introduces a formal language, derived from your JSON Schema, that defines which sequences are legal. At each step, the system masks out every token that would lead to an illegal state and renormalizes the distribution over the remainder. The model never steps outside the allowed language.

The formal language comes from compiling your JSON Schema into a context-free grammar. OpenAI's implementation converts the schema into a CFG, pre-processes it into a cached data structure, and consults that structure on every token generation step. source A schema like:

{
  "type": "object",
  "properties": {
    "name": { "type": "string" },
    "order_id": { "type": "integer" },
    "category": { "enum": ["billing", "shipping", "returns"] }
  },
  "required": ["name", "order_id", "category"],
  "additionalProperties": false
}

becomes a grammar where an extraction object must contain exactly those three keys with those types, and the category field must be one of three string literals. The grammar tracks position: inside an object, expecting a property name; inside a value, constrained to a specific type; inside an enum, only allowing listed options. At any prefix, the grammar state encodes exactly which tokens keep the partial sequence on a path that can still complete to a valid JSON document matching the schema. source

This means the model cannot omit required fields, cannot add extra properties when additionalProperties is false, cannot use a string where an integer is expected, and cannot pick a category outside the enum. These become token-level impossibilities.

Why is JSON mode different from full structured outputs?

JSON mode, available in OpenAI's API (application programming interface) by setting text.format to {"type": "json_object"}, constrains the model to generate syntactically valid JSON. The grammar is generic: any valid JSON document is allowed. You are guaranteed that JSON.parse will succeed, but the resulting object can have arbitrary keys, types, and nesting. The model might call the field orderId instead of order_id or return extra commentary alongside the JSON. source

Structured outputs with json_schema and strict: true replace the generic JSON grammar with one derived specifically from your schema. The model cannot produce anything that fails schema validation at the structural level required fields, types, enum membership are all enforced during generation. source The internal evaluations for gpt-4o-2024-08-06 report 100% adherence to complex schemas under this setup.

For our customer-support assistant, this distinction matters. With JSON mode, the assistant might correctly extract name and order_id but put the category in a field called issue_type. Your downstream code breaks, and you are back to writing defensive parsers. With structured outputs, the category field will exist and will contain exactly one of billing, shipping, or returns because those are the only tokens the grammar allows at that position.

How do function calling and structured outputs relate?

Function calling is structured outputs in disguise. When you define a tool with a JSON Schema describing its parameters, the provider uses constrained decoding to guarantee that any function call arguments match that schema. The model chooses a function name and fills in arguments; the grammar ensures the arguments conform to the declared types and required fields. source

For data extraction tasks, you can define a function whose only purpose is to accept the structured data you want. The function never executes. It exists purely to give the decoding engine a schema to enforce. This works well when your extraction maps cleanly onto a function's argument list. For our assistant, you might define log_customer_issue(name: string, order_id: int, category: enum) and treat the function call as your structured output. source

Response-level structured outputs with json_schema are better when you need arbitrary nesting, want to reuse a schema across many calls without the function abstraction, or find the tool-calling paradigm awkward for pure data extraction. Both mechanisms rely on the same underlying constrained decoding stack. The choice is ergonomic, not technical.

What happens when constraints clash with what the model wants to say?

Constrained decoding forces the model into a narrower token space. When the schema is strict or the model is small, this can degrade output quality because the model's preferred tokens keep getting masked. The Draft-Conditioned Constrained Decoding paper formalizes this as a projection loss: you are discarding probability mass assigned to illegal sequences, and that "tax" grows when the schema excludes many high-probability paths. source

The fix is to separate reasoning from formatting. Generate an unconstrained draft first, letting the model think freely, then feed that draft as context into a constrained decoding step that maps the content into the schema. This two-phase approach improved structured accuracy on GSM8K from 15.2% to 39.0% using a 1B parameter model. source

For our assistant, this means: if the extraction requires reasoning (this customer mentioned three separate issues, which one is primary?), do that reasoning in an unconstrained first pass. Then constrain the formatting pass. OpenAI's reasoning tokens and Anthropic's extended thinking features achieve something similar by keeping internal reasoning outside the schema envelope and only constraining the final visible output. source source

What guarantees do I actually get, and where do I still need validation?

Structured outputs guarantee structural correctness: valid JSON, required keys present, types correct, enum values within the allowed set. What they do not guarantee is semantic correctness. The model can put a plausible but wrong name in the name field or assign billing to a shipping complaint. The grammar only constrains form, not meaning.

Numeric bounds (minimum, maximum) are also tricky. A CFG cannot enforce that an integer falls within a range at the token level because the constraint depends on the full value, and tokens are generated incrementally. Libraries like Guidance explicitly note that numeric bounds "cannot really be supported in the context of LLM (large language model) generation." source Providers implement a practical subset and leave the rest to post-hoc validation. source

Production systems layer semantic validation on top of syntactic constraints. Guardrails AI wraps any LLM call with JSON Schema validation and Python-level validators. If a field must be a valid email or a URL must be reachable, you write a validator. When validation fails, Guardrails re-prompts the model with error feedback. source This adds latency but catches the errors that grammars cannot express. Instructor takes a similar approach with Pydantic models and automatic retries. source

For the customer-support assistant, you would use structured outputs to guarantee that order_id is an integer and category is one of the three enum values. Then you would add a validation layer that checks the order_id against your database and flags cases where the extracted category contradicts the order's actual status. Structured outputs eliminate the parsing headache. Validation catches the business logic violations.

Anatomy of a reliable structured output pipeline

Here is what a production-grade pipeline looks like for our support assistant, combining decoding-time constraints with post-hoc validation:

The unconstrained pass does the reasoning. The constrained pass enforces the schema. The validator checks business rules. If anything fails, the feedback loop re-prompts. This architecture treats the LLM as a component in a reliable data pipeline, not as a magic box you hope behaves.

OpenAI caches the compiled grammar per schema, so only the first request with a new schema pays the compilation cost. Subsequent requests reuse the cached artifact. source The token-level masking adds modest per-step overhead, but the elimination of parsing failures and retries typically makes the overall system faster and cheaper than best-effort JSON mode with defensive code.

Quick Reference

Frequently Asked Questions

Q: Does structured outputs cost more in tokens or latency?

The schema itself counts against input tokens, and the first request with a new schema incurs a one-time compilation cost for building the grammar artifact. Per-token generation overhead is small because the provider caches the pre-processed grammar and uses efficient data structures for token masking. The net cost is often lower because you eliminate retries from parsing failures. source

Q: Can I use structured outputs with streaming?

Yes. Providers like OpenAI support streaming with structured outputs. The model sends tokens as they are generated, and those tokens are guaranteed to be prefixes of a valid schema-conforming JSON document. You may need to buffer partial output and parse only on completion, depending on your parser's tolerance for incomplete JSON. source

Q: What if my schema is too complex for the grammar compiler?

Providers support a practical subset of JSON Schema. Nested objects, arrays, enums, required fields, and additionalProperties are well supported. Features like $ref, allOf, and anyOf have varying levels of support depending on the provider. Check the provider's documentation for the exact supported subset. For constraints that cannot be expressed in a CFG (numeric ranges, cross-field dependencies), use post-hoc validation. source

Q: How do I handle optional fields in my schema?

Define them in your JSON Schema without listing them in the required array. The grammar will allow the model to include or omit optional fields. If the model omits an optional field, the resulting JSON simply will not have that key. If it includes the field, the value must match the declared type. This works identically to standard JSON Schema semantics.

Q: What is the fallback if I cannot use provider-native structured outputs?

Use a library like Guardrails or Instructor. These parse the model's output (handling common formatting issues like markdown code fences), validate against your schema, and re-prompt with error feedback on failure. You get schema adherence at the cost of potentially multiple LLM calls, but this works with any provider, including local models. source

Test yourself

Your customer-support assistant uses structured outputs to extract order_id (integer, required) and refund_amount (number, required, with minimum: 0). On one call, the model returns {"order_id": 12345, "refund_amount": -50.00}. The JSON is structurally valid: both fields exist, types are correct. But refund_amount is negative, which violates the minimum: 0 constraint. Your downstream code processes the negative refund and accidentally charges the customer. What went wrong, and how do you fix it?

Answer: The grammar compiled from your JSON Schema enforces required keys and types but cannot enforce numeric bounds at the token level because a CFG has no mechanism to check that a completed number falls within a range while tokens are being generated incrementally. The model can output -50.00 as a valid number token sequence that satisfies the number type constraint, and the grammar will allow it. The minimum: 0 constraint is part of JSON Schema but lives outside what the CFG can express. The fix is to add a post-hoc validation layer: after receiving the structured output, validate refund_amount >= 0 in your application code or using a library like Guardrails with a field-level validator. If validation fails, re-prompt the model with an explicit error like "refund_amount must be non-negative." Structured outputs guarantee structural correctness, not semantic correctness. Always validate business constraints in code.

If getting this kind of breakdown every week on how real systems work under the hood sounds useful, subscribe to Internals Decoded at internalsdecoded.com. Next up in the series: what happens when you need the model to use external APIs, databases, or your own code during generation.

Sources

• OpenAI Structured Outputs documentation
• OpenAI guide on how structured outputs works
• OpenAI JSON mode documentation
• OpenAI function calling documentation
• Anthropic extended thinking documentation
• Draft-Conditioned Constrained Decoding paper
• Guidance documentation on JSON Schema limitations
• Guardrails AI documentation
• Instructor documentation
• OpenAI reasoning documentation
• platform.openai.com · Function Calling
• platform.openai.com · Structured Outputs
• platform.openai.com · Structured Outputs