Getting a model to return data your code can parse used to be an exercise in defensive programming — prompt carefully, hope, then write a repair function for the cases where hope failed. That is no longer necessary. Constrained decoding makes invalid output structurally impossible. This page covers how that works, what it still cannot promise, and how to design a schema the model handles well.
The three approaches
Prompting for JSON — unreliable, fails a few percent of the time in ways that are hard to predict. JSON mode — guarantees it parses, not that it has your fields. Structured outputs with a schema — guarantees shape and types. Use the third whenever it is available.
Why prompting alone fails
A model generates one token at a time, each sampled from a probability distribution. Nothing in that process is aware of JSON grammar — the model produces what looked likely given its training, and JSON in training data is overwhelmingly presented inside markdown code fences.
| Failure | What you get |
|---|---|
| Markdown fences | ```json\n{…}\n``` |
| Preamble | Here is the JSON you requested: |
| Trailing commentary | Note that the second field was inferred. |
| Trailing comma | {"a": 1, "b": 2,} |
| Single quotes | {'a': 1} |
| Unescaped quotes | {"note": "she said "yes""} |
| Comments | {"a": 1 // the count} |
| Wrong field names | {"total_amount"} when you asked for total |
| Truncation | Output stops mid-object |
Each is individually rare. Across a production workload they are collectively common, and the failure rate rises exactly when you least want it to — on unusual inputs, long outputs and edge cases.
How constrained decoding works
The mechanism is simple and worth understanding, because it explains precisely what the guarantee covers.
At each step the model produces a probability for every token in its vocabulary. Normally the decoder samples from that distribution. With constrained decoding, a mask is applied first: every token that would make the output invalid against the grammar is set to zero probability.
The key property: invalid output is not detected and corrected — it is never generated. There is no retry, no repair, no post-processing. The token that would have opened a markdown fence had zero probability at that position.
💡 Why this needs provider support
Masking happens inside the sampling loop, so it requires access to the logits before a token is chosen. That is why structured outputs are a provider feature rather than something a client library can add — and why self-hosted stacks use libraries that hook the sampler directly.
The schema is compiled into a state machine once, then consulted at every step. The runtime cost is small; the compilation happens on first use, which is why an unusual schema can add latency to its first request.
JSON mode versus structured outputs
| JSON mode | Structured outputs | |
|---|---|---|
| Output parses | Guaranteed | Guaranteed |
| Field names correct | No | Guaranteed |
| Types correct | No | Guaranteed |
| Required fields present | No | Guaranteed |
| No extra fields | No | Guaranteed with additionalProperties: false |
| Values are correct | No | No |
That last row is the one to internalise. Everything these features guarantee is structural. A schema requiring {"invoice_total": number} will produce a number. Whether it is the right number is entirely unaddressed.
🚨 Structural validity can disguise nonsense
Before structured outputs, a model unable to answer might produce malformed JSON or a refusal — visibly broken, easy to catch. Under constraint it must produce something matching the schema, so it produces a well-formed object with invented values.
The failure moved from loud to silent. Always include a way for the model to signal uncertainty — a nullable field, a confidence value, or an explicit could_not_determine option in an enum — otherwise you are forcing it to guess and formatting the guess neatly.
Designing a schema the model handles well
Constrained decoding guarantees conformance. It does not make a badly designed schema easy to fill correctly.
Keep it flat
Reshape into your own nested structure after parsing. Extraction accuracy is what the model controls; object shape is what your code controls.
Use enums wherever the answer is closed
An enum eliminates a whole class of downstream normalisation. Without it you will receive "Billing", "billing issue" and "BILLING" across a large enough sample. Always include a catch-all option so the model is not forced into a wrong category.
Put descriptions in the schema
Descriptions are part of the prompt the model sees, and they are frequently more effective than the same instruction in the system message because they sit adjacent to the field being generated. Handling rules for ambiguous cases belong here.
Order fields so reasoning comes first
This is the highest-leverage trick in schema design, and it follows from how generation works.
A model generates left to right and cannot revise. If the first field is the answer, it commits before any reasoning. If a reasoning field comes first, the reasoning is in context when the answer is generated.
The improvement on classification and judgement tasks is consistent and often substantial. Note that JSON object key order is not semantically meaningful to a parser but is meaningful to generation — the model fills fields in the order the schema presents them.
Make uncertainty expressible
Without the null option and the explicit instruction, a model asked for a VAT number on a document that has none will produce something VAT-number-shaped. It has no other legal move.
What the constraint cannot do
| Guaranteed | Not guaranteed |
|---|---|
| Parses as JSON | Values are accurate |
| Field names match | The right field was chosen |
| Types match | Numbers are in a sensible range |
| Required fields present | They were populated meaningfully |
| Enum values are from the list | The correct one was picked |
| Strings are strings | Dates are valid dates |
JSON Schema's format, pattern, minimum and maximum keywords are also frequently unenforced by structured output implementations — the grammar covers structure and types, not value constraints. Validate after parsing regardless:
Failures that still happen
Truncation
The most common remaining failure. Constrained decoding guarantees validity of what it emits, not that it will finish. Hit max_tokens mid-object and you get an incomplete, unparseable fragment.
Set the limit generously, and check the finish reason rather than only catching parse errors:
Unbounded arrays
An array with no maxItems is an invitation to generate until the budget runs out. Bound them, and where the real count could be large, paginate the task instead.
Refusals
A model declining to answer must still produce schema-conforming output, so the refusal appears inside your fields — a name field containing "I cannot extract personal information from this document". Check for this explicitly rather than trusting that a populated field means a real value.
Streaming structured output
Structured output can be streamed, and each chunk is a fragment of JSON rather than a complete object. Rendering progressively requires a parser that tolerates incomplete input.
Ordering the schema so user-visible fields come first pays off here: put the summary before the metadata and something useful appears immediately rather than after the whole object completes.
A working checklist
- Use structured outputs where the provider supports them, JSON mode as a fallback.
- Set
additionalProperties: falseso you get only your fields. - Flatten the schema. Reshape in code afterwards.
- Enums for closed sets, always with a catch-all option.
- Descriptions on every ambiguous field, including how to handle missing data.
- Reasoning fields before conclusion fields.
- Make uncertainty expressible — nullable types and explicit unknown options.
- Bound arrays with
maxItems. - Check the finish reason before parsing.
- Validate semantics after parsing. Structure is not correctness.
Inspecting or validating JSON?
Format, validate and explore JSON in your browser — useful for checking model output and designing schemas, with nothing uploaded.
Open the JSON Formatter →Summary
- Prompting for JSON fails a few percent of the time, unpredictably.
- Constrained decoding masks invalid tokens during generation — bad syntax is never produced.
- JSON mode guarantees parsing; structured outputs guarantee shape.
- Neither guarantees the values are right. The failure mode became silent.
- Flat schemas, enums and field descriptions materially improve accuracy.
- Put reasoning fields before answer fields — the model cannot revise.
- Always allow uncertainty or you are formatting a guess.
- Truncation is the main remaining failure. Check the finish reason.
Frequently Asked Questions
Why does an LLM return JSON wrapped in markdown fences?
Because its training data is full of JSON presented inside code blocks, so that is the most probable continuation when asked for JSON. Prompting against it reduces the frequency but never eliminates it. Constrained decoding removes the possibility entirely, because the fence characters are not valid at that position.
What is the difference between JSON mode and structured outputs?
JSON mode guarantees the output parses as JSON but says nothing about its shape — you can get valid JSON with the wrong fields. Structured outputs constrain generation against your schema, so the field names, types and required properties are guaranteed as well.
How does constrained decoding work?
At each generation step the model produces probabilities for every token, and the decoder masks out every token that would make the output invalid against the schema. The model picks from what remains. Invalid syntax is not corrected after the fact — it is never generated.
Does a schema guarantee the values are correct?
No. It guarantees structure, not truth. A schema requiring a numeric invoice_total will produce a number, and that number can still be wrong. Constrained decoding solves parsing failures and does nothing for accuracy, which still needs validation and evaluation.
Why does my structured output get truncated mid-object?
The generation hit the max_tokens limit. Constrained decoding guarantees validity of what it produces, not that it will finish — if the budget runs out mid-object the result is incomplete and unparseable. Set a limit that comfortably exceeds your largest expected response.