The demo parsed cleanly forty times in a row. Then it hit production, and on request 41 the model opened with “Sure! Here’s the JSON you asked for:” and json.loads() threw. Structured outputs exist because that failure is not an edge case — it is the default behaviour of a system trained to be conversational.
Structured outputs are techniques that force a language model to return data conforming to a defined schema instead of free-form prose. The three approaches, in increasing order of reliability, are prompt instruction, function or tool calling, and constrained decoding at the sampling layer. Only the third makes malformed output structurally impossible.
Why does asking nicely stop working?
A language model samples one token at a time from a probability distribution. When you write “respond only with valid JSON,” you are shifting that distribution — you are not constraining it. The tokens for a friendly preamble still carry non-zero probability, and across enough requests, non-zero probability becomes a production incident.
The failures cluster into four shapes, and each one breaks a different part of your parser:
- Conversational wrapping. A greeting before the object, or a “Let me know if you need anything else!” after it.
- Markdown fencing. The object arrives inside triple backticks, sometimes with a language hint, sometimes without.
- Schema drift. Valid JSON, wrong shape. A field the model decided to rename, a string where you specified an integer, an array collapsed into a single object.
- Truncation. The response hits the token ceiling mid-object and the closing brace never arrives.
Teams usually respond by writing a defensive parser: strip the fences, find the first brace, find the last brace, try again. That works. It also means your data layer now depends on a regular expression, and schema drift slips straight past it.
The three approaches to structured outputs
These are not competing products. They sit at different layers of the stack, and the further down you push the constraint, the fewer failure modes survive.
| Approach | Where the constraint lives | What still breaks |
|---|---|---|
| Prompt only “Return JSON” |
In the instruction text. The model is asked, not restricted. | Everything — preambles, fences, drift, truncation. Expect a 1–5% failure rate at scale. |
| Tool calling function schemas |
In the API contract. You declare a function signature; the model fills the arguments. | Field-level drift on loosely typed schemas, and the model choosing not to call the tool at all. |
| Constrained decoding grammar-guided |
In the sampler. Tokens that would violate the grammar are masked before selection. | Nothing structural. Semantic quality can drop when the schema fights the model’s natural phrasing. |
What tool calling looks like in practice
Tool calling is the pragmatic middle. You describe the shape you want as a function schema, and the provider returns the arguments as a parsed object rather than a string you have to interrogate.
1# The schema is the contract. Every field is required and typed. 2extract_ticket = { 3 "name": "extract_ticket", 4 "description": "Extract fields from a support email.", 5 "input_schema": { 6 "type": "object", 7 "properties": { 8 "severity": {"type": "string", 9 "enum": ["low", "medium", "high"]}, 10 "product_area": {"type": "string"}, 11 "needs_human": {"type": "boolean"} 12 }, 13 # Without this line the model will happily omit fields 14 "required": ["severity", "product_area", "needs_human"] 14 } 15} 16 17# Force the call rather than leaving it optional 18response = client.messages.create( 19 model="claude-sonnet-4-6", 20 max_tokens=512, 21 tools=[extract_ticket], 22 tool_choice={"type": "tool", "name": "extract_ticket"}, 23 messages=[{"role": "user", "content": email_body}] 24) 25 26# Already a dict. No string parsing, no fence stripping. 27ticket = response.content[0].input
The two lines that do the work are required and tool_choice. Omit the first and fields go missing on ambiguous inputs. Omit the second and the model may answer in prose instead of calling the tool — which is exactly the failure you were trying to eliminate.
Schema design decides your failure rate
Once the mechanism is in place, output quality becomes a schema design problem. A few habits move the number more than switching providers does.
- Enumerate wherever the answer is closed. An
enumof three values cannot drift. A free string field labelled “severity” will eventually return “critical”, “urgent”, and “High”. - Flatten aggressively. Deeply nested objects raise both truncation risk and the chance the model attaches a value to the wrong parent. Two shallow calls beat one four-level schema.
- Write descriptions for the model, not for your teammates. The
descriptionfield is read at inference time. “ISO 8601 date, or null if the email gives no date” resolves ambiguity that the type alone cannot. - Give the model an exit. Add a nullable field or an explicit
"unknown"enum member. Without one, a model facing missing information will invent a plausible value rather than return nothing. - Validate after parsing anyway. Run the object through Pydantic or an equivalent validator. Structural validity is not semantic validity — a well-formed ticket can still carry a nonsense date.
The cost nobody budgets for: rigid schemas can degrade reasoning quality. When a model must emit tokens in an order the grammar dictates, it loses the room to think through the problem in prose first. If accuracy drops after you tighten a schema, add a reasoning string field as the first property and let the model fill it before the structured fields.
Which one should you actually use?
Prompt-only JSON is fine for prototypes and for anything where a human reads the output before it moves downstream. The moment the output feeds another system without review, move to tool calling — the migration is usually an afternoon, and it removes three of the four failure shapes outright. Reach for constrained decoding when malformed output is genuinely unacceptable: financial records, medical extraction, or any pipeline where a parse failure costs more than a retry.
This matters most in agent loops, where one step’s output becomes the next step’s input. A single malformed object does not fail one request; it corrupts the rest of the chain. Teams building on RAG architectures hit this at the point where retrieval results get routed by an LLM-produced decision object, and the debugging bill arrives well after the design decision was made.
Key Takeaways
- Prompt instructions shift a model’s output distribution; they never constrain it. A 1% malformed rate is a daily incident at 10,000 requests.
- Tool calling moves the contract from the prompt into the API and eliminates preambles, fences, and most truncation in one step.
- Constrained decoding masks invalid tokens at the sampler, making malformed output structurally impossible rather than merely rare.
- Schema design drives the residual error rate — enums over free strings, shallow over nested, and an explicit escape hatch for missing data.
- Structural validity is not semantic validity. Keep a validation layer after parsing regardless of which approach you use.
Conclusion
Structured outputs are the boundary between a language model as a chat interface and a language model as a component in a system. Everything downstream — retries, monitoring, cost, the confidence to run without a human in the loop — depends on how firmly that boundary is drawn. Pick the strictest mechanism your provider supports, then spend your remaining effort on the schema rather than on the parser. Teams moving from experiments to production will find the same principle in fine-tuning LLMs for industry use cases: constrain the model’s behaviour at the layer where the constraint is enforceable, not at the layer where it is merely requested.
Frequently Asked Questions
What are structured outputs in an LLM?
Structured outputs are techniques that make a language model return data matching a predefined schema, such as JSON, rather than free-form text. They are implemented through prompt instructions, function or tool calling, or constrained decoding at the sampling layer, with reliability increasing at each level.
Is JSON mode the same as structured outputs?
No. JSON mode guarantees the response is syntactically valid JSON but says nothing about which fields it contains. Structured outputs with a schema guarantee both valid syntax and the correct shape, which is what downstream code actually depends on.
Do structured outputs make an LLM less accurate?
They can. A rigid schema removes the model’s opportunity to reason in prose before committing to an answer. Adding a reasoning field as the first property in the schema usually recovers the lost accuracy at the cost of a few extra tokens.
Should I still validate the output if I use a strict schema?
Yes. Constrained decoding guarantees the structure is correct, not that the values are true. A date field will always contain a well-formed date, but it may be the wrong date, so a semantic validation layer stays necessary.