What Are Structured Outputs? Forcing LLMs to Return Valid JSON

While standard prompting leans on the model's "best effort" to follow instructions, structured outputs use a deterministic mechanism that gets you 100% schema compliance. That difference is why they are now the industry standard for building reliable AI-to-program interfaces, and why the fragile regex-based parsers and repetitive retry loops we all wrote around them are worth deleting.

You no longer have to tolerate "chatty" models that ignore your formatting constraints. With structured outputs you stop asking nicely in a prompt and start enforcing the constraint at the engine level, where the model has no vote. This allows you to treat the large language model (LLM) as a type-safe compute primitive within your production stack.

You can then wire these models into your existing CI/CD pipelines with roughly the confidence you give a standard REST API. Because the defensive prompting and the error-handling middleware around it stop earning their keep, the overhead that comes with non-deterministic text generation drops, and you get to spend your attention on business logic instead.

What are structured outputs?

Structured outputs use a method called "constrained decoding," or guided generation. The technique narrows the model's vocabulary during sampling by checking the partial output against a specific grammar or schema. Instead of letting the model pick from its entire vocabulary of tokens, the system offers only those tokens that keep the response valid.

Under the hood, the system keeps a state machine that decides which tokens are legal at any given step. The dominant production case is JSON Schema, used for defining objects and required fields, with Regular Expressions (Regex) covering simpler patterns like ISO dates or phone numbers. When you need something more specialized, full grammars like GBNF or EBNF can pin the model to a language like SQL or to a custom domain-specific language.

A prompted model can still hand you markdown fences or a chatty preamble no matter how firmly you asked, while constrained decoding makes that deviation architecturally impossible. Because the constraint is enforced during generation rather than as a post-processing step, you are guaranteed an output that is valid by construction.

{
  "type": "object",
  "properties": {
    "name": { "type": "string" },
    "age": { "type": "integer" },
    "email": { "type": "string", "format": "email" }
  },
  "required": ["name", "age", "email"],
  "additionalProperties": false
}

Why prompting alone isn't enough

Pure prompting is non-deterministic by nature, and the failure modes it produces are exactly the ones that break production. Models wrap JSON in markdown blocks, omit required fields, or return the wrong data type, such as a string when your database expects an integer. Even with highly capable models, a 1–5% parse error rate is common at scale, which is what pays for all those expensive retry loops and hand-written validators.

OpenAI's evaluation data puts a number on the reliability gap. On complex JSON schema-following tasks, the older gpt-4-0613 model managed less than 40% reliability when relying on prompting alone, while a newer model like gpt-4o-2024-08-06 scores a perfect 100% once the strict: true parameter is enabled. Read that as the end of an argument: an engineering constraint beats a politely worded instruction, and it is not close.

The engineering overhead of "defensive prompting," where you keep telling the model to "only return raw JSON," is a significant tax on development. Structured outputs take that burden off you by moving the responsibility for format compliance from the developer to the inference engine. What you get back is the "provably correct" behavior mission-critical applications need, and I would much rather have that living in the engine than in my prompt.

How the model gets forced into the schema

The enforcement happens through a process called logit masking during model sampling. At each decoding step, the model generates logits (raw scores) for every token in its vocabulary, which typically spans 32,000 to 128,000 tokens. Left unconstrained, it samples from that entire distribution.

With structured outputs, the inference engine maintains a state machine—a pushdown automaton for JSON or a Deterministic Finite Automaton (DFA) for regex. It asks that machine which tokens are legal given the current text, then masks every invalid one by setting its logits to negative infinity. Once renormalized, the model samples only from the legal continuations, and it keeps its relative preferences among the valid choices. Nothing about the model's judgment changed, only the set of options it was allowed to vote on.

Provider implementations like OpenAI's build on Context-Free Grammars (CFG) rather than simpler Finite State Machines (FSM), because an FSM or DFA cannot express recursive types or match parentheses in deeply nested JSON structures. You pay for that in a "compile-time" overhead on the first request with a new schema, while the system generates the necessary grammar artifacts, though these are typically cached for subsequent requests.

How OpenAI, Claude and Gemini differ

The three big providers went three different ways, and if you build on more than one you will feel it. OpenAI uses the response_format parameter with strict: true, which grew out of their basic "JSON Mode" into full "Structured Outputs." OpenAI also introduced a refusal string value in the API response, so your code can detect the case where a model has deviated from the schema to issue a safety refusal, instead of guessing from a parse failure.

Anthropic's Claude uses output_config.format for JSON and strict: true for tool use. One detail will bite SDK users: the Python SDK (v1.0+) now raises a TypeError if you reach for the legacy output_format field instead of output_config. Claude also caches compiled grammars for 24 hours, but any change to the schema structure or to the set of tools provided in the request throws that cache away.

Google's Gemini, with the recommended gemini-3.8-flash model, provides native support for Pydantic (Python) and Zod (JavaScript). Its implementation holds up best of the three on recursive structures, which makes it my first pick for schemas like organizational charts, where an "Employee" object contains a list of other "Employee" objects referencing the root.

The cost: valid JSON, wrong answer

Structured outputs guarantee syntactic validity, but they can also induce "reasoning degradation," and that is the bill nobody reads before signing. The study "Let Me Speak Freely?" found that forcing a model to adhere to a rigid format from the very first token can suppress its internal reasoning chain. When a model is forced to commit to a specific token—like the first number of a math answer—it may bypass the hidden state processing that the logic required.

The effect shows up worst on complex reasoning tasks such as GSM8K, where the study measured stricter format constraints performing significantly worse than free-form text. The finding I would tape to your monitor is that the order of keys in a schema is critical: put the "answer" key before the "reasoning" key and you have forced the model into zero-shot direct answering, that is, committing to a result before writing any reasoning, and what comes back is often incorrect but perfectly formatted.

Stricter formats also increase a model's sensitivity to prompt perturbations. Small changes in wording that would not affect a natural language response can cause significant performance swings when the model is operating under heavy constraints. So the two things engineers keep collapsing into one have to stay apart: "syntax validity" means the JSON is valid, "semantic correctness" means the answer is right, and the first guarantees you nothing about the second.

Reason first, constrain last

The fix for reasoning degradation is the "Reasoning Field" pattern, and it is cheap enough that I default to it. Instead of a schema that only requests the final result, you define a schema that forces the model to perform its chain-of-thought analysis inside the structured output. Put a reasoning or explanation field at the top of the JSON object and the model gets a "scratchpad" within the token stream, before the state machine forces a categorical or numerical choice.

Because the model can "think" in unconstrained natural language first, semantic accuracy on complex tasks improves significantly; the reasoning field is simply the token space it needs to process the logic before generation reaches its terminal state. In high-stakes environments, you can even split the work in two: one call for unconstrained reasoning, then a second, cheaper call that does nothing but constrained extraction.

In the following example, the step_by_step_reasoning field is defined as a string so the model has ample space to process the logic before committing to the final_answer. That ordering is the whole trick, and it is what makes the final output both architecturally compliant and semantically sound.

{
  "type": "object",
  "properties": {
    "step_by_step_reasoning": { "type": "string" },
    "final_answer": { "type": "number" }
  },
  "required": ["step_by_step_reasoning", "final_answer"],
  "additionalProperties": false
}

When you still need your own validation layer

Structured outputs are not a complete replacement for a validation layer. They guarantee the shape of the data, and that is all they guarantee: hallucinations, mathematical errors, and safety-based deviations all travel through a valid schema without a scratch. If a model triggers a safety refusal, it will typically return a refusal string or a specific stop reason rather than the requested JSON object, which your code must be prepared to handle.

Libraries like Instructor complement structured outputs by providing a Pydantic-based validation layer. Instructor allows you to implement llm_validator for semantic checks (keeping a model from saying objectionable things, for instance) and retries automatically when validation fails. That covers the business logic structured outputs cannot enforce, such as checking if a generated date is in the future.

Your application logic must also manage the boring technical interruptions. If a response is cut off by max_tokens, or comes back with a finish_reason other than "stop", the resulting JSON will likely be incomplete and invalid, however good the grammar was. A validation layer is what stands between "valid syntax" and data you are willing to put in a database.

When to use structured outputs — and when not to

Structured outputs are essential for data extraction, classification, and agentic tool use, anywhere downstream systems must consume the data programmatically, and that is what a reliable AI-to-program interface is for. Avoid them for creative writing or initial brainstorming, where strict format constraints may stifle the model's performance for no return. And when the reasoning is hard and you cannot include a reasoning field, the "NL-to-Format" pattern—generating text first and then converting it to a schema—remains the safer trade for maintaining semantic quality.

References

• Introducing Structured Outputs in the API — OpenAI
• Structured Outputs — Claude Platform Docs
• Increase Output Consistency — Claude Platform Docs
• Structured Outputs — Gemini API, Google AI for Developers
• Constrained Decoding: Forcing LLM Output to a Grammar — ZeroEntropy
• Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models
• The Constraint Tax: Measuring Validity-Correctness Tradeoffs in Structured Outputs for Small Language Models
• Instructor — Multi-Language Library for Structured LLM Outputs