# What Are Structured Outputs? Forcing LLMs to Return Valid JSON

> Structured outputs ensure 100% JSON schema compliance in LLM responses, eliminating parsing errors and improving AI-to-program reliability.

Source: https://camnangai.com/en/engineering/what-are-structured-outputs  
Published: 2026-09-03

**Structured outputs** are an inference-time technique that forces model responses to adhere
  strictly to a developer-defined schema, such as JSON.

While standard prompting leans on the model's "best effort" to follow instructions, structured
outputs use a deterministic mechanism that gets you 100% schema compliance. That difference is why
they are now the industry standard for building reliable AI-to-program interfaces, and why the
fragile regex-based parsers and repetitive retry loops we all wrote around them are worth deleting.

You no longer have to tolerate "chatty" models that ignore your formatting constraints. With
structured outputs you stop asking nicely in a prompt and start enforcing the constraint at the
engine level, where the model has no vote. This allows you to treat the
[large language model (LLM)](https://camnangai.com/en/knowledge/what-is-an-llm) as a type-safe compute primitive within
your production stack.

You can then wire these models into your existing CI/CD pipelines with roughly the confidence you
give a standard REST API. Because the defensive prompting and the error-handling middleware around it
stop earning their keep, the overhead that comes with non-deterministic text generation drops, and
you get to spend your attention on business logic instead.

![Structured outputs illustrated: a large language model's free-form output forced into a rigid JSON shape](https://camnangai.com/articles/structured-outputs-la-gi/01-infographic-en.webp)

## What are structured outputs?

**Structured outputs** use a method called "constrained decoding," or guided generation. The
technique narrows the model's vocabulary during sampling by checking the partial output against a
specific grammar or schema. Instead of letting the model pick from its entire vocabulary of
[tokens](https://camnangai.com/en/knowledge/what-are-ai-tokens), the system offers only those tokens that keep the
response valid.

![Three levels of output constraint: regular expressions, JSON Schema and GBNF grammars](https://camnangai.com/articles/structured-outputs-la-gi/02-en.webp)

Under the hood, the system keeps a state machine that decides which tokens are legal at any given
step. The dominant production case is JSON Schema, used for defining objects and required fields,
with Regular Expressions (Regex) covering simpler patterns like ISO dates or phone numbers. When you
need something more specialized, full grammars like GBNF or EBNF can pin the model to a language like
SQL or to a custom domain-specific language.

A prompted model can still hand you markdown fences or a chatty preamble no matter how firmly you
asked, while constrained decoding makes that deviation architecturally impossible. Because the
constraint is enforced _during_ generation rather than as a post-processing step, you are guaranteed
an output that is valid by construction.

```json
{
  "type": "object",
  "properties": {
    "name": { "type": "string" },
    "age": { "type": "integer" },
    "email": { "type": "string", "format": "email" }
  },
  "required": ["name", "age", "email"],
  "additionalProperties": false
}
```

## Why prompting alone isn't enough

Pure prompting is non-deterministic by nature, and the failure modes it produces are exactly the ones
that break production. Models wrap JSON in markdown blocks, omit required fields, or return the wrong
data type, such as a string when your database expects an integer. Even with highly capable models, a
1–5% parse error rate is common at scale, which is what pays for all those expensive retry loops and
hand-written validators.

![Four common failure modes when prompting alone for JSON: added preamble, markdown fences, wrong data types, missing required fields](https://camnangai.com/articles/structured-outputs-la-gi/03-en.webp)

OpenAI's evaluation data puts a number on the reliability gap. On complex JSON schema-following
tasks, the older `gpt-4-0613` model managed less than 40% reliability when relying on prompting
alone, while a newer model like `gpt-4o-2024-08-06` scores a perfect 100% once the `strict: true`
parameter is enabled. Read that as the end of an argument: an engineering constraint beats a politely
worded instruction, and it is not close.

The engineering overhead of "defensive prompting," where you keep telling the model to "only return
raw JSON," is a significant tax on development. **Structured outputs** take that burden off you by
moving the responsibility for format compliance from the developer to the inference engine. What you
get back is the "provably correct" behavior mission-critical applications need, and I would much
rather have that living in the engine than in my prompt.

## How the model gets forced into the schema

The enforcement happens through a process called logit masking during model sampling. At each
decoding step, the model generates logits (raw scores) for every token in its vocabulary, which
typically spans 32,000 to 128,000 tokens. Left unconstrained, it samples from that entire
distribution.

![Logit masking: invalid tokens are set to negative infinity before softmax so their probability becomes zero](https://camnangai.com/articles/structured-outputs-la-gi/04-en.webp)

With structured outputs, the inference engine maintains a state machine—a pushdown automaton for JSON
or a Deterministic Finite Automaton (DFA) for regex. It asks that machine which tokens are legal
given the current text, then masks every invalid one by setting its logits to negative infinity. Once
renormalized, the model samples only from the legal continuations, and it keeps its relative
preferences among the valid choices. Nothing about the model's judgment changed, only the set of
options it was allowed to vote on.

Provider implementations like OpenAI's build on Context-Free Grammars (CFG) rather than simpler
Finite State Machines (FSM), because an FSM or DFA cannot express recursive types or match
parentheses in deeply nested JSON structures. You pay for that in a "compile-time" overhead on the
first request with a new schema, while the system generates the necessary grammar artifacts, though
these are typically cached for subsequent requests.

## How OpenAI, Claude and Gemini differ

The three big providers went three different ways, and if you build on more than one you will feel
it. OpenAI uses the `response_format` parameter with `strict: true`, which grew out of their basic
"JSON Mode" into full "Structured Outputs." OpenAI also introduced a `refusal` string value in the
API response, so your code can detect the case where a model has deviated from the schema to issue a
safety refusal, instead of guessing from a parse failure.

![Comparison of how OpenAI, Claude and Gemini implement structured outputs through different API parameters](https://camnangai.com/articles/structured-outputs-la-gi/05-en.webp)

Anthropic's Claude uses `output_config.format` for JSON and `strict: true` for tool use. One detail
will bite SDK users: the Python SDK (v1.0+) now raises a `TypeError` if you reach for the legacy
`output_format` field instead of `output_config`. Claude also caches compiled grammars for 24 hours,
but any change to the schema structure or to the set of tools provided in the request throws that
cache away.

Google's Gemini, with the recommended `gemini-3.8-flash` model, provides native support for Pydantic
(Python) and Zod (JavaScript). Its implementation holds up best of the three on recursive structures,
which makes it my first pick for schemas like organizational charts, where an "Employee" object
contains a list of other "Employee" objects referencing the root.

## The cost: valid JSON, wrong answer

Structured outputs guarantee syntactic validity, but they can also induce "reasoning degradation,"
and that is the bill nobody reads before signing. The study "Let Me Speak Freely?" found that forcing
a model to adhere to a rigid format from the very first token can suppress its internal reasoning
chain. When a model is forced to commit to a specific token—like the first number of a math answer—it
may bypass the hidden state processing that the logic required.

![Syntactic validity versus semantic correctness: a schema-valid JSON object can still carry the wrong answer](https://camnangai.com/articles/structured-outputs-la-gi/06-en.webp)

The effect shows up worst on complex reasoning tasks such as GSM8K, where the study measured stricter
format constraints performing significantly worse than free-form text. The finding I would tape to
your monitor is that the order of keys in a schema is critical: put the "answer" key before the
"reasoning" key and you have forced the model into zero-shot direct answering, that is, committing to
a result before writing any reasoning, and what comes back is often incorrect but perfectly
formatted.

Stricter formats also increase a model's sensitivity to prompt perturbations. Small changes in
wording that would not affect a natural language response can cause significant performance swings
when the model is operating under heavy constraints. So the two things engineers keep collapsing into
one have to stay apart: "syntax validity" means the JSON is valid, "semantic correctness" means the
answer is right, and the first guarantees you nothing about the second.

## Reason first, constrain last

The fix for reasoning degradation is the "Reasoning Field" pattern, and it is cheap enough that I
default to it. Instead of a schema that only requests the final result, you define a schema that
forces the model to perform its chain-of-thought analysis _inside_ the structured output. Put a
`reasoning` or `explanation` field at the top of the JSON object and the model gets a "scratchpad"
within the token stream, before the state machine forces a categorical or numerical choice.

![Schema field order compared: placing the reasoning field before the answer instead of answering first](https://camnangai.com/articles/structured-outputs-la-gi/07-en.webp)

Because the model can "think" in unconstrained natural language first, semantic accuracy on complex
tasks improves significantly; the reasoning field is simply the token space it needs to process the
logic before generation reaches its terminal state. In high-stakes environments, you can even split
the work in two: one call for unconstrained reasoning, then a second, cheaper call that does nothing
but constrained extraction.

In the following example, the `step_by_step_reasoning` field is defined as a string so the model has
ample space to process the logic before committing to the `final_answer`. That ordering is the whole
trick, and it is what makes the final output both architecturally compliant and semantically sound.

```json
{
  "type": "object",
  "properties": {
    "step_by_step_reasoning": { "type": "string" },
    "final_answer": { "type": "number" }
  },
  "required": ["step_by_step_reasoning", "final_answer"],
  "additionalProperties": false
}
```

## When you still need your own validation layer

**Structured outputs** are not a complete replacement for a validation layer. They guarantee the
_shape_ of the data, and that is all they guarantee: hallucinations, mathematical errors, and
safety-based deviations all travel through a valid schema without a scratch. If a model triggers a
safety refusal, it will typically return a `refusal` string or a specific stop reason rather than the
requested JSON object, which your code must be prepared to handle.

![Two layers of defence: structured outputs guarantee the shape, a validation layer checks the business logic](https://camnangai.com/articles/structured-outputs-la-gi/08-en.webp)

Libraries like [**Instructor**](https://python.useinstructor.com/) complement structured outputs by
providing a Pydantic-based validation layer. Instructor allows you to implement `llm_validator` for
semantic checks (keeping a model from saying objectionable things, for instance) and retries
automatically when validation fails. That covers the business logic structured outputs cannot
enforce, such as checking if a generated date is in the future.

Your application logic must also manage the boring technical interruptions. If a response is cut off
by `max_tokens`, or comes back with a `finish_reason` other than `"stop"`, the resulting JSON will
likely be incomplete and invalid, however good the grammar was. A validation layer is what stands
between "valid syntax" and data you are willing to put in a database.

## When to use structured outputs — and when not to

**Structured outputs** are essential for data extraction, classification, and
[agentic](https://camnangai.com/en/knowledge/what-is-an-ai-agent) tool use, anywhere downstream systems must consume the
data programmatically, and that is what a reliable AI-to-program interface is for. Avoid them for
creative writing or initial brainstorming, where strict format constraints may stifle the model's
performance for no return. And when the reasoning is hard and you cannot include a reasoning field,
the "NL-to-Format" pattern—generating text first and then converting it to a schema—remains the safer
trade for maintaining semantic quality.

## References

- [Introducing Structured Outputs in the API — OpenAI](https://openai.com/index/introducing-structured-outputs-in-the-api/)
- [Structured Outputs — Claude Platform Docs](https://platform.claude.com/docs/en/build-with-claude/structured-outputs)
- [Increase Output Consistency — Claude Platform Docs](https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/increase-consistency)
- [Structured Outputs — Gemini API, Google AI for Developers](https://ai.google.dev/gemini-api/docs/structured-output)
- [Constrained Decoding: Forcing LLM Output to a Grammar — ZeroEntropy](https://zeroentropy.dev/concepts/constrained-decoding/)
- [Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models](https://arxiv.org/abs/2408.02442)
- [The Constraint Tax: Measuring Validity-Correctness Tradeoffs in Structured Outputs for Small Language Models](https://arxiv.org/abs/2605.26128)
- [Instructor — Multi-Language Library for Structured LLM Outputs](https://python.useinstructor.com/)