Never trust the model's JSON: validate and repair

Ask a language model for JSON and, most of the time, you get JSON. "Most of the time" is the problem. In a system that feeds that output to other code, the occasional missing bracket, extra sentence or wrong field name becomes an outage. The rule that keeps applications stable is simple: never trust the model's structured output; validate it, and repair or reject it in code.

How structured output fails

• The JSON is wrapped in prose ("Sure! Here is the data: ...") or in a code fence.
• A required field is missing, misspelled or renamed.
• A number arrives as a string, or a date in a different format.
• A list is empty, truncated, or cut off mid-object because the response hit its length limit.
• The value is valid JSON but wrong: a category that is not in your allowed list, or a value that contradicts the source.

Syntactically valid is not the same as semantically valid. You need both checks.

Define the contract as a schema

Write down what a good response looks like, in a machine-checkable form. In Python, a validation library makes this short.

from pydantic import BaseModel, Field, ValidationError
from typing import Literal

class TicketTriage(BaseModel):
    category: Literal["billing", "technical", "account", "other"]
    urgency: int = Field(ge=1, le=5)
    summary: str = Field(min_length=5, max_length=200)
    needs_human: bool

The schema is documentation for the model, a test for its output, and the type your other code can rely on.

Ask clearly, then validate

Tell the model the exact shape, give one example, and say what to do when unsure. Many providers also offer a structured-output or JSON mode, or let you pass a schema; use it when available, because it constrains the response format. It still needs validation, since constraints on shape do not guarantee the content is correct.

import json

def parse_ticket(raw: str) -> TicketTriage:
    text = raw.strip()
    if text.startswith("```"):
        text = text.strip("`")
        text = text.split("\n", 1)[1] if "\n" in text else text
    data = json.loads(text)          # raises on invalid JSON
    return TicketTriage(**data)      # raises on wrong fields or values

A repair loop with a limit

When validation fails, you have three sensible options, in order:

1. Cheap local repair. Strip code fences, trim text before the first { and after the last }, fix trailing commas.
2. Ask the model to fix it. Send back the error message and the original output: "Your response failed validation with: ... Return corrected JSON only."
3. Fall back. After a small, fixed number of attempts, stop. Route to a default, a human, or a clear error.

def get_triage(prompt: str, call_model, max_attempts: int = 3) -> TicketTriage | None:
    message = prompt
    for attempt in range(max_attempts):
        raw = call_model(message)
        try:
            return parse_ticket(raw)
        except (json.JSONDecodeError, ValidationError) as err:
            message = f"{prompt}\n\nYour previous answer was invalid: {err}\nReturn corrected JSON only."
    return None   # caller decides what to do

Always bound the loop. An unbounded "try again" can burn cost and time on an input that will never work.

Handle truncation explicitly

If the response reached the maximum length, the JSON is probably cut off. Check the finish reason your provider reports, and treat a truncated response as a failure rather than parsing whatever arrived. Raise the limit, shorten the requested output, or split the task.

Check meaning, not just shape

Add checks that a schema cannot express:

• Does a returned quote actually appear in the source text?
• Does a referenced ID exist in your database?
• Are numbers consistent (a total that matches its parts)?
• Is a "confidence" field really tied to something, or decorative?

A small function per rule is enough, and it is where many real bugs are caught.

Log what fails

Record the raw output, the validation error, and which attempt succeeded. The pattern of failures tells you what to change in the prompt or schema, and gives you regression tests for free.

Common mistakes

1. Using json.loads on raw output with no error handling.
2. Accepting extra or missing fields silently.
3. An unbounded retry loop.
4. Treating a truncated response as complete.
5. Validating shape but never meaning.

Try it yourself

Write a schema for something small (a product review with sentiment and topics). Send 30 varied inputs to a model, validate every response, and count how many needed repair. Look at the failures and improve either the prompt or the schema, then measure again.