Stop parsing prose: structured output patterns that survive production

You ask the model for the priority and the category. It answers in a clean sentence, so you reach for a regex — /priority:\s*(\w+)/ — and it works. In the demo. For about a week. Then the model rephrases: "high priority" becomes "this looks pretty urgent," the regex returns null, the downstream code throws, and a support ticket routes to nowhere. I've shipped that bug. You probably have too.

The instinct is to make the regex smarter. Wrong layer. You don't have a parsing problem; you have a contract problem. The model emits prose because prose is what it was trained to emit, and prose is the one thing your pipeline can't lean on. So stop sharpening the parser. Decide the exact shape of data you'll accept, up front, and reject everything else. Four techniques get you there, and they stack. Here they are in the order you should reach for them.

1. Define the contract first

Before you write a line of the prompt, write the schema. First, deliberately, as the spec the rest of the pipeline builds against. If the downstream code needs a priority of low | medium | high | urgent and a category from a fixed list, that constraint is a fact about your system, and the schema is where the fact lives.

This inverts the usual flow. The naive version prompts first, then writes code to cope with whatever comes back — so the model's whims define your data model. Schema-first flips it: the schema is the authority, the prompt's job is to satisfy it, and everything past the boundary assumes it held. A good schema makes invalid states unrepresentable. There's no priority: "kind of high", because the type won't let it exist.

When to skip it: almost never. Even when you want a freeform paragraph back — a summary, a draft reply — wrap it in a one-field object. The cost is one line. The payoff is that "the output" stays addressable and testable.

2. Constrain generation — don't just hope

A schema you only check after the fact still lets the model emit garbage; you just catch it later. Better to make invalid output hard to generate in the first place. Think of it as a ladder, roughly in order of strength:

• JSON mode — the provider guarantees syntactically valid JSON. It stops unparseable junk, but says nothing about your fields: well-formed JSON with the wrong keys still gets through.
• Tool / function calling — you hand the model a JSON Schema for the arguments and it fills them in, so it constrains the shape itself, beyond the raw syntax. This is the production sweet spot, and the layer a library like Instructor rides on — now at v1.15, built on Pydantic v2, no longer Python-only. See the glue layer in the 2026 open-source AI stack.
• Grammar-constrained decoding — the decoder is masked at each step so only tokens that fit a formal grammar can be sampled, which makes invalid output impossible to emit. The most powerful, most flexible versions are self-hosted: Outlines, XGrammar, and llguidance, running inside vLLM or SGLang against weights you serve yourself.

That ladder used to sort cleanly into hosted versus self-hosted. That's the part that aged. By mid-2026 the top two rungs have merged on the big hosted APIs: OpenAI (GA), Anthropic (beta, added November 2025), and Gemini all expose schema-strict Structured Outputs, where you pass a JSON Schema and the provider holds the model to it — a real step up from the loose JSON mode that only promised valid syntax. OpenAI's strict mode is itself a grammar (CFG) engine under the hood, and it takes raw Lark or regex grammars on custom tools. Self-hosting still owns the exotic end; grammar-level constraints have plainly reached the hosted APIs too.

A tool-call schema is just your contract, expressed in the provider's dialect:

const triageTool = {
  name: "triage_ticket",
  description: "Classify a support ticket.",
  parameters: {
    type: "object",
    properties: {
      priority: { type: "string", enum: ["low", "medium", "high", "urgent"] },
      category: { type: "string", enum: ["billing", "bug", "howto", "other"] },
      needsHuman: { type: "boolean" },
    },
    required: ["priority", "category", "needsHuman"],
    additionalProperties: false,
  },
} as const;

When NOT to reach for it: constraint isn't free. Deeply nested schemas cost latency, and over-constraining can quietly degrade answer quality, because you're narrowing the model's path while it's still reasoning. Support still varies by provider and model, so confirm what yours actually does before you design around it. Constraint lowers the failure rate; it doesn't remove the need to validate. That's the next step.

3. Validate at the boundary

One rule makes everything downstream safe: treat model output as untrusted external input. Untrusted the way a request body or a third-party API response is — never assumed safe because it "usually is," or because the tool call "guarantees" the shape. The boundary is the single place your code earns the right to trust the data, and it earns that by parsing it.

In TypeScript, Zod is the clean way. Define the contract once, run the raw output through it, and you come out the far side with a fully typed value or a precise error:

import { z } from "zod";
 
const Triage = z.object({
  priority: z.enum(["low", "medium", "high", "urgent"]),
  category: z.enum(["billing", "bug", "howto", "other"]),
  needsHuman: z.boolean(),
  // normalize deliberately: trim + cap a free-text note, default to "".
  note: z.string().trim().max(280).default(""),
});
 
type Triage = z.infer<typeof Triage>;
 
function parseTriage(raw: string): Triage {
  const json = JSON.parse(raw); // throws on non-JSON — that's fine, catch upstream
  return Triage.parse(json); // throws ZodError with field-level detail
}

Three things earn their place here. The enums reject any value outside the contract, so a hallucinated "critical" fails loudly instead of slipping through. The .trim().max(280).default("") is deliberate normalization — you decide how to coerce, instead of discovering downstream that the model returned 4,000 characters. And Triage.parse fails fast with a typed error: when it throws, you know exactly which field broke and why. That typed error is the most valuable thing in the pipeline, because it's what you feed back in step four.

One schema, two jobs. Zod 4 (what zod resolves to now — the code above is unchanged for it) added z.toJSONSchema(), so the same Triage object can emit the JSON Schema you hand the provider in step two. The validator and the tool definition come from one source and can't drift apart.

When NOT to over-do it: validate, but don't re-validate the same value at every function three layers deep. Parse once, at the boundary, then pass the typed value around and trust it.

4. Retry with the error fed back

When validation fails, most code throws and moves on. But the ZodError you just caught is the most useful signal you have: it says, in plain terms, what was wrong with the output. The model that produced the bad answer can usually produce a good one if you tell it what broke. So hand the error back as the next turn's input:

async function triageWithRetry(
  ticket: string,
  maxAttempts = 3,
): Promise<Triage> {
  let lastError = "";
 
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    const correction = lastError
      ? `\n\nYour previous reply failed validation:\n${lastError}\nReturn only valid JSON matching the schema.`
      : "";
 
    // callModel and triagePrompt are stand-ins for your own API call and
    // prompt builder — swap in whatever your stack uses.
    const raw = await callModel(triagePrompt(ticket) + correction);
 
    // safeJsonParse: a non-throwing JSON.parse that returns undefined on bad
    // JSON, so a malformed reply becomes a retryable failure, not a throw.
    const result = Triage.safeParse(safeJsonParse(raw));
    if (result.success) return result.data;
 
    // Feed the exact validation failure back into the next attempt.
    lastError = result.error.issues
      .map((i) => `- ${i.path.join(".")}: ${i.message}`)
      .join("\n");
  }
 
  throw new Error(`Triage failed after ${maxAttempts} attempts: ${lastError}`);
}

The loop is bounded on purpose. An unbounded retry just burns money and latency while the model fails the same way forever. Three attempts is my default; pick yours from the cost and the criticality of the call.

When NOT to lean on it: two cautions. First, cost — every retry is another full inference, so a 10% failure rate with retries is real money at scale. Measure it. Second, idempotency — a retry has to be safe to repeat, so the model step must not have already fired a side effect. Classify first, then act.

This is the closed loop that separates a real coding agent from a confident guesser: generate, check against ground truth, feed the failure back, try again. It's the throughline of the field guide to AI coding agents — the loop is only as good as the check that closes it.

When to fall back to a second pass

Retries have a ceiling. A retry re-rolls the same approach — same prompt, same schema, same model, hoping the dice land differently. A second pass changes the approach. Reach for the wrong one and you waste both money and time.

The tell is in how the retries fail. If they fail differently each time — valid once, missing a field the next, malformed after that — you're fighting noise, and one more retry is genuinely the cheaper fix. If they fail the same way every time — the model keeps refusing to emit one nested field, or keeps inventing an enum value that isn't in the list — retrying just pays to watch it lose the same fight. That's your signal to change the approach.

A second pass is usually one of three moves. Shrink the schema: ask for the two fields the model produces reliably, derive the rest in code. Split extract-then-format: one cheap call to pull the raw facts, a second to shape them into the contract, so neither call has to do two hard things at once. Escalate the model: route the cases that keep failing to a stronger one, and keep the cheap model for the easy majority. The common thread: when retries plateau, reduce the difficulty of what you're asking for.

When NOT to bother: if a single retry already clears 99% of cases, a second-pass pipeline is complexity you don't need yet. Build the fallback when the failure logs ask for it.

The cache

A few things worth keeping:

• Make invalid states unrepresentable. A tight schema with enums and required fields does more for reliability than any amount of defensive parsing bolted on afterward.
• Validate once, at the boundary. Model output is untrusted external input. Parse it at the edge, fail fast with a typed error, and let everything downstream trust the result.
• The validator's error is your retry prompt. It's the clearest correction signal you'll get. Feed it back, bound the loop, and measure what the retries cost.
• Know retry from second pass. Different-every-time failures want one more re-roll; same-every-time failures want a simpler schema or a different model.

Underneath all of it: the schema is the contract, and the model is a fallible external system. Wrap it the way you'd wrap any unreliable dependency — constrain what you can, validate what comes back, correct with the error in hand, and keep a simpler path ready for when it won't cooperate.

Write the schema first. Everything downstream follows from that one decision.