Write the eval before the prompt: eval-driven development for AI features

I've shipped an AI feature this way: wire up a model, write a prompt, run it on the two inputs that came to mind, eyeball the output, nudge the wording until the demo looks right. Ship it. It feels like progress — plausible text fills the screen and the thing appears to work.

It quietly doesn't. "Tweak the prompt until the demo works" has no definition of done, only a definition of looked fine once. You never said what correct means, so you can't tell whether your last edit improved the feature or just moved the failure somewhere you didn't look. And model output drifts run to run: the input you didn't try, or the one that passed yesterday, may already be broken. The regressions are real. You just never see them.

Ordinary software solved this shape of problem decades ago with test-driven development: write the failing check first, then make it pass. Point the same move at an AI feature and you get eval-driven development — write the eval before the prompt.

What an eval actually is

An eval is a scored check on an AI feature's behavior: a set of inputs, the outputs your system produces for them, and a grader that says how good those outputs are. It's the AI-shaped cousin of a unit test, and the two differences are the whole point.

First, the output is non-deterministic. A unit test asserts that add(2, 2) is 4, today and forever. Ask a prompt the same question twice and you can get two different answers — both fine, or one fine and one not. So you stop reasoning about a single assertion and start reasoning about a distribution of behavior across many cases. One green run proves nothing. What you're reading is the shape of the scores over a set.

Second, the result is scored, not binary. "Correct" for a summary, a label, or an agent's plan isn't always a clean true/false. Sometimes it is — valid JSON or not, the right label or the wrong one — and you should grab that certainty whenever the output allows it. But quality often lives on a gradient, and your grader has to produce a number you can compare, not just a pass stamp.

Hold those together and the unit is simple: input → output → grader. That triple is to an AI feature what a test case is to a function.

# one eval case, framework-neutral
input:  "Refund the $40 order paid by card"
output:  (whatever the model returns)
grade:
  - parses as a RefundAction
  - action is "refund"
  - amount is 40.00

Write that down and you've defined what the feature is supposed to do. Skip it and write the prompt first, and you've defined nothing. That's why prompt-first feels productive and ages so badly.

Eval-driven development is TDD for AI

Kent Beck's loop is three beats — red, green, refactor — and it ports to AI features almost without translation.

Red: write the failing eval first. Before you touch the prompt, write the cases: a handful of representative inputs and what a good output looks like for each. Run them against whatever you have — an empty prompt, a stub, last week's version. It should fail, or score low. That low score is the first honest definition of done the feature has ever had.

Green: build the minimal prompt or agent to pass it. Write the smallest thing that moves the score: the plainest prompt, the simplest tool loop, the cheapest model that clears the bar. Clever can wait. The eval says when you're done, and that's the whole trick — you stop arguing about prompt wording in the abstract, because the score settles it.

Refactor: iterate against the score. A green bar is what finally lets you change things safely. Swap in a cheaper model and watch the score move. Trim the prompt, run it again. Add a retrieval step, check the delta. Every change is a measurement now instead of a vibe, and the eval catches the regression the moment a "harmless" edit breaks a case you'd forgotten.

The punchline runs through the field guide to AI coding agents: a system is only as good as its ability to check its own work. An agent that can run your tests has a ground truth to move toward. A feature with an eval has the same. Most of a good prompt is just a good definition of done — and the eval is that definition, written down and runnable.

The grading ladder

The hard part of an eval is the grader, and graders come in rungs. Climb only as high as the output forces you to. Cheaper, more deterministic grading wins whenever you can get it.

Exact and structured assertions are the bottom rung and the first to reach for. If the feature emits JSON, assert it parses and matches a schema. If it classifies, assert the label. If it calls a tool, assert the arguments. These checks are fast, free, deterministic, and impossible to argue with — the same certainty an ordinary unit test gives you. They've also gotten easier to lean on: OpenAI, Anthropic, and Google all ship schema-strict structured-output modes now, so "the model usually returns valid JSON" is closer to a guarantee than a hope. The limit is honesty. They tell you the shape is right, not whether the content is any good. A confidently wrong answer in perfect JSON sails straight through.

LLM-as-judge grades what a regex can't, and by 2026 it's a standard, well-documented move: hand a second model the input, the output, and a rubric, and ask it to score. Is this summary faithful to the source? Did this answer actually resolve the question? Is the tone right? Powerful, and the rung to trust least. Judges have biases — they can favor longer answers, or whatever sounds confident, or outputs written in their own style. And they can be gamed: optimize hard enough against a judge and you'll learn to please the judge instead of the user. Treat it as a component that itself needs evaluating. Spot-check its scores against human judgment, keep the rubric concrete, and prefer pairwise calls ("is A better than B") over absolute scores, which drift. Never quote a judge's number as if it were ground truth.

The regression set is the rung that compounds. Every real failure that slips through — a hallucinated field, a refusal that shouldn't have fired, an edge case nobody pictured — becomes a case you add to the set. The suite stops being something you wrote once and turns into a growing memory of every mistake the feature has ever made, the way a good bug tracker turns incidents into regression tests. It's the cheapest rung to maintain and quietly the highest-value, because the bug you already fixed is the one most likely to creep back after an innocent prompt edit.

Most real features use all three: structured assertions where there's a right answer, a judge where there isn't, and a regression set growing underneath both.

Where it shines, and where it bites

Eval-driven development pays off most where behavior is scoped, repeatable, and checkable: extraction, classification, routing, structured generation, retrieval answers you can check against a source, tool-calling agents whose actions you can assert on. Anywhere you can say concretely what a good output contains is in the strike zone, the same property that makes a task safe to hand a coding agent.

It bites where "good" is genuinely subjective and open-ended: a brainstorm, a poem, a long free-form draft where ten different outputs are all fine and the difference is taste. You can still eval the guardrails — it stayed on topic, it didn't leak the system prompt — but don't pretend a score captures the quality. Force a number onto something inherently subjective and you've bought a confident metric that measures the wrong thing.

And sometimes an eval is overkill. For a throwaway script or a one-shot internal tool you'll run twice, a quick smoke check — does it run, does the output look sane — is the honest amount of rigor. The skill is knowing which situation you're in, not applying maximum ceremony to everything.

Gate every change, mine every failure

The reason to pay the upfront cost is that an eval suite is an asset that appreciates. The loop has two halves.

Build the set once, then make it the gate. Wire the suite into CI so no prompt edit, model swap, or dependency bump merges without running it. That's what turns "we think the new model is better" into "the score went up on the set we agreed matters." A CI-friendly harness runs your cases on every change the way your unit tests do — promptfoo is one open-source example. (It picked up an OpenAI owner in March 2026 and stayed open source under the same MIT license; if vendor neutrality matters for your eval tooling, note it and move on.)

Trace production and feed the failures back. Your set only knows the inputs you thought of; production knows the ones you didn't. A tracer like Langfuse — MIT-licensed core, now on v3 — captures real prompts, outputs, and costs on live traffic, so when something breaks in the wild you can lift the offending case straight out of the trace and drop it into the regression set. Real failures become permanent tests. (For retrieval features, Ragas scores the RAG-specific things a generic grader misses, like faithfulness and answer relevance, and it's grown past pure RAG into general agent and LLM evaluation.)

That's the flywheel the evals and observability layer of the 2026 open-source AI stack is built on: trace what production actually does, and let it keep schooling your eval set. Each turn makes the next change safer. Skip it and every prompt tweak is back to a roll of the dice.

The cache

A few keepers, the way we keep everything here — examples, not endorsements.

• A version-controlled eval harness beats a spreadsheet. promptfoo, or something you grow yourself — either way the win is cases in source control, run on every change, diffed like code. The tool matters less than the habit.
• Treat the judge as code that needs review. LLM-as-judge is a real, now-standard technique, and the model labs' own writing on evals and judging is worth reading for the patterns. But a judge you never check is just a second model you're trusting blindly. Calibrate it against human ratings before you lean on its scores.
• The best eval case is a real bug. Up-front brainstorms give you the cheap cases. The valuable ones are the failures you catch in production and refuse to let recur. Mine your traces.

The tools will keep moving: promptfoo wears an OpenAI badge now, Langfuse is on v3, next year's judge models will beat this year's. The habit underneath doesn't move. You don't know what "working" means until you've written down how you'd check it, and you can't improve what you refuse to measure.

So before the next prompt, write the eval. Let the score, not the demo, say when you're done, and stop shipping AI changes you can't measure.