A minimal evaluation harness for LLM apps

Why you need one

An LLM feature that "looks fine" in a demo can break quietly when you change a prompt, swap a model or update your documents. A minimal evaluation harness is a small script that runs a fixed set of test cases through your app and reports what changed. It does not need a platform. It needs discipline.

The four parts

1. Cases: a file of inputs, each with what a good answer must contain or avoid.
2. Runner: code that calls your app exactly as production does and stores every output.
3. Checks: functions that score each output.
4. Report: a summary and a comparison with the last run.

Step 1: collect cases

Start with 30 to 50. Take them from real use: support questions, tricky inputs, past failures. Include:

• Typical cases that must always work.
• Edge cases: empty input, very long input, another language.
• Adversarial cases: attempts to make the model ignore instructions.
• Should refuse or say "I don't know" cases, because a confident wrong answer is a failure.

Store them as JSON lines with an id, the input, and expectations.

{"id": "refund-01", "input": "Can I return an opened item?",
 "must_include": ["30 days"], "must_not_include": ["guarantee"]}

Step 2: run everything

Call the app through the same code path production uses, with fixed settings such as low temperature. Save the raw output, the prompt version, the model name, latency and token counts next to each case. Without those you cannot explain a change later.

Step 3: cheap checks first

Prefer deterministic checks. They are fast, free and do not disagree with themselves.

• Does the output parse as the required JSON schema?
• Does it contain required phrases and avoid forbidden ones?
• Is it within length limits?
• For retrieval apps, did it cite a passage that actually contains the answer?

def check(case, out):
    ok = all(s in out for s in case.get("must_include", []))
    ok &= not any(s in out for s in case.get("must_not_include", []))
    return ok

Step 4: add a model as judge, carefully

Some qualities, such as tone or helpfulness, are hard to test with string matching. A second model can grade against a written rubric. Treat it as a measurement you must validate:

• Give it a short, specific rubric and ask for a score with a reason.
• Hand-label 20 or so outputs and check the judge agrees with you.
• Do not let a model grade its own family's output without spot checks.

Step 5: make it a gate

Print a table: pass rate overall and by tag, cases that flipped from pass to fail, and cost. Then decide a rule, for example "no previously passing case may fail" or "overall pass rate must not fall". Run the harness in your CI pipeline so a prompt edit cannot ship untested.

Keep it healthy

• Add a case for every bug you fix, so it cannot return.
• Version the case file with your code.
• Refresh the set as usage changes; a stale set gives false comfort.
• Look at outputs by eye regularly. Scores hide surprises.

Quick summary

Cases, a runner, cheap checks, a validated judge for the rest, and a rule that blocks regressions. That is enough to change prompts and models with confidence.