Write the evaluation set before the promptWrite the evaluation set before the prompt

Teams building with language models usually arrive at evaluation in the same order: build something that works on a few examples, demo it, discover it fails in ways nobody anticipated, then assemble an evaluation set from those failures.

That set is useful — it prevents the specific regressions it was built from. But it is not a specification. It describes the system that exists rather than the behaviour that was wanted, and it inherits every assumption that was never questioned during the build.

The set is the requirements document

Writing examples before implementation forces the questions that are otherwise deferred. What should happen when the document is a credit note rather than an invoice? When the question rests on a metric nobody has defined? When two tools disagree? Each of these becomes a row with an expected outcome, and answering them is the design work.

This is the same argument as test-first development, with one difference that makes it stronger: for a deterministic function you can often reason your way to correct behaviour, whereas for a model-driven system the only way to know what it does is to run it. The evaluation set is the only specification that can actually be checked.

Assert on the decision, not the prose

Scoring free text against a reference answer is unreliable and expensive. The productive move is to make the system emit a structured decision alongside its prose, and to assert on the structure.

interface EvalCase {
  id: string
  input: string
  /** The decision the system must reach. */
  expect: {
    action: 'answer' | 'escalate' | 'refuse'
    /** Ids that must appear in the cited sources. */
    citesAnyOf?: string[]
    /** Fields that must be null — i.e. the system must admit it doesn't know. */
    nullFields?: string[]
  }
}

const cases: EvalCase[] = [
  {
    id: 'credit-note-misfiled-as-invoice',
    input: 'fixtures/credit-note-0042.pdf',
    expect: { action: 'escalate', nullFields: ['po_reference'] },
  },
  {
    id: 'uncertified-metric',
    input: 'How healthy is the north region?',
    expect: { action: 'refuse' },
  },
]

The second case is the one teams leave out. A set that contains only questions the system should answer cannot detect over-reach, and over-reach is the failure that reaches users. Correct refusal has to be scored as a pass.

Score failure types separately

A single accuracy number hides the only tradeoff that matters. In an operations workflow, acting when the system should have escalated is a safety failure; escalating when it could have acted is a cost failure. Both reduce one score, and they are not equally bad.

Report them separately and the tuning conversation becomes concrete: we will accept more escalations to drive safety failures toward zero. That is a decision a business can make. "Accuracy is 91%" is not.

Keep it small enough to run constantly

An evaluation set of forty well-chosen cases that runs on every change is worth far more than four hundred cases that run before a release. Choose cases for coverage of distinct failure modes rather than volume, and add a new one each time production surprises you.