Designing an LLM Evaluation That Actually Tells You Something

A public benchmark score tells you how a model ranks against other models on someone else’s task. It rarely tells you whether the model will work for yours. Building your own evaluation is the only way to answer that, and most of the difficulty is in design rather than in tooling.

Start from the failure you care about

Before writing any harness, define the specific behaviour you are trying to measure and what a failure looks like in production. An eval that produces a single aggregate score across mixed capabilities hides exactly the information you need. Break the task into categories that fail for different reasons, for example retrieval misses, formatting violations, refusals, and factual errors, and report each separately.

Build the set from real inputs where you can. Synthetic cases generated by a model tend to cluster around what that model finds natural, which systematically misses the inputs that break things.

Choosing a grader

Graders fall into three broad classes.

Programmatic graders check something exactly: does the output parse as valid JSON, does the code compile and pass unit tests, does the extracted field match the reference string. These are cheap, deterministic and reproducible. Use them wherever the task allows.

Model graders use a language model to judge an output against a rubric. They handle open ended tasks that no assertion can check, but they inherit the judge model’s biases. Known effects include favouring longer answers, favouring answers stylistically similar to the judge’s own outputs, and position bias when comparing two candidates. Randomise the order of candidates, keep rubrics concrete and testable, and validate the judge against human labels on a sample before trusting it at scale.

Human grading remains the reference standard for subjective quality. It is slow and expensive, so use it to calibrate the automated graders rather than to run the whole suite.

Sampling, variance and pass@k

Generation is stochastic, so a single sample per prompt gives a noisy estimate. Pass@k measures the probability that at least one of k samples is correct. It is the right metric when a downstream system can filter or verify candidates, for example running generated code against tests. It is the wrong metric when the user sees only one answer, in which case the single sample success rate is what matters.

Report uncertainty. Two models differing by a small margin on a small eval set may not be distinguishable at all. Confidence intervals on the score, and holding decoding parameters fixed across compared runs, are what keep an eval from producing conclusions the data does not support.

Contamination and overfitting

Public benchmark items appear in training data. When a model scores unusually well on a well known set relative to how it behaves on fresh problems of the same difficulty, contamination is the first explanation to check. Keeping a private held out set that has never been published anywhere is the practical defence.

The equivalent problem in your own work is tuning prompts against the test set. Once you have iterated on the same examples repeatedly, the score reflects your fitting rather than the model’s capability. Keep a development set for iteration and a separate test set touched only for final measurement.

Making it a regression suite

An eval that runs once is a report. An eval that runs on every prompt change, model version and retrieval configuration is infrastructure. Version the dataset, pin the grader, log per item outputs rather than only the aggregate, and diff item level results between runs. Most real regressions show up as a handful of newly failing items, which an aggregate score can easily hide.

Common mistakes

Evaluating on a set too small to detect the differences you care about. Grading with the same model that produced the outputs. Mixing capability categories into one number. Changing the dataset and the model at the same time, which makes the comparison meaningless.