Static benchmarks vs production evals, what to measure, the eval pyramid — and why choosing the wrong eval is worse than having none at all.
Traditional software testing is a closed function: given input x, does the system return the unique correct answer y? For LLMs the answer space is open-ended, context-dependent, and distribution-shifted between training and deployment. There is often no ground truth — only better and worse.
At temperature > 0, the same prompt produces different outputs on each call. Even temperature 0 is not strictly reproducible across hardware, batch size, or quantisation level. Your eval harness must run multiple samples and report distributions, not single values.
Once a benchmark becomes a target, it ceases to be a good measure. GPT-4 and Claude 3 Opus both score >86 % on original MMLU; the benchmark no longer separates frontier models. Every number you publish will eventually be gamed — intentionally or by data contamination.
High benchmark score ≠ high user satisfaction. A model can memorise MMLU answers during pre-training (contamination) yet fail routine helpdesk summarisation. The proxy task must match the production task or the eval is noise.
Evaluating can it do X is different from evaluating does it do X safely, consistently, and within policy. Conflating the two produces models that ace reasoning benchmarks but hallucinate citations in production.
An eval is a bet that a metric predicts business value. State that bet explicitly and revisit it after every model version. If you cannot articulate what a 3-point MMLU improvement buys your users, you are measuring the wrong thing.
Six benchmarks dominate leaderboards. Know their structure, scoring methodology, and known failure modes before trusting any headline number.
| Benchmark | Tasks / Size | Metric | Known Issues |
|---|---|---|---|
| MMLU Hendrycks 2020 | 57 subjects, 14 042 MCQs | Accuracy (4-choice) | Saturated at frontier; heavy contamination risk; 4-choice inflates random baseline to 25 % |
| MMLU-Pro Wang 2024 | 12 032 harder MCQs, 10-choice | Accuracy (10-choice) | Random baseline 10 %; still MCQ, so format gaming possible |
| GPQA Rein 2023 | 448 PhD-level science Qs | Accuracy (4-choice) | Small N; domain experts score ~65 %; suited for hard capability ceiling tests only |
| HumanEval Chen 2021 | 164 Python functions, unit tests | pass@k (k=1,10,100) | Tiny; test suite quality varies; no multi-file, no imports |
| SWE-bench Jimenez 2023 | 2 294 GitHub issues + test suites | % resolved | Expensive (~$2–5 per run per agent); repo contamination; scaffold-sensitive |
| MT-Bench Zheng 2023 | 80 two-turn conversations, 8 categories | GPT-4 score 1–10 | Judge bias (positional, verbosity); 80 items is low N for statistical confidence |
HumanEval's pass@k is the probability that at least one of k sampled completions passes all unit tests, estimated without running all k: pass@k = 1 − C(n−c, k) / C(n, k) where n is total samples and c is passing samples. This unbiased estimator avoids exponential sample cost for large k.
Static benchmarks are snapshots taken in the lab. Production LLM systems exhibit failure modes that flat MCQ or single-turn code-gen tests simply cannot expose.
User queries shift over time: new product launches, changing jargon, seasonal topics, emerging adversarial patterns. A model fine-tuned in Q1 on RLHF data may degrade silently by Q3 without any model change. Static benchmarks don't capture this; online eval does.
MT-Bench has only 2 turns. Real support conversations run 8–15 turns. Models lose track of earlier context, contradict themselves, or hallucinate earlier user statements. No standard benchmark measures reference-to-prior-turn accuracy at depth > 4.
Generating a syntactically valid JSON tool-call is not the same as calling the right tool with correct argument types, honouring schema constraints, handling errors, and retrying sensibly. ToolTalk and ToolBench exist but neither is a universal standard. Teams must roll bespoke tool-call evals against their own API surfaces.
WebArena Zhou 2023 and OSWorld Xie 2024 benchmark web-agent completion rates on real browser tasks. SWE-bench Verified 2024 adds human-validated test cases. These are expensive (>$10/task) and scaffold-sensitive; small architectural changes can swing scores by ±15 %.
Studies of deployed chatbot logs consistently find that the top failure categories are: refusal over-trigger (safe topics refused), hallucinated citations (plausible but wrong URLs/authors), instruction ignore (model ignores explicit constraints in system prompt), and format drift (correct content, wrong structure). None of these appear prominently in MMLU or HumanEval scores.
A model may possess a capability under ideal prompting conditions yet fail to deploy it reliably under production system-prompt + context-window + tool-call overhead. This distinction matters enormously for eval design.
Concrete examples of the gap: a model that solves GPQA diamond-tier chemistry under zero-shot chain-of-thought may fail to answer a simpler chemistry question when the system prompt is 4 000 tokens of customer-service instructions. The instructions crowd out reasoning capacity without the model flagging any uncertainty.
Always evaluate in your deployment context. Run your eval suite with the full production system prompt, any injected tool schemas, and representative conversation histories. A benchmark run against a bare-model endpoint tells you about capability; it tells you almost nothing about capability-in-context. The gap can easily be 15–25 % on complex tasks.
Every eval method falls into one of three classes. Your choice determines cost, scalability, and what failure modes you can catch.
| Class | Good for | Breaks down when |
|---|---|---|
| Reference-based | Code correctness, structured output, factual QA with known answers, regression CI | Free-text generation, paraphrase-valid answers, creative tasks — ROUGE penalises correct but differently-worded outputs |
| Reference-free | Summarisation quality, instruction-following, style consistency, scalable production eval | Judge biases (verbosity, self-preference) corrupt scores; requires calibration against human ground truth |
| Human-graded | Ground-truth anchor for judge calibration, nuanced policy evaluation, final sign-off before deploy | Too slow and expensive for regression CI; annotator disagreement inflates variance; hard to reproduce |
The eval pyramid borrows from software testing: fast, cheap, local checks at the bottom; slow, expensive, production signals at the top. Run all four layers but weight your investment heavily at the bottom — each layer should gate the next.
Eval cost is almost never free. Budget it explicitly or watch it silently inflate your inference bill.
| Component | Unit cost (2025 indicative) | Scale driver | Optimisation lever |
|---|---|---|---|
| SUT inference (output tokens) | $0.003–$0.015 / 1K tokens (Sonnet–Opus) | N items × avg response length | Cache eval inputs; batch API (−50 % Anthropic batch) |
| LLM judge (scoring call) | $0.003–$0.015 / 1K tokens + prompt overhead | N items × judge prompt + response | Smaller judge model; multi-shot prompt compression; sample eval 10 % |
| Human annotation | $0.05–$0.50 / item (crowdsource–expert) | Items needing human review | Active learning: only annotate uncertain samples; dual-annotate 10 % |
| SWE-bench / agent eval | $2–$10 / task (compute + tokens) | N agent tasks × avg turns | Checkpoint-based resumption; parallelize sandboxes |
| Human A/B experiment | $5–$20K / experiment (MTurk or vendor) | Statistical power needed | Use online implicit signals first; human A/B only for ambiguous cases |
For a mid-size team running 50 deploys/month: target <$500/month total eval spend. That means: ~1 500-item golden set, Sonnet-class judge, batch API, no SWE-bench on every PR. Reserve the expensive evals (agent tasks, human A/B) for monthly capability audits and major version gates.
A well-constructed 200-item eval set beats a 10 000-item dump of noisy data. These are the engineering steps that matter.
from dataclasses import dataclass
from enum import StrEnum
import json, hashlib
class Split(StrEnum):
TRAIN = "train" # prompt-development / few-shot selection
DEV = "dev" # tuning judge thresholds
TEST = "test" # held-out; never touched during development
@dataclass
class EvalItem:
id: str # stable hash of (category, prompt)
category: str # e.g. "summarisation", "tool_call", "refusal"
split: Split
prompt: str # full prompt as sent to SUT
reference: str | None # gold answer if available
rubric: str | None # judge prompt snippet for this item
tags: list[str] # e.g. ["long-context", "tool-use", "en"]
added_by: str # "human_seed" | "prod_sample_2025-Q1" | "adversarial"
def make_id(category: str, prompt: str) -> str:
return hashlib.sha1(f"{category}|{prompt}".encode()).hexdigest()[:12]Never let the golden set leak into fine-tuning data. Version-stamp every release (eval-set-v3.2.jsonl). Never delete items — mark as deprecated with a reason. Re-annotate a random 5 % sample each quarter to catch annotation drift. Report inter-annotator agreement (Cohen's kappa ≥ 0.7 is the minimum bar for a trusted set).
Deck 02 goes deep on LLM-as-Judge: pairwise vs scoring, positional and verbosity biases, calibration against human ground truth, and constitutional eval with multi-criteria rubrics.