Production Evals

Topics We'll Cover

• Online Eval — Sampling Production Traffic
• Drift Detection — Input & Output Distributions
• Golden Sets — Curating, Versioning, Growing
• Regression CI — Gating Deploys on Eval Pass
• Sampled Human Review — Triage and Agreement
• Feedback Loops — Thumbs Up/Down to Eval Signal
• Eval Observability — Dashboards & Alerts
• Cost Models for Continuous Eval
• What to Take Away

Online Eval — Sampling Production Traffic

You cannot score every production request — inference cost, latency budget, and annotation throughput all prevent exhaustive eval. The goal is a statistically representative sample that gives you signal on tail failures without dominating your budget.

Sampling strategies

Drift Detection — Input & Output Distributions

Distribution drift is the silent quality killer. No model change happened, no deploy occurred — yet user query distribution shifted and your quality metrics degraded. You won't see it without monitoring both the input and output distributions.

Input drift signals

Output drift signals

Golden Sets — Curating, Versioning, Growing

A golden set is your ground truth: a fixed, annotated corpus of (prompt, expected_quality) pairs against which every model version is regression-tested. It is the most valuable artefact your eval programme produces.

# Each line in golden_set_v4.2.jsonl
{
  "id":          "gs-00a3f9",           # stable, never reused
  "version_added": "v4.0",             # golden-set release that added this item
  "category":    "rag_summarisation",
  "split":       "test",              # "dev" or "test"
  "prompt":      "...",               # full prompt including system prompt
  "reference":   "...",               # gold answer or rubric
  "human_score": 4,                   # 1-5 from annotator
  "annotator":   "a12",
  "tags":        ["long-context", "en"],
  "deprecated":  false,
  "deprecation_reason": null
}

Growth cadence

• Weekly: promote 10–20 items from sampled human review to the golden set (those that reveal new failure modes or cover underrepresented categories).
• Monthly: quarterly review of deprecated items; rebalance category distribution; add adversarial items from red-team session.
• Never delete — mark items deprecated: true with a reason. Deprecated items stay in the file and can be reactivated. Deletion loses the failure history.

Regression CI — Gating Deploys on Eval Pass

Regression CI runs your golden-set eval on every candidate model before it reaches production. The gate is: all category scores must be within tolerance of the current production baseline.

# .github/workflows/eval_gate.yml (excerpt)
jobs:
  eval-gate:
    runs-on: ubuntu-latest
    steps:
      - name: Run golden-set eval
        run: |
          python ci/run_eval.py \
            --golden-set golden_set_v4.2.jsonl \
            --model "${{ env.CANDIDATE_MODEL }}" \
            --baseline-scores baselines/prod_v4.1.json \
            --tolerance 0.02 \
            --fail-on-regression

def check_regression(candidate: dict, baseline: dict, tol: float) -> list[str]:
    failures = []
    for category, baseline_score in baseline.items():
        cand_score = candidate.get(category, 0.0)
        if cand_score < baseline_score - tol:
            failures.append(
                f"{category}: {cand_score:.3f} < {baseline_score:.3f} - {tol}"
            )
    return failures  # empty = pass

Gate configuration

• Tolerance per category — tighter for safety-adjacent categories (refusal, toxicity: ±0.5 %); looser for style (summarisation length: ±5 %).
• Baseline anchoring — the baseline is the current production model's scores, not a historical best. Ratchet baseline upward only on deliberate improvement; never ratchet down.
• Soft vs hard gate — regressions of <1 pp generate a warning; >2 pp block the deploy; >5 pp on any safety category page on-call immediately.

Sampled Human Review — Triage and Agreement

Human review closes the loop that automated evals cannot close: it catches systematic judge biases, surfaces qualitatively novel failure modes, and provides the ground truth used to calibrate and retrain judge models.

Weekly review workflow

Annotation taxonomy

Feedback Loops — Thumbs Up/Down to Eval Signal

Implicit user feedback is the highest-volume signal you have. Converting it into structured eval data requires careful denoising — raw thumbs-down data is noisy and biased toward edge cases.

Noise sources in thumbs-down data

• Frustration clicks — users rate down the model when the system was slow, even if the response was good. Correlate thumbs-down rate with latency percentiles.
• Preference vs quality — user disagreed with model opinion; response was high-quality but unwanted. These should be tagged separately from quality failures.
• Prompt ambiguity — the prompt was underspecified; no model could answer it correctly. Filter by prompt clarity score before adding to golden set.

Eval Observability — Dashboards & Alerts

Eval data is only useful if it is visible. A well-designed dashboard gives on-call engineers a 30-second read on system health and a 5-minute drill path to root cause.

Recommended dashboard panels

Alert runbook (minimum viable)

Cost Models for Continuous Eval

Production eval has four cost levers: sample rate, judge model tier, human review headcount, and scoring frequency. Model these explicitly before committing to a continuous eval cadence.

# Inputs
daily_requests       = 100_000
sample_rate          = 0.01           # 1%
judge_input_tokens   = 800            # system + prompt + response
judge_output_tokens  = 200            # score + rationale
judge_cost_per_1k_in = 0.003         # Claude 3.5 Sonnet
judge_cost_per_1k_out= 0.015
human_items_per_week = 200
human_cost_per_item  = 0.25          # trained annotator (USD)

# Monthly judge cost
scored_per_day   = daily_requests * sample_rate      # 1000
judge_in_cost    = (scored_per_day * 30 * judge_input_tokens / 1000) * judge_cost_per_1k_in
judge_out_cost   = (scored_per_day * 30 * judge_output_tokens / 1000) * judge_cost_per_1k_out
human_cost_month = human_items_per_week * 4.3 * human_cost_per_item

# Results: ~$72 judge + ~$215 human = $287/month
print(f"Judge: ${judge_in_cost+judge_out_cost:.0f}/month")
print(f"Human: ${human_cost_month:.0f}/month")
print(f"Total: ${judge_in_cost+judge_out_cost+human_cost_month:.0f}/month")

What to Take Away

• Sample, don't score everything. 1–2 % uniform + trigger-based oversampling for high-stakes categories is a practical starting point. Route through an async scoring queue.
• Monitor both input and output distributions. KL divergence on embeddings for input; refusal rate, hallucination rate, and output length for output. Alert on >0.3 KL and >2 pp score shifts.
• Golden sets are the crown jewel. Never contaminate them with training data. Version every release. Grow weekly from human review. Never delete — deprecate.
• Regression CI gates deploys. Tolerance ±2 pp per category; tighter for safety. Anchor baseline to current production model, not a historical best.
• Human review converts 15–30 % of negatives. Filter noise (latency frustration, preference disagreement). IAA κ ≥ 0.7 before trusting labels.
• Dashboards need 30-second read time. Quality scorecard, drift monitor, feedback funnel on one page. Alert runbook with named owners and SLAs.
• Budget explicitly. $300–$500/month is achievable at 100K req/day with 1 % sample, Sonnet judge, and 200 human items/week.