Every team knows they should have evals. Most do not, and the reason is almost always the same: evaluation sounds like a project — benchmarks, labeling pipelines, judge models, dashboards. So it stays on the backlog while prompt changes ship on the strength of three good-looking outputs in a Slack thread.
Here is the uncomfortable truth that unblocks it: an eval suite of fifty real, labeled examples that actually runs is worth more than any framework-shaped plan for five hundred. You can build the fifty this week.
Start from traces, not imagination
Synthetic test questions have a way of testing the system you wish you had. Production traces test the one you have. Pull recent traffic from your tracing tool — LangSmith, or whatever you log with — and sample three kinds of interactions:
- Known failures: anything a user flagged, retried, or abandoned; anything an engineer screenshotted into Slack.
- Typical traffic: the boring median requests that represent what the system does all day.
- Edge cases you already worry about: the long documents, the ambiguous phrasings, the queries in the domain vocabulary your embeddings might not know.
Aim for roughly half failures, half representative traffic. A suite made only of failures will make every change look like an improvement; a suite made only of successes will wave regressions through.
Label with a small, honest rubric
For each example, record what a good response requires — not a golden string to match, but a judgment a person can make in under a minute. Three fields carry most of the value:
- Verdict on the actual production output: pass / fail / marginal.
- Failure category, from a short controlled list you grow as you go: retrieval-miss, wrong-synthesis, hallucinated-detail, refused-wrongly, format-broken, truncated-context.
- What correct looks like: a sentence, the passage that should have been cited, or the key fact the answer must contain.
The failure categories are the quiet payoff. The first time you tally them, you find out whether you have a retrieval problem, a prompting problem, or a formatting problem — and teams are wrong about this surprisingly often. We have seen "the model hallucinates" turn out to be "the retriever returned nothing relevant and the model improvised" in the majority of labeled failures.
Split retrieval from generation
If your system is RAG, evaluate the layers separately from day one. For every example where you identified the answering passage, you can score retrieval mechanically: did that passage appear in the top k? At what rank? No judge model needed, runs in seconds, and it localizes failures precisely.
Then evaluate generation conditioned on retrieval: given the right context was present, was the answer faithful to it? This separation keeps you from burning a week tuning prompts to compensate for a retriever that never surfaced the evidence.
Use an LLM judge — but calibrate it before you trust it
Human-grading fifty examples per change does not scale, so you will want an LLM-as-judge for the generation-side checks: faithfulness to retrieved context, presence of required facts, correct refusal behavior. Judges are genuinely useful and genuinely fallible, so calibrate:
- Write the judge prompt against your rubric, one criterion per judgment — small focused judges beat one omnibus "rate this 1–10" prompt.
- Run it over the examples you already hand-labeled and measure agreement.
- Read the disagreements. Some will be judge errors; some will be your labels being wrong; both are worth knowing.
Agreement in the ~90% range on a clear rubric is achievable and sufficient for regression detection. Below that, tighten the rubric before trusting the judge, and keep the criteria you cannot make reliable as human spot-checks instead.
Wire it into CI while it is small
A suite that requires remembering to run it will stop being run. Fifty examples is small enough to execute on every prompt or pipeline change — minutes of wall-clock, pennies of tokens. Report three things per run:
- retrieval recall-at-k,
- per-criterion generation pass rates,
- and a per-example diff against the last accepted run — which examples flipped, in which direction.
The diff is what makes the suite debuggable rather than just alarming. "Overall pass rate dropped two points" starts an argument; "these four examples regressed, all in the refused-wrongly category" starts a fix.
Let it grow from production
The suite stays alive by feeding on real failures: every production incident becomes a labeled example the day it is triaged. Re-sample typical traffic quarterly so drift in your users' behavior shows up in the suite. Retire examples when the feature they cover is gone — not when they become inconvenient.
Fifty traces, a short rubric, a calibrated judge, a CI gate. It fits in a week, and it changes how the team ships: model upgrades become an afternoon of reading a diff, and "which prompt is better" stops being a matter of who argues loudest.