You put AI into the workflow. Your quality control is still a feeling.

Ask a marketing team how they know their AI-written output is any good and you get one of three answers. Somebody reads a few. The client has not complained. Or, most honestly, nobody knows.

That was a defensible position when a model drafted a dozen things a week and a human touched all of them. It stops being defensible at the volume most teams now run, because the failure you are exposed to is not a bad output. It is a slow, unremarkable drift in the tail that nobody is sampling, in a pipeline nobody has changed, discovered eventually by somebody outside the company.

An eval is the standing answer to that. It has exactly three parts, and teams that struggle with evals are almost always missing the third.

Build the golden set out of your own history

The input set is the part you cannot buy and should not invent. It comes from work you have already done, which means the sourcing job is archaeology rather than authorship.

Pull 30 to 50 real cases first. Take the ordinary ones that represent most of your volume, then deliberately add the cases that have burned you: the product with the confusing name, the regulated category, the client who hates a particular word, the input that arrives empty. Anthropic’s guidance on eval design is explicit that edge cases belong in the distribution, naming irrelevant or nonexistent input data, overly long inputs, and “ambiguous test cases where even humans would find it hard to reach an assessment consensus.” [5]

Size follows the question you are asking. A few dozen cases will catch the kind of breakage that matters most, which is something obviously wrong shipping quietly. Detecting a small quality change, a few percentage points, needs a few hundred. The same guidance is refreshingly direct about the tradeoff most teams get backwards: prioritize volume over quality, because “more questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals.” [5]

Then freeze it. A golden set that quietly gains cases is not a baseline, and the whole point is that this month’s number means the same thing as last month’s.

Rubrics: binary where you can, scaled where you must

The instinct is to write “rate this ad copy from 1 to 10.” It is the worst available option, because it produces a number nobody can act on and two people cannot reproduce.

Split the rubric by the nature of the question instead. Anything with a right answer becomes a rule-based binary check, and those are cheap, fast, and perfectly reliable: is it within the character limit, does it contain a banned claim, is every figure present in the source, does it include the required disclosure, is the product name spelled correctly. Only what is genuinely subjective goes to a scaled judgment, on a short anchored scale where each point has a written description.

This is not a theoretical preference. Amazon’s production system for generating marketing copy at scale combines rule-based metrics with a model judge that scores relevance and generalization on a 0 to 5 scale, and reports 89.57% agreement with human reviewers against large-scale human annotation data. [4] The lesson is the architecture as much as the number: hard constraints are handled by rules, and the judge is reserved for what rules cannot see. The same work describes a second loop worth copying, in which representative cases are sampled for human review and used to refine the evaluation prompt over time, so the rubric improves instead of ossifying. [6]

Keep the scaled part short. A 0 to 3 scale with written anchors produces more agreement than a 1 to 10 scale where the difference between a 6 and a 7 has never been defined by anyone.

The judge, and what the research actually found

Using a model to grade model output is the only thing that makes this affordable, and the evidence for it is genuinely better than skeptics assume. The MT-Bench work found that strong judges like GPT-4 “can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans.” [1] Human raters do not agree with each other perfectly either, which reframes the standard: the question is not whether a judge is flawless but whether it is as good as the human you would otherwise have used, at a fraction of the cost.

That same paper names the three failure modes you have to design against: position bias, verbosity bias, and self-enhancement bias. [1] Each has a cheap countermeasure, and none of them is optional.

Then the uncomfortable part. Rating Roulette found that “LLM judges have low intra-rater reliability in their assigned scores across different runs,” with variance making “their ratings inconsistent, almost arbitrary in the worst case.” [2] A single scoring pass is a sample, not a measurement.

And the finding that should change how you validate a judge before trusting it. A systematic evaluation of 21 judges from nine providers, across 118 runs and roughly 541,000 individual judgments, found judge rankings shifting by up to 14 positions between benchmarks, and, most pointedly, that “high test-retest reliability (>0.95) coexists with severe position bias (>0.10) in two production-deployed judges.” [3]

• Run every pairwise comparison in both orders and keep only the verdicts that survive the swap. Position bias is the cheapest bias to eliminate and the most likely to invert a decision.
• Score more than once and aggregate, because a single run has real variance.
• Never let a model judge its own family’s output when you are choosing between models.
• Calibrate against humans once, on 50 cases, and record the agreement rate. A judge you have not calibrated is an opinion with a decimal point.
• Recalibrate when you change the judge model, and treat the judge as a dependency with a version.

Is your eval actually load-bearing?

Regression is the point

The reason to build any of this is not to produce a quality score. It is to know when something got worse without anyone deciding it should.

Rerun the eval on every change to the prompt, the model, the retrieval source, or the tools available. Then rerun it on a schedule anyway, because the dependency you did not change can change underneath you when a provider ships an update. A pipeline that nobody has touched in two months is not a pipeline that has been stable for two months. Nobody has checked.

Wire the threshold to something that actually blocks. If the eval score dropping below a line does not stop a deployment or force a review, you will discover that the number is advisory the first time it is inconvenient, which is invariably the moment it mattered.

Where this earns its cost

Three workflows justify the build almost immediately, and they share a shape: high volume, low individual stakes, and an ugly aggregate if quality slips.

Generated ad copy and product text, where the rules are strict and the volume is enormous, and where a factual grounding check against your own product data catches most of what actually damages you. Classification and tagging, where accuracy is directly measurable against a labeled set and a drift of a few points quietly reshapes reporting. And written summaries of data, where the failure mode is not bad prose but a confident sentence about a number the source never contained.

None of this is exotic. It is the same discipline the rest of the measurement stack already runs on: decide the rule before you look at the result, hold the baseline fixed, and be honest about the size of the effect you can actually detect. The teams pulling ahead with AI right now are not the ones with better prompts. They are the ones who can tell you, with a number, whether this month is worse than last.

Sources

1. arXiv · Judging LLM-as-a-judge with MT-Bench and Chatbot ArenaZheng et al.: strong judges matching controlled and crowdsourced human preferences at over 80% agreement, the same level humans reach with each other, and the naming of position, verbosity, and self-enhancement bias
2. ACL Anthology · Rating Roulette: self-inconsistency in LLM-as-a-judge frameworksHaldar and Hockenmaier, Findings of EMNLP 2025: low intra-rater reliability across runs, with ratings inconsistent and almost arbitrary in the worst case
3. arXiv · Reliability without validity: a systematic, large-scale evaluation of LLM-as-a-judge models21 judges from nine providers, 118 runs and roughly 541,000 judgments: rankings shifting by up to 14 positions, and test-retest reliability above 0.95 coexisting with severe position bias in two production judges
4. arXiv · LLMs for customized marketing content generation and evaluation at scaleAmazon: rule-based metrics combined with a judge scoring relevance and generalization on a 0 to 5 scale, reporting 89.57% agreement with human reviewers
5. Anthropic · Create strong empirical evaluationsbe task-specific, automate when possible, prioritize volume over quality, and the explicit list of edge cases to include
6. Amazon Science · LLMs for customized marketing content generation and evaluation at scalethe publication record for the AutoEval framework, including the human-in-the-loop update loop that refines the evaluation prompt over time

MS

Editor-in-chief

Mikołaj Salecki

Writes about media, tech, and AI business for people who actually run digital. Former agency lead. Skeptic of frameworks that read better than they perform.

LinkedIn ↗Facebook ↗salecki.digital ↗

More articles →