Why this lesson exists

In Lesson 10 you learned the four pieces of any eval — golden set, rubric, judge, gate. That's the skeleton. This lesson is the muscle. We zoom in on the two pieces a PM actually has to author with their own hands: the golden set and the judge. Get either one wrong and your pass rate is a number nobody should believe.

The uncomfortable truth most PMs discover three months in: an LLM-judge will happily lie to you. Not because it's broken — because you wrote a vague rubric, fed it ambiguous examples, and forgot to ever check its grades against a human. A judge that is wrong in the same direction every day is worse than no judge at all. It's a smiling dashboard that hides regressions until users tweet about them.

So this lesson teaches three things, in order. One: how to curate a golden set that actually represents your users — size, selection, refresh. Two: how to write a rubric a stranger could grade with. Three: how to set up the weekly human-sample review that keeps your LLM-judge honest. We'll walk Priya through all three on her SouthBank India support bot.

One-line takeaway:
A golden set is your test data. A rubric is your test definition. A judge is your test runner. Human review is the audit that proves the runner isn't lying.

The six things a PM authors

If Lesson 10 gave you the four-box pipeline, this lesson breaks the middle two boxes — set + judge — into the six artefacts a PM is on the hook for. Screenshot this for your next standup. Anyone who can't name all six is shipping on vibes.

📋
1. Golden set
100–500 real inputs paired with ideal answers. Sourced from prod logs, not imagination.
📏
2. Rubric
A 1–5 scale with written anchors per grade. Two PMs reading it must agree on what 4 means.
🤖
3. LLM-judge
A second model that grades each output against the rubric. Cheap, fast, biased toward leniency.
👤
4. Human review
10–20% sample re-graded by a person each cycle. The audit that keeps the LLM-judge honest.
🗓️
5. Review cadence
Daily offline run, weekly human sample, monthly set refresh, quarterly rubric overhaul.
🚨
6. Re-eval trigger
Any prompt edit, retrieval change, model swap, or judge-vs-human gap >10% — the eval reruns.
The chain rule:
the golden set bounds what you can measure, the rubric bounds what "good" means, the judge bounds how often you can grade, and human review bounds how much you can trust the judge. Weakest link sets your real quality bar — not your headline pass rate.

How to actually build a golden set

Three questions every PM has to answer before the engineer can wire anything: how many examples, where do they come from, and how often do they change. The answers below are battle-tested defaults for a chat-style support feature like SouthBank's. Adjust by 30% for your domain — don't reinvent.

Size — 200 is the magic number

Below 100 examples, your pass rate moves 4–6 points per run on noise alone. You'll spend half your meetings arguing about whether the model regressed or three examples got lucky. Above 500 the cost per full eval starts hurting: a 500-row golden set with an LLM-judge at ₹15 per row is ₹7,500 per run, and you'll run it on every PR. 200 is the sweet spot — small enough to run cheaply, large enough that a 5-point delta is real.

Selection — stratify, don't sample uniformly

If you grab 200 random tickets from prod, 70% will be "what's my balance" and your eval will tell you the bot is great at one thing. Stratify instead. For Priya's SouthBank bot, the 200 break down into 6 buckets she defined with her ops lead: balance & statements (40), card disputes (40), KYC & onboarding (30), loans & FDs (30), refusals & safety (30), and edge cases / multilingual (30). Each bucket has its own minimum pass rate. A 90% headline that hides a 60% on safety is the kind of thing that ends careers.

Refresh — monthly diff, quarterly overhaul

The world drifts. New products launch, fee structures change, scams evolve, users phrase things differently. Once a month, Priya pulls the last 30 days of thumbs-down tickets, clusters them, and adds 10–20 new examples in the buckets that show drift. Once a quarter, she retires the 30 oldest examples that no longer reflect real traffic. The set stays around 200, but it's never the same 200 it was last quarter.

What Priya does with this: she stops treating the golden set as a one-time deliverable. It's a living artefact, owned by her, refreshed on a calendar her engineer can't touch without her sign-off. When her VP asks "how do you know quality is holding?", she points at the refresh log, not the dashboard.

Scenario — Priya's judge is grading too leniently

🏦
Real-world scenario
SouthBank India support bot — the LLM-judge says 91%, but Priya's gut says it's 78%

It's Tuesday morning, ten weeks after Priya shipped the SouthBank India support bot in Mumbai. She has a clean 200-example golden set, stratified across her six buckets, refreshed last week. The nightly LLM-judge — running gpt-4o on each output — has been quietly reporting 91% pass rate for three weeks. Green dashboard. Bored VP. Boring standup.

Then a customer-care lead forwards her three transcripts where the bot gave wrong UPI dispute timelines. Priya re-reads them, flips back to the dashboard: each one was graded 4/5 by the LLM-judge. They are clearly 2s. She picks 10 more answers at random and re-grades them herself. Her grades average 3.4. The LLM-judge averaged 4.2 on the same ten. The judge is grading 0.8 points high — systematically — and the real pass rate is closer to 78%, below her 85% gate.

She has 10 weeks until the next bank-wide review. She doesn't need to fix the bot — she needs to fix how she grades the bot. She's going to add a 10% human-sample review loop, recalibrate the judge against it, and rebuild trust in her own dashboard. Here's how.

The judge prompt — anatomy of an honest grader

A lenient LLM-judge is almost always a vague-rubric problem. The judge isn't lying; you didn't tell it precisely enough what to do. Below is the kind of prompt Priya rewrites her judge to use. You don't write Python — you write the prompt. This is a PM-owned artefact, not an engineering one.

ROLEYou are a strict QA reviewer for SouthBank India's support bot. You grade against an ideal answer, not against your own knowledge of banking.

RUBRICGrade 1–5. 5 = same intent + same facts + right tone + cites the policy doc. 4 = right facts and intent, tone slightly off OR citation missing. 3 = partly right, one key fact wrong or omitted. 2 = mostly wrong, misleading, or off-topic. 1 = unsafe, gives loan eligibility, exposes PII, or contradicts RBI policy.

FORMATReturn ONLY a JSON object: {"grade": <int 1-5>, "reason": "<≤15 words>"}. No prose outside the JSON.

REFUSALIf the ideal answer or the bot's answer is missing, return {"grade": 0, "reason": "input incomplete"}. Do not guess.
Why this fixes Priya's lenient judge:
the original prompt said "grade quality 1 to 5." That's a vibe check. The new prompt forces the judge to anchor each grade to concrete properties (facts, tone, citation, safety). When a grade is 4, there's a written reason it isn't a 5. When she audits 20 grades next week, she can argue with the judge's reason, not just its number — and that's the only way recalibration ever lands.

LLM-as-judge vs human review — pick the right tool

Both are part of every healthy eval. The mistake is treating them as alternatives instead of layers. Below is the comparison Priya keeps pinned in her PRD.

Dimension LLM-as-judge Human review
Cost ₹5–20 per example ₹40–120 per example
Speed 200 rows in ~3 minutes 200 rows in ~6–8 hours of reviewer time
Consistency High within a run, drifts across runs as vendor updates the model Lower — two reviewers may differ by 0.5 points unless rubric is sharp
Bias Lenient by default, prefers fluent answers, soft on safety nuance Strict on tone, slow on factual accuracy, fatigues after ~50 rows
Best for Scaling — every PR, every nightly run, all 200 rows Audit — 10–20% sample weekly, plus all refusal & safety rows
Limits Can't see novel failure modes; grades to its training distribution Doesn't scale; can't run on every PR; expensive after ~50 rows/week
What Priya does with this: LLM-judge runs on all 200 every night. A reviewer she contracts in Pune re-grades 20 random rows every Friday. Refusal & safety bucket (30 rows) is re-graded by the human every week, regardless of the random sample, because the LLM-judge is weakest there.

Where the judge actually disagrees with humans

Priya's first three weeks of audits showed the LLM-judge agrees with humans 92% of the time on factual answers, but only 64% on subjective ones. The chart below is from her week-12 calibration report. Read it left to right: judge-vs-human agreement collapses on subjective and refusal questions — exactly where bad answers do the most reputational damage.

LLM-judge vs human agreement (n=200, week 12) Factual 92% Subjective 64% Multi-step 72% Refusal 56% ↓ Trust threshold: 85% Below 85% the LLM-judge cannot be trusted alone. Refusal & subjective rows must be human-reviewed every week.

The review loop — daily, weekly, quarterly

The cadence is what makes the whole thing work. Without it, Priya is just running a script. With it, she has a system that catches drift before users do.

Daily · 2am LLM-judge runs all 200 pass rate posted to Slack Weekly · Friday Human re-grades 20 random + all 30 refusal rows Quarterly · review week Rubric & bucket overhaul retire 30, add 30 PRIYA OWNS THE CADENCE. ENGINEERS WIRE THE SCRIPTS. ↑ catches regressions ↑ catches judge drift ↑ catches world drift

Three loops, three different things being checked. The daily loop catches code-side regressions — someone tweaked the prompt, retrieval index moved, model version flipped. The weekly loop catches judge drift — the LLM-judge starting to grade differently than humans. The quarterly loop catches world drift — your golden set itself going stale relative to real traffic.

The AI connection

How real AI teams keep their judges honest

Every serious AI product team — at Anthropic, at the AI orgs inside global banks, at fintechs like NairaSwift in Lagos — runs the same triple-loop. LLM-judge for scale, weekly human-sample audit for calibration, quarterly rubric review for drift. The PM owns the rubric and the cadence. When a vendor pushes a model update, the first thing rebuilt is the judge prompt — because a judge calibrated on last quarter's model is a judge that's already lying to you. golden-set rubric llm-as-judge human-review-loop judge-calibration stratified-sampling

Mini challenge — set up Priya's review loop in 30 minutes

Bring your judge back into agreement with humans

1. Stratify your golden set into 6 buckets

Open any AI feature you can get logs from. Pull 60 examples and bin them into 6 user-intent buckets — at least one of which is "refusal / safety". Write down the per-bucket minimum pass rate you would defend in a launch review. The number you set for the safety bucket should be higher than your headline gate.

2. Rewrite the judge prompt against the rubric anatomy

Take a vague prompt — "grade this answer 1–5" — and rewrite it into ROLE / RUBRIC / FORMAT / REFUSAL spans like the SouthBank example above. Each grade must have a one-line written anchor. The judge must return JSON with both a grade and a ≤15-word reason.

3. Pick a 10% sample and re-grade it yourself

Run your judge on 50 examples. Pick 5 at random. Grade them yourself, blind to the LLM-judge's score. Compare. If your average is more than 0.5 points different from the judge's, your judge is uncalibrated — and you've just earned the right to argue about which one of you is wrong. That argument is the calibration loop.

Interview questions

Easy How big should a golden set be?
+
For most chat or summarisation features, 200 stratified examples is the working default. Below 100, run-to-run noise drowns out real changes — a 5-point delta might just be three lucky examples. Above 500 the cost per full eval starts to bite when you're running it on every PR. Stratify across 4–8 user-intent buckets so a strong overall pass rate can't hide a weak one inside a single category like safety or refusal.
Easy What's a rubric and why does it have written anchors?
+
A rubric is the written definition of what a 1, 2, 3, 4, and 5 mean for your feature — usually anchored to concrete properties like facts, tone, citation, and safety. Anchors matter because without them every grader (LLM or human) drifts toward their own private definition of "good", and your pass rate becomes a vibe. The test is simple: hand the rubric and one example to two strangers. If they can't agree on the grade within 1 point, your anchors aren't tight enough yet.
Medium Your LLM-judge says 91% but your gut says 78%. What do you do?
+
Don't argue with the dashboard — calibrate against it. Pull 20 random rows from the latest run, re-grade them yourself blind to the LLM-judge's score, then compare. If your average is more than 0.5 points lower, the judge is grading lenient — almost always because the rubric is vague or the judge prompt lacks anchors. Rewrite the judge prompt to force a written reason alongside every grade, re-run on the same 20 rows, and check whether the gap closes. Then stand up a weekly 10–20% human-sample audit so this never gets discovered by accident again.
Medium How often should you refresh a golden set, and how do you decide what to retire?
+
Monthly diff, quarterly overhaul. Each month, pull the last 30 days of thumbs-down or escalation tickets, cluster by topic, and add 10–20 new examples in the buckets that show drift. Each quarter, retire the 30 oldest examples that no longer reflect real traffic — for example a SouthBank ticket about a fee structure that was discontinued. Keep total size near constant (~200) so pass rates stay comparable across quarters. Always log what you changed and why; if a regression hits, you need to know whether the drop is the bot getting worse or the bar getting harder.
Hard A vendor pushes a new model version. The LLM-judge agreement with humans drops from 90% to 72% on the same golden set, even though the bot's answers look fine. Walk me through the next 48 hours.
+
First, separate three things that this could be: the bot got worse, the judge got worse, or the rubric is missing a property the new model surfaces. Hour 0–4: pull 30 rows where the LLM-judge and human disagreed, re-grade them yourself, and tag each disagreement with a reason — fact, tone, citation, safety, formatting. If 80% of disagreements cluster on one property, your rubric is missing or vague on that property. Hour 4–12: rewrite the judge prompt to add an explicit anchor for the missed property, re-run on the same 30 rows, and check whether agreement recovers. Hour 12–24: if the gap stays, swap the judge model itself (often the cheapest fix — a different model family is less correlated with the production model). Hour 24–48: lock in whichever judge prompt + judge model combination gets human agreement back above 85%, document the change in your rubric changelog, and pause the launch gate for 24 hours of fresh eval data before you trust the dashboard again. The mistake is letting the gate keep firing in the meantime — a judge you no longer trust must not be allowed to ship anything.

Test your knowledge

Question 1 of 5
What's the working-default size of a golden set for a chat-style AI feature?
Question 2 of 5
Priya's LLM-judge reports 91% but her own re-grade of 10 random rows averages 3.4 (vs the judge's 4.2). What's the most likely root cause?
Question 3 of 5
Which is the right cadence for a healthy review loop?
Question 4 of 5
Which question type has the worst LLM-judge vs human agreement, and therefore needs human review every week?
Question 5 of 5
A vendor model update lands and your LLM-judge agreement with humans drops from 90% to 72%. What do you NOT do?
0
/5
You can now grade the graders.

Common mistakes first-time AI PMs make

Trusting one LLM-judge forever
Judges drift the moment a vendor updates the model, and they're systematically lenient on subjective and refusal questions. Without a weekly human audit, your dashboard becomes confidently wrong.
Sampling the golden set uniformly
If 70% of prod traffic is balance checks, a uniform sample makes you great at balance checks and blind to refusals. Stratify into 4–8 user-intent buckets and set a per-bucket minimum.
Rubrics without written anchors
"Grade quality 1 to 5" is a vibe check. Without one-line written anchors per grade, two graders will disagree by 1 point on half your examples and you'll never know your judge is lying.
Treating the golden set as a one-time deliverable
The world drifts. New products, new fees, new scams, new phrasings. Without a monthly diff and quarterly overhaul, your eval slowly stops measuring the feature your users actually use.

Key takeaways

  • Golden set + rubric + LLM-judge + human review + cadence + re-eval trigger — six artefacts the PM authors. Engineers wire them, they don't define them.
  • 200 stratified examples is the default size. Below 100 you measure noise; above 500 you bleed cost on every PR.
  • A rubric without written anchors is a vibe check. Force every grade to have a one-line written reason and your judge stops lying to you.
  • LLM-judge for scale, human review for audit. Weekly 10–20% sample plus all refusal & safety rows — never less.
  • Three loops: daily eval (catches code drift), weekly audit (catches judge drift), quarterly refresh (catches world drift).
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You now build evals that scale.
Next — design the harness that ships them.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →