Why this lesson exists
In Lesson 10 you learned the four pieces of any eval — golden set, rubric, judge, gate. That's the skeleton. This lesson is the muscle. We zoom in on the two pieces a PM actually has to author with their own hands: the golden set and the judge. Get either one wrong and your pass rate is a number nobody should believe.
The uncomfortable truth most PMs discover three months in: an LLM-judge will happily lie to you. Not because it's broken — because you wrote a vague rubric, fed it ambiguous examples, and forgot to ever check its grades against a human. A judge that is wrong in the same direction every day is worse than no judge at all. It's a smiling dashboard that hides regressions until users tweet about them.
So this lesson teaches three things, in order. One: how to curate a golden set that actually represents your users — size, selection, refresh. Two: how to write a rubric a stranger could grade with. Three: how to set up the weekly human-sample review that keeps your LLM-judge honest. We'll walk Priya through all three on her SouthBank India support bot.
The six things a PM authors
If Lesson 10 gave you the four-box pipeline, this lesson breaks the middle two boxes — set + judge — into the six artefacts a PM is on the hook for. Screenshot this for your next standup. Anyone who can't name all six is shipping on vibes.
How to actually build a golden set
Three questions every PM has to answer before the engineer can wire anything: how many examples, where do they come from, and how often do they change. The answers below are battle-tested defaults for a chat-style support feature like SouthBank's. Adjust by 30% for your domain — don't reinvent.
Size — 200 is the magic number
Below 100 examples, your pass rate moves 4–6 points per run on noise alone. You'll spend half your meetings arguing about whether the model regressed or three examples got lucky. Above 500 the cost per full eval starts hurting: a 500-row golden set with an LLM-judge at ₹15 per row is ₹7,500 per run, and you'll run it on every PR. 200 is the sweet spot — small enough to run cheaply, large enough that a 5-point delta is real.
Selection — stratify, don't sample uniformly
If you grab 200 random tickets from prod, 70% will be "what's my balance" and your eval will tell you the bot is great at one thing. Stratify instead. For Priya's SouthBank bot, the 200 break down into 6 buckets she defined with her ops lead: balance & statements (40), card disputes (40), KYC & onboarding (30), loans & FDs (30), refusals & safety (30), and edge cases / multilingual (30). Each bucket has its own minimum pass rate. A 90% headline that hides a 60% on safety is the kind of thing that ends careers.
Refresh — monthly diff, quarterly overhaul
The world drifts. New products launch, fee structures change, scams evolve, users phrase things differently. Once a month, Priya pulls the last 30 days of thumbs-down tickets, clusters them, and adds 10–20 new examples in the buckets that show drift. Once a quarter, she retires the 30 oldest examples that no longer reflect real traffic. The set stays around 200, but it's never the same 200 it was last quarter.
Scenario — Priya's judge is grading too leniently
It's Tuesday morning, ten weeks after Priya shipped the SouthBank India support bot in Mumbai. She has a clean 200-example golden set, stratified across her six buckets, refreshed last week. The nightly LLM-judge — running gpt-4o on each output — has been quietly reporting 91% pass rate for three weeks. Green dashboard. Bored VP. Boring standup.
Then a customer-care lead forwards her three transcripts where the bot gave wrong UPI dispute timelines. Priya re-reads them, flips back to the dashboard: each one was graded 4/5 by the LLM-judge. They are clearly 2s. She picks 10 more answers at random and re-grades them herself. Her grades average 3.4. The LLM-judge averaged 4.2 on the same ten. The judge is grading 0.8 points high — systematically — and the real pass rate is closer to 78%, below her 85% gate.
She has 10 weeks until the next bank-wide review. She doesn't need to fix the bot — she needs to fix how she grades the bot. She's going to add a 10% human-sample review loop, recalibrate the judge against it, and rebuild trust in her own dashboard. Here's how.
The judge prompt — anatomy of an honest grader
A lenient LLM-judge is almost always a vague-rubric problem. The judge isn't lying; you didn't tell it precisely enough what to do. Below is the kind of prompt Priya rewrites her judge to use. You don't write Python — you write the prompt. This is a PM-owned artefact, not an engineering one.
RUBRICGrade 1–5. 5 = same intent + same facts + right tone + cites the policy doc. 4 = right facts and intent, tone slightly off OR citation missing. 3 = partly right, one key fact wrong or omitted. 2 = mostly wrong, misleading, or off-topic. 1 = unsafe, gives loan eligibility, exposes PII, or contradicts RBI policy.
FORMATReturn ONLY a JSON object: {"grade": <int 1-5>, "reason": "<≤15 words>"}. No prose outside the JSON.
REFUSALIf the ideal answer or the bot's answer is missing, return {"grade": 0, "reason": "input incomplete"}. Do not guess.
LLM-as-judge vs human review — pick the right tool
Both are part of every healthy eval. The mistake is treating them as alternatives instead of layers. Below is the comparison Priya keeps pinned in her PRD.
| Dimension | LLM-as-judge | Human review |
|---|---|---|
| Cost | ₹5–20 per example | ₹40–120 per example |
| Speed | 200 rows in ~3 minutes | 200 rows in ~6–8 hours of reviewer time |
| Consistency | High within a run, drifts across runs as vendor updates the model | Lower — two reviewers may differ by 0.5 points unless rubric is sharp |
| Bias | Lenient by default, prefers fluent answers, soft on safety nuance | Strict on tone, slow on factual accuracy, fatigues after ~50 rows |
| Best for | Scaling — every PR, every nightly run, all 200 rows | Audit — 10–20% sample weekly, plus all refusal & safety rows |
| Limits | Can't see novel failure modes; grades to its training distribution | Doesn't scale; can't run on every PR; expensive after ~50 rows/week |
Where the judge actually disagrees with humans
Priya's first three weeks of audits showed the LLM-judge agrees with humans 92% of the time on factual answers, but only 64% on subjective ones. The chart below is from her week-12 calibration report. Read it left to right: judge-vs-human agreement collapses on subjective and refusal questions — exactly where bad answers do the most reputational damage.
The review loop — daily, weekly, quarterly
The cadence is what makes the whole thing work. Without it, Priya is just running a script. With it, she has a system that catches drift before users do.
Three loops, three different things being checked. The daily loop catches code-side regressions — someone tweaked the prompt, retrieval index moved, model version flipped. The weekly loop catches judge drift — the LLM-judge starting to grade differently than humans. The quarterly loop catches world drift — your golden set itself going stale relative to real traffic.
The AI connection
How real AI teams keep their judges honest
Every serious AI product team — at Anthropic, at the AI orgs inside global banks, at fintechs like NairaSwift in Lagos — runs the same triple-loop. LLM-judge for scale, weekly human-sample audit for calibration, quarterly rubric review for drift. The PM owns the rubric and the cadence. When a vendor pushes a model update, the first thing rebuilt is the judge prompt — because a judge calibrated on last quarter's model is a judge that's already lying to you. golden-set rubric llm-as-judge human-review-loop judge-calibration stratified-sampling
Mini challenge — set up Priya's review loop in 30 minutes
Bring your judge back into agreement with humans
Open any AI feature you can get logs from. Pull 60 examples and bin them into 6 user-intent buckets — at least one of which is "refusal / safety". Write down the per-bucket minimum pass rate you would defend in a launch review. The number you set for the safety bucket should be higher than your headline gate.
Take a vague prompt — "grade this answer 1–5" — and rewrite it into ROLE / RUBRIC / FORMAT / REFUSAL spans like the SouthBank example above. Each grade must have a one-line written anchor. The judge must return JSON with both a grade and a ≤15-word reason.
Run your judge on 50 examples. Pick 5 at random. Grade them yourself, blind to the LLM-judge's score. Compare. If your average is more than 0.5 points different from the judge's, your judge is uncalibrated — and you've just earned the right to argue about which one of you is wrong. That argument is the calibration loop.
Interview questions
Test your knowledge
Common mistakes first-time AI PMs make
Key takeaways
- Golden set + rubric + LLM-judge + human review + cadence + re-eval trigger — six artefacts the PM authors. Engineers wire them, they don't define them.
- 200 stratified examples is the default size. Below 100 you measure noise; above 500 you bleed cost on every PR.
- A rubric without written anchors is a vibe check. Force every grade to have a one-line written reason and your judge stops lying to you.
- LLM-judge for scale, human review for audit. Weekly 10–20% sample plus all refusal & safety rows — never less.
- Three loops: daily eval (catches code drift), weekly audit (catches judge drift), quarterly refresh (catches world drift).
You now build evals that scale.
Next — design the harness that ships them.
Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Agentic Engineer →