The four pieces every AI PM owns — golden set, rubric, judge, gate. Stop shipping on vibes.
🎯 PM🤖 AI-relevant📏 Evals⚡ Beginner📖 ~22 min
Lesson 10 of AI Product Manager Track
Why evals, not unit tests
In your last PM life, "how do you know it works?" had a clean answer: QA ran 47 unit tests, all green, merge the PR. The contract was assert equality. Given input X, the system produces output Y, always. If it doesn't, someone's code is wrong.
An AI feature breaks that contract on day one. Ask the same question twice and you'll get two different answers — same intent, different wording, sometimes different facts. Run the same prompt tomorrow against the vendor's "same" model and the distribution shifts again. You can't assert output == "Yes, you can pause your loan". The string won't match.
So PMs in AI replace unit tests with evals. An eval doesn't ask "is the output exactly this?" It asks "across a representative set of inputs, what fraction of outputs are acceptable?" You don't get a pass/fail. You get a pass rate — a number between 0 and 100. And you decide, up front, what number is good enough to ship.
This lesson gives you the four pieces that make up every eval you'll ever run. Learn them once, reuse them forever.
One-line takeaway:
Unit tests prove a system does the right thing. Evals prove a system tends to do the right thing — and quantify how often it misses.
The four pieces — at a glance
Every eval you'll run, in every AI product on earth, is some version of this pipeline. Read it left to right.
Miss any one of these and your "eval" is theatre. A golden set with no rubric is a folder of examples. A rubric with no judge is a Google Doc. A judge with no gate is a dashboard no one acts on. All four, wired together, is an eval.
What each piece is — and who owns it
Four cards. One line of definition, one line of PM ownership. Screenshot this for your next standup.
📋
1. Golden set
What it is: 100–500 real inputs paired with ideal answers. PM owns: sourcing examples from real users, not imagined ones.
📏
2. Rubric
What it is: a 1–5 scale with written anchors for each grade. PM owns: the definition of "good" — tone, accuracy, safety.
⚖️
3. Judge
What it is: LLM-as-judge plus a human-reviewed sample. PM owns: auditing the judge against humans every two weeks.
🚦
4. Gate
What it is: the minimum pass rate needed to ship. PM owns: setting the number before seeing the score.
The ownership rule:
If your eval breaks, an engineer cannot fix it without you. Golden-set examples, rubric wording, judge calibration, and the launch gate number all require product judgment. This is why "PM owns the eval" is written into the AI PM job description.
Scenario — Priya wakes up to a 13-point drop
🏦
Real-world scenario
SouthBank India support bot, one month after launch — the pass rate just collapsed overnight
Priya is the PM for SouthBank India's support bot (you met her in Lesson 00). She shipped four weeks ago. Launch gate was 85% on a 200-ticket golden set; she hit 87% and rolled to 100% of users. For three weeks, the eval sat at 86–88%. Boring. Green. Good.
This Monday morning she opens the eval dashboard. Pass rate: 74%. Thirteen points below gate. Nobody changed the code over the weekend. The engineer is already in her DMs: "???"
Priya has a budget of one day to figure this out before her VP asks for a rollback decision. Here's what her eval tells her — and what a PM without one would be guessing.
Priya's golden-set pass rate — last week vs this week
What Priya does with this: she opens the 26 newly-failing tickets, groups them by topic, and finds 19 of the 26 are about the new fixed-deposit scheme launched Friday. The bot answers from stale knowledge-base docs and gets the interest rate wrong. The eval didn't regress because of the model — it regressed because the world regressed. She ships a retrieval fix, re-runs the eval, and is back at 86% by Tuesday afternoon. No rollback needed.
The only reason Priya could diagnose the problem in hours instead of weeks is that she had an eval already wired up. A PM without one would be reading random thumbs-down transcripts, hoping a pattern emerges, defending the feature to their VP on gut feel. You never want to be that PM.
How it actually works — offline vs online
Evals split into two families, and you need both. Offline evals run on your golden set, before a release, in a controlled script. Online evals run on real users, in production, after the feature is live. Offline catches regressions early; online catches the things you didn't know to put in your golden set.
A useful way to reason about eval tools is a 2×2: offline vs online on one axis, leading vs lagging on the other. Leading signals tell you something before users suffer; lagging tell you after.
How to read this:
a healthy eval programme lives in all four quadrants. Offline golden-set runs catch obvious regressions. Manual reviews catch subjective taste issues. Shadow traffic catches weird edge cases. Thumbs data catches the things none of the above saw. If you only have one quadrant — usually thumbs — you'll learn about quality problems from angry tweets.
The minimum viable eval harness — 25 lines a PM can read
You don't write this code. Your engineer does. But if you can't read it, you can't argue about the rubric, the judge, or the gate. Here's the version Priya uses at SouthBank — the same loop from Lesson 00, now fully wired.
eval_harness.pyRuns nightly · posts pass rate to Slack
What this gives Priya: one number she can defend in any meeting. "Pass rate is 86% against our 200-ticket golden set, judged by Gemini, cross-checked weekly on 30 samples by a human reviewer, gate is 85%." That sentence is the entire AI PM job compressed to 20 seconds.
Eval types compared — pick the right tool
Not every eval needs the heaviest machinery. Here are the five you'll reach for, sorted roughly by when you'd use them in a feature's life.
Eval type
What it catches
Cost
When you use it
Assert-style (rules)
Format errors, banned words, missing citations
~₹0 · runs in ms
Always, as the first filter. Fast sanity check before the judge runs.
Golden-set pass rate
Regressions, prompt drift, model swaps
₹100–500 per full run
Every PR that touches the prompt, retrieval, or model.
LLM-as-judge
Tone, accuracy, partial-credit quality
₹5–20 per example
Inside the golden-set loop for anything subjective or free-form.
Human review
Edge cases, judge miscalibration, "vibes"
~₹30–80 per example
Weekly sample of 30–50. Also before every major launch.
Live thumbs / handoff rate
What you forgot to put in the golden set
Free (built into UI)
Always-on in prod. Feeds back into the golden set monthly.
The sequencing rule:
cheap evals first, expensive last. Run rule-based asserts → golden-set with LLM-judge → human review on disagreements → thumbs in prod. Don't spend human minutes on something a regex could catch.
The AI connection
How real AI teams run evals
Anthropic, OpenAI, Google and every serious AI product team treat evals as first-class infrastructure — versioned, CI-gated, on-call-paged when they regress. The PM owns the contents (examples, rubric, gate), the platform team owns the runner. When a vendor pushes a model update, the first question in every review meeting is the same: "did the eval move?" Everything else — the demo, the thumbs data, the CEO's gut feel — is secondary to that number.
golden-setrubricllm-as-judgelaunch-gateshadow-trafficregression-alert
Mini challenge — design an eval in 20 minutes
Stand up a tiny eval for any AI feature you touch
1. Collect 20 real inputs
Open your feature's logs (or your competitor's product, or a support queue). Copy 20 real user questions into a spreadsheet. Not imagined ones — real. For each, in the next column, write the ideal answer in one or two sentences. That's a micro golden set.
2. Write a 5-line rubric
One paragraph. Grade 5 = perfect. Grade 4 = right answer, slightly off tone. Grade 3 = partly right. Grade 2 = mostly wrong. Grade 1 = unsafe. Add one concrete example for each grade, pulled from your 20 rows. Your engineer can paste this into a judge prompt as-is.
3. Pick your gate — before you run it
Write one sentence: "I'll ship this when ≥X% of the 20 examples score 4 or 5, and no single one scores 1." Do this before running the first eval — that way the score can't retroactively talk you into lowering the bar. This sentence is the backbone of every good AI PRD you'll ever write.
Interview questions
EasyWhat's a golden set?
+
A curated set of 100–500 realistic inputs paired with ideal (or at least acceptable) answers, used to score an AI feature's quality before release. It's not a test suite in the classical sense — you don't expect exact string matches. You run your current prompt/model over the inputs, have a judge (LLM or human) grade each output against the ideal, and report a pass rate. The PM owns the contents — real user queries, diverse coverage, refreshed monthly.
EasyWhy can't you use unit tests for an LLM feature?
+
Because LLM outputs are non-deterministic. The same prompt produces different strings on different runs, and the vendor can change the model underneath you. Unit tests assert exact equality, which breaks immediately. Evals replace pass/fail on one output with pass rate across many outputs, graded against a written rubric. The number becomes your quality bar.
MediumLLM-as-judge vs human review — when do you pick each?
+
Use LLM-as-judge for the bulk of your grading because it's cheap (₹5–20 per example) and fast enough to run on every PR. Use human review for (a) rubric calibration — sample 30–50 examples weekly to check the LLM-judge agrees with humans; (b) anything with subjective or high-stakes quality where the judge might miss nuance — creative writing, legal wording, medical tone; (c) before a major launch, as a final smoke test. The right answer is almost never "one or the other." It's LLM-judge for scale + weekly human audit to keep the judge honest.
MediumYour eval pass rate drops 13 points overnight. How do you triage?
+
First, check whether anything changed on your side: prompt, retrieval index, model version pin, post-processing. If code is stable, it's almost always one of three things — (1) the vendor silently updated the model (rare if you pinned the version, common if you didn't), (2) your knowledge base is now stale relative to the real world (new product, policy change, pricing update), or (3) a new class of user query started arriving that the golden set doesn't cover. Open the newly-failing examples, group them by topic, and look for the cluster. Priya's drop was stale retrieval after a new fixed-deposit product launch — 19 of 26 new failures were on that one topic. The triage takes hours with an eval, weeks without.
HardDesign an eval harness for an AI feature where the right answer is subjective — like an AI writing assistant. What does "pass" even mean?
+
You can't anchor to a single "ideal answer" for subjective work, so you shift the rubric from correctness to properties. First, decompose "good writing" into measurable dimensions: on-tone (matches voice guide), on-brief (addresses the user's intent), safe (no PII, no claims outside policy), and useful (saves the user edit time). Grade each dimension 1–5 separately. Second, the golden set becomes input + style target + do-not-do list rather than input + ideal output. Third, the judge is a calibrated LLM-judge plus human reviewers on a larger sample than usual — maybe 20% instead of 5% — because subjective grading drifts faster. Fourth, add online evals that proxy for usefulness: edit distance between AI draft and final shipped copy (smaller = more useful), time-to-ship, and a lightweight "was this helpful?" on the draft. The gate is multi-dimensional: e.g. ≥80% on-tone, ≥85% on-brief, <1% unsafe, median edit-distance < 30%. The key shift from the SouthBank bot is that there's no "right" answer — just a distribution of acceptable ones — so your rubric and your online signals both matter more, and your human-review cadence doubles.
Test your knowledge
Question 1 of 5
Which list names the four pieces of every eval?
Question 2 of 5
Why don't unit tests work for AI features?
Question 3 of 5
Priya's eval drops from 87% to 74%. Nothing in the app code changed. Most likely cause?
Question 4 of 5
Which is a leading online signal in the offline/online × leading/lagging matrix?
Question 5 of 5
When should you set the launch gate?
0
/5
You now have an eval mindset.
Common mistakes first-time AI PMs make
Building the eval after launch
The eval is the PRD. If you ship before the golden set exists, you're shipping on vibes — and you have no way to diagnose regressions when (not if) quality drops.
Using only thumbs-up in prod
Thumbs are noisy, biased toward happy users, and lag real failures by weeks. They're one quadrant of four — never run your quality bar off thumbs alone.
Letting the prompt drift without re-eval
Every prompt change, retrieval tweak, or model swap needs a full golden-set run before merge. "Just a small wording fix" is how you quietly lose 8 points of pass rate.
Trusting one LLM-judge forever
Judges drift too. Sample 30–50 examples weekly and have a human re-score them. If the LLM-judge and humans disagree by more than 10%, recalibrate the rubric before the number becomes a lie.
Key takeaways
Evals replace unit tests for AI. You measure a pass rate, not a pass/fail.
Every eval has four pieces: golden set · rubric · judge · gate. Miss one and it's theatre.
The PM owns all four — engineering wires them together but cannot define "good" for you.
Run evals in all four quadrants: offline/leading (manual review), offline/lagging (golden set), online/leading (shadow traffic), online/lagging (thumbs).
Set the launch gate before you see the score. That's how you avoid rationalizing the bar down.
When pass rate drops, open the failing examples and cluster by topic. The eval makes diagnosis a day's work, not a quarter's.