From a "just make it accurate" ask to a working harness in 90 minutes. Step by step.
🎯 PM🤖 AI-relevant🛠️ Mini-build⚡ Beginner📖 ~24 min
Lesson 12 of AI Product Manager Track
What this mini-build is
Lessons 10 and 11 gave you the parts. Golden set, rubric, judge, gate — and the human review loop that keeps the judge honest. Useful, but the moment you're handed a real feature and a calendar invite that says "eval harness — kickoff", those parts don't tell you what to do first. Or second. Or how to defend any of it to a leadership team that just wants to hear "90% accuracy by Friday."
This lesson is the bridge. You'll sit next to Rohan, a PM at TaxSetu in Bangalore, and walk through every design choice for the eval harness on his AI tax-notice summarizer. Six decisions. Each one has a question Rohan must answer and an answer he picks — with the trade-off written down so you'd defend the same call in your next interview. By the end you'll have a working harness on paper, ready to hand to an engineer.
This is a mini-build, not a reference. It's longer than a normal lesson because each step is a real decision with real downside. Read it once with a coffee, screenshot the design-step blocks, and you have a checklist you can reuse for every AI feature you ship after this one.
One-line takeaway:
An eval harness is six product decisions in a trench coat. Not a Python file — a contract about what good means and how often you'll re-check it.
Scenario — Rohan has 90 minutes and a 90% target
🧾
Real-world scenario
TaxSetu Bangalore — summarize a tax notice into 3 plain-English bullets, leadership wants 90%
Rohan is the PM for TaxSetu, a Bangalore-based tax-filing app. Indian taxpayers receive intimidating notices from the Income Tax Department — five pages of legal language, section numbers, deadlines buried in paragraph six. TaxSetu's new feature takes a notice (uploaded photo or PDF), runs OCR, and returns a 3-bullet plain-English summary: "What it says · What you owe · What you must do by when." The bot has been demoed three times. Everyone loves it. Nobody can ship it.
Yesterday's leadership review gave Rohan one number: "90% accuracy or we don't launch." No definition of accuracy. No method for measuring it. He has 10 weeks until tax-filing season opens in Bangalore, and his next quarterly review is in 90 minutes. He needs a harness, not a research paper.
By the end of this lesson, Rohan walks out of that 90-minute meeting with a one-page harness design — six decisions, every one defensible, all sized to ship inside 10 weeks. We'll make those six calls together.
The six design choices — at a glance
Every eval harness is six bets a PM places, in order. The order matters. Skip ahead and you'll redesign three steps later when the answer to step 1 changes. Here are the six, with one line each. We'll spend the rest of the lesson making each call for Rohan.
🌾
1. Golden-set source
Where do real tax notices come from? Synthetic samples, prod uploads, or a partnered firm's archive.
📐
2. Golden-set size
How many notices in v1? The number that's stable enough to trust and cheap enough to refresh.
📊
3. Rubric dimensions
What does "accurate" mean? Faithfulness, completeness, clarity, brevity — each scored on a 1–5 scale.
⚖️
4. Judge type
LLM-judge, humans, or a hybrid? Cost vs scale vs the safety dimensions humans alone can grade.
🔬
5. Sample rate
What fraction goes to humans for audit? Enough to catch judge drift, few enough to fit a weekly budget.
🚨
6. Re-eval trigger
When does the harness rerun? Every PR, every model swap, monthly refresh, after every leadership ask.
Why these six and no others:
these are the only six knobs a PM owns. Engineers wire the pipeline; vendors swap the model; legal sets compliance. But who picks examples, defines good, picks the grader, sizes the audit, and decides re-run rules? That's product. Anything you can't influence isn't on the harness — it's a constraint you design around.
The harness as a pipeline
Once Rohan has made the six calls, his harness becomes a pipeline an engineer can wire in a day. Read it left to right — every box is a thing the harness does on every run.
Every box in the strip maps to one of the six choices. Source + Sample come from choices 1 and 2. Label and Rubric from choice 3 (someone has to author the ideal answers and the dimensions to score). Judge and Score come from choices 4 and 5. The gate decision wraps choice 6's re-run rules. We're going to walk through them now, in order.
How Rohan designs the harness — six steps in 90 minutes
Each step has the question Rohan asks himself, the option he picks, and a one-line reason a sceptical engineer would accept. Read each block as a unit — they're independent decisions, but the order is load-bearing.
Step 1 · Golden-set source
Where do the notices come from?
"Should I generate synthetic notices, scrape prod, or partner with a CA firm?"
Synthetic notices look uniform — the model will memorise the format and the eval will lie. Prod uploads are the real distribution but they're scarce in pre-launch and legally noisy (PII, consent). A partnered chartered-accountant firm in Bangalore has years of redacted notices already on file, with the original client questions already answered.
Rohan picks: partner with one CA firm in Bangalore for 250 redacted notices spanning the four most common section types (143(1), 139(9), 245, 156). He'll layer in real prod uploads after launch — but the v1 set is human-vetted and legal-clean.
Step 2 · Golden-set size
How many notices does v1 hold?
"50 to be quick, 200 to be stable, or 1000 to be thorough?"
Below 100 the pass rate jumps 4–6 points per run on noise alone. Above 500, every full eval run is its own meeting cost — and Rohan will need to refresh quarterly. The rule from Lesson 11 holds: 200 is the sweet spot for chat-style features. For a structured task like notice summarisation, where outputs are tightly bounded (3 bullets), 200 still works because the failure modes are concentrated.
Rohan picks: 200 notices in v1, stratified across the four section types (50 each). Compare-table below explains why he didn't go smaller or larger.
Step 3 · Rubric dimensions
What does "accurate" actually mean?
"Leadership said 90% accuracy. 90% of what?"
This is the call most PMs flunk. "Accuracy" is not a dimension — it's a wishful aggregate. Rohan unpacks it into four dimensions, each scored 1–5 with anchors. Faithfulness: does every claim in the 3 bullets appear in the source notice? Completeness: are the three load-bearing facts present (what · how much · by when)? Clarity: would a non-CA understand it on first read? Brevity: is each bullet ≤ 18 words and free of legalese? Pass = ≥ 4 on all four. He'll publish the anchors so two reviewers grade the same notice within 0.5 points.
Rohan picks: 4-dimension rubric (faithfulness, completeness, clarity, brevity), 1–5 with written anchors per grade. Headline pass rate = % of notices scoring ≥ 4 on every dimension. Faithfulness is the hard floor; a 3 there blocks the launch even if the other three are perfect.
Step 4 · Judge type
LLM, human, or hybrid?
"Can a model grade tax notices, or do I need a CA in the loop?"
Pure human grading: 200 notices × 4 dimensions × 6 minutes = 80 reviewer hours per run. Unfeasible weekly. Pure LLM-judge: cheap, fast, but for faithfulness it will sometimes flatter the bot — both models trained on the same notice corpus drift in the same direction. Hybrid wins: an LLM-judge (a different vendor than the bot — cross-family is the discipline from Lesson 11) grades all 200, a CA reviewer grades a 15% sample of 30 notices weekly, the harness flags any per-dimension gap > 0.7 points and forces recalibration.
Rohan picks: hybrid. LLM-judge runs every PR; CA reviewer audits 30 notices weekly with a calibration check on faithfulness specifically. The judge prompt is a PM artefact — Rohan will rewrite the rubric prose into the judge prompt himself.
Step 5 · Sample rate for human audit
How many go to the human?
"5%, 15%, or every notice on the safety dimension?"
The audit isn't there to grade everything — it's there to grade the grader. 15% (30 notices) gives Rohan a statistically meaningful gap signal between LLM and human within ±2.5 points. He layers one rule: every notice that scored ≤ 2 on faithfulness goes to a human regardless of the sample. Hallucinations on a tax notice are the kind of thing that triggers a tribunal. You don't sample those — you review every one.
Rohan picks: 15% random sample (30 notices) reviewed weekly + 100% of any notice the LLM-judge scored ≤ 2 on faithfulness. CA cost: ~₹4,000 per week — about ₹50k per quarter, which fits a Bangalore quality budget.
Step 6 · Re-eval trigger
When does the harness rerun?
"Daily? Every PR? Only when the model changes?"
Rohan defines five triggers. Every PR that touches the prompt, the OCR pipeline, or the post-processor — automatic, blocks merge if pass rate drops > 3 points. Every model swap (vendor pushed an update). Every retrieval change (new section types or year-over-year tax-rule shifts). Monthly golden-set refresh — add 20 new notices, retire the 20 oldest. Anytime leadership asks — the harness should be a one-command run, not a project.
Rohan picks: all five triggers, with a 3-point regression bar on PR runs and a 5-point bar on monthly. The on-call PM (himself) gets the Slack ping. There is no "re-eval is engineering's job" clause — that's how you wake up to a 13-point drop with no audit trail.
The 90-minute timeline
Rohan didn't pull these from thin air; he ran a 90-minute working session with his engineer and ops lead. Roughly: 10 min on source (step 1), 10 min on size (step 2), 25 min on the rubric (step 3 — always the longest), 15 min on judge type (step 4), 15 min on sample rate (step 5), 15 min on triggers (step 6). The next morning the engineer wires the pipeline. By Friday Rohan has v1 numbers on a real golden set.
Why 200 — the size trade-off
The most-asked question in any eval-harness review is "why that number?" Rohan keeps this table pinned in his PRD. Every number is illustrative for a 200-row chat/summarisation feature; adjust ±30% by domain.
Dimension
Small — 50 rows
Medium — 200 rows
Large — 1000 rows
Cost per full run
₹750 (LLM-judge)
₹3,000
₹15,000
Signal stability
Pass rate ±5 pts on noise — you'll argue every run
Pass rate ±2 pts — a 5-pt drop is real
Pass rate ±0.7 pts — clean signal but expensive
Refresh effort
1 hour/month — easy to keep current
4 hours/month — manageable
20+ hours/month — needs a dedicated annotator
Statistical power
Detects 12-point regressions only
Detects 5-point regressions reliably
Detects 2-point regressions
Human-audit cost (15%)
~₹1,000/week
~₹4,000/week
~₹20,000/week
Best for
Hackday prototypes, internal tools
Customer-facing v1 to v3
Mature features, regulated domains
What Rohan does with this: he writes the 200 number into his PRD with the row "signal stability ±2 pts" copied verbatim as the justification. When his VP asks "why not 1000?" he points at refresh effort. When the engineer asks "why not 50?" he points at signal stability. The table replaces six follow-up emails.
The 4-dimension rubric — what "good" actually means
Step 3 was the longest call for a reason — it's the part of the harness that turns a vague leadership ask ("90% accuracy") into something an LLM-judge or a CA reviewer can actually score. Below is the rubric Rohan ships. Each dimension is 1–5 with anchors; the dots show the threshold for "pass" on that row (≥ 4 = pass, all four required).
Dimension
What the grader checks
Pass threshold
Faithfulness
Every claim in the 3 bullets is supported by the source notice. No fabricated section numbers, dates, or rupee amounts. 5 = perfect, 3 = one paraphrase that subtly distorts, 1 = a fact that isn't in the notice at all.
≥ 4 required
Completeness
All three load-bearing facts present: what the notice says, what the taxpayer owes / is owed, what they must do by when. 5 = all three present and correct, 3 = one missing or vague, 1 = two or more missing.
≥ 4 required
Clarity
A non-CA could understand the bullets on first read. No untranslated section numbers, no Latin, no acronyms without expansion. 5 = grandparent-readable, 3 = needs one re-read, 1 = legalese.
≥ 4 required
Brevity
Each bullet ≤ 18 words, total ≤ 55 words. No filler ("As per the notice…"). 5 = within budget, well-shaped, 3 = over by 20–40%, 1 = essay-length.
≥ 4 required
Why four and not one:
a single "accuracy" score lets a fluent-but-wrong answer hide. With four dimensions, a notice that hallucinated one rupee figure scores 5/5/5/2 and gets blocked even if it reads beautifully. Faithfulness is the hard floor — Rohan writes that into the gate explicitly: headline pass rate = % of notices scoring ≥ 4 on all four dimensions, AND no notice scoring < 4 on faithfulness.
Rohan's first four prompt versions
The harness pays for itself the moment Rohan starts iterating. Across his first week of post-design work, he tries four prompt versions on the same 200-notice golden set. Same set, same rubric, same judge — only the prompt changes. The harness gives him a single comparable number per version. Without it, he'd be choosing between four prompts on vibes.
What Rohan ships: v4. He documents that the lift from v3 to v4 came almost entirely from completeness (more bullets correctly capture the deadline) — clarity and brevity were already at 4.6 and 4.8 averages by v3. That's the kind of breakdown only a multi-dimension rubric gives you. A single-number eval would have told him "v4 is better" without telling him why.
The minimal eval loop, adapted for a summarizer
You don't write this. Your engineer does. But you need to read it well enough to argue when the rubric, judge, or gate gets watered down in a sprint. This is the version of the Lesson 10 harness adapted for Rohan's 4-dimension rubric — about 30 lines a PM can defend in any meeting.
taxsetu_eval.pyRuns on every PR · pings Rohan on Slack
Three things to notice as the PM. One: the gate condition is Rohan's product call, not the engineer's — the all(... >= 4) + faithfulness floor lives in product code only because Rohan demanded it. Two: the judge is explicitly cross-vendor (Anthropic grades a bot built on a different family). Three: there is no fallback. If the JSON parse fails, the run fails — which is intentional. A silent skip is how lenient evals are born.
The AI connection
How real teams design eval harnesses
Production AI teams at Anthropic, OpenAI, and Indian fintechs run almost exactly Rohan's six-decision template. The vocabulary varies — some call faithfulness "groundedness," some call the audit "blind regrade" — but the shape is identical: a stratified golden set in the low hundreds, a multi-dimension rubric where one dimension is the safety floor, a cross-vendor LLM-judge, a 10–20% human audit, and a documented set of triggers. Frameworks like OpenAI's Evals, Promptfoo, and LangSmith all encode some subset of this loop. The PM's job isn't to pick the framework — it's to author the six choices the framework asks for.
golden-setrubricfaithfulnesscross-vendor-judgere-eval-triggerstratified-sample
Mini challenge — design a harness for your own feature
Pick any AI feature you've shipped or want to ship. Take 20 minutes. By the end you'll have a one-page harness design.
Run the six-step design on a feature you care about
1. Pick the feature and the leadership ask
One sentence each. "VidyaPath shows AI-explained answers to physics doubts. Leadership wants 95% explanations rated 'helpful' by students." Be specific — vague asks make vague harnesses.
2. Make calls 1, 2, and 3 in 10 minutes
Source (synthetic / prod / partner), size (50 / 200 / 1000), rubric dimensions (write 3–5 with one-line anchors). Don't optimise — pick a defensible default and move on. You can revise.
3. Make calls 4, 5, and 6 in 10 minutes
Judge type (LLM / human / hybrid), sample rate (% audit + any 100% rules), triggers (which 3–5 events rerun the harness). Write each as a single-sentence answer with a one-sentence reason. That's your harness PRD section.
Interview questions on this lesson
If a recruiter is hiring an AI PM, these are the questions about evals they actually ask. Read the answers as templates — adjust to your own feature.
EasyHow big should a golden set be?
+
For a customer-facing v1 of a chat or summarisation feature, 200 examples is the default — small enough to run on every PR for a few thousand rupees, large enough that the pass rate moves only ±2 points on noise. Below 100 you'll spend half your reviews arguing whether a 4-point drop is real. Above 500 the cost of refresh starts choking the cadence. For mature, regulated features you go to 1000+ to detect smaller regressions; for hackday prototypes 50 is fine. The rule isn't "more is better" — it's "smallest set that still gives you a stable signal," and that's around 200 for most consumer AI features.
EasyWhat's the difference between an eval and a unit test?
+
A unit test asserts equality: given input X, the output is exactly Y. An eval asserts a distribution: across a representative set of inputs, what fraction of outputs are acceptable? Unit tests give you pass/fail. Evals give you a pass rate, and you decide upfront what number is good enough to ship. AI features can't pass unit tests because the same input produces different outputs; they need evals because the question becomes "does it tend to do the right thing?" not "did it do exactly this?"
MediumFaithfulness vs hallucination — what's the metric?
+
Hallucination is the failure mode — the bot states a fact that isn't in the source. Faithfulness is the metric you measure to detect it. Practically, faithfulness is scored 1–5 against a single question: "is every claim in the output supported by the source material?" 5 = perfect grounding, 3 = one paraphrase that subtly distorts a fact, 1 = a claim with no source support. You measure faithfulness on every output in the golden set, and you make it a hard floor in the gate — for a tax summarizer, no notice can score below 4 on faithfulness even if every other dimension is perfect, because a hallucinated rupee figure can trigger a tribunal. Hallucination is what users feel; faithfulness is what PMs ship.
MediumWhen would you use a pure human judge instead of an LLM-judge?
+
Three situations. One: when the dimension you're grading isn't represented in the LLM-judge's training distribution — local-language tone in Swahili customer support, regional politeness markers in Tamil, regulatory nuance in DPDP-compliant outputs. Two: when the cost of a wrong grade is catastrophic — refusal correctness on a NairaSwift dispute bot, safety on a SouthBank India loan-eligibility refusal. Three: when you suspect judge-model drift and you've lost calibration. In all three, you bias toward humans. Otherwise, pay the LLM-judge for scale and audit it weekly with a human sample. Pure-human is rarely the right call for everything; it's the right call for one or two dimensions plus the audit.
HardRubric drift — how do you detect and fix it?
+
Rubric drift is when your written rubric stays the same but the way it gets applied silently changes — the LLM-judge's interpretation of "faithfulness" today isn't what it was three months ago, because the vendor pushed a model update or your reviewer pool turned over. You detect it three ways. One: the calibration gap — track the average difference between LLM-judge grades and human-audit grades each week; if the gap drifts beyond ±0.5 points on any dimension, drift is happening. Two: anchor-example regression — keep a fixed set of 10 "this is exactly a 5" and 10 "this is exactly a 2" examples; if the judge's grades on these stable anchors move month-over-month, the rubric isn't being applied the same way. Three: dimension correlation — if faithfulness and clarity grades suddenly become highly correlated (the judge is conflating them), the rubric needs sharper anchors. You fix it by rewriting the anchors with sharper one-line definitions, regrading the audit sample under the new anchors, and force-running calibration before the next launch. The hard part is that drift is invisible until you're already wrong; that's why the weekly audit and the anchor set exist as standing infrastructure, not one-time deliverables.
Quiz — 5 questions
Question 1 of 5
Leadership tells Rohan "we want 90% accuracy." What's the first thing he should do?
Question 2 of 5
Why is 200 the default golden-set size for a v1 customer-facing AI feature?
Question 3 of 5
Rohan's rubric has four dimensions, all scored 1–5. Why does he insist on a hard floor of ≥ 4 on faithfulness specifically?
Question 4 of 5
Rohan picks a 15% human-audit sample plus one extra rule. What's the rule and why?
Question 5 of 5
Across v1 → v2 → v3 → v4 of his prompt, Rohan's pass rate goes 71% → 79% → 86% → 92%. The gate is 90%. Which version ships, and what does the harness give him that a demo wouldn't?
0
/ 5
Harness designer mode: unlocked.
You can now sit in the room while the engineer wires this — and argue about the rubric, not just the model.
Common mistakes when designing your first harness
Letting "accuracy" stay one number
If leadership says 90% accuracy and you hand back a single-dimension grade, fluent-but-wrong outputs will hide. Always unpack into 3–5 named dimensions with anchors.
Synthetic golden set as v1
Generating 200 fake notices yourself feels productive. The model memorises your format, your eval reports 95%, prod traffic looks nothing like it, and you ship a regression.
Same vendor for bot and judge
A judge trained on the same family as the bot will flatter it on subtle errors. Always cross-vendor — bot on one family, judge on another, audited weekly.
No re-eval triggers in writing
"We'll rerun when something changes" isn't a trigger; it's a wish. Write the five events that force a rerun — every PR, every model swap, every retrieval change, monthly refresh, leadership ask.
Key takeaways
Six choices, in order. Source → size → rubric → judge → sample rate → re-eval triggers. Skip ahead and you'll redesign three steps later.
"Accuracy" is never the metric. Unpack it into 3–5 dimensions with anchors before you measure anything.
200 is the default size. Stable signal, manageable refresh, affordable audits. Adjust ±30% by domain, not by gut.
Faithfulness is the safety floor. Make it a hard gate condition, not just one of many dimensions averaged.
The harness is six product decisions in a trench coat. The Python is one day of engineering — the design is the PM's whole job.