What this capstone is
Capstone 1 was the easy one. You sat in Chidi's chair and wrote a PRD from scratch — eight clean sections, your prose, your call. This capstone is the one that decides whether you're actually senior. Today you don't write the harness. You review one a junior PM dropped on your desk, and you find the things that are quietly broken before they ship anything to a real declined sender in Lagos.
The skill being tested is design critique. Reviewing an eval harness is harder than writing one because the bugs don't compile. A leaky synthetic dataset looks fine. A 30-row golden set returns a number. An LLM-judge with no human calibration produces a clean dashboard. None of this throws an error — and that is exactly why a senior PM has to catch it. By the time users tweet, the harness has been confidently wrong for six weeks.
Today's draft was written by Funmi, Chidi's junior PM at NairaSwift. She's three months into her first AI-PM job, took the eval-harness mini-build seriously, and produced a draft that is genuinely good for a junior — but has seven issues a senior reviewer must catch in the first read. We'll walk through each, score them by severity, and rebuild the harness section by section.
The six axes a senior reviews on
Every eval harness review follows the same checklist. Screenshot this for your next review session. If you can't fill in concrete evidence under each axis, you haven't actually reviewed the harness — you've nodded at the headline number.
The scenario — Friday afternoon, week 4
Same NairaSwift "explain-my-decline" feature you saw in Capstone 1. Chidi's PRD is locked, engineering is wiring the prompts, and the launch ladder says shadow runs in week 5 — meaning the eval harness has to be live and trusted by Wednesday. Three weeks ago Chidi delegated the harness draft to Funmi, his junior PM (3 months in, sharp, eager, hasn't yet shipped an AI feature solo). Funmi's draft landed in Notion this morning. The doc is well-formatted, the prose is confident, and the headline pass-rate on the dummy run was 89% — above the 84% gate.
Chidi reads it once at 2pm and the back of his neck goes warm. The number is too clean. The 30-row golden set is described in two sentences. The "tone" rubric is the words "tone: 1–5". There is no mention of the fraud-trip refusal class. There is no comparison to last week. The judge is a single LLM-as-judge call with no human sample. By 2:15 he has a list of seven issues across the six axes — five P0/P1, two P2 — and he books a 30-minute walkthrough with Funmi for Monday morning. This lesson is that walkthrough.
Your job: sit in Chidi's seat. Read the draft. Find the issues before §6 spoils them. Score them. Then watch the harness get rebuilt section by section in §7. By the end you will have done a senior-PM review without writing a single line of code.
Funmi's draft eval harness v0.1 · for review
Below is Funmi's draft as it appeared in Notion this morning. Read it once, slowly. Mentally tag each row with what's missing or wrong before scrolling down. The seven issues are all hiding in plain sight — none of them require knowing what model NairaSwift is on or which library Funmi is using. They're all PM-shaped issues.
Author: Funmi · Reviewer: Chidi · Status: Draft · Headline result: 89% pass on dummy run · Gate target: ≥ 84%
The review flow — six steps, ninety minutes
Before we list the issues, here's the choreography of the actual review. This is the cadence Chidi runs every time a junior or an engineer hands him a draft harness. It's the same shape for a one-page draft or a 20-page one — the time per step changes, the steps don't.
Step 3 is where the lesson lives. The rest is choreography you can copy from any senior PM. Steps 4–5 only work if step 3 is honest. Below is what step 3 produces when Chidi sits down with Funmi's draft on Monday morning.
Seven issues, scored and explained
Here are the seven issues Chidi flags, in the order a senior would surface them in a review. Severity uses the same P0/P1/P2 scale as engineering bug triage — P0 means the harness will mislead the launch decision, P1 means the harness will miss real regressions, P2 means the harness is correct but fragile. Three are P0, three are P1, one is P2.
What's wrong
30 rows is below the noise floor. Run the same eval twice and the pass rate moves 5–8 points on luck alone — meaning the 89% headline could just as easily be 81% next Tuesday. You cannot defend a 5-point delta in a launch review when the run-to-run variance is bigger than the delta.
How to fix
Grow to 250 rows minimum, sourced from real production decline events from the previous quarter. Lock the size for v1.0 — never debate "is it big enough" mid-launch. Below 100 rows, the harness measures noise. 200–300 is the sweet spot for a chat-style explainer.
What's wrong
Funmi used gpt-4o-mini (the production model) to generate the "ideal" answers, then graded the production model's outputs against those same ideal answers. This is the textbook leakage trap: the model is being graded on how well it matches itself. The 89% says nothing about whether real declined senders will be helped.
How to fix
Replace synthetic ideals with human-written ideal answers from CX agents who currently handle these tickets. Sample 250 real declines from prod logs, hand them to two ops leads, have each one write the ideal explanation independently, then reconcile. Slow, expensive, the only way the eval can ever be trusted.
What's wrong
An LLM-judge is systematically lenient on subjective rows and silently drifts every time the vendor pushes a model update. Funmi's plan ("we'll add humans later if time permits") is the line that ends careers. Without a weekly human sample, the dashboard becomes confidently wrong somewhere between weeks 4 and 8.
How to fix
Add a weekly human spot-check: 20% random sample (50 rows of 250) re-graded by a CX lead, plus 100% re-grading of the refusal class. If LLM-judge vs human agreement falls below 85%, the harness is paused, not the launch — but you only know if the audit exists.
What's wrong
"Tone 1–5 based on whether it sounds calm and helpful" is a vibe check, not a rubric. Two graders (LLM or human) reading the same answer will diverge by 1–2 points. The 89% becomes uninterpretable — you can't argue with the grade because there's nothing concrete to argue against.
How to fix
Write one-line anchors per grade: 5 = calm + factual + no apology before facts + matches sender's language; 4 = ≤1 violation; 3 = 2 violations; 2 = ≥3 or apologises before stating cause; 1 = blames sender or escalates anxiety. Force the judge to return the violation list alongside the grade.
What's wrong
Funmi excluded the fraud-trip-high-confidence class because "the prompt already returns a refusal string". That logic is exactly backwards. Refusals are the highest-stakes outputs — a wrong-tone or fact-leaking refusal is a regulatory event in fintech. An eval that doesn't grade refusals doesn't grade the riskiest 10% of traffic.
How to fix
Add a dedicated refusal subset (40 rows) covering high-confidence fraud trips, KYC-change requests, and any out-of-scope intents from PRD §2. Rubric checks: (a) did it refuse, (b) did it use the verbatim refusal string, (c) did it leak any allow-list data. Subset must score 100% — zero tolerance, AND'd with the headline gate.
What's wrong
The 89% headline is the average across all decline codes. NairaSwift's traffic is dominated by "insufficient funds" and "FX cap" — together 70% of declines. If the bot is great on those and 60% on fraud-trip-low-confidence, the headline still clears the gate while the highest-cost segment quietly fails. Headline pass-rate without segment slicing is a confidence trap.
How to fix
Slice by 5 decline-code segments × 2 cause types (declined-by-bank vs declined-by-fraud). Report a per-segment pass-rate alongside the headline. Set a per-segment minimum — fraud-cause segments require ≥ 90% pass to launch, not 84%. Headline is a courtesy; the segments are the gate.
What's wrong
"We just look at the latest run" means a 4-point drop next Wednesday is invisible unless someone remembers what last Wednesday's number was. Without a stored run history, you cannot tell whether a prompt edit improved or regressed quality — and you definitely cannot tell whether the vendor pushed a silent model update.
How to fix
Persist every run row-by-row to a run-history table (run id, model SHA, prompt SHA, per-segment pass-rates, judge model, judge agreement). Trend the headline AND each segment over the last 8 runs. Auto-flag any segment that drops > 3 points run-over-run. P2 because the harness is correct without it — just fragile to silent drift.
What the fix looks like — issue × severity × fix complexity
One table, seven rows. This is what Chidi paste-shares to Funmi after the walkthrough. Severity tells her what to fix first. Fix complexity tells her what to fight engineering for. The "why critical" column is the one she's expected to defend in the launch review on Friday.
| # | Issue | Severity | Why critical | Fix complexity |
|---|---|---|---|---|
| 1 | Golden set too small (30) | P1 | Run-to-run noise > expected effect size; deltas can't be defended | Medium — 2 weeks of CX-lead labelling time |
| 2 | Leaky synthetic ideals (model grades self) | P0 | Grade is on intra-model match, not user helpfulness — invalidates 89% | High — full re-source from prod, two human authors |
| 3 | Single LLM-judge, no human | P0 | Lenient by default + drifts on vendor updates → silent regression | Low — schedule weekly 50-row review with CX lead |
| 4 | "Tone" rubric is a vibe check | P1 | Two graders disagree by 1–2 points; grade is not interpretable | Low — write 5 anchor lines + violation list format |
| 5 | No refusal coverage | P0 | Highest-stakes traffic ungraded; fraud-trip wrong-tone = T&S incident | Medium — author 40-row refusal subset + 100% gate |
| 6 | No segment slicing | P1 | Headline hides per-segment failure on highest-cost decline classes | Low — per-segment pass-rate, per-segment floor |
| 7 | No regression tracking | P2 | Drift becomes invisible — vendor updates and prompt regressions hide | Low — persist run rows + 8-run trend chart |
What the fixes do to the signal
Below is the illustrative before/after when Funmi reruns the harness with all seven fixes applied. Two numbers move: the false-pass rate (how often the dashboard reports a passing run when humans would have failed it) and the segment coverage (how many distinct user-intent slices the harness reports separately). The headline pass-rate goes down, which is exactly what a senior wants to see — the harness is now telling the truth.
The harness, before and after — side by side
Optional code block. This is the only block in the lesson — the broken eval-runner pseudocode on the left and the fixed version on the right. PMs don't write this; PMs review it. The point is the structural diff, not the syntax.
v0.1 (Funmi) — broken
- load 30 synthetic rows from notion
- for each row → call gpt-4o-mini → store output
- for each output → call gpt-4o judge with rubric
- parse {grade: int, tone: int}
- pass = grade ≥ 4 AND tone ≥ 4
- print headline = passes / total
- post number to slack — done
v1.0 (after fixes) — trustworthy
- load 250 prod rows + 40 refusal rows from labelled set
- tag each row with segment (code × cause)
- call production prompt → store output + model SHA
- call judge with anchored rubric → grade + violation list
- once a week: human re-grades 50 random + all 40 refusals
- compute per-segment pass + headline + judge-vs-human agreement
- append row to run-history table; trend last 8 runs
- gate = headline ≥ 84% AND every segment ≥ floor AND refusals = 100%
The AI connection
How real AI teams run eval reviews
Every serious AI org — from BoraPay's risk team in Nairobi to the AI cores inside global banks — runs the same review cadence on every harness landing in their codebase. Six axes (set, rubric, judge, refusal, slicing, regression). Severity tagged at P0/P1/P2. The reviewer is always a senior PM or staff engineer, never the author. The fix list lives in the harness's own README, version-controlled with the run history. A harness without a review log is a harness that hasn't been reviewed. eval-review segment-slicing judge-calibration refusal-coverage regression-tracking data-leakage
Mini challenge — review a harness in 30 minutes
Run the six-axis review on a real harness
Open §4 of this lesson on a screen, draft on one side, blank Notion page on the other. Without scrolling to §6, list every concern you'd raise. Tag each P0 / P1 / P2. Compare to Chidi's seven. The ones you missed are your real growth areas — write them down somewhere you'll see again.
Take any AI feature you've shipped or shadowed. Open whatever passes for its eval (a Google sheet, a CI script, a vibe check). Score it on the six axes — set integrity, rubric anchoring, judge calibration, refusal coverage, segment slicing, regression tracking. If you can't fill all six, the harness is broken; the next sentence is the fix list.
For the harness you just scored, write a one-page review note in Notion. Three sections: headline severity (one sentence), the issues table (issue × severity × why critical × fix complexity), the fix sprint (which two issues land this week). Send it to whoever owns the harness with a "30 min Monday morning?" calendar invite. That is the review loop.
Interview questions
Test your knowledge
Common mistakes when reviewing an eval harness
Key takeaways
- Eval review is a six-axis checklist — set integrity, rubric anchoring, judge calibration, refusal coverage, segment slicing, regression tracking. AND'd, not OR'd. 5/6 is broken on 1.
- Synthetic ideals graded by the production model are P0 leakage. Human-written ideals from CX agents are the only trustworthy source. Slow, expensive, the only honest path.
- An LLM-judge alone is lying to you. Schedule the weekly 20% human sample before week one of shadow, not after the first regression scare.
- Refusals get their own subset and a 100% gate. The fact that the prompt hardcodes the refusal is what makes it eval-able, not what makes it skippable.
- Headline is a courtesy, segments are the gate. Per-segment minimums on highest-cost slices are AND'd with the headline. A weighted average will hide a high-cost failure every time.
- Persist run history or you're guessing. A run with no comparison is a snapshot, not an eval. Trend per-segment over the last 8 runs and auto-flag drops > 3 points.
- Reviews are loops, not gates. Lead with what's good, walk one issue at a time, separate severity from blame, and stay correctable in public — that's how juniors become seniors.
You now review evals like a senior.
Pick up the cheatsheet — and ship your first AI feature.
Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Agentic Engineer →