What this capstone is

Capstone 1 was the easy one. You sat in Chidi's chair and wrote a PRD from scratch — eight clean sections, your prose, your call. This capstone is the one that decides whether you're actually senior. Today you don't write the harness. You review one a junior PM dropped on your desk, and you find the things that are quietly broken before they ship anything to a real declined sender in Lagos.

The skill being tested is design critique. Reviewing an eval harness is harder than writing one because the bugs don't compile. A leaky synthetic dataset looks fine. A 30-row golden set returns a number. An LLM-judge with no human calibration produces a clean dashboard. None of this throws an error — and that is exactly why a senior PM has to catch it. By the time users tweet, the harness has been confidently wrong for six weeks.

Today's draft was written by Funmi, Chidi's junior PM at NairaSwift. She's three months into her first AI-PM job, took the eval-harness mini-build seriously, and produced a draft that is genuinely good for a junior — but has seven issues a senior reviewer must catch in the first read. We'll walk through each, score them by severity, and rebuild the harness section by section.

How to read this lesson:
first pass — read Funmi's draft (§4), close the tab, and try to list the issues yourself. Second pass — open the issues list (§6) and see which you caught. The gap is your real growth area. The point of a capstone isn't getting the right answer; it's noticing where you stopped looking.

The six axes a senior reviews on

Every eval harness review follows the same checklist. Screenshot this for your next review session. If you can't fill in concrete evidence under each axis, you haven't actually reviewed the harness — you've nodded at the headline number.

📋
1. Golden-set integrity
Size, source, stratification, refresh. Synthetic data is leakage until proven otherwise.
📏
2. Rubric anchoring
Every grade has a written anchor. "Tone" without anchors is a vibe check, not a metric.
⚖️
3. Judge calibration
LLM-judge alone is lying to you. Weekly human sample is non-negotiable.
🚫
4. Refusal coverage
Does the set actually contain rows the bot must refuse? Most drafts forget this entirely.
🔬
5. Segment slicing
Headline pass-rate hides per-segment disasters. Slice by user-intent and failure-class.
📈
6. Regression tracking
A run with no history is a snapshot, not an eval. Trend lines per segment over time.
The senior-PM read:
the six axes are AND'd, not OR'd. A harness that scores 5/6 isn't 83% good — it's broken on the one axis that wasn't checked. Funmi's draft fails on five of the six. Your job over the next twenty minutes is to find them in the order a senior would.

The scenario — Friday afternoon, week 4

💳
Capstone scenario
Chidi at NairaSwift Lagos has 90 minutes to review his junior PM's draft eval harness for "explain-my-decline".

Same NairaSwift "explain-my-decline" feature you saw in Capstone 1. Chidi's PRD is locked, engineering is wiring the prompts, and the launch ladder says shadow runs in week 5 — meaning the eval harness has to be live and trusted by Wednesday. Three weeks ago Chidi delegated the harness draft to Funmi, his junior PM (3 months in, sharp, eager, hasn't yet shipped an AI feature solo). Funmi's draft landed in Notion this morning. The doc is well-formatted, the prose is confident, and the headline pass-rate on the dummy run was 89% — above the 84% gate.

Chidi reads it once at 2pm and the back of his neck goes warm. The number is too clean. The 30-row golden set is described in two sentences. The "tone" rubric is the words "tone: 1–5". There is no mention of the fraud-trip refusal class. There is no comparison to last week. The judge is a single LLM-as-judge call with no human sample. By 2:15 he has a list of seven issues across the six axes — five P0/P1, two P2 — and he books a 30-minute walkthrough with Funmi for Monday morning. This lesson is that walkthrough.

Your job: sit in Chidi's seat. Read the draft. Find the issues before §6 spoils them. Score them. Then watch the harness get rebuilt section by section in §7. By the end you will have done a senior-PM review without writing a single line of code.

Why critique, not creation: in your first AI-PM job you will spend more time reviewing other people's harnesses (engineers', juniors', vendors') than writing your own. The job is to spot the silent failure before the dashboard hides it. That's the muscle this capstone trains.

Funmi's draft eval harness v0.1 · for review

Below is Funmi's draft as it appeared in Notion this morning. Read it once, slowly. Mentally tag each row with what's missing or wrong before scrolling down. The seven issues are all hiding in plain sight — none of them require knowing what model NairaSwift is on or which library Funmi is using. They're all PM-shaped issues.

Notion · NairaSwift · Risk & Trust · Funmi (junior PM)
Eval harness v0.1 — Explain-My-Decline (DRAFT)

Author: Funmi · Reviewer: Chidi · Status: Draft · Headline result: 89% pass on dummy run · Gate target: ≥ 84%

1
Golden set
30 examples. I drafted them by writing realistic-looking decline prompts in a Google Doc and then asking the same gpt-4o-mini we'll ship to generate the "ideal" answer for each. Then I corrected obvious typos. Quick to build, covers the 5 named decline codes from the PRD.
2
Rubric
Two axes: code-correctness (1–5) and tone (1–5). Code-correctness has anchors per grade based on whether the right decline code is named. Tone is graded 1–5 by the judge "based on whether it sounds calm and helpful". Pass = both axes ≥ 4.
3
Judge
Single LLM-as-judge: gpt-4o, given the rubric and the bot's answer, returns a JSON grade. No human spot-check planned in v0.1 — we'll add one later if time permits. Cheap, fast, runs on every PR in CI.
4
Run cadence & reporting
Run on demand or on every PR. Output a single headline pass-rate posted to Slack. Latest run: 89% pass — clears the 84% gate. No history kept across runs (we just look at the latest).
5
Coverage
All 5 in-scope decline codes from PRD §2 are represented (6 examples each). I haven't added the fraud-trip refusal class yet because the prompt already returns a refusal string and there's no user-visible LLM output to grade.
Pause here: close your laptop lid for two minutes. Try to list 5+ things wrong with this draft. Chidi found seven. The scoring rubric for this capstone is honest with yourself about which ones you'd have missed.

The review flow — six steps, ninety minutes

Before we list the issues, here's the choreography of the actual review. This is the cadence Chidi runs every time a junior or an engineer hands him a draft harness. It's the same shape for a one-page draft or a 20-page one — the time per step changes, the steps don't.

1. Receive draft do not skim block 90 min 2. Read end-to-end no notes yet trust your gut 3. Find issues six-axis check tag P0 / P1 / P2 4. Discuss walk through one row each 5. Fix list P0/P1 in this sprint 6. Re-eval & trust REVIEW IS A LOOP, NOT A GATE — SHIP THE FIXED HARNESS, KEEP REVIEWING

Step 3 is where the lesson lives. The rest is choreography you can copy from any senior PM. Steps 4–5 only work if step 3 is honest. Below is what step 3 produces when Chidi sits down with Funmi's draft on Monday morning.

Seven issues, scored and explained

Here are the seven issues Chidi flags, in the order a senior would surface them in a review. Severity uses the same P0/P1/P2 scale as engineering bug triage — P0 means the harness will mislead the launch decision, P1 means the harness will miss real regressions, P2 means the harness is correct but fragile. Three are P0, three are P1, one is P2.

1
Golden set is too small (30 rows)
P1 · trust

What's wrong

30 rows is below the noise floor. Run the same eval twice and the pass rate moves 5–8 points on luck alone — meaning the 89% headline could just as easily be 81% next Tuesday. You cannot defend a 5-point delta in a launch review when the run-to-run variance is bigger than the delta.

How to fix

Grow to 250 rows minimum, sourced from real production decline events from the previous quarter. Lock the size for v1.0 — never debate "is it big enough" mid-launch. Below 100 rows, the harness measures noise. 200–300 is the sweet spot for a chat-style explainer.

2
Synthetic data is leaky (model graded by itself)
P0 · invalid

What's wrong

Funmi used gpt-4o-mini (the production model) to generate the "ideal" answers, then graded the production model's outputs against those same ideal answers. This is the textbook leakage trap: the model is being graded on how well it matches itself. The 89% says nothing about whether real declined senders will be helped.

How to fix

Replace synthetic ideals with human-written ideal answers from CX agents who currently handle these tickets. Sample 250 real declines from prod logs, hand them to two ops leads, have each one write the ideal explanation independently, then reconcile. Slow, expensive, the only way the eval can ever be trusted.

3
Single LLM-judge with no human calibration
P0 · invalid

What's wrong

An LLM-judge is systematically lenient on subjective rows and silently drifts every time the vendor pushes a model update. Funmi's plan ("we'll add humans later if time permits") is the line that ends careers. Without a weekly human sample, the dashboard becomes confidently wrong somewhere between weeks 4 and 8.

How to fix

Add a weekly human spot-check: 20% random sample (50 rows of 250) re-graded by a CX lead, plus 100% re-grading of the refusal class. If LLM-judge vs human agreement falls below 85%, the harness is paused, not the launch — but you only know if the audit exists.

4
"Tone" rubric has no written anchors
P1 · vibe

What's wrong

"Tone 1–5 based on whether it sounds calm and helpful" is a vibe check, not a rubric. Two graders (LLM or human) reading the same answer will diverge by 1–2 points. The 89% becomes uninterpretable — you can't argue with the grade because there's nothing concrete to argue against.

How to fix

Write one-line anchors per grade: 5 = calm + factual + no apology before facts + matches sender's language; 4 = ≤1 violation; 3 = 2 violations; 2 = ≥3 or apologises before stating cause; 1 = blames sender or escalates anxiety. Force the judge to return the violation list alongside the grade.

5
No refusal coverage in the golden set
P0 · safety

What's wrong

Funmi excluded the fraud-trip-high-confidence class because "the prompt already returns a refusal string". That logic is exactly backwards. Refusals are the highest-stakes outputs — a wrong-tone or fact-leaking refusal is a regulatory event in fintech. An eval that doesn't grade refusals doesn't grade the riskiest 10% of traffic.

How to fix

Add a dedicated refusal subset (40 rows) covering high-confidence fraud trips, KYC-change requests, and any out-of-scope intents from PRD §2. Rubric checks: (a) did it refuse, (b) did it use the verbatim refusal string, (c) did it leak any allow-list data. Subset must score 100% — zero tolerance, AND'd with the headline gate.

6
No segment slicing (declined-by-bank vs declined-by-fraud)
P1 · blind

What's wrong

The 89% headline is the average across all decline codes. NairaSwift's traffic is dominated by "insufficient funds" and "FX cap" — together 70% of declines. If the bot is great on those and 60% on fraud-trip-low-confidence, the headline still clears the gate while the highest-cost segment quietly fails. Headline pass-rate without segment slicing is a confidence trap.

How to fix

Slice by 5 decline-code segments × 2 cause types (declined-by-bank vs declined-by-fraud). Report a per-segment pass-rate alongside the headline. Set a per-segment minimum — fraud-cause segments require ≥ 90% pass to launch, not 84%. Headline is a courtesy; the segments are the gate.

7
No regression tracking (no run history)
P2 · fragile

What's wrong

"We just look at the latest run" means a 4-point drop next Wednesday is invisible unless someone remembers what last Wednesday's number was. Without a stored run history, you cannot tell whether a prompt edit improved or regressed quality — and you definitely cannot tell whether the vendor pushed a silent model update.

How to fix

Persist every run row-by-row to a run-history table (run id, model SHA, prompt SHA, per-segment pass-rates, judge model, judge agreement). Trend the headline AND each segment over the last 8 runs. Auto-flag any segment that drops > 3 points run-over-run. P2 because the harness is correct without it — just fragile to silent drift.

The senior pattern:
notice that none of the seven issues required reading any code. They are all PM-shaped — set source, rubric anchors, judge calibration, refusal coverage, segment slicing, regression history. This is the discipline. An eval review is a documentation review, performed with the same eye that drafted the PRD.

What the fix looks like — issue × severity × fix complexity

One table, seven rows. This is what Chidi paste-shares to Funmi after the walkthrough. Severity tells her what to fix first. Fix complexity tells her what to fight engineering for. The "why critical" column is the one she's expected to defend in the launch review on Friday.

# Issue Severity Why critical Fix complexity
1 Golden set too small (30) P1 Run-to-run noise > expected effect size; deltas can't be defended Medium — 2 weeks of CX-lead labelling time
2 Leaky synthetic ideals (model grades self) P0 Grade is on intra-model match, not user helpfulness — invalidates 89% High — full re-source from prod, two human authors
3 Single LLM-judge, no human P0 Lenient by default + drifts on vendor updates → silent regression Low — schedule weekly 50-row review with CX lead
4 "Tone" rubric is a vibe check P1 Two graders disagree by 1–2 points; grade is not interpretable Low — write 5 anchor lines + violation list format
5 No refusal coverage P0 Highest-stakes traffic ungraded; fraud-trip wrong-tone = T&S incident Medium — author 40-row refusal subset + 100% gate
6 No segment slicing P1 Headline hides per-segment failure on highest-cost decline classes Low — per-segment pass-rate, per-segment floor
7 No regression tracking P2 Drift becomes invisible — vendor updates and prompt regressions hide Low — persist run rows + 8-run trend chart

What the fixes do to the signal

Below is the illustrative before/after when Funmi reruns the harness with all seven fixes applied. Two numbers move: the false-pass rate (how often the dashboard reports a passing run when humans would have failed it) and the segment coverage (how many distinct user-intent slices the harness reports separately). The headline pass-rate goes down, which is exactly what a senior wants to see — the harness is now telling the truth.

Funmi's harness — before vs after the 7 fixes False-pass rate (audit disagreed with judge) Before 22% After 4% Segment coverage (slices reported separately) Before 1 segment After 5 segments
How to read this chart in a launch review: false-pass rate dropping from 22% to 4% means the harness now agrees with humans on ~96% of grades — above the 85% trust threshold. Segment coverage going from 1 to 5 means a fraud-cause regression no longer hides inside the headline. Both numbers are the harness, not the bot. The bot itself didn't change.

The harness, before and after — side by side

Optional code block. This is the only block in the lesson — the broken eval-runner pseudocode on the left and the fixed version on the right. PMs don't write this; PMs review it. The point is the structural diff, not the syntax.

v0.1 (Funmi) — broken

  • load 30 synthetic rows from notion
  • for each row → call gpt-4o-mini → store output
  • for each output → call gpt-4o judge with rubric
  • parse {grade: int, tone: int}
  • pass = grade ≥ 4 AND tone ≥ 4
  • print headline = passes / total
  • post number to slack — done

v1.0 (after fixes) — trustworthy

  • load 250 prod rows + 40 refusal rows from labelled set
  • tag each row with segment (code × cause)
  • call production prompt → store output + model SHA
  • call judge with anchored rubric → grade + violation list
  • once a week: human re-grades 50 random + all 40 refusals
  • compute per-segment pass + headline + judge-vs-human agreement
  • append row to run-history table; trend last 8 runs
  • gate = headline ≥ 84% AND every segment ≥ floor AND refusals = 100%
eval_harness_v1.py — pseudocode, for review only
Why one code block, not two:
the broken version is in the side-by-side panel above as a bullet list — the structural shape is what matters, not the bad syntax. Showing the broken Python verbatim teaches the wrong thing. PMs read code for structural review; they don't critique line-by-line.

The AI connection

How real AI teams run eval reviews

Every serious AI org — from BoraPay's risk team in Nairobi to the AI cores inside global banks — runs the same review cadence on every harness landing in their codebase. Six axes (set, rubric, judge, refusal, slicing, regression). Severity tagged at P0/P1/P2. The reviewer is always a senior PM or staff engineer, never the author. The fix list lives in the harness's own README, version-controlled with the run history. A harness without a review log is a harness that hasn't been reviewed. eval-review segment-slicing judge-calibration refusal-coverage regression-tracking data-leakage

Mini challenge — review a harness in 30 minutes

Run the six-axis review on a real harness

1. Print Funmi's draft and red-pen it

Open §4 of this lesson on a screen, draft on one side, blank Notion page on the other. Without scrolling to §6, list every concern you'd raise. Tag each P0 / P1 / P2. Compare to Chidi's seven. The ones you missed are your real growth areas — write them down somewhere you'll see again.

2. Score a harness from your own work

Take any AI feature you've shipped or shadowed. Open whatever passes for its eval (a Google sheet, a CI script, a vibe check). Score it on the six axes — set integrity, rubric anchoring, judge calibration, refusal coverage, segment slicing, regression tracking. If you can't fill all six, the harness is broken; the next sentence is the fix list.

3. Write a one-page review note

For the harness you just scored, write a one-page review note in Notion. Three sections: headline severity (one sentence), the issues table (issue × severity × why critical × fix complexity), the fix sprint (which two issues land this week). Send it to whoever owns the harness with a "30 min Monday morning?" calendar invite. That is the review loop.

Interview questions

Easy What's segment slicing in evals?
+
Segment slicing is reporting your eval pass-rate not just as one headline number but broken down by user-intent or failure-class. For NairaSwift's "explain-my-decline" you'd slice by decline code (insufficient funds, FX cap, fraud-trip-low, KYC limit, batch closed) and by cause type (declined-by-bank vs declined-by-fraud). Without slicing, the headline is a weighted average dominated by your most common segment — meaning a 10-point failure on a rare-but-high-stakes segment hides under a passing headline. The senior-PM rule: headline is a courtesy, segments are the gate. Set a per-segment minimum, gate the launch on every segment clearing it, and the headline becomes a redundant summary of the truth, not a comfort blanket over it.
Easy How do you give junior-PM eval feedback?
+
Three rules. One: lead with what the draft does well — a junior who fears the review will write defensive harnesses, and defensive harnesses hide failures. Two: walk one issue at a time, not a list dump — humans absorb fix-by-fix, not bulk. Three: separate severity from blame. P0 means "this will mislead the launch", not "you screwed up". Use the same six-axis checklist every time so the junior internalises the framework, not just this one fix. By the third or fourth review they should be self-flagging issues before you do — that's the real KPI of how good your reviews are.
Medium A draft eval harness reports 89% pass on a 30-row golden set. What's your first reaction?
+
Healthy scepticism, in this exact order. First, where did the rows come from? 30 hand-typed rows is a noise generator — the same eval rerun next Tuesday could land at 81% on luck alone. Second, where did the ideal answers come from? If the same model that's being graded also wrote the ideals, the 89% is intra-model self-agreement, not helpfulness. Third, how is the judge calibrated? If there's no human spot-check, the 89% might be the judge's lenient distribution, not the bot's quality. Fourth, what segments? An average of 89% across 5 decline codes might hide a 60% on the riskiest one. The number itself is irrelevant until those four questions have crisp answers. Senior PM move: in the review meeting you write "89%" on the whiteboard and circle the four question marks around it — then start working clockwise.
Medium Why is excluding refusals from the eval the most dangerous shortcut?
+
Because refusals are the highest-cost output class and the lowest-volume one. A wrong-tone refusal in a fraud-trip flow can produce a Trust & Safety incident, a regulatory question, or a public escalation — outcomes that dwarf the cost of a slightly-off explanation in the bulk traffic. Funmi's logic ("the prompt already returns a refusal string") is exactly backwards: the fact that there's a hardcoded refusal is what makes it eval-able — you check (a) was the refusal triggered when it should have been, (b) was it the verbatim string with no addition, (c) did it leak any allow-list data into the surrounding prose. Without those three checks in the harness, a single prompt edit can quietly start mixing decline reasoning into refusals and you'd never know until a regulator asked. Refusals get a dedicated subset, a 100% gate, and weekly human review — non-negotiable on a fintech harness.
Hard A junior PM disagrees with your top issue. They have data. What do you do?
+
First, take the disagreement seriously — not as politeness, as practice. The whole point of training a junior is that they eventually catch things you miss; a review where the junior never wins is a review that's not actually working. So: hour zero, ask them to walk you through the data. What exactly are they measuring, on which rows, against which baseline? In Funmi's case, suppose she pushes back on issue #2 (leaky synthetic ideals) because she ran a small experiment showing the model-generated ideals match human-written ideals 87% of the time. That's a real data point. Now you have three branches. Branch one: she's right, the leakage matters less for this domain than the priors say — you adjust the issue from P0 to P1 and document why in the harness README so the next reviewer doesn't re-fight it. Branch two: she's right on the 87% number but wrong on the inference — model-generated and human-generated ideals can be 87% identical and still produce a systematically wrong eval signal because the 13% disagreement clusters on the highest-stakes rows; in that case you sit with her and trace the disagreement on five specific rows. Branch three: the experiment itself has a flaw — small sample, biased subset, ideal-vs-ideal agreement isn't the same as bot-vs-ideal agreement. In all three cases, the tone is "let's get to the truth", not "I'm the senior". The thing you must not do is overrule on hierarchy ("we're going with the standard practice") — that breaks the review loop forever. The junior either changes their mind because the evidence flipped, or you change yours because their evidence is real, or both of you write a one-paragraph "we deferred this" note and move on. The harness gets better either way; the relationship survives; the next review is sharper. The hard thing about being a senior reviewer isn't being right more often — it's being correctable in public.

Test your knowledge

Question 1 of 5
Funmi's draft uses gpt-4o-mini to generate the "ideal" answers, then grades the production model (also gpt-4o-mini) against those ideals. What's the senior-PM read?
Question 2 of 5
Funmi excluded the fraud-trip refusal class because "the prompt already returns a refusal string". Why is this exactly the wrong call?
Question 3 of 5
The headline is 89% across 5 decline codes. Why is the senior PM still uneasy without a per-segment slice?
Question 4 of 5
Funmi defers human review with "we'll add humans later if time permits". What's the senior-PM response?
Question 5 of 5
A junior pushes back on your top issue with real data. What's the wrong move?
0
/5
You can run a senior-PM eval review.

Common mistakes when reviewing an eval harness

Reviewing only the headline number
If your review starts and ends with "89% — clears 84%, ship it", you're not reviewing the harness; you're rubber-stamping it. The number is the last thing you check, not the first.
Trusting any synthetic dataset
Synthetic golden sets and synthetic ideals are leakage until proven otherwise. If the same model is on both sides of the grading, you're measuring intra-model agreement, not user helpfulness.
Deferring human review to "later"
"We'll add humans if time permits" is the line that ends careers. Without a weekly spot-check, the LLM-judge drifts silently and the dashboard becomes confidently wrong somewhere between weeks 4 and 8.
Letting hierarchy win the review
Overruling a junior with "we're going with standard practice" instead of working through their data breaks the review loop forever. The next harness will be defensive, not better.

Key takeaways

  • Eval review is a six-axis checklist — set integrity, rubric anchoring, judge calibration, refusal coverage, segment slicing, regression tracking. AND'd, not OR'd. 5/6 is broken on 1.
  • Synthetic ideals graded by the production model are P0 leakage. Human-written ideals from CX agents are the only trustworthy source. Slow, expensive, the only honest path.
  • An LLM-judge alone is lying to you. Schedule the weekly 20% human sample before week one of shadow, not after the first regression scare.
  • Refusals get their own subset and a 100% gate. The fact that the prompt hardcodes the refusal is what makes it eval-able, not what makes it skippable.
  • Headline is a courtesy, segments are the gate. Per-segment minimums on highest-cost slices are AND'd with the headline. A weighted average will hide a high-cost failure every time.
  • Persist run history or you're guessing. A run with no comparison is a snapshot, not an eval. Trend per-segment over the last 8 runs and auto-flag drops > 3 points.
  • Reviews are loops, not gates. Lead with what's good, walk one issue at a time, separate severity from blame, and stay correctable in public — that's how juniors become seniors.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You now review evals like a senior.
Pick up the cheatsheet — and ship your first AI feature.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →