Why use this

This is the page you screenshot, pin to your second monitor, and reach for the next time someone says "can we ship an AI feature by Q3?" It is the working PM's compression of the 22 lessons in the track — the diagrams, the tables, the rules — laid out so you can scan, not read.

Use it during PRD reviews to remember which sections must exist. Use it during eval design to remember what a good harness looks like. Use it during launch debates to remember why you do not, ever, lower the gate.

The promise: if your AI PRD, your eval harness, and your rollout plan all match what's on this page, you have a feature ready for production review. If any one of them doesn't, you have homework — and the section that's missing tells you exactly which lesson to revisit.

The 5-layer AI stack

Every AI feature is five layers. Bugs cluster in Layers 2 and 4. PMs spend the most time in Layers 5 and 1.

5
UX
How the user meets the AI — chat, inline, copilot, agentic. PM-owned.
4
Orchestration
Prompt + retrieval + tool calls + memory. Where most quality bugs hide.
3
Model
The LLM you don't own. Pin versions. Re-eval on every vendor upgrade.
2
Data
What the model sees — knowledge base, freshness, PII boundaries, audit logs.
1
Evals & ops
Did it work — golden sets, judges, gates, online metrics. PM owns the eval. Full stop.
PM owns L5 + L1 Eng leads L4 + L3 Data lead leads L2

The 8-section AI PRD

Every production AI PRD has these eight headings, in this order. Reviewers should reject any PRD missing one. Each card below is what you actually write under that heading.

1🎯 Problem & users

Write: the user pain in user words, the top 3 intents, one business KPI plus one trust metric.

2🧭 Scope (in / out)

Write: what intents the AI handles, what it refuses, and the named handoff for everything else.

3📝 Prompts & behavior

Write: role, tone, format, refusal strings — the behavioural contract the model is held to.

4🗂️ Data & retrieval

Write: allowed fields, freshness SLA, PII boundaries, retrieval config, audit-log retention.

5🚦 Eval & launch gate

Write: golden set size, rubric, pass-rate threshold, zero-tolerance subsets that block launch.

6🚀 Rollout plan

Write: shadow → canary → 100%, with a measurable gate and a rollback rule between each step.

7🛡️ Risks & fallbacks

Write: each risk gets a fallback, an owner, and a detection signal — not a paragraph of hedging.

8📊 Metrics & success

Write: north-star, guardrails, cost per task — what you watch on the dashboard at week 1, 4, 12.

The skip-test:
the only section you can compress is Rollout, and only if v0 already runs at 100%. Every other heading is a precondition for shipping. Missing one usually maps to a class of post-launch incident.

The 5-part prompt anatomy

Every system prompt — for support bots, summarizers, copilots, agents — has the same five colour-coded parts. If your prompt is missing one, find out which, then write it.

ROLE You are a SouthBank India support agent. TONE Calm, factual, never defensive.

FORMAT Answer in ≤3 sentences. Cite the policy doc. Never invent fees.

EXAMPLES Show 2 ideal answers and 1 bad answer the model must avoid.

REFUSAL If the question is about loan eligibility, respond exactly: "I can't help with that — tapping <a> will connect you to an agent."
Rule of thumb: ROLE + TONE shape the voice. FORMAT + EXAMPLES shape the structure. REFUSAL is the only line a regulator reads. Never ship without all five.

Eval harness checklist

A useable eval harness has these twelve pieces. Tick them before you talk to anyone about launching.

Golden set ≥ 200 inputs. Drawn from real user logs, not synthetic.
Slices defined. Easy / hard / regulated / multi-lingual — pass rate per slice.
Rubric is anchored. 1-5 with worked examples for each score; not "rate vibes."
Judge is calibrated. LLM-as-judge agreement ≥80% with human review on a 50-item sample.
Launch gate written down. "Ship at ≥85% overall, ≥95% on regulated slice."
Zero-tolerance subset. Any failure on this set blocks launch — no exceptions.
Re-runs on every prompt change. Eval is in the CI for prompt commits.
Re-runs on every model upgrade. Vendor pushes a new version → harness re-runs same day.
Online sample ≥1%/week. Real traffic graded weekly — offline eval is necessary, not sufficient.
Refusal rate tracked. Both refusal & over-refusal — refusing valid questions is a bug.
Cost per task logged. Pass rate × cost per task = your real launch cost.
Owner is named. Eval has a single PM owner, not "the team." If everyone owns it, no one does.

Rollout ladder

Shadow → 1% → 10% → 100%. Each step has a measurable gate before you climb. Each step has a rollback rule before you fall.

Shadow 0% user-facing eval live Canary 1% real users thumbs & logs Canary 10% quality holds cost in budget 100% gate cleared ship it ↑ build golden set ↑ gate: >80% pass ↑ gate: no P0 ↑ gate: 85%+
Shadow = no risk, full signal Canary = small risk, real users 100% = gate cleared, on-call ready

Decision rules table

The rules a working AI PM defaults to when the room disagrees. Memorise them. Cite them by number in PRD reviews.

# The situation The rule
1 Pass rate is below the gate the day before launch Never lower the gate. Slip the date or shrink the scope. The gate is your trust contract — moving it costs more than missing the date.
2 The model refuses a question Every refusal needs a named handoff — agent, form, doc link. A bare "I can't help with that" is a bug, not a feature.
3 Quality is great in the demo, the eval is "in progress" No eval, no launch. A demo is one happy path — an eval is the distribution of the next 100,000.
4 Vendor pushes a model upgrade Re-run the harness before promoting. "Same model name, new behaviour" is the most common cause of silent regressions.
5 Cost per task is over budget at 1% canary Cut scope or downgrade the model — never assume scale will fix unit economics. Cost per task only goes up with traffic.
6 Eng wants to "just tweak the prompt" mid-rollout Prompts are versioned like code. Any prompt change re-runs the eval and re-starts the rollout ladder from canary.
7 A regulated topic (loans, tax, medical) is in scope Regulated slice gets its own gate (≥95%) and a zero-tolerance set. Hallucination here is the lawsuit, not the bug.
8 The PRD has no Risks section Reject the PRD. "We didn't think of any" means you didn't think. Every AI feature has prompt injection, cost spike, vendor-upgrade and PII risks.
9 Launching on vibes — "users seem to like it" Trust score, task success, refusal rate, cost per task. Four numbers. If you can't quote them, you don't know if it works.
10 Build vs buy is unclear Buy the layer where you have no edge (model, infra). Build the layer where the moat lives (data, evals, UX). Default: buy L3, own L1.

Common gotchas

Six red flags. If any of these are true on your feature, fix them before the launch review, not after the incident.

Eval set is from QA, not real users
QA writes the questions they expect. Users ask the questions QA never thought of. Always sample from production logs.
No zero-tolerance slice
Average pass rate hides the worst 5%. Without a regulated/safety slice gate, your tail risk is invisible.
Prompt and code share a repo, no versioning
Three engineers tweak the prompt in three branches. You can't reproduce the eval result anyone quotes.
Cost per task only measured at v0
v0 ran on internal traffic. Production has 10× the tokens per session. Re-measure at every rollout step.
Refusal copy is bare
"I can't help with that" with no handoff is a dead-end. Refusal is a UX surface — design it like an empty state.
Success metric is "DAU"
DAU does not measure whether the AI helped anyone. Track task success, trust, time saved, refusal rate — and one business KPI.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You have the playbook.
Ship the feature.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →