The AI PM one-pager — every framework, table, and decision rule from the track, on a single screen.
📄 Reference🎯 PM🤖 AI-relevant⚡ Intermediate📖 ~10 min
Reference — One-pager
Screenshot · Pin to Notion · Ship the feature
Why use this
This is the page you screenshot, pin to your second monitor, and reach for the next time someone says "can we ship an AI feature by Q3?" It is the working PM's compression of the 22 lessons in the track — the diagrams, the tables, the rules — laid out so you can scan, not read.
Use it during PRD reviews to remember which sections must exist. Use it during eval design to remember what a good harness looks like. Use it during launch debates to remember why you do not, ever, lower the gate.
The promise: if your AI PRD, your eval harness, and your rollout plan all match what's on this page, you have a feature ready for production review. If any one of them doesn't, you have homework — and the section that's missing tells you exactly which lesson to revisit.
The 5-layer AI stack
Every AI feature is five layers. Bugs cluster in Layers 2 and 4. PMs spend the most time in Layers 5 and 1.
5
UX
How the user meets the AI — chat, inline, copilot, agentic. PM-owned.
4
Orchestration
Prompt + retrieval + tool calls + memory. Where most quality bugs hide.
3
Model
The LLM you don't own. Pin versions. Re-eval on every vendor upgrade.
2
Data
What the model sees — knowledge base, freshness, PII boundaries, audit logs.
1
Evals & ops
Did it work — golden sets, judges, gates, online metrics. PM owns the eval. Full stop.
Every production AI PRD has these eight headings, in this order. Reviewers should reject any PRD missing one. Each card below is what you actually write under that heading.
1🎯 Problem & users
Write: the user pain in user words, the top 3 intents, one business KPI plus one trust metric.
2🧭 Scope (in / out)
Write: what intents the AI handles, what it refuses, and the named handoff for everything else.
3📝 Prompts & behavior
Write: role, tone, format, refusal strings — the behavioural contract the model is held to.
Write: golden set size, rubric, pass-rate threshold, zero-tolerance subsets that block launch.
6🚀 Rollout plan
Write: shadow → canary → 100%, with a measurable gate and a rollback rule between each step.
7🛡️ Risks & fallbacks
Write: each risk gets a fallback, an owner, and a detection signal — not a paragraph of hedging.
8📊 Metrics & success
Write: north-star, guardrails, cost per task — what you watch on the dashboard at week 1, 4, 12.
The skip-test:
the only section you can compress is Rollout, and only if v0 already runs at 100%. Every other heading is a precondition for shipping. Missing one usually maps to a class of post-launch incident.
The 5-part prompt anatomy
Every system prompt — for support bots, summarizers, copilots, agents — has the same five colour-coded parts. If your prompt is missing one, find out which, then write it.
ROLE
You are a SouthBank India support agent.
TONE
Calm, factual, never defensive.
FORMAT
Answer in ≤3 sentences. Cite the policy doc. Never invent fees.
EXAMPLES
Show 2 ideal answers and 1 bad answer the model must avoid.
REFUSAL
If the question is about loan eligibility, respond exactly: "I can't help with that — tapping <a> will connect you to an agent."
Rule of thumb: ROLE + TONE shape the voice. FORMAT + EXAMPLES shape the structure. REFUSAL is the only line a regulator reads. Never ship without all five.
Eval harness checklist
A useable eval harness has these twelve pieces. Tick them before you talk to anyone about launching.
Golden set ≥ 200 inputs. Drawn from real user logs, not synthetic.
Slices defined. Easy / hard / regulated / multi-lingual — pass rate per slice.
Rubric is anchored. 1-5 with worked examples for each score; not "rate vibes."
Judge is calibrated. LLM-as-judge agreement ≥80% with human review on a 50-item sample.
Launch gate written down. "Ship at ≥85% overall, ≥95% on regulated slice."
Zero-tolerance subset. Any failure on this set blocks launch — no exceptions.
Re-runs on every prompt change. Eval is in the CI for prompt commits.
Re-runs on every model upgrade. Vendor pushes a new version → harness re-runs same day.
Online sample ≥1%/week. Real traffic graded weekly — offline eval is necessary, not sufficient.
Refusal rate tracked. Both refusal & over-refusal — refusing valid questions is a bug.
Cost per task logged. Pass rate × cost per task = your real launch cost.
Owner is named. Eval has a single PM owner, not "the team." If everyone owns it, no one does.
Rollout ladder
Shadow → 1% → 10% → 100%. Each step has a measurable gate before you climb. Each step has a rollback rule before you fall.
Shadow = no risk, full signalCanary = small risk, real users100% = gate cleared, on-call ready
Decision rules table
The rules a working AI PM defaults to when the room disagrees. Memorise them. Cite them by number in PRD reviews.
#
The situation
The rule
1
Pass rate is below the gate the day before launch
Never lower the gate. Slip the date or shrink the scope. The gate is your trust contract — moving it costs more than missing the date.
2
The model refuses a question
Every refusal needs a named handoff — agent, form, doc link. A bare "I can't help with that" is a bug, not a feature.
3
Quality is great in the demo, the eval is "in progress"
No eval, no launch. A demo is one happy path — an eval is the distribution of the next 100,000.
4
Vendor pushes a model upgrade
Re-run the harness before promoting. "Same model name, new behaviour" is the most common cause of silent regressions.
5
Cost per task is over budget at 1% canary
Cut scope or downgrade the model — never assume scale will fix unit economics. Cost per task only goes up with traffic.
6
Eng wants to "just tweak the prompt" mid-rollout
Prompts are versioned like code. Any prompt change re-runs the eval and re-starts the rollout ladder from canary.
7
A regulated topic (loans, tax, medical) is in scope
Regulated slice gets its own gate (≥95%) and a zero-tolerance set. Hallucination here is the lawsuit, not the bug.
8
The PRD has no Risks section
Reject the PRD. "We didn't think of any" means you didn't think. Every AI feature has prompt injection, cost spike, vendor-upgrade and PII risks.
9
Launching on vibes — "users seem to like it"
Trust score, task success, refusal rate, cost per task. Four numbers. If you can't quote them, you don't know if it works.
10
Build vs buy is unclear
Buy the layer where you have no edge (model, infra). Build the layer where the moat lives (data, evals, UX). Default: buy L3, own L1.
Common gotchas
Six red flags. If any of these are true on your feature, fix them before the launch review, not after the incident.
Eval set is from QA, not real users
QA writes the questions they expect. Users ask the questions QA never thought of. Always sample from production logs.
No zero-tolerance slice
Average pass rate hides the worst 5%. Without a regulated/safety slice gate, your tail risk is invisible.
Prompt and code share a repo, no versioning
Three engineers tweak the prompt in three branches. You can't reproduce the eval result anyone quotes.
Cost per task only measured at v0
v0 ran on internal traffic. Production has 10× the tokens per session. Re-measure at every rollout step.
Refusal copy is bare
"I can't help with that" with no handoff is a dead-end. Refusal is a UX surface — design it like an empty state.
Success metric is "DAU"
DAU does not measure whether the AI helped anyone. Track task success, trust, time saved, refusal rate — and one business KPI.
Africa & India's Best AI Academy
🤖 AI-first🎓 Placement Assistance
You have the playbook. Ship the feature.
Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.