Classical PM playbooks crack the moment your feature starts hallucinating. Here's what changes — and what you have to relearn to ship AI.
🎯 PM🤖 AI-relevant🧠 Foundations⚡ Beginner📖 ~14 min
Lesson 00 of AI Product Manager Track
What's actually different?
Your last feature probably had a spec like: "When user taps Pay, debit ₹500 and show a success toast." You shipped it, wrote two test cases, and moved on. The behavior was deterministic — same input, same output, forever.
Now your PRD says: "When the user asks a question, the AI answers it using the knowledge base." Suddenly you can't write a single test case that locks in correctness. The model might give a right answer today, a half-right one tomorrow, and a confident fabrication next week. Your QA lead is confused. Your legal team is horrified. Your engineer is asking what "good" means.
That's AI PM. It isn't regular PM with a chatbot bolted on. It's a different operating model. Five things break at once:
🎲
Outputs are probabilistic
Same input → different outputs. You can't assert equality, you assert a distribution.
📉
Quality needs evals, not tests
You swap unit tests for graded rubrics over a golden set of examples.
🪞
Hallucination is a feature of the tech
LLMs generate confidently wrong answers. You design mitigations, not wish it away.
💸
Unit economics live in tokens
Every prompt costs money. Pricing, margin, and rate-limits become PM decisions.
🧭
The model is a dependency you don't own
A silent GPT-4 → GPT-4o upgrade can change your product overnight. You version prompts like code.
🔐
Safety & privacy move into the spec
Prompt injection, PII leaks, IP issues — not edge cases, core PRD sections.
Classical PM vs AI PM — side by side
If you're coming from payments, marketplaces, SaaS or e-commerce, here's the mental remap.
Task success rate · trust · time saved · cost per task
Core artifact
PRD + Figma
PRD + prompt + eval set + UX
Who owns "correctness"
QA + engineering
PM owns the eval. Full stop.
The one-line takeaway:
Classical PM is about making the product do the right thing. AI PM is about making the product trend toward the right thing — and knowing, with data, how often it misses.
Scenario — SouthBank India ships a support bot
🏦
Real-world scenario
SouthBank wants to deflect 40% of tier-1 support tickets with an AI assistant
Priya is the PM for SouthBank India's support app. Leadership hands her a target: reduce tier-1 ticket volume by 40% using an AI chat assistant. She has 10 weeks, one engineer, and a pile of 5-year-old FAQ documents. She's never shipped an AI feature.
The classical PM version of this PRD would be short. Add a search bar, wire up FAQ articles, done. The AI version is much harder.
Priya's four uncomfortable questions
Question 1
"How do I know it works?"
Classical: QA runs 50 test cases.
AI PM: Priya needs a golden set — 200 real past tickets with ideal answers — and a rubric. Every release runs against it.
Question 2
"What if it gives a wrong loan answer?"
Classical: "Follow the logic in the code."
AI PM: Add retrieval (RAG), hard refusals for regulated topics, a human-handoff escape hatch, and a weekly review of 1% of transcripts.
Question 3
"How much will this cost per month?"
Classical: "A server, maybe ₹50k/month."
AI PM: Model cost ≈ ₹0.4 per conversation × 200k conversations/month = ₹80k + infra. Priya must pick a cheaper model for easy questions.
Question 4
"What if the vendor changes the model?"
Classical: Not her problem.
AI PM: She needs pinned model versions, a re-eval pipeline when vendors push upgrades, and a fallback model.
Why this matters: Priya's actual job the first three weeks is not writing Figma. It's assembling 200 real tickets, writing the rubric, and getting the eval script working. Without that, every demo is vibes.
The AI PM loop — how the work actually flows
Classical PMs live in a Discover → Define → Deliver loop. AI PMs add three ingredients: Prompt, Eval, Watch.
A minimal eval loop in plain Python
You don't have to code this yourself — but a PM who can read it is worth 3x one who can't. Here's what your eval engineer is actually writing.
eval_loop.pyHow you measure "did it work"
What Priya does with this number: If pass rate is 62%, she does not ship. She cuts scope ("no loan questions yet"), improves the prompt, or adds retrieval. She ships when pass rate clears her launch gate — say 85%.
The AI connection
Everything downstream in this track — prompt design, guardrails, UX patterns, pricing, rollouts — plugs into the loop you just read. golden-setrubricllm-as-judgelaunch-gate are the four words that separate AI PMs from PMs who say "AI" a lot.
What an AI PM actually owns
If you flip to the last page of a well-run AI feature spec, you'll see these artifacts listed. They are the PM's deliverables, not engineering's.
📝
The prompt
Versioned in git. PM owns wording, engineer owns plumbing.
📋
The golden set
100–500 realistic examples. PM curates and updates monthly.
📏
The rubric
How "good" is defined — in writing — for graders and LLM-judges.
🚦
The launch gate
The minimum eval score, refusal rate, and cost per task to ship.
🎛️
The fallback plan
What happens when the model is down, slow, or wrong.
📊
The in-product metrics
Thumbs up/down, handoff rate, task completion — wired before launch.
Mini challenge — score your current product
Pick any AI feature — yours or a competitor's
1. Find a probabilistic surface
Name one AI output in the product (summary, suggestion, chat reply). Write down three reasonable answers the model could give. Now write what a classical PM's acceptance criteria would look like for it. Notice how badly they fit.
2. Draft 10 golden examples
Write 10 real user inputs and, for each, the ideal AI answer. Yes, by hand. This is the starting shape of every AI PM's job.
3. Name your launch gate
In one sentence: "I'll ship this when ___% of the golden set scores ≥ 4, and cost per task is under ₹___." That sentence is the heart of your PRD.
Interview questions
EasyWhat's the difference between a classical PRD and an AI PRD?
+
A classical PRD specifies deterministic behavior: given X, do Y. An AI PRD specifies intent, examples, guardrails, and an eval — because the system is probabilistic. The AI PRD also lists the prompt, the golden set, and the launch gate as first-class artifacts.
EasyWhy can't you test an LLM feature with unit tests alone?
+
LLM outputs are non-deterministic — the same prompt can produce different outputs. Unit tests assert exact equality, which won't hold. You need evals: graded rubrics over a curated set of inputs with expected-ish answers, and you measure a pass rate rather than pass/fail.
MediumHow would you decide whether to use a smaller, cheaper model vs a flagship one?
+
Run both against the same golden set and measure pass rate, latency, and cost per task. Pick the cheapest model that clears your launch gate. Often you'll route: easy questions → cheap model, hard/edge questions → flagship. The routing decision itself is a PM call because it changes user experience.
MediumYour engineer says the feature is ready. Your eval score is 78%. What do you do?
+
Depends on the launch gate you set upfront. If the gate was 85%, you don't ship — you either improve the prompt/retrieval, cut scope (refuse risky questions), or lower the gate consciously with stakeholders. What you never do is ship anyway and hope. Set the gate before you know the score, so the score can't retroactively lower the bar.
HardA vendor just upgraded their flagship model. Your feature's quality drops 9 points overnight. How do you respond as PM?
+
Short term: pin the old model version (most vendors let you), re-run the eval on the new version, and triage which examples regressed. Medium term: update the prompt and retrieval for the new model, re-eval, and switch back when the gate is cleared. Long term: your spec should treat the model as a dependency — list the pinned version, the eval diff budget (e.g. "≤3 point regression OK"), and the fallback model. This is why AI PMs version prompts in git and keep a parallel "shadow" eval running against candidate models.
Test your knowledge
Question 1 of 5
Which of these is the clearest reason AI PM is a new discipline?
Question 2 of 5
What is a "golden set"?
Question 3 of 5
Priya's eval pass rate is 62% and her launch gate is 85%. What should she do?
Question 4 of 5
Which of these is NOT a normal artifact owned by an AI PM?
Question 5 of 5
Why do AI PMs care about token cost so much?
0
/5
You now think like an AI PM.
Common mistakes first-time AI PMs make
Shipping without an eval
Demos feel great, then 10% of prod traffic shows the feature is wrong half the time. If you don't have an eval, you don't have a product.
Writing the prompt, then the rubric
The rubric defines "good." Write it before the prompt so the prompt optimizes for the right thing.
Ignoring refusals
Refusing is a feature. For regulated domains (health, loans, legal), refusing a risky question is safer than answering it wrong.
Picking a model before the task
"We'll use GPT-4o" is not a strategy. Pick a cheap baseline, measure, then upgrade only where the gate demands it.
Key takeaways
AI features are probabilistic, so the PM's job shifts from "specify behavior" to "shape a distribution of behaviors."
The four artifacts that matter: prompt, golden set, rubric, launch gate.
Set the launch gate before you measure — that's how you avoid rationalizing the bar down.
Token cost is a PM decision. Every product meeting should discuss cost per task alongside conversion.
The model is a dependency. Pin versions, shadow-eval upgrades, and keep a fallback model ready.