What's actually different?

Your last feature probably had a spec like: "When user taps Pay, debit ₹500 and show a success toast." You shipped it, wrote two test cases, and moved on. The behavior was deterministic — same input, same output, forever.

Now your PRD says: "When the user asks a question, the AI answers it using the knowledge base." Suddenly you can't write a single test case that locks in correctness. The model might give a right answer today, a half-right one tomorrow, and a confident fabrication next week. Your QA lead is confused. Your legal team is horrified. Your engineer is asking what "good" means.

That's AI PM. It isn't regular PM with a chatbot bolted on. It's a different operating model. Five things break at once:

🎲
Outputs are probabilistic
Same input → different outputs. You can't assert equality, you assert a distribution.
📉
Quality needs evals, not tests
You swap unit tests for graded rubrics over a golden set of examples.
🪞
Hallucination is a feature of the tech
LLMs generate confidently wrong answers. You design mitigations, not wish it away.
💸
Unit economics live in tokens
Every prompt costs money. Pricing, margin, and rate-limits become PM decisions.
🧭
The model is a dependency you don't own
A silent GPT-4 → GPT-4o upgrade can change your product overnight. You version prompts like code.
🔐
Safety & privacy move into the spec
Prompt injection, PII leaks, IP issues — not edge cases, core PRD sections.

Classical PM vs AI PM — side by side

If you're coming from payments, marketplaces, SaaS or e-commerce, here's the mental remap.

Dimension Classical PM AI PM
Spec Deterministic rules & flows Intent + examples + guardrails
Quality bar Unit tests, E2E tests, bug count Eval score on a golden set, human review sampling
Failure mode Crash, 500, wrong calc Hallucination, unsafe output, wrong tone, cost spike
Rollout Feature flag → 1% → 100% Shadow mode → canary → eval gate → 100%
Cost model Mostly fixed infra per user Variable per request (tokens in + tokens out)
North-star metric DAU / conversion / retention Task success rate · trust · time saved · cost per task
Core artifact PRD + Figma PRD + prompt + eval set + UX
Who owns "correctness" QA + engineering PM owns the eval. Full stop.
The one-line takeaway:
Classical PM is about making the product do the right thing. AI PM is about making the product trend toward the right thing — and knowing, with data, how often it misses.

Scenario — SouthBank India ships a support bot

🏦
Real-world scenario
SouthBank wants to deflect 40% of tier-1 support tickets with an AI assistant

Priya is the PM for SouthBank India's support app. Leadership hands her a target: reduce tier-1 ticket volume by 40% using an AI chat assistant. She has 10 weeks, one engineer, and a pile of 5-year-old FAQ documents. She's never shipped an AI feature.

The classical PM version of this PRD would be short. Add a search bar, wire up FAQ articles, done. The AI version is much harder.

Priya's four uncomfortable questions

Question 1
"How do I know it works?"
Classical: QA runs 50 test cases.
AI PM: Priya needs a golden set — 200 real past tickets with ideal answers — and a rubric. Every release runs against it.
Question 2
"What if it gives a wrong loan answer?"
Classical: "Follow the logic in the code."
AI PM: Add retrieval (RAG), hard refusals for regulated topics, a human-handoff escape hatch, and a weekly review of 1% of transcripts.
Question 3
"How much will this cost per month?"
Classical: "A server, maybe ₹50k/month."
AI PM: Model cost ≈ ₹0.4 per conversation × 200k conversations/month = ₹80k + infra. Priya must pick a cheaper model for easy questions.
Question 4
"What if the vendor changes the model?"
Classical: Not her problem.
AI PM: She needs pinned model versions, a re-eval pipeline when vendors push upgrades, and a fallback model.
Why this matters: Priya's actual job the first three weeks is not writing Figma. It's assembling 200 real tickets, writing the rubric, and getting the eval script working. Without that, every demo is vibes.

The AI PM loop — how the work actually flows

Classical PMs live in a Discover → Define → Deliver loop. AI PMs add three ingredients: Prompt, Eval, Watch.

A minimal eval loop in plain Python

You don't have to code this yourself — but a PM who can read it is worth 3x one who can't. Here's what your eval engineer is actually writing.

eval_loop.py How you measure "did it work"
What Priya does with this number: If pass rate is 62%, she does not ship. She cuts scope ("no loan questions yet"), improves the prompt, or adds retrieval. She ships when pass rate clears her launch gate — say 85%.

The AI connection

Everything downstream in this track — prompt design, guardrails, UX patterns, pricing, rollouts — plugs into the loop you just read. golden-setrubricllm-as-judgelaunch-gate are the four words that separate AI PMs from PMs who say "AI" a lot.

What an AI PM actually owns

If you flip to the last page of a well-run AI feature spec, you'll see these artifacts listed. They are the PM's deliverables, not engineering's.

📝
The prompt
Versioned in git. PM owns wording, engineer owns plumbing.
📋
The golden set
100–500 realistic examples. PM curates and updates monthly.
📏
The rubric
How "good" is defined — in writing — for graders and LLM-judges.
🚦
The launch gate
The minimum eval score, refusal rate, and cost per task to ship.
🎛️
The fallback plan
What happens when the model is down, slow, or wrong.
📊
The in-product metrics
Thumbs up/down, handoff rate, task completion — wired before launch.

Mini challenge — score your current product

Pick any AI feature — yours or a competitor's

1. Find a probabilistic surface

Name one AI output in the product (summary, suggestion, chat reply). Write down three reasonable answers the model could give. Now write what a classical PM's acceptance criteria would look like for it. Notice how badly they fit.

2. Draft 10 golden examples

Write 10 real user inputs and, for each, the ideal AI answer. Yes, by hand. This is the starting shape of every AI PM's job.

3. Name your launch gate

In one sentence: "I'll ship this when ___% of the golden set scores ≥ 4, and cost per task is under ₹___." That sentence is the heart of your PRD.

Interview questions

Easy What's the difference between a classical PRD and an AI PRD?
+
A classical PRD specifies deterministic behavior: given X, do Y. An AI PRD specifies intent, examples, guardrails, and an eval — because the system is probabilistic. The AI PRD also lists the prompt, the golden set, and the launch gate as first-class artifacts.
Easy Why can't you test an LLM feature with unit tests alone?
+
LLM outputs are non-deterministic — the same prompt can produce different outputs. Unit tests assert exact equality, which won't hold. You need evals: graded rubrics over a curated set of inputs with expected-ish answers, and you measure a pass rate rather than pass/fail.
Medium How would you decide whether to use a smaller, cheaper model vs a flagship one?
+
Run both against the same golden set and measure pass rate, latency, and cost per task. Pick the cheapest model that clears your launch gate. Often you'll route: easy questions → cheap model, hard/edge questions → flagship. The routing decision itself is a PM call because it changes user experience.
Medium Your engineer says the feature is ready. Your eval score is 78%. What do you do?
+
Depends on the launch gate you set upfront. If the gate was 85%, you don't ship — you either improve the prompt/retrieval, cut scope (refuse risky questions), or lower the gate consciously with stakeholders. What you never do is ship anyway and hope. Set the gate before you know the score, so the score can't retroactively lower the bar.
Hard A vendor just upgraded their flagship model. Your feature's quality drops 9 points overnight. How do you respond as PM?
+
Short term: pin the old model version (most vendors let you), re-run the eval on the new version, and triage which examples regressed. Medium term: update the prompt and retrieval for the new model, re-eval, and switch back when the gate is cleared. Long term: your spec should treat the model as a dependency — list the pinned version, the eval diff budget (e.g. "≤3 point regression OK"), and the fallback model. This is why AI PMs version prompts in git and keep a parallel "shadow" eval running against candidate models.

Test your knowledge

Question 1 of 5
Which of these is the clearest reason AI PM is a new discipline?
Question 2 of 5
What is a "golden set"?
Question 3 of 5
Priya's eval pass rate is 62% and her launch gate is 85%. What should she do?
Question 4 of 5
Which of these is NOT a normal artifact owned by an AI PM?
Question 5 of 5
Why do AI PMs care about token cost so much?
0
/5
You now think like an AI PM.

Common mistakes first-time AI PMs make

Shipping without an eval
Demos feel great, then 10% of prod traffic shows the feature is wrong half the time. If you don't have an eval, you don't have a product.
Writing the prompt, then the rubric
The rubric defines "good." Write it before the prompt so the prompt optimizes for the right thing.
Ignoring refusals
Refusing is a feature. For regulated domains (health, loans, legal), refusing a risky question is safer than answering it wrong.
Picking a model before the task
"We'll use GPT-4o" is not a strategy. Pick a cheap baseline, measure, then upgrade only where the gate demands it.

Key takeaways

  • AI features are probabilistic, so the PM's job shifts from "specify behavior" to "shape a distribution of behaviors."
  • The four artifacts that matter: prompt, golden set, rubric, launch gate.
  • Set the launch gate before you measure — that's how you avoid rationalizing the bar down.
  • Token cost is a PM decision. Every product meeting should discuss cost per task alongside conversion.
  • The model is a dependency. Pin versions, shadow-eval upgrades, and keep a fallback model ready.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You see why AI PM is different.
Now ship features that actually work.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →