The five layers
Strip away the hype and every AI feature — chatbot, copilot, summarizer, agent — is the same five-layer sandwich. If you can point to the layer a bug lives in, you can route it to the right fix. If you can't, you'll ask the model team to solve a UX problem and the UX team to solve an eval problem.
Each layer, what the PM actually decides
The cheat for each layer: one sentence about what it is, and two columns — decisions you make, and questions you ask.
🖼️ How the user meets the AI
The surface and interaction pattern. Do they chat with it? Does it appear inline as a suggestion? Does it operate as an agent in a side-panel? The UX decision shapes everything below — latency tolerance, prompt length, refusal design.
You decide
- Interaction pattern (chat / inline / agent)
- Streaming vs full response
- Trust signals (sources, confidence, "AI-generated" badge)
- Handoff path when the AI can't help
You ask engineering
- "What's the P95 latency budget?"
- "Can we show citations inline?"
- "How do we render a refusal?"
- "Where does the user undo?"
🧩 Prompt, retrieval, tools
The scaffolding around the model. The system prompt ("you are a helpful support agent…"), retrieval (pulling the right docs), tools (calling APIs), routing (small model for easy, flagship for hard). This is where most of your time actually goes.
You decide
- System prompt — tone, refusals, format
- What to retrieve and from where
- Which tools the model can call
- Routing rules across models
You ask engineering
- "What's in the context window right now?"
- "What does the retrieval return for this query?"
- "When does the router fall back?"
- "Are we logging the full prompt for debugging?"
🧠 The reasoning engine
The LLM itself. Pick your vendor, your model tier, your context size, your temperature. This layer is deceptively small in code but huge in cost and risk. A silent upgrade can change your whole product.
You decide
- Vendor (OpenAI / Anthropic / Google / open-source)
- Pinned model version (never just "latest")
- Temperature & max tokens
- When to fine-tune vs prompt
You ask engineering
- "What model is production running today?"
- "When this vendor upgrades, what's our process?"
- "What's the fallback model if the vendor is down?"
- "Do we need data isolation for enterprise tenants?"
🗂️ What the model sees
Your moat. A great model with bad data is a bad product. A mediocre model with great proprietary data is a business. PMs decide what to index, refresh, own, and protect.
You decide
- Which documents to embed & index
- Refresh cadence (daily? per change?)
- What session memory the AI keeps
- PII redaction & tenant isolation
You ask engineering
- "How fresh is the knowledge base?"
- "When a doc is deleted, how fast is the index updated?"
- "Do we log raw user inputs? For how long?"
- "Can tenant A's data ever leak into tenant B's answers?"
📏 Did it actually work?
The layer you wish you'd started on. Offline evals before launch, online monitoring after. Without it you're shipping on vibes. With it, you ship on data.
You decide
- The golden set & rubric
- The launch gate (min pass rate, max cost)
- Human review sample rate
- Regression budget when model upgrades
You ask engineering
- "How do we trace a single bad response end-to-end?"
- "What alerts fire on cost & quality regressions?"
- "Is thumbs-down wired to a review queue?"
- "How long until we notice silent quality drop?"
Dissecting a real feature — SouthBank's support bot
Same product as Lesson 00. Let's map it layer by layer.
| Layer | SouthBank's choice | Why |
|---|---|---|
| 5 · UX | Chat bubble in the mobile app, with "Talk to a human" always visible | Users already know chat. The handoff link protects trust. |
| 4 · Orchestration | System prompt + RAG over the FAQ. Hard refuse on loan/credit questions. | Retrieval keeps answers grounded. Refusal avoids regulatory exposure. |
| 3 · Model | GPT-4o-mini for 90% of queries, GPT-4o for escalations | Cost: ₹0.4/conversation average. Quality stays above gate. |
| 2 · Data | FAQ + product terms, refreshed nightly. No PII in the prompt. | Stale docs = wrong answers. PII = compliance nightmare. |
| 1 · Evals & ops | 200-ticket golden set, LLM-as-judge + 1% human review. Launch gate: 85% pass, ₹0.5 max/conversation. | Lets Priya ship confidently and catch regressions within 24 hours. |
When a bug comes in, which layer is it?
The fastest way to internalize the stack is to debug with it. A user pastes a bad transcript in Slack. Which layer do you open first?
Diagnostic questions in plain Python
This is a trimmed version of what AI PM "triage scripts" look like. You don't write this, but you read its output with the engineer on calls.
The AI connection
Every AI horror story you read — from the airline chatbot that invented a refund policy to the news summarizer that fabricated quotes — can be traced to a specific layer. Chatbot made up a policy? Layer 2 (no grounding data) or Layer 4 (no refusal rule). News summarizer fabricating? Layer 1 (no eval caught it). layer-2layer-4layer-1 are the layers where most AI PM bugs actually live.
Mini challenge — dissect a product you use
Pick an AI product you use daily
For a product like ChatGPT's search, Gmail's smart reply, or Notion AI — write one sentence describing each layer. Which layer did you have the hardest time naming? That's usually the PM's weakest lens.
Which layer gives this product its defensibility? (Hint: for Notion AI it's Layer 2 — your workspace data. For ChatGPT it's Layer 3 + scale. For Gmail Smart Reply it's Layer 1 + implicit feedback.)
Where would you invest if you were their PM next quarter? Why? Write the one-line hypothesis you'd put in a doc.
Interview questions
Test your knowledge
Common mistakes
Key takeaways
- Every AI feature stands on five layers: UX, Orchestration, Model, Data, Evals & Ops.
- Build top-down (what does the user need?), measure bottom-up (do evals say it works?).
- Most bugs live in Layers 2 and 4, not Layer 3. Don't swap models first.
- The moat is almost always Layer 2 (your data) or Layer 1 (your feedback loop), rarely Layer 3.
- A PM who can route a bug report to the right layer saves their team weeks.
You can now diagram any AI product.
Next step — build one.
Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Agentic Engineer →