The five layers

Strip away the hype and every AI feature — chatbot, copilot, summarizer, agent — is the same five-layer sandwich. If you can point to the layer a bug lives in, you can route it to the right fix. If you can't, you'll ask the model team to solve a UX problem and the UX team to solve an eval problem.

5
UX — how the user meets the AI
Chat · inline suggestion · agentic side-panel · copilot · embedded. Includes trust signals, undo, handoff, refusal UI.
4
Orchestration — prompt + tools + retrieval
The logic around the model: system prompts, function calls, RAG, routing between models, retry loops.
3
Model — the reasoning engine
The LLM itself: vendor choice (OpenAI, Anthropic, Google, Meta), version, temperature, fine-tunes.
2
Data — what the model sees
Documents, embeddings, structured records, user profile, session memory. What you index, own, and protect.
1
Evals & ops — did it actually work?
Golden sets, offline eval, LLM-as-judge, human review, logging, tracing, cost monitoring, rollout gates.
Reading order matters:
you build the stack top-down (UX first — what does the user actually need?) and measure it bottom-up (evals tell you whether the layers work together). A PM who gets this is worth 3 PMs who don't.

Each layer, what the PM actually decides

The cheat for each layer: one sentence about what it is, and two columns — decisions you make, and questions you ask.

Layer 5 · UX

🖼️ How the user meets the AI

The surface and interaction pattern. Do they chat with it? Does it appear inline as a suggestion? Does it operate as an agent in a side-panel? The UX decision shapes everything below — latency tolerance, prompt length, refusal design.

You decide

  • Interaction pattern (chat / inline / agent)
  • Streaming vs full response
  • Trust signals (sources, confidence, "AI-generated" badge)
  • Handoff path when the AI can't help

You ask engineering

  • "What's the P95 latency budget?"
  • "Can we show citations inline?"
  • "How do we render a refusal?"
  • "Where does the user undo?"
Layer 4 · Orchestration

🧩 Prompt, retrieval, tools

The scaffolding around the model. The system prompt ("you are a helpful support agent…"), retrieval (pulling the right docs), tools (calling APIs), routing (small model for easy, flagship for hard). This is where most of your time actually goes.

You decide

  • System prompt — tone, refusals, format
  • What to retrieve and from where
  • Which tools the model can call
  • Routing rules across models

You ask engineering

  • "What's in the context window right now?"
  • "What does the retrieval return for this query?"
  • "When does the router fall back?"
  • "Are we logging the full prompt for debugging?"
Layer 3 · Model

🧠 The reasoning engine

The LLM itself. Pick your vendor, your model tier, your context size, your temperature. This layer is deceptively small in code but huge in cost and risk. A silent upgrade can change your whole product.

You decide

  • Vendor (OpenAI / Anthropic / Google / open-source)
  • Pinned model version (never just "latest")
  • Temperature & max tokens
  • When to fine-tune vs prompt

You ask engineering

  • "What model is production running today?"
  • "When this vendor upgrades, what's our process?"
  • "What's the fallback model if the vendor is down?"
  • "Do we need data isolation for enterprise tenants?"
Layer 2 · Data

🗂️ What the model sees

Your moat. A great model with bad data is a bad product. A mediocre model with great proprietary data is a business. PMs decide what to index, refresh, own, and protect.

You decide

  • Which documents to embed & index
  • Refresh cadence (daily? per change?)
  • What session memory the AI keeps
  • PII redaction & tenant isolation

You ask engineering

  • "How fresh is the knowledge base?"
  • "When a doc is deleted, how fast is the index updated?"
  • "Do we log raw user inputs? For how long?"
  • "Can tenant A's data ever leak into tenant B's answers?"
Layer 1 · Evals & ops

📏 Did it actually work?

The layer you wish you'd started on. Offline evals before launch, online monitoring after. Without it you're shipping on vibes. With it, you ship on data.

You decide

  • The golden set & rubric
  • The launch gate (min pass rate, max cost)
  • Human review sample rate
  • Regression budget when model upgrades

You ask engineering

  • "How do we trace a single bad response end-to-end?"
  • "What alerts fire on cost & quality regressions?"
  • "Is thumbs-down wired to a review queue?"
  • "How long until we notice silent quality drop?"

Dissecting a real feature — SouthBank's support bot

Same product as Lesson 00. Let's map it layer by layer.

Layer SouthBank's choice Why
5 · UX Chat bubble in the mobile app, with "Talk to a human" always visible Users already know chat. The handoff link protects trust.
4 · Orchestration System prompt + RAG over the FAQ. Hard refuse on loan/credit questions. Retrieval keeps answers grounded. Refusal avoids regulatory exposure.
3 · Model GPT-4o-mini for 90% of queries, GPT-4o for escalations Cost: ₹0.4/conversation average. Quality stays above gate.
2 · Data FAQ + product terms, refreshed nightly. No PII in the prompt. Stale docs = wrong answers. PII = compliance nightmare.
1 · Evals & ops 200-ticket golden set, LLM-as-judge + 1% human review. Launch gate: 85% pass, ₹0.5 max/conversation. Lets Priya ship confidently and catch regressions within 24 hours.
The punchline: Priya's hardest decisions aren't in Layer 3 (model). They're in Layer 4 (should loan questions refuse or route to a human?) and Layer 1 (where's the launch gate?). That's where AI PMs earn their pay.

When a bug comes in, which layer is it?

The fastest way to internalize the stack is to debug with it. A user pastes a bad transcript in Slack. Which layer do you open first?

Diagnostic questions in plain Python

This is a trimmed version of what AI PM "triage scripts" look like. You don't write this, but you read its output with the engineer on calls.

triage.py Which layer owns this bug?

The AI connection

Every AI horror story you read — from the airline chatbot that invented a refund policy to the news summarizer that fabricated quotes — can be traced to a specific layer. Chatbot made up a policy? Layer 2 (no grounding data) or Layer 4 (no refusal rule). News summarizer fabricating? Layer 1 (no eval caught it). layer-2layer-4layer-1 are the layers where most AI PM bugs actually live.

Mini challenge — dissect a product you use

Pick an AI product you use daily

1. Name the five layers

For a product like ChatGPT's search, Gmail's smart reply, or Notion AI — write one sentence describing each layer. Which layer did you have the hardest time naming? That's usually the PM's weakest lens.

2. Spot the moat

Which layer gives this product its defensibility? (Hint: for Notion AI it's Layer 2 — your workspace data. For ChatGPT it's Layer 3 + scale. For Gmail Smart Reply it's Layer 1 + implicit feedback.)

3. Find a weak layer

Where would you invest if you were their PM next quarter? Why? Write the one-line hypothesis you'd put in a doc.

Interview questions

Easy Name the five layers of the AI product stack.
+
UX (how the user meets the AI), Orchestration (prompt + retrieval + tools), Model (the LLM), Data (what the model sees), and Evals & Ops (did it work and is it still working). The mnemonic: U-O-M-D-E.
Easy Which layer is usually the moat?
+
Layer 2 — data — for most enterprise AI products. The model is a commodity; your users' data, workflow traces, and feedback loops are not. Fine-tuning and private indexes compound this over time.
Medium Your chatbot hallucinates an answer about a pricing plan. Which layer do you fix first?
+
Usually Layer 2 or 4. First check retrieval (Layer 2): did the model even have the pricing page in context? If not, the index is the bug. If yes, the prompt (Layer 4) may not be forcing grounding — "answer only from the documents provided; if not present, refuse." Only after both layers are clean do you touch the model.
Medium A new model version from your vendor promises 20% better benchmarks. Should you upgrade?
+
Not on a press release. Run your Layer 1 evals (golden set) on the new model in shadow mode. Compare pass rate, cost, and latency to the current model. Ship only if the new one clears your launch gate. Benchmarks from the vendor are a signal, not a verdict — your golden set is.
Hard You're asked to cut AI costs by 50% in a month without hurting quality. Which layers do you attack, in what order?
+
Most leverage is in Layers 4 and 3. Start with Layer 4 (orchestration): trim the prompt, compress retrieved context, cache frequent answers — cost scales linearly with tokens. Then Layer 3 (model): route easy queries to a cheaper model (GPT-4o-mini, Haiku, Flash), reserve the flagship for harder ones. Layer 1 evals gate the whole change: you must prove quality didn't drop. Avoid touching Layer 2 (data) first — smaller indexes rarely cut cost much but often hurt quality.

Test your knowledge

Question 1 of 5
Which layer covers system prompts, retrieval, and tool-calling?
Question 2 of 5
An AI support bot confidently cites a pricing tier that doesn't exist. Most likely culprit?
Question 3 of 5
Which layer is usually the strongest moat for an enterprise AI product?
Question 4 of 5
Your vendor pushes a new model. Safest way to decide whether to upgrade?
Question 5 of 5
To cut AI cost 50% quickly, which layers give you the most leverage first?
0
/5
You speak stack. That's the AI PM lingua franca.

Common mistakes

"Let's just swap the model"
Upgrading Layer 3 without updating Layer 4 prompts often breaks behavior. Every model has quirks your prompt was written against.
Over-indexing on model choice
PMs spend a month debating OpenAI vs Anthropic. The actual quality gap for most tasks is closed by Layer 2 (data) and Layer 4 (prompt).
Skipping Layer 1 until launch
By launch day, it's too late to build a golden set. Start Layer 1 on day 1, even with 20 examples.
Forgetting UX = trust
Same answer, shown without a citation vs with one, produces completely different trust outcomes. UX is not polish, it's product.

Key takeaways

  • Every AI feature stands on five layers: UX, Orchestration, Model, Data, Evals & Ops.
  • Build top-down (what does the user need?), measure bottom-up (do evals say it works?).
  • Most bugs live in Layers 2 and 4, not Layer 3. Don't swap models first.
  • The moat is almost always Layer 2 (your data) or Layer 1 (your feedback loop), rarely Layer 3.
  • A PM who can route a bug report to the right layer saves their team weeks.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You can now diagram any AI product.
Next step — build one.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →