What this lesson is

Most AI dashboards lie politely. Pass rate is up, thumbs-up is at 90%, the launch slide deck looks great — and three months later somebody notices the feature is being used half as much as it was the week after launch. The honest metric was always there. Nobody had wired it.

This lesson is about choosing the small set of numbers that genuinely prove an AI feature is working — and the trap-shaped ones that don't. By the end, you can walk into a metrics review with six numbers, not sixty, and defend each of them against a sceptical leader who asks "so what?"

The mental model: AI metrics split cleanly into two worlds. Offline eval metrics (pass rate on a golden set, run before launch, decided by your team) and in-product metrics (task success, time-to-task, escalation, retention — observed on real users, after launch). Most teams ship strong on the first and weak on the second. Then they get surprised.

The PM's measurement job, in one line:
wire the in-product metrics before the feature ships, set a baseline in the first week, and watch the leading indicators weekly so the lagging ones don't surprise anyone in quarter three.

The six metrics every AI feature actually needs

You don't need a metrics catalogue. You need six numbers a PM can recite from memory and explain to a non-PM in a sentence. Each one catches a different failure mode that the others can't.

✅
Task success rate
Of conversations that started a task, what % ended with the task done — the user got their answer, the form was filled, the dispute was logged. The single most honest number on the page.
🚫
Refusal rate
% of requests where the AI refused or said "I can't help with that". Should be non-zero (good guardrails) but not climbing — a rising refusal rate means the AI is stepping back from things it should be handling.
⏱️
Time-to-task
Median seconds from "user starts" to "task done". The promise of AI is mostly compression of time — if this number isn't shrinking versus the non-AI baseline, you don't have an AI win, you have a chatbot.
👍
Thumbs-up rate
% of responses with explicit positive feedback (thumbs, stars, in-line CSAT). Cheapest trust signal you'll ever ship. Read it as a directional indicator, not an absolute — it's biased and noisy, but the trend is honest.
🆘
Escalation rate
% of AI sessions that bounced to a human agent (or to a non-AI flow). The most under-rated metric. If escalation is high, the AI is solving the easy 30% and shovelling the hard 70% to support — which is sometimes fine and sometimes a disaster.
🔁
Retention impact
Do users who used the AI feature this week come back next week / next month at higher rates than matched users who didn't? The lagging answer to "is this a real product or just a demo?".
The PM's first instinct should be: watch task success and time-to-task weekly, watch retention impact monthly, and treat thumbs-up as a thermometer not a thermostat. If you have to pick three to put on the wall, those are the three.

What each metric catches — and where it lies

The same six numbers, expanded. Read each row as: what does this metric catch that nothing else does, would it lie if you only looked at it alone, and which team owns the number when it moves.

Metric What it catches Vanity if alone? Owned by
Task success rate Whether the user actually got their job done. End-to-end outcome, not surface politeness. Low risk. The most honest single number — but defining "task done" is hard and political. PM (defines the success event); engineering instruments it.
Refusal rate Guardrail health. Catches both a bot that's gone too cautious and one that's broken its refusal rules. Yes alone. A "low" refusal rate could mean great coverage or a model that's hallucinating instead of refusing — needs a paired safety metric. PM + safety/ML; reviewed in weekly quality readout.
Time-to-task Whether AI compressed effort versus the previous flow. The "did this actually save time" sanity check. Yes alone. A fast bad answer is faster than a slow good one — must be paired with task success. PM; compared against pre-AI baseline maintained by analytics.
Thumbs-up rate Direct user trust signal. Cheapest in-product CSAT you'll ever wire. Highest vanity risk. Selection-biased (only happy or angry people click), gameable by tone changes, and silent on the users who quietly leave. PM watches the trend; design owns the prompt for feedback.
Escalation rate The "AI handed it back to a human" signal. Catches feature-creep and over-promised coverage. Low alone. But "low escalation" can hide users who gave up entirely instead of escalating — pair with abandonment. PM + support ops; both have a stake in the number.
Retention impact Whether the AI feature is genuinely sticky — do users come back because of it or in spite of it. Lagging — by the time it moves you're usually two quarters late. Pair with weekly leading indicators. PM + analytics; the number that matters most in QBRs.
Rule of thumb:
never report a single AI metric in isolation. Each row above lies a particular way; the others catch the lie. Task success without time-to-task is a slow win. Thumbs-up without retention is theatre. Escalation without abandonment is a half-truth.

Where each metric sits — leading vs lagging, user vs business

The same six numbers, plotted on the only 2×2 that matters for measurement. Y-axis: how fast does the number move when something breaks (leading at the top, lagging at the bottom). X-axis: who feels it — the user, or the business. The quadrant a metric lives in tells you how often to look at it and who to escalate to.

↑ Leading (moves first) Lagging (moves last) ↓ User-felt Business-felt Watch weekly 👍 Thumbs-up rate ⏱️ Time-to-task ✅ Task success Watch weekly 🆘 Escalation rate 🚫 Refusal rate (eval pass rate sits here too) Watch monthly 🔁 Retention impact (NPS, repeat use) Watch quarterly 💰 Cost per task (ticket deflection ₹)
How to read this:
the top row is what you stare at every Monday — these are the canaries. The bottom row is the QBR slide. If you only watch the bottom row, every problem is already a quarter old before you see it. If you only watch the top row, you'll miss the slow leak that's burning money.

Scenario — Priya at SouthBank India, one month post-launch

🏦
Real-world scenario
"Eval pass rate is 88%. Why does leadership want a deeper review?"

Priya is the PM for the SouthBank India support bot, launched four weeks ago across the Mumbai retail-banking app. Her launch slide showed an offline eval pass rate of 88% on a 200-question golden set. The bot is live for 100% of users. The launch went smoothly. She has 10 weeks left in the quarter.

Last week the COO sent a calendar invite titled "AI bot — month-one metrics review". Priya opens her dashboard and finds exactly two numbers: eval pass rate (still 88%) and DAU (steady). The thing she's missing — and the thing the COO will absolutely ask about — is whether the bot is actually working for users. Pass rate on a frozen test set says one thing. Real users in a real app say another. She has one week to wire the in-product metrics that should have been on day one.

The dashboard Priya builds for the review

By the end of the week, Priya has six numbers wired and four weeks of trailing data backfilled from logs. She picks four to put on the leadership slide — the rest live in the backup. For each, she shows the pre-launch goal she set in the PRD next to the actual from the first month. Goal-vs-actual is the only honest framing for a metrics review: it tells you what you committed to, not just what happened.

SouthBank bot — month-one dashboard (goal vs actual) Eval pass rate goal 90% 88% Thumbs-up rate goal 70% 76% — beat goal Time-to-task (min) goal 3.5 min saved 4.2 min saved Escalation rate goal <10% 14% — over goal Trailing 4 weeks · n ≈ 41,200 sessions · 28% repeat-user cohort Pre-AI baseline time-to-task: 7.6 min (live agent flow)
Goal (set in PRD) Actual — beat goal Actual — at/near goal Actual — missed goal
What Priya does with this: leads the review with the two greens (thumbs-up and time-to-task), is honest about the one red (escalation), and reframes the eval pass rate as directionally on-target rather than as the headline. The COO walks out with a clear picture; nobody is surprised in month three because nothing was hidden in month one.

How metrics get wired in — before launch, not after

The meta-mistake is treating in-product metrics as something the analytics team will "add later". By "later" the user behaviour has already adapted to the feature, and the only baseline you have is the one with AI in it. Wire metrics into the same PR that ships the feature, not the one after.

1. Define name the success event 2. Wire in product log it on every session 3. Set baseline week-1 number = floor 4. Watch weekly leading first, then lagging 5. Tune prompt · model ↑ in PRD ↑ before ship ↑ first week ↑ Monday standup ↑ on drift

The difference between offline eval and in-product metrics

Offline eval — pass rate on the golden set — is the permission to ship. It tells you the feature behaves correctly on a controlled distribution your team chose. In-product metrics tell you whether real users, asking the questions you didn't think of, actually got their job done.

Three differences worth burning into the team: distribution (golden sets are curated; real traffic is messy), incentives (eval is graded by your team; in-product is graded by an irritated user at 11pm), and cadence (eval runs on every change; in-product runs all the time). Use eval to gate launches; use in-product to decide whether the feature has earned its keep.

Priya's wire-in checklist:
session ID on every turn, an explicit task_started and task_completed event, the human-handoff event, the thumbs-up/down click, latency timestamps, and a flag for "user closed the chat without resolving". That's eight log events. Everything else can be derived from those.

The AI connection

How real AI teams actually run this

Mature AI teams keep two metric stacks side by side. The quality stack — golden-set pass rate, refusal rate, hallucination rate, latency p50/p95 — runs on every model or prompt change before the change reaches users. The product stack — task success, time-to-task, escalation, retention — runs continuously on real traffic. The PM watches both weekly. When the quality stack moves and the product stack doesn't (or vice versa), that mismatch is the most valuable signal in the dashboard. It's where eval design needs to catch up to reality. task-success-rate refusal-rate time-to-task escalation-rate retention-lift offline-vs-online

Mini challenge — wire the metrics for one feature

Pick an AI feature you (or your team) ships

1. Define the success event in one sentence

"The user got X done." Be specific — "answered the question" is too soft; "the user did not need to escalate to support and did not return to ask the same thing within 24 hours" is a real definition. Write it down. If two PMs on your team write different sentences, you don't have a metric yet, you have a debate.

2. Pick three leading and one lagging

From the six in this lesson, choose three that should move within a week if something breaks (task success, time-to-task, escalation, thumbs-up — pick three) plus one lagging metric (retention impact). Write the goal number for each, taken from your PRD if you have one or your gut if you don't.

3. Name the watcher and the cadence

For each metric, write who looks at it and when. "PM, every Monday 9am, in the standup deck" is a metric. "The team will keep an eye on it" is not. If you can't name the human and the time, the metric will quietly stop being watched within three weeks.

Interview questions

Easy What's the difference between an offline eval metric and an in-product metric?
+
An offline eval metric (e.g. pass rate on a golden set of 200 curated questions) is run by your team, on a distribution your team chose, before the change ships. It's the permission to ship — it gates launches. An in-product metric (e.g. task success rate, time-to-task, escalation) is observed on real users in production, on the messy distribution real users actually create. Eval tells you the feature can work; in-product tells you whether it does. Healthy AI teams ship strong on both — and watch the gap between them as a signal, because a big gap means the eval doesn't reflect real traffic.
Easy What's a vanity metric in AI?
+
A vanity metric is one that only moves up — and tells you nothing about whether users actually got their job done. The classic AI vanity metric is thumbs-up rate read in isolation: it's selection-biased (only happy or angry people click), gameable by softening the bot's tone, and silent on the users who quietly stopped using the feature. DAU on the AI surface is another — usage isn't success, especially when "I had to use the bot three times to get one answer" looks the same as "I used the bot three times because it's great". The fix is never to drop the metric; it's to pair it with a counterweight. Thumbs-up paired with task success and retention is fine. Thumbs-up alone is theatre.
Medium How do you connect AI feature metrics to business metrics?
+
Three steps, in order. (1) Pick the business metric the AI feature is supposed to move — at SouthBank that might be "support tickets deflected" (a cost line) or "weekly active users in the app" (a retention line). Don't ship without naming it. (2) Build the bridge metric: the in-product number that should mechanically move the business number. For deflection, it's escalation rate × volume. For retention, it's matched-cohort week-2 retention of AI users vs non-users. (3) Validate the link with a holdout — keep 5–10% of users out of the AI experience for the first quarter and compare. The holdout tells you whether the AI metric and the business metric are actually causally connected, or whether you got lucky on a seasonal trend. If you skip step 3, you'll attribute every win to AI for a year and then fail to defend it when the CFO asks.
Medium Escalation rate is going up — is that bad?
+
Not always. Escalation rising is bad if (a) task success is falling at the same time, (b) the new escalations are on questions the bot used to answer, or (c) handoff messages show frustration ("this bot is useless"). Escalation rising is fine — sometimes good — if you've recently tightened the refusal rules to push borderline cases to humans, or if a new product launch added question types the bot was never trained on. The PM's first move when escalation moves is to slice it: by topic, by user segment, by recency. The headline number is almost never the answer; the slice that's moving is.
Hard Thumbs-up is at 90%, but retention is dropping. What's happening?
+
This is the most common AI dashboard pattern in the wild, and it's almost always one of three things. (1) Selection bias on the thumbs. The 10% of sessions that bothered to click are the ones where the bot was either delightful or catastrophic; the 70% where the user shrugged and left never registered an opinion. The 90% is real, but it's 90% of a small, biased sample. Fix: instrument an implicit success signal (did the user complete the task? did they return to the same question?) and watch that instead. (2) Friction on the unhappy path. The bot is great when it works; on the cases it can't handle, the handoff to a human is broken — long wait, lost context, repeated questions. Users leave silently rather than rate it. Fix: instrument the handoff itself; thumbs-up only rates the AI turn, not the recovery flow. (3) Substitution effect. The bot is solving the easy questions that used to be a reason for users to log in regularly (browse, ask a casual question, feel served). Now they get a quick answer and leave — task success is high, retention drops, both are real, both are caused by the same change. Fix: the AI is doing its narrow job correctly; the product job is to give users a different reason to come back. The wrong move in all three cases is to celebrate the 90% and ignore the retention. The right move is to treat the divergence as the most valuable signal on the dashboard — it's telling you exactly where to look.

Test your knowledge

Question 1 of 5
Which of these is the most honest single-number measure of whether an AI feature is working?
Question 2 of 5
A team reports thumbs-up at 92% but won't show task success rate. What's the most likely problem?
Question 3 of 5
Where does retention impact sit on the leading-vs-lagging × user-vs-business matrix?
Question 4 of 5
Eval pass rate is 88% but in-product task success is only 61%. What does that gap most likely mean?
Question 5 of 5
When should in-product metrics be wired in?
0
/5
You can defend an AI feature on the metrics that matter.

Common mistakes

Reporting eval pass rate as the headline
Pass rate on a frozen golden set is the launch-permission number, not the working-feature number. Show it on slide three, not slide one. The headline belongs to a real-user metric — task success or time-to-task.
Treating thumbs-up as the trust score
Thumbs-up is biased and noisy. It's a directional thermometer, never an absolute thermostat. Pair it with task success and retention, or it will tell you the feature is loved while users quietly stop using it.
Wiring metrics after launch
"We'll add analytics in the next sprint" means you'll never see a clean week-1 baseline. By sprint two the user behaviour has adapted to the feature. Wire the events in the same PR as the feature itself.
Ignoring escalation rate
A bot that solves 50% of cases and escalates 50% can look fine — until support is drowning in the hard half. Always read escalation rate alongside task success and abandonment. The unhappy path is where AI features quietly fail.

Key takeaways

  • Six metrics, not sixty. Task success, refusal rate, time-to-task, thumbs-up, escalation, retention impact. If you can't recite them you don't have a measurement plan.
  • Offline eval is the permission to ship; in-product metrics are proof it shipped well. Watch the gap between them as its own signal.
  • Top-row of the 2×2 (leading × user/business) gets watched weekly. Bottom-row gets watched monthly or quarterly. Don't confuse the two cadences.
  • Thumbs-up alone is theatre; thumbs-up paired with task success and retention is honest. No metric reports alone — every row in the table lies a particular way.
  • Wire the events in the same PR that ships the feature, set the week-one baseline before tuning, and name a human and a cadence for every metric — or it will quietly stop being watched.
  • If the AI feature can't be tied to a business metric (deflected tickets, retained users, GMV lift) with a holdout cohort, you don't yet know whether it's working — you know it's busy.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You now measure AI features that earn trust.
Next — keep them safe and accurate.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →