A task-map you can take into a roadmap meeting. Know before you commit.
🎯 PM🤖 AI-relevant🗺️ Task map⚡ Beginner📖 ~14 min
Lesson 03 of AI Product Manager Track
The question that saves a quarter
Every AI ask from your CEO arrives sounding the same: "Add AI to search." "Let's use AI for fraud." "Can we make the app smarter with AI?" By week three of the roadmap you've discovered two uncomfortable things — half of what you committed to doesn't actually need an LLM, and the other half an LLM can't do without scaffolding nobody scoped.
The skill this lesson teaches is one question you ask before the roadmap slot: is this task one LLMs are actually good at? Get a crisp answer in ten minutes, route the parts that fit to an LLM, route the parts that don't to traditional search, rules, or a flat "no", and you've just bought back a month.
The frame:
an LLM is a confident junior who read the whole internet once, can't remember today's date, and will never tell you they're unsure. Scope every feature with that colleague in mind.
The LLM task map
Plot any candidate task on two axes: how well LLMs suit it and how much business value it creates. Most PM regret lives in the bottom-right — tasks you committed to that look AI-shaped on a slide but fall apart in production. Your job is to push everything into the top-left quadrant, or push it off the roadmap.
How to read this: top-right is where you push. Top-left is where LLMs are a feature of the feature (rewrite the query, explain the result — the core still needs classical systems). Bottom-right is what kills PMs — a shiny demo that makes no revenue move. Bottom-left you kill on sight.
Where LLMs actually shine
Six task shapes LLMs handle with minimal scaffolding. If your feature reduces to one of these, zero-shot is usually enough. These are the tasks that pay the rent.
📝
Summarize
Condense 500 product reviews into "fits true to size, runs narrow at the heel". Shape: long text in, short text out. Very reliable.
🏷️
Classify
Put a short piece of text into one of N buckets. "Is this query about delivery, sizing, or returns?" Beats regex + keyword on messy user text.
🔍
Extract
Pull structured fields out of unstructured text. Invoice total, shipping address, warranty date from a PDF. Cheap and accurate with good examples.
✏️
Rewrite
Turn "red shoe size 42 mans" into a clean search query. Turn a customer rant into a polite ticket. Same meaning, different shape.
🌍
Translate
Nigerian-Pidgin to English, Swahili to English, casual to formal. Excellent for major languages, shakier for niche dialects.
📚
Answer-from-context
Given a passage, answer questions about it. The beating heart of every RAG product. Strong when grounded, dangerous when ungrounded.
The pattern:
every green task is "short text out, grounded in text in". Anything that fits that template has a high baseline success rate. Anything that doesn't is a project, not a prompt.
Where LLMs wobble or break
Six task shapes where an LLM alone will disappoint you. You can still ship these features — you just can't ship them with an LLM as the core. The LLM plays a supporting role, or none at all.
📰
Fresh facts
"What's the price right now?" "Is this item in stock?" Training data is frozen. Without retrieval or tools, expect confident fiction.
🧮
Precise math
Multi-step arithmetic, currency conversion, tax calculations. LLMs stumble at small numeric precision. Call a calculator tool, don't prompt-and-pray.
🧩
Multi-step reasoning
"Plan a 3-stop delivery route that minimises fuel." LLMs can mimic reasoning but drop constraints silently. Use them to orchestrate real solvers, not replace them.
📏
Enforceable rules
"Never show products above ₦50,000 to users flagged low-limit." A prompt is a suggestion. A rules engine is a contract. Don't confuse them.
🗂️
Long-tail entity memory
Every SKU you stock, every seller you onboarded last week. The model doesn't know your catalog — your database does. LLM + retrieval, not LLM alone.
🚫
Hard refusals
Regulatory "you must not answer" cases. System prompts degrade. A deterministic filter outside the LLM is the only thing auditors accept.
The recurring error: PMs assume "newer model = solved". Frontier models move the red tasks from "broken" to "wobbly" — they never move them to "solid". Design for the wobble.
Scenario — Amara has 10 weeks and a CEO mandate
🛒
Real-world scenario
NaijaCart Nigeria wants "AI-powered product search" in the app by Q3
Amara is the search PM at NaijaCart's Lagos office. The CEO just returned from a fintech conference and posted in the #product channel: "Every competitor has AI search. We need it in the app by Q3." Amara has 10 weeks, two backend engineers, one data scientist shared with the fraud team, and a cost budget that boils down to ₦15 per search session.
When she sits down to scope, "AI-powered product search" explodes into at least six tasks: query rewriting ("red shoe 42 mans" → clean query), intent classification (is this a product query, a delivery-status question, or a complaint?), product re-ranking across 2 million SKUs, surfacing a one-line review summary beneath each result, explaining a price drop to the user, and filtering out fraudulent listings.
The uncomfortable thing she realises on Tuesday: only two of the six tasks are clean LLM work. Two need an LLM plus classical infrastructure, and two shouldn't touch an LLM at all. If she prompts her way through all six, she'll ship a demo that wobbles at launch, miss the fraud SLA, and blow past ₦15 per session by week two.
Amara's task-by-task triage
Before she writes a single PRD line, Amara builds the table below. One row per task. One column for the LLM verdict. One column for the right tool. One column for notes her engineers will actually use. This is the artefact that turns a fuzzy CEO ask into an honest scope.
Hard refusal territory. LLM can assist as a second opinion, but the blocking decision must be deterministic and auditable.
The honest scope:
out of six tasks, two are pure LLM, two are LLM + classical scaffolding, two should be rules or classical ML. Amara now has a defensible answer for her CEO and her engineers — she's not shipping "AI search", she's shipping six distinct features with three different underlying technologies.
The triage workflow, drawn
The map Amara sketches on her whiteboard for every AI ask — and the one you should internalise before the next roadmap cycle.
The AI connection
How real AI teams triage tasks
Inside Anthropic, OpenAI, and every serious AI product team, new feature proposals get shoved through the same mental filter: is this task zero-shot (prompt only), few-shot (prompt with 3–5 examples), RAG-shaped (needs grounded retrieval), or tool-shaped (needs a calculator, a database, a real-time API)? A PM who can name which category a task lives in before the meeting starts is the PM whose roadmaps ship on time. The same triage map is what NaijaCart, BoraPay, and NairaSwift teams keep pinned next to their Figma boards.
zero-shotfew-shotRAGtool-usetask-triage
Mini challenge — triage five asks before lunch
Turn any AI request into a routed plan in 20 minutes
1. Collect five "add AI" asks from your own backlog
Pull the last five AI-flavoured requests from your Slack, Jira, or leadership emails. Write each as a one-verb, one-object sentence. "Summarize invoices." "Classify support tickets." "Answer billing questions." If you can't reduce it to that, it's not one task — it's three.
2. Place each on the 2×2
For each task: pick a quadrant (ship now, wrap, demo trap, ignore). Be honest about business value — "the CEO would like it" is not business value. If two asks land in demo-trap, you now have material for a "what we're not building, and why" doc.
3. Write the verdict column
For each "ship now" or "wrap" task, fill the triage table: LLM? · Right tool · PM notes. Three columns, one row per task. Send it to your eng lead with the subject "before we commit to the roadmap". Watch how fast the fuzzy AI ask sharpens into a real plan.
Interview questions
EasyName three task shapes LLMs handle well zero-shot.
+
Summarize (many reviews to one line), classify (query into one of N buckets), extract (pull structured fields from unstructured text). All three share the same template: short text out, grounded in text in. Two more equally strong: rewrite (clean up messy queries) and answer-from-context (read a passage, answer about it).
EasyWhen would you NOT reach for an LLM?
+
Four flags: (1) the answer depends on data fresher than the training cutoff and you don't have retrieval, (2) the task demands numeric precision or a multi-step plan that must be correct, (3) the feature must enforce a rule (regulatory, tier-gated, safety), not suggest one, or (4) the business already has a keyword, regex, or classical ML baseline that works at 95%+. In that last case the LLM is a cost increase, not a quality increase.
MediumA stakeholder wants LLMs for everything. How do you push back without sounding like a blocker?
+
Don't say no — re-scope. Take their ask ("AI-powered search") and break it into underlying tasks on a whiteboard. For each task, mark LLM-suited, LLM-plus-tool, or not-LLM. Now you're having a technical conversation, not a political one. The stakeholder sees exactly which parts get AI and which don't, and why. Close with "here are three LLM tasks we can ship in 10 weeks; the other three are a separate project" — that's a yes, not a no, and it keeps the roadmap honest.
MediumWhat's the difference between a task an LLM can do zero-shot vs one that needs fine-tuning?
+
Zero-shot works when the task is generic and well-represented in the model's training — classification into common categories, general summarization, mainstream translation. Fine-tuning earns its keep when (a) you have a domain vocabulary the model doesn't know well (niche legal terms, product SKUs, internal jargon), (b) you need a very specific output format at high reliability, or (c) you have 10k+ labelled examples that encode tacit knowledge. Rule of thumb: try zero-shot, then few-shot (3–5 examples in the prompt), then RAG, then fine-tune. Fine-tuning is the last 10% of quality for 10x the ops cost — earn it.
HardYour CEO demos an LLM answering live product-availability questions perfectly on three examples and wants it launched next sprint. What do you do in the next 24 hours?
+
This is a classic demo-trap. Three cherry-picked examples don't reflect the long tail — and "live product availability" is textbook "fresh facts" territory, which LLMs break on without grounding. In 24 hours: (1) run 50 real queries drawn from yesterday's logs, report pass rate honestly (it will drop to 40–60%); (2) diagnose — is the model hallucinating stock levels? Inventing SKUs? Stale pricing? (3) come back with two options — option A is the LLM as a natural-language shell over a real inventory API (LLM rephrases, the API answers); option B is full-LLM with a hard refusal rule for availability questions. Option A is the right answer 9 times out of 10. The answer your CEO wants to hear is not "we can't do it" — it's "we can do it, and here's how, and it takes 10 weeks not 2." Never let a three-example demo become a production commitment.
Test your knowledge
Question 1 of 5
Which task is LLMs' natural sweet spot (high fit, minimal scaffolding)?
Question 2 of 5
Amara's "price-drop explanation" feature should be built as:
Question 3 of 5
Your CEO wants LLM-based product search across 2M SKUs. The correct scoping move is:
Question 4 of 5
Which is a "red" task — where an LLM alone will likely disappoint?
Question 5 of 5
A CEO demo shows the LLM getting 3/3 product-availability questions right. Correct response?
0
/5
You can now triage any AI ask before it eats your roadmap.
Common mistakes
Using an LLM for hard math
GST, tax, currency, inventory counts. LLMs drift at 3 decimal places and don't flag it. Call a calculator tool and let the LLM phrase the result.
Asking for real-time data without tools
"Is this in stock?" "What's today's price?" The training cutoff lies frozen. Without retrieval or an API call, the model will invent a confident answer.
Skipping the baseline
Regex, keyword search, and a 100-line rules file often beat the LLM at 10x the speed and 100x the cost efficiency. Build the classical version first; LLM is the upgrade, not the default.
Assuming a demo = a production feature
Three cherry-picked examples hide the long tail. A 90% demo becomes a 55% launch. Always re-run on 50+ real queries before committing a sprint.
Key takeaways
LLMs shine at summarize, classify, extract, rewrite, translate, answer-from-context — short text out, grounded in text in.
They wobble or break on fresh facts, precise math, multi-step reasoning, enforceable rules, long-tail entity memory, and hard refusals.
Before any "add AI" ask, triage each underlying task into one of three routes: pure LLM, LLM + tool/RAG, or no LLM at all.
A demo is not a product. Three cherry-picked examples hide the long-tail failures that show up in week one of launch.
The cheapest AI win is usually the LLM-as-shell pattern — classical system does the work, LLM handles the language around it.
Frontier models move red tasks from broken to wobbly, never to solid. Design for the wobble.