What a model card actually is

A model card is a vendor's self-published spec sheet for an LLM. OpenAI, Anthropic, Meta, Google, Mistral — they all ship one with every release. Think: a 15–30 page PDF, a GitHub README, or a long blog post. It lists what the model can do, what it shouldn't do, how it was trained, and what evals the lab ran.

PMs treat model cards like they treat an RFP response: part truth, part marketing, all legally-careful. You don't read it cover to cover. You go in with a checklist, pull three numbers, ignore most of the narrative, and come out with an answer to the only question that matters: does this model clear my product's quality gate at a price I can pay?

The rule:
the vendor is graded on their own benchmarks. You're graded on yours. Never conflate the two.

The six things a PM extracts

A model card has fifty sections. Six of them change your decision. Screenshot this grid and paste it into your next vendor eval doc.

🧠
Capabilities & intended use
What the lab says it was built for — chat, tool-use, coding, RAG. Skip if your use-case isn't on the list.
📏
Context window
Max tokens in + out. Caps how much of your knowledge base and conversation history you can stuff in a prompt.
📅
Training cutoff
The date its knowledge stops. Ask a model with a 2024-06 cutoff about this year's tax slab changes and it will invent.
⚖️
License & data terms
Commercial use OK? Outputs yours? Can they train on your prompts? One line that can kill your launch.
📊
Eval numbers
MMLU, GSM8K, HumanEval, MT-Bench. Vendor-selected, vendor-scored. Useful as a floor — never as a verdict.
🛡️
Safety & limits
Known failure modes, refusal behavior, jailbreak tests, bias cards. The section PMs under-read the most.
What you ignore: architecture diagrams, parameter counts, training GPU hours, tokenization notes. Fun trivia, zero PM decisions.

Three candidates side by side

A PM's model card reading never happens for one model in isolation. You're always comparing. Below is the exact grid you should build for any shortlist — these illustrative numbers reflect the shape of what you'd pull in April 2026.

Dimension gpt-4o-mini claude-haiku Llama-3-70B (open-weight)
Context window 128k tokens 200k tokens 128k tokens
MMLU (vendor-reported) 82.0% 80.1% 82.9%
Price per 1M tokens (in/out) $0.15 / $0.60 $0.25 / $1.25 $0.60 / $0.60 (self-host; GPU infra)
Knowledge cutoff Oct 2023 Apr 2024 Dec 2023
License Commercial via API. Outputs yours. Retained 30d for abuse only. Commercial via API. Outputs yours. Zero retention option. Llama 3 Community License. Commercial OK <700M MAU. Derivative models named "Llama-…".
Refusal behavior Refuses with short template. Can be system-prompted to redirect. Refuses with reasoning. Harder to jailbreak; sometimes over-refuses tax-law edge cases. Weaker default refusals. You ship your own guardrails.
Best-fit signal Cheap high-volume Q&A, strong tool-use Long docs, nuanced reasoning, safety-sensitive Data residency, fine-tune control, zero per-token fees
How to actually fill this in:
open the three cards in tabs, ctrl-F for "context window", "cutoff", "license", "MMLU". Takes 10 minutes. The remaining 20 pages are for your ML teammate, not you.

Scenario — Rohan has 10 weeks and three cards

🧾
Real-world scenario
TaxSetu wants a tax Q&A bot live before the July 31 ITR deadline

Rohan is the PM for TaxSetu's new AI tax assistant out of Bangalore. The brief: handle 70% of basic taxpayer questions ("Can I claim HRA if I own a house in another city?") so human CAs can focus on complex filings. He has 10 weeks to ship, one backend engineer, one ML engineer, and a legal-risk budget that is essentially zero.

Three model cards are open on his monitor: gpt-4o-mini, claude-haiku, and Llama-3-70B. His ML engineer keeps asking "which one?" and Rohan keeps stalling because every card is 25 pages long and half of them are architecture diagrams he doesn't need.

The uncomfortable thing he realises by Tuesday: none of the three cards actually tell him whether the model knows Indian tax law as of April 2026. All three cutoffs are 2023-ish. The answer to "which model" is not in the cards — it's in what he'll build on top of them.

The 10-minute read, step by step

Here is the pass Rohan actually runs. No code. No calls to his ML engineer. Just three browser tabs and a notepad.

Step 1 · Filter

🔎 Skim the "intended use" paragraph

If the card lists your use-case (Q&A, RAG, tool-use) and your domain risk-class isn't on the "not suitable for" list, keep the tab open. Otherwise close it. For TaxSetu, all three pass this filter — but if Rohan were building medical triage, claude-haiku's card explicitly says "not for clinical decisions" and that tab closes in 30 seconds.

You extract

  • Intended use list
  • "Not suitable for" list
  • Any regulated-industry warnings

You ignore

  • Marketing sentence about "state-of-the-art"
  • Architecture overview
  • Parameter counts
Step 2 · Constraints

📐 Grab context window, cutoff, and license

These are three ctrl-F searches. Rohan writes them in a sticky-note sheet. The cutoff matters most for TaxSetu: tax law changes every union budget. If the cutoff is more than 12 months old, he knows before he runs a single eval that he'll need RAG over the latest IT-Act amendments — no matter which model wins.

You extract

  • Context window (tokens)
  • Training cutoff (month/year)
  • Commercial-use clause
  • Data retention terms

You ask the ML team

  • "Does the API let us disable logging?"
  • "What happens if we hit the context ceiling?"
  • "Any enterprise contract that overrides the public card?"
Step 3 · Evals

📈 Read vendor evals as a floor, not a verdict

The card will show MMLU, GSM8K, HumanEval, MT-Bench, maybe a new in-house one. Rohan reads these only to confirm the model is in the right class. 82% on MMLU vs 80% on MMLU does not tell him anything about Indian tax questions. He trusts one chart only: his own golden-set pass rate (Lesson 10 — evals 101).

He builds a 30-question TaxSetu golden set from real historical queries, runs all three models, and gets this:

TaxSetu 30-question golden-set pass rate gpt-4o-mini 56% claude-haiku 67% Llama-3-70B 50% ← Rohan's launch gate: 70%
What Rohan does with this: nobody clears his gate yet. That is not a model choice — it's a signal that none of them know current ITR rules. The real next move is RAG over the updated Income Tax portal, not another model.

Mapping the three candidates on cost vs quality

Once the golden-set numbers are in, Rohan drops the three candidates into a 2×2. This is the diagram he takes to his Friday review meeting — one slide, one decision.

↑ Higher quality on your task Lower quality on your task ↓ Cheaper Pricier Sweet spot high quality, cheap Premium tier ship for hard cases only Retire cheap but doesn't clear gate Money pit expensive and bad fit gpt-4o-mini 56% · cheapest claude-haiku 67% · mid-price Llama-3-70B 50% · infra-heavy
Rohan's call: ship claude-haiku + RAG for the hard tax reasoning, route simple queries ("what's the last date to file ITR?") to gpt-4o-mini via a router. Keep Llama-3 in the back pocket for a future on-prem enterprise tier, not for v1.

The AI connection

How real teams actually use model cards

Inside Anthropic, OpenAI, and Google, the card is a compliance and trust artefact — written by a joint policy + research team, reviewed by legal. On the outside, AI PM teams at companies like TaxSetu, NaijaCart, and BoraPay keep a living "model shortlist" spreadsheet that merges card facts with their own golden-set pass rates. When a vendor ships a new version, the spreadsheet gets a fresh row, not a fresh debate. model-card knowledge-cutoff license golden-set

Mini challenge — read three cards in 30 minutes

Pick three models, extract six rows each

1. Open the cards, set a 10-minute timer per model

Go to the official model card page for gpt-4o-mini (OpenAI), claude-haiku (Anthropic), and Llama-3-8B (Meta). Ctrl-F your way to context window, cutoff, license, MMLU, safety notes. Paste six bullets each into a doc. No reading paragraphs.

2. Flag the one license clause you'd email legal

Every card has one line that would make your GC wince. Find it in each. Hint: look for retention, training on inputs, named-entity restrictions, or MAU caps. One sentence per model — that's the email you'd send.

3. Write the three questions you'd ask your ML team

Turn the unknowns into PM-shaped questions: "What's our fallback if the context window is exceeded?" "Can we pin this version?" "How do we prove the cutoff-to-today gap doesn't hurt us?" Bring them to your next standup. You've just done AI PM reading comprehension.

Interview questions

Easy What are the six things a PM extracts from a model card?
+
Capabilities & intended use, context window, training cutoff, license & data terms, vendor eval numbers, and safety / known-limits. Architecture, parameter counts, and training hardware are noise for a PM. The mnemonic: C-C-C-L-E-S (capabilities, context, cutoff, license, evals, safety).
Easy Why doesn't MMLU tell you whether a model is good for your product?
+
MMLU is a general-knowledge multiple-choice benchmark. It does not resemble a TaxSetu user asking about HRA exemptions, a NaijaCart customer asking where their order is, or an BoraPay user asking why a transaction failed. Two models within a few points of each other on MMLU can differ by 15+ points on your golden set. MMLU is a floor — use it to reject models that are clearly weaker, never to pick the winner.
Medium Your vendor's card says "knowledge cutoff: Oct 2023". Your product launches in May 2026. What do you do?
+
First, list everything in your domain that changed after Oct 2023 — for TaxSetu that's every union budget, every new ITR form, every Section 87A tweak. Second, decide whether those changes are in-scope for the questions users will ask. If yes (they almost always are), the model alone can't answer; you need RAG over a fresh, authoritative source. Third, write the refusal rule: if the user's question touches a post-cutoff topic and retrieval returns nothing, refuse and hand off. The cutoff isn't a dealbreaker — it's a design constraint you must name.
Medium Open-weight vs API vendor — what changes in how you read the card?
+
API vendor cards lean on safety evals and usage policy — you inherit their guardrails and liability shell. Open-weight cards (Llama, Mistral) lean on architecture and fine-tune recipes — you inherit the model but own the hosting, filtering, abuse policy, and refusal behavior. The PM read shifts: for API, focus on license retention clauses and enterprise data terms; for open-weight, focus on the community license fine print (MAU caps, derivative naming) and the fact that you're now responsible for every safety eval the vendor didn't run.
Hard A vendor releases a new version of the model you already ship on. Their card claims "20% better on reasoning". Walk me through the exact 48-hour decision process before you decide to upgrade.
+
Hour 0–2: diff the new card vs the old one — cutoff, context window, license, refusal behavior, known limits. A silent refusal-policy change is what actually breaks your prompt, not the headline reasoning number. Hour 2–6: run your golden set against both versions in shadow mode (Lesson 10), report pass rate, refusal rate, cost, and P95 latency. Hour 6–12: diff cost. If the new one is pricier, compute the monthly delta in rupees — ₹0.4/conversation × 200k/month = ₹80k matters. Hour 12–24: ship a 1% canary and watch thumbs-down + task completion. Hour 24–48: if golden-set + canary both clean, ramp to 10%, then 100%. Never upgrade on the card's number alone — the card is marketing, your eval is the verdict. Also: pin both versions so you can roll back in one config flip if production smells off.

Test your knowledge

Question 1 of 5
Which of these is least useful to a PM reading a model card?
Question 2 of 5
A model card shows 82% MMLU. What does that tell you about its fit for your product?
Question 3 of 5
TaxSetu's target model has a knowledge cutoff of Oct 2023 and ships in May 2026. The right response is:
Question 4 of 5
Open-weight Llama vs API claude-haiku — which PM question shifts the most?
Question 5 of 5
A vendor's new model card promises "+20% on reasoning". You should:
0
/5
You can now read a model card without falling for the marketing.

Common mistakes

Trusting the vendor's benchmark table
MMLU, GSM8K, MT-Bench are selected by the lab, scored by the lab, and cherry-picked to flatter the release. Floor, not verdict.
Ignoring the training cutoff
Cutoff older than 12 months = guaranteed RAG + refusal work. Ship without it and you'll hallucinate last year's tax slab to real users.
Reading only the first page
The first page is a press release. The decision-critical stuff — retention clauses, known-limits, refusal behavior — lives 15 pages deep.
Skipping the safety card
Jailbreak evals, bias tables, refusal logs — the section most PMs skim. It's also where your legal & PR exposure is hiding.

Key takeaways

  • A model card is a vendor spec sheet — half truth, half marketing. Read with a six-item checklist, not cover to cover.
  • Extract: capabilities, context window, cutoff, license, vendor evals, and safety notes. Ignore params, GPU hours, and architecture.
  • Vendor MMLU is a floor. Your golden-set pass rate is the verdict. Never confuse the two.
  • Training cutoff drives more product decisions than anyone admits — it usually mandates RAG plus a refusal rule.
  • Always read cards as a shortlist comparison (three tabs, 10 min each). Single-model reads breed model-crush, not PM decisions.
  • The card gives you questions for your ML team. The answers — not the card — pick the model.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You can now read any model card.
Now pick the right one for your product.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →