Why "seems to work" is not a metric

You build an agent that calls a ticker lookup tool. You test it manually on RELIANCE.NS, it returns price and news—looks good. Ship it. Then a user asks for SBI.NS, and your agent crashes because the tool endpoint timeout logic wasn't tested. Another user asks for an obscure ticker; the agent hallucinates a stock price instead of saying "I don't know."

📈
Real-world scenario
Arjun (Hyderabad): manual testing is not testing

Arjun ships a ticker agent for SouthBank India's retail trading platform. On his machine, it works fine. But in production, 15% of user requests fail or return wrong data. Why? Because he never built a systematic test set. He never measured latency or token costs. He never asked: "What should my agent return for ticker X, and did it?"

He rewrites the agent with a 20-ticker eval set, deterministic scoring (price match, news count, sentiment accuracy), and LLM-as-judge grading. Now he catches regressions before they ship. Every time he tweaks the system prompt, he runs the eval suite. Score goes from 65% to 78% in two weeks.

Evaluation is how you separate "my agent works on my test" from "my agent works reliably." It's the backbone of any AI system that touches production.

Four types of agent evaluation

Not all tests are equal. Build a stack:

Type Method When to use Cost
Unit tests Test each tool independently: does `get_price(RELIANCE)` return a dict with price, symbol, timestamp? Every tool change. Catches silent failures. Zero (no LLM calls)
End-to-end Feed the agent a user query, check final output against ground truth After system prompt changes. Measures real user experience. 1 LLM call per test case
Deterministic scoring Exact match, substring match, numeric tolerance, schema validation When you have ground truth (price must be between ₹2500–2600) Zero (rule-based)
LLM-as-judge Use a stronger model (Gemini, GPT-4) to grade your agent's output When ground truth is fuzzy ("Does this summary capture the key point?") 1 judge call per test case

A robust eval harness uses all four. Start with unit tests (free), then add end-to-end tests (cheap), then layer in deterministic scoring and LLM judges for subjective cases.

Step 1: Build a realistic test set

Your test set is a list of (input, expected_output) pairs. For a ticker agent, that might be 20 tickers with known prices, news counts, and sentiment. For a customer support agent, it might be 30 customer emails with known categories and priorities.

test_set.py Define a test set for a ticker briefing agent

Key principles for test set design:

  • Realistic scope. 20–50 test cases. Small enough to run in seconds, large enough to catch patterns.
  • Known outputs. For each test case, you must know what a correct answer looks like.
  • Boundary conditions. Include edge cases: empty results, timeouts, ambiguous tickers (is "TCS" unambiguous?), malformed inputs.
  • Representative diversity. Don't test only the stocks you're familiar with. Include rare tickers, cross-sectors, unstable prices.

Step 2: Deterministic scoring (rule-based, zero LLM cost)

When you have ground truth, score with rules. No LLM needed. Fast and reproducible.

eval_harness.py Deterministic scoring functions

The scoring rubric is tuned to your domain. For financial data, price tolerance matters. For news summaries, you might score on keyword presence or sentiment accuracy. The key: each rubric item is deterministic.

Step 3: LLM-as-judge for subjective quality

When ground truth is fuzzy—"Is this summary good?"—ask a stronger LLM to grade. Use a model like Gemini 2.0 Flash (faster, cheaper) to judge your Ollama agent's outputs.

judge.py Grade agent outputs with an LLM judge

LLM judges are expensive (1 judge call per test case), so use them sparingly. Combine with deterministic scoring: run deterministic tests first (free), then judge only the borderline cases.

Step 4: Measure cost and latency

Wrap every agent call with time and token accounting. Over time, you'll see patterns: "This system prompt causes 2x token usage, no quality gain."

metrics.py Track latency and token usage for each eval run

Track these in a spreadsheet or database. Over 10 eval runs, you'll see cost drift. If your average tokens per call went from 150 to 300, your system prompt change doubled token usage. This is actionable feedback.

Step 5: Regression testing—rerun eval suite on every change

Every time you change the system prompt or add a tool, rerun the full eval suite. Track results in a log. If the score drops, you caught a regression before it ships.

Sample regression log
Run 1: 74% | Run 2 (new prompt): 68% | Run 3 (reverted): 74%
Action
Revert prompt, investigate. 6% drop is significant—not worth it.

Keep a CSV log:

Timestamp Change Avg Score Avg Latency (ms) Avg Tokens
2026-04-01 10:00 Baseline 0.74 245 180
2026-04-01 11:30 Add sentiment check 0.81 268 195
2026-04-01 14:00 Verbose system prompt 0.68 450 380
2026-04-01 15:00 Revert to add sentiment 0.81 268 195

Worked example: Full eval harness for ticker briefing agent

Let's build a complete eval pipeline for the Lesson 05 ticker agent. We'll define 10 tickers, score deterministically, run the agent, and measure quality.

full_eval.py Complete eval harness for ticker briefing

Step 6: A/B testing prompt variants

Build two system prompts. Run them against the same test set. Which one scores higher?

ab_test.py A/B test two system prompts

When to trust LLM-as-judge vs when to need humans

Scenario Trust LLM Judge? Also need humans?
Pricing accuracy
("Is price within ±2%?")
✗ No Deterministic rule only. LLM might hallucinate.
News relevance
("Does news relate to the company?")
✓ Yes LLM is good at this. Sample human review monthly.
Tone/sentiment classification
("Is the briefing friendly?")
~ Moderate Use 2-judge approach: Claude + GPT. Agree = trust.
Format correctness
("Is JSON valid? All fields present?")
✗ No Use Pydantic schema validation. Rule-based only.
Safety/toxicity
("Does briefing contain slurs?")
✓ Yes, but sample Judge 20% of test set weekly. Audit failures monthly.

Rule: Use LLM judges for subjective quality (relevance, clarity, completeness). Use deterministic scoring for objective facts (format, price range, field presence). When stakes are high (financial, healthcare), always include human audit loop.

Mini Challenge: Build a 5-ticker eval harness

Task: Define a test set for 5 stocks (RELIANCE.NS, HDFCBANK.NS, INFY.NS, TCS.NS, SBIN.NS) with price ranges and minimum news counts. Write a scoring function that gives each test case a 0–1 score based on price accuracy and news count. Run your ticker agent from Lesson 05 on all 5 tickers and report average score and latency.

Hint 1: Use the test set structure from this lesson. Define price_range and min_news per ticker.

Hint 2: Scoring: 0.5 points for price in range, 0.5 for news count >= min_news.

Hint 3: Wrap agent calls with time.time() to measure latency.

Interview Q&A: Agent Evaluation

Q1 (Easy)
Why is manual testing insufficient for agents?
Answer
Manual tests cover only the happy path. Edge cases (rare tickers, timeouts, malformed inputs) are missed. Systematic test sets catch regressions before they reach users.
Q2 (Easy)
What's the difference between deterministic scoring and LLM-as-judge?
Answer
Deterministic scoring uses rules (price in range? news count >= 3?). LLM judges answer fuzzy questions ("Is this summary clear?"). Deterministic is free and reproducible; LLM judges are expensive but good at nuance.
Q3 (Medium)
You change your agent's system prompt. Avg score drops from 78% to 71%. What do you do?
Answer
Regression detected. Analyze which test cases failed. Did the new prompt help on some tickers but hurt others? If the quality drop is not worth it, revert. If it improves quality on critical cases, investigate the failures and iterate.
Q4 (Medium)
How do you know if your test set is good?
Answer
Good test sets catch regressions and reflect real usage. If your test set never fails, it's too easy. If every change fails a few tests, that's good signal. Audit 10% of test cases monthly with humans.
Q5 (Hard)
Your agent scores 92% on your test set but users report 60% of requests fail. Why?
Answer
Test set doesn't match real distribution. Users ask for tickers not in your 20-ticker set, or they use ambiguous queries ("tech stock today"). Collect real user queries, add them to test set, retrain eval. This is why production monitoring is essential.

Test your knowledge

Question 1 of 5
What is the main risk of only manual testing for an agent?
Question 2 of 5
When should you use deterministic scoring vs LLM-as-judge?
Question 3 of 5
What does a good test set contain?
Question 4 of 5
What is regression testing in the context of agents?
Question 5 of 5
Your agent scores 90% on your eval set but users report 50% failure rate. What's most likely?
0
/5
Great! You understand agent evaluation. Now build reliable systems.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You've built eval harnesses.
Now ship agents that scale.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →