The problem: silent failures
An agent fails. The user sees "Error." You get a log line: "Agent crashed." That's it. You don't know which LLM call failed. You don't know what tokens cost. You don't know if it was a hallucination or a real error. You certainly can't replay the failure and debug it.
In production—especially in Kenya and India on 3G networks where latency spikes matter—agents fail in subtle ways. A tool call hangs. An LLM response is truncated mid-token. The trace disappears. If you haven't instrumented your agent with structured logging and tracing, you're flying blind.
This lesson teaches you to see everything: every LLM call, every token, every tool invocation, every error with full context.
Structured logging with Python's logging module
The foundation is Python's built-in logging module with a JSON formatter. Instead of free-text logs like "Agent thought for 2.3 seconds," log structured JSON: one log line per action, parseable by machines.
One log = one action. Log when:
- An LLM API call starts (model, tokens)
- An LLM API call finishes (latency, cost)
- A tool call starts (name, args)
- A tool call finishes (result, duration)
- An error occurs (stack, context)
Logging every LLM and tool call
Wrap your LLM and tool calls with logging. Capture what went in, what came out, how long it took, and how much it cost.
Distributed tracing: one trace per user turn
Logs are events. Traces are the full call graph. One trace correlates all logs (LLM, tool, errors) into a single narrative. Assign a trace ID at the start of a user turn, pass it through every call, and you can replay the entire failure.
Trace anatomy:
- Trace ID: UUID for the user turn (e.g., "turn-2026-04-16-a7f3d8")
- Parent span: The agent's decision-making (e.g., "agent_turn")
- Child spans: LLM call, tool 1, tool 2, error (if any)
- Attributes: model, tokens, latency, cost, result
You can build traces with OpenTelemetry (the open standard) or a homegrown Tracer class. Here's a simple tracer:
Real scenario: replay a failed trace
A customer on 3G in Nairobi asks: "Transfer ₹500 to my friend's account." The bot should reject it (wrong currency). Instead, it says "Done." A trace shows:
- LLM turn 1: "I need the account number" (correct)
- Customer: "Send to 254-XXXXX" (provides number)
- LLM turn 2: Latency 8,500ms (high—network lag?)
- LLM output truncated at "I'll transfer KSh 500 to ..." (hung mid-response?)
- Agent assumes it succeeded and logs "Transfer complete" (hallucination!)
Without traces, you'd never know the response was truncated. With traces + logs, you see the 8.5s latency spike and the incomplete response. You can reproduce it locally, fix the network timeout, and deploy.
PII redaction: scrub before logging
Never log phone numbers, emails, or account IDs directly. Redact them before sending logs to external systems.
Free tools: Langfuse, Weights & Biases, ELK stack
Logging and tracing are useless without a backend to store and query them. Here are three free options:
| Tool | Cost | Best for |
|---|---|---|
| Langfuse (open source) | Free / self-hosted | LLM tracing, cost tracking, replay UI |
| Weights & Biases | Free tier + paid | Experiment tracking, dashboards, real-time monitoring |
| ELK Stack (Elasticsearch + Logstash + Kibana) | Free / self-hosted | Raw log storage, ad-hoc queries, full-text search |
| CloudWatch (AWS) | Pay-as-you-go (~$0.50/GB) | If you're already on AWS, built-in integrations |
For startups in Kenya and India, self-hosted Langfuse or a local ELK stack saves costs and keeps data private.
Token and cost dashboards with pandas
Write a simple Python notebook to group logs by model, day, and agent. Spot trends. Catch runaway costs.
Why observability matters in Kenya and India
On 3G networks with variable latency, LLM calls can hang. Traces reveal:
- Which calls timeout first (hint: it's often the longest prompts)
- Latency percentiles (p50, p99) by time of day
- Partial responses (e.g., LLM response cut short mid-sentence)
- Tool call failures due to network interrupts
With traces, you can set intelligent timeouts: if an LLM call exceeds p99 latency + 2σ, fallback to a cached response or escalate to a human.
How AI systems use observability
Large language models are black boxes. They hallucinate. They fail silently. Observability is your window into the model's reasoning. Every major AI company (OpenAI, Anthropic, Google) logs and traces internally. Langfuse was built by engineers who realized every LLM app needs traces. Structured logs + traces are non-negotiable for production agents.
Task: Take the agent from Lesson 18 (multi-turn chat). Add structured JSON logging for:
1. When an LLM call starts (model name, num messages)
2. When an LLM call ends (latency, token counts)
3. When a tool call starts/ends (tool name, result)
4. Any errors (exception message, context)
Run the agent and verify the logs are valid JSON on each line.
Hint: Use the JSONFormatter from structured_logging.py. Wrap your LLM call with logger.info(..., extra={...}). Test by piping output through `jq` or parsing in Python: `json.loads(line)`.
Test your knowledge
You've learned to see into agents.
Now deploy them with confidence.
Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Agentic Engineer →