The two core problems: context limits and hallucination

By Lesson 10, your agents had tools. They could fetch live stock prices, current weather, real-time search results. But what about static documents? A customer-support bot needs to read the company's policy PDF. A research agent needs to analyze a 10-Q filing. A loan officer's assistant needs to reference the bank's lending guidelines.

The naive approach: dump the entire document into the prompt. But this hits two walls instantly.

📄 Full PDF (50 pages)
→
⚠️ Context limit exceeded
→
❌ Agent fails

Problem 1: Context window limits. An LLM has a fixed input size measured in tokens. Claude can handle 200K tokens (roughly 150,000 words). Ollama's 7B model is more like 4K tokens (3,000 words). A typical 100-page PDF is 30,000+ tokens. You can't fit it. And even if you could, there's a deeper issue:

Problem 2: Hallucination. LLMs confidently invent facts when they don't know the answer. Ask Ollama "What is SouthBank India's policy on loan-to-value ratios?" and it will give you a plausible-sounding answer—completely made up. It has no access to the policy PDF. It's just predicting what a reasonable answer might sound like.

🏦
Real-world scenario
SouthBank India customer support bot
A customer in Mumbai calls asking, "What's your policy on cashback for international transactions?" The agent (Ollama LLM) has never seen the actual policy PDF. So it invents: "We offer 2% cashback on international purchases." But the real policy says 1%. Damage: lost customer trust. Revenue: negative.

Context window limits: why you can't dump everything

A context window is the maximum number of tokens an LLM can read. Think of it as the model's working memory. You send in a prompt (tokens), the model reads it, and returns a completion (more tokens). The total—prompt + completion—must fit in the context window.

Ollama 7B
4K tokens (~3K words)
Ollama 13B
8K tokens (~6K words)
Claude 3.5
200K tokens (~150K words)
Typical 100-page PDF
30K+ tokens

Even if you have a large context window, there's a second issue: attention dilution. The model's "attention"—where it focuses—spreads thin. If you dump 100 pages but the relevant answer is on page 47, the model might miss it buried under noise. Plus, every token costs money. Sending 50,000 tokens to Claude costs more than sending 1,000.

The insight:
You don't need the entire document. You need the relevant 1-2 pages. RAG solves this by fetching only the parts that matter.

Hallucination: LLMs confidently invent facts

Here's a concrete example. I asked Ollama: "What is SouthBank India's policy on loan defaults?" Ollama replied: "SouthBank India requires borrowers to maintain a debt-to-income ratio below 40% and provides a 6-month grace period for missed payments."

Sounds authoritative, right? Completely fabricated. Ollama has no idea what SouthBank India's actual policy is. It's predicting what sounds reasonable based on patterns in its training data. This is hallucination—confident, false, and dangerous in production.

❓ Question: Policy on defaults
→
🧠 LLM guesses (no data)
→
😟 Hallucinated answer

Why it happens: LLMs are text predictors. They learned from internet text—blogs, books, training data cut off in early 2024. They have zero knowledge of:

  • Your company's internal policies
  • Recent filings or documents published after training
  • Proprietary data (customer data, medical records, employee handbooks)
  • Private PDFs, internal wikis, or company databases

RAG defined: retrieval-augmented generation

RAG is a simple idea: fetch relevant chunks from your documents, inject them into the prompt, and ask the LLM to answer based only on those chunks.

❓ Question
→
🔍 Retrieve relevant docs
→
💉 Inject into prompt
→
✅ Grounded answer

The LLM no longer hallucinates because it has context. It reads the retrieved chunks and answers based on them—or says "I don't know" if the answer isn't there. You get citations. You get accuracy. You get trust.

The definition:
RAG = Retrieval (find the right chunks) + Augmented (add them to the prompt) + Generation (LLM answers based on those chunks, not from memory).

The RAG pipeline: five steps

Here's how it works end-to-end. You'll build this in Lessons 12–14.

Step 1: Chunk the documents

Split your PDF into small pieces. A chunk might be 1–2 paragraphs (500 tokens). Why? Because when you retrieve, you want focused, relevant pieces—not entire chapters.

Step 2: Embed each chunk

Convert each chunk into a vector—a list of numbers that captures its meaning. Two similar chunks have similar vectors. Tools like `sentence-transformers` or OpenAI's embedding API do this. You run this once, then store the vectors.

Step 3: Store in a vector database

Save the vectors in a database (ChromaDB, Pinecone, Weaviate). When you query, you search by vector similarity—find the chunks most similar to your question.

Step 4: Retrieve (at query time)

User asks a question. Convert the question to a vector. Find the top 3–5 most similar chunks in the database. Return them.

Step 5: Inject and prompt the LLM

Format the retrieved chunks into the prompt. Ask the LLM: "Answer this question based only on the text below. If you don't know, say 'I don't know.'"

rag_pipeline.py High-level RAG flow

When RAG beats fine-tuning

You might be thinking: "Why not just fine-tune the model on my documents?" Good question. RAG wins in these scenarios:

RAG is better when:
✅ Choose RAG
Docs change: New policies every quarter? RAG updates instantly. Fine-tuning requires retraining.
Citations matter: "Where did you get that?" RAG shows the source chunk. Fine-tuning can't cite.
Compute budget: RAG is cheap (retrieve + call LLM). Fine-tuning is GPU-hours.
Scale: RAG handles 10,000 PDFs. Fine-tuning is all-or-nothing.
Fine-tuning is better when:
🎯 Choose Fine-tuning
Style/tone: You want a specific personality. Fine-tuning trains it in.
Latency: Your app needs responses in 100ms. Fine-tuning is faster (no retrieval step).
One-off domain: Medical diagnosis model trained on millions of cases.
Cost at scale: Millions of queries. Fine-tuning amortizes training cost.

For most agents—customer support, document Q&A, research assistants—RAG wins. It's faster to build, easier to maintain, and safer (citations!).

India & Kenya scenarios: RAG in the wild

Scenario 1: HDFC Bank customer support (India)

An HDFC customer calls with a question about the home loan policy. Your agent:

  • Takes the question
  • Searches the internal policy PDF (chunked + embedded)
  • Retrieves the 3 most relevant sections
  • Injects them into the LLM prompt

The LLM answers based on the actual policy, not hallucination. "Your loan-to-value ratio must not exceed 80% as per section 3.2 of our housing loan policy (retrieved from internal doc)."

Scenario 2: Jumia Kenya returns policy bot (Kenya)

A customer in Nairobi bought a shirt and wants to know about returns. Your bot:

  • Queries: "What is the return window for clothing items?"
  • Retrieves from the returns policy PDF
  • Answers: "30 days from purchase, unworn, with tags attached (per our Clothing Returns section)."

Fast, accurate, traceable. Every answer is grounded in the document.

How LLMs use this concept

Every time an LLM responds to a user without access to live data, it's hallucinating based on training data patterns. RAG fixes this by giving the model a "ground truth" source—the retrieved chunks. The model still uses its language skills to rephrase and reason, but now it's anchored to facts.

What you'll build in lessons 12–14

Lesson 12: Embeddings. Understand vector representations. Learn how sentence-transformers work. Build your first embedding model locally.

Lesson 13: ChromaDB. Spin up a vector database. Chunk a real PDF. Store embeddings. Write your first retrieval query.

Lesson 14: Full RAG pipeline. Build an agent that reads a 10-Q filing, retrieves sections, and answers investor questions grounded in the document.

Mini Challenge

Think of a real document your agent needs to read

An employee handbook, a pricing guide, a legal contract, a customer policy. Imagine chunking it into 500-token pieces and storing them in a vector database. How many chunks would you have? What queries would users ask?

Test your knowledge

Question 1 of 5
Why can't you just dump a 100-page PDF into an LLM prompt?
Question 2 of 5
What is hallucination in the context of LLMs?
Question 3 of 5
In the RAG pipeline, what does "retrieval" mean?
Question 4 of 5
RAG beats fine-tuning when documents change frequently. Why?
Question 5 of 5
Which is a core advantage of RAG that fine-tuning can't provide?
0
/5
Excellent! You understand why RAG is critical for document-aware agents.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You've learned why documents need RAG.
Now build agents that read and cite them.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →