The two core problems: context limits and hallucination
By Lesson 10, your agents had tools. They could fetch live stock prices, current weather, real-time search results. But what about static documents? A customer-support bot needs to read the company's policy PDF. A research agent needs to analyze a 10-Q filing. A loan officer's assistant needs to reference the bank's lending guidelines.
The naive approach: dump the entire document into the prompt. But this hits two walls instantly.
Problem 1: Context window limits. An LLM has a fixed input size measured in tokens. Claude can handle 200K tokens (roughly 150,000 words). Ollama's 7B model is more like 4K tokens (3,000 words). A typical 100-page PDF is 30,000+ tokens. You can't fit it. And even if you could, there's a deeper issue:
Problem 2: Hallucination. LLMs confidently invent facts when they don't know the answer. Ask Ollama "What is SouthBank India's policy on loan-to-value ratios?" and it will give you a plausible-sounding answer—completely made up. It has no access to the policy PDF. It's just predicting what a reasonable answer might sound like.
Context window limits: why you can't dump everything
A context window is the maximum number of tokens an LLM can read. Think of it as the model's working memory. You send in a prompt (tokens), the model reads it, and returns a completion (more tokens). The total—prompt + completion—must fit in the context window.
Even if you have a large context window, there's a second issue: attention dilution. The model's "attention"—where it focuses—spreads thin. If you dump 100 pages but the relevant answer is on page 47, the model might miss it buried under noise. Plus, every token costs money. Sending 50,000 tokens to Claude costs more than sending 1,000.
Hallucination: LLMs confidently invent facts
Here's a concrete example. I asked Ollama: "What is SouthBank India's policy on loan defaults?" Ollama replied: "SouthBank India requires borrowers to maintain a debt-to-income ratio below 40% and provides a 6-month grace period for missed payments."
Sounds authoritative, right? Completely fabricated. Ollama has no idea what SouthBank India's actual policy is. It's predicting what sounds reasonable based on patterns in its training data. This is hallucination—confident, false, and dangerous in production.
Why it happens: LLMs are text predictors. They learned from internet text—blogs, books, training data cut off in early 2024. They have zero knowledge of:
- Your company's internal policies
- Recent filings or documents published after training
- Proprietary data (customer data, medical records, employee handbooks)
- Private PDFs, internal wikis, or company databases
RAG defined: retrieval-augmented generation
RAG is a simple idea: fetch relevant chunks from your documents, inject them into the prompt, and ask the LLM to answer based only on those chunks.
The LLM no longer hallucinates because it has context. It reads the retrieved chunks and answers based on them—or says "I don't know" if the answer isn't there. You get citations. You get accuracy. You get trust.
The RAG pipeline: five steps
Here's how it works end-to-end. You'll build this in Lessons 12–14.
Step 1: Chunk the documents
Split your PDF into small pieces. A chunk might be 1–2 paragraphs (500 tokens). Why? Because when you retrieve, you want focused, relevant pieces—not entire chapters.
Step 2: Embed each chunk
Convert each chunk into a vector—a list of numbers that captures its meaning. Two similar chunks have similar vectors. Tools like `sentence-transformers` or OpenAI's embedding API do this. You run this once, then store the vectors.
Step 3: Store in a vector database
Save the vectors in a database (ChromaDB, Pinecone, Weaviate). When you query, you search by vector similarity—find the chunks most similar to your question.
Step 4: Retrieve (at query time)
User asks a question. Convert the question to a vector. Find the top 3–5 most similar chunks in the database. Return them.
Step 5: Inject and prompt the LLM
Format the retrieved chunks into the prompt. Ask the LLM: "Answer this question based only on the text below. If you don't know, say 'I don't know.'"
When RAG beats fine-tuning
You might be thinking: "Why not just fine-tune the model on my documents?" Good question. RAG wins in these scenarios:
For most agents—customer support, document Q&A, research assistants—RAG wins. It's faster to build, easier to maintain, and safer (citations!).
India & Kenya scenarios: RAG in the wild
Scenario 1: HDFC Bank customer support (India)
An HDFC customer calls with a question about the home loan policy. Your agent:
- Takes the question
- Searches the internal policy PDF (chunked + embedded)
- Retrieves the 3 most relevant sections
- Injects them into the LLM prompt
The LLM answers based on the actual policy, not hallucination. "Your loan-to-value ratio must not exceed 80% as per section 3.2 of our housing loan policy (retrieved from internal doc)."
Scenario 2: Jumia Kenya returns policy bot (Kenya)
A customer in Nairobi bought a shirt and wants to know about returns. Your bot:
- Queries: "What is the return window for clothing items?"
- Retrieves from the returns policy PDF
- Answers: "30 days from purchase, unworn, with tags attached (per our Clothing Returns section)."
Fast, accurate, traceable. Every answer is grounded in the document.
How LLMs use this concept
Every time an LLM responds to a user without access to live data, it's hallucinating based on training data patterns. RAG fixes this by giving the model a "ground truth" source—the retrieved chunks. The model still uses its language skills to rephrase and reason, but now it's anchored to facts.
What you'll build in lessons 12–14
Lesson 12: Embeddings. Understand vector representations. Learn how sentence-transformers work. Build your first embedding model locally.
Lesson 13: ChromaDB. Spin up a vector database. Chunk a real PDF. Store embeddings. Write your first retrieval query.
Lesson 14: Full RAG pipeline. Build an agent that reads a 10-Q filing, retrieves sections, and answers investor questions grounded in the document.
Mini Challenge
An employee handbook, a pricing guide, a legal contract, a customer policy. Imagine chunking it into 500-token pieces and storing them in a vector database. How many chunks would you have? What queries would users ask?
Test your knowledge
You've learned why documents need RAG.
Now build agents that read and cite them.
Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Agentic Engineer →