🎯 What you will learn

RAG (Retrieval-Augmented Generation) is the most practical technique for building LLM-powered apps that answer questions accurately from your own documents. Instead of relying on the LLM's training data (which can be stale or wrong), you retrieve relevant passages first and inject them as context. By the end of this lesson you will understand:

  • The three-step RAG pipeline: Retrieve → Augment → Generate
  • Keyword-based retrieval with Java Streams (for demos and prototypes)
  • Prompt augmentation — building a context-enriched prompt string
  • Calling Ollama's llama3.2:3b model with the augmented prompt
  • Production RAG with pgvector and the nomic-embed-text embedding model
  • Why chunking and embedding consistency matter for retrieval quality
💼 Interview reality: "How does RAG work?" is asked in every AI/ML backend round in 2025. Interviewers probe: retrieval strategy, how you prevent hallucination, and how you evaluate quality. This lesson covers all three.

🎓 Real-world scenario

🎓
VidyaPath · Bangalore, India
Answers grounded in your own docs — no hallucinations

VidyaPath's LLM kept hallucinating Java facts. Riya implements RAG: before calling the LLM, the app retrieves relevant Java tutorial snippets from a knowledge base and adds them to the prompt. The LLM is instructed to answer only from the provided context, eliminating false facts.

🔍 Step 1 — Knowledge base + retrieval

The retrieval step finds documents relevant to the user's question. In production this is a vector similarity search (pgvector). For this demo we use keyword matching with Java Streams — simple, runs locally, no database required.

javac -cp ".:jackson-core.jar:jackson-databind.jar" SimpleRAG.java
SimpleRAG.java — knowledge base + retrieval stepLesson 36 · RAG pipeline
verified output
Compiled successfully — knowledge base loaded with 4 documents

Production retrieval: Replace keyword matching with pgvector cosine similarity search. The query is embedded to a float vector, then PostgreSQL finds the K nearest document vectors. This works even when the user's words differ from the document's words — "how do pipelines work in Java?" retrieves the Streams doc via semantic proximity.

🤖 Steps 2 + 3 — Augment prompt + generate answer

The augmented prompt wraps the retrieved context with explicit instructions: "answer ONLY from the context below." This constraint is what prevents hallucination. The LLM is then called via Ollama's REST API.

java -cp ".:jackson-core.jar:jackson-databind.jar" SimpleRAG
SimpleRAG.java — augment + generateLesson 36 · full RAG pipeline with llama3.2:3b
verified output — full RAG pipeline with llama3.2:3b
=== VidyaPath — RAG Demo === Q: How do Java Streams work? [Retrieved 1 doc(s)] A: Java Streams work by creating a pipeline of operations that are executed lazily, meaning they only occur when a terminal operation is encountered. Intermediate operations like filter() and map() are executed when their corresponding terminal operations (like forEach(), collect(), or reduce()) are called. This lazy execution allows for efficient processing of large collections without loading them all into memory at once. Q: What is Spring dependency injection? [Retrieved 1 doc(s)] A: Spring dependency injection allows components to receive dependencies or objects they need to function, eliminating the need for explicit 'new' statements. This is achieved through the use of annotations such as @Service, @Repository, and @Component, which automatically register beans in an ApplicationContext. Q: When does @Transactional roll back? [Retrieved 1 doc(s)] A: @Transactional rolls back when a RuntimeException is thrown, causing Spring to roll back all changes made by the transaction. This ensures database consistency and prevents partial data corruption.

🐘 Production RAG with pgvector (PostgreSQL)

For production, replace keyword matching with vector similarity search. pgvector adds a vector data type and the <=> cosine distance operator to PostgreSQL. Documents are embedded with nomic-embed-text via Ollama and stored as float vectors. Retrieval is a single SQL query.

# pgvector + Ollama nomic-embed-text
pgvector schema + similarity searchLesson 36 · production semantic search
Java — generate embedding + store in pgvectorLesson 36 · nomic-embed-text via Ollama
verified output — pgvector similarity search result
-- pgvector similarity search result: content: "Java Streams API provides a functional pipeline..." similarity: 0.9234 content: "Spring manages objects called beans..." similarity: 0.4821

Why nomic-embed-text? It is a dedicated embedding model — optimised for encoding semantic meaning into dense float vectors. llama3.2:3b is a text generation model. Always use an embedding-specific model for the Embed step; use the generation model only for the Generate step.

📋 RAG pipeline stages

StageWhat happensDemoProduction
RetrieveFind relevant documents for the queryKeyword match with Java Streamspgvector cosine similarity
AugmentBuild context-enriched promptbuildPrompt()Same — token-budget truncation added
GenerateCall LLM with the augmented promptOllama llama3.2:3bOllama or OpenAI API
Embed (index)Convert documents to vectors for storageN/A (keyword)Ollama nomic-embed-text

📝 Quiz — 5 questions

Question 1 of 5
What are the three steps of a RAG pipeline?
Question 2 of 5
Why does RAG reduce LLM hallucinations?
Question 3 of 5
What PostgreSQL extension enables vector similarity search?
Question 4 of 5
Which Ollama model is optimised for generating text embeddings?
Question 5 of 5
What does the <=> operator do in pgvector?
0
/ 5
Quiz complete!

Next: Prompt Versioning and Tests →

🎤 Interview Q&A

EasyWhat is the difference between keyword search and vector search?
+

Keyword search (SQL LIKE / full-text search) matches exact words. Vector search compares semantic meaning — "car" and "automobile" are close in vector space even with no shared words. RAG systems benefit from semantic search to retrieve relevant context even when the user's exact words differ.

EasyWhat is chunking and why does it matter for RAG?
+

Long documents are split into smaller chunks (e.g., 500 tokens each) before embedding. Embedding a 100-page PDF as one vector loses granularity. Chunking lets you retrieve the specific section that answers a question, not the entire document.

MediumHow do you keep the vector database in sync with updated documents?
+

Re-embed and re-upsert when a document changes. Use a hash (SHA-256 of content) to detect changes. On upsert: UPDATE kb_documents SET content=?, embedding=? WHERE source=?. Many teams use event-driven pipelines: document change → reindex event → embedding service.

MediumWhat is a context window limit and how does RAG respect it?
+

Every LLM has a maximum context length (llama3.2:3b: ~128K tokens). If you retrieve too many documents, the augmented prompt may exceed this limit. RAG implementations set a max-context budget: retrieve top-K docs, truncate if total tokens exceed the limit, then generate.

HardHow would you evaluate RAG quality systematically?
+

Use RAGAS (a Python framework): it measures Faithfulness (is the answer grounded in retrieved docs?), Answer Relevance (does it answer the question?), and Context Precision/Recall (did retrieval find the right docs?). For Java: store (question, expected_answer, retrieved_docs, generated_answer) tuples and write JUnit tests that assert keyword presence and cosine similarity of generated vs. expected answers.

⚠️ Common mistakes

Telling the LLM 'use the context' but not enforcing it
If the prompt says 'here is some context' without 'ONLY use the context', the LLM mixes context with its training knowledge. Be explicit: 'If the answer is not in the context, say: I don't know.'
Retrieving too few or too many docs
Too few: the right answer is missing. Too many: the context window fills up and irrelevant docs distract the LLM. Start with top-3 and tune based on evaluation metrics.
Embedding questions and documents with different models
Query and document embeddings must come from the same model. Using nomic-embed-text for documents and text-embedding-ada-002 for queries produces incompatible vector spaces — similarity scores are meaningless.
Not chunking before embedding
Embedding a 5,000-word document as one vector averages all its semantics into one point. A question about paragraph 3 won't match this average well. Always chunk to 256–512 tokens before embedding.

✅ Key takeaways

  • RAG = Retrieve context → Augment prompt → Generate grounded response
  • Keyword matching works for demos; use pgvector + nomic-embed-text for production semantic search
  • Instruct the LLM: "Answer ONLY from the context below" to prevent hallucination
  • Chunk documents (256–512 tokens) before embedding for precise retrieval
  • Evaluate RAG quality with RAGAS: Faithfulness, Answer Relevance, Context Precision
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

From RAG to production-grade AI services

Join the Best AI Programs — three 5-month courses, fully online, with placement assistance. Week 1 is free.

Explore our AI-era courses →