🎯 What you will learn
RAG (Retrieval-Augmented Generation) is the most practical technique for building LLM-powered apps that answer questions accurately from your own documents. Instead of relying on the LLM's training data (which can be stale or wrong), you retrieve relevant passages first and inject them as context. By the end of this lesson you will understand:
- The three-step RAG pipeline: Retrieve → Augment → Generate
- Keyword-based retrieval with Java Streams (for demos and prototypes)
- Prompt augmentation — building a context-enriched prompt string
- Calling Ollama's
llama3.2:3bmodel with the augmented prompt - Production RAG with pgvector and the
nomic-embed-textembedding model - Why chunking and embedding consistency matter for retrieval quality
🎓 Real-world scenario
VidyaPath's LLM kept hallucinating Java facts. Riya implements RAG: before calling the LLM, the app retrieves relevant Java tutorial snippets from a knowledge base and adds them to the prompt. The LLM is instructed to answer only from the provided context, eliminating false facts.
🔍 Step 1 — Knowledge base + retrieval
The retrieval step finds documents relevant to the user's question. In production this is a vector similarity search (pgvector). For this demo we use keyword matching with Java Streams — simple, runs locally, no database required.
Production retrieval: Replace keyword matching with pgvector cosine similarity search. The query is embedded to a float vector, then PostgreSQL finds the K nearest document vectors. This works even when the user's words differ from the document's words — "how do pipelines work in Java?" retrieves the Streams doc via semantic proximity.
🤖 Steps 2 + 3 — Augment prompt + generate answer
The augmented prompt wraps the retrieved context with explicit instructions: "answer ONLY from the context below." This constraint is what prevents hallucination. The LLM is then called via Ollama's REST API.
🐘 Production RAG with pgvector (PostgreSQL)
For production, replace keyword matching with vector similarity search. pgvector adds a vector data type and the <=> cosine distance operator to PostgreSQL. Documents are embedded with nomic-embed-text via Ollama and stored as float vectors. Retrieval is a single SQL query.
Why nomic-embed-text? It is a dedicated embedding model — optimised for encoding semantic meaning into dense float vectors. llama3.2:3b is a text generation model. Always use an embedding-specific model for the Embed step; use the generation model only for the Generate step.
📋 RAG pipeline stages
| Stage | What happens | Demo | Production |
|---|---|---|---|
| Retrieve | Find relevant documents for the query | Keyword match with Java Streams | pgvector cosine similarity |
| Augment | Build context-enriched prompt | buildPrompt() | Same — token-budget truncation added |
| Generate | Call LLM with the augmented prompt | Ollama llama3.2:3b | Ollama or OpenAI API |
| Embed (index) | Convert documents to vectors for storage | N/A (keyword) | Ollama nomic-embed-text |
📝 Quiz — 5 questions
<=> operator do in pgvector?Next: Prompt Versioning and Tests →
🎤 Interview Q&A
Keyword search (SQL LIKE / full-text search) matches exact words. Vector search compares semantic meaning — "car" and "automobile" are close in vector space even with no shared words. RAG systems benefit from semantic search to retrieve relevant context even when the user's exact words differ.
Long documents are split into smaller chunks (e.g., 500 tokens each) before embedding. Embedding a 100-page PDF as one vector loses granularity. Chunking lets you retrieve the specific section that answers a question, not the entire document.
Re-embed and re-upsert when a document changes. Use a hash (SHA-256 of content) to detect changes. On upsert: UPDATE kb_documents SET content=?, embedding=? WHERE source=?. Many teams use event-driven pipelines: document change → reindex event → embedding service.
Every LLM has a maximum context length (llama3.2:3b: ~128K tokens). If you retrieve too many documents, the augmented prompt may exceed this limit. RAG implementations set a max-context budget: retrieve top-K docs, truncate if total tokens exceed the limit, then generate.
Use RAGAS (a Python framework): it measures Faithfulness (is the answer grounded in retrieved docs?), Answer Relevance (does it answer the question?), and Context Precision/Recall (did retrieval find the right docs?). For Java: store (question, expected_answer, retrieved_docs, generated_answer) tuples and write JUnit tests that assert keyword presence and cosine similarity of generated vs. expected answers.
⚠️ Common mistakes
✅ Key takeaways
- RAG = Retrieve context → Augment prompt → Generate grounded response
- Keyword matching works for demos; use pgvector + nomic-embed-text for production semantic search
- Instruct the LLM: "Answer ONLY from the context below" to prevent hallucination
- Chunk documents (256–512 tokens) before embedding for precise retrieval
- Evaluate RAG quality with RAGAS: Faithfulness, Answer Relevance, Context Precision
From RAG to production-grade AI services
Join the Best AI Programs — three 5-month courses, fully online, with placement assistance. Week 1 is free.
Explore our AI-era courses →