What is ChromaDB?

ChromaDB is an open-source vector database designed for developers. Unlike Pinecone or Weaviate (which require servers), ChromaDB runs locally. You embed documents, store them as vectors, and query with semantic similarity—all on your machine or a single server.

The workflow is simple: embed a document chunk → store it in ChromaDB with metadata → later, embed a user question → query for similar chunks → pass those chunks to an LLM. This is the foundation of RAG (Retrieval-Augmented Generation).

🏦
Real-world scenario
SouthBank India (Mumbai): customer support with policy docs

SouthBank India gets 500 customer emails per day asking about refund windows, fraud limits, and card replacement policies. The policies live in a 50-page PDF. Manually searching is slow.

Solution: chunk the PDF into 200 snippets (e.g., "M-Pesa refunds are valid within 30 days"), embed each with nomic-embed-text (from Lesson 12), store in ChromaDB. When a customer asks "What is the M-Pesa refund window?", the agent embeds the question, queries ChromaDB for the top-3 similar chunks, passes those to the LLM. Answer in 0.2 seconds.

Install ChromaDB and set up persistence

ChromaDB has an ephemeral mode (in-memory, lost on restart) and a persistent mode (on-disk). For production, use persistent.

setup_chromadb.py Install and initialize
Key insight:
PersistentClient stores all vectors, metadata, and embeddings as files. Next time you restart, all your data is there. No setup, no database server, no credentials.

Create a collection and add documents

A collection is a named container for documents. ChromaDB auto-embeds documents using a default embedder. When you call .add(), it embeds each document and stores the vector + metadata.

add_documents.py Add M-Pesa policy snippets to a collection
What just happened:
ChromaDB embedded all 10 documents using the default embedder. Each document now has a 384-dimensional vector. Those vectors are stored on disk in ./chroma_db/.

Query: find semantically similar documents

Call .query() with a text question. ChromaDB embeds your question and finds the top-N closest vectors using cosine similarity. Returns ids, distances, documents, and metadatas.

query_documents.py Query for similar policy chunks
Distance metric:
Lower distance = more similar. ChromaDB uses cosine similarity, so 0.0 is identical, 1.0 is opposite.

Combine semantic search with metadata filters

Query only documents matching specific metadata. E.g., find M-Pesa policies but not credit card policies, or only documents from Q3 2025. Filters combine with semantic similarity.

filtered_query.py Query with a where filter
Power combo:
Metadata filters narrow the search space (fast), then semantic similarity ranks within that space (accurate). Much more efficient than searching all 10,000 documents.

Wire nomic-embed-text (custom embedder) from Lesson 12

ChromaDB's default embedder is good, but you can use nomic-embed-text (from Lesson 12) for better performance. Define an embedding function and pass it to get_or_create_collection().

custom_embedder.py Use nomic-embed-text with ChromaDB

Worked example 1: Agent queries M-Pesa policies

Load 10 M-Pesa policy snippets, embed them, then query with a customer question and return the top-3 most relevant chunks. This is the foundation of a customer support agent.

mpesa_support_agent.py Customer support with ChromaDB
Sample output
Customer question: I sent money 45 days ago... Top 3 matching policies: 1. [relevance: 0.85] M-Pesa refunds are valid within 30 days... 2. [relevance: 0.72] Fraud claims must be reported within 7 days... 3. [relevance: 0.61] Dispute resolution takes 15-20 working days...

Worked example 2: Query a mini SouthBank 10-Q filing

Chunk a fictional SouthBank India quarterly filing (10-Q) into 10 sections: revenue, expense, profit, risk factors, etc. Query with "What was Q3 revenue?" and retrieve the relevant section.

southbank_10q_rag.py RAG over financial filings
Sample output
Query: 'What was Q3 revenue?' Answer: Q3 2025 Total Revenue: ₹12,500 crores (USD 1.5B equivalent). YoY growth 18%.

When to upgrade to Qdrant, Pinecone, or Weaviate

ChromaDB is great for local development and small-to-medium workloads (100K-1M documents). But production RAG systems often outgrow it. Here's when to upgrade:

  • Scale: If you have >10M documents across multiple teams, Qdrant or Pinecone auto-scale. ChromaDB (even on a single server) struggles.
  • Multi-tenancy: Pinecone/Weaviate isolate data per customer. ChromaDB is simpler for single-user or single-app use.
  • Advanced filtering: Weaviate excels at GraphQL queries and complex metadata filters. ChromaDB is simpler but less flexible.
  • Collaboration: If your team shares embeddings across microservices, a hosted vector DB (Pinecone, Qdrant Cloud) is easier than managing a ChromaDB server.
  • Real-time sync: Some apps need vector updates sub-second. Pinecone guarantees that; ChromaDB file writes are slower.
Recommendation:
Start with ChromaDB for prototyping. Once you hit 500K documents or multiple teams, evaluate Qdrant (self-hosted, open-source) or Pinecone (managed, pay-as-you-go).

Next: Lesson 14 — End-to-end RAG with real filings

In Lesson 14, you'll build a full RAG pipeline: load a real SEC filing (or SouthBank annual report), chunk it smartly (respecting paragraph boundaries), embed it with an LLM-specific embedder, store in ChromaDB, then chain an LLM on top to answer multi-turn questions about the document. By Lesson 15, your agent will read any PDF, ask intelligent follow-ups, and cite specific sections—all without training.

Mini challenge: Build a custom policy bot

Task: Create a ChromaDB collection for Tata Insurance policies (create your own 5 snippets about coverage, claims, deductibles). Add them to ChromaDB. Then query: "What is the claims process?" and print the top result.

Hint 1: Use get_or_create_collection() with a unique name.

Hint 2: Call .add() with documents, ids, and metadatas.

Hint 3: Call .query() with your question and n_results=1.

Test your knowledge

Question 1 of 5
What does ChromaDB's PersistentClient store?
Question 2 of 5
How does ChromaDB's .query() method return results?
Question 3 of 5
What is the advantage of combining metadata filtering with semantic search?
Question 4 of 5
How do you wire a custom embedder like nomic-embed-text to ChromaDB?
Question 5 of 5
When should you upgrade from ChromaDB to Qdrant or Pinecone?
0
/5
Great! You understand ChromaDB fundamentals and RAG patterns.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You've embedded your first documents.
Now scale to production RAG.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →