What is ChromaDB?
ChromaDB is an open-source vector database designed for developers. Unlike Pinecone or Weaviate (which require servers), ChromaDB runs locally. You embed documents, store them as vectors, and query with semantic similarity—all on your machine or a single server.
The workflow is simple: embed a document chunk → store it in ChromaDB with metadata → later, embed a user question → query for similar chunks → pass those chunks to an LLM. This is the foundation of RAG (Retrieval-Augmented Generation).
SouthBank India gets 500 customer emails per day asking about refund windows, fraud limits, and card replacement policies. The policies live in a 50-page PDF. Manually searching is slow.
Solution: chunk the PDF into 200 snippets (e.g., "M-Pesa refunds are valid within 30 days"), embed each with nomic-embed-text (from Lesson 12), store in ChromaDB. When a customer asks "What is the M-Pesa refund window?", the agent embeds the question, queries ChromaDB for the top-3 similar chunks, passes those to the LLM. Answer in 0.2 seconds.
Install ChromaDB and set up persistence
ChromaDB has an ephemeral mode (in-memory, lost on restart) and a persistent mode (on-disk). For production, use persistent.
Create a collection and add documents
A collection is a named container for documents. ChromaDB auto-embeds documents using a default embedder. When you call .add(), it embeds each document and stores the vector + metadata.
./chroma_db/.
Query: find semantically similar documents
Call .query() with a text question. ChromaDB embeds your question and finds the top-N closest vectors using cosine similarity. Returns ids, distances, documents, and metadatas.
Combine semantic search with metadata filters
Query only documents matching specific metadata. E.g., find M-Pesa policies but not credit card policies, or only documents from Q3 2025. Filters combine with semantic similarity.
Wire nomic-embed-text (custom embedder) from Lesson 12
ChromaDB's default embedder is good, but you can use nomic-embed-text (from Lesson 12) for better performance. Define an embedding function and pass it to get_or_create_collection().
Worked example 1: Agent queries M-Pesa policies
Load 10 M-Pesa policy snippets, embed them, then query with a customer question and return the top-3 most relevant chunks. This is the foundation of a customer support agent.
Worked example 2: Query a mini SouthBank 10-Q filing
Chunk a fictional SouthBank India quarterly filing (10-Q) into 10 sections: revenue, expense, profit, risk factors, etc. Query with "What was Q3 revenue?" and retrieve the relevant section.
When to upgrade to Qdrant, Pinecone, or Weaviate
ChromaDB is great for local development and small-to-medium workloads (100K-1M documents). But production RAG systems often outgrow it. Here's when to upgrade:
- Scale: If you have >10M documents across multiple teams, Qdrant or Pinecone auto-scale. ChromaDB (even on a single server) struggles.
- Multi-tenancy: Pinecone/Weaviate isolate data per customer. ChromaDB is simpler for single-user or single-app use.
- Advanced filtering: Weaviate excels at GraphQL queries and complex metadata filters. ChromaDB is simpler but less flexible.
- Collaboration: If your team shares embeddings across microservices, a hosted vector DB (Pinecone, Qdrant Cloud) is easier than managing a ChromaDB server.
- Real-time sync: Some apps need vector updates sub-second. Pinecone guarantees that; ChromaDB file writes are slower.
Task: Create a ChromaDB collection for Tata Insurance policies (create your own 5 snippets about coverage, claims, deductibles). Add them to ChromaDB. Then query: "What is the claims process?" and print the top result.
Hint 1: Use get_or_create_collection() with a unique name.
Hint 2: Call .add() with documents, ids, and metadatas.
Hint 3: Call .query() with your question and n_results=1.
Test your knowledge
You've embedded your first documents.
Now scale to production RAG.
Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Agentic Engineer →