What is OpenSearch?

OpenSearch — Apache 2.0 fork of Elasticsearch

OpenSearch is a community fork of Elasticsearch, released under the Apache 2.0 licence. It is powered by Lucene under the hood and exposes a full-text search engine plus analytics capabilities over an HTTP REST API. The fork was created in 2021 when Elastic changed its licence terms; AWS and the open-source community maintained the Apache 2.0 fork as OpenSearch.

Used for: log analytics (EFK/ELK-style stacks), e-commerce product search, application monitoring, and security event analytics. In this track's Docker Compose setup, OpenSearch 2.x runs with security disabled for local development — no credentials needed on localhost:9200.

Tunde, a data engineer at NaijaCart, uses OpenSearch to power the product search bar that serves millions of product queries per day across Lagos, Abuja, and Port Harcourt. This lesson walks through the exact API calls he uses every day — from creating an index to running aggregations.

OpenSearch vs Cassandra vs Elasticsearch

All three are popular in big data stacks. Understanding when to pick OpenSearch versus Cassandra is a common interview topic.

Dimension OpenSearch Cassandra
Primary use Full-text search, log analytics Time-series, write-heavy workloads
Query Lucene full-text + aggregations CQL (SQL-like, limited)
Licence Apache 2.0 (OpenSearch) / Elastic Licence (ES) Apache 2.0
Schema Dynamic (auto-detect types) Strict (schema-first)
Data model Documents (JSON) Wide-column rows
Rule of thumb: If the question is "does this text contain X?" use OpenSearch. If the question is "give me all records for user ID 123 in the last 30 days" at high write throughput, use Cassandra.

Health check

Before running any API calls, verify OpenSearch is up and healthy. Start your Docker stack with ./start.sh, then run:

shell Lesson 27 · cluster health

OpenSearch must be running at localhost:9200. Output will vary.

sample output
{ "cluster_name": "opensearch", "status": "green", "number_of_nodes": 1, "number_of_data_nodes": 1, "active_primary_shards": 1, "active_shards": 1, "unassigned_shards": 0 }

status: "green" means all primary and replica shards are assigned. On a single-node cluster with number_of_replicas: 0, this is the expected healthy state. yellow means replicas are unassigned — normal on single-node if you accidentally set replicas to 1.

Create an index

An index in OpenSearch is analogous to a table in a relational database. Wanjiku created the naijacart-products index with explicit field mappings so that OpenSearch does not guess types incorrectly.

shell Lesson 27 · create naijacart-products index

OpenSearch must be running at localhost:9200. Output will vary.

sample output
{"acknowledged":true,"shards_acknowledged":true,"index":"naijacart-products"}
text vs keyword — the most important mapping decision

text fields are tokenised and analysed — perfect for full-text search (e.g., searching "5000mAh battery" inside a description). keyword fields are stored as-is — perfect for exact match filters and aggregations (e.g., filtering by category = "Electronics"). Using text on a field you want to aggregate is a common mistake covered in the mistakes section below.

Index a document

Indexing a document is the OpenSearch equivalent of inserting a row. The _doc endpoint accepts a JSON body. OpenSearch assigns an auto-generated _id if you do not specify one.

shell Lesson 27 · index one product document

OpenSearch must be running at localhost:9200. Output will vary.

sample output
{"_index":"naijacart-products","_id":"abc123","result":"created","_shards":{"successful":1}}

Filter + range query

The bool query combines multiple clauses. Using filter instead of must means OpenSearch skips relevance scoring — filters are cached and significantly faster for exact-match and range conditions.

shell Lesson 27 · bool filter with range

OpenSearch must be running at localhost:9200. Output will vary.

sample output
{ "hits": { "total": {"value": 2}, "hits": [ {"_source": {"name": "Samsung Galaxy A54", "price_ngn": 185000}}, {"_source": {"name": "Tecno Spark 10", "price_ngn": 62000}} ] } }

Aggregation — terms bucket

Aggregations are OpenSearch's analytics engine. A terms aggregation groups documents by a keyword field value — like GROUP BY in SQL. The avg aggregation computes a numeric average across all matched documents.

shell Lesson 27 · terms + avg aggregation

OpenSearch must be running at localhost:9200. Output will vary.

sample output
{ "aggregations": { "by_category": { "buckets": [ {"key": "Electronics", "doc_count": 5}, {"key": "Fashion", "doc_count": 3} ] }, "avg_price": {"value": 124500} } }
What does size: 0 mean?

Setting size: 0 at the top level tells OpenSearch to return zero documents in the hits array — only the aggregation results are returned. This is a best practice for aggregation-only queries: it reduces response size and avoids Lucene fetching document source unnecessarily.

Python client

The opensearch-py library wraps the HTTP REST API with a Pythonic interface. Install it with pip install opensearch-py==2.6.0 (already included in the Lesson 00 pip install block).

python Lesson 27 · opensearch-py client — reference, no output needed

The Python client is used in later lessons where OpenSearch is the consumer destination for Kafka events. For development, the curl commands above are easier to iterate on quickly.

Chidi's search tip

Chidi (NaijaCart) — text vs keyword in production

"At NaijaCart, I use text type for product descriptions — they need full-text tokenisation so a search for '5000mAh' finds '5000mAh battery'. I use keyword for category and city — exact match and aggregatable without any sub-field tricks. Never use text for fields you want to aggregate — it won't work without a .keyword sub-field, and dynamic mapping sometimes adds one silently. Explicit mappings eliminate the confusion."

Common mistakes

Using text type for aggregations
Aggregations require keyword fields. A text field is tokenised and cannot be bucketed without a .keyword sub-field. Always declare keyword type for category, city, status, and other exact-value fields.
Over-sharding on a single node
Each shard is a Lucene instance with overhead. A single-node development cluster with 5 shards just burns memory for no benefit. Start with number_of_shards: 1 in local dev. Scale shards when you actually need horizontal distribution.
Not setting number_of_replicas: 0
The default is 1 replica. On a single-node cluster, there is nowhere to place the replica shard, so the cluster stays yellow (unassigned replicas) forever. Always set number_of_replicas: 0 for local single-node development.
Mapping explosion from nested JSON
Dynamic mapping automatically creates field entries for every key it sees. If you index deeply nested or highly variable JSON, OpenSearch can create thousands of field mappings, hitting the default limit and causing index rejection errors. Define explicit mappings for production indices.

Quiz — 3 questions

Question 1 of 3
Which OpenSearch field type supports full-text search (tokenised, analysed)?
Question 2 of 3
Which field type should you use for aggregations and exact-match filters (e.g., category = "Electronics")?
Question 3 of 3
What does setting size: 0 at the top level of an OpenSearch search request mean?
0
/ 3
OpenSearch fundamentals checked.

Next: pipe Kafka events into OpenSearch in real time.

Interview Q&A

Easy What is the difference between OpenSearch and Elasticsearch?
+
OpenSearch is an Apache 2.0 open-source fork of Elasticsearch, created in 2021 after Elastic changed the Elasticsearch licence to the Server Side Public License (SSPL) and the Elastic Licence — neither of which is OSI-approved open source. AWS led the fork to maintain a truly open-source search engine. Functionally, OpenSearch 1.x and 2.x are very close to Elasticsearch 7.10 (the last Apache 2.0 ES release). The REST API, query DSL, index mappings, and aggregation syntax are nearly identical. Key differences: OpenSearch includes a built-in security plugin (free), OpenSearch Dashboards replaces Kibana, and OpenSearch has diverged in features over time — particularly around machine learning, anomaly detection, and the k-NN vector search capabilities. For a data engineering interview, the key point is: same Lucene engine, same API feel, different governance model.
Medium When would you use OpenSearch instead of a relational database?
+
Use OpenSearch when: (1) you need full-text search — SQL LIKE queries do not scale; they require a full table scan, while OpenSearch maintains an inverted index that returns results in milliseconds even on millions of documents. (2) The query shape is not known in advance — OpenSearch handles ad hoc queries across any combination of fields without schema changes or index additions. (3) You need relevance ranking — a relational database returns matching rows with no notion of which result is "most relevant"; OpenSearch returns ranked results via BM25 scoring. (4) Log and event analytics — OpenSearch is designed for semi-structured JSON with variable fields, while a relational database requires a fixed schema. Use a relational database when: you need ACID transactions, complex multi-table joins, or authoritative record storage. A common architecture: PostgreSQL stores the authoritative product catalog; OpenSearch indexes a denormalised copy for the search bar.
Medium What is the difference between a filter and a query in OpenSearch?
+
In OpenSearch's bool query, a must (or should) clause contributes to relevance scoring — documents that match the clause score higher. A filter clause is a binary yes/no check: the document either matches or it does not, and it does not affect the _score. The performance implication is significant: filter clauses are cached by OpenSearch at the segment level in a bitset (bitfield of matching document IDs). Subsequent queries with the same filter retrieve the cached bitset instead of re-running the filter. This makes filters dramatically faster for things like term: {"status": "active"} or range: {"price": {"gte": 100}}. The best practice is: use must only for full-text queries where ranking matters; use filter for everything else (status, category, date range, etc.).

Key takeaways

  • OpenSearch is an Apache 2.0 fork of Elasticsearch, powered by Lucene, accessed entirely via HTTP REST. The Docker stack runs it at localhost:9200 with security disabled for local development.
  • Field type is the most important mapping decision: text for full-text search (tokenised), keyword for exact match and aggregations. Never aggregate on a text field without a .keyword sub-field.
  • Set number_of_replicas: 0 and number_of_shards: 1 for single-node local dev — avoids yellow cluster health and unnecessary overhead.
  • Use filter clauses (not must) for exact-match and range conditions — they are cached and orders of magnitude faster than scored queries for high-frequency filters.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Search, stream, and analyse at scale

Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Data Engineer →