What we're building

This mini-build creates a centralized log analytics pipeline for a fintech API. Application services (payments-api, auth-service, wallet-service) emit structured JSON logs. Those logs flow into Kafka, then a Python consumer bulk-indexes them into OpenSearch, where operations teams can query, visualize, and alert on error rates in real time.

This architecture is the open-source equivalent of the ELK (Elasticsearch/Logstash/Kibana) stack, but using Kafka as a durable buffer between log producers and the search index — a critical improvement because OpenSearch can become a bottleneck under log bursts.

Step 1
log_producer.py
→
Kafka topic
api.logs
→
Step 2
opensearch_consumer.py
→
OpenSearch
logs-2025-01
Why Kafka in front of OpenSearch? OpenSearch indexing throughput is limited by segment merging and refresh cycles. A spike in error logs (e.g., a database outage causing thousands of timeout errors per second) can overwhelm OpenSearch if logs are sent directly. Kafka acts as a durable buffer: consumer lag grows during the spike and naturally drains when OpenSearch recovers. No logs are lost.

Index mapping — the foundation of fast queries

OpenSearch mappings define how each field is stored and indexed. Getting this wrong causes either bloated indices (mapping everything as text with full-text analysis) or broken queries (using text for fields you need to filter exactly). The key distinction: keyword for exact-match fields, text for full-text search, date for timestamps.

Field Type Rationale
@timestamp date Enables time-range queries and efficient time-based sorting
service keyword Exact-match filter: service = "payments-api"
level keyword Exact-match filter: level = "ERROR"; low cardinality (INFO, WARN, ERROR, DEBUG)
message text Full-text search: message CONTAINS "timeout"
trace_id keyword Exact-match lookup to correlate logs across services for one request
latency_ms integer Range queries: latency_ms > 1000; aggregations: p99 latency
status_code integer Numeric filter: status_code = 503; aggregation: count by code
error boolean Fast filter: error = true without parsing the level field
create_index.sh Lesson 28 · PUT index with explicit mappings
dynamic: "strict" prevents mapping explosions

By default, OpenSearch dynamically maps every new field it sees. A single malformed log event with 200 unexpected fields can create 200 new mappings, bloating the cluster state and slowing down every operation. Setting "dynamic": "strict" rejects documents with unexpected fields — you get a 400 error instead of a runaway mapping growth. In development, use "dynamic": true to discover fields, then lock down the mapping for production.

Step 1 — Log producer

The producer simulates a fleet of microservices generating structured log events. Each event matches the index mapping exactly — all fields are present and correctly typed. In a real system, your application services would call a logging library that serializes to this format.

log_producer.py Lesson 28 · produce structured log events
python log_producer.py --rate 200
[INFO] Produced 5,000 log events … [INFO] Produced 10,000 log events … [INFO] Produced 15,000 log events … ^C [INFO] Stopped. Total produced: 18,341

Step 2 — OpenSearch consumer with bulk indexing

Indexing one document per Kafka message would be catastrophic for performance — each index call is an HTTP round trip. Instead, the consumer accumulates a local buffer of up to 500 documents, then flushes them to OpenSearch in a single bulk request. The index name rotates by date (logs-2025-01) to enable efficient lifecycle management.

opensearch_consumer.py Lesson 28 · consume + bulk index with date rotation
python opensearch_consumer.py
[INFO] Consumer started. group=opensearch-sink-v1 topic=api.logs [INFO] Flushed 500 docs. ok=500 err=0 cumulative: ok=500 err=0 [INFO] Flushed 500 docs. ok=500 err=0 cumulative: ok=1,000 err=0 [INFO] Flushed 341 docs. ok=341 err=0 cumulative: ok=1,341 err=0 [INFO] Shutdown. total_ok=18,341 total_errors=0

Querying — find all ERROR logs in the last hour

OpenSearch Query DSL combines filters (exact match, time range) with aggregations (counts, percentiles). The query below finds all ERROR logs from the payments-api in the last hour and returns a count by host, allowing operations to quickly identify which pod is generating the most errors.

query_errors.py Lesson 28 · DSL query — ERROR logs last 1 hour
python query_errors.py
=== ERROR count by host (last 1h) === payments-api-pod-03 142 errors payments-api-pod-07 98 errors payments-api-pod-01 61 errors payments-api-pod-05 44 errors === Latency percentiles (ms) === p50=2147ms p95=3201ms p99=3481ms === Total hits: 345 ===
Use filter context, not query context, for exact matches

Wrapping term and range clauses inside filter (not must) tells OpenSearch to skip relevance scoring — the query only checks "does this doc match?" without computing a float score. This is significantly faster and allows the results to be cached. Always use filter for exact-match and time-range conditions. Only use must with match when you need full-text relevance scoring.

Index lifecycle — rollover after 10 GB or 30 days

A single OpenSearch index that grows indefinitely becomes slow to query and impossible to delete selectively. Index Lifecycle Management (ILM) — called ISM in OpenSearch — automates rollover: when an index exceeds 10 GB or 30 days old, a new index is created and the alias is updated. Old indices move to warm/cold storage or are deleted automatically.

rollover_policy.json Lesson 28 · ISM policy via curl or OpenSearch Dashboards
Consumer side of rollover: the consumer uses logs-* as a wildcard when querying across all historical indices. When writing, it targets the current month's index (e.g., logs-2025-01). When ISM rolls over, OpenSearch creates logs-2025-01-000002 and updates the alias — the consumer does not need to change. Write to the alias, not the concrete index name.

Backpressure — what happens when OpenSearch is slow

OpenSearch can become a bottleneck during heavy indexing: segment merging competes with writes, the JVM pauses for GC, or a cluster rebalance moves shards. When this happens, the bulk API returns 429 (Too Many Requests) or times out. The consumer's response determines whether logs are lost or safely buffered.

Scenario Kafka consumer lag Correct response
OpenSearch bulk returns 200 OK Low — consumer keeps up Commit Kafka offsets, clear buffer, continue
OpenSearch returns 429 Too Many Requests Growing — consumer slows Back off with exponential delay; do NOT commit offsets; retry the same batch
OpenSearch connection timeout Growing — consumer stalls Do NOT commit offsets; Kafka retains messages; consumer resumes when OpenSearch recovers
Mapping conflict (new field type mismatch) Low — one doc fails per batch Log the failed document, produce it to a dead-letter topic, continue with the rest of the batch
Kafka consumer lag is your observability signal

Monitor the consumer group lag on api.logs with kafka-consumer-groups.sh --describe --group opensearch-sink-v1. A growing lag (e.g., from 0 to 50 000 in 5 minutes) is an early warning that OpenSearch is falling behind. Alert when lag exceeds 10 000 messages so the on-call engineer can investigate before logs are delayed by more than a few minutes. The alert threshold depends on your log ingestion rate: at 1 000 logs/sec, lag of 10 000 = 10 seconds of delay.

Challenge — extend the pipeline

Two extensions to build
Challenge 1 — Filter DEBUG logs with Kafka Streams before indexing

Currently all log levels (DEBUG, INFO, WARN, ERROR) are indexed into OpenSearch, which wastes storage on low-value DEBUG logs. Add a Kafka Streams topology (or a simple Python filter consumer) that reads from api.logs, drops any message where level == "DEBUG", and produces the remaining messages to a new topic api.logs.filtered. Update opensearch_consumer.py to subscribe to api.logs.filtered instead of api.logs. Verify that no DEBUG logs appear in OpenSearch after the change.

Hint: a Python filter consumer is a standard Kafka consumer that reads from one topic and a producer that writes to another. No Kafka Streams library is needed for a simple filter like this.

Challenge 2 — Alert when ERROR rate exceeds 5%

Add a monitoring loop to opensearch_consumer.py (or as a separate script) that runs every 60 seconds: query OpenSearch for the count of ERROR logs and the total log count in the last 5 minutes, compute the error rate as a percentage, and print an alert if it exceeds 5%. Use the OpenSearch count API or an aggregation query. Bonus: distinguish error rate by service so you can alert on payments-api independently of auth-service.

Hint: use client.count(index="logs-*", body={"query": {"bool": {"filter": [...]}}})["count"] to get counts without returning documents.

Quiz — 4 questions

Question 1 of 4
You need to filter logs exactly by service = "payments-api". What OpenSearch field type should the service field use?
Question 2 of 4
Why does the consumer buffer 500 documents and send them in one bulk request rather than indexing each document individually?
Question 3 of 4
OpenSearch returns a 429 error during a bulk index call. What should the consumer do?
Question 4 of 4
An index rollover policy is configured to roll over when size exceeds 10 GB or age exceeds 30 days. The current index is 3 GB and 15 days old. What happens?
0
/ 4
Log analytics pipeline deployed.

Next: Airflow Fundamentals (Lesson 29).

Interview Q&A

Medium What is the difference between keyword and text field types in OpenSearch?
+

A keyword field stores the value as-is, without any tokenization or analysis. It is used for exact-match filtering (term query), sorting, and aggregations. You would use keyword for fields like service, level, trace_id, or environment — any field where you need level = "ERROR" to match exactly and not "error" or "ERRORS".

A text field is tokenized by an analyzer (by default the standard analyzer: lowercased, split on whitespace and punctuation). It is used for full-text search where you want message CONTAINS "timeout" to match "timeout connecting to db" even though the query word is not the entire field value. text fields cannot be used for aggregations or sorting — for that you need a keyword sub-field (a .keyword multi-field). The mapping decision has a direct performance impact: using text for low-cardinality exact-match fields like level wastes analysis CPU on every index call and makes aggregations impossible.

Hard How does Kafka help with backpressure when OpenSearch is slow?
+

When OpenSearch slows down (due to segment merging, GC pauses, or a node rebalance), the Python consumer's bulk calls take longer. Because the consumer commits Kafka offsets only after a successful bulk flush, no messages are lost. The consumer's poll rate drops, and the consumer group lag on the Kafka topic grows — messages accumulate in Kafka's durable log rather than being dropped.

This is the key advantage of Kafka as a buffer: Kafka retains messages for the configured retention period (default 7 days) regardless of how fast consumers are reading. When OpenSearch recovers, the consumer resumes from its last committed offset and processes the backlog. The lag metric (visible via kafka-consumer-groups.sh --describe) acts as a natural backpressure signal — it tells operators exactly how far behind the indexing pipeline is, in messages and in estimated time. Without Kafka, a direct log-shipper to OpenSearch would either block the application logging thread (causing latency) or drop logs when OpenSearch is too slow to accept them.

Medium What is a mapping explosion and how do you prevent it?
+

A mapping explosion occurs when an OpenSearch index accumulates thousands of field mappings, usually because dynamic: true allows arbitrary JSON fields to be mapped automatically. Each new field creates a new entry in the cluster state — a shared data structure that must fit in memory on every node. When cluster state becomes too large (thousands of field mappings across dozens of indices), every cluster operation slows down, and the master node can run out of heap memory.

Prevention: (1) Set "dynamic": "strict" in production — reject unknown fields at index time so a rogue log event cannot add new mappings; (2) Define an explicit mapping for every expected field before going to production; (3) Set index.mapping.total_fields.limit (default: 1000) to cap the maximum number of fields per index; (4) Use a structured log format (like the one in this lesson) so all services emit the same fields. If you truly need to store arbitrary JSON (e.g., a context blob), use object type with dynamic disabled for that sub-object only, and store the raw JSON as a single keyword or text field.

Easy What is the purpose of index rollover and how does it differ from deleting an old index?
+

Index rollover creates a new, empty index and updates an alias to point to it, while leaving the old index intact and queryable. The alias (logs-write) always points to the "active" index receiving new writes, while queries can use a wildcard pattern (logs-*) to search across all indices simultaneously. Rollover solves two problems: (1) preventing a single index from growing too large (large indices are slow to query, rebalance, and recover from failure); (2) enabling age-based data retention — after rollover, the old index is in a "warm" state and can be moved to cheaper storage or deleted automatically by the lifecycle policy.

Deleting an old index is a separate step that happens after rollover, controlled by the lifecycle policy's delete state. The key advantage of rollover + lifecycle policy over a cron job that deletes old data: it is declarative, auditable, and handles index age consistently regardless of when the data arrived.

Key takeaways

  • Index mapping is the most important design decision in OpenSearch — use keyword for exact-match fields (service, level, trace_id), text for full-text search (message body), and date for timestamps. Set "dynamic": "strict" in production to prevent mapping explosions from malformed log events.
  • Bulk indexing is essential for throughput: accumulate 200–1 000 documents and send them in a single HTTP request rather than one call per document. Add a time-based flush (every 1–2 seconds) so the buffer does not hold events indefinitely during low-volume periods.
  • Use Kafka as a durable buffer between log producers and OpenSearch — growing consumer lag during an OpenSearch slowdown is far better than lost logs. Never commit Kafka offsets until the bulk call succeeds.
  • Rotate indices by date or size with an ISM (Index State Management) policy. A logs-2025-01 index is easier to manage, delete, and archive than a single logs index that grows forever. Set rollover conditions at 10 GB or 30 days, whichever comes first.
  • Monitor consumer lag on the Kafka topic as the primary backpressure signal. Alert when lag exceeds a threshold equivalent to 30–60 seconds of ingestion delay — this catches OpenSearch slowdowns before they become multi-hour data gaps in dashboards.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

The foundation our AI platform engineers stand on

Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Data Engineer →