What we're building
This mini-build creates a centralized log analytics pipeline for a fintech API. Application services (payments-api, auth-service, wallet-service) emit structured JSON logs. Those logs flow into Kafka, then a Python consumer bulk-indexes them into OpenSearch, where operations teams can query, visualize, and alert on error rates in real time.
This architecture is the open-source equivalent of the ELK (Elasticsearch/Logstash/Kibana) stack, but using Kafka as a durable buffer between log producers and the search index — a critical improvement because OpenSearch can become a bottleneck under log bursts.
Index mapping — the foundation of fast queries
OpenSearch mappings define how each field is stored and indexed. Getting this wrong causes either bloated indices (mapping everything as text with full-text analysis) or broken queries (using text for fields you need to filter exactly). The key distinction: keyword for exact-match fields, text for full-text search, date for timestamps.
| Field | Type | Rationale |
|---|---|---|
@timestamp |
date | Enables time-range queries and efficient time-based sorting |
service |
keyword | Exact-match filter: service = "payments-api" |
level |
keyword | Exact-match filter: level = "ERROR"; low cardinality (INFO, WARN, ERROR, DEBUG) |
message |
text | Full-text search: message CONTAINS "timeout" |
trace_id |
keyword | Exact-match lookup to correlate logs across services for one request |
latency_ms |
integer | Range queries: latency_ms > 1000; aggregations: p99 latency |
status_code |
integer | Numeric filter: status_code = 503; aggregation: count by code |
error |
boolean | Fast filter: error = true without parsing the level field |
By default, OpenSearch dynamically maps every new field it sees. A single malformed log event with 200 unexpected fields can create 200 new mappings, bloating the cluster state and slowing down every operation. Setting "dynamic": "strict" rejects documents with unexpected fields — you get a 400 error instead of a runaway mapping growth. In development, use "dynamic": true to discover fields, then lock down the mapping for production.
Step 1 — Log producer
The producer simulates a fleet of microservices generating structured log events. Each event matches the index mapping exactly — all fields are present and correctly typed. In a real system, your application services would call a logging library that serializes to this format.
Step 2 — OpenSearch consumer with bulk indexing
Indexing one document per Kafka message would be catastrophic for performance — each index call is an HTTP round trip. Instead, the consumer accumulates a local buffer of up to 500 documents, then flushes them to OpenSearch in a single bulk request. The index name rotates by date (logs-2025-01) to enable efficient lifecycle management.
Querying — find all ERROR logs in the last hour
OpenSearch Query DSL combines filters (exact match, time range) with aggregations (counts, percentiles). The query below finds all ERROR logs from the payments-api in the last hour and returns a count by host, allowing operations to quickly identify which pod is generating the most errors.
Wrapping term and range clauses inside filter (not must) tells OpenSearch to skip relevance scoring — the query only checks "does this doc match?" without computing a float score. This is significantly faster and allows the results to be cached. Always use filter for exact-match and time-range conditions. Only use must with match when you need full-text relevance scoring.
Index lifecycle — rollover after 10 GB or 30 days
A single OpenSearch index that grows indefinitely becomes slow to query and impossible to delete selectively. Index Lifecycle Management (ILM) — called ISM in OpenSearch — automates rollover: when an index exceeds 10 GB or 30 days old, a new index is created and the alias is updated. Old indices move to warm/cold storage or are deleted automatically.
logs-* as a wildcard when querying across all historical indices. When writing, it targets the current month's index (e.g., logs-2025-01). When ISM rolls over, OpenSearch creates logs-2025-01-000002 and updates the alias — the consumer does not need to change. Write to the alias, not the concrete index name.
Backpressure — what happens when OpenSearch is slow
OpenSearch can become a bottleneck during heavy indexing: segment merging competes with writes, the JVM pauses for GC, or a cluster rebalance moves shards. When this happens, the bulk API returns 429 (Too Many Requests) or times out. The consumer's response determines whether logs are lost or safely buffered.
| Scenario | Kafka consumer lag | Correct response |
|---|---|---|
| OpenSearch bulk returns 200 OK | Low — consumer keeps up | Commit Kafka offsets, clear buffer, continue |
| OpenSearch returns 429 Too Many Requests | Growing — consumer slows | Back off with exponential delay; do NOT commit offsets; retry the same batch |
| OpenSearch connection timeout | Growing — consumer stalls | Do NOT commit offsets; Kafka retains messages; consumer resumes when OpenSearch recovers |
| Mapping conflict (new field type mismatch) | Low — one doc fails per batch | Log the failed document, produce it to a dead-letter topic, continue with the rest of the batch |
Monitor the consumer group lag on api.logs with kafka-consumer-groups.sh --describe --group opensearch-sink-v1. A growing lag (e.g., from 0 to 50 000 in 5 minutes) is an early warning that OpenSearch is falling behind. Alert when lag exceeds 10 000 messages so the on-call engineer can investigate before logs are delayed by more than a few minutes. The alert threshold depends on your log ingestion rate: at 1 000 logs/sec, lag of 10 000 = 10 seconds of delay.
Challenge — extend the pipeline
Currently all log levels (DEBUG, INFO, WARN, ERROR) are indexed into OpenSearch, which wastes storage on low-value DEBUG logs. Add a Kafka Streams topology (or a simple Python filter consumer) that reads from api.logs, drops any message where level == "DEBUG", and produces the remaining messages to a new topic api.logs.filtered. Update opensearch_consumer.py to subscribe to api.logs.filtered instead of api.logs. Verify that no DEBUG logs appear in OpenSearch after the change.
Hint: a Python filter consumer is a standard Kafka consumer that reads from one topic and a producer that writes to another. No Kafka Streams library is needed for a simple filter like this.
Add a monitoring loop to opensearch_consumer.py (or as a separate script) that runs every 60 seconds: query OpenSearch for the count of ERROR logs and the total log count in the last 5 minutes, compute the error rate as a percentage, and print an alert if it exceeds 5%. Use the OpenSearch count API or an aggregation query. Bonus: distinguish error rate by service so you can alert on payments-api independently of auth-service.
Hint: use client.count(index="logs-*", body={"query": {"bool": {"filter": [...]}}})["count"] to get counts without returning documents.
Quiz — 4 questions
service = "payments-api". What OpenSearch field type should the service field use?Next: Airflow Fundamentals (Lesson 29).
Interview Q&A
A keyword field stores the value as-is, without any tokenization or analysis. It is used for exact-match filtering (term query), sorting, and aggregations. You would use keyword for fields like service, level, trace_id, or environment — any field where you need level = "ERROR" to match exactly and not "error" or "ERRORS".
A text field is tokenized by an analyzer (by default the standard analyzer: lowercased, split on whitespace and punctuation). It is used for full-text search where you want message CONTAINS "timeout" to match "timeout connecting to db" even though the query word is not the entire field value. text fields cannot be used for aggregations or sorting — for that you need a keyword sub-field (a .keyword multi-field). The mapping decision has a direct performance impact: using text for low-cardinality exact-match fields like level wastes analysis CPU on every index call and makes aggregations impossible.
When OpenSearch slows down (due to segment merging, GC pauses, or a node rebalance), the Python consumer's bulk calls take longer. Because the consumer commits Kafka offsets only after a successful bulk flush, no messages are lost. The consumer's poll rate drops, and the consumer group lag on the Kafka topic grows — messages accumulate in Kafka's durable log rather than being dropped.
This is the key advantage of Kafka as a buffer: Kafka retains messages for the configured retention period (default 7 days) regardless of how fast consumers are reading. When OpenSearch recovers, the consumer resumes from its last committed offset and processes the backlog. The lag metric (visible via kafka-consumer-groups.sh --describe) acts as a natural backpressure signal — it tells operators exactly how far behind the indexing pipeline is, in messages and in estimated time. Without Kafka, a direct log-shipper to OpenSearch would either block the application logging thread (causing latency) or drop logs when OpenSearch is too slow to accept them.
A mapping explosion occurs when an OpenSearch index accumulates thousands of field mappings, usually because dynamic: true allows arbitrary JSON fields to be mapped automatically. Each new field creates a new entry in the cluster state — a shared data structure that must fit in memory on every node. When cluster state becomes too large (thousands of field mappings across dozens of indices), every cluster operation slows down, and the master node can run out of heap memory.
Prevention: (1) Set "dynamic": "strict" in production — reject unknown fields at index time so a rogue log event cannot add new mappings; (2) Define an explicit mapping for every expected field before going to production; (3) Set index.mapping.total_fields.limit (default: 1000) to cap the maximum number of fields per index; (4) Use a structured log format (like the one in this lesson) so all services emit the same fields. If you truly need to store arbitrary JSON (e.g., a context blob), use object type with dynamic disabled for that sub-object only, and store the raw JSON as a single keyword or text field.
Index rollover creates a new, empty index and updates an alias to point to it, while leaving the old index intact and queryable. The alias (logs-write) always points to the "active" index receiving new writes, while queries can use a wildcard pattern (logs-*) to search across all indices simultaneously. Rollover solves two problems: (1) preventing a single index from growing too large (large indices are slow to query, rebalance, and recover from failure); (2) enabling age-based data retention — after rollover, the old index is in a "warm" state and can be moved to cheaper storage or deleted automatically by the lifecycle policy.
Deleting an old index is a separate step that happens after rollover, controlled by the lifecycle policy's delete state. The key advantage of rollover + lifecycle policy over a cron job that deletes old data: it is declarative, auditable, and handles index age consistently regardless of when the data arrived.
Key takeaways
- Index mapping is the most important design decision in OpenSearch — use
keywordfor exact-match fields (service, level, trace_id),textfor full-text search (message body), anddatefor timestamps. Set"dynamic": "strict"in production to prevent mapping explosions from malformed log events. - Bulk indexing is essential for throughput: accumulate 200–1 000 documents and send them in a single HTTP request rather than one call per document. Add a time-based flush (every 1–2 seconds) so the buffer does not hold events indefinitely during low-volume periods.
- Use Kafka as a durable buffer between log producers and OpenSearch — growing consumer lag during an OpenSearch slowdown is far better than lost logs. Never commit Kafka offsets until the bulk call succeeds.
- Rotate indices by date or size with an ISM (Index State Management) policy. A
logs-2025-01index is easier to manage, delete, and archive than a singlelogsindex that grows forever. Set rollover conditions at 10 GB or 30 days, whichever comes first. - Monitor consumer lag on the Kafka topic as the primary backpressure signal. Alert when lag exceeds a threshold equivalent to 30–60 seconds of ingestion delay — this catches OpenSearch slowdowns before they become multi-hour data gaps in dashboards.
The foundation our AI platform engineers stand on
Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Data Engineer →