💼 Business context
Equity Stream Bank's fraud operations team needs to action on flagged transactions within 30 seconds of a transaction occurring — before the customer leaves the merchant. Their ML model runs inline with the payment gateway and produces a fraud score for every transaction. Transactions with fraud_score > 0.7 are written to a Kafka topic for downstream handling.
The data engineering team owns what happens after Kafka: the fraud signals must be stored in Cassandra for instant point lookups by the ops team (when a customer calls in), and simultaneously indexed in OpenSearch so the analytics team can build dashboards showing fraud patterns by geography, time, and transaction type.
A Spark Structured Streaming job that reads the transactions Kafka topic at 100 events/second, filters for fraud signals, writes to Cassandra via foreachBatch for sub-second lookup, and simultaneously bulk-indexes into OpenSearch for analytical queries. End-to-end latency target: under 5 seconds from Kafka produce to Cassandra commit.
🏗️ Architecture
This pipeline fans out from a single Kafka source to two sinks with very different access patterns — Cassandra for operational point lookups (low latency, high write throughput) and OpenSearch for analytical range queries (full-text search, aggregations, dashboards).
Equity Stream Bank — Real-Time Fraud Detection Pipeline
100 events/sec
3 partitions
foreachBatch
ops lookups
analyst dashboards
🌊 Step 1 — Kafka producer at 100 events/sec
The producer simulates the ML gateway output. It produces JSON events to the transactions topic, with approximately 8% of events flagged as fraud (matching typical bank fraud rates). The event schema is fixed — any consumer relying on this schema must not change without coordinating schema evolution.
⚡ Step 2 — Spark Structured Streaming job
The Spark job reads from Kafka, parses the JSON payload, applies a watermark for late data handling, filters for fraud signals, and uses foreachBatch to write to both sinks in a single micro-batch pass.
🗄️ Step 3 — Cassandra DDL & idempotent writes
Cassandra is chosen for the ops sink because it delivers single-digit millisecond reads by primary key — the exact access pattern the fraud ops team needs when a customer calls in. The table is modelled for the primary query: lookup by txn_id.
🔍 Step 4 — OpenSearch index mapping
Create the index template before the first Spark micro-batch lands. The template applies automatically to any index matching fraud-events-*. Explicit field mappings prevent OpenSearch from auto-detecting fraud_score as long (integer) instead of float.
✅ Step 5 — Verify both sinks
After the first micro-batch commits, verify data in both Cassandra and OpenSearch. These commands serve as smoke tests you can add to your CI/CD pipeline.
maxOffsetsPerTrigger.
⏱ Watermarking & checkpointing
Two Spark Structured Streaming features ensure correctness under failure and late data.
Watermarking for late data
Mobile transactions in East Africa are sometimes delayed — a transaction initiated at 14:30 might reach Kafka at 14:31 due to intermittent connectivity. The 30-second watermark instructs Spark to keep late-arriving events in state for up to 30 seconds past the current event-time watermark before dropping them.
Checkpoint location
The checkpoint directory records two things: the last committed Kafka offsets, and any stateful aggregation state (like the windowed fraud rate above). On restart, Spark reads the checkpoint and resumes exactly where it left off — no events are lost or reprocessed.
In production, store checkpoints on a durable, distributed filesystem — S3 (s3a://your-bucket/checkpoints/fraud_pipeline) or HDFS (hdfs://namenode:9000/checkpoints/fraud_pipeline). Local disk checkpoints are lost if the worker node is replaced. An S3 checkpoint adds <50ms per micro-batch for the metadata write — a worthwhile trade for durability.
🔐 Production considerations
Before going to production — security checklist
- Kafka SSL/SASL: configure
security.protocol=SASL_SSL,sasl.mechanism=SCRAM-SHA-512in Spark's Kafka options. Never run Kafka without auth in a bank environment. - Cassandra authentication: enable
authenticator: PasswordAuthenticatorin cassandra.yaml. Create a dedicated service account with WRITE-only permissions on thefraud_detectionkeyspace. - OpenSearch security: enable the security plugin. Create a service account with index:data/write/* permissions on
fraud-events-*and index:admin/create on the same pattern for auto-index creation. - Checkpoint on S3: replace
./checkpoints/withs3a://equitystream-spark-checkpoints/fraud_pipeline/. Configure IAM role or access keys via Hadoop configuration in the SparkSession. - Secrets management: store Cassandra and OpenSearch credentials in Vault or AWS Secrets Manager. Retrieve at runtime — never hard-code credentials in job code or DAG config.
- Monitoring: expose
query.lastProgressmetrics to Prometheus via the Spark metrics sink. Alert whenbatchDuration> 80% of trigger interval ornumInputRowsdrops to zero (producer failure).
🏆 Challenge tasks
Implement the 5-minute / 1-minute sliding window aggregation from the watermarking section. Write the windowed results to a separate Cassandra table user_fraud_windows with primary key (user_id, window_start). A fraud ops analyst should be able to query: "which users had a fraud rate above 50% in any 5-minute window today?"
Hint: you need outputMode("update") for windowed aggregations with watermarks, not "append". Append mode does not emit updated window results.
Inside write_batch(), add a check: if any row in the batch has fraud_score > 0.98, send a Slack alert via webhook.
import requests
SLACK_WEBHOOK = "https://hooks.slack.com/services/T.../B.../..."
high_risk = [r for r in rows if r.fraud_score > 0.98]
if high_risk:
requests.post(SLACK_WEBHOOK, json={
"text": f":rotating_light: {len(high_risk)} extreme fraud signals (score > 0.98) in batch {batch_id}. "
f"Top: txn_id={high_risk[0].txn_id}, score={high_risk[0].fraud_score:.4f}"
})
🧠 Quiz — 4 questions
foreachBatch used to write to Cassandra and OpenSearch instead of the built-in Kafka or console sinks?IF NOT EXISTS. What does this ensure in the context of Spark Structured Streaming?outputMode("update") instead of outputMode("append")?You have completed the Big Data Engineering track. Congratulations!
🎤 Interview Q&A
INSERT IF NOT EXISTS keyed on txn_id ensures that replaying a batch does not create duplicate rows — the second insert is a lightweight no-op. OpenSearch's bulk index with an explicit document _id set to txn_id also ensures idempotency — indexing the same document twice with the same _id overwrites the first, not appends. So the pipeline achieves effectively exactly-once semantics through at-least-once delivery from Spark plus idempotent writes to both sinks. True EOS (Spark's EXACTLY_ONCE trigger) is available but requires a transactional sink that Cassandra and OpenSearch do not support natively.
withWatermark("ts", "30 seconds") tells Spark how long to retain state for late-arriving events. Without a watermark, Spark would keep state for every window forever — which would cause unbounded memory growth for long-running streaming jobs. With withWatermark("ts", "30 seconds"), Spark tracks the maximum event time seen so far in the stream and drops any event whose ts is more than 30 seconds behind that maximum. This means: events that arrive up to 30 seconds late are still counted in the correct window; events that arrive more than 30 seconds late are silently dropped. The watermark also signals to Spark when it is safe to finalise and emit window results — a window result is emitted once the watermark advances past the window's end time, meaning no more late events can fall into it. Choosing the watermark delay requires understanding your data: mobile transactions in Africa with intermittent connectivity might need a 30-second to 2-minute watermark, while a bank's in-datacenter payment gateway might need only 5 seconds.
✅ Key takeaways
- Spark Structured Streaming's
foreachBatchAPI is the production pattern for writing to systems that do not have a native Spark sink — it gives you a full Spark DataFrame and the Python ecosystem per micro-batch, enabling fan-out writes to Cassandra and OpenSearch in a single pass. - Idempotent writes are the key to safety in streaming pipelines. Cassandra's
IF NOT EXISTSand OpenSearch's explicit document_idboth ensure that Spark batch retries (which happen on any failure before checkpoint commit) do not create duplicates — achieving effective exactly-once semantics without transactional overhead. - The watermark
.withWatermark("ts", "30 seconds")bounds the state Spark keeps in memory and defines how long to wait for late events. Choose the watermark delay based on your network characteristics — too short drops legitimate late events; too long grows state memory indefinitely. - Checkpoints are not optional in production streaming. Store them on a durable distributed filesystem (S3 or HDFS) so the job can survive driver restarts without replaying from the beginning of the Kafka topic or losing windowed state.
- The polyglot persistence pattern — Cassandra for operational point reads, OpenSearch for analytical range queries and dashboards — is the correct architecture when two consumer teams have fundamentally different access patterns. Writing to both in the same micro-batch adds negligible latency versus the alternative of two separate streaming jobs.
- Monitor three streaming metrics proactively: input rate (tracks producer health), processing rate (must exceed input rate to stay stable), and batch duration (must stay below the trigger interval). When batch duration approaches the trigger interval, the job is falling behind and will begin accumulating lag — scale up executors before the lag becomes unrecoverable.
Track Complete!
You have completed all 35 lessons of the Big Data Engineering with Spark & Kafka track at Ramayai AI Academy. From HDFS fundamentals to real-time fraud detection — you have built the full stack.
You finished the track. Now build it in production.
Join the Best AI Program: The AI-Era Data Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Data Engineer →