Stream Processing: Apache Flink vs Spark Streaming
Modern data platforms require real-time processing of high-volume event streams for fraud detection, real-time analytics dashboards, and automated alerting. Traditional batch processing architectures process data in scheduled nightly chunks, introducing hours of latency before insights become available.
Distributed Stream Processing Frameworks process continuous event streams in sub-second timeframes. Apache Flink is a true event-driven stream processor that handles records individually with sub-millisecond latency and fine-grained RocksDB state management. Spark Structured Streaming uses a micro-batch architecture that processes incoming events in small configurable time windows (e.g. 100ms triggers). This guide compares Apache Flink and Spark Streaming on processing latency, state management, event-time watermarking, and fault tolerance.
Mental Model: Real-Time Stateful Event-Driven Processing vs Micro-Batch Execution
Stream processing engines handle infinite event streams arriving from Apache Kafka or Apache Pulsar:
1. Event-Driven Streaming (Flink): Processes every incoming record immediately as it arrives in memory. Delivers sub-10ms processing latency and supports complex event processing (CEP). 2. Micro-Batch Streaming (Spark): Batches incoming records into tiny datasets collected over a short trigger interval ($100\text{ms} - 1\text{s}$) and executes them as miniature Spark RDD jobs. For real-time analytics and message queue integrations, review building realtime analytics pipelines apache pinot druid and building high throughput message queues apache pulsar kafka.
Quick reference
- Flink event-driven engine processes individual events with sub-millisecond latency SLAs.
- Spark Structured Streaming micro-batching achieves high throughput with ~100ms latency.
- Both engines integrate natively with Apache Kafka, Apache Pulsar, and Cloud Storage.
- Decouples stream ingestion from downstream real-time analytics storage engines.
- Powers real-time data pipelines at Uber, Netflix, Stripe, Alibaba, and CoreConcept.
Remember this
Choose Apache Flink for sub-millisecond event processing and Spark Streaming for unified batch/stream analytics.
Apache Flink State Backend (RocksDB) & Chandy-Lamport Checkpointing
Stateful stream processing requires maintaining intermediate state (e.g. running account totals or active session windows) across millions of events:
- RocksDB State Backend: Flink stores active key-value state in local embedded RocksDB instances on worker nodes, allowing state sizes to exceed available JVM heap memory. - Chandy-Lamport Checkpointing: Flink injects barrier markers into input event streams to take consistent, asynchronous snapshots of operator state without pausing stream execution.
Quick reference
- RocksDB embedded KV store enables multi-terabyte state management without JVM GC pauses.
- Chandy-Lamport algorithm takes consistent distributed snapshots during live processing.
- Incremental checkpointing uploads only modified state blocks to Cloud Storage (S3/GCS).
- Provides exactly-once processing guarantees across source, operator, and sink failures.
- Supports dynamic state schema migration for long-running streaming applications.
Remember this
Deploy Flink RocksDB state backends and Chandy-Lamport checkpointing for fault-tolerant stateful streaming.
Spark Structured Streaming Micro-Batching vs Continuous Processing Engine
Spark Structured Streaming unifies stream and batch processing using the Catalyst Optimizer and Tungsten execution engine:
- Micro-Batch Execution: Evaluates streaming queries as incremental computations on continuously growing unbounded DataFrame tables. - Continuous Processing Engine (Experimental): Bypasses RDD micro-batch scheduling to achieve sub-millisecond end-to-end latency for simple map-like transformations:
1# PySpark Structured Streaming Example2from pyspark.sql import SparkSession3from pyspark.sql.functions import expr4 5spark = SparkSession.builder.appName("KafkaStreamProcessor").getOrCreate()6 7df = spark.readStream \8 .format("kafka") \9 .option("kafka.bootstrap.servers", "localhost:9092") \10 .option("subscribe", "orders") \11 .load()12 13query = df.writeStream \14 .format("console") \15 .outputMode("append") \16 .trigger(processingTime='1 second') \17 .start()Quick reference
- Catalyst Optimizer optimizes streaming DataFrame queries with the same engine as batch SQL.
- Tungsten cache-conscious memory layout optimizes CPU L1/L2 cache usage.
- Unifies batch ETL pipelines and real-time streaming queries under a single codebase.
- Supports stateful operations via mapGroupsWithState for custom session aggregation.
- Ideal for data teams with existing Apache Spark infrastructure.
Remember this
Deploy Spark Structured Streaming to unify batch ETL and real-time streaming under one framework.
Windowing Operations (Tumbling, Sliding, Session) & Watermarking Event Time
Handling out-of-order events caused by network delays requires managing Event Time (when the event occurred) versus Processing Time (when the stream processor received it):
1. Watermarking: A watermark $W(t)$ is a monotonic timestamp barrier indicating that no further events with event timestamp $t' le W(t)$ will arrive. 2. Window Types: Tumbling (non-overlapping fixed intervals), Sliding (overlapping intervals), and Session (gap-based user activity windows).
Quick reference
- Event Time processing guarantees correct calculation despite network latencies.
- Watermarking bounds out-of-order event delays and triggers window evaluations safely.
- Tumbling windows group data into non-overlapping fixed time buckets (e.g. 5-minute sales).
- Session windows group user actions based on periods of inactivity (e.g. 30-min idle gap).
- Handles late-arriving data gracefully via allowed lateness window extensions.
Remember this
Configure event-time watermarking to process out-of-order event streams accurately.
Key takeaway
To test Apache Flink locally, run docker run -d --name flink-jobmanager -p 8081:8081 apache/flink:latest jobmanager and access the web UI at http://localhost:8081.
Related Articles
Explore this topic