A Comprehensive Benchmarking Analysis of Fault Recovery in Stream Processing Frameworks

1 INTRODUCTION

• Fault tolerance remains under-measured despite extensive research on stream-processing performance and its importance for continuously operating systems.Failures can occur across infrastructure and software layers, making swift recovery essential for continuous, low-latency processing.Source: §1 · p. 1
• The paper asks how quickly executions recover, how failures affect performance, how correlated worker failures behave, and how stable recovery is across recurrent failures.Source: §1 · p. 1
• The study extends ShuffleBench with fault-tolerance measurements, automated execution and data collection in Kubernetes-based cloud environments.The extension builds on the prior benchmark and provides an experimental foundation for comparing modern frameworks.Source: §1 · pp. 1–2
• Its methodology measures throughput, latency, recovery impact on performance, stability, and recovery time using chaos-engineering-inspired recurrent failure injection.Source: §1 · p. 2
• The evaluation covers Flink, Kafka Streams, and Spark Structured Streaming because of their industry acceptance and academic relevance.Experimental data and an extended replication package are available as supplementary material.Source: §1 · p. 2

2.1 Distributed Stream Processing

• Stream-processing frameworks distribute continuous filtering, transformation, and aggregation across workers while abstracting cluster, state, and time management.Stateful operations route records with the same key to the same worker, enabling distributed processing of dataflow subgraphs.Source: §2.1 · p. 2

2.2 Fault Tolerance

• Fault tolerance applies distributed-systems advances to preserve correct results despite failures, addressing a longstanding consistency challenge.Source: §2.2 · p. 2
• At-least-once semantics prevent data loss but may produce duplicate output during recovery, whereas exactly-once semantics avoid lost or duplicated results.Exactly-once processing requires additional implementation and affects processing performance.Source: §2.2 · p. 2
• The paper focuses on output semantics because required guarantees vary by application and use case.For example, duplicate alerts may be unacceptable, while occasional duplicate dashboard metrics may be tolerable.Source: §2.2 · p. 2
• Fault recovery restores normal operation after failure, whereas fault tolerance aims to prevent downtime for high availability.Source: §2.2 · p. 2

3.1 State-of-the-art

• Existing studies examine fault recovery across frameworks, fault-tolerance taxonomies, single-node systems, deployment trade-offs, checkpoint protocols, and correlated-failure recovery.Relevant evaluations include multiple fault types and settings across Spark, Flink, Spark Structured Streaming, and Kafka Streams.Source: §3.1 · p. 3

3.2 Research Gaps

• Updated measurements are needed because frameworks continue developing fault-tolerance features and optimizations.Source: §3.2 · p. 3
• Few studies compare failures affecting multiple worker instances, although a physical-machine crash can affect several virtual machines or pods.The impact of such more critical failures on running stream-processing applications remains unclear.Source: §3.2 · p. 3
• The literature mainly studies one-time failures, leaving recovery stability across recurrent and unpredictable failures insufficiently characterized.Recovery can vary with checkpoint timing, affected workers, and the effectiveness of subsequent recovery.Source: §3.2 · p. 3

4.1 Benchmark

• ShuffleBench evaluates state-local aggregation workloads in which records are shuffled by key before custom aggregation, using configurable Kubernetes tooling.The benchmark abstracts aggregation logic as a black-box component and supports implementations for state-of-the-art frameworks.Source: §4.1 · p. 3
• The fault-tolerance extension adds benchmark components for failure scenarios and recovery metrics in an Amazon EKS cluster with dedicated broker, manager, generator, and worker instances.The deployment uses eight worker pods with 1 CPU and 3 GB of RAM on c5.large instances.Source: §4.1 · pp. 3–4, Figure 2 · p. 4
• The workload uses 20,000 consumers with 50% aggregate selectivity, 40 input and output Kafka partitions, and 56-minute experiment runs.The duration supports quantitative measurement, recurrent failure injection, and subsequent recovery.Source: §4.1 · p. 4

4.2 Frameworks Deployment

• Flink and Spark use checkpointing for at-least-once semantics, while Kafka Streams uses Kafka-backed changelogs to restore failed task state.Flink and Spark store checkpoint state on an Amazon EFS volume mounted in the pods.Source: §4.2 · p. 4
• Flink checkpointing and Kafka Streams commits use 2-second intervals, while Spark uses comparable post-warmup micro-batching; Flink also runs in reactive mode.Spark additionally uses write-ahead logs and 16 SQL shuffle partitions.Source: §4.2 · p. 4

5.1 Goals

• The evaluation creates comparable framework deployments, tests periodic and recurrent failures, and emphasizes realistic worker-instance faults.The goals include comparable throughput capabilities, comprehensive failure evaluation, and scenarios reflecting likely real-world failures.Source: §5.1 · p. 4

5.2 Failure Injection Scenarios

• Chaos Mesh injects failures in two scenarios: worker-pod kills and operating-system crashes affecting worker pods.The experiments are explicitly inspired by chaos engineering.Source: §5.2 · p. 4
• Pod kills remove worker state and records, trigger replacement pods within usually 2–10 seconds, and recur every 5 minutes to assess recovery stability.The study varies simultaneous kills across 1, 2, and 4 worker pods to represent simple and critical failures.Source: §5.2 · p. 4
• Pod crashes remove local state without creating replacement pods; two workers remain unavailable for 5 minutes, with failures repeated every 10 minutes.This scenario is intended to represent a critical failure and test resilience under reduced resources.Source: §5.2 · pp. 4–5

5.3 Metrics

• The benchmark measures input and output throughput, event-latency percentiles, consumer lag, fault-recovery performance, guarantee impact, stability, recovery time, and system resources.Throughput is collected as a five-second moving average, latency over ten-second windows, and resource indicators every two seconds.Source: §5.3 · p. 5
• The method is applied across the results sections, with supplementary material containing complementary findings beyond the most insightful reported results.The benchmark tooling supports automated execution and data collection in Kubernetes-based cloud environments.Source: §5.3 · p. 5

6.1 ALOS Recovery Characterization

• After worker-pod kills, Flink and Kafka Streams rebalance partitions while Spark Structured Streaming assigns tasks to replacement workers.Kubernetes creates replacement pods, after which frameworks reschedule computation onto the worker pool.Source: §6.1 · p. 5

6 FAULT RECOVERY CHARACTERIZATION

• Flink generally recovers fastest and most consistently, while Kafka Streams shows recurrent performance instability and Spark Structured Streaming remains stable with higher latency.The comparison spans application throughput, latency, lag, resource behavior, and both ALOS and EOS scenarios.Source: §6.1 · p. 5, §6.2 · p. 9

6.1 ALOS Recovery Characterization

• Kafka Streams’ partition assignment can overload some workers and underutilize others, producing lag accumulation, unstable latency, and variable recovery across recurrent failures.A rebalance can sometimes process lag quickly, but suboptimal assignments often prolong recovery and increase resource fluctuations.Source: §6.1 · pp. 5–6
• Spark Structured Streaming stabilizes its dynamic micro-batching after several minutes, but its event latency remains significantly higher than event-driven frameworks.The self-regulated interval settles around 2 seconds per batch, while fine-grained task scheduling produces dynamic CPU usage and potentially adds overhead.Source: §6.1 · p. 5
• Operating-system memory comparisons should be interpreted cautiously because JVM allocation and computation differences make operating-system memory usage an unreliable basis for cross-framework conclusions.The authors specifically caution against treating Spark’s larger observed memory allocation as definitive evidence.Source: §6.1 · p. 5
• Under critical crashes without replacement pods, all frameworks recover when the failure ends, but Flink recovers faster, Kafka Streams accumulates the most lag, and Spark fluctuates during task reprocessing.Kafka Streams took almost 3 minutes to reduce latency, whereas Spark increased its micro-batching interval to around 3 seconds.Source: §6.1 · p. 7

6.2 EOS Recovery Characterization

• Across repeated failures, Flink maintains stable recovery, Spark shows similar performance impacts across failures, and Kafka Streams exhibits phase-dependent recovery lasting several minutes.The recurrent-failure results extend the single-failure characterization to ten kills and five crash episodes.Source: Figure 7 · p. 9, Figure 8 · p. 9
• For exactly-once processing, Flink provides better performance and fault recovery than Kafka Streams; Flink recovered in less than a minute, while Kafka Streams remained unstable because of accumulated lag.Spark Structured Streaming is excluded from EOS output experiments because it cannot natively guarantee exactly-once semantics when writing to Kafka.Source: §6.2 · pp. 7–9

• Kafka Streams recovery follows multistage patterns lasting several minutes, with variable lag peaks, throughput drops, and latency impacts across failures.After one failure, recovery took around 4 minutes to return to a comparable state.Source: §7 · p. 9
• Kafka Streams’ instability is mainly associated with which worker fails and how subsequent rebalances are enacted.Source: §7 · p. 9
• Spark Structured Streaming shows minor fluctuations after recurrent failures, attributed to dynamic micro-batching and fine-grained task-to-executor mapping, while performance impact remains similar.Source: §7 · p. 9
• Flink and Spark Structured Streaming provide stable fault recovery, whereas Kafka Streams has unstable behavior with volatile recovery times.Source: §7 · p. 9
• Across repeated worker-instance crashes, Flink and Spark recoveries remain similar, while Kafka Streams shows differing impacts on latency, lag accumulation, and lag reduction.Source: §7 · pp. 9–10

• The detector identifies failures from moving-average and standard-deviation deviations, then marks recovery when throughput remains within 15% of normal for at least 40 seconds.Source: §8 · p. 10
• The method defines normal processing using the interval between warm-up and the first failure, typically the second through fifth execution minutes.Source: §8 · p. 10
• The detector is customizable but uses parameter values without ground truths, making its recovery estimates suitable for confirmation through visual inspection.Source: §8 · p. 10
• Flink achieves the fastest overall recovery time for both throughput-related metrics, while Kafka Streams is slower and more variable.Source: §8 · p. 10, Figure 10 · p. 10
• In four of ten Kafka Streams failures with four pods killed, complete recovery was not achieved within the five-minute window, so recovery time was set to the failure interval.This convention can underestimate recovery when completion would require longer than the interval.Source: §8 · p. 10

9.1 Observations Summary and Takeaways

• Flink and Spark Structured Streaming recover consistently fast and stably under at-least-once guarantees, while Kafka Streams recovers more slowly and variably.Source: §9.1 · p. 11
• Spark Structured Streaming can be suitable when low latency is not required, while its native support for exactly-once guarantees is limited.Source: §9.1 · p. 11

9.2 Findings discussion

• The study’s novelty includes measuring stability across recurrent failures while accounting for newer Flink enhancements such as reactive mode, checkpointing, and load-management optimizations.Source: §9.2 · p. 11
• Kafka Streams provides limited fault recovery because recurrent failure injection and recovery-time measurement exposed weaknesses not captured by earlier single-failure evaluations.Source: §9.2 · p. 11

9.1 Observations Summary and Takeaways

• Flink provides fast, stable recovery and is the most suitable framework for use cases requiring exactly-once semantics.Source: §9.2 · p. 11, §9.1 · p. 11

10 CONCLUSIONS

• Fault recovery materially affects application performance and quality of service, making it a crucial criterion for production framework selection.Source: §10 · p. 11

10.1 Results Implications

• Kafka Streams needs fault-recovery improvements because unstable post-failure behavior can compromise quality of service and service-level agreements.Source: §10.1 · p. 11
• Flink’s positive recovery results indicate strong suitability for resilience and continuous service availability, while recovery performance can differ substantially across failures.Source: §10.1 · p. 11
• The method may inform runtime self-adaptation and evaluation of other long-running distributed systems because such adaptations involve fault-recovery procedures.Source: §10.1 · p. 11

10.2 Limitations and Threats to Validity

• The study remains bounded by prior internal-validity threats and lacks comprehensive statistical evidence about failure rates in large-scale public-cloud deployments.Source: §10.2 · p. 11

10.3 Future Work

• Five minutes between injected failures is insufficient for measuring recovery time in every scenario, motivating longer failure intervals in future work.Source: §10.3 · p. 12
• Future extensions should cover manager failures, larger state sizes, Kafka resource and cost effects, more exactly-once scenarios, output correctness, and execution costs.Source: §10.3 · p. 12