Source-linked AI summary
Benchmarking Distributed Stream Data Processing Systems
Jeyhun Karimov, Tilmann Rabl, Asterios Katsifodimos, Roman Samarev, Henri Heiskanen, Volker Markl
TL;DR
Detailed performance analysis is needed because streaming engines have diverging capabilities and characteristics, while batch processing is insufficient when new data must be processed quickly. The paper proposes a benchmarking framework with defined metrics and realistic workloads, evaluates Apache Storm, Spark, and Flink, and finds system-specific performance patterns across workloads.
Problem
Existing comparisons provide limited analysis of streaming engines' performance characteristics, despite the need to process newly arriving data quickly.
Method
The paper develops a generic framework using latency and throughput metrics, sustainable-throughput testing, separated drivers, and industrial online-gaming workloads to evaluate Storm, Spark, and Flink.
Results
Flink has better overall throughput for aggregation and join queries, while the best system varies by workload, latency objective, data skew, arrival-rate fluctuations, and window size.
Takeaways & Limitations
The benchmark identifies distinct system characteristics and supports choosing among Storm, Spark, and Flink according to workload and performance priorities.
Abstract
from arXiv · showhide
The need for scalable and efficient stream analysis has led to the development of many open-source streaming data processing systems (SDPSs) with highly diverging capabilities and performance characteristics. While first initiatives try to compare the systems for simple workloads, there is a clear gap of detailed analyses of the systems' performance characteristics. In this paper, we propose a framework for benchmarking distributed stream processing engines. We use our suite to evaluate the performance of three widely used SDPSs in detail, namely Apache Storm, Apache Spark, and Apache Flink. Our evaluation focuses in particular on measuring the throughput and latency of windowed operations, which are the basic type of operations in stream analytics. For this benchmark, we design workloads based on real-life, industrial use-cases inspired by the online gaming industry. The contribution of our work is threefold. First, we give a definition of latency and throughput for stateful operators. Second, we carefully separate the system under test and driver, in order to correctly represent the open world model of typical stream processing deployments and can, therefore, measure system performance under realistic conditions. Third, we build the first benchmarking framework to define and test the sustainable performance of streaming systems. Our detailed evaluation highlights the individual characteristics and use-cases of each system.
I. INTRODUCTION
The paper presents a benchmarking framework for distributed stream processing systems, motivated by gaps in production-realistic measurements. It evaluates Apache Storm, Apache Spark, and Apache Flink using online gaming workloads and introduces metrics and designs for accurate, sustainable evaluation.
- Workloads: The workloads target online video-game scenarios involving windowed aggregations and joins for monitoring purchases, advertising, and user feeds.These operations support use cases such as tracking in-application purchases and comparing user feeds across distribution channels.
- Framework and Evaluation: The framework evaluates Apache Storm, Apache Spark, and Apache Flink with latency and throughput as its major performance indicators.Latency is measured from source data production to output production, while throughput counts ingested and processed records per time unit.
- Motivation: Existing comparisons did not measure latency and throughput achievable in production settings, especially for stateful operators and separated benchmark drivers.Previous measurements were often calculated within the system under test, potentially producing incorrect metrics.
- Contributions: The paper introduces accurate latency measurement for stateful operators and applies the method to varied use cases.Stateful-operator latency is a central contribution because joins and other windowed operations retain state before producing results.
- Contributions: The framework separates the test driver from the system under test and measures maximum sustainable throughput while handling system-specific backpressure.These designs aim to evaluate systems under controlled workloads without allowing unprocessable input to cause failures.
II. RELATED WORK
Prior stream-processing benchmarks used varied setups and metrics, leaving concerns about bottlenecks, inaccurate latency measurements, and inconsistent throughput definitions. This work distinguishes itself by separating the system under test and driver and by introducing sustainable throughput.
- Prior Benchmarks: Earlier benchmarks covered batch and stream processing, including broad suites, cloud experiments, and informal comparisons of Storm, Spark, and Flink.The reviewed efforts span Hadoop, Spark applications, end-to-end big-data workloads, and streaming benchmarks using external systems.
- Measurement Concerns: External components such as Kafka, Redis, or mediator systems could become bottlenecks or affect measurement accuracy.The paper argues that benchmark infrastructure must be designed so auxiliary systems do not determine the measured performance.
- Measurement Concerns: Previous work defined latency from source ingestion to sink emission, but this can hide backpressure and other effects visible in event-time latency.The paper therefore measures event-time latency rather than relying only on operator-ingestion timestamps.
- Throughput Definition: Because aggregation can produce fewer result tuples than input tuples, the paper measures ingestion throughput and introduces sustainable throughput.Sustainable throughput controls the input rate to avoid prolonged latency growth rather than simply ingesting at the maximum possible rate.
- Novelty: The framework claims to be the first to separate the SUT and driver, use a scalable data generator, define system- and event-time metrics, and use sustainable throughput throughout experiments.These design choices distinguish the framework from the related benchmarks discussed in the paper.
A. On-the-fly Data Generation vs. Message Brokers
The benchmark generates data on the fly and uses queues instead of message brokers to avoid infrastructure bottlenecks and smooth differences between generation and ingestion rates. The driver remains isolated from the system under test.
- Data Generation: Message brokers can become benchmark bottlenecks because data exchange with streaming systems may limit measured performance.The framework therefore chooses on-the-fly generation rather than reading data from a broker or filesystem.
- Rate Fluctuations: Variable ingestion rates can result from transient network issues, garbage collection, and other runtime effects during experiments.Such fluctuations complicate performance measurement when the arrival rate is not constant.
- Rate Fluctuations: Queues between data generators and source operators smooth differences between data-generation and data-ingestion rates.The queues are intended to even out transient rate fluctuations before events reach the streaming system.
- Driver Isolation: The driver is isolated from the SUT, and measurements are performed outside the SUT rather than using internal system statistics.The driver includes data generators, queues, and measurements, and driver instances run on separate machines.
- Metrics: The framework distinguishes throughput and latency, including event-time and processing-time latency and maximum and sustainable throughput.These distinctions organize the benchmark’s measurement model.
A. Latency
The paper distinguishes event-time latency from processing-time latency for windowed operators, where output timing depends on multiple contributing tuples and window completion. It assigns output times from contributing events to measure these latencies consistently.
- Latency Concepts: Event-time records when an event is captured, whereas processing-time records when an operator processes a tuple.The two notions of time motivate distinct latency measures.
- Event-Time Latency: Event-time latency is the interval between a tuple’s event-time and its emission from the SUT output operator.For windowed output events, the event-time is based on the contributing events.
- Processing-Time Latency: Processing-time latency is the interval between a tuple’s ingestion time at the input operator and its emission from the SUT output operator.Each tuple receives a processing-time field when it reaches the first operator.
- Latency Interpretation: Event-time latency includes queue waiting, while processing-time latency measures time spent processing within the streaming system.Event-time latency reflects the interval from user action or source production to completed output processing.
- Windowed Operators: Windowed operators make latency measurement non-trivial because tuples wait until the window is formed completely before results are returned.This waiting time affects the latency of stateful aggregates and other windowed operations.
- Windowed Operators: For a windowed output, event-time and processing-time are respectively the maximum event-time and processing-time of all contributing events.The output latency is then computed using the corresponding assigned time and emission time.
- Join Example: In the join example, ads and purchases are joined in a 10-minute window, and result tuples receive the maximum event-time of tuples in their corresponding windows.This same assignment supports event-time latency calculation for join outputs.
B. Throughput
The paper distinguishes ordinary throughput from sustainable throughput by accounting for event-time latency and prolonged backpressure. Sustainable throughput is the highest traffic load a deployment can handle without continuously increasing event-time latency.
- Throughput measures the number of events processed per unit time, but it may not correlate with event-time latency.Batching can increase throughput while the batching delay increases event-time latency.
- Sustainable throughput reflects whether a system can handle rising data-arrival rates without exhibiting prolonged backpressure.Deployments may need to adapt, such as by scaling out, as arrival rates increase.
- Sustainable throughput is the highest event-traffic load a system handles without continuously increasing event-time latency.
- The benchmark finds sustainable throughput by starting with a very high generation rate and decreasing it until the system sustains the rate without a continuously growing queue.Some queue fluctuation is allowed as long as the queue does not continuously increase.
V. WORKLOAD DESIGN
The workload design models online video-game advertising and purchasing, using windowed aggregation and join queries over synthetic event streams. These queries represent revenue tracking and correlating advertisements with resulting purchases.
- The benchmark workload is derived from Rovio’s online video-game application and its monitoring of user actions and service behavior.Examples include monitoring active users and checking newly added game features.
- The dataset contains purchase events and advertisement proposals for gem packs, generated synthetically.Purchases include user, gem-pack, price, and time fields; advertisements include user, gem-pack, and time fields.
- The windowed aggregation query computes total purchase revenue grouped by gem-pack identifier.It uses a sliding window over the purchases stream.
- The windowed join correlates advertisement proposals with purchases using matching user and gem-pack identifiers.The query identifies proposed gem packs that users subsequently bought.
VI. EVALUATION
The evaluation compares three streaming systems using selected experimental results because the study includes many parameters and variables. The evaluated versions are Storm 1.0.2, Spark 2.0.1, and Flink 1.1.3.
- The evaluation measures the performance of Storm 1.0.2, Spark 2.0.1, and Flink 1.1.3.
- The paper presents the most interesting results rather than all experimental results because the evaluation has many parameters and variables.
A. System setup
The system setup uses a multi-node cluster, synchronized clocks, parallel data generators, warmup, and backpressure controls. The experiments tune system parameters while treating dropped connections as failures and acknowledging Storm’s backpressure limitation.
- A. System setup: The cluster has 20 nodes with 16GB RAM, 1Gb/s networking, and 2, 4, or 8 workers and driver nodes.Each node uses a 2.40GHz Intel Xeon E5620 processor, and a dedicated master serves the streaming systems.
- A. System setup: Data generators produce 100M events at constant speed using 16 parallel instances, with synchronized cluster clocks and normally distributed keys.
- A. System setup: The experiments use a 25% warmup and enable backpressure so systems cannot ingest more input than they can process.A dropped connection from the SUT to the data-generator queue halts the experiment and indicates unsustainable throughput.
- A. System setup: The paper reports selected figures or features when behavior is similar across parameters because of space limitations.
- A. System setup: Configuration tuning balances throughput and latency, while parallelism must balance resource utilization against network or resource exhaustion.Relevant settings include buffer size, RDD partitions, batch or block intervals, workers, and executors.
- A. System setup: Storm’s backpressure feature is described as immature and capable of stalling topologies under high workloads.With backpressure disabled, Storm also drops data-queue connections at high workloads, which the paper treats as a system failure.
B. Performance Evaluation
The evaluation shows that performance depends strongly on workload, window configuration, cluster size, and engine architecture. Storm, Spark, and Flink exhibit distinct throughput, latency, scalability, and skew-handling characteristics across windowed operations.
- Windowed Aggregations: Storm and Spark have comparable aggregation throughput, with Storm outperforming Spark by approximately 8% across configurations.Flink’s performance is bounded by network bandwidth with four or more nodes.
- Windowed Aggregations: Spark has higher aggregation latency than Storm and Flink but less variation, partly because mini-batches give tuples within a batch similar latencies.Spark’s block-manager-to-DStream transfers and RDD creation add overhead, increasing average latency.
- Windowed Aggregations: Storm’s latency increases with workload and cluster size, whereas Spark’s decreases in larger distributed environments; Storm’s maximum latencies can be outliers.The results are attributed to Spark partitioning RDDs more effectively as the environment grows.
- Windowed Joins: Windowed joins cause Spark fluctuations and increase Flink latency relative to aggregations, although 90% workload significantly reduces spikes.Flink outperforms Spark on all reported windowed-join latency parameters, while Spark’s blocking operators and cross-RDD scheduling add overhead.
- Large Windows: Larger windows can sharply degrade Spark: a 60-second window and slide halves throughput and increases average latency tenfold at a 4-second batch size.Storm requires spill-capable user data structures for large windows; otherwise, the experiments encountered memory exceptions.
- Skewed Data: For skewed aggregation data, Spark reaches 0.53 M/s sustainable throughput in a 4-node cluster and outperforms both engines in clusters with at least four nodes.For skewed joins, Flink often becomes unresponsive and Spark exhibits very high latencies as memory is consumed quickly and backpressure performs inefficiently.
- System Architectures: Spark performs worse with three or fewer nodes because of blocking operators, while Flink avoids this limitation through non-blocking operators and Storm has a similar architecture.The evaluation links architectural choices to differing performance across cluster sizes and operation types.
C. Discussion
The discussion presents system-selection guidelines: Spark suits skewed inputs, while Flink is favored for average latency, rate fluctuations on joins, and overall throughput.
- Spark is the best choice for streams containing skewed data.
- Flink behaves better than Spark under fluctuations in data arrival rate for join queries.
- Flink is the best choice when average latency is the priority and provides better overall throughput for aggregation and join queries.
- The paper defines event-time and processing-time latency and shows a significant difference between them.
D. Future work
Future work extends the framework toward concurrent query workloads and broader system compatibility through a generic interface.
- The authors plan to develop TPC-style benchmarks with concurrently executed query workloads and a small operator set.
- They are developing a generic interface for systems including Apache Samza, Heron, and Apache Apex.
VII. CONCLUSIONS
The paper introduces a benchmark for streaming engines using online video-game scenarios, separating measurement from the systems under test and evaluating sustainable workloads. Experiments compare Storm, Spark, and Flink, revealing system-specific advantages and challenges and yielding use-case selection guidelines.
- The framework benchmarks streaming engines with online video-game scenarios motivated by industry’s need for real-time data processing.
- It defines stateful-operator latency, separates the driver from the systems under test, and measures the highest sustainable workloads.
- Extensive experiments with Apache Storm, Apache Spark, and Apache Flink reveal specific advantages and challenges for each system.
- The discussion provides rules for determining system requirements for a use-case.