Source-linked AI summary
RIoTBench: A Real-time IoT Benchmark for Distributed Stream Processing Platforms
Anshu Shukla, Shilpa Chaturvedi, Yogesh Simmhan
TL;DR
RIoTBench addresses the lack of rigorous, realistic benchmarks for evaluating distributed stream processing systems on IoT applications and data streams. It combines reusable IoT tasks, reference applications, real-world workloads, and performance metrics, then validates the suite on Apache Storm with empirical observations.
Problem
Contemporary DSPS performance has not been rigorously studied for IoT applications and data streams because no well-defined benchmark captures their domain features, compositional capabilities, and real data streams.
Method
RIoTBench combines representative micro-benchmark tasks, four reference IoT applications, four real-world IoT data streams, and throughput-based performance metrics for DSPS evaluation.
Results
The suite was validated on Apache Storm and showed substantial variation across micro-benchmark task throughput, with most tasks supporting at least 3,000 messages/sec and some reaching 68,000 messages/sec.
Takeaways & Limitations
RIoTBench provides reusable micro and application benchmarks for evaluating DSPS efficacy on common IoT processing tasks and dataflows using real-world streams.
Takeaways & Limitations
The study does not emphasize durability or fault-tolerance metrics, although those metrics can be added.
Abstract
from arXiv · showhide
The Internet of Things (IoT) is an emerging technology paradigm where millions of sensors and actuators help monitor and manage, physical, environmental and human systems in real-time. The inherent closedloop responsiveness and decision making of IoT applications make them ideal candidates for using low latency and scalable stream processing platforms. Distributed Stream Processing Systems (DSPS) hosted on Cloud data-centers are becoming the vital engine for real-time data processing and analytics in any IoT software architecture. But the efficacy and performance of contemporary DSPS have not been rigorously studied for IoT applications and data streams. Here, we develop RIoTBench, a Realtime IoT Benchmark suite, along with performance metrics, to evaluate DSPS for streaming IoT applications. The benchmark includes 27 common IoT tasks classified across various functional categories and implemented as reusable micro-benchmarks. Further, we propose four IoT application benchmarks composed from these tasks, and that leverage various dataflow semantics of DSPS. The applications are based on common IoT patterns for data pre-processing, statistical summarization and predictive analytics. These are coupled with four stream workloads sourced from real IoT observations on smart cities and fitness, with peak streams rates that range from 500 to 10000 messages/sec and diverse frequency distributions. We validate the RIoTBench suite for the popular Apache Storm DSPS on the Microsoft Azure public Cloud, and present empirical observations. This suite can be used by DSPS researchers for performance analysis and resource scheduling, and by IoT practitioners to evaluate DSPS platforms.
1 Introduction
IoT applications require low-latency, scalable processing of massive real-time sensor streams, but DSPS performance for these workloads lacked rigorous, domain-specific evaluation. RIoTBench addresses this gap with reusable tasks, applications, workloads, and performance metrics validated on Apache Storm.
- Millions of sensors generate real-time observations that require scalable cloud processing and rapid feedback for IoT decision-making.The workload spans distributed sensors and thousands of events per second.
- DSPS provide distributed dataflow engines for continuous IoT analytics, but their efficacy on IoT applications and streams had not been rigorously studied.Existing systems include Apache Storm, Spark Streaming, and Flink.
- RIoTBench classifies streaming applications, composition semantics, data sources, essential IoT tasks, and DSPS performance metrics.The classifications target latency and scalability requirements.
- The suite contains reusable micro-benchmark tasks, four reference applications, and four real-world IoT streams with diverse distributions.Applications cover data preprocessing, statistical analytics, and predictive analytics.
- RIoTBench was validated with empirical results on Apache Storm, while supporting evaluation by IoT practitioners and reproducible comparison by DSPS researchers.The benchmark is intended for performance, scalability, resource-management, and resiliency studies.
2 Background and Related Work
Prior benchmarks cover stream processing, Big Data, IoT data generation, RDF streams, or embedded hardware, but they often target different platforms, workloads, or evaluation goals. RIoTBench focuses specifically on realistic IoT workloads and combines micro- and application-level DSPS benchmarks.
- DSPS differ in programming and execution semantics, including micro-batch processing in Spark Streaming versus per-tuple processing in Storm.These differences create latency-throughput trade-offs that motivate domain-specific evaluation.
- Earlier DSPS benchmarks often used limited metrics and generic workloads, while StreamBench covered synthetic web and network streams with seven micro-benchmarks.StreamBench included performance, durability, and fault-tolerance metrics and compared Storm with Spark Streaming.
- Other benchmarks target Apache Spark, CEP systems, synthetic IoT time series, RDF streams, or embedded-system connectivity rather than platform-agnostic IoT DSPS evaluation.Their metrics and composition models reflect their respective platforms and domains.
- StreamBench did not cover larger applications, several dataflow patterns, selectivity ratios, or a broad range of realistic input rates.The paper also notes that RIoTBench does not emphasize durability or fault tolerance, though those metrics can be added.
- RIoTBench combines realistic IoT application logic, data-stream workloads, and micro- and application-level benchmarks for reproducible DSPS comparison.It is designed to be simple to implement across diverse DSPS.
3 Characteristics of DSPS Applications and Streams
DSPS applications are modeled as task-and-stream dataflow graphs whose execution depends on message composition, routing, parallelism, and stream characteristics. Selectivity, throughput, rate variation, and message size determine workload behavior and downstream processing demands.
- Dataflow Composition Semantics: A DSPS application is a directed dataflow graph whose vertices are user-defined tasks and edges carry continuous messages.Graphs may be cyclic, and tasks execute when adequate inputs are available.
- Dataflow Composition Semantics: Selectivity σ measures output messages per input message and can amplify or attenuate rates, multiplying load on downstream tasks.The paper expresses it as σ = input rate : output rate.
- Dataflow Composition Semantics: Source, sink, transform, filter, flatmap, and aggregate tasks implement distinct message-generation and consumption patterns.Their selectivities include 1:1, N:M, 1:N, and N:1 forms.
- Dataflow Composition Semantics: Routing semantics determine whether messages are duplicated, sent to one downstream task by round robin or hashing, or merged across incoming streams.These choices shape how task instances receive and distribute work.
- Application Dimensions: Data parallelism allocates concurrent resources to task instances, while application size reflects the number of tasks in the graph.Parallel execution is typically suitable for tasks without cross-message state.
- Input Stream Characteristics: Input throughput, its time-varying distribution, and message size influence processing load, adaptation requirements, and communication cost.Real streams may exhibit bursts, saw-tooth patterns, or diurnal variation.
4 Characteristics of IoT Applications and Streams
RIoTBench organizes IoT processing around reusable task categories, application dataflows, and stream characteristics. These streams vary in sampling rates, sensor populations, and temporal event patterns, requiring benchmarks that reflect real-time domain diversity.
- Benchmark Scope: The benchmark categorizes tasks, applications, and streams to define a search space for meaningful validation of IoT dataflows on DSPS.The scope is intentionally focused on common real-time processing and analytics tasks rather than every possible IoT scenario.
- Task Categories: IoT task categories include parsing, filtering, statistical analytics, predictive analytics, pattern detection, and visual analytics.Tasks operate on incoming messages to transform, clean, aggregate, predict, detect, or present stream information.
- Application Dataflows: IoT dataflows span observation, orientation, decision, and action activities, including preprocessing, summarization, prediction, classification, notification, and external service access.These activities loosely correspond to the OODA cycle and may use storage, databases, or publish-subscribe systems.
- Stream Characteristics: IoT streams are typically time-series data whose sampling rates range from once daily to hundreds per second, while sensor populations range from hundreds to millions.These characteristics create substantially different workload scales across application domains.
- Stream Characteristics: Event rates may vary over time because sensors transmit on changes, batch delayed observations, or follow diurnal and bimodal human-activity patterns.Examples include smart-meter reporting bursts and fitness-device synchronization at intervals unlike the underlying sampling rate.
5 Performance Metrics
The paper defines DSPS metrics covering resource use, latency, throughput, and jitter. These measurements are interpreted under stable operating conditions and account for both task-level processing and end-to-end dataflow behavior.
- Latency: Latency measures the time required to process inputs into outputs, while application latency includes the time from source consumption to causally dependent sink generation.End-to-end latency also includes network and queuing time, and varies with input rate, resources, and message type.
- Throughput: Throughput is the aggregated sink output rate in messages per second; ideally, output throughput ω_o = σ × ω_i for selectivity σ and input throughput ω_i.The paper also considers the peak throughput a dataflow can sustain.
- Measurement Conditions: Throughput and latency are meaningful only when the DSPS sustains the input rate without growing latency or input-buffer queues.The stability condition excludes unsustainable backlog from these measurements.
- Jitter: Jitter measures variation between expected and actual output throughput, normalized by the expected long-term average output rate.For an ideal DSPS, jitter tends toward zero even when input rates change instantaneously.
- Resource Metrics: CPU and memory utilization are averaged across the virtual machines used by a dataflow to quantify distributed resource use and overhead.These measures also relate to the VM resources and price required to run the application.
6 RIoTBench IoT Benchmark Suite
RIoTBench combines reusable IoT micro-benchmarks, composed application dataflows, and real-world stream workloads to evaluate DSPS behavior. Its design covers common IoT processing logic and workload diversity, with scaled peak rates from 500 to 10,000 messages per second.
- Suite Design: The benchmark workloads contain DSPS-executed dataflow logic and input streams for evaluating the defined performance metrics on emerging IoT applications.The workload design targets comparisons of contemporary DSPS on Cloud infrastructure.
- Micro-benchmarks: Micro-benchmarks cover common IoT tasks and patterns, measuring peak input throughput sustainable on a unit computing resource.The resulting baseline supports comparisons across DSPS and resource scheduling for larger composed dataflows.
- Micro-benchmarks: The task suite includes parsing, statistical analytics, predictive machine learning, storage, messaging, compression, and visualization operations.Tasks are implemented as source-task, benchmark-task, and sink-task dataflows capturing patterns such as transform, filter, aggregate, and flat map.
- Application Benchmarks: Four application dataflows combine reusable tasks for ETL, statistical analytics, model training, and predictive analytics.ETL cleans and archives streams; STATS computes parallel summaries; TRAIN periodically builds models; PRED applies models to incoming data and communicates decisions.
7 Evaluation of Proposed Benchmarks
The evaluation implements RIoTBench dataflows as Apache Storm topologies on Azure and measures throughput, latency, jitter, and resource use across micro-benchmarks and application workloads. Results show substantial task-level performance variation, stable processing at peak rates, and workload-dependent resource requirements.
- Experimental Setup: RIoTBench’s 27 Java micro-benchmarks were composed into ETL, STATS, TRAIN, and PRED dataflows and evaluated on Apache Storm.Tasks consume and produce objects, while Storm bolts, topologies, and a replaying source task implement the benchmark dataflows.
- Experimental Setup: Storm ran on Microsoft Azure VMs, with separate configurations for micro-benchmarks and application dataflows.Micro-benchmarks used an exclusive D1 task VM plus a D4 support VM; application benchmarks used D3 task VMs and additional D4 VMs.
- Micro-benchmark Results: 68,000 msg/sec was the peak throughput reached by several tasks, whereas XML parsing supported 310 msg/sec and plotting supported 25 msg/sec.DTT and MLT supported 50 and 70 msg/sec, respectively, because model training was CPU-bound.
- Micro-benchmark Results: 1 msg/min was supported by ATR, while RZP and LZP supported 300 msg/sec and 3,000 msg/sec, respectively.ATR scanned a full Azure table; RZP wrote to a remote shared directory, whereas LZP used a local disk.
- Micro-benchmark Results: 2,600 ms was the maximum observed end-to-end latency for Q3, despite mean latencies expected to be sub-millisecond at observed throughputs.Latency variability was associated with queuing, nonuniform task execution, library costs, and framework overheads for high-rate tasks.
- Micro-benchmark Results: Jitter remained close to zero across tasks, indicating sustainable peak-rate processing, while CPU utilization was at least 70% for most CPU-bound tasks.SML, MQS, and Azure tasks were exceptions because they were I/O-bound; memory use was higher for high-throughput or buffering-intensive tasks.
- Application Results: Application latency depended on critical-path tasks, with ETL latency near 30 millisec for CITY, FIT, and TAXI and higher variation for GRID.TRAIN had higher latency because it encoded batch processing as a streaming dataflow and scanned Azure tables.
- Application Results: Application resource needs varied by workload, while the generally liberal allocation strategy left resources under-utilized.STATS used more CPU and memory because of AVG, DAC, and GroupViZ, whereas FIT had the lowest CPU usage at 500 msg/sec.
8 Conclusion
The paper concludes that RIoTBench provides a broad benchmark suite for evaluating DSPS on representative IoT tasks, applications, and real-world streams. It validates the suite with Apache Storm and identifies extensions for future coverage and cross-platform evaluation.
- Conclusion: RIoTBench evaluates DSPS using micro-benchmarks and complete dataflows for pre-processing, statistical summarization, and predictive analytics.The applications align with the OODA interaction model and use four spatially and temporally scaled IoT streams.
- Conclusion: The benchmark combines four real-world streams from Smart Grid, Smart Transportation, Urban Sensing, and personal fitness domains.The streams are scaled to recreate contemporary IoT stream profiles.
- Conclusion: The suite was validated on Apache Storm, with performance metrics reported for the benchmark workloads.
- Future Work: Future work includes adding event pattern detection and notification tasks and evaluating other DSPS such as Apache Spark Streaming and Flink.
- Future Work: The benchmark applications use real and accurate business logic and are being integrated into a smart-campus project for water and power management.
A Configurations used in Application Dataflows
The application configurations specify which dataset attributes support each task, while footnotes document dataset-specific exclusions and interpretation constraints. These constraints reflect task requirements and data characteristics rather than a single uniform configuration.
- ETL: ETL task configurations are documented in Table 4, with annotations describing dataset-specific attribute handling.
- STATS: STATS task configurations are documented in Table 5, with dataset-specific attribute selections and annotations.
- ETL: DAC is not configured for individual Taxi trips because the operation is not meaningful at that granularity.
- PRED: PRED task configurations are documented in Table 6, including prediction classes used by DTC.
- PRED: GRID is univariate, so DTC and MLR tasks requiring multiple fields have no configured fields for that dataset.
- TRAIN: TRAIN task configurations are documented in Table 7, with GRID exclusions for DTT and MLT because those tasks require multiple fields.