Source-linked AI summary
Beyond Batch Processing: Towards Real-Time and Streaming Big Data
Saeed Shahrivari, Saeed Jalili
TL;DR
Hadoop’s batch design does not suit demands for interactive jobs, real-time queries, and streams. The article surveys real-time and stream-processing approaches and reports that newer query solutions generally outperform Hive in its benchmark, with Hive matching Impala on ETL-like queries.
Problem
Hadoop is designed for high-throughput batch jobs, while interactive jobs, real-time queries, and streaming data are increasingly needed.
Source: §1 · p. 2, §6 · p. 10Method
The article surveys in-memory computing, optimized real-time query systems, and stream-processing approaches, and compares selected solutions experimentally.
Source: §1 · p. 2Results
New-generation solutions generally perform better than Hive, while Hive matches Impala on the ETL-like query; the article identifies in-memory computing, real-time query systems, Storm, and S4 as approaches for non-batch demands.
Source: §5 · p. 9, §6 · p. 10Takeaways & Limitations
The article presents Spark as an example combining in-memory computing, interactive querying through Shark, and stream processing through micro-batching.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Today, big data is generated from many sources and there is a huge demand for storing, managing, processing, and querying on big data. The MapReduce model and its counterpart open source implementation Hadoop, has proven itself as the de facto solution to big data processing. Hadoop is inherently designed for batch and high throughput processing jobs. Although Hadoop is very suitable for batch jobs but there is an increasing demand for non-batch processes on big data like: interactive jobs, real-time queries, and big data streams. Since Hadoop is not proper for these non-batch workloads, new solutions are proposed to these new challenges. In this article, we discuss two categories of these solutions: real-time processing, and stream processing for big data. For each category, we discuss paradigms, strengths and differences to Hadoop. We also introduce some practical systems and frameworks for each category. Finally, some simple experiments are done to show effectiveness of some solutions compared to available Hadoop-based solutions.
1. Introduction
MapReduce and Hadoop serve large-scale batch processing, but emerging interactive, real-time, and streaming workloads call for alternatives. The article surveys real-time and stream-processing approaches and reports experiments comparing some solutions with Hadoop-based systems.
1. Introduction
- MapReduce supports large-scale data processing with a simple programming model, scalability, and built-in fault tolerance.Its components include an execution engine, Google File System, and BigTable.Source: §1 · pp. 1–2
- Hadoop is designed for high-throughput batch jobs that may take hours or days, unlike newer jobs and queries expected to finish in seconds or minutes.Hadoop is an open-source counterpart to MapReduce.Source: §1 · p. 2
- The article focuses on real-time processing and stream processing, including interactive queries and processing data that has not been fully stored.It discusses paradigms, strengths, differences from Hadoop, practical systems, and experiments comparing solutions.Source: §1 · p. 2
2. The MapReduce framework
MapReduce expresses distributed computation through map and reduce phases, while Hadoop implements the model with additional platform features. Batch constraints and job-startup overhead limit Hadoop for iterative, online, interactive, and continuous workloads.
2. The MapReduce framework
- MapReduce transforms input key/value pairs into intermediate pairs with map, then groups values by key for reduce to produce final pairs.A distributed execution assigns tasks to workers, which write intermediate and final files [7].Source: §2 · pp. 2–3, Figure 1 · p. 4
2.1. Apache Hadoop
- Hadoop is a popular open-source MapReduce implementation that uses a master/slave architecture and stores files on HDFS by default.It also supports pluggable input and output sources, including HBase, Cassandra, and relational databases.Source: §2.1 · p. 3
- Hadoop adds schedulers, YARN job definitions, high-availability masters, and pluggable I/O to MapReduce’s scalability and fault-tolerance features.HBase, Pig, Hive, and Mahout provide storage, warehousing, and machine-learning capabilities on the platform.Source: §2.1 · p. 3
- MapReduce requires all input to be ready before a job begins, preventing online and stream use; startup overhead also impedes interactive and near-real-time queries.It also lacks inherent support for recursive or iterative jobs and continuous computations.Source: §2.1 · p. 4
3. Real-time big data processing
Real-time big-data processing approaches either reduce MapReduce overhead or provide optimized querying. The section describes in-memory computing and specialized query systems as distinct approaches.
3. Real-time big data processing
- Real-time solutions either reduce MapReduce overhead for sub-second jobs or optimize queries over structured and unstructured big data.The article discusses both categories.Source: §3 · p. 4
3.1. In-memory computing
- Hadoop’s startup procedures are not optimized for seconds-scale execution, and HDFS favors throughput over I/O performance.The passage attributes HDFS’s performance limits to large blocks on hard drives, with transfer rates of 100–200 megabytes per second.Source: §3.1 · p. 5
- In-memory computing uses distributed main memory to store and process big data, addressing the I/O limits of disk-based processing.The passage reports memory bandwidth above 10 gigabytes per second versus 200 megabytes per second for hard disks, and nanosecond versus millisecond access latency.Source: §3.1 · p. 5
- In-memory computing can use caching without keeping the whole dataset in memory; caching helps iterative jobs and can help handle large streams.Spark’s RDDs can draw on HDFS and HBase, while GridGain’s file system can operate with HDFS as a caching layer.Source: §3.1 · p. 5
- In-memory computing is a computing paradigm, not just an in-memory database or data grid, and data grids can serve as building blocks.The distinction includes scheduling and moving code to data rather than data to code.Source: §3.1 · pp. 5–6
3.2. Real-time queries over big data
- Dremel combines columnar storage for nested structures with scalable parallel aggregation for real-time complex queries.Impala applies columnar storage and parallel-DBMS techniques to ad hoc queries; Hive remains a stable option for long-running analytics.Source: §3.2 · p. 6
- Drill supports real-time queries across storage systems, while Shark uses Spark and Hive compatibility and claims up to 100x faster responses than Hive.Stinger targets a 100x improvement for future Hive versions, and Redshift targets petabyte-scale warehousing.Source: §3.2 · p. 6
4. Streaming big data
Stream workloads need to process continually arriving input and may need continuous computations, unlike batch MapReduce. The section contrasts micro-batching with stream frameworks Storm and S4 and describes differences in their programming models and trade-offs.
4. Streaming big data
- Batch MapReduce requires complete input before computation and returns output only after the job finishes, unlike streams that arrive continuously.Some applications also require continuously running queries, such as detecting anomalies in incoming events.Source: §4 · p. 6
- Micro-batching treats a stream as small batches delivered at intervals to a batch system, but is inadequate for true stream-processing demands.Some real-time MapReduce implementations, including Spark and GridGain, support this technique.Source: §4 · p. 7
- S4 expresses programs as processing elements per key, whereas Storm uses source spouts and processing bolts arranged in a topology.Both frameworks run on the JVM and process keyed streams.Source: §4 · p. 7
- S4 offers simpler programming and automated execution, while Storm offers more control but requires programmers to manage load balancing and tuning.Storm guarantees processing of tuples that successfully enter its system, whereas S4 can drop messages when a processing-element buffer fills and may lose work since its latest checkpoint after a crash.Source: §4 · p. 7
5. Experimental results
The experiments compare Spark with Hadoop on simple data-processing tasks and report Berkeley AMP Lab aggregation-query benchmark results. Spark outperforms Hadoop in WordCount and Grep, while Shark and Redshift outperform Hive across result sizes, with an ETL-like exception for Impala.
5. Experimental results
- The article tests Spark against Hadoop on 40-GB WordCount and Grep jobs, and excludes stream-processing systems because the authors say they are not comparable to Hadoop.The experiments are illustrative rather than detailed technology comparisons.Source: §5 · p. 8
- 1 second versus about 160 seconds: Spark runs Grep that fast with the input cached in RAM, while Hadoop takes about 160 seconds.Spark also outperforms Hadoop in WordCount and Grep with both on-disk and cached inputs.Source: §5 · p. 8, Figure 2 · p. 9
- The AMP Lab benchmark compares Redshift, Hive, Shark, and Impala on scans, aggregations, and joins using BI-like, intermediate, and ETL-like result sizes.The article reports only the aggregation-query results.Source: §5 · pp. 8–9
- Shark and Redshift outperform Hive for all result sizes; for ETL-like queries, Hive performs the same as Impala because the result is too large for Impala to handle properly.The reported Berkeley AMP Lab results cover aggregation queries.Source: §5 · p. 9, Figure 3 · p. 9
6. Conclusion
The conclusion argues that Hadoop suits batch, high-throughput jobs but not interactive queries, real-time queries, or streams. It presents in-memory computing, real-time query systems, and stream frameworks as alternatives, highlighting Spark’s combination of these approaches while leaving their practical popularity unresolved.
6. Conclusion
- Hadoop is mature for batch, high-throughput work but is not suitable enough for interactive jobs, real-time queries, and stream data.The conclusion says these non-batch demands have led to new solutions.Source: §6 · pp. 9–10
- In-memory computing can reduce execution time when data fits in distributed memory or jobs iterate over input; real-time query systems use custom storage and parallel-DBMS techniques for sub-second responses.Storm and S4 are identified as stream-processing frameworks with distinct programming models, strengths, and weaknesses.Source: §6 · p. 10
- Spark is presented as combining in-memory computing through RDDs, interactive querying through Shark, and stream processing through fast micro-batching.The authors leave open which approach will become popular in practice.Source: §6 · p. 10