An Experimental Survey on Big Data Frameworks

• Existing storage, processing, and analysis tools are inadequate for massive heterogeneous data, raising challenges of scalability, fault tolerance, and efficiency.Source: §1 · p. 1
• Big Data is characterized by four Vs—Volume, Variety, Veracity, and Velocity —covering dataset size, data types, quality issues, and incoming-flow speed.Source: §1 · p. 1
• The survey categorizes popular frameworks by features including programming model and support for iterative processing of streaming data.Source: §1 · pp. 1–2
• The experimental study compares frameworks using representative batch, stream, and iterative workloads.Source: §1 · pp. 1–2
• Although prior surveys exist, this study compares popular frameworks across both batch and stream processing.Source: §1 · p. 1
• The paper presents best practices for using popular Big Data frameworks in several application domains.Source: §1 · p. 2

• Prior experiments assessed MapReduce implementations for performance and scalability and studied configuration effects on Spark, Hadoop, and Flink runtime.Source: §2 · p. 2
• 2.5x, 5x, and 5x faster than MapReduce on WordCount, k-means, and PageRank, respectively, Spark was reported in.Source: §2 · p. 2
• Other surveys discussed Big Data fundamentals and framework features, while an in-memory review covered RDD persistence, immutability, and data placement.Source: §2 · p. 2
• Spark and Flink studies considered scalability, disk-size effects, and functionality, including machine-learning and batch-processing settings.Source: §2 · p. 2
• A Storm–Flink study examined how stream-system design interacts with modern processors on scale-up architectures, rather than comparing individual-system performance.Source: §2 · p. 2
• Many prior surveys were limited in the frameworks or features evaluated, while this study compares batch and stream processing and includes representative batch and iterative workloads.Source: §2 · p. 2

3.1.1 Hadoop system overview

• Hadoop combines HDFS distributed storage with MapReduce, which processes large datasets through user-defined Map and Reduce functions.HDFS uses a Name Node and Data Nodes; the Hadoop WordCount job maps text chunks and reduces word counts.Source: §3.1.1 · p. 3, §3.1.2 · p. 4

3.2.1 Spark system overview

• Spark represents cluster data as immutable RDDs, applies transformations and actions, and models each job as a graph of operations.For WordCount, Spark reads input into an RDD, generates word-count pairs, aggregates them with reduceByKey, and writes the results.Source: §3.2.1 · p. 4, §3.2.2 · p. 5

3.3.1 Storm system overview

• Storm targets real-time processing with fault tolerance; its topologies are DAGs of source spouts and processing bolts distributed across nodes.The WordCount stream passes text lines through stages that extract words and count their occurrences.Source: §3.3.1 · p. 5, §3.3.2 · p. 5

3.4.1 Samza system overview

• Samza uses Kafka for messaging and YARN for distributed resource management, with its core providing APIs to create and run stream tasks.Its WordCount example sends messages through a splitter task and then a counter task to produce word frequencies.Source: §3.4.1 · p. 6, §3.4.2 · p. 6

3.5.1 Flink system overview

• Flink supports batch and real-time stream processing, iterative computation, and higher-level operations such as joins, filters, and aggregations.Its compiler converts programs into DAGs that an optimizer can reorder; WordCount uses map, groupBy, and sum to count words.Source: §3.5.1 · p. 6, §3.5.2 · p. 7

3.6 Categorization of Big Data Frameworks

• Hadoop, Flink, and Storm use key-value data, Spark combines RDD and key-value formats, and Hadoop alone is batch-only while the others support streaming.All studied frameworks run in clusters with framework-specific managers and APIs for languages including Java, Scala, and Python.Source: §3.6 · p. 7, Table 1 · p. 8
• Spark and Flink provide machine-learning libraries, while Hadoop and Samza can use external tools; Spark emphasizes in-memory iterative processing, and Flink supports complex graph workloads.Hadoop is widely used for large clusters and text processing but performs poorly for iterative jobs without in-memory caching; Spark’s RDD caching supports iterative workloads, and Spark and Flink offer machine-learning and graph-analysis tools.Source: §3.6 · p. 7

3.7.1 Healthcare applications

• For healthcare sensor streams requiring iterative processing, the original MapReduce model is unsupported, while Spark’s RDD caching supports repeated analysis of large in-memory datasets.The cited discussion notes that MapReduce extensions such as Haloop and Twister require reloading and reprocessing data across iterations, potentially wasting network and processor resources.Source: §3.7.1 · pp. 7–8

3.7.2 Recommendation systems

• Emerging recommenders must adapt to high-volume item data and streaming ratings; Hadoop cannot handle these fast streams, while Mahout, Flinkml, and Sparkmllib offer collaborative filtering.Changing news items and user interests require continuous adjustment of recommendations.Source: §3.7.2 · p. 8

3.7.3 Social media

• Social-media analytics needs real-time processing, iterative learning, and methods such as text analysis and community detection; Storm and Flink provide in-stream frameworks.Source: §3.7.3 · p. 8

3.7.4 Smart cities

• Smart-city workloads span slow planning and traffic control suited to Hadoop batch, micro-batch public services, and millisecond healthcare decisions suited to Storm; combining frameworks may support cross-domain services.Sensors and connected devices generate data used across transport, healthcare, agriculture, and other urban services.Source: §3.7.4 · pp. 8–9

4. EXPERIMENTS

• Five frameworks—Spark, Hadoop, Flink, Samza, and Storm—were evaluated for scalability, configuration effects, and resource usage.The workloads include batch, stream, and iterative processing.Source: §4 · p. 9

4.1 Experimental environment

• The tests used a 10-machine Ubuntu 16.04 cluster with 4 CPUs, 8 GB memory, and 500 GB local storage per machine.All frameworks ran under YARN; framework versions were Hadoop 2.9.0, Flink 1.3.2, Spark 1.6.0, Samza 0.10.3, and Storm 1.1.1.Source: §4.1 · p. 9

4.2 Experimental protocol

• Batch tests compared Hadoop, Spark, and Flink on WordCount, K-means, and PageRank using real and synthetic datasets.WordCount used tweets collected with Flume; K-means used synthetic datasets, and PageRank used seven real graph datasets.Source: §4.2 · p. 9
• Stream tests evaluated Storm, Flink, Samza, and Spark by sending tweets from HDFS through Kafka into an ETL workload.The pipeline stored processed messages in Elasticsearch, with Kibana available for visualization.Source: §4.2 · pp. 9–10
• A custom monitoring tool collected CPU, RAM, disk I/O, and bandwidth histories, then stored them in Elasticsearch for Kibana visualization.Monitoring workloads used 50 GB for WordCount, 10 million examples for K-means, and the G5 dataset for PageRank.Source: §4.2 · p. 10

4.3.1 Batch mode

• The batch-mode evaluation measures scalability, CPU, RAM, disk I/O, and bandwidth, and varies data size and other performance settings.WordCount is run on small and big datasets.Source: §4.3.1 · p. 10, Scalability · p. 10

Scalability

• Spark is fastest on small datasets and retains its lead on big datasets; Flink beats Hadoop only for very small datasets.The experiments cover 250 MB–2 GB small datasets and 1–100 GB big datasets.Source: Scalability · p. 10, Figure 15 · p. 11, Figure 16 · p. 11
• As clusters grow, Spark remains faster than Hadoop and Flink, while the Spark–Hadoop gap narrows at larger cluster sizes.Spark takes approximately 450–300 seconds on 2–6 nodes and about 290 seconds as node counts increase further.Source: Scalability · pp. 10–11, Figure 17 · p. 12
• Flink’s runtime slope is unstable because network traffic affects intermediate results transferred between Map and Reduce functions.Source: Scalability · p. 10
• Spark stops improving from 6 to 9 nodes when intermediate-result processing requires more RAM; Hadoop reaches 400 seconds at 9 nodes versus 600 seconds at 4.Hadoop stores intermediate results on disk.Source: Scalability · p. 11

Iterative processing

• For K-means, Spark and Flink have similar response times and are faster than Hadoop, which writes output to disk after each iteration.Source: Iterative processing · p. 12
• For PageRank, Spark’s memory consumption degrades performance relative to Flink and Hadoop when it creates new RDDs and replaces those that no longer fit in memory.Source: Iterative processing · p. 12
• On 10 million K-means examples, Hadoop’s runtime curve rises exponentially with iteration count, while Spark’s and Flink’s curves are linear and show no significant iteration-count effect.Source: Iterative processing · p. 12, Figure 19 · p. 12

Data partitioning

• Hadoop and Spark runtime curves rise with HDFS block size, while Flink’s runtime does not vary; below 16 MB, more input splits than cluster cores reduce processing time.Source: Data partitioning · p. 12
• Hadoop’s mapper count follows input splits, Spark maps HDFS input splits to RDD partitions, and Flink’s streaming input mechanism makes it independent of HDFS block size.Source: Data partitioning · pp. 12–13

Impact of the cluster manager

• Standalone mode is faster than YARN and Mesos in the tested WordCount, K-means, and PageRank workloads.The authors attribute this to standalone using all resources for a job, whereas YARN and Mesos schedulers share cluster resources among applications [32].Source: Impact of the cluster manager · p. 13

Impact of bandwidth

• Flink’s response time decreases as bandwidth rises because its jobs send data across the network from sources to processing units.The experiment uses WordCount with 50 GB of data and varies bandwidth from 128 MB to 1 GB.Source: Impact of bandwidth · p. 13

Impact of some configuration parameters

• The study varies RAM and thread-count settings because framework configuration parameters can affect both performance and result quality.Hadoop configures parallel Map and Reduce tasks with tasktracker parameters; Spark uses executor-cores, and Flink uses slots.Source: Impact of some configuration parameters · p. 13
• For Spark, adding workers per machine increases processing time, while increasing executor memory reduces response time, subject to other resource and network constraints.The authors advise limiting workers on memory-constrained machines and scaling their number with available memory.Source: Impact of some configuration parameters · p. 13
• For Flink, adding slots reduces WordCount response time, but the same effect is not observed for K-means and PageRank; memory size has little effect on those workloads.The authors relate WordCount's response-time reduction to its high CPU consumption.Source: Impact of some configuration parameters · p. 13
• Hadoop performs best with two Map and two Reduce slots on the tested four-core machines, and processing time falls as io.sort.mb increases to 100 MB.The remaining cores are reserved for DataNode and NodeManager daemons; performance stabilizes once computing units have sufficient buffer memory.Source: Impact of some configuration parameters · p. 13

Resources consumption

• In batch WordCount, Flink maximizes CPU utilization relative to Spark and Hadoop, while its execution time is higher than theirs.The authors attribute Flink's CPU use to its three-stage pipeline, which keeps slots listening to data streams.Source: Resources consumption · p. 14, Figure 29 · p. 16
• In batch mode, Hadoop writes about 90 MB/s to disk, compared with about 30 MB/s for Spark and about 50 MB/s for Flink.The authors characterize Spark as memory-oriented and say Flink first loads processing data into memory before distributing it to workers.Source: Resources consumption · p. 15, Figure 31 · p. 16
• Batch bandwidth use is 37 MB/s for Hadoop, 120 MB/s for Flink, and 70 MB/s for Spark.The authors attribute Flink's high bandwidth use to streaming data between sources and processing nodes.Source: Resources consumption · p. 15, Figure 32 · p. 16

4.3.2 Stream mode

• For 100 KB stream messages, Flink, Samza, and Storm process events at higher rates than Spark; with roughly 500 KB messages, Samza and Flink are especially efficient relative to Spark.The authors relate the first comparison to window times, which are milliseconds for Flink, Samza, and Storm versus seconds for Spark.Source: Figure 33 · p. 16, §4.3.2 · p. 16, Figure 34 · p. 17
• In the 100 KB stream scenario, Flink uses about 10% of available CPU to process 8,320 events, while Storm uses 15%–18% to process 10,085 events.The authors note that Flink may provide better results than Storm when CPU resources are more exploited.Source: §4.3.2 · pp. 16–17
• For 100 KB stream processing, Spark reaches 6 GB RAM use, while Flink, Samza, and Storm each stay at or below 5 GB.The cited passage attributes Spark's use to in-memory micro-batch processing and contrasts it with the other frameworks' single-message processing.Source: §4.3.2 · p. 17
• Stream bandwidth is reported as 375–385 KB/s for Flink, 387–390 KB/s for Storm, about 400 Mb/s for Samza, and no more than 220 KB/s for Spark.The authors relate Spark's lower bandwidth use to processing groups of messages per operation, reducing serialization and migration frequency.Source: §4.3.2 · p. 17

4.4 Summary of the evaluation

• Spark handles large datasets better than Hadoop and Flink, but intensive memory processing can degrade its performance as SparkContext searches for and removes unused RDDs.The paper says this workload creates RDD objects at each transformation operation.Source: §4.4 · p. 17
• Hadoop performs well overall, but writing intermediate results to disk limits it and can require considerable processing time as data grows, especially for iterative applications.Source: §4.4 · p. 17
• In batch mode, Flink's pipeline minimizes idle-resource periods and maximizes CPU use relative to Spark and Hadoop, while demanding more network resources than Hadoop.The paper relates this network use to Flink's faster performance than Hadoop.Source: §4.4 · p. 17
• Flink, Samza, and Storm are similar in stream processing; the paper describes Flink as low-latency, while Spark processes in micro-batches and cluster managers can automate resource allocation.In standalone mode, users specify Spark and Flink resource allocation when submitting jobs; with Mesos or YARN, allocation is automatic.Source: §4.4 · p. 17

5. BEST PRACTICES

• Framework selection is challenging given the range of frameworks and their similar, varied services; the section discusses strengths for stream, batch, machine-learning, and graph applications.Source: §5 · pp. 17–18

5.1 Stream processing

• Storm is presented as a strong choice for high-volume streams, and the experiments show good performance and resource savings even as event volume grows.The passage describes the target stream scale as billions of events per second per core.Source: §5.1 · p. 18

5.2 Micro-batch processing

• Spark may suit periodic batch tasks, while Flink and Spark are candidates for applications that combine batch and stream processing in a lambda architecture.Spark micro-batch processing groups datasets into larger time windows before analysis.Source: §5.2 · p. 18

5.3 Machine learning algorithms

• For machine learning, Spark MLlib is described as suitable for general needs and most areas; FlinkML and Hadoop's Mahout are also identified as framework libraries.FlinkML includes k-Means, logistic regression, and ALS implementations.Source: §5.3 · p. 18

5.4 Big graph processing

• Hadoop is not an optimal graph-processing model because its coarse-grained tasks are heavyweight for graph and iterative algorithms and it cannot cache intermediate data in memory.The paper also identifies graph libraries for Spark and Flink, including GraphX and Flink Gelly.Source: §5.4 · pp. 18–19

• The study compares Hadoop, Spark, Storm, and Flink on a cluster of machines.Source: §6 · p. 19
• The frameworks are categorized by programming model, data sources, supported languages, and support for iterative processing.Source: §6 · p. 19
• The survey highlights best practices for using the studied frameworks.Source: §6 · p. 19