Source-linked AI summary
BigDataBench: a Big Data Benchmark Suite from Internet Services
Lei Wang, Jianfeng Zhan, Chunjie Luo, Yuqing Zhu, Qiang Yang, Yongqiang He, Wanling Gao, Zhen Jia, Yingjie Shi, Shujie Zhang, Chen Zheng, Gang Lu, Kent Zhan, Xiaona Li, Bizhu Qiu
TL;DR
Big data systems require benchmarks that reflect diverse workloads and data, but existing efforts often target narrow applications or software stacks. This paper presents BigDataBench, an industrially developed suite with scalable real-world data and 19 characterized workloads, revealing low operation intensity, input-volume effects, and cache implications for big data architecture research.
Problem
Existing big data benchmarks inadequately cover the diversity of workloads and real-world data sets needed to evaluate broadly used big data systems and architectures.
Method
BigDataBench combines broad internet-service application scenarios, diverse real-world data sets, scalable data generation, and 19 workloads characterized with varying inputs.
Results
Big data applications show very low operation intensity, input volume affects micro-architecture characteristics, and L1 instruction-cache misses are high while LLC caches are effective.
Takeaways & Limitations
Architecture research on big data should account for realistic data volumes and the distinct cache and operation-intensity behavior of big data workloads.
Takeaways & Limitations
The authors plan to add other real-world data sets to study their impact on the same workloads.
Abstract
from arXiv · showhide
As architecture, systems, and data management communities pay greater attention to innovative big data systems and architectures, the pressure of benchmarking and evaluating these systems rises. Considering the broad use of big data systems, big data benchmarks must include diversity of data and workloads. Most of the state-of-the-art big data benchmarking efforts target evaluating specific types of applications or system software stacks, and hence they are not qualified for serving the purposes mentioned above. This paper presents our joint research efforts on this issue with several industrial partners. Our big data benchmark suite BigDataBench not only covers broad application scenarios, but also includes diverse and representative data sets. BigDataBench is publicly available from http://prof.ict.ac.cn/BigDataBench . Also, we comprehensively characterize 19 big data workloads included in BigDataBench with varying data inputs. On a typical state-of-practice processor, Intel Xeon E5645, we have the following observations: First, in comparison with the traditional benchmarks: including PARSEC, HPCC, and SPECCPU, big data applications have very low operation intensity; Second, the volume of data input has non-negligible impact on micro-architecture characteristics, which may impose challenges for simulation-based big data architecture research; Last but not least, corroborating the observations in CloudSuite and DCBench (which use smaller data inputs), we find that the numbers of L1 instruction cache misses per 1000 instructions of the big data applications are higher than in the traditional benchmarks; also, we find that L3 caches are effective for the big data applications, corroborating the observation in DCBench.
1 Introduction
BigDataBench addresses the limited diversity of existing big data benchmarks by covering broad application scenarios and representative real-world data sets. It characterizes 19 workloads and reports distinctive architectural behaviors on Intel Xeon processors.
- Existing benchmarks often target specific applications or software stacks, failing to represent diverse workloads and real-world data sets.
- BigDataBench uses an incremental, real-system methodology developed with industrial partners to cover broad application scenarios and representative data sets.
- Six real-world data sets span structured, semi-structured, and unstructured data from text, graph, and table sources.
- 19 big data workloads are comprehensively characterized with varying data inputs on an Intel Xeon E5645 processor.
- BigDataBench workloads have floating-point operation intensity two orders of magnitude lower than traditional benchmarks.
2. Big Data Benchmarking Requirements
The paper defines benchmark requirements around fair coverage of applications and data, data-centricity, workload diversity, software-stack representation, technical currency, and usability.
- Benchmarks should measure and compare systems across broad application scenarios and diverse, representative real-world data sets.
- Data-centric benchmarks should address volume, variety, velocity, and veracity while preserving raw data characteristics during synthesis.
- Workloads should represent diverse application scenarios and types to provide comprehensive characteristics for systems design and optimization.
- Benchmark suites should cover representative software stacks, emerging techniques, future changes, and easy deployment, configuration, execution, and measurement.
3 Related work
Prior big data benchmarks generally focus on narrow application classes, software stacks, or data types, leaving limited coverage of diverse real-world workloads and scalable data.
- Most prior efforts evaluate specific application types or software stacks rather than broadly measuring big data systems and architectures.
- AMP Lab benchmarks target real-time analytics but cover limited workloads and only table data.
- HiBench and GridMix focus on Hadoop-related workloads, with incomplete data types or only text microbenchmarks.
- YCSB targets simple Cloud OLTP storage workloads, while LinkBench characterizes Facebook social-graph data and database workloads.
- CloudSuite and DCBench use small data sets and lack diversity of real-world data sets and workloads, including scalable data generation preserving veracity.
4 Our Benchmarking Methodology and Decisions
BigDataBench builds a broad benchmark suite from dominant internet-service domains, diverse real-world data, multiple application types, and representative software stacks. Its data-generation process scales volume while preserving data characteristics.
- Application domains: The methodology uses an incremental, iterative process centered on search engines, social networks, and e-commerce.
- Data sets: Data diversity covers structured, semi-structured, and unstructured types, with text, graph, and table sources.
- Workloads: The suite includes online services, offline analytics, and real-time analytics, reflecting different latency and computation requirements.
- Data sets: Six representative real-world data sets are collected across data types, sources, and application domains.
- Workloads: BigDataBench contains 19 benchmarks spanning application scenarios, algorithms, data types, sources, software stacks, and application types.
- Workloads: Users can select specific workload types for focused comparisons, while comprehensive system evaluation should cover all benchmarks.
5 Synthetic Data Generation Approaches and Tools
BigDataBench addresses the challenge of obtaining large, representative data by scaling application-specific real-world data while preserving their characteristics. Its BDGS tool supports diverse domains, data types, formats, and user-specified sizes.
- Application-specific generators synthetically scale real-world data while preserving their significant characteristics.
- The generation procedure estimates data-model parameters from representative real-world data before producing synthetic data.
- BDGS generates synthetic big data while preserving the four V properties across search, e-commerce, and social-network domains.
- Six real-life data sets cover structured, semi-structured, and unstructured data from text, graph, and table sources.
- Users can specify data size, with practical limits determined by storage capacity, generator parallelism, and runtime.
6 Workload Characterization Experiments
The workload-characterization experiments examine experiment configurations, data-volume effects on micro-architecture events, and characterization of big data benchmarks.
- The experiments cover configurations and methodology, data-volume effects on micro-architecture events, and workload characterization.
6.1 Experiments Configurations and Methodology
The experiments characterize BigDataBench workloads across multiple software stacks, cluster scales, profiling methods, metrics, and established benchmark suites.
- Hadoop is the basic software stack, supplemented by HBase, Nutch, MPICH2, and Rubis for different workload studies.
- Experiments use up to 1 TB of input on a matching 14-node cluster, with each node containing two Xeon E5645 processors, 16 GB memory, and 8 TB disk.
- Perf collects about 20 hardware performance-counter events, while the proc filesystem supplies operating-system performance data after a roughly 30-second ramp-up.
- User-perceivable metrics include requests per second for online services, operations per second for Cloud OLTP, and data processed per second for analytics.
- SPEC CPU2006, HPCC, and PARSEC provide comparison workloads using their specified applications, inputs, versions, and benchmark sets.
6.2 The Implication of Data Volume for Architecture Research
Data volume materially changes both micro-architectural behavior and user-perceivable performance, and the direction and magnitude of change depend on the workload.
- Simulation-based architecture studies must consider input scale because larger data increases runtime, while small inputs are valid only if events remain similar.
- Different workloads exhibit different MIPS trends as data volume increases; Grep and WordCount increase after the 16-times baseline.
- K-means L3 cache MPKI rises from 0.8 with small input to 2 with large input, whereas Sort decreases, Grep increases, and Index shows no obvious difference.
- Larger inputs can degrade Sort performance because memory cannot hold all data, increasing I/O, shuffling, disk access, and network congestion.
- These workload-specific trends make narrow application coverage and small data sets insufficient for evaluating big data systems and architecture.
- The MIPS of Grep differs by 2.9 times and K-means L3 cache MPKI by 2.5 times between baseline and 32X data volume.
6.3 Workload Characterization
BigDataBench workloads exhibit low computation-to-memory-access ratios and distinctive cache and TLB behavior compared with traditional benchmarks. These characteristics vary with processor cache hierarchy and workload data-access patterns.
- Instruction behavior: 75 is the average integer-to-floating-point instruction ratio for BigDataBench, versus 1.4 for PARSEC, 1.0 for HPCC, and 0.67 for SPECFP.Across BigDataBench workloads, the ratio ranges from 10 for Bayes to 179 for Grep.
- Operation intensity: 0.007 and 0.05 are BigDataBench’s average floating-point operation intensities on Xeon E5310 and E5645, respectively.These values are two orders of magnitude lower than traditional workloads; L3 caching partly explains the higher E5645 value.
- Operation intensity: 0.5 and 1.8 are BigDataBench’s average integer operation intensities on Xeon E5310 and E5645, respectively, remaining in the same order of magnitude as other benchmarks.The comparison includes PARSEC, HPCC, SPECFP, and SPECINT.
- Operation intensity: BigDataBench workloads have low computation-to-memory-access ratios because they process large data volumes with relatively simple algorithms ranging from O(n) to O(n*lgn).Traditional HPCC and PARSEC workloads commonly range from O(n*lgn) to O(n3).
- Memory hierarchy: 23 is BigDataBench’s average L1 instruction-cache MPKI, at least four times higher than traditional benchmarks, while its average L2 MPKI is 21.The authors associate high L1I MPKI with huge code size and deep software stacks.
- Memory hierarchy: 1.5 is BigDataBench’s average L3-cache MPKI, indicating effective Xeon E5645 LLC caching, while its average ITLB and DTLB MPKI are 0.54 and 2.5.DTLB MPKI ranges from 0.2 for Nutch server to 14 for BFS, reflecting diverse data-access patterns.
- Lessons learned: The findings imply that processors should improve L1I performance while emphasizing cache area and energy efficiency for big-data workloads.The authors conjecture that deep software stacks contribute to high front-end stalls.
7 Conclusion
The paper presents BigDataBench as a broad big-data benchmarking effort built from real systems, diverse data, and scalable data generation. Its characterization finds low operation intensity, input-size-sensitive microarchitecture, effective LLC caching, and high L1I cache MPKI.
- Conclusion: BigDataBench covers broad application scenarios, diverse real-world data sets, scalable 4V-preserving data generation, and nineteen benchmarks across multiple workload dimensions.The methodology was developed jointly with several industrial partners from real systems.
- Conclusion: BigDataBench workloads have very low operation intensity compared with traditional benchmarks.The conclusion reports this as a principal characterization result.
- Conclusion: Input volume has a non-negligible impact on big-data microarchitecture characteristics, making simple applications and small data sets insufficient for big-data architecture scenarios.This finding concerns architecture research using representative workloads and data volumes.
- Conclusion: On Intel Xeon E5645, the LLC is effective for big-data workloads, while high L1I cache MPKI indicates demand for better L1I performance.These findings are reported as conclusions from the workload characterization.