PaperlayerView exact revision on arXiv

Source-linked AI summary

Solving Big Data Challenges for Enterprise Application Performance Management

Tilmann Rabl, Mohammad Sadoghi, Hans-Arno Jacobsen, Sergio Gómez-Villamor, Victor Muntés-Mulero, Serge Mankowskii

arXiv:1208.4167v1cs.DB

TL;DR

APM produces high-rate monitoring data that must be stored long-term while remaining available for current monitoring. The paper benchmarks six open-source data stores with APM-like workloads and reports scalability, latency, and configuration findings. Cassandra generally provides the highest throughput, while its latency is high; the study also identifies practical setup difficulties.

  • Problem

    APM can generate millions of measurements per second, creating a need for storage that sustains high ingestion while supporting current monitoring and historical analysis.

  • Method

    The paper benchmarks six data stores using YCSB-based APM workloads on memory-bound and disk-bound clusters, measuring throughput and latency.

  • Results

    Cassandra’s throughput dominated in all tests and showed linear scalability in most experiments, but its latency was peculiarly high.

  • Takeaways & Limitations

    The results characterize trade-offs among storage architectures for APM and related high-rate monitoring workloads, including scalability, latency, and operational configuration.

  • Takeaways & Limitations

    HBase benchmark runs frequently failed nondeterministically and often required repeated execution after broken test runs.

Abstract

from arXiv · show

As the complexity of enterprise systems increases, the need for monitoring and analyzing such systems also grows. A number of companies have built sophisticated monitoring tools that go far beyond simple resource utilization reports. For example, based on instrumentation and specialized APIs, it is now possible to monitor single method invocations and trace individual transactions across geographically distributed systems. This high-level of detail enables more precise forms of analysis and prediction but comes at the price of high data rates (i.e., big data). To maximize the benefit of data monitoring, the data has to be stored for an extended period of time for ulterior analysis. This new wave of big data analytics imposes new challenges especially for the application performance monitoring systems. The monitoring data has to be stored in a system that can sustain the high data rates and at the same time enable an up-to-date view of the underlying infrastructure. With the advent of modern key-value stores, a variety of data storage systems have emerged that are built with a focus on scalability and high data rates as predominant in this monitoring use case. In this work, we present our experience and a comprehensive performance evaluation of six modern (open-source) data stores in the context of application performance monitoring as part of CA Technologies initiative. We evaluated these systems with data and workloads that can be found in application performance monitoring, as well as, on-line advertisement, power monitoring, and many other use cases. We present our insights not only as performance results but also as lessons learned and our experience relating to the setup and configuration complexity of these data stores in an industry setting.

1. INTRODUCTION

Enterprise systems generate complex, high-volume monitoring data that must support both sustained ingestion and timely analysis. The paper benchmarks six data stores across two hardware setups and reports performance and industry configuration experience.

  • Enterprise systems can span thousands of heterogeneous, interdependent servers, making administration and monitoring complex.
  • 10 million individual measurements per second can arise from monitoring 10K nodes at 10K average metrics per node every 10 seconds.
  • The study compares Apache Cassandra, Apache HBase, Project Voldemort, Redis, VoltDB, and MySQL Cluster.The systems represent a broad range of modern storage architectures.
  • The benchmark measures maximum sustainable throughput using memory-bound and disk-bound hardware setups.
  • The paper contributes an APM use case specification, an up-to-date comparison of six data-store architectures, and industry experience with setup and configuration.

2. APPLICATION PERFORMANCE MANAGEMENT

Application performance management monitors complex enterprise software through detailed instrumentation, producing millions of measurements per second. Its storage system must archive data long-term while keeping recent data available for real-time monitoring and notifications.

  • A transaction may span more than a thousand components across physical machines, while response time remains critical.
  • APM addresses systems where heterogeneous components and unavailable source code prevent in-depth component analysis or profiling.
  • APM can capture method invocations and aggregate thousands of measurements per agent at fixed intervals, yielding millions of measurements per second in larger deployments.
  • APM storage must support long-term archival, online access to recent data, sliding-window aggregates, and emergency notifications.
  • 100:1 or more is the typical write-to-read ratio, with online queries processed in subsecond ranges and historical queries taking minutes.
  • Monitoring should impose no more than 5% overhead, and only tens of nodes may be dedicated to archiving data in an enterprise system with hundreds of nodes.

3. BENCHMARK AND SETUP

The benchmark models APM’s append-oriented monitoring workload with YCSB and evaluates key-value stores on memory-bound and disk-bound clusters. It uses multiple workload mixes, realistic records, and repeated long-running executions.

  • The storage queries comprise current-value lookups and small scans for health information and time-window aggregates.
  • The benchmark uses YCSB to generate configurable CRUD workloads mapped to key-value or column-based data models.
  • APM records contain a 25-byte key and five 10-byte value fields, producing a 75-byte raw record.
  • Five workloads vary insert, read, and scan operations, including read-intensive mixes and write-to-read ratios relevant to append-only APM data.
  • The experiments use independent memory-bound and disk-bound clusters with different data footprints and hardware configurations.
  • Each test ran for 600 seconds, with results averaged over at least three independent executions.

4. BENCHMARKED KEY-VALUE STORES

The benchmarked stores cover key-value, extensible record, and scalable relational architectures with differing consistency, replication, partitioning, and memory-use designs. Their implementations also vary in maturity, configuration, and cluster support.

  • The six stores comprise Project Voldemort and Redis, HBase and Cassandra, and MySQL Cluster and VoltDB.
  • The comparison excludes document stores because no candidate was found that appeared to match the study’s requirements.
  • 4.1 HBase: HBase is a distributed, column-oriented database based on BigTable, running on Hadoop and HDFS for fault tolerance and replication.
  • 4.2 Cassandra: Cassandra is a distributed key-value store designed for large data volumes, high availability, replication, and linear read and write scalability.
  • 4.3 Voldemort: Voldemort is a distributed, fault-tolerant persistent hash table that automatically partitions and replicates data without central coordination.
  • 4.4 Redis: Redis is an in-memory key-value store with optional durability and master-slave replication, while its tested cluster version was unstable.
  • 4.5 VoltDB: VoltDB is an ACID-compliant relational in-memory database using shared-nothing partitioning and stored procedures as transaction units.

5. EXPERIMENTAL RESULTS

The experiments compare throughput and latency across six data stores under Workload R, a 95%-read workload, using logarithmic latency scales. Results show distinct trade-offs in scalability, throughput, and read/write latency across architectures.

  • Workload R setup: Workload R used 95% reads and 5% writes, and each run loaded freshly installed systems before a 10-minute maximum-throughput test.The workload size was scaled with cluster size using 10 million records per node.
  • Workload R throughput: More than 50K ops/sec: Redis achieved the highest single-node throughput, while HBase reached 2.5K operations per second.Cassandra and MySQL achieved about 25K ops/sec, and Voldemort 12K ops/sec.
  • Workload R throughput: Cassandra, Voldemort, and HBase demonstrated a roughly linear increase in maximum throughput as cluster size grew.These three web data stores were explicitly designed for scalability in web-scale settings.
  • Workload R read latency: 230 µs: Voldemort had the lowest read latency for one node, increasing to 260 µs for 12 nodes.Cassandra’s average read latency was 5–8 ms, while HBase’s was 50–90 ms.
  • Workload R read latency: Redis and MySQL showed decreasing read latency with scale, whereas VoltDB’s read latency increased as throughput decreased.The sharded stores’ decreasing latency was attributed to reduced load per system.
  • Workload R write latency: HBase had the best write latency but was less stable, while Cassandra had the highest stable write latency among the benchmarked systems.Voldemort had roughly equal write and read latency, making it a compromise between write and read speed.

5.2 Workload RW

Workload RW evaluates systems under a 50% write rate, revealing distinct throughput scaling and latency patterns across the six stores.

  • VoltDB achieves the highest single-node throughput, while Redis is slightly lower and 20% below its Workload R throughput.
  • Cassandra, HBase, and Project Voldemort scale linearly across multiple nodes; MySQL scales well through 8 nodes before leveling off.
  • Read latency is essentially unchanged from Workload R, except MySQL is 75% lower at one node and 40% lower at 12 nodes.
  • Write-latency trends resemble Workload R, but HBase is almost 50% lower and MySQL is twice as high across scales.

5.3 Workload W

Workload W most closely represents the APM use case without scans and stresses the stores with a 99% write rate, producing markedly different performance behavior.

  • 99% write rate makes Workload W closest to the APM use case without scans, but too write-heavy for web information systems’ production workloads.
  • HBase nearly doubles throughput at 12 nodes, whereas Cassandra increases only 2% under Workload W.
  • At 12 nodes, HBase read latency reaches 1 second on average, while Voldemort’s read latency is almost twice its Workload R and RW level.
  • HBase write latency increases by a factor of 20, whereas other systems increase by 5-15% and Voldemort remains almost identical to Workload RW.
  • Workload RS: MySQL has the best single-node throughput for Workload RS but does not scale with additional nodes, while Cassandra and HBase scale linearly.
  • Workload RS: MySQL scans become slow beyond two nodes because the YCSB client translates scans into inefficient key-range SQL queries.
  • Workload RS: HBase scan latency is almost in the second range, while Cassandra scans remain constant at 20-25 milliseconds and are four times slower than reads.

5.5 Workload RSW

Workload RSW combines reads, scans, and writes, showing that reduced scan frequency benefits HBase and Cassandra while severely hurting MySQL as the cluster grows.

  • Workload RSW contains 50% reads, with 25% of all operations scans, and its throughput resembles Workload RS except for sharper MySQL degradation.
  • 20 operations per second is MySQL’s one-node throughput, falling below one operation per second from four nodes onward.
  • Twice as high as Workload RS is the throughput achieved by HBase and Cassandra under the lower scan rate.
  • VoltDB achieves the best one-node throughput, but its throughput only slightly decreases from two to four nodes.
  • Scan latencies are stable for most systems, with MySQL as the exception; Cassandra and Voldemort have the scan-operation latency noted for Workload RSW.
  • Varying Throughput: At reduced load, Cassandra’s normalized read latency decreases almost linearly, while HBase decreases linearly below 80% but remains constant above that level.
  • Varying Throughput: HBase write latency is unstable but always below 0.1 milliseconds in the reduced-load tests.

5.7 Disk Usage

Disk usage grows linearly with records and cluster size, but storage overhead differs substantially across the evaluated disk-based systems.

  • 700 megabytes for one node to 8.4 gigabytes for 12 nodes is the raw-data growth for 10 million 75-byte records per node.
  • 2.5 gigabytes per node is Cassandra’s post-load disk usage, compared with 5 gigabytes for MySQL and 5.5 gigabytes for Project Voldemort.
  • VoltDB was omitted from the bounded-throughput test because its latency was prohibitive above 4 nodes.
  • 7.5 gigabytes per node makes HBase the least storage-efficient system, using 10 times the raw data size.
  • Additional schema and version information drives overhead, while compression can reduce disk usage at the cost of throughput.

5.8 Disk-bound Cluster (Cluster D)

On disk-bound Cluster D, Cassandra, HBase, and Project Voldemort showed lower throughput and higher latency than in-memory tests, while workload write ratios strongly affected throughput. HBase provided very low write latency, whereas Voldemort offered the lowest read latency among the detailed results.

  • Test setup: Disk-bound Cluster D excluded Redis, VoltDB, and MySQL and tested only workloads R, RW, and W at one scale.The inserted dataset exceeded available memory, forcing all tested systems to use disk.
  • Throughput: 26× and 15×: Cassandra’s throughput increased by a factor of 26 and HBase’s by 15 from Workload R to Workload W.The throughput gains occurred as write ratios increased.
  • Read latency: 40 ms, 70 ms, and 5–6 ms: Cassandra, HBase, and Voldemort respectively showed distinct read-latency patterns across workloads.Cassandra reached 25 ms for Workload W; HBase exceeded 200 ms for Workload W; Voldemort rose to 20 ms for Workload W.
  • Write latency: Below 1 ms: HBase had very low write latency, especially for mixed Workload RW.Cassandra and Voldemort showed stable write latency with a slight decrease for Workload RW.
  • Overall comparison: Cassandra, HBase, and Project Voldemort scaled near-linearly, but disk-bound operation produced much lower throughput and higher latency.Cassandra had the highest throughput, HBase low write latency but high read latency, and Voldemort stable low-level read and write latency.
  • Sharded and single-instance systems: Sharded MySQL achieved throughput almost as high as Cassandra’s, while sharded Redis and single-instance VoltDB did not scale comparably across nodes.Redis throughput suffered from workload imbalance, and VoltDB showed no throughput increase beyond one node.

6. EXPERIENCES

The industry evaluation found that configuration, tuning, client behavior, and debugging materially shaped the benchmarking process. Cassandra was comparatively straightforward to deploy, while HBase and several client or sharding setups introduced substantial operational complications.

  • Industry experience: At least a week was devoted to configuring and tuning each system beyond its default configuration.The evaluation explicitly examined setup, configuration, and tuning difficulty from an industry perspective.
  • Benchmarking: YCSB initially failed to saturate high-performing systems, requiring a recent patch and a client-to-storage-node ratio reduced to 1:2.The adjustment addressed benchmark-client scalability rather than datastore functionality.
  • Cassandra: Cassandra’s symmetric architecture made one common setup virtually sufficient, but random default tokens frequently produced highly unbalanced workloads.The benchmark therefore assigned an optimized token set to the nodes.
  • HBase: HBase required Hadoop installation, service-placement decisions, and difficult debugging of client memory leaks and misleading HDFS permission errors.The setup placed master services with data nodes because the benchmark used at most 12 nodes.
  • HBase: HBase benchmark failures were frequent, nondeterministic, and often required repeated test runs despite no obvious issue.The reported failures contrasted with the other systems evaluated.
  • Client and sharding issues: Redis required a custom sharded YCSB client, while connection counts forced fewer threads and client tuning affected system saturation.Project Voldemort similarly reached its default client limits of 10 threads and 50 connections in maximum-throughput tests.
  • MySQL: MySQL installation was smooth, but the sharded JDBC client required fewer threads to avoid saturating the systems.An alternative would have been a different YCSB client or the MySQL Cluster version.
  • VoltDB: VoltDB configuration followed community guidance, but the benchmark’s synchronous communication differed from prior tests using asynchronous communication.The paper notes that asynchronous communication may better fit VoltDB’s execution model.

7. RELATED WORK

Related work situates the study within the rise of specialized key-value stores and prior YCSB-based evaluations. The paper emphasizes that earlier studies did not provide a comparable broad, multi-system evaluation at this node scale.

  • Specialized data stores: Specialized data stores emerged after relational systems had dominated both the market and academic research.Key-value architectures were adopted by major companies including Google, Facebook, LinkedIn, Yahoo!, and Amazon.
  • Scope of comparison: The study compared six publicly available modern data stores spanning a broad range of architectures.The selected systems reflected different architectural categories and performance claims.
  • Prior benchmarks: YCSB provided a simple, widely used benchmark that fit the APM use case and supported comparisons involving systems such as PNUTS, Cassandra, HBase, and MySQL.Prior work also included scan-focused YCSB evaluations and the YCSB++ extension for enhanced monitoring.
  • Research gap: No other study known to the authors compared such a wide selection of systems at a scale of up to 12+ nodes.The paper used small records and reported comprehensive results across a broad range of workloads.

8. CONCLUSION

The paper evaluates six storage systems for APM and related high-rate workloads, combining maximum-throughput benchmarking with operational observations. Cassandra, HBase, and Project Voldemort generally scaled linearly, but their latency and throughput trade-offs differed, and further work is needed for reliable APM support.

  • Contribution: Six open-source storage systems were evaluated for storing APM monitoring data and related workloads such as online advertising, click streams, and power monitoring.The study focused on maximum achievable throughput and used two differently structured compute clusters.
  • Results: Cassandra, HBase, and Project Voldemort showed linear scalability in most tests, with Cassandra dominating throughput but exhibiting peculiarly high latency.Voldemort maintained much lower stable latency, while HBase traded the lowest throughput for low write and high read latency.
  • Results: Sharded Redis and MySQL achieved good throughput but were less scalable than Cassandra, HBase, and Project Voldemort.The throughput of multi-node sharded systems depended substantially on the sharding approach.
  • APM applicability: 240K inserts per second: the projected 12-monitoring-node scenario exceeds Cassandra’s maximum Workload W throughput on the in-memory cluster, though not drastically.The authors note that further improvements are needed to reliably sustain APM requirements because Cluster M stored data in memory.
  • Future work: Replication and compression remain future-work variables whose impact on throughput was not determined.The authors also plan to extend the range of tested architectures.
Loading 1208.4167v1…