PaperlayerView exact revision on arXiv

Source-linked AI summary

Can the Elephants Handle the NoSQL Onslaught?

Avrilia Floratou, Nikhil Teletia, David J. Dewitt, Jignesh M. Patel, Donghui Zhang

arXiv:1208.4166v1cs.DB

TL;DR

The paper asks whether relational systems can compete with NoSQL systems in interactive data-serving and decision-support workloads. It benchmarks SQL Server against MongoDB and Hive, finding that SQL systems retain performance advantages while NoSQL systems remain competitive in many cases.

  • Problem

    The paper asks how RDBMS performance compares with NoSQL systems across interactive data-serving and decision-support workloads, extending prior comparisons that covered only simple DSS queries and MapReduce.

    Source: §1 · p. 1

  • Method

    The authors compare SQL Server with MongoDB using YCSB for interactive serving, and PDW with Hive using TPC-H for decision-support analytics.

    Source: §1 · p. 2, §3.4.1 · p. 9

  • Results

    SQL systems retain significant performance advantages across the compared NoSQL classes, including PDW at approximately 9X faster than Hive on 16TB TPC-H.

    Source: §1 · p. 2, §3.5 · p. 12

  • Takeaways & Limitations

    SQL systems may need to preserve their performance advantages and expand functionality, including automatic sharding and a more flexible data model such as JSON.

    Source: §1 · p. 2

    The results are unaudited and unofficial, did not meet all benchmark requirements, and support relative comparisons only within this paper.

    Source: §1 · p. 2

Abstract

from arXiv · show

In this new era of "big data", traditional DBMSs are under attack from two sides. At one end of the spectrum, the use of document store NoSQL systems (e.g. MongoDB) threatens to move modern Web 2.0 applications away from traditional RDBMSs. At the other end of the spectrum, big data DSS analytics that used to be the domain of parallel RDBMSs is now under attack by another class of NoSQL data analytics systems, such as Hive on Hadoop. So, are the traditional RDBMSs, aka "big elephants", doomed as they are challenged from both ends of this "big data" spectrum? In this paper, we compare one representative NoSQL system from each end of this spectrum with SQL Server, and analyze the performance and scalability aspects of each of these approaches (NoSQL vs. SQL) on two workloads (decision support analysis and interactive data-serving) that represent the two ends of the application spectrum. We present insights from this evaluation and speculate on potential trends for the future.

1. INTRODUCTION

The paper asks whether relational systems can remain competitive as NoSQL systems challenge them in both interactive data-serving and large-scale analytics. It compares SQL Server with MongoDB and Hive using benchmarks for those two workload classes.

1. INTRODUCTION

  • Document-store systems such as MongoDB offer an alternative for interactive applications, while cluster-oriented MapReduce systems such as Hive target large-scale analytics.NoSQL systems may include built-in sharding or partitioning primitives for distributing data across nodes.Source: §1 · p. 1
  • Interactive data-serving workloads use small reads or updates, while decision-support workloads run complex queries over massive datasets.The former resembles traditional OLTP; Hive on Hadoop represents the newer MapReduce-based analytics alternative.Source: §1 · p. 1
  • The evaluation asks how RDBMS performance compares with NoSQL across interactive serving and decision-support workloads, extending earlier work limited to simple DSS queries and MapReduce.The cited earlier study did not consider Hive, a more sophisticated MapReduce query-processing system.Source: §1 · p. 1
  • SQL systems retained significant performance advantages over both NoSQL classes, although the NoSQL systems were competitive in many cases.The paper also argues that SQL systems may need broader functionality, including automatic sharding and a more flexible data model such as JSON.Source: §1 · p. 2
  • The evaluation covers one system per class, so other systems could produce different comparative results.The authors describe the study as an initial understanding of the emerging big-data landscape.Source: §1 · p. 2
  • The benchmark results are unaudited and unofficial, and should be used only for relative comparisons within this paper.The runs did not meet all benchmark requirements and should not be compared with official results.Source: §1 · p. 2

2. BACKGROUND

The background introduces PDW and Hive for parallel analytics, MongoDB for document-oriented data serving, and client-sharded SQL Server and MongoDB variants used in the comparison.

2.1 Parallel Data Warehouse (PDW)

  • PDW is a shared-nothing parallel database whose control node plans parallel operations and whose compute nodes execute them, with DMS moving data between nodes.The control node post-processes and reintegrates results for delivery.Source: §2.1 · p. 2

2.2 Hive

  • Hive provides a structured data model and HiveQL, which it translates into a directed acyclic graph of MapReduce jobs.Its data can be organized into databases, tables, partitions, and buckets, with multiple storage formats available.Source: §2.2 · p. 2

2.3 MongoDB

  • MongoDB stores BSON documents in collections without requiring every document in a collection to share the same structure.It supports B-tree indexes, auto-sharding, asynchronous replication, and failover through replica sets.Source: §2.3 · p. 2

2.4 Client-side Sharded SQL Server and MongoDB

  • The study implements client-side sharding for SQL Server and MongoDB to compare them in a cluster environment.Unlike Mongo-AS, both client-sharded variants use hash rather than range partitioning; Mongo-CS also lacks automatic balancing and failover.Source: §2.4 · pp. 2–3
  • The compared systems have different feature sets: MongoDB offers a flexible data model and single-entity read/write atomicity, while SQL Server provides full ACID semantics and multiple isolation levels.SQL Server also has better manageability and performance-analysis tools, according to the paper.Source: §2.4 · p. 3

3. EVALUATION

The evaluation compares parallel SQL Server PDW with Hive for decision support and compares MongoDB with client-sharded SQL Server and MongoDB for interactive serving.

3. EVALUATION

  • TPC-H evaluates PDW and Hive for decision support, while YCSB compares Mongo-AS with SQL-CS and Mongo-CS for interactive serving.The section introduces the hardware and software configurations used in the experiments.Source: §3 · p. 3
  • The DSS comparison uses Microsoft Parallel Data Warehouse and Hive, representing relational and NoSQL systems in the evaluation.The passage identifies TPC-H as the benchmark for this comparison.Source: §3 · p. 3
  • The interactive-serving comparison includes MongoDB with auto-sharding, client-sharded SQL Server, and client-sharded MongoDB.The passage identifies YCSB as the benchmark for these systems.Source: §3 · p. 3

3.1 Hardware Configuration

The experiments used a 16-node cluster, with distinct storage arrangements and node roles for the DSS and interactive-serving evaluations.

3.1 Hardware Configuration

  • All experiments used 16 nodes connected by a 1 Gbit Ethernet switch, each with dual quad-core Xeon processors, 32 GB memory, and ten SAS disks.One hard drive on each node was reserved for the operating system.Source: §3.1 · p. 3
  • For DSS, eight data disks formed one RAID 0 volume for Hive but separate logical volumes for PDW; each PDW node also stored its log on a separate disk.Source: §3.1 · p. 3
  • Hive used an extra namenode/jobtracker node, while PDW used separate control and landing nodes; the operating systems also differed between them.The landing node handled data loading and did not participate in query execution.Source: §3.1 · p. 3
  • YCSB used eight server nodes and eight client nodes, with MongoDB and SQL Server using different disk arrangements.Mongo-AS additionally used an extra configuration server to hold cluster-state metadata.Source: §3.1 · p. 3

3.2 Software Configuration

The software configuration describes Hive and Hadoop, PDW, and MongoDB, including settings chosen for the benchmark comparison.

3.2 Software Configuration

  • The section sets out the software configurations used for the systems tested.Source: §3.2 · p. 3

3.2.1 Hive and Hadoop

  • Hive 0.7.1 on Hadoop 0.20.203 used 8 map and reduce tasks per node, 2GB per-task heaps, 256MB HDFS blocks, and 128 reducers per job.TPC-H tables were stored as compressed RCFile, with map-side aggregation and join features enabled.Source: §3.2.1 · p. 3

3.2.2 PDW

  • PDW AU3 pre-release ran SQL Server 2008 with 24GB buffer pools per compute node and 128 partitions; Hive used identical hardware to avoid an apples-to-oranges comparison.PDW was tested on hardware unlike its larger-memory, Infiniband-connected commercial appliance configuration.Source: §3.2.2 · p. 3

3.2.3 MongoDB (Mongo-AS)

  • MongoDB 1.8.2 used 128 shards and 16 mongod processes per server, a choice intended to raise write concurrency around per-process global locks.The experiments did not configure replica sets.Source: §3.2.3 · pp. 3–4

3.3 Traditional DSS Workload: Hive vs. PDW

The study compares Hive and PDW on TPC-H across four scale factors, describing workload and layout choices before reporting performance and scaling results.

3.3.1 Workload Description

  • The workload ran 22 TPC-H queries sequentially at 250GB, 1000GB, 4000GB, and 16000GB; refresh functions were omitted because that Hive version lacked required insert and delete support.The generator was modified to use a 64-bit random number generator after its partkey and custkey values overflowed at 16TB.Source: §3.3.1 · p. 4

3.3.2 Data Layout

  • Hive partitioning can reduce scans for partition-column selections, while bucketing can help joins but not selections; PDW hash-distributes or replicates tables.The PDW comparison omitted indexes because the tested Hive optimizer could not automatically generate plans that used them; the authors planned to revisit the comparison when Hive considered indexes.Source: §3.3.2 · pp. 4–5

3.3.3 Data Preparation and Load Times

  • Hive loading used a two-phase process—copying text data into HDFS, then converting it to compressed RCFile—while PDW split landing-node files for parallel loading across 16 compute nodes.Table 2 presents load times, but the supplied passages do not give the values.Source: §3.3.3 · p. 5, Table 2 · p. 5

3.3.4 Experimental Evaluation

  • Table 3 reports per-query runtimes, PDW-over-Hive speedups, and scale-up factors; Figure 1 plots normalized arithmetic and geometric mean response times.The figure normalizes results to PDW at scale factor 250.Source: §3.3.4 · p. 5, §3.3.4.1 · p. 5, Figure 1 · p. 5

3.3.4.1 Performance Analysis

  • 34.1X was PDW’s average speedup over Hive at 250 GB; PDW also had lower normalized means and was faster on every query at every scale.Figure 1 plots normalized arithmetic and geometric mean response times; Query 9 did not complete in Hive at 16TB because of insufficient disk space.Source: §3.3.4.1 · p. 5, Figure 1 · p. 5, §3.3.4 · p. 5

Query 5

PDW HIVE

The section pairs PDW’s local-join planning advantages with Hive’s scaling behavior, showing an 18x PDW advantage on Query 19 while tracing Hive’s workload-specific overheads.

PDW HIVE

  • For Query 5, PDW’s cost-based repartitioning enables local joins, while Hive’s common joins repartition tables and its RCFile reads are CPU-bound at about 70 MB/sec.Hive’s join order is user-determined, missing optimization opportunities; the cited read bandwidth is far below the setup’s sequential I/O capacity.Source: PDW HIVE · pp. 6–7

Query 19

  • At the 16TB scale factor, Query 19 ran approximately 18 times faster in PDW than in Hive.PDW replicates the part table and joins it locally; Hive redistributes both tables for a common join.Source: Query 19 · p. 7

3.3.4.2 Scalability Analysis

  • Hive’s response times scale sub-linearly for some queries when dataset size grows fourfold, and it scales well as dataset size increases.The section analyzes queries where execution time does not rise proportionally with each fourfold dataset increase.Source: §3.3.4.2 · p. 7

Query 1

  • 2.3X, 3.7X, and 4X were the successive map-phase time increases as dataset size grew fourfold; sparse bucketing also left many files empty.At 250 GB, mixed allocation of empty and non-empty files increased map-phase time beyond the ideal schedule; the empty-file overhead declined at larger scales.Source: Query 1 · p. 7

Query 22

  • For Sub-query 1, 4× data growth from 1TB to 4TB increased job time only 47 to 102 seconds as startup overhead dominated; Sub-query 2 took equal time at 1TB and 4TB.At those scale factors, 3 and 12 map tasks fit in 128 slots in one round. The join failed after about 400 seconds due to Java heap errors, then Hive’s common-join backup completed it.Source: Query 22 · p. 8

3.3.4.3 Discussion

  • PDW’s cost-based join ordering and repartitioning reduce network transfers; Hive’s bucketing and map-side joins face optimizer, memory, and co-location constraints.Buckets must fit each map task’s memory, while non-colocated HDFS buckets can still require network I/O.Source: §3.3.4.3 · pp. 8–9
  • Hive has lower scaling factors than PDW, partly because its small-dataset overheads diminish and some tasks take the same time across scale factors.Examples include empty-file and map-task startup costs, available parallelism, and fixed-duration filesystem-related jobs.Source: §3.3.4.3 · p. 9

3.4 “Modern” OLTP Workload: MongoDB vs. SQL Server

In the tested read-only and read-heavy YCSB workloads, SQL-CS achieved higher throughput than the MongoDB variants, while Mongo-AS led short-range scans but had much higher append latency. The evaluation compared three systems across five workloads, with MongoDB run without durability support.

3.4.1 Workload Description

  • The YCSB evaluation compared Mongo-AS, Mongo-CS, and SQL-CS across five workloads using 640 million 1-KB records and 800 client threads.MongoDB was run without logging or disk-flush durability, while clients waited for server acknowledgement after writes.Source: §3.4.1 · pp. 9–10

3.4.3 Experimental Evaluation

  • 125,457 ops/sec was SQL-CS’s peak in Workload C, versus 68,533 for Mongo-AS and 60,907 for Mongo-CS.SQL-CS’s average read latency at its peak was 6.4 ms; Mongo-AS and Mongo-CS peaked at 11.8 ms and 13.2 ms, respectively.Source: Figure 2 · p. 9, §3.4.3 · p. 10
  • 103,789 ops/sec was SQL-CS’s achieved throughput in Workload B, while MongoDB systems missed the 40,000 ops/sec target.Checkpointing reduced SQL-CS throughput from about 15,000 ops/sec per server to 7,000–8,000 ops/sec, limiting its maximum relative to Workload C.Source: Figure 3 · p. 9, §3.4.3 · pp. 10–11

3.4.2 Data Preparation

  • SQL-CS and Mongo-CS loaded the data in 146 and 45 minutes, respectively; SQL-CS used separate insertion transactions rather than bulk insert.Mongo-AS’s manually distributed chunks took 114 minutes to load.Source: §3.4.2 · p. 10

3.4.3 Experimental Evaluation

  • Workload A’s greater update share coincided with 25%–45% of time spent at MongoDB’s global lock, versus 4%–12% in Workload B.At a 40,000 ops/sec target, SQL-CS under read uncommitted had 69 ms update latency and 15 ms read latency; the authors attributed the lower read latency to reads not being blocked by writes.Source: Figure 4 · p. 10, §3.4.3 · p. 11
  • At Workload D’s highest achievable throughput of 224,271 ops/sec, Mongo-CS had higher read and append latencies than SQL-CS.Mongo-AS had 320 ms append latency at a 20,000 ops/sec target and crashed above that target during this workload.Source: Figure 5 · p. 11, §3.4.3 · pp. 11–12
  • 6,337 ops/sec was Mongo-AS’s highest throughput on Workload E, with 30.4 ms scan latency and 1,832 ms append latency.Range partitioning let Mongo-AS scan typically one partition per short-range query; SQL-CS and Mongo-CS used hash partitioning and scanned more partitions.Source: Figure 6 · p. 11, §3.4.3 · p. 12

3.5 Discussion

Across DSS and interactive data-serving workloads, the evaluated relational systems outperform their NoSQL counterparts, though the NoSQL systems remain competitive in many cases. The discussion highlights both PDW’s advantage over Hive and SQL-CS’s advantage over MongoDB.

3.5 Discussion

  • The evaluation compared PDW and SQL-CS with Hive and MongoDB on DSS and OLTP workloads.Source: §3.5 · p. 12
  • At 16TB, PDW was approximately 9X faster than Hive on TPC-H, even without PDW indexing.The authors attribute the advantage to relational systems’ mature cost-based optimization and query-evaluation techniques, which MapReduce systems could adopt.Source: §3.5 · p. 12
  • SQL-CS achieved higher throughput than MongoDB at the same client count and lower latency in almost every YCSB test, even without MongoDB durability.This contrasts with the belief that relational databases are too heavyweight for simple single-operation requests.Source: §3.5 · p. 12

4. CONCLUSIONS AND FUTURE WORK

The relational systems retain a significant performance advantage over the evaluated NoSQL alternatives, which are nevertheless competitive in some cases. The paper notes that system choice also involves non-performance features and proposes broader comparisons in future work.

4. CONCLUSIONS AND FUTURE WORK

  • Relational systems retained a significant performance advantage over NoSQL counterparts, while NoSQL alternatives were competitive in some cases.Source: §4 · p. 12
  • NoSQL systems tend toward more flexible data models, auto-sharding, automatic load balancing, and different consistency models.Source: §4 · p. 12
  • Future work should evaluate other SQL and NoSQL systems and revisit their performance differences in a few years.Source: §4 · p. 12
Loading 1208.4166v1…