Source-linked AI summary
No Silver Bullet: Boosting GaussDB Performance on the 30TB TPC-H Workload
Tim Zeyl, Jason Lam, Shu Lin, Reza Pournaghi, Qi Cheng, Calvin Wong, Kaixiang Du, Yuliang He, Yang Sun, Weicheng Wang, Paul Lee, Chen Ruo, Yang Xinyi, Li Qunan, Wang Junjie, Hu Dongxing, Chong Chen, Per-Ake Larson
TL;DR
Large-scale analytical workloads exposed scalability bottlenecks in GaussDB’s execution, shuffle, and optimization paths. The paper adapts multiple techniques, including pipeline execution, scalable distributed data movement, and Bloom-filter enhancements, then evaluates the integrated system on 30 TB TPC-H. GaussDB achieved QphH@30TB = 39,508,107, over 40% above the historical published record.
Problem
GaussDB needed to address execution, communication, and optimization challenges that limited analytical-query scalability as clusters grew to tens of terabytes.
Method
The paper integrates a pipeline execution model, scalable shuffle using Unified Bus and remote-memory mechanisms, distributed Bloom-filter optimization and streaming, near-FK pruning, and improved statistics.
Results
QphH@30TB = 39,508,107, over 40% higher than the historical published TPC-H record at 30 TB.
Takeaways & Limitations
The integrated enhancements enabled GaussDB to achieve elite analytical performance on TPC-H at scale.
Takeaways & Limitations
The reported TPC-H results were unaudited, and the throughput deployment used fewer vCPUs and less memory than Hologres.
Abstract
from arXiv · showhide
GaussDB is Huawei's premier database system, designed for large-scale deployments and the most demanding workloads. It is a distributed shared-nothing system, capable of handling all types of workloads. This paper outlines a series of modifications to GaussDB aimed at improving its performance on large-scale and complex analytical workloads. After these changes, its performance on the TPC-H workload exceeded the best published result by 40% at 30 TB. The key enhancements to achieve this elite performance include adopting a pipeline execution model, a faster and more scalable inter-node data shuffle, exploiting a unified bus and unified remote memory access. We also expanded the support of cost-based Bloom filter placement and implemented several Bloom filter streaming strategies, enabling their use across nodes.
1 INTRODUCTION
GaussDB’s scale-out analytical performance was limited by execution, communication, and optimization challenges. The paper addresses these bottlenecks with coordinated engine, shuffle, Bloom-filter, and statistics improvements, achieving elite 30 TB TPC-H performance.
- Scalability challenges emerged as clusters expanded to store and process tens of terabytes across more worker nodes.
- Volcano execution caused CPU overhead through many threads and OS-level scheduling for inter-thread communication at high parallelism.GaussDB therefore implemented a pipeline execution model to maintain performance at scale.
- Naïve shuffle communication created severe I/O bottlenecks because logical channels grew across SMP threads and nodes.The scalable shuffle reduced the number of connections by an exponential factor.
- TCP-based shuffle communication could make the network a bottleneck as database size and shuffle traffic increased.The design used Huawei’s Unified Bus communication channels to speed network traffic during shuffle.
- Distributed Bloom-filter support was expanded with new streaming strategies and cost-based optimization across data nodes.
- Optimizer cost modeling was aligned with streaming improvements, while HyperLogLog synopses improved distinct-value estimates and inclusion-dependency verification.
- 40% higher than Hologres, GaussDB achieved QphH@30TB = 39,508,107 on the 30 TB TPC-H workload.The result used ARM-based Kunpeng servers, whereas reference systems mainly used Intel-based servers.
2 GAUSSDB ARCHITECTURE
GaussDB is a shared-nothing distributed database that integrates transactional and analytical processing within each node. Its HTAP architecture combines row storage, per-node in-memory columnar views, specialized engines, and distributed execution.
- GaussDB separates coordination, storage, and execution across nodes using coordinator nodes and data nodes in a shared-nothing architecture.Coordinator nodes parse and optimize SQL, while data nodes store shards and execute query fragments in parallel.
- IMCV adds a local, in-memory columnar representation to each data node without introducing a separate analytical cluster or external columnar engine.
- GaussDB uses separate row and vector execution engines for transactional and analytical queries, with analytical plans decomposed and pushed to data nodes.The supplied passage states that analytical queries are classified and dispatched through the coordinator.
- Transactional updates remain in row storage and delta tables, which analytical execution merges with IMCV data for transactionally consistent snapshots.
- Per-node IMCV enables distributed analytical execution through local columnar scans, inter-node shuffling, and coordinator aggregation.This avoids centralized analytical bottlenecks while colocating OLTP and OLAP infrastructure at nodes.
- The paper’s enhancements target the vector engine for analytical queries and are not expected to affect transactional-query performance.
3 PIPELINE EXECUTION FRAMEWORK
GaussDB’s pipeline framework decomposes physical plans into parallel task pipelines and combines pull execution within pipelines with push-based communication between them. Event-driven dispatching and local exchange mechanisms support efficient execution and resource use.
- 3.1 Pull and push model: A pipeline builder decomposes Volcano-optimized physical plans into linear pipelines separated by materialization and exchange operators.Each pipeline runs from a source operator to a sink operator.
- 3.1 Pull and push model: Pipelines expose parallelism unavailable under Volcano; independent pipeline tasks process disjoint data and exchange it locally or remotely.In the hash-join example, pipelines P2 and P3 can run in parallel when CPU and memory resources are available.
- 3.1 Pull and push model: Within a pipeline, the pull model lets the driver exhaust downstream output before requesting upstream data, replacing recursion with a loop.
- 3.1 Pull and push model: The pull model transfers VectorBatch memory ownership downstream and permits safe reuse after downstream operators no longer need the batch.
- 3.1 Pull and push model: Between pipelines, push-based execution blocks downstream pipelines at materialization barriers but allows producer and consumer pipelines to overlap through exchange caches.
- 3.2 Local-exchange cache: Local exchange caches use up to 64 VectorBatch slots, atomic occupancy tracking, and reference counts for lock-free producer-consumer coordination.
- 3.2 Local-exchange cache: Cache memory is allocated as needed and released by the last consumer to reduce peak memory consumption.
- 3.3 Event-driven pipeline dispatching: Pipeline tasks self-dispatch when events remove dependencies, avoiding a dedicated dispatcher thread.Dispatch requires available source output, an accepting sink, and a zero sideways-dependency counter.
4 SCALABLE SHUFFLE
GaussDB’s scalable shuffle system redistributes tables across data nodes while addressing communication growth from high DOP and many DNs. The design combines two-level mailboxes, scalable inter-node communication, and UB/URMA-based data movement.
- Shuffle overview: Shuffle redistributes tables among data nodes using table attributes as redistribution keys, with partitions scaling as data nodes multiplied by per-node DOP.This supports partitioned distributed joins, where both join sides may need redistribution on join keys.
- Mailbox design: Two-level mailboxes reduce communication channels by managing quotas at the data-node level instead of the DOP-partition level.The design replaces flat producer-consumer connections and scales better as DOP or the number of data nodes increases.
- Mailbox design: The two-level-mailbox design achieves faster data transfer and reaches query-processing transition points earlier than the flat-mailbox design.The comparison used TPC-H Q9 across varying data-node and DOP configurations at 3.75 TB and 7.5 TB; the transfer-rate example used 32 data nodes and DOP=32.
- UB-based shuffle: UB replaces TCP channels for shuffle to exploit high concurrency, large bandwidth, and low latency, while URMA provides remote-memory communication primitives.Reliable messaging was selected for performance, with receiver-side ordering buffers preserving per-stream order.
- UB-based shuffle: UB improved performance by 7.18% versus TCP overall in an 8-node, 7.5 TB TPC-H evaluation, but some low-shuffle queries regressed.The regressions were primarily attributed to kernel-level CPU overhead during URMA memory allocation and reclamation; adaptive communication selection was proposed as future work.
5 DISTRIBUTED BLOOM FILTERS
GaussDB extends Bloom-filter optimization and streaming to distributed MPP deployments, balancing filtering benefits against network and memory costs. The distributed strategy scales better as node counts increase, while Bloom-filter-aware planning substantially reduces TPC-H execution time.
- Bloom filters reduce downstream processing by filtering rows early, improving query performance and limiting distributed data transfer.
- GaussDB extends cost-based Bloom-filter optimization from single-node SMP to MPP-SMP by modeling Bloom-filter streaming costs across data nodes.
- Distributed Bloom filters exploit aligned partitioning to merge per-thread filters into node-level filters that are serialized and transmitted across the cluster.
- MPP-merge Bloom-filter traffic scales as m d^2/8 bytes, versus m d/8 bytes for MPP-distributed filters, making distributed filters preferable at higher node counts.Merging remains necessary when the probe-side partitioning column is unavailable.
- The implementation adds bucket Bloom filters, serialized hash transmission, compression, and explicit prefetching to reduce cache misses, network traffic, and memory stalls.
- 50-60% reduction in total execution time is achieved by Bloom-filter-aware cost-based optimization across all 22 TPC-H queries, compared with no Bloom filters.Planner post-processing produced an approximate 20-40% reduction.
6 IMPROVING STATISTICS AND FOREIGN KEY APPROXIMATION
GaussDB uses approximate foreign-key relationships discovered from HLL statistics to guide aggregation rewrites and Bloom-filter pruning during optimization. The lightweight detector is fast and showed no false positives or false negatives in the reported TPC-H planning evaluation.
- Near-FK constraints provide approximate inclusion relationships that support cost-based optimization without the integrity guarantees of strict foreign keys.
- Automatic near-FK detection enables eager GROUP-BY pushdown and prevents unnecessary Bloom filters in several queries.
- 6.1 Exploiting near-FKs for eager aggregation: For TPC-H query 10, near-FK information lets the optimizer estimate downstream join selectivity before deciding whether to push aggregation through the join.
- 6.2 Using near-FKs for Bloom filter pruning: Near-FK detection prunes Bloom filters when an unfiltered primary-key build side and foreign-key probe side cannot yield filterable rows.
- 6.3 Detecting near-FK constraints: HLL synopses support a lower-cost detector that compares bucket-level trailing-zero maxima to test candidate inclusion relationships during planning.The procedure uses M comparisons, with additional handling for HLL shards.
- 6.3 Detecting near-FK constraints: The detector’s false-positive probability rises when the candidate column’s NDV is much smaller than the referenced column’s NDV, requiring special handling for small k.Using M=1024 buckets and a true primary key keeps the reported false-positive probability low; near-FKs tolerate non-zero false positives.
- 6.3 Detecting near-FK constraints: Zero false positives and zero false negatives were found among join clauses during planning of a TPC-H 7.5 TB deployment.The detector executes in tens to hundreds of microseconds.
7 ADDITIONAL IMPROVEMENTS
GaussDB adds vectorization, vector-form Numeric storage, and lightweight IMCV compression as further analytical-query optimizations. Compression reduces memory substantially while maintaining nearly unchanged end-to-end TPC-H latency at the reported scale.
- Vectorized execution was added for expression evaluation, join and aggregation hashing, Bloom-filter hashing, and selected LLVM-generated expressions.
- GaussDB stores Numeric columns in vector form using integers with a shared scale, enabling basic arithmetic through integer instructions under stated constraints.
- IMCV automatically selects lightweight compression methods, including delta encoding, dictionary compression, RLE, LZ4, and ZSTD, based on data characteristics.
- Compression can improve performance for some queries by enabling late reading and operations directly on compressed vectors.
- 60% reduction in IMCV memory size was measured at 7.5 TB TPC-H, from approximately 10 TB uncompressed to approximately 4 TB compressed.End-to-end latency across all TPC-H queries showed negligible difference from the uncompressed configuration at this scale factor.
8 EVALUATION
GaussDB was evaluated on a 30 TB TPC-H deployment using 32 servers and 128 data nodes. Its composite score exceeded the next-best reference system by 40%, while query-level results highlighted strengths in local exchange, remote shuffle, Bloom filters, and near-FK detection.
- Evaluation setup: 32 servers deployed GaussDB at SF=30,000, with 128 DNs, 64 TB memory, and 5120 CPU cores.The evaluation used Kunpeng 920 servers with 160 cores, 2048 GB RAM, and 5 TB SSD storage interconnected through a UB network.
- TPC-H results: QphH@30TB = 39,508,107, 40% better than Hologres, the next-best reference system.GaussDB achieved 7716 QphH@30TB/vCPU versus Hologres at 2533 QphH@30TB/vCPU; Exasol remained highest at 9837 QphH@30TB/vCPU.
- TPC-H results: The evaluation reported a relatively higher percentage improvement in the power test than in the throughput test.The authors speculate that fewer vCPUs and less memory than Hologres may have constrained the deployment relatively more under high throughput-test concurrency.
- TPC-H results: GaussDB had the best performance on most power-test queries and generally matched the best-performing system on the remainder.The reported strengths include query types using local exchange cache, requiring remote shuffle or many Bloom filters, and benefiting from near-FK determination.
- TPC-H results: These query-level observations support the importance of GaussDB’s local exchange, scalable shuffle, distributed Bloom-filter support, and near-FK improvements.The evaluation connects strong query performance with specific analytical query-processing enhancements.
9 CONCLUSION
GaussDB’s analytical query-processing improvements enabled elite TPC-H performance at scale. The conclusion presents the system as an option for enterprises seeking reliable and performant HTAP database capabilities.
- 9 CONCLUSION: GaussDB’s pipeline engine, scalable shuffle, distributed cost-based Bloom-filter support, and near-FK detection enabled elite TPC-H performance at scale.The conclusion groups these among the analytical query-processing improvements made to GaussDB.