PaperlayerView exact revision on arXiv

Source-linked AI summary

Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems

Weijie Zhao, Deping Xie, Ronglai Jia, Yulei Qian, Ruiquan Ding, Mingming Sun, Ping Li

arXiv:2003.05622v1cs.DCcs.LGstat.ML

TL;DR

Massive-scale advertising models have sparse inputs but terabyte-scale parameters that exceed GPU and CPU memory. The paper introduces a distributed hierarchical parameter server spanning GPU HBM, CPU memory, and SSDs, with GPU-contained training computation. On real CTR workloads, the system was faster and more cost-effective than MPI-cluster solutions while retaining comparable accuracy.

  • Problem

    Terabyte-scale sparse-input advertising models do not fit in GPU or CPU memory, while MPI-based training incurs substantial cluster, communication, synchronization, and maintenance costs.

  • Method

    A distributed hierarchical parameter server uses GPU HBM, CPU main memory, and SSDs, with pipelined transfers and GPU-contained training computation.

  • Results

    A 4-node system was 1.8-4.8X faster than the MPI solution, with a 4.4-9.0X better price-performance ratio and relative accuracy difference within 0.1%.

  • Takeaways & Limitations

    Hierarchical GPU parameter storage enables scalable training of massive CTR models without requiring the large CPU clusters used by the MPI solution.

  • Takeaways & Limitations

    The paper focuses on CTR models in the final stage of Baidu’s ads system, while earlier-stage ad recall using ANNS and MIPS is separate.

Abstract

from arXiv · show

Neural networks of ads systems usually take input from multiple resources, e.g., query-ad relevance, ad features and user portraits. These inputs are encoded into one-hot or multi-hot binary features, with typically only a tiny fraction of nonzero feature values per example. Deep learning models in online advertising industries can have terabyte-scale parameters that do not fit in the GPU memory nor the CPU main memory on a computing node. For example, a sponsored online advertising system can contain more than $10^{11}$ sparse features, making the neural network a massive model with around 10 TB parameters. In this paper, we introduce a distributed GPU hierarchical parameter server for massive scale deep learning ads systems. We propose a hierarchical workflow that utilizes GPU High-Bandwidth Memory, CPU main memory and SSD as 3-layer hierarchical storage. All the neural network training computations are contained in GPUs. Extensive experiments on real-world data confirm the effectiveness and the scalability of the proposed system. A 4-node hierarchical GPU parameter server can train a model more than 2X faster than a 150-node in-memory distributed parameter server in an MPI cluster. In addition, the price-performance ratio of our proposed system is 4-9 times better than an MPI-cluster solution.

1 INTRODUCTION

Massive-scale CTR models combine sparse, high-dimensional inputs with parameters that exceed practical GPU and CPU memory capacities. The paper introduces a hierarchical GPU parameter-server architecture using distributed GPU memory, CPU memory, and SSDs to train these models efficiently.

  • Motivation: CTR prediction uses ultra-high-dimensional sparse inputs, with only a tiny fraction of features nonzero per example.The model references parameters associated with the active sparse features.
  • Evaluation context: The paper evaluates the architecture on real CTR systems and positions it as an alternative to costly MPI-based training clusters.The introduction frames communication, synchronization, hardware, and maintenance costs as limitations of the existing approach.
  • Motivation: Massive CTR models can exceed GPU memory, making direct GPU training impractical because of limited capacity and CPU-GPU transfer overhead.The paper targets models whose parameter volume exceeds the aggregate GPU HBM available in practical clusters.
  • Approach: The proposed parameter server distributes working parameters across GPUs and uses direct inter-GPU communication to reduce CPU-GPU data-transfer overhead.A distributed hash table spans multiple GPUs, while hierarchical storage extends capacity beyond GPU and main memory.
  • Approach: A four-stage pipeline overlaps network communication, SSD I/O, CPU-GPU transfers, and GPU computation to hide data-loading latency.The design addresses slow data movement and storage relative to GPU training speed.
  • Contributions: The hierarchical system uses GPU HBM, CPU main memory, and SSDs to store working, out-of-GPU-memory, and out-of-main-memory parameters.The paper presents the architecture as a distributed GPU parameter server for terabyte-scale sparse-input deep learning.

2 PRIOR EFFORT: HASHING METHODS FOR REDUCING CTR MODELS

The paper’s prior hashing effort sought to reduce CTR model size while preserving accuracy, but results differed sharply between image-search and web-search ads. OP+OSRP was efficient for sparse binary data, yet hashing remained unacceptable for the main web-search CTR model.

  • OP+OSRP method: OP+OSRP permutes features, divides them into k bins, applies independent sign projections, and expands each sign into binary features of dimension 2k.The method preserves binary inputs and is designed for sparse data.
  • OP+OSRP method: OP+OSRP is efficient for sparse data because it essentially touches each nonzero entry once.
  • Web-search results: Hash+DNN could not be proposed for web-search CTR models because it reduced the accuracy of current DNN models and consequently affected company revenue.
  • Implication: The authors therefore motivate a lossless distributed hierarchical GPU parameter server instead of relying on hashing to shrink massive CTR models.

3 DISTRIBUTED HIERARCHICAL PARAMETER SERVER OVERVIEW

The proposed system organizes parameter storage across HBM-PS, MEM-PS, and SSD-PS, while a four-stage pipeline overlaps network, CPU, SSD, and GPU work. Training uses data parallelism and synchronizes GPU parameters after each minibatch.

  • Architecture: The architecture contains three components: HBM-PS, MEM-PS, and SSD-PS.
  • Workflow: Each node processes its own HDFS training batches in a data-parallel workflow and prepares local and remote referenced parameters for its GPUs.
  • Workflow: After each minibatch, GPU workers pull required parameters from HBM-PS, train, and update referenced parameters on other GPUs.Inter-GPU synchronization is performed after each minibatch to eliminate parameter staleness.
  • Storage hierarchy: HBM-PS provides direct GPU access to working parameters, while MEM-PS and SSD-PS accommodate parameters beyond GPU capacity.This hierarchy supports terabyte-scale parameters distributed across GPUs, memory, and SSDs.
  • Pipeline: A four-stage pipeline overlaps data transfer, parameter partitioning, materialized data loading or dumping, and neural-network training across network, CPU, SSD, and GPU resources.Each stage uses a prefetch queue and worker thread to hide latency.

4 HBM-PS

HBM-PS stores working parameters in distributed GPU memory and exposes them through a multi-GPU hash table. GPU communication and RDMA-based synchronization enable parameter updates without CPU-mediated transfers.

  • HBM-PS role: HBM-PS stores working parameters across GPU HBMs and lets GPU workers request and update them without GPU-to-CPU memory transfers.The design targets higher training throughput while acknowledging that HBM capacity is limited and expensive.
  • Distributed hash table: The multi-GPU distributed hash table partitions referenced batch parameters across local GPU hash tables and supports insert, get, and accumulate operations.
  • Parameter updates: The accumulate operation partitions key-value updates by GPU and asynchronously sends each partition to its owning GPU for accumulation.It is called after backpropagation to update parameters referenced during training.
  • Cross-node sharing: Shared working parameters can be referenced and updated by multiple nodes because nodes process separate training batches.
  • Synchronization: Inter-node synchronization uses RDMA-based all-reduce so GPUs receive and reduce updates without CPU and operating-system memory involvement.Synchronization is required to guarantee model convergence.

5 MEM-PS

MEM-PS gathers referenced parameters from local SSDs and remote memory servers, caches parameters to reduce SSD I/O, and applies GPU updates in memory. It partitions parameters by key using a fixed node mapping.

  • Parameter gathering: MEM-PS identifies referenced parameters and communicates with local SSD-PS and remote MEM-PS to gather them.A parameter cache reduces SSD I/O.
  • Parameter partitioning: Parameters are partitioned by a predefined parameter-to-node mapping using modulo hashing; remote parameters are pulled from other MEM-PS instances.Local parameters are fetched from SSDs.
  • Parameter updates: After GPU computation, MEM-PS pulls updated parameters from HBM-PS and applies the changes in memory.Working parameters are pinned in memory, and remote MEM-PS instances pull updates from their own GPUs because HBM-PS keeps remote parameters synchronized.

6 SSD-PS

SSD-PS stores parameters that exceed GPU and main-memory capacity on SSDs, using file-level organization and batched file I/O to maintain them efficiently.

  • SSD-PS maintains materialized parameters that are out of main memory on SSDs.
  • File-level organization keeps a parameter-to-file mapping in main memory while storing parameter collections in files.File descriptors require much less memory than parameter values, and each node stores only a parameter shard.
  • SSD-PS gathers requested keys and reads the entire parameter file containing them as its SSD I/O unit.
  • Evicted parameters are written as new chunked files instead of being updated in place, avoiding random disk writes.

7 EXPERIMENTAL EVALUATION

Experiments on five real-world CTR models evaluate performance, scalability, execution-time components, and accuracy against a production MPI-cluster baseline. The 4-node hierarchical GPU parameter server is faster and more cost-effective while maintaining nearly identical accuracy.

  • Experimental setup: The evaluation uses four GPU nodes and measures training execution time and prediction accuracy against a distributed in-memory MPI parameter server.Training data come from search-engine click-history logs and models are evaluated online through A/B testing.
  • Experimental setup: Five real-world CTR models range from 300 GB to 10 TB, with 75–150 CPU-only MPI nodes used as production baselines.Sparse parameters range from 8 × 10^9 to 10^11, while dense parameters are 4–5 orders of magnitude smaller.
  • Comparison with MPI solution: 1.8–4.8X faster training is achieved by HPS-4 than by the MPI solution across all five models.The comparison uses the production MPI-cluster solution as its baseline.
  • Comparison with MPI solution: 4.4–9.0X better price-performance is achieved by the proposed system than by the MPI cluster.One GPU node costs roughly as much as 10 CPU-only MPI nodes, while the baseline uses 75–150 nodes.
  • Comparison with MPI solution: Relative AUC differences across all five models remain within 0.1%, indicating lossless training relative to the MPI solution.For Model C, relative AUC loss is below 0.01%; the other four models are slightly better than MPI-128.
  • Execution-time analysis: The pipelined batch time is dominated by the slowest stage among reading examples, parameter pull/push, and GPU DNN training.HBM-PS pull/push time follows the number of non-zero input features, while training time follows dense-parameter size.
  • SSD-PS analysis: Model E’s cache hit rate reaches 46% after 40 training batches and then remains stable.The cache begins cold and rapidly captures frequently visited parameters during the first 10 batches.
  • Scalability: A speedup of 3.57 out of 4 is obtained on four nodes, with sub-linear scaling attributed to increased network communication.

8 RELATED WORK

Prior work spans CTR models, parameter-server synchronization, and hardware-aware architectures for communication and memory bottlenecks. This paper extends that direction to models too large for CPU memory through a three-layer GPU training design.

  • CTR prediction models: Deep CTR models use embedding layers and factorization-machine components to model sparse-feature correlations.
  • Parameter servers: Parameter-server research studies synchronization patterns such as BSP and architectures tailored to different hardware and network conditions.
  • Parameter servers: GeePS addresses limited GPU memory with a two-layer architecture that moves required parameters between GPU memory and CPU memory.
  • Parameter servers: The proposed system distributes a hash table across GPU HBM for direct GPU peer-to-peer communication, reducing CPU-GPU movement and synchronization overhead relative to GeePS.
  • Parameter servers: A three-layer architecture using SSD-PS, MEM-PS, and HBM-PS supports training a 10 TB model when parameters exceed cluster CPU memory.

9 CONCLUSIONS

The paper presents a hierarchical GPU parameter server that places out-of-GPU-memory parameters in main memory and out-of-main-memory parameters on SSD. Experiments on real-world sponsored-advertising CTR models report substantial speedups over MPI-based systems.

  • Conclusions: Model hashing is not fully applicable to accuracy-crucial advertising applications, while massive models may not fit in CPU or GPU memory.
  • Conclusions: The hierarchical design uses main memory for out-of-GPU-memory parameters and SSDs for out-of-main-memory parameters.
  • Conclusions: Experiments on 5 real-world CTR prediction models confirm the proposed system’s effectiveness and scalability.
  • Conclusions: 1.8-4.8X faster than the MPI solution in the production environment is achieved by the 4-node distributed hierarchical GPU parameter server.
  • Conclusions: The system is being integrated with PaddlePaddle as PaddleBox.

C HBM-PS IMPLEMENTATION

HBM-PS partitions parameters across GPUs and uses RDMA plus staged all-reduce communication to reduce data movement. The implementation also pins or partitions dense parameters to keep frequently referenced values available during training.

  • Partition policy: Modulo hashing maps parameter keys to GPU ids with balanced partitioning and constant memory space in general cases.
  • GPU Communication mechanism: The communication baseline copies data from sender HBM through CPU memory and the network before transferring it into receiver HBM.
  • GPU Communication mechanism: RDMA enables zero-copy transfer directly between device memories, eliminating CPU and operating-system buffer involvement.
  • All-reduce communication: All-reduce synchronizes parameter updates through parallel inter-node exchanges followed by intra-node GPU tree communication.
  • Dense parameters: Dense parameters can be pinned in every GPU’s HBM, or sharded when HBM cannot replicate them.

D MEM-PS IMPLEMENTATION

MEM-PS reduces SSD I/O by caching parameters according to both recency and frequency of use.

  • Cache policy: The cache combines Least Recently Used and Least Frequently Used policies to retain recently and frequently accessed parameters in memory.

E SSD-PS IMPLEMENTATION

SSD-PS manages parameter loading, dumping, and storage growth through parameter-to-file mappings, sequential writes, and background compaction. Its file-size choice balances SSD I/O bandwidth against unnecessary parameter reads, while the paper distinguishes its caching and storage design from prior systems.

  • Parameter loading: SSD-PS loads requested parameters by consulting MEM-PS and a parameter-to-file mapping to locate files on SSD.Because entire files must be read, file organization affects read efficiency.
  • Parameter loading: File size trades off SSD I/O bandwidth against unnecessary parameter reading: larger files increase extra reads, whereas smaller files underutilize bandwidth.The implementation tunes file size for optimal performance.
  • Parameter dumping: Evicted parameters are chunked into new SSD files and written sequentially instead of being updated in place through random disk writes.The parameter-to-file mapping is updated after the new files are written.
  • File compaction: A background worker triggers leveled file compaction at a preset disk-usage threshold, merging old files that contain many stale values.Compaction reduces SSD usage while running in the background.
  • Related designs: The paper positions SSD-PS alongside prior cache-management algorithms and SSD key-value stores using in-memory indexes with append-only SSD structures.The related systems include cache policies based on recency and frequency, plus several SSD key-value-store designs.
Loading 2003.05622v1…