Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems

Motivation

• CTR prediction uses ultra-high-dimensional sparse inputs, with only a tiny fraction of features nonzero per example.The model references parameters associated with the active sparse features.Source: §1 · p. 1, Figure 1 · p. 1

Evaluation context

• The paper evaluates the architecture on real CTR systems and positions it as an alternative to costly MPI-based training clusters.The introduction frames communication, synchronization, hardware, and maintenance costs as limitations of the existing approach.Source: §1.1 · pp. 1–2, §1.2 · p. 2

Motivation

• Massive CTR models can exceed GPU memory, making direct GPU training impractical because of limited capacity and CPU-GPU transfer overhead.The paper targets models whose parameter volume exceeds the aggregate GPU HBM available in practical clusters.Source: §1.3 · p. 2

Approach

• The proposed parameter server distributes working parameters across GPUs and uses direct inter-GPU communication to reduce CPU-GPU data-transfer overhead.A distributed hash table spans multiple GPUs, while hierarchical storage extends capacity beyond GPU and main memory.Source: §1.3 · p. 2
• A four-stage pipeline overlaps network communication, SSD I/O, CPU-GPU transfers, and GPU computation to hide data-loading latency.The design addresses slow data movement and storage relative to GPU training speed.Source: §1.3 · p. 2

Contributions

• The hierarchical system uses GPU HBM, CPU main memory, and SSDs to store working, out-of-GPU-memory, and out-of-main-memory parameters.The paper presents the architecture as a distributed GPU parameter server for terabyte-scale sparse-input deep learning.Source: §1.3 · pp. 2–3

OP+OSRP method

• OP+OSRP permutes features, divides them into k bins, applies independent sign projections, and expands each sign into binary features of dimension 2k.The method preserves binary inputs and is designed for sparse data.Source: §2 · p. 3
• OP+OSRP is efficient for sparse data because it essentially touches each nonzero entry once.Source: §2 · p. 3

Web-search results

• Hash+DNN could not be proposed for web-search CTR models because it reduced the accuracy of current DNN models and consequently affected company revenue.Source: §2 · p. 4

Implication

• The authors therefore motivate a lossless distributed hierarchical GPU parameter server instead of relying on hashing to shrink massive CTR models.Source: §2 · p. 4

Architecture

• The architecture contains three components: HBM-PS, MEM-PS, and SSD-PS.Source: §3 · p. 4

Workflow

• Each node processes its own HDFS training batches in a data-parallel workflow and prepares local and remote referenced parameters for its GPUs.Source: §3 · pp. 4–5
• After each minibatch, GPU workers pull required parameters from HBM-PS, train, and update referenced parameters on other GPUs.Inter-GPU synchronization is performed after each minibatch to eliminate parameter staleness.Source: §3 · p. 5

Storage hierarchy

• HBM-PS provides direct GPU access to working parameters, while MEM-PS and SSD-PS accommodate parameters beyond GPU capacity.This hierarchy supports terabyte-scale parameters distributed across GPUs, memory, and SSDs.Source: §3 · p. 5

Pipeline

• A four-stage pipeline overlaps data transfer, parameter partitioning, materialized data loading or dumping, and neural-network training across network, CPU, SSD, and GPU resources.Each stage uses a prefetch queue and worker thread to hide latency.Source: §3 · p. 5

HBM-PS role

• HBM-PS stores working parameters across GPU HBMs and lets GPU workers request and update them without GPU-to-CPU memory transfers.The design targets higher training throughput while acknowledging that HBM capacity is limited and expensive.Source: §4 · p. 5

Distributed hash table

• The multi-GPU distributed hash table partitions referenced batch parameters across local GPU hash tables and supports insert, get, and accumulate operations.Source: §4.1 · pp. 5–6

Parameter updates

• The accumulate operation partitions key-value updates by GPU and asynchronously sends each partition to its owning GPU for accumulation.It is called after backpropagation to update parameters referenced during training.Source: §4.1 · p. 6

Cross-node sharing

• Shared working parameters can be referenced and updated by multiple nodes because nodes process separate training batches.Source: §4.2 · p. 6

Synchronization

• Inter-node synchronization uses RDMA-based all-reduce so GPUs receive and reduce updates without CPU and operating-system memory involvement.Synchronization is required to guarantee model convergence.Source: §4.2 · p. 6

Parameter gathering

• MEM-PS identifies referenced parameters and communicates with local SSD-PS and remote MEM-PS to gather them.A parameter cache reduces SSD I/O.Source: §5 · p. 6

Parameter partitioning

• Parameters are partitioned by a predefined parameter-to-node mapping using modulo hashing; remote parameters are pulled from other MEM-PS instances.Local parameters are fetched from SSDs.Source: §5 · p. 6

Parameter updates

• After GPU computation, MEM-PS pulls updated parameters from HBM-PS and applies the changes in memory.Working parameters are pinned in memory, and remote MEM-PS instances pull updates from their own GPUs because HBM-PS keeps remote parameters synchronized.Source: §5 · p. 6

• SSD-PS maintains materialized parameters that are out of main memory on SSDs.Source: §6 · p. 6
• File-level organization keeps a parameter-to-file mapping in main memory while storing parameter collections in files.File descriptors require much less memory than parameter values, and each node stores only a parameter shard.Source: §6 · p. 7
• SSD-PS gathers requested keys and reads the entire parameter file containing them as its SSD I/O unit.Source: §6 · p. 7
• Evicted parameters are written as new chunked files instead of being updated in place, avoiding random disk writes.Source: §6 · p. 7

Experimental setup

• The evaluation uses four GPU nodes and measures training execution time and prediction accuracy against a distributed in-memory MPI parameter server.Training data come from search-engine click-history logs and models are evaluated online through A/B testing.Source: §7 · p. 7, §7.1 · p. 7
• Five real-world CTR models range from 300 GB to 10 TB, with 75–150 CPU-only MPI nodes used as production baselines.Sparse parameters range from 8 × 10^9 to 10^11, while dense parameters are 4–5 orders of magnitude smaller.Source: §7 · p. 7

Comparison with MPI solution

• 1.8–4.8X faster training is achieved by HPS-4 than by the MPI solution across all five models.The comparison uses the production MPI-cluster solution as its baseline.Source: §7.1 · p. 7
• 4.4–9.0X better price-performance is achieved by the proposed system than by the MPI cluster.One GPU node costs roughly as much as 10 CPU-only MPI nodes, while the baseline uses 75–150 nodes.Source: §7.1 · pp. 7–8
• Relative AUC differences across all five models remain within 0.1%, indicating lossless training relative to the MPI solution.For Model C, relative AUC loss is below 0.01%; the other four models are slightly better than MPI-128.Source: §7.1 · p. 8

Execution-time analysis

• The pipelined batch time is dominated by the slowest stage among reading examples, parameter pull/push, and GPU DNN training.HBM-PS pull/push time follows the number of non-zero input features, while training time follows dense-parameter size.Source: §7.2 · p. 8, §7.3 · p. 8

SSD-PS analysis

• Model E’s cache hit rate reaches 46% after 40 training batches and then remains stable.The cache begins cold and rapidly captures frequently visited parameters during the first 10 batches.Source: §7.4 · p. 8

Scalability

• A speedup of 3.57 out of 4 is obtained on four nodes, with sub-linear scaling attributed to increased network communication.Source: §7.6 · p. 9

CTR prediction models

• Deep CTR models use embedding layers and factorization-machine components to model sparse-feature correlations.Source: §8 · pp. 9–10

Parameter servers

• Parameter-server research studies synchronization patterns such as BSP and architectures tailored to different hardware and network conditions.Source: §8 · p. 10
• GeePS addresses limited GPU memory with a two-layer architecture that moves required parameters between GPU memory and CPU memory.Source: §8 · p. 10
• The proposed system distributes a hash table across GPU HBM for direct GPU peer-to-peer communication, reducing CPU-GPU movement and synchronization overhead relative to GeePS.Source: §8 · p. 10
• A three-layer architecture using SSD-PS, MEM-PS, and HBM-PS supports training a 10 TB model when parameters exceed cluster CPU memory.Source: §8 · p. 10

Conclusions

• Model hashing is not fully applicable to accuracy-crucial advertising applications, while massive models may not fit in CPU or GPU memory.Source: §9 · p. 10
• The hierarchical design uses main memory for out-of-GPU-memory parameters and SSDs for out-of-main-memory parameters.Source: §9 · p. 10
• Experiments on 5 real-world CTR prediction models confirm the proposed system’s effectiveness and scalability.Source: §9 · p. 10
• 1.8-4.8X faster than the MPI solution in the production environment is achieved by the 4-node distributed hierarchical GPU parameter server.Source: §9 · p. 10
• The system is being integrated with PaddlePaddle as PaddleBox.Source: §9 · p. 10

Partition policy

• Modulo hashing maps parameter keys to GPU ids with balanced partitioning and constant memory space in general cases.Source: §C.1 · p. 14

GPU Communication mechanism

• The communication baseline copies data from sender HBM through CPU memory and the network before transferring it into receiver HBM.Source: §C.2 · p. 15
• RDMA enables zero-copy transfer directly between device memories, eliminating CPU and operating-system buffer involvement.Source: §C.2 · p. 15

All-reduce communication

• All-reduce synchronizes parameter updates through parallel inter-node exchanges followed by intra-node GPU tree communication.Source: §C.3 · pp. 15–16

Dense parameters

• Dense parameters can be pinned in every GPU’s HBM, or sharded when HBM cannot replicate them.Source: §C.4 · p. 16

Cache policy

• The cache combines Least Recently Used and Least Frequently Used policies to retain recently and frequently accessed parameters in memory.Source: D MEM-PS IMPLEMENTATION · p. 16

Parameter loading

• SSD-PS loads requested parameters by consulting MEM-PS and a parameter-to-file mapping to locate files on SSD.Because entire files must be read, file organization affects read efficiency.Source: E SSD-PS IMPLEMENTATION · p. 16
• File size trades off SSD I/O bandwidth against unnecessary parameter reading: larger files increase extra reads, whereas smaller files underutilize bandwidth.The implementation tunes file size for optimal performance.Source: E SSD-PS IMPLEMENTATION · p. 16

Parameter dumping

• Evicted parameters are chunked into new SSD files and written sequentially instead of being updated in place through random disk writes.The parameter-to-file mapping is updated after the new files are written.Source: E SSD-PS IMPLEMENTATION · p. 16

File compaction

• A background worker triggers leveled file compaction at a preset disk-usage threshold, merging old files that contain many stale values.Compaction reduces SSD usage while running in the background.Source: E SSD-PS IMPLEMENTATION · p. 16, Figure 10 · p. 17

Related designs

• The paper positions SSD-PS alongside prior cache-management algorithms and SSD key-value stores using in-memory indexes with append-only SSD structures.The related systems include cache policies based on recency and frequency, plus several SSD key-value-store designs.Source: F ADDITIONAL RELATED WORK · pp. 16–17