PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Training Efficiency of Deep Learning Recommendation Models at Scale

Bilge Acun, Matthew Murphy, Xiaodong Wang, Jade Nie, Carole-Jean Wu, Kim Hazelwood

arXiv:2011.05497v1cs.ARcs.LG

TL;DR

Recommendation models combine compute-intensive work with embedding tables whose memory demands and model-dependent configurations complicate GPU training. The paper characterizes production-scale training across model configurations and systems, finding that efficiency gains depend on the model and that Zion helps when embedding tables exceed one Big Basin's capacity.

  • Problem

    Recommendation models have large embedding-table capacity and bandwidth requirements, and their diverse configurations create varying system-resource demands that complicate choosing efficient training hardware.

    Source: §I · pp. 1–2

  • Method

    The paper characterizes Facebook production recommendation models across configurations and training systems, including embedding-table placement and system interactions.

    Source: §I · pp. 1–2, §VIII · p. 12

  • Results

    Big Basin training efficiency improves by 4.3x for M1prod and 2.8x for M2prod versus baseline production CPU systems, while M3prod scales poorly; Zion helps when tables exceed one Big Basin.

    Source: §I · p. 2

  • Takeaways & Limitations

    The production-scale characterization can guide training-infrastructure design as embedding tables grow into multiple terabytes and high throughput must be balanced with model quality.

    Source: §VIII · p. 12

Abstract

from arXiv · show

The use of GPUs has proliferated for machine learning workflows and is now considered mainstream for many deep learning models. Meanwhile, when training state-of-the-art personal recommendation models, which consume the highest number of compute cycles at our large-scale datacenters, the use of GPUs came with various challenges due to having both compute-intensive and memory-intensive components. GPU performance and efficiency of these recommendation models are largely affected by model architecture configurations such as dense and sparse features, MLP dimensions. Furthermore, these models often contain large embedding tables that do not fit into limited GPU memory. The goal of this paper is to explain the intricacies of using GPUs for training recommendation models, factors affecting hardware efficiency at scale, and learnings from a new scale-up GPU server design, Zion.

I. INTRODUCTION

The introduction frames recommendation training as a heterogeneous-systems problem: embedding-table capacity and irregular accesses constrain GPU use, while model configurations produce different resource demands. The paper characterizes these differences across CPU, Big Basin, and Zion and reports model-dependent gains from hardware and embedding placement.

I. INTRODUCTION

  • The paper enables production-scale training on Big Basin and prototypes Zion, implementing embedding-placement strategies and characterizing how model configurations interact with system architectures.The characterization identifies a broad design space and performance-optimization opportunities across CPU, memory, and networking utilization.Source: §I · pp. 1–2
  • Throughput generally rises from dual-socket CPUs to Big Basin to Zion, but M3prod scales less because its embedding tables require more memory.Embedding placement also significantly affects throughput; M1prod and M2prod use GPU memory on Big Basin, while M3prod uses remote CPU memory and Zion uses system memory.Source: §I · pp. 1–2
  • Recommendation models' large embedding-table capacity and bandwidth requirements help explain why most are trained on CPU servers rather than 8-GPU systems.The paper notes that embedding tables have grown from tens of GBs to TBs across the industry.Source: §I · p. 1
  • Recommendation models with similar overall structure can vary widely in resource use, and the best hardware depends on features, embedding sizes, interaction types, and MLP dimensions.Analysis of over five hundred training runs found varied CPU, memory-capacity, memory-bandwidth, and network-bandwidth utilization.Source: §I · p. 2
  • Embedding tables impose both large memory-capacity demands and irregular vector-access bottlenecks, and tables that do not fit on one Big Basin benefit from Zion's 2 TB system memory and 1 TB/s memory bandwidth.As model sizes grow into multiple terabytes, the paper calls for solutions spanning compute, networking, and storage.Source: §I · p. 2
  • Big Basin training efficiency improves by 4.3x for M1prod and 2.8x for M2prod versus baseline production CPU systems, while M3prod scales poorly.M3prod has large embedding tables and more frequent embedding-vector lookups.Source: §I · p. 2

II. BACKGROUND ON ML TRAINING AT FACEBOOK

Facebook’s ML pipeline moves data through preprocessing, training, and inference. Recommendation workloads are trained especially frequently, and their training runs grew rapidly.

A. Machine Learning Training at Facebook

B. Training Systems for Deep Learning Recommendation

Recommendation workloads show variable resource utilization across runs, even with the same model type and system configuration. Their training systems combine data parallelism for MLP parameters with model parallelism for large embedding tables, motivating GPU and distributed-system design choices.

B. Training Systems for Deep Learning Recommendation

  • Recommendation models combine MLP layers for continuous features with embedding tables for categorical features, and their configurations affect training efficiency.Figure 3 highlights configurable model components that affect efficiency.Source: B. Training Systems for Deep Learning… · p. 3
  • Facebook’s production training system uses data parallelism for MLP parameters and partitions embedding tables across sparse parameter servers.Trainers use mini-batches and update dense parameters with a center parameter server; embedding tables are not directly replicated because of their memory requirements.Source: B. Training Systems for Deep Learning… · p. 3
  • CPU training offers memory capacity for recommendation models, but leaves accelerator parallelism unused; Big Basin GPU training requires embedding-placement optimization.Big Basin has eight NVIDIA Tesla GPUs and was originally designed for non-recommendation AI workloads.Source: B. Training Systems for Deep Learning… · p. 3

III. OVERVIEW OF TRAINING RECOMMENDATION MODEL ARCHITECTURES AND PARAMETERS

  • Parameter servers show a wider utilization distribution, with a lower mean and longer tail than trainer servers.The comparison covers runs of the same model type and system configuration; model architecture may partly explain the variability.Source: §III · pp. 3–4, Figure 5 · p. 4
  • The paper excludes learning rate, warm-up iterations, and optimizer choice because they affect model quality but have minor or no performance effect.It focuses instead on commonly configured architecture components that affect hardware efficiency.Source: §III · p. 4

A. Model Architecture Configurations

Recommendation-model efficiency depends on dense and sparse feature configuration, embedding-table characteristics, feature interactions, MLP dimensions, batch size, and gradient synchronization. Sparse-table sizing and access patterns vary across production models and influence resource placement and balance.

A. Model Architecture Configurations

  • Dense features are scalar inputs processed by dense layers, while sparse categorical features map indices to embeddings using feature-specific hash sizes.Sparse-feature embedding parameters scale with embedding dimension and hash size.Source: 1) Feature Selection · p. 4
  • Sparse features can share embedding tables to reduce model size when they are semantically similar and can use a shared hash size.Dense and sparse embedding outputs can be concatenated or combined through pairwise dot products to capture feature interactions.Source: 2) Embedding Tables · p. 4, 3) Feature Interaction · p. 5
  • Embedding-table hash sizes vary by sparse feature and production model, and lower sizes can reduce table size while risking accuracy degradation from collisions.Figure 6 plots hash size against mean feature length for M1prod, M2prod, and M3prod.Source: 2) Embedding Tables · p. 4, Figure 6 · p. 5
  • Feature-length distributions are power-law-like, so a few tables receive many more accesses; access frequency does not always track table size.Uneven access can imbalance servers if tables are not carefully partitioned, while the characterization suggests caching and quantization as optimization opportunities.Source: 2) Embedding Tables · p. 5, Figure 7 · p. 5
  • Recommendation models tune the depth and width of bottom and top MLP stacks, with the bottom stack reducing dense inputs to a denser vector.The top stack follows feature interaction.Source: 4) MLP Dimensions · p. 5
  • Larger GPU mini-batches can improve throughput roughly linearly up to a limit, while gradient synchronization choices affect both model quality and scaling performance.Distributed training may use synchronous or asynchronous gradient synchronization.Source: 5) Batch Size · p. 5, 6) Gradient Synchronization Method · pp. 5–6

IV. HARDWARE AND SYSTEM CONFIGURATIONS

Hardware and system configuration follows model input and architecture choices. The most efficient setup can change as those configurations change.

IV. HARDWARE AND SYSTEM CONFIGURATIONS

  • Hardware and system configuration are chosen after model input and architecture configurations.The paper describes hardware platforms in heterogeneous datacenters and their major system configurations.Source: §IV · p. 6
  • A workflow often has a de-facto hardware type, but changing model configurations can change the most efficient hardware and system setup.Source: §IV · p. 6
  • The section describes hardware platforms in heterogeneous datacenters and outlines their major system configurations.Source: §IV · p. 6

A. Hardware Platforms

The paper compares CPU, Big Basin, and Zion hardware and explains how embedding placement and server counts are configured to meet training throughput and memory needs. Zion provides substantially more system memory and bandwidth than Big Basin, while placement choices trade off fit, communication, and CPU work.

A. Hardware Platforms

  • Zion combines eight CPU sockets with eight GPUs, about 2 TB of system memory, and about 1 TB/s memory bandwidth; Big Basin has eight GPUs and 256 GB of DRAM.Big Basin's V100 GPUs connect through NVLink; each provides 15.7 teraflops of single-precision compute and 900 GB/s HBM2 bandwidth.Source: A. Hardware Platforms · p. 6

B. System Configurations

  • Embedding tables can reside in GPU memory, GPU-server system memory, remote CPU-server memory, or a hybrid of system and GPU memory.GPU placement can offload model operations and reduce CPU use and CPU-GPU copies, while remote placement adds network latency and CPU send/receive work.Source: 1) Embedding Table Placement · pp. 6–7
  • Trainer, parameter-server, and reader counts are set according to training-throughput and embedding-memory requirements, and can vary between runs.Reader servers are scaled so data reading does not bottleneck training; more performant hardware may use more readers.Source: 2) Number of Servers · p. 7
  • More than 40% of CPU training workflows use the same trainer count, while parameter-server counts vary widely as feature experiments change memory needs.Increasing trainer count generally raises training speedup approximately linearly up to a point, but extra trainers can reduce model quality during parameter synchronization.Source: 2) Number of Servers · p. 7

V. EFFICIENCY OF MODEL SETUP CONFIGURATIONS

Because the possible model and system architecture combinations are exponential, the paper studies a subset of important configurations systematically. It builds a customizable test suite from basic recommendation-model components to compare throughput behavior on CPU and GPU systems.

V. EFFICIENCY OF MODEL SETUP CONFIGURATIONS

  • The study parameterizes a subset of important model configurations because the full model-and-system combination space is exponential.The goal is to understand effects on throughput and compare CPU and GPU behavior.Source: §V · p. 7
  • The test suite varies dense features from 64 to 4096 and sparse features from 4 to 128, a range described as representative of Facebook recommendation models.It fixes hash size across sparse features and truncates lookups per table to 32 to limit noise and outliers.Source: §V · p. 7
  • The customizable suite is built from basic components of recommendation models to study configuration effects systematically.Source: §V · p. 7

A. Number of Sparse and Dense Features: As the number

Feature counts, batch size, and embedding hash size affect training throughput differently across CPU and GPU setups. The reported comparisons show higher GPU throughput across tested feature configurations, but power efficiency and throughput responses depend on the configuration.

A. Number of Sparse and Dense Features: As the number

  • As dense and sparse feature counts rise, training throughput falls; Big Basin has higher throughput, though its performance-per-watt is lower in some cases.The feature-scaling comparison uses fixed batch sizes of 200 on CPU and 1600 on GPU, hash size 100000, and MLP dimensions 5123.Source: A. Number of Sparse and Dense… · pp. 7–8, Figure 10 · p. 8
  • GPU throughput exceeds CPU throughput across the tested feature combinations, but GPU power efficiency is highest for models with more dense features.The Big Basin server's power capacity requirement is 7.3 times that of the dual-socket CPU server.Source: A. Number of Sparse and Dense… · p. 8, Figure 10 · p. 8

B. Batch size: There exists a different throughput-optimal

  • Batch size has a configuration-dependent throughput optimum; on GPUs, larger batches reduce CUDA-call overhead until communication costs offset compute benefits.Throughput saturates beyond a point, while further batch-size increases can affect model quality, so batch size must be selected for each model configuration.Source: B. Batch size: There exists a… · p. 8, Figure 11 · p. 8
  • As hash size grows, Big Basin throughput drops significantly because more GPUs are needed to hold embeddings, increasing inter-GPU communication.The comparison uses one CPU parameter server with 256 GB memory and one Big Basin server with 256 GB GPU memory.Source: B. Batch size: There exists a… · p. 9, Figure 12 · p. 9

D. MLP Dimensions: Increasing the width of the MLP layers

The paper examines throughput as MLP dimensions increase on CPU and GPU systems. Sparse-feature operations dominate at smaller dimensions, while CPU throughput falls more than GPU throughput as dimensions grow.

D. MLP Dimensions: Increasing the width of the MLP layers

VI. CASE STUDY WITH PRODUCTION-SCALE MODELS

The case study compares three production recommendation models across CPU and Big Basin GPU setups, with performance depending on embedding-table fit and placement. Big Basin improves throughput and power efficiency for models whose embedding tables fit, while M3prod remains bottlenecked when they do not.

VI. CASE STUDY WITH PRODUCTION-SCALE MODELS

  • The evaluated models differ in sparse and dense feature counts and embedding-table size, with M3prod tables in the hundreds of GBs rather than tens of GBs.The production models also capture varying lookups per table and diverse hash sizes.Source: §VI · p. 9

A. CPU vs Big Basin GPU Performance: When embedding

  • M1prod achieved 2.25 times higher throughput on GPU than its production CPU setup and was 4.3 times more power efficient; M2prod reached 0.85 times CPU throughput and 2.8 times higher power efficiency.For M3prod, Big Basin reached 0.67x the throughput of the production setup on 8 CPU trainers.Source: A. CPU vs Big Basin GPU Performance:… · p. 10, Table III · p. 10
  • When embedding tables exceed Big Basin GPU memory, remote CPU parameter servers leave CPU resources and data copies on the GPU server as bottlenecks.The authors could not test distributing tables across multiple Big Basins because fast inter-node GPU-GPU communication was unavailable.Source: A. CPU vs Big Basin GPU Performance:… · p. 10

B. Big Basin vs Zion GPU Performance: Zion’s large mem-

The section compares embedding placements on Big Basin and Zion and examines accuracy as GPU batch sizes and hyperparameters are tuned. Zion’s advantage is strongest when embedding tables exceed a single GPU server’s memory, while accuracy evaluation and multi-server options have stated constraints.

B. Big Basin vs Zion GPU Performance: Zion’s large mem-

  • With system-memory placement, Zion performed best; GPU-memory placement favored Big Basin, while absent direct GPU-GPU communication forced Zion’s cross-GPU traffic through CPUs.Remote system-memory placement did not surpass other approaches, with lookup latency and GPU-server CPU resources becoming bottlenecks.Source: B. Big Basin vs Zion GPU Performance:… · p. 10, Figure 14 · p. 10
  • For RM-3, analytical modeling found Zion several orders of magnitude more efficient than multiple Big Basins with embedding tables placed in GPU memory.Zion’s roughly 2 TB system memory and roughly 1 TB/s bandwidth support large tables and fast lookups.Source: B. Big Basin vs Zion GPU Performance:… · p. 10
  • Despite manual tuning, GPU accuracy gaps grew with batch size, and 0.1-0.2% model-loss regression may be unacceptable for recommendation models.Bayesian hyperparameter retuning yielded higher accuracy, measured as lower negative entropy, for M1prod (-0.2%) and M2prod (-0.1%).Source: B. Big Basin vs Zion GPU Performance:… · p. 11, Figure 15 · p. 11
  • Accuracy was not evaluated for M3prod because its throughput was low and running a hyperparameter search sweep was costly.Tuning to minimize loss took around a week using high volumes of training data.Source: B. Big Basin vs Zion GPU Performance:… · p. 11

VII. RELATED WORK

Prior work reports accelerator gains for recommendation training, but industry-scale recommendation models and their system challenges remain under-studied. The paper characterizes Facebook production models and training systems, concluding that its design-space analysis can guide training-infrastructure design as embedding-table capacity grows.

VII. RELATED WORK

  • Prior accelerator examples include Google training 14 times faster on TPUs, Baidu training a 10 TB model twice as fast with a hierarchical GPU parameter server [58], and MLPerf DLRM training results on TPUv4 and DGX-A100,.The paper notes that MLPerf-DLRM represents a medium-scale click-through-rate model, while production model architectures and parameters vary.Source: §VII · p. 11
  • The paper identifies industry-scale recommendation training as under-studied and notes that near-memory designs RecNMP and TensorDIMM [30] are not optimized for gradient aggregation.Their benefits therefore do not translate well into training performance improvement.Source: §VII · p. 11

VIII. CONCLUSION

  • The paper characterizes Facebook production-model properties, training throughput, model quality, and techniques for enabling recommendation training on Big Basin and Zion.Its conclusion highlights effects of dense and sparse features, batch sizes, embedding hash sizes, and MLP dimensions on throughput.Source: §VII · pp. 11–12, §VIII · p. 12
  • As embedding-table capacity grows into multiple terabytes, the paper presents Zion and its production-scale design-space analysis as guidance for training-infrastructure design.The conclusion frames high throughput alongside high accuracy as a driver for specialized training hardware.Source: §VIII · p. 12
Loading 2011.05497v1…