Source-linked AI summary
Deep Learning Recommendation Model for Personalization and Recommendation Systems
Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G. Azzolini, Dmytro Dzhulgakov, Andrey Mallevich, Ilia Cherniavskii, Yinghai Lu, Raghuraman Krishnamoorthi, Ansha Yu, Volodymyr Kondratenko, Stephanie Pereira, Xianjie Chen, Wenlin Chen, Vijay Rao, Bill Jia, Liang Xiong, Misha Smelyanskiy
TL;DR
Neural recommendation models must handle categorical features, yet this network class remains comparatively understudied. The paper introduces DLRM and a specialized hybrid parallelization scheme, then evaluates its accuracy and execution performance while releasing implementations for experimentation and benchmarking.
Problem
Recommendation networks handle categorical features differently from other deep learning networks and have received relatively little academic attention.
Method
DLRM combines embeddings, multilayer perceptrons, explicit second-order feature interactions, and hybrid model parallelism for embeddings with data parallelism for MLPs.
Results
DLRM achieves slightly higher training and validation accuracy than the compared model and runs in around 256 seconds on CPU and 62 seconds on GPU in Caffe2.
Takeaways & Limitations
The open-source DLRM description and implementations support further algorithmic experimentation, system co-design, and benchmarking of recommendation networks.
Abstract
from arXiv · showhide
With the advent of deep learning, neural network-based recommendation models have emerged as an important tool for tackling personalization and recommendation tasks. These networks differ significantly from other deep learning networks due to their need to handle categorical features and are not well studied or understood. In this paper, we develop a state-of-the-art deep learning recommendation model (DLRM) and provide its implementation in both PyTorch and Caffe2 frameworks. In addition, we design a specialized parallelization scheme utilizing model parallelism on the embedding tables to mitigate memory constraints while exploiting data parallelism to scale-out compute from the fully-connected layers. We compare DLRM against existing recommendation models and characterize its performance on the Big Basin AI platform, demonstrating its usefulness as a benchmark for future algorithmic experimentation and system co-design.
1 Introduction
Personalization and recommendation systems have recently adopted neural networks, drawing on both recommendation-system methods and predictive analytics. The paper introduces DLRM by combining these perspectives for categorical and dense features.
- Personalization systems support tasks including ad click-through rate prediction and rankings at large internet companies.
- Recommendation methods evolved from expert-driven content filtering to collaborative filtering based on past user behavior.
- Predictive models progressed from linear and logistic regression toward deep networks that use embeddings for categorical data.
- DLRM combines embeddings for sparse categorical features, an MLP for dense features, explicit feature interactions, and a final MLP for event probabilities.Implementations are provided in PyTorch and Caffe2 for testing and experimentation.
2 Model Design and Architecture
DLRM combines embeddings for categorical features, MLPs for dense features, and explicit feature interactions to produce predictions. Its architecture draws on recommendation methods, factorization machines, and multilayer perceptrons.
- Embeddings: Embeddings map categorical data to dense representations, while weighted multi-item lookups can be represented with multi-hot vectors.A mini-batch of embedding lookups is represented using a sparse matrix A.
- Matrix Factorization: Matrix factorization represents users and products with embedding vectors whose dot products predict ratings, motivating DLRM's interaction design.The formulation addresses settings where only some entries of the rating matrix are known.
- Factorization Machines: Factorization machines incorporate second-order interactions into a linear model while factorizing interactions into latent vectors for sparse data.This reduces the complexity of second-order interactions to linear computational complexity.
- DLRM Architecture: DLRM processes categorical features with embeddings and continuous features with a bottom MLP, producing representations of the same dimension.The categorical embeddings generalize latent factors, while the bottom MLP produces a matching dense representation.
- DLRM Architecture: The model computes second-order interactions by taking dot products between embedding vectors and processed dense features.These interactions may optionally pass through MLPs before concatenation with the processed dense features.
- DLRM Architecture: A top MLP post-processes concatenated interactions and dense features before a sigmoid produces an event probability.
3 Parallelism
DLRM’s large embedding tables create memory-capacity and bandwidth constraints that make pure data parallelism impractical. The implementation therefore combines model parallelism for embeddings with data parallelism for MLP computation and uses personalized all-to-all communication between devices.
- Memory constraints: DLRM embeddings can require multiple GBs per table, making the model memory-capacity and bandwidth intensive.Replicating these embeddings with data parallelism is prohibitive, so the model may need distribution across devices.
- Combined parallelism: The parallelized design uses model parallelism for embeddings and data parallelism for MLPs.This mitigates embedding memory bottlenecks while parallelizing MLP forward and backward propagation.
- Communication: Embedding distribution requires personalized all-to-all communication because devices must exchange embedding vectors for the top MLP and interaction operator.The implementation uses a butterfly shuffle to slice embedding vectors and transfer them to target devices.
- Combined parallelism: MLP parameters are smaller in memory but require substantial computation, making data parallelism preferable for concurrent sample processing.Data-parallel MLP updates are accumulated with synchronous allreduce before each iteration.
4 Data
The implementation supports random, synthetic, and public datasets for evaluating accuracy and system behavior. Custom generators create continuous and categorical inputs, while synthetic traces model access distributions and public datasets provide real recommendation data.
- Datasets: The implementation supplies random, synthetic, and public datasets for operator characterization, overall performance testing, and accuracy measurement.Random and synthetic data exercise hardware properties without storage dependencies, whereas public data supports accuracy experiments.
- Random data: Continuous features are generated as random vectors, and mini-batches are formed as matrices whose rows represent samples.Uniform or normal distributions can generate the continuous values.
- Random data: Categorical features use fixed or random numbers of indices per multi-hot vector, with lengths or offsets delimiting embedding lookups.Indices are sampled within the embedding table’s row range and concatenated into mini-batch lookup representations.
- Synthetic data: Synthetic traces are generated from unique accesses and the observed probability distribution of distances between repeated accesses.The profiling and generation algorithms respectively estimate the distribution and sample stack distances to construct a trace.
- Synthetic data: Adjusted synthetic traces produce cache hit/miss rates similar to original traces in the experiments.The adjustment raises probabilities for unique accesses and removes them from the support after they have been observed.
- Public data: The Criteo Ad Kaggle and Terabyte datasets contain 13 continuous and 26 categorical features for ad CTR prediction.The Kaggle dataset covers approximately 45 million samples over 7 days, while the Terabyte dataset spans 24 days.
5 Experiments
Experiments evaluate DLRM accuracy on Criteo data and profile its execution on the Big Basin platform. DLRM slightly outperforms an untuned DCN in accuracy, while profiling shows embedding lookups and fully connected layers dominate runtime.
- Experimental setup: Experiments use PyTorch and Caffe2 implementations on a Big Basin system with dual Xeon 6138 CPUs and eight Tesla V100 GPUs.The model uses fp32 parameters and int32 or int64 indices depending on the framework.
- Model accuracy: DLRM is compared with an as-is Deep and Cross network on the Criteo Ad Kaggle dataset, with model sizes adapted to the dataset’s features.DLRM includes bottom and top MLPs for processing dense features and producing predictions.
- Model accuracy: DLRM obtains slightly higher training and validation accuracy than DCN after one epoch with SGD and Adagrad, without extensive hyperparameter tuning.Training accuracy is shown with solid lines and validation accuracy with dashed lines.
- Performance profiling: A sample DLRM with eight categorical features, 512 continuous features, and 1M-vector embedding tables per categorical feature is profiled on 2048K samples.The samples are organized into 1K mini-batches on a single socket/device.
- Performance profiling: The Caffe2 implementation runs in around 256 seconds on CPU and 62 seconds on GPU.Profiling attributes most runtime to embedding lookups and fully connected layers; fully connected layers are nearly negligible on GPU.
6 Conclusion
The paper proposes and open-sources a deep learning recommendation model for categorical data, aiming to make these industry-relevant networks more accessible for research and system co-design.
- The authors proposed and open-sourced a novel deep learning-based recommendation model that exploits categorical data.
- The work addresses the limited academic attention given to recommendation and personalization networks despite their practical success in industry.
- The model and implementation are intended to support algorithmic experimentation, modeling, system co-design, and benchmarking.
- A documented configuration uses eight 1,000,000-entry embedding tables, 64-dimensional sparse features, and specified bottom and top MLP architectures.