PaperlayerView exact revision on arXiv

Source-linked AI summary

The Architectural Implications of Facebook's DNN-based Personalized Recommendation

Udit Gupta, Carole-Jean Wu, Xiaodong Wang, Maxim Naumov, Brandon Reagen, David Brooks, Bradford Cottel, Kim Hazelwood, Bill Jia, Hsien-Hsin S. Lee, Andrey Malevich, Dheevatsa Mudigere, Mikhail Smelyanskiy, Liang Xiong, Xuan Zhang

arXiv:1906.03109v4cs.DCcs.LG

TL;DR

Production recommendation systems are important data-center workloads, but existing systems research and public benchmarks provide limited production-representative evidence. This paper releases production-scale models and open-source workloads, defines relevant metrics, and analyzes their execution across Intel servers. The results show distinct latency, throughput, co-location, and optimization tradeoffs driven by batching, server architecture, and model composition.

  • Problem

    Existing recommendation benchmarks and much DNN systems research do not adequately represent production-scale recommendation workloads with strict latency constraints, large embeddings, irregular memory access, and substantial fully connected computation.

  • Method

    The paper presents production-scale personalized recommendation models, quantitative metrics, synthetic workloads, open-source implementations, and performance analyses on Haswell, Broadwell, and Skylake servers.

  • Results

    Broadwell achieves up to 40% lower latency while Skylake achieves 30% higher throughput for the servers considered.

  • Takeaways & Limitations

    Latency-bounded throughput, batching, co-location, server heterogeneity, and model-specific optimization are central considerations for at-scale recommendation systems.

Abstract

from arXiv · show

The widespread application of deep learning has changed the landscape of computation in the data center. In particular, personalized recommendation for content ranking is now largely accomplished leveraging deep neural networks. However, despite the importance of these models and the amount of compute cycles they consume, relatively little research attention has been devoted to systems for recommendation. To facilitate research and to advance the understanding of these workloads, this paper presents a set of real-world, production-scale DNNs for personalized recommendation coupled with relevant performance metrics for evaluation. In addition to releasing a set of open-source workloads, we conduct in-depth analysis that underpins future system design and optimization for at-scale recommendation: Inference latency varies by 60% across three Intel server generations, batching and co-location of inferences can drastically improve latency-bounded throughput, and the diverse composition of recommendation models leads to different optimization strategies.

I. INTRODUCTION

Personalized recommendation dominates production AI inference, yet public benchmarks and existing DNN optimizations inadequately represent its production-scale workload diversity. The paper presents representative models, metrics, open-source workloads, and analyses connecting model composition and server architecture to latency, throughput, and scheduling.

  • Up to 79% of AI inference cycles in a production-scale data center are consumed by DNN-based personalized recommendation models, with RMC1–RMC3 accounting for up to 65%.
  • Production-scale recommendation workloads differ from public benchmarks through stricter application constraints, larger embedding tables, more irregular memory access, and larger fully-connected layers.MLPerf-NCF has fewer and smaller fully-connected layers, so its insights may not represent production systems.
  • The paper presents production-scale personalized recommendation models, quantitative performance metrics, synthetic models, and Intel Haswell, Broadwell, and Skylake analyses.The study also examines recommendation-model performance in production environments.
  • Latency-bounded throughput is more representative than latency alone because it measures how many items can be ranked under service-level agreement requirements.
  • With unit batch size, Broadwell optimizes inference latency, whereas increasing batch size favors Skylake throughput through higher compute density and AVX-512 acceleration.The comparison spans Haswell, Broadwell, and Skylake servers.
  • Co-locating recommendation models can improve throughput but trades off single-model latency, with Haswell and Broadwell particularly susceptible to latency degradation.The tradeoff creates additional scheduling optimization opportunities.
  • 30% to 95% of execution time is spent on compute-intensive operations such as fully connected layers across models and server architectures.Consequently, optimizing fully connected layers alone provides limited end-to-end latency improvement for embedding-dominated models.

II. BACKGROUND

Personalized recommendation ranks new content according to user preferences and interactions. Its inputs combine dense continuous features with sparse categorical features because users interact with only a small subset of a large content space.

  • Personalized recommendation recommends new content to users based on their preferences.Recommendations account for content-ranking services such as movies, videos, news-feed posts, and entertainment.
  • Recommendation systems rank content using users’ preferences and previous interactions, including clicks, ratings, and purchases.
  • Video recommendation is sparse because millions of viewers may encounter tens of thousands of videos, while each user interacts with only a handful.
  • Sparse features encode categorical inputs such as device type or content-genre preference, while dense features encode continuous inputs such as user age.Categorical inputs can be represented as multi-hot vectors.

B. Recommendation Models

Recommendation models separately transform dense and sparse features, combine their representations, and produce a predicted click-through rate. Their architecture and bottlenecks vary with embedding-table, sparse-lookup, fully connected, and batch-size configurations.

  • Model architecture: Dense features pass through Bottom-FC layers, while sparse features become dense representations through embedding-table lookups and aggregation.Each sparse ID selects an embedding-table row, and selected rows are combined into a vector.
  • Model architecture: The concatenated embedding and Bottom-FC outputs pass through Top-FC layers to produce a predicted click-through rate for each user-post pair.Pairs with the highest predicted click-through rate are prioritized.
  • Execution: At data-center scale, requests for many user-post pairs are batched to improve overall throughput.
  • Model diversity: The depth and width of FC layers, embedding-table count and size, sparse IDs per table, and typical batch size depend on the recommendation use case.

C. Embedding Tables

Embedding tables distinguish recommendation DNNs from CNNs and RNNs, but their size, low compute density, and irregular accesses create execution challenges.

  • Embedding tables transform sparse input features into dense representations but require large storage, low compute density, and irregular memory accesses.These properties make efficient execution difficult compared with conventional CNN and RNN layers.
  • Embedding tables can consume tens of GB of memory per recommendation model, requiring large off-chip memory systems.Individual tables range from tens of MBs to several GBs, and each model can contain 4 to 40 tables.
  • 0.25 FLOPs/Byte: SparseLengthsSum has much lower operational intensity than RNN, FC, and CNN layers.The comparison values are 5.5 FLOPs/Byte for RNN, 18 FLOPs/Byte for FC, and 141 FLOPs/Byte for CNN.
  • 8 MPKI: a typical SparseLengthsSum operator has a higher LLC cache miss rate than RNN, FC, and CNN layers on Broadwell.The comparison rates are 0.5 MPKI, 0.2 MPKI, and 0.06 MPKI, respectively.

III. AT-SCALE PERSONALIZATION

Production personalization hierarchically filters and ranks content using diverse recommendation models whose embedding, lookup, and fully connected-layer configurations shape their resource demands.

  • A. Production Recommendation Pipeline: Thousands of candidate posts are filtered down by orders of magnitude before DNN-based models rank the remaining subset under strict latency constraints.Filtering may use lightweight methods or RMC1, while finer-grained ranking uses larger models such as RMC2 and RMC3.
  • B. Diversity of Recommendation Models: Recommendation models vary in embedding-table counts and sizes, lookup counts, and Bottom-FC and Top-FC dimensions.Table I normalizes each parameter to the smallest instance across the three configurations, with lookup counts normalized to RMC3.
  • B. Diversity of Recommendation Models: RMC2 can have up to an order of magnitude more embedding tables than RMC1 and RMC3.Aggregate embedding-table storage is 100MB, 10GB, and 1GB for RMC1, RMC2, and RMC3, respectively.
  • B. Diversity of Recommendation Models: RMC1 and RMC2 perform more embedding lookups than RMC3, leading to more irregular memory accesses and higher cache miss rates on Intel servers.RMC1 and RMC2 serve workloads with many sparse features, whereas RMC3 serves applications with fewer sparse features.
  • B. Diversity of Recommendation Models: RMC3 has generally much wider Bottom-FC layers and is more compute intensive than RMC1 and RMC2.Its wider layers reflect the use of more dense features in social-media-post ranking.

IV. EXPERIMENTAL SETUP

The evaluation studies production-scale recommendation inference across heterogeneous Intel servers using representative synthetic models and a benchmark that preserves their execution flow.

  • Server Architectures: Haswell, Broadwell, and Skylake servers provide the heterogeneous CPU platforms used to study recommendation inference performance.Skylake adds AVX-512, more parallel cores, and larger L2 caches, while the cache hierarchies also differ across generations.
  • Synthetic Recommendation Models: Representative RMC1, RMC2, and RMC3 implementations are analyzed with a benchmark that accurately represents production-scale model execution flow.The benchmark is implemented in Caffe2 with Intel MKL as a backend library.
  • Inference Analysis: An order of magnitude: unit-batch inference latency varies across RMC1, RMC2, and RMC3 on an Intel Broadwell server.The operator-level time breakdown also varies significantly across the three models.
  • Experimental Method: Inference performance is evaluated with a single Caffe2 worker and a single Intel MKL thread, using fp32 data and model parameters.The setup also studies non-unit batch sizes and model co-location to process inputs and models in parallel for throughput under SLA constraints.

V. UNDERSTANDING INFERENCE PERFORMANCE OF A SINGLE MODEL

Single-model recommendation inference is shaped by model composition, batch size, and server architecture. Broadwell favors low-batch latency, while Skylake gains with larger batches but remains constrained by irregular embedding-table accesses.

  • Model composition: 15×: inference latency varies across production-scale recommendation models, with embedding-table size driving RMC2 latency and fully connected layers driving RMC3 latency.On Broadwell at unit batch size, RMC1, RMC2, and RMC3 measure 0.04ms, 0.30ms, and 0.60ms, respectively.
  • Model composition: No single operator determines runtime bottlenecks: compute-intensive RMC3 spends over 96% of execution time in BatchMatMul or FC, whereas RMC1 spends 20% in SparseLengthsSum.BatchMatMul and FC comprise only 61% of RMC1 runtime, with additional time in embedding operations, concatenation, and activations.
  • Server architecture: At batch size 16, Broadwell improves performance over Haswell and Skylake by 1.4× and 1.5× for RMC1, 1.3× and 1.4× for RMC2, and 1.32× and 1.65× for RMC3.Broadwell’s advantage at low batch sizes comes from its 20% higher clock frequency, while small batches do not efficiently exploit Skylake’s AVX-512.
  • Batching: Skylake outperforms Haswell and Broadwell starting at batch size 64, but its gains are limited by irregular embedding-table accesses.Skylake runs memory-intensive RMC1 1.3× faster than Broadwell, while SparseLengthsSum becomes the runtime bottleneck at sufficiently high batch sizes.
  • Design implications: Recommendation inference requires balancing latency, throughput, memory capacity, bandwidth, and FLOPs across diverse model characteristics.The paper argues for full-stack optimization rather than standalone FC acceleration, including efficient embedding-table execution.

VI. UNDERSTANDING EFFECTS OF CO-LOCATING MODELS

Co-location trades per-model latency for aggregate throughput, with the tradeoff depending on model memory behavior and cache hierarchy. Skylake handles high co-location better than Haswell and Broadwell, while RMC2 is especially sensitive to interference.

  • Latency degradation: Co-locating eight models degrades latency by 1.3× for RMC1, 2.6× for RMC2, and 1.6× for RMC3.RMC2 is most affected because its irregular memory accesses increase pressure on shared memory resources.
  • Latency degradation: For RMC2, co-location increases FC and SparseLengthsSum time by 1.6× and 3×, respectively.SparseLengthsSum is more affected because its irregular accesses provide less cache reuse; RMC2 LLC-MPKI rises from 0.06 to 0.8 when instances increase from 1 to 8.
  • Cache hierarchy: Inclusive L2/L3 cache hierarchies make Haswell and Broadwell more susceptible to per-model latency degradation than Skylake’s exclusive hierarchy.Broadwell’s L2 miss rate increases by 29% with 16 co-located inferences, compared with a 10% increase on Skylake.
  • Throughput tradeoff: At low co-location, Broadwell has 10% higher throughput and lower latency than Skylake, but Skylake outperforms both architectures as co-location increases.Under a strict 3ms latency bound, Skylake provides the highest throughput by accommodating multiple co-located models.
  • Concurrency: Hyperthreading degrades p99 latency for recommendation models, especially compute-intensive RMC3, despite generally improving system throughput.The reported results use one model per physical core without hyperthreading.

A. Recommendation Inference in Production

Production co-location increases throughput but also creates substantial latency variability, particularly on Broadwell. Skylake’s exclusive cache hierarchy moderates both average and tail-latency degradation under increasing co-location.

  • Production behavior: Co-location increases overall throughput while introducing significant performance variability in production recommendation serving.The paper identifies a tradeoff between aggregate throughput and predictable SLA satisfaction.
  • Latency distributions: Broadwell shows a multimodal FC latency distribution at 40µs, 58µs, and 75µs, whereas Skylake shows a single mode at 45µs.The distributions arise despite fixed operator input and output dimensions in the production environment.
  • Latency distributions: On Broadwell, FC latency forms regions near 40µs with no co-location, 60µs with 5–15 jobs, and 100µs with over 20 jobs.The p99 latency rises significantly above 20 co-located jobs, while Skylake’s average and p99 latency increase more gradually.
  • Architecture and scheduling: Skylake’s exclusive L2/L3 cache hierarchy reduces the shared-memory impact of co-locating recommendation models with irregular memory accesses.The paper presents scheduling optimizations as an opportunity to balance latency, throughput, and performance variability.

VII. OPEN-SOURCE BENCHMARK

Public recommendation benchmarks do not represent production-scale workloads because their embeddings and fully connected layers are substantially smaller. Consequently, they emphasize compute behavior unlike production models.

  • Benchmark representativeness: MLPerf-NCF has orders-of-magnitude smaller embedding tables and fewer fully connected parameters than production-scale recommendation workloads.These differences make the benchmark’s resource demands unlike those of deployed recommendation systems.
  • Benchmark representativeness: Fully connected layers comprise over 90% of MLPerf-NCF execution time, whereas SparseLengthsSum comprises around 80% of production RMC cycles.The contrasting breakdowns reflect the different scale and composition of the benchmark and production workloads.

A. Configuring the open-source benchmark

The open-source benchmark is designed to represent production-scale and diverse recommendation models through configurable end-to-end system parameters. It supports performance analysis, scheduling studies, and memory-system optimization for workloads whose sparse and dense components create distinct bottlenecks.

  • Benchmark flexibility: Configurable end-to-end parameters let the benchmark represent RMC1, RMC2, RMC3, and other realistic recommendation models.The parameters define system components such as embedding tables, fully connected layers, batches, and sparse lookups.
  • Production-scale representation: Production-scale recommendation differs from MLPerf-NCF through stricter latency and throughput constraints, larger embedding tables, and more compute-intensive fully connected layers.These differences produce larger storage requirements, irregular memory accesses, and more floating-point operations.
  • System studies: The benchmark supports studies of distributed inference, threading, and scheduling by varying batch, fully connected, and embedding-table configurations.It can analyze recommendation models running across many nodes or threads.
  • Memory-system optimization: Input-dependent sparse accesses create opportunities for intelligent caching and prefetching despite their irregularity.Embedding trace generators expose these access patterns for memory-system studies.
  • Optimization challenges: Recommendation workloads combine memory capacity demands, irregular accesses, and diverse compute- and memory-intensive operators, limiting direct reuse of CNN- and RNN-oriented optimizations.The paper positions its benchmark and analysis as a basis for broader recommendation-system optimization.

IX. CONCLUSION

The paper analyzes production-scale recommendation models on server-scale systems and identifies their distinctive execution challenges and optimization opportunities. Across evaluated servers, Broadwell provides lower latency while Skylake provides higher throughput, and sparse-access traces support memory-system studies.

  • Conclusion: Recommendation DNNs require larger storage capacity, produce irregular memory accesses, and exhibit diverse operator-level performance bottlenecks.These characteristics distinguish them from traditional CNNs and RNNs.
  • Conclusion: Broadwell achieves up to 40% lower latency while Skylake achieves 30% higher throughput on the servers considered.The comparison highlights different platform-level performance trade-offs for recommendation inference.
  • Conclusion: Embedding-access traces enable future memory-system optimization studies for recommendation workloads.The traces expose the fraction of unique sparse IDs used across production recommendation use cases.
Loading 1906.03109v4…