PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inference

Udit Gupta, Samuel Hsia, Vikram Saraph, Xiaodong Wang, Brandon Reagen, Gu-Yeon Wei, Hsien-Hsin S. Lee, David Brooks, Carole-Jean Wu

arXiv:2001.02772v1cs.DC

TL;DR

Neural personalized recommendation consumes substantial datacenter inference capacity, while heterogeneous models and heavy-tailed query sizes complicate efficient serving. The paper introduces DeepRecInfra and its dynamic DeepRecSched scheduler to co-design execution for model, query, latency, and hardware characteristics. Across eight models, the system doubles throughput, with production evaluation also reporting substantial latency improvements.

  • Problem

    Recommendation dominates production inference capacity, but heterogeneous model behavior and heavy-tailed query sizes require specialized at-scale efficiency solutions.

  • Method

    DeepRecInfra characterizes representative models and production query patterns, while DeepRecSched dynamically chooses request- and batch-level parallelism for latency-bounded throughput.

  • Results

    DeepRecSched doubles system throughput under strict latency targets across eight recommendation models.

  • Takeaways & Limitations

    At-scale recommendation inference benefits from scheduling that adapts to query distributions, model architectures, latency targets, and hardware platforms.

Abstract

from arXiv · show

Neural personalized recommendation is the corner-stone of a wide collection of cloud services and products, constituting significant compute demand of the cloud infrastructure. Thus, improving the execution efficiency of neural recommendation directly translates into infrastructure capacity saving. In this paper, we devise a novel end-to-end modeling infrastructure, DeepRecInfra, that adopts an algorithm and system co-design methodology to custom-design systems for recommendation use cases. Leveraging the insights from the recommendation characterization, a new dynamic scheduler, DeepRecSched, is proposed to maximize latency-bounded throughput by taking into account characteristics of inference query size and arrival patterns, recommendation model architectures, and underlying hardware systems. By doing so, system throughput is doubled across the eight industry-representative recommendation models. Finally, design, deployment, and evaluation in at-scale production datacenter shows over 30% latency reduction across a wide variety of recommendation models running on hundreds of machines.

I. INTRODUCTION

Neural recommendation dominates production inference capacity yet has heterogeneous architectures, memory behavior, and query-size distributions that require specialized at-scale optimization. DeepRecInfra and DeepRecSched co-design infrastructure and scheduling around these characteristics to improve latency-bounded throughput.

  • Motivation: More than 70% of Facebook datacenter machine-learning inference cycles serve recommendation and ranking, motivating specialized optimization.Facebook recommendation use cases require more than 10× the inference capacity of common computer-vision and natural-language-processing tasks.
  • Recommendation heterogeneity: Recommendation models are memory intensive, storage demanding, and irregular in memory access compared with CNNs and RNNs.Their inputs include both continuous and categorical features, whose processing operations differ fundamentally.
  • Recommendation heterogeneity: Model-level heterogeneity arises because use cases size major recommendation components differently, producing distinct dense and sparse memory-access patterns.Dense processing dominates some models, while sparse categorical processing dominates others.
  • At-scale effects: 1.7× higher system throughput is achieved by optimizing against production recommendation query-size distributions instead of log-normal distributions.Personalized recommendation working-set sizes have heavier tails than standard web-service distributions.
  • Approach: DeepRecInfra characterizes eight industry-representative models and production query arrivals, while DeepRecSched co-designs request- and batch-level parallelism.The scheduler considers query rate and size, model architecture, hardware platform, and service-level latency targets.
  • Results: DeepRecSched doubles system throughput under strict latency targets across the evaluated recommendation models.The infrastructure also supports production-datacenter evaluation and includes representative models and query distributions for follow-on work.

II. NEURAL RECOMMENDATION MODELS

Personalized recommendation models combine dense and sparse features because user interactions cover only a small subset of possible items. Their architectures process embeddings, feature interactions, and large-scale item rankings before presenting top-N content.

  • Overview: Recommendation models personalize content from user preferences and prior interactions, but users interact with only a small subset of possible items.This sparsity distinguishes recommendation inputs from traditional DNN inputs.
  • Features: Dense features represent continuous inputs and are commonly processed with fully connected MLP layers.Sparse features represent categorical inputs such as purchased products or liked movies.
  • Embedding tables: Embedding tables map sparse categorical features to latent vectors, with table sizes ranging from tens to billions of rows.The vector dimension is determined by the number of latent features used for category representation.
  • Embedding-table access: Sparse embedding-table operations use one-hot or multi-hot indices and incur irregular memory accesses during pooling.Pooling combines selected embedding vectors using operations such as sum, dot product, or multiplication.
  • Feature interaction: Dense and sparse outputs are combined through feature interaction before subsequent predictor-DNN stacks.Typical interactions include concatenation, summation, and averaging.
  • Product ranking: Predictor-DNN outputs estimate click-through-rate probabilities for thousands of user-item pairs, which are ranked to select top-N content.This ranking workload requires non-unit inference batch sizes.

III. DEEPRECINFRA: AT-SCALE RECOMMENDATION

DeepRecInfra is an extensible framework for studying at-scale recommendation inference across diverse models and use cases. It combines representative models with production-informed execution behavior to support system evaluation.

  • Infrastructure: DeepRecInfra is a highly extensible infrastructure for modeling and evaluating at-scale recommendation inference.It is designed to support diverse recommendation models and use cases.
  • Infrastructure: The generalized recommendation architecture can be customized through key parameters to realize industry implementations with distinct performance characteristics.Recent models from Google, Facebook, and Alibaba exhibit notable architectural differences.

1) State-of-the-art neural recommendation models:

DeepRecInfra covers eight diverse recommendation architectures whose differing feature-processing structures create distinct optimization needs. Their performance and accelerator behavior vary across models and batch sizes.

  • DeepRecInfra composes eight state-of-the-art recommendation models to capture architectural diversity.The collection spans commercial video recommendation, e-commerce, and social-media use cases.
  • Model architectures: NCF uses four embedding tables and a small MLP predictor, processing only one-hot sparse features without a dense fully connected stack.
  • Model architectures: WnD combines sparse and dense features, while MT-WnD extends it with parallel prediction stacks for multiple objectives.MT-WnD models objectives including click-through rate, comment rate, likes, and ratings.
  • Model architectures: DLRM-RMC1, RMC2, and RMC3 vary embedding lookups and fully connected layer sizes, producing distinct architectures.
  • Model architectures: DIN applies attention over embedding-table outputs, whereas DIEN adds gated recurrent units to capture evolving user interests.
  • Hardware behavior: GPU speedup and the batch size needed to outperform CPUs vary across recommendation models.The comparison considers representative models across multiple batch sizes.

2) Operator diversity:

Recommendation inference exhibits substantial operator and workload diversity. These differences shape bottlenecks, CPU/GPU benefits, and the latency-throughput tradeoffs used for system design.

  • Operator diversity: MLP layers dominate dense-feature models, embedding lookups dominate sparse-feature models, and attention-based models have different bottlenecks.DIN divides runtime across concatenation, embedding, sum, and fully connected operations, while DIEN is dominated by recurrent layers.
  • Query diversity: Recommendation query sizes follow a heavy-tailed production distribution unlike the normal or log-normal distributions commonly assumed for web workloads.
  • Acceleration opportunity: GPUs accelerate larger queries, but the optimal inflection point and speedup differ across models.
  • Acceleration opportunity: GPU data loading consumes 60∼80% of end-to-end inference time on average across recommendation models and batch sizes.GPU benefits are especially significant at higher batch sizes for compute-intensive models such as WnD.
  • Latency and throughput: Recommendation throughput is measured as queries per second under a p95 tail-latency requirement.
  • Latency and throughput: Tail-latency targets vary across applications and service-level agreements, leading to distinct system design decisions.

C. Real-Time Query Serving for Recommendation Inference

Real-time recommendation serving must model both query arrivals and working-set sizes. Production arrivals are Poisson-like, while query sizes have a heavy tail that affects execution time and accelerator choices.

  • Serving model: DeepRecInfra models real-time serving through query arrival rate and working-set size.
  • Query arrivals: Production profiling finds recommendation query arrival rates follow a Poisson distribution.
  • Query sizes: Recommendation query sizes have a heavier tail than the fixed, normal, or log-normal distributions often assumed for web services.
  • Execution impact: Queries below the p75 size account for over half of CPU execution time, while 25% of large queries contribute nearly 50% of total execution time.
  • Scale modeling: A handful of machines can approximate datacenter-scale tail-latency trends within 10% for recommendation inference.The comparison uses server-class Intel Skylake and Broadwell machines.

E. Putting it Altogether

DeepRecSched combines request/batch parallelism with accelerator offloading to maximize QPS under strict tail-latency targets. The best configuration depends on models, latency targets, hardware, and query sizes.

  • E. Putting it Altogether: DeepRecInfra provides an extensible framework for modeling recommendation models, query arrivals, and query-size distributions at scale.
  • E. Putting it Altogether: DeepRecSched splits large queries into smaller requests and offloads suitable large queries to specialized AI hardware.The design balances batch-level and SIMD-level parallelism, cache contention, and queuing delay.
  • E. Putting it Altogether: Splitting queries across cores exploits hardware parallelism but reduces parallelism within each individual request.
  • E. Putting it Altogether: For DLRM-RMC3, the optimal batch size increases from 128 to 256 as the tail-latency target relaxes from 66ms to 100ms.
  • E. Putting it Altogether: Optimal batch sizes are 64 for DIEN, 128 for DLRM-RMC3, and 256 for DLRM-RMC1.
  • E. Putting it Altogether: Hardware heterogeneity changes the tradeoff between request-level and batch-level parallelism because Broadwell and Skylake provide different SIMD capabilities and cache behavior.Broadwell uses AVX-256, while Skylake uses AVX-512.
  • E. Putting it Altogether: The optimal query-size threshold, and therefore the GPU-processed query fraction, varies among embedding-, MLP-, and attention-dominated models.

B. Leverage parallelism with specialized hardware

DeepRecSched uses dynamic batch-size and CPU/GPU offloading decisions to optimize latency-bounded recommendation throughput across query patterns, models, and hardware platforms.

  • Scheduler design: A hill-climbing algorithm identifies optimal batch and query request sizes across recommendation models and hardware platforms.
  • Evaluation setup: Figure 11 compares DeepRecSched-CPU and DeepRecSched-GPU with a static scheduler using throughput across latency targets and QPS/Watt power efficiency.
  • Scheduler design: DeepRecSched balances batch- and request-level parallelism while tuning query-size thresholds for specialized-hardware offloading.It increases batch size while QPS improves under the target tail latency, then hill-climbs the query-size threshold for accelerator processing.
  • Evaluation setup: DeepRecSched is evaluated with DeepRecInfra against a production-scale fixed-batch baseline across hardware systems and platforms.The setup includes CPU and GPU implementations, configurable latency targets, production query-size distributions, and Poisson arrival patterns.
  • Evaluation setup: Optimal batch size varies across SLA targets, query-size distributions, recommendation models, and hardware platforms.

VI. DEEPRECSCHED EVALUATION

Across eight recommendation models, DeepRecSched substantially improves throughput and power efficiency over a static scheduler, while GPU benefits depend on model characteristics.

  • Performance: 1.7×, 2.1×, and 2.7× higher QPS are achieved by DeepRecSched-CPU at low, medium, and high tail-latency targets.DeepRecSched-GPU increases performance improvement to 4.0×, 5.1×, and 5.8× at the corresponding targets.
  • Power efficiency: 1.7×, 2.1×, and 2.7× higher QPS/Watt are achieved by DeepRecSched-CPU at low, medium, and high tail-latency targets.
  • Power efficiency: GPU offloading degrades power efficiency for memory-intensive models DLRM-RMC1 and DIN because its power overhead outweighs the performance gain.

A. Balance of Request and Batch Parallelism

DeepRecSched improves recommendation throughput by selecting request- versus batch-level parallelism according to latency targets, query distributions, model characteristics, and hardware platforms.

  • Overall tradeoff: DeepRecSched-CPU improves QPS over a fixed static baseline by balancing request- versus batch-level parallelism across varied operating conditions.
  • Optimizing across SLA targets: Lower tail-latency targets favor smaller batches and request-level parallelism, whereas relaxed targets favor larger batches and batch-level parallelism.
  • Optimizing across query size distributions: Applying lognormal-distribution batch sizes to production query sizes degrades DLRM-RMC1 performance by 1.2×, 1.4×, and 1.7× at low, medium, and high tail latencies.Production query sizes have a heavier tail, requiring distribution-specific optimization.
  • Optimizing across recommendation models: Compute-intensive models use smaller optimal batches, while memory-intensive models use larger batches to exploit SIMD, core parallelism, and memory bandwidth.At high SLA targets, optimal batch sizes are 256 for DLRM-RMC3, 128 for WnD, and 1024 for DLRM-RMC1 and DIN.
  • Production validation: Across models and servers, batch-size optimization reduces p95 latency by 1.39× and p99 latency by 1.31× in production.
  • Optimizing across hardware platforms: At a 175ms latency target, optimal batch sizes are 1024 on Intel Broadwell and 256 on Intel Skylake.The difference reflects their distinct cache hierarchies and cache-contention behavior.
  • Specialized hardware: Joint CPU/GPU scheduling increases system throughput, while the power-optimal configuration varies with tail-latency targets.GPUs are optimal for power efficiency at low tail latencies, whereas CPUs are better at higher targets.
  • Overall tradeoff: The optimal batch size depends on tail-latency targets, query-size distributions, models, and underlying hardware platforms.

B. Tail Latency Reduction for At-Scale Production Execution

Production deployment shows that specialized hardware can reduce achievable tail latency and increase throughput, but power efficiency depends on latency targets and model architecture.

  • Production execution: In production, batch-size optimization is evaluated across recommendation models and server-class Intel CPUs to measure tail-latency effects.
  • Tail latency: GPUs reduce DLRM-RMC1’s lowest achievable tail-latency target from 57ms on CPUs to 41ms, a 1.4× reduction.
  • Throughput: DeepRecSched-GPU achieves higher QPS than DeepRecSched-CPU across tail-latency targets by executing larger queries on GPUs.
  • Throughput: At a 120ms tail-latency target, the optimal query-size threshold is 324 and GPU-processed work falls to 18%.
  • Infrastructure efficiency: QPS/Watt is maximized by CPU/GPU parallelization at low tail latencies, while relaxed targets can favor CPU execution.
  • Infrastructure efficiency: Power efficiency is co-optimized across tail-latency targets and recommendation models, with specialized hardware outperforming for compute-intensive models.
  • Infrastructure efficiency: Infrastructure efficiency requires considering model architecture and tail-latency targets across recommendation use cases.

VII. RELATED WORK

Prior work largely targets traditional DNN inference or isolated recommendation optimizations, leaving at-scale recommendation query patterns and system specialization insufficiently addressed. DeepRecInfra and DeepRecSched provide an end-to-end framework and specialized scheduling design, achieving 2× throughput across eight models under varied SLA targets.

  • Prior DNN optimization: Prior DNN benchmarks and accelerator designs primarily target fully connected, convolutional, and recurrent networks rather than the recommendation models studied here.These prior solutions do not apply to the paper’s broad collection of state-of-the-art recommendation models.
  • Recommendation optimization: Recommendation-focused prior work explores near-memory processing and embedding-table compression, whereas this paper pursues broader system design optimization.The cited approaches address selected model families or model representation rather than end-to-end at-scale inference.
  • Machine learning at-scale: Prior at-scale machine-learning studies examine warehouse-scale execution, but recommendation inference properties and query patterns were not their focus.This leaves the interaction between recommendation workloads and at-scale system behavior insufficiently explored.
  • Paper contribution: DeepRecInfra supplies an end-to-end infrastructure and models production-like real-time query patterns and working-set distributions for recommendation inference.DeepRecSched is designed from these workload characteristics for at-scale recommendation performance improvement.
  • Paper contribution: 2× throughput improvement is demonstrated across eight recommendation models and varied SLA targets using DeepRecSched.The design is additionally evaluated in a real production datacenter, where similar performance benefits are observed.
Loading 2001.02772v1…