RecNMP: Accelerating Personalized Recommendation with Near-Memory Processing

Liu Ke; Udit Gupta; Carole-Jean Wu; Benjamin Youngjae Cho; Mark Hempstead; Brandon Reagen; Xuan Zhang; David Brooks; Vikas Chandra; Utku Diril; Amin Firoozshahian; Kim Hazelwood; Bill Jia; Hsien-Hsin S. Lee; Meng Li; Bert Maher; Dheevatsa Mudigere; Maxim Naumov; Martin Schatz; Mikhail Smelyanskiy; Xiaodong Wang

Source-linked AI summary

RecNMP: Accelerating Personalized Recommendation with Near-Memory Processing

Liu Ke, Udit Gupta, Carole-Jean Wu, Benjamin Youngjae Cho, Mark Hempstead, Brandon Reagen, Xuan Zhang, David Brooks, Vikas Chandra, Utku Diril, Amin Firoozshahian, Kim Hazelwood, Bill Jia, Hsien-Hsin S. Lee, Meng Li, Bert Maher, Dheevatsa Mudigere, Maxim Naumov, Martin Schatz, Mikhail Smelyanskiy, Xiaodong Wang

arXiv:1912.12953v1Dec 30, 2019cs.DCcs.AR

TL;DR

Personalized recommendation inference is dominated by irregular, memory-bound sparse embedding operations that challenge conventional acceleration. The paper proposes RecNMP, a lightweight commodity-DRAM near-memory architecture with production-aware hardware/software co-optimization. It reports up to 4.2× throughput improvement and 45.8% memory energy savings, alongside up to 9.8× sparse-embedding performance improvement.

Problem
Sparse embedding lookup and pooling dominate recommendation inference while exhibiting irregular access patterns and memory bandwidth bottlenecks.
Method
RecNMP performs embedding operations near commodity DRAM and combines lightweight hardware with caching, packet scheduling, and hot-entry profiling.
Results
4.2× throughput improvement and 45.8% memory energy savings are reported overall, with up to 9.8× performance improvement for sparse embedding operations.
Takeaways & Limitations
RecNMP provides a practical, scalable near-memory solution for personalized recommendation across production-relevant models and configurations.

Abstract

from arXiv · show

Personalized recommendation systems leverage deep learning models and account for the majority of data center AI cycles. Their performance is dominated by memory-bound sparse embedding operations with unique irregular memory access patterns that pose a fundamental challenge to accelerate. This paper proposes a lightweight, commodity DRAM compliant, near-memory processing solution to accelerate personalized recommendation inference. The in-depth characterization of production-grade recommendation models shows that embedding operations with high model-, operator- and data-level parallelism lead to memory bandwidth saturation, limiting recommendation inference performance. We propose RecNMP which provides a scalable solution to improve system throughput, supporting a broad range of sparse embedding models. RecNMP is specifically tailored to production environments with heavy co-location of operators on a single server. Several hardware/software co-optimization techniques such as memory-side caching, table-aware packet scheduling, and hot entry profiling are studied, resulting in up to 9.8x memory latency speedup over a highly-optimized baseline. Overall, RecNMP offers 4.2x throughput improvement and 45.8% memory energy savings.

I. INTRODUCTION

Personalized recommendation models are important but difficult to accelerate because sparse embedding operations are irregular, memory-bound, and dominant in production inference. RecNMP addresses this bottleneck with a lightweight commodity-DRAM near-memory design and production-aware hardware/software co-optimization.

Motivation: More than 72% of AI inference cycles across Facebook’s production datacenters come from top recommendation models.Recommendation systems serve search, social networking, retail, and streaming services.
Motivation: Recommendation models have low compute-intensity and little regularity, limiting existing accelerators that rely on reusable dataflow and spatial locality.These characteristics make techniques developed for FC, CNN, and RNN workloads poorly suited to recommendation models.
Bottleneck: Sparse embedding operations combine irregular lookup with pooling over tables of tens to hundreds of GBs, defeating prefetching and exhausting on-chip memory.Their low operational intensity creates a fundamental memory bottleneck that standard caching, batching, and hardware acceleration do not overcome.
Approach: RecNMP performs embedding lookup and pooling near commodity DRAM while leaving compute-intensive FC operators on the CPU or an accelerator.The DIMM-based design targets production-scale capacities without specialized 2.5D/3D integration.
Results: 9.8× memory latency speedup, 4.2× end-to-end throughput improvement, and 45.9% memory energy savings are reported for RecNMP.The system combines memory-side caching, table-aware packet scheduling, hot entry profiling, and customized NMP instructions.
Evaluation: A production-aware evaluation framework accounts for model co-location and load balancing when assessing RecNMP.The design exploits distinctive spatial and temporal reuse patterns from production embedding traces.

II. CHARACTERIZING DEEP LEARNING PERSONALIZED RECOMMENDATION MODELS

Production-scale recommendation models combine dense processing with sparse embedding lookup and pooling, whose Gather-Reduce behavior creates distinctive memory requirements. Characterization of these models motivates near-memory acceleration for their dominant embedding operations.

A. Overview of Personalized Recommendation Models: Recommendation models combine dense features processed by DNN layers with sparse features indexed from large embedding tables.The architecture reflects production-scale models and includes representative model parameters.
A. Overview of Personalized Recommendation Models: Embedding tables contain potentially millions of vectors, and lookup plus pooling operations provide the learned representation of sparse features.These operations follow a Gather-Reduce pattern, with model-specific element-wise reductions.
A. Overview of Personalized Recommendation Models: Figure 2 distinguishes recommendation models by their dense and sparse components and by parameters such as the number of embedding tables.The listed configurations represent production-scale recommendation models.

B. A Case Study—Facebook’s DLRM Benchmark

Facebook’s DLRM benchmark represents production recommendation workloads through two canonical model classes, RM1 and RM2, configured for multiple levels of parallel inference.

Model Structure: DLRM processes dense features with BottomFC, sparse features through embedding-table lookups, and combines them in TopFC to predict click-through rate.
Benchmark Models: RM1 and RM2 represent two canonical recommendation-model classes used for acceleration analysis.RM1 has few embedding tables, whereas RM2 has tens of embedding tables.
Production Relevance: RM1 accounts for over 30% and RM2 over 25% of machine-learning execution cycles in Facebook’s production datacenter.
Parallelism: Production recommendation systems use model-, operator-, and data-level parallelism to achieve high throughput under strict latency constraints.Model-level parallelism increases concurrent inferences, while operator-level parallelism adds threads per model.

C. Operator Bottleneck Study

Sparse embedding operations dominate recommendation-model latency and push execution toward the system’s memory-bandwidth limit, leaving little headroom without additional bandwidth.

Latency Breakdown: 73.5% and 68.9% of execution time is spent on SLS for RM2-small and RM2-large at batch size 8.For RM1-small and RM1-large, SLS accounts for 37.2% and 50.6%, respectively; its share rises with batch size.
Roofline Analysis: Recommendation models operate in the memory-bandwidth-constrained region of the roofline.SLS has low compute and high memory requirements, while FC has higher compute needs.
Roofline Analysis: SLS performance approaches the theoretical system bound as batch size increases.
Roofline Analysis: 35.1% from the theoretical performance bound leaves little room for improvement without increasing system memory bandwidth.Near-memory execution can lift the bandwidth roofline by performing embedding lookup and pooling before the pin-limited interface.
Bandwidth Saturation: 67.4% of available bandwidth is consumed by SLS at batch size 256 with 30 threads.Higher saturation becomes undesirable because memory latency starts increasing significantly.

F. Embedding Table Locality Analysis

Production embedding-table traces exhibit meaningful temporal reuse but little spatial locality, motivating locality-aware caching and scheduling in RecNMP.

Locality Characterization: Production traffic has modest locality, mostly from temporal reuse, contrary to the assumption that embedding lookups are always random.The analysis evaluates both random traces and production traces from embedding tables T1–T8.
Temporal Locality: 20%–60% hit rates occur for combined production traces, compared with less than 5% for random traces.Production-trace hit rates increase as cache capacity grows from 8MB to 64MB.
Spatial Locality: Increasing cacheline size from 64B to 512B decreases hit rate, indicating little spatial locality in embedding lookups.The same trend appears with a fully associative cache, isolating conflict misses as the explanation.
Design Implication: RecNMP is designed to exploit recommendation workloads’ sparse, irregular accesses by computing locally on data fetched from concurrently activated DRAM ranks.Its design maximizes DRAM rank-level parallelism for the dominated embedding operations.
Design Implication: RecNMP uses lightweight specialized logic and a rank-level cache while retaining commodity DRAM compatibility.The hardware supports SLS-family operators rather than general-purpose computation.

A. Hardware Architecture

RecNMP places lightweight processing units in DIMM buffer chips and rank modules to perform embedding lookups and pooling near commodity DRAM. Its compressed instructions and concurrent rank activation address bandwidth and command-interface limits caused by irregular sparse accesses.

Architecture: Each RecNMP-equipped DIMM contains a processing unit with one DIMM-NMP module and multiple rank-NMP modules.Multiple DIMMs and memory channels can be coordinated to scale capacity and parallelism.
Processing hierarchy: Rank-NMP modules translate instructions into DRAM commands, manage memory-side caching, and compute SLS-family operators concurrently.DIMM-NMP aggregates rank-level partial sums and returns the final pooled result to the host.
Memory-side caching: A RankCache uses LocalityBit hints to cache reusable embedding vectors while bypassing the cache for other requests.This targets temporal reuse because caching final SLS results is ineffective when operands vary across operators.
Bandwidth expansion: Compressed NMP instructions interleave commands across ranks, enabling up to 8× bandwidth expansion for 64B embedding vectors with low spatial locality.The mechanism addresses the C/A bandwidth limit that otherwise restricts concurrent rank activation.

C. Programming Model and Execution Flow

RecNMP uses a heterogeneous programming model in which CPU host code launches NMP kernels that execute embedding pooling near memory. The memory controller packages, schedules, decodes, and routes these instructions while preserving packet-level ordering constraints.

Programming model: The programming model divides applications into CPU host calls and NMP kernels offloaded to RecNMP processing units.NMP kernels compile into instruction packets, and their results return to the host CPU.
Execution flow: Host and NMP memory regions are marked separately to simplify coherence between CPU caches and RecNMP.Host-visible arrays such as Indices and Lengths remain cacheable by the CPU.
Packetization: A single SLS kernel can be split into multiple packets containing pooling operations tagged by PsumTag.The design uses a 4-bit PsumTag, which determines the maximum number of poolings per packet.
Memory-controller execution: The NMP-enabled memory controller queues concurrent packets, maps addresses, and reorders instructions within each packet rather than across packets.The NMP extension adds scheduling and arbitration logic to avoid out-of-order packet scheduling under FR-FCFS.

D. HW/SW Co-optimization

RecNMP combines memory-side caching with table-aware scheduling and hot-entry profiling to preserve embedding-table locality under concurrent recommendation traffic. These optimizations improve cache effectiveness and reduce memory latency and energy.

Optimization overview: 33.7% memory latency improvement and 45.8% memory access energy saving result from locality-aware optimization.The optimization combines RankCache with table-aware packet scheduling and hot-entry profiling.
Table-aware scheduling: Table-aware packet scheduling prioritizes packets from the same batch and embedding table to preserve intra-table temporal locality.This counters interference from lookups to multiple tables and co-located recommendation models.
Hot-entry profiling: Hot-entry profiling marks frequently reused embedding entries with LocalityBit so RankCache stores them while other requests bypass the cache.The hint is generated from the indices used in an NMP kernel.
Cache effectiveness: The combined optimization makes measured cache hit rates closely approach the ideal infinite-cache case across embedding tables.This remains true even for the limited-locality T8 trace.

IV. EXPERIMENTAL METHODOLOGY

The evaluation combines real-system measurements on production-scale recommendation models with cycle-level DDR4 and RecNMP simulations. It examines scalability, placement, caching, scheduling, profiling, and comparisons with other near-memory systems.

Methodology: The evaluation uses real server-class CPU experiments and cycle-level memory simulations to measure embedding acceleration and end-to-end effects.The simulation models address mapping, packet generation, locality optimization, DRAM, RankCache, and rank-NMP logic.
Overall results: RecNMP accelerates end-to-end personalized recommendation inference by up to 4.2×.The evaluation compares optimized RecNMP with a host baseline and state-of-the-art NMP designs.
Scalability: SLS latency scales linearly with active ranks, while more pooling operations per NMP packet further reduce latency.The study evaluates 1 × 2, 1 × 4, 2 × 2, and 4 × 2 DIMM-by-rank configurations.
Placement: Page coloring achieves 1.96×, 3.83×, and 7.35× speedup over the DRAM baseline in 2-, 4-, and 8-rank systems.The technique allocates an entire embedding table to the same rank to address load imbalance.
Optimization results: A 128KB RankCache improves latency by 14.2%, while same-table and same-batch scheduling adds 15.4% improvement.The final schedule-plus-profile configuration reduces cache contention by using software-provided cacheability hints.
Cache sizing: A 128KB RankCache is the optimal design point because larger caches provide marginal hit-rate gains while increasing access latency.An 8KB cache reaches only a 24.9% hit rate in the reported sweep.
Trace sensitivity: RecNMP extracts 40% more performance from production traces than from fully random traces.The comparison highlights RecNMP's ability to exploit differing amounts of data reuse.

B. FC Operator Speedup

RecNMP improves co-located FC performance by offloading SLS work and reducing cache contention, with benefits depending on FC size, co-location, and pooling.

Offloading SLS with RecNMP reduces cache contention for co-located FC operators.SLS data can evict reusable FC weights from the cache hierarchy, degrading FC performance.
12% to 30% FC performance improvement occurs for larger FCs whose weights exceed L2-cache capacity.These weights reside mainly in the LLC cache, making them more exposed to cache contention.
∼4% relative improvement occurs for smaller FCs whose working sets fit inside the L2 cache.The cited examples include all BottomFC and RM1’s TopFC operators.
FC degradation varies with FC size, co-location degree, and pooling values.Baseline FC performance worsens with larger weights, higher co-location, and higher pooling values.

C. End-to-end Model Speedup

RecNMP’s end-to-end benefits increase when models spend more time in SLS operators, while co-location trades latency for throughput. The system also reduces memory energy use through near-memory accumulation and caching.

Models spending more time in SLS operators receive higher end-to-end speedup, with RM2-large achieving the highest model-level speedup.End-to-end speedup weights the speedups of SLS and non-SLS operators across four representative model configurations.
Larger batch sizes generally increase model-level speedup because accelerated SLS operators occupy a larger share of execution time.Figure 18(b) reports single-model speedup across different batch sizes.
Co-location generally increases system throughput at the cost of degrading latency.Production-trace locality improves performance over random traces, but its bonus diminishes as co-location increases because of cache interference.
45.8% memory energy savings are achieved versus the baseline DRAM system.Savings come from local accumulation near DRAM, reduced latency leakage, and fewer DRAM accesses enabled by memory-side caching and RankCache optimization.
RecNMP’s processing-unit area and power overhead scale linearly across multiple DIMM ranks while embedding speedup scales linearly as well.The design is described as compatible with existing DRAM devices and accommodable within a DIMM’s area and power budget.

VI. RELATED WORK

Prior work has characterized recommendation models and explored near-memory or memory-oriented acceleration, but RecNMP targets production-scale sparse embeddings with a practical commodity-DRAM design.

Recommendation models have storage, memory-bandwidth, and compute requirements distinct from CNNs, RNNs, and FCs.RecNMP reports a locality study using production-scale models and representative embedding traces.
3D- and 2.5D-stacked near-memory schemes are poorly suited to production recommendation models because of limited 16−32GB capacity and high ownership cost.Production recommendation models require embedding capacities in the tens to hundreds of gigabytes.
NVM-based embedding techniques improve effective read bandwidth by 2−3× to 2.3GB/s, remaining below typical DRAM bandwidth of 76.8GB/s.The passage contrasts these results with the bandwidth demands of larger personalized recommendation embeddings.
TensorDIMM relies on high spatial locality in large embedding vectors, limiting its applicability to the irregular accesses targeted by RecNMP.Its address mapping interleaves consecutive 64B across DIMM modules, so performance scales at the DIMM level.
RecNMP combines production-model characterization with a lightweight, scalable, commodity-DRAM-compliant solution for sparse embedding acceleration.The reported evaluation reaches up to 9.8× SLS performance improvement and up to 4.2× system throughput improvement.