PaperlayerView exact revision on arXiv

Source-linked AI summary

RecSSD: Near Data Processing for Solid State Drive Based Recommendation Inference

Mark Wilkening, Udit Gupta, Samuel Hsia, Caroline Trippel, Carole-Jean Wu, David Brooks, Gu-Yeon Wei

arXiv:2102.00075v1cs.ARcs.LG

TL;DR

Large recommendation embedding tables demand costly memory, while SSDs offer greater capacity but slower access. RecSSD offloads embedding operations to SSDs and combines this with caching; it reports lower end-to-end inference latency than SSD baselines, with results varying by comparison system.

  • Problem

    Large embedding tables require substantial memory, and although SSDs offer higher capacity than DRAM, their slower access and coarse block transfers challenge sparse recommendation lookups.

    Source: §3.1 · p. 3, §3.4 · p. 5

  • Method

    RecSSD moves embedding-table operations, including gather and aggregation, into SSDs and combines near-data processing with host-side and SSD-side caching.

    Source: §1 · pp. 1–2, §8 · p. 11

  • Results

    RecSSD reduces end-to-end inference latency by 4× versus off-the-shelf SSD systems and by up to 2× atop highly optimized hybrid DRAM-SSD systems.

    Source: §1 · p. 2, §8 · p. 11

  • Takeaways & Limitations

    A real OpenSSD implementation demonstrates the feasibility of SSD-based recommendation inference, with performance described as comparable to DRAM-based memories.

    Source: §1 · p. 2, §8 · p. 11

    The prototype is limited to single-model, single-SSD systems and evaluates direct request latencies rather than latency-bounded throughput.

    Source: §5 · p. 8

Abstract

from arXiv · show

Neural personalized recommendation models are used across a wide variety of datacenter applications including search, social media, and entertainment. State-of-the-art models comprise large embedding tables that have billions of parameters requiring large memory capacities. Unfortunately, large and fast DRAM-based memories levy high infrastructure costs. Conventional SSD-based storage solutions offer an order of magnitude larger capacity, but have worse read latency and bandwidth, degrading inference performance. RecSSD is a near data processing based SSD memory system customized for neural recommendation inference that reduces end-to-end model inference latency by 2X compared to using COTS SSDs across eight industry-representative models.

1 INTRODUCTION

Large embedding tables make recommendation inference memory-intensive, while DRAM capacity is costly and SSDs have slower reads and writes. RecSSD moves embedding computation into SSDs and reports lower embedding-operation and end-to-end latency.

1 INTRODUCTION

  • In Facebook’s datacenter, recommendation accounts for 10× the infrastructure capacity of other AI-driven applications [20] [28] [30]; embedding tables can consume up to hundreds of GBs [20] [32], while SSDs offer orders of magnitude more capacity but slower reads and writes.Source: §1 · p. 1
  • RecSSD offloads the entire embedding-table operation, including gather and aggregation, to SSDs to use internal bandwidth and reduce host–SSD communication overhead.The reported embedding-operation latency improvement is 4× compared with baseline SSD.Source: §1 · p. 1
  • Host-side and SSD-side caching complement near-data processing by exploiting recommendation-query locality, and the design was implemented on Cosmos+OpenSSD with Micron UNVMe.The implementation targets a real system and uses the existing SSD hardware.Source: §1 · p. 2
  • Up to 2× end-to-end inference latency reduction is reported on highly optimized hybrid DRAM-SSD systems; the real OpenSSD prototype showed a 2× improvement.The reported embedding-operation latency improvement is up to 4×.Source: §1 · pp. 1–2

2 BACKGROUND

Recommendation models combine compute-oriented fully connected layers with memory-intensive embedding lookups for categorical features. Embedding tables impose large-capacity, irregular-access, low-compute-intensity demands, while SSDs offer dense storage but bring latency, bandwidth, and access-granularity trade-offs.

2.1 Recommendation Systems

  • Fully connected layers perform regular multiply-accumulate operations, whereas embedding references make irregular memory lookups that stress bandwidth.Recommendation models vary in operator composition and in the sizes of their fully connected layers and embedding tables.Source: §2.1 · p. 2, Figure 1 · p. 2
  • Embedding operations gather ID-selected vectors for categorical inputs and aggregate them by summing, averaging, concatenating, or matrix multiplication.Embedding vectors typically contain 16, 32, or 64 learned features, and inference requests are often batched.Source: §2.1 · p. 2
  • Embedding tables can require up to ~10GB each and TBs in aggregate; sparse, user-varying IDs create irregular access, while low compute intensity makes flash a potential target.The passages also state that embedding-table compute intensity is much lower than in other deep-learning workloads, limiting efficient execution on some proposed accelerators.Source: §2.1 · pp. 2–3

2.2 Flash memory systems

  • The SSD’s FTL runs on an on-device microprocessor, maps host logical blocks to flash, and manages functions including garbage collection and wear leveling.The SSD architecture uses channel-organized flash controllers and an on-board DRAM buffer for DMA transfers.Source: §2.2 · p. 3, Figure 2 · p. 3
  • Flash offers orders-of-magnitude greater storage density than DRAM at around 4–8× lower cost per bit, but each package provides 32–40MB/sec and writes incur O(ms) latency.SSDs aggregate bandwidth by striping logical blocks across multiple flash packages and conducting accesses in parallel.Source: §2.2 · p. 3

3 SSD STORAGE FOR NEURAL RECOMMENDATION

The section characterizes why SSDs are challenging for recommendation embeddings: coarse block transfers penalize sparse accesses, while reuse is concentrated in hot regions but varies across tables. SSDs can impose severe embedding-operation latency, motivating near-data processing and locality-aware optimizations.

3.1 Embedding access patterns in production models

  • A few hundred pages capture 30% of embedding-table reuse, while a few thousand pages can capture more than 50%.The production access patterns follow a power law, with reuse concentrated in a few hot regions.Source: §3.1 · p. 3, Figure 3 · p. 4
  • With a 16 MB page cache per embedding table, more than 50% of reuse is achieved across all analyzed tables, despite hit rates otherwise ranging from under 10% to over 90%.The measurements use a 16-way LRU cache with 4 KB pages; the best per-table capacity may vary.Source: §3.1 · p. 4, Figure 4 · p. 4

3.2 Performance of individual embedding operations

  • Three orders of magnitude longer latency separates SSD from optimized DRAM SparseLengthsSum for the specified one-million-row, 32-dimension table configuration.The comparison uses 80 lookups per table; SSD random-read bandwidth is 10K IOPS or 10 MB/s versus 1 GB/s on DRAM.Source: §3.2 · p. 4, Figure 5 · p. 4

3.3 Performance of end-to-end recommendation models

  • At batch size 64, storing embedding tables in SSD rather than DRAM increases latency for WND, MTWND, DIEN, and NCF by 1.01×, 1.01×, 1.09×, and 1.01×, while embedding-dominated models degrade by several orders of magnitude.The eight evaluated recommendation models include MLP-dominated models, whose execution time remains largely unaffected between DRAM and SSD.Source: §3.3 · p. 5

3.4 Opportunities for acceleration and optimization

  • NDP targets the SSD performance gap for embedding-dominated models by co-locating lightweight gather-reduce computation to reduce PCIe round trips and improve SSD bandwidth utilization.The work also frames SSD capacity as an opportunity for MLP-dominated models, whose execution time is largely spent on matrix operations.Source: §3.4 · p. 5

4 RECSSD DESIGN

RecSSD moves embedding-table operations into the SSD FTL and adds scheduling and caching for multi-page requests and temporal locality. Its design uses the existing NVMe protocol and commodity SSD hardware.

4 RECSSD DESIGN

  • RecSSD implements embedding operations in the SSD FTL, with custom host-side NDP drivers, while retaining compatibility with commodity SSD hardware and existing NVMe protocol.The SSD microprocessor must handle reduction computation without becoming overly delayed in its scheduling functions.Source: §4 · p. 5, §4.1 · p. 5, §4.3 · p. 7

4.1 Mapping Embedding Operations to the FTL

  • An SLS scheduling layer feeds page requests from multiple in-flight SLS operations to the existing page scheduler, which continues to maximize internal memory parallelism.This layer adds buffers and request queues above the page-level scheduler to support multi-page SLS operations.Source: §4.1 · pp. 5–6
  • Each SLS operation uses a pending-request buffer and a specialized embedding cache; the buffer tracks configuration, request status, pending page requests, host requests, and results.The components support the multi-page operation shown in Figure 7.Source: §4.1 · p. 6, Figure 7 · p. 7
  • The FTL preprocesses SLS inputs into per-page embedding requests, checks the embedding cache, then extracts and accumulates required embeddings as Flash pages arrive.Cache hits can avoid Flash page reads; each completed page updates the corresponding result embedding and completion counters.Source: §4.1 · p. 6

4.2 Exploiting temporal locality in embedding operations

  • SLS worker threads overlap embedding I/O with neural-network computation; host DRAM caching statically partitions profiled hot embeddings from those kept on SSD.SLS worker count matches independent I/O queues, while neural-network workers match available CPU resources.Source: §4.2 · p. 6
  • The SSD-side cache is direct-mapped because the FTL’s limited DRAM and simple CPU make the overhead of maintaining LRU or pseudo-LRU state a design concern.The FTL environment favors compact code with low computational overhead and consistent performance.Source: §4.2 · pp. 6–7

4.3 Feasibility and Implementation: NDP SLS Interface

  • The interface sends embedding parameters with a write-like command and retrieves results with a read-like command, associating each pair through a request ID encoded in the starting logical block address.The input-ID list is sorted to improve processing on the SSD’s less powerful CPU.Source: §4.3 · p. 7

5 METHODOLOGY AND IMPLEMENTATION

RecSSD is implemented and evaluated on an OpenSSD platform with a compatible Micron UNVMe interface, using eight recommendation models and synthetic traces with varied locality. The prototype measures direct request latency on a single-model, single-SSD system.

OpenSSD

  • The evaluation uses a 2TB Cosmos+ OpenSSD with a customizable flash controller and FTL, implementing RecSSD in FTL firmware without hardware changes.The platform provides a fully functional NVMe Flash SSD.Source: §5 · p. 7

Micron UNVMe

  • The Micron UNVMe driver adds two commands using existing NVMe read/write structures and an unused command bit, retaining host-controller compatibility.A low-latency userspace library polls read-command completion and uses the maximum number of threads and command queues.Source: §5 · pp. 7–8

Neural recommendation models

  • The evaluation integrates RecSSD's SparseLengthsSum embedding operations with eight industry-representative DeepRecInfra models through a Python-level CTypes interface.The paper notes these operations could later be ported into a custom Caffe2 operator.Source: §5 · p. 8

Input traces and Performance Metrics

  • Synthetic traces vary embedding-access locality using K values of 0, 1, and 2, producing 13%, 54%, and 72% unique accesses, respectively.The evaluation assumes one embedding vector per 16KB SSD page, 1 million vectors per table, and host DRAM caches of up to 2K entries per table.Source: §5 · p. 8
  • Because the prototype is limited to a single model and SSD, the study reports direct request latencies rather than latency-bounded throughput.Latency results are averaged across batches to represent steady-state behavior.Source: §5 · p. 8

Physical Compute Infrastructure

  • Experiments run on a quad-core Intel Skylake desktop with 64GB DRAM; the prototype SSD supports eight channels and up to 1.4GB/s sequential-read throughput.The paper notes newer SSD systems will have higher throughput.Source: §5 · p. 8

6 EVALUATING RECSSD

RecSSD achieves up to 2× speedup over baseline SSD for recommendation inference, with results varying by access locality, caching, and model parameters. Its strongest advantages arise when SSD pages must be accessed and host caching is comparably effective.

6 EVALUATING RECSSD

  • Up to 2× speedup over baseline SSD is reported for recommendation inference overall.The evaluation includes embedding-operation microbenchmarks and end-to-end model comparisons.Source: §6 · p. 8

6.1 Accelerating embedding table operations

  • Up to 4× performance improvement over baseline SSD occurs for random, low-locality embedding lookups.RecSSD raises memory-level parallelism by issuing concurrent Flash requests; roughly half the FTL time is spent on Translation.Source: Figure 8 · p. 8, §6.1 · p. 9
  • For sequential, high-locality accesses, conventional SSDs outperform RecSSD because host-side aggregation outweighs RecSSD's reduced NVMe-command overhead.Translation on the dual-core ARM A9 accounts for over half of RecSSD execution time in this setting.Source: Figure 8 · p. 8, §6.1 · p. 9

6.2 Accelerating end-to-end recommendation models

  • Up to 7× speedup over a conventional SSD appears in some end-to-end models with naive RecSSD, while other models show no observable benefit.These results use random input indices without operator pipelining or caching.Source: §6.2 · p. 9, Figure 9 · p. 9

6.3 Exploiting Locality in end to end models

  • With SSD-side caching alone, conventional SSDs do better at high locality, whereas RecSSD wins at low locality and reaches up to 1.5× without host DRAM caching.The comparison spans batch sizes and locality conditions; the baseline uses host-side fully associative LRU caching while RecSSD uses a direct-mapped SSD cache.Source: §6.3 · p. 9, Figure 10 · p. 10
  • With static host partitioning and SSD caching, RecSSD achieves 2× performance improvement when baseline LRU hit rates are relatively low and RecSSD's host hit rates are comparable.At higher batch sizes and more indices per request, the static-partition hit rate approaches 25%.Source: §6.3 · pp. 9–10, Figure 10 · p. 10

6.4 Sensitivity study: Impact of model architecture

  • Absolute embedding-table size does not affect lookup performance in these results, although larger tables motivate SSD storage over capacity-constrained DRAM.Performance depends on access patterns rather than absolute table size.Source: §6.4 · p. 10, Table 1 · p. 10
  • Larger embedding vectors relative to page size reduce RecSSD's relative performance, while more indices per lookup can outweigh the performance loss from increasing table count.The paper attributes table-count overhead to a separate NDP call per table and describes indices-per-lookup as amortizing control overhead and SSD-side aggregation.Source: §6.4 · p. 10

7 RELATED WORK

Related work frames SSDs as increasingly useful for datacenter storage and describes prior near-data processing and recommendation-acceleration efforts. RecSSD combines SSD near-data processing with NAND flash for recommendation inference, trading generality for application-specific simplicity.

SSD systems

  • SSDs offer 2.6× and 3.2× bandwidth per Watt and bandwidth per dollar, respectively, and are used in datacenters as DRAM replacements and for large-scale applications.The passage also notes SSD storage in terabyte-scale recommendation training platforms.Source: §7 · p. 11
  • Prior general SSD near-data processing systems use hardware modifications, complex programming frameworks, and heavily modified drivers; this system trades generality for application-specific simplicity and cost efficiency.Source: §7 · p. 11

Accelerating recommendation inference

  • Prior recommendation work addresses GPU address translation, near-memory processing, and flash embedding storage with caching; RecSSD explores combining near-data processing and NAND flash and evaluates real systems across recommendation workloads.The passage says the prior flash caching techniques can be used alongside RecSSD.Source: §7 · p. 11

8 CONCLUSION

RecSSD offloads embedding-table computation to SSDs and uses host-side and SSD-side caching. The conclusion reports reduced end-to-end latency versus off-the-shelf SSDs and comparable performance to DRAM-based memories.

8 CONCLUSION

  • RecSSD offloads key embedding-table operations to reduce host–SSD round-trip communication and improve internal SSD bandwidth utilization.Source: §8 · p. 11
  • Host-side and SSD-side caching are used with RecSSD to support high-performance embedding-table operations.Source: §8 · p. 11
  • The conclusion reports 4× lower end-to-end neural recommendation inference latency than off-the-shelf SSD systems and comparable performance to DRAM-based memories.The paper says this enables efficient and scalable datacenter neural recommendation inference.Source: §8 · p. 11

A.1 Abstract

RecSSD’s artifacts implement an NDP SLS operator on Cosmos+OpenSSD and provide an NVMe-compatible driver interface. Evaluation uses eight DeepRecInfra recommendation models and DLRM-based synthetic trace generators instrumented with production locality analysis.

A.1 Abstract

  • RecSSD implements a fully functional NDP SLS operator in the open-source Cosmos+OpenSSD system.The implementation is provided in the RecSSD-OpenSSDFirmware repository.Source: §A.1 · p. 11
  • Its interface is implemented within Micron’s UNVMe driver library to maintain NVMe protocol compatibility.The driver library is provided in the RecSSD-UN-VMeDriver repository.Source: §A.1 · p. 11
  • Evaluation uses eight DeepRecInfra recommendation models in Caffe2 and DLRM synthetic trace generators instrumented with production locality analysis.The models and trace generators are included in RecSSD-RecInfra.Source: §A.1 · p. 11

A.2 Artifact check-list (meta-information) •

The artifact checklist specifies the software, model, runtime, and hardware used, along with estimated preparation and experiment times and availability information.

A.2 Artifact check-list (meta-information) •

  • The listed compilation and model tools include GCC, Python3, PyTorch, Caffe2, and DeepRecInfra.The runtime environment is Ubuntu 14.04.Source: §A.2 · p. 11
  • The hardware setup comprises Cosmos+ OpenSSD, two Linux desktop machines, and a remote PDU.Source: §A.2 · p. 11
  • Workflow preparation is estimated at 4–8 hours after hardware acquisition, and experiments at 10+ hours.The software is described as publicly available; the required hardware platform is potentially still purchasable through its original developers.Source: §A.2 · p. 11

A.3 Description

The artifact description covers public repositories, hardware and software dependencies, setup, workflow, and expected evaluation outputs. It also notes that Figures 3 and 4 rely on proprietary industry data and cannot be reproduced with the open infrastructure.

A.3 Description

  • RecSSD is available through several GitHub repositories and a public Zenodo archive.The archive DOI is 10.5281/zenodo.4321943.Source: §A.3.1 · p. 11

A.3.1 How to access.

  • Access requires a Cosmos+ OpenSSD, two Linux desktop-class machines, and a remote PDU; software dependencies include Xilinx SDK 2014.4, OpenSSD firmware and bitstream, Python3, PyTorch, and Caffe2.The models use DeepRecInfra benchmarks and Facebook DLRM trace generation.Source: §A.3.2 · p. 12, §A.3.3 · p. 12, §A.3.4 · p. 12

A.4 Installation

  • Installation replaces the OpenSSD GreedyFTL source with RecSSD firmware, binds the UNVMe driver to the device, and places libflashrec.so where the model runtime can load it.The instructions also require downloading RecSSD-RecInfra and installing Python3 and PyTorch.Source: §A.4 · p. 12

A.5 Experiment workflow

  • The workflow launches OpenSSD firmware, binds the UNVMe driver, generates synthetic locality-pattern traces, then runs model sweeps with baseline SSD or NDP interfaces.Repository documentation and the OpenSSD tutorial provide the detailed steps.Source: §A.5 · p. 12

A.6 Evaluation and expected results

  • Model latency is reported from SSDHost across batches at steady state, while Figure 8’s NDP-request FTL timing is measured and recorded from the SSDDev SDK.Figures 3 and 4 use proprietary industry data and are not reproducible with the open-sourced infrastructure.Source: §A.6 · p. 12

Related papers

Loading 2102.00075v1…