Source-linked AI summary
Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implications
Jongsoo Park, Maxim Naumov, Protonu Basu, Summer Deng, Aravind Kalaiah, Daya Khudia, James Law, Parth Malani, Andrey Malevich, Satish Nadathur, Juan Pino, Martin Schatz, Alexander Sidorov, Viswanath Sivakumar, Andrew Tulloch, Xiaodong Wang, Yiming Wu, Hector Yuen, Utku Diril, Dmytro Dzhulgakov, Kim Hazelwood, Bill Jia, Yangqing Jia, Lin Qiao, Vijay Rao, Nadav Rotem, Sungjoo Yoo, Mikhail Smelyanskiy
TL;DR
Data-center deep-learning inference is growing while model characteristics change quickly, limiting the usefulness of hardware optimized for older workloads. The paper characterizes Facebook inference models, evaluates optimizations on existing systems, and derives hardware and software co-design requirements. It identifies high embedding-memory demands, matrix and vector computation, small-batch inference needs, half-precision support, and a gap between commonly studied models and data-center workloads.
Problem
Growing and rapidly changing DL inference workloads make it difficult to optimize hardware effectively for current and future data-center models.
Method
The paper characterizes Facebook’s current and projected inference workloads, profiles operator performance, and evaluates individual-operator and whole-graph optimizations.
Results
The characterization identifies high embedding-memory bandwidth and capacity, powerful matrix and vector engines, large on-chip memory, half-precision support, and a gap between studied and deployed workloads.
Takeaways & Limitations
DL inference hardware and its software ecosystem should be co-designed around current and future model characteristics, including benchmarking, interfaces, compilers, and graph scheduling.
Takeaways & Limitations
Embedding storage on NVM is constrained by low bandwidth and poor locality, while current x86 systems lack VNNI instructions for higher-throughput int8 accumulation.
Abstract
from arXiv · showhide
The application of deep learning techniques resulted in remarkable improvement of machine learning models. In this paper provides detailed characterizations of deep learning models used in many Facebook social network services. We present computational characteristics of our models, describe high performance optimizations targeting existing systems, point out their limitations and make suggestions for the future general-purpose/accelerated inference hardware. Also, we highlight the need for better co-design of algorithms, numerics and computing platforms to address the challenges of workloads often run in data centers.
1. Introduction
Deep learning inference is expanding across Facebook services, increasing data-center power and compute demands. Rapid model evolution makes hardware optimization difficult, motivating characterization of current and future workloads and hardware requirements.
- Motivation: DL models must scale visual, speech, and language services to billions of Facebook users.
- Motivation: DL inference demand is expected to become a significant share of future data-center demand as applications expand and models improve.Improved model quality is often associated with higher compute and memory requirements.
- Challenge: Hardware optimized for older models can become inefficient because DL application characteristics change rapidly.AlexNet is given as an example of a once-relevant model that no longer represents current computer-vision systems.
- Approach: The paper characterizes current systems and projected applications, emphasizing CPUs because many inference workloads require flexibility, availability, and low latency.
- Implications: The characterization suggests high embedding memory bandwidth and capacity, powerful matrix and vector engines, large on-chip memory, and half-precision support for future DL hardware.
2. Characterization of DL Inference
Facebook’s inference workloads span recommendation, computer vision, and language applications with rapidly changing models and diverse computational characteristics. Their performance depends on memory capacity and bandwidth, matrix shapes, operator mix, and workload-specific constraints.
- Inference workloads cover personalized recommendations, visual understanding, and natural-language processing, including ranking, detection, video, translation, and related tasks.
- Ranking and Recommendation: Embedding tables can contain billions of parameters and require memory capacities exceeding tens of GBs, while embedding lookups often dominate recommendation-model execution.Recommendation models combine sparse-feature embeddings with dense features and higher-order interactions; their execution is often memory-bandwidth bound.
- Computer Vision: 43M parameters and 8B multiply-adds characterize ResNeXt-101-32x4d, whereas ResNeXt-101-32x48d uses 829M parameters and 153B multiply-adds while reaching 85.4% ImageNet-1K Top-1 accuracy.The larger model improves validation accuracy by 4% and relies on group convolutions.
- Computer Vision: Object detection uses substantially larger inputs than classification, with a typical 3×800×600 input being 9.5× larger than a typical classification input.Detection also batches final convolutional blocks over multiple region proposals, increasing computational cost.
- Computer Vision: Larger activations in detection and video workloads pressure on-chip memory capacity and off-chip bandwidth, while small batches make batching difficult for complex vision tasks.Distorting aspect ratios to a fixed 224×224 shape can cause huge accuracy drops in object detection.
- Compute Characteristics: Recent models benefit from larger on-chip memory, but performance can remain sensitive to on-chip bandwidth when layers have few operations per activation.For ShuffleNet and ResNeXt-3D, assuming only 2K operations per weight would incorrectly suggest that 1 TB/s on-chip bandwidth saturates 100 TOP/s compute.
- Computation Kernels: Common inference kernels include fully connected layers, embedding lookups, tensor manipulations, and narrow matrix multiplications from group or depth-wise convolutions.Tensor manipulation includes concatenation, splitting, and slicing; FC and group/depth-wise convolution shapes can be tall and skinny or narrow.
- Compute Characteristics: Standard benchmarks may miss data-center inference behavior because they emphasize larger batches and omit small reduction dimensions and large activation tensors.These omissions are especially relevant to depth-wise convolutions, image detection, and video models.
3. Performance Optimizations
The paper characterizes rapidly changing data-center inference workloads and optimizes CPU execution through reduced-precision kernels, operator monitoring, quantization, and whole-graph fusion. These techniques improve performance while exposing accuracy, processor, interface, and library constraints.
- CPU inference remains important because Facebook workloads require availability, flexibility, and low latency while serving fluctuating demand.
- Fleet-wide monitoring combines operator observers, layer specifications, hardware counters, telemetry, and bottleneck-tuned benchmarks.
- Data-center reduced-precision inference is constrained by accuracy requirements, limited CPU support, immature libraries, and absent native instructions.
- FBGEMM provides reduced-precision linear algebra optimized for deep-learning inference on general-purpose processors.
- Outlier-aware quantization makes the outlier matrix sparse, so its 32-bit-accumulation computation contributes only a small fraction of total time.
- ResNet-50 int8 quantization achieves 75.6% Top-1 and 92.8% Top-5 ImageNet accuracy, with drops of 0.3% and 0.1% versus fp32.
- DL inference interfaces need convolution, fused post-processing, and reduced-precision pipelines beyond standard BLAS GEMM abstractions.
- Whole-graph optimization targets operator fusion, data movement elimination, scheduling, and threading, with manual and compiler-generated fused kernels.
4. Application Driven HW Co-design Directions
The paper frames server-side inference optimization as co-design across models, numerics, and hardware. Rapid workload change and diverse compute and memory demands require up-to-date performance modeling and adaptable architectures.
- Server-side DL optimization should co-design models, numerics, and hardware, with the process driven by rapidly changing and diverse models.
- Workload Diversity: Diverse matrix shapes and long-tail operators motivate hardware that combines matrix multiplication engines with powerful vector engines.
- Memory Systems: Larger activations and models increase pressure on on-chip memory capacity and off-chip memory bandwidth.
- System Design: A hypothetical 100 TOP/s accelerator may require several GB/s of PCIe or network bandwidth unless image decompression occurs on the accelerator or host.
- DL Model and Hardware Co-design: When performance is bandwidth bound, increasing FLOPs without increasing bandwidth consumption can improve accuracy while maintaining performance.
5. Related Work
Related work spans holistic data-center workload characterization, specialized accelerator platforms, energy-efficient reduced-precision and sparse computation, and benchmarking efforts. The paper distinguishes its focus on inference software and hardware co-design.
- Hazelwood et al. characterize inference, training, and data acquisition, whereas this paper focuses on DL inference characteristics for software and hardware co-design.
- NVIDIA GPUs, Google TPUs, and Microsoft Brainwave are established data-center ML platforms, but their generality for diverse workloads remains uncertain.
- Prior energy-efficiency techniques exploit reduced precision, sparsity, and early stopping in inference accelerators.
- Emerging benchmark suites, including MLPerf, support system-level performance measurement across software platforms and device types.
6. Conclusion
The paper concludes that rapidly evolving models and rising computation and memory requirements make co-designing DL inference hardware for present and future workloads challenging. Its characterization and optimization experience informs hardware, benchmarking, linear-algebra, compiler, and graph-optimization discussions.
- Co-designing inference hardware for current and future DL models is important but challenging as models and resource requirements evolve rapidly.
- The paper contributes experience relevant to inference hardware design and the software ecosystem supporting benchmarking, DL linear algebra, and whole-graph optimization.