Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implications

Inference demand

• A significant fraction of future data-center power demand is expected to come from DL inference as applications expand and model quality improvements increase compute and memory needs.The paper points to Figure 1 for the projected server demand for DL inference across data centers.Source: §1 · p. 1

Workload change

• DL models change so quickly that hardware optimized for older models can become inefficient for newer workloads, as AlexNet no longer represents today’s computer vision systems.This rapid application change complicates hardware optimization for future workloads.Source: §1 · p. 1

Hardware implications

• The characterization motivates four hardware requirements: high-bandwidth, high-capacity embedding memory; powerful matrix and vector engines; large on-chip memory for small-batch inference; and half-precision floating-point support.These requirements reflect current and projected model characteristics and limitations encountered when optimizing applications on existing platforms.Source: §1 · p. 1

Characterization and optimization

• The authors characterize DL models using direct access to current systems and applications projected to drive future workloads, targeting optimizations mostly at CPUs because inference often needs flexibility, availability, and low latency.They identify a gap between models commonly studied by the systems community and those running in Facebook data centers, with implications for processor design.Source: §1 · p. 1

2.1. Representative Models

• Inference workloads serve personalized feeds, rankings, and recommendations, as well as visual and language content understanding that supports recommendations, integrity, and security.Source: §2.1 · p. 2

2.1.1. Ranking and Recommendation

• Recommendation models combine dense features with sparse embedding lookups; embedding tables can contain billions of parameters and drive memory capacity beyond tens of GBs.Multiple candidates are often scored within 100s ms, enabling batching in FCs, but random embedding accesses make execution memory-bandwidth bound. More signals and larger embeddings may improve quality while increasing memory pressure and the need for distributed training and inference.Source: §2.1.1 · p. 2

2.1.2. Computer Vision

• Computer-vision workloads span classification, detection, and video; ResNeXt-101-32x48d has 829M parameters, requires 153B multiply-adds, and reaches 85.4% ImageNet-1K Top-1 accuracy.Detection uses higher-resolution inputs and many region proposals, while ResNeXt-3D with depth-wise convolutions requires 3× less FLOPs than Res(2+1)D [66]. Larger activations and small preferred batches increase pressure on on-chip memory and off-chip bandwidth.Source: §2.1.2 · pp. 2–3

2.1.3. Language Models

• Language models commonly use encoder-decoder NMT with recurrent GRU or LSTM cells, but recurrence limits parallelism and has motivated convolutional and attention-based alternatives.Larger models and ensembles can improve translation quality, while multilingual models may need more capacity [32] [58]. Sequential beam-search decoding remains typical at inference; small batches suit instant translation, whereas offline localization can benefit from larger batches.Source: §2.1.3 · pp. 3–4

2.2. Compute Characteristics

• Embedding workloads can exceed tens of GBs and have low arithmetic intensity, while larger on-chip memory improves projected performance and bandwidth demands vary across workloads.Embedding lookups are difficult to serve with either high-bandwidth memory, whose capacity is limited, or non-volatile memory, whose bandwidth and access granularity are problematic. Small batches and low operations per activation can also make performance sensitive to on-chip bandwidth, motivating both matrix-matrix and vector engines.Source: §2.2 · pp. 4–5, Figure 3 · p. 4

2.3. Computation Kernels

• FCs are the most time-consuming CPU kernels, but narrow matrix shapes can underutilize matrix engines, and depth-wise convolutions can take comparable time despite few FLOPs.Tensor manipulation, including concatenation, splitting, and slicing, is also a target for whole-graph optimization. Standard benchmarks may miss small batches, small depth-wise-convolution reductions, and large detection or video activations.Source: §2.3 · pp. 5–6

3.1. Data Center Fleet-wide DL Inference Profiling

• Fleet-wide observers combine operator metrics and layer specifications to classify compute- versus bandwidth-bound execution, prioritize optimizations, and validate roofline predictions against sampled telemetry.The fleet’s diverse and changing models motivate continuous monitoring, while bottleneck-specific benchmarks help set optimization goals.Source: §3.1 · p. 6

3.2. Reduced Precision Inference

• Data-center inference favors accurate, compute-intensive models and typically accepts less than 1% accuracy change, but CPUs lack strong reduced-precision hardware support and mature libraries.Current x86 processors support fp16-to-fp32 conversion but not native fp16 computation; available 8-bit multiplication approaches have throughput and accuracy tradeoffs, and VNNI was unavailable in the described microarchitectures.Source: §3.2 · p. 6, §3.2.1 · p. 6

3.2.1. Performance Challenges

• Against MKL fp32, FBGEMM fp16 and i8-acc32 GEMMs reached up to 2× and 4× speedups at low intensity, while i8-acc16 reached 1.7× on ResNet-50.For memory-bound cases, fp16 or i8-acc32 increases arithmetic intensity; for instruction-throughput-bound cases, i8-acc16 uses outlier-aware quantization and periodic spills. The fp16 GEMM also reduced recommendation-model latency by 15%, while i8-acc32 delivered 2.4× overall speedup on Faster-RCNN-Shuffle.Source: §3.2.1 · pp. 6–7, Figure 6 · p. 7

3.2.2. Accuracy Challenges:

• ResNet-50 int8 quantization except softmax achieved 75.6% Top-1 and 92.8% Top-5 ImageNet-1K accuracy, drops of 0.3% and 0.1% versus fp32.The effective techniques included fine-grain, quantization-aware, selective, outlier-aware, and net-aware quantization.Source: §3.2.2 · pp. 7–8

3.2.3. Software Challenges:

• DL inference needs packed-weight interfaces for tall, skinny matrices and native convolution and fused output processing, since standard BLAS packing and im2col can add overhead.The FBGEMM interface accepts a pre-packed weight matrix and supports specialized input packing and output processing.Source: §3.2.3 · p. 8

3.3. Whole Graph Optimization

• Merging batched matrix multiplies with tensor-manipulation operations saved over 10% runtime; those operations account for about 17% of inference CPU time.The authors mine frequent graph subgraphs and rank fusion opportunities using roofline estimates of performance before and after fusion.Source: §3.3 · p. 9

4. Application Driven HW Co-design Directions

• Co-design should span DL models, numerics, and hardware platforms, with rapidly changing and diverse models driving the process.The numerics include quantization, Winograd/FFT convolution, and sparsity.Source: §4 · p. 9
• Diverse matrix shapes, long-tail operators, and conflicting memory demands call for general vector engines, larger on-chip capacity, and high-bandwidth memory support.Designers cannot rely on on-chip capacity alone: future demand is difficult to project and some recommendation models already exceed on-chip capacity while requiring high bandwidth.Source: §4 · p. 9
• 2–3% accuracy drops from quantization or pruning are often too costly; a 32x32d model losing >1% for <2× speedup may be less advantageous than an unquantized 32x16d model.Data-center inference hardware should support per-channel quantization and fp16 or bfloat16 fallback for accuracy-sensitive parts such as some models' last layer.Source: §4 · p. 9
• Disaggregating inference into a separate tier can protect co-located workloads from DL's memory, bandwidth, and power pressure while allowing request pooling for larger batches.Tier design must account for network bandwidth and latency, and compression; a hypothetical 100 TOP/s accelerator would require a few GB/s of PCIe and/or network bandwidth for the listed models unless image decompression is done on the accelerator or same host.Source: §4 · pp. 9–10
• Hardware-aware model design should account for scarce power, on-chip memory, and off-chip bandwidth rather than optimizing floating-point operations alone.When performance is bandwidth-bound, adding FLOPs without increasing bandwidth consumption can improve accuracy while maintaining performance; adding 2× FLOPs to recommendation-model FC layers may be more economical than doubling embedding dimensions for similar accuracy gains. Int8 multiplication uses more than 5× less energy and area than fp16, allowing wider models to recover quantization accuracy while retaining energy savings [44]; transformer NMT is another cited example of hardware-aware trade-offs.Source: §4 · p. 10

• Hazelwood et al. characterize data-center ML across inference, training, and data acquisition, whereas this paper focuses on DL inference characteristics for software/hardware co-design.Source: §5 · p. 10
• NVIDIA GPUs, Google TPUs, and Microsoft Brainwave serve data-center ML [33], but TPU small-batch and group/depth-wise convolution utilization and Brainwave’s generality remain challenges.Brainwave uses dot-product engines, on-chip weight storage, and block floating point; the paper questions whether such architectures efficiently target diverse inference workloads.Source: §5 · p. 10
• SCNN [49], early-stopping convolution, Bit-Fusion [61], OLAccel, and Fleischer et al.’s AI core exploit sparsity or reduced and variable precision for inference efficiency.The paper notes that low-precision and sparse computation must meet server-side inference accuracy requirements for widespread data-center use; OLAccel’s reduced precision with a high-precision fraction is also used in this paper’s optimizations.Source: §5 · p. 10
• Emerging benchmarks include a remotely measured mobile model-zoo framework and MLPerf for server and mobile devices, facilitating system-level comparisons across software platforms and hardware architectures.The passage names TensorFlow and PyTorch as example software platforms.Source: §5 · p. 10

6. Conclusion

• Co-designing inference hardware for current and future deep-learning models is important but challenging as their computation and memory requirements increase.Source: §6 · p. 10
• The authors’ deep-learning inference characterization and optimization experience can provide insights for inference-hardware design.Source: §6 · p. 10
• The paper aims to contribute to discussion of co-design software, including benchmarking suites, deep-learning-optimized linear algebra interfaces, and compilers that optimize and schedule entire model graphs.Source: §6 · p. 10