Source-linked AI summary
Distributed Inference Performance Optimization for LLMs on CPUs
Pujiang He, Shan Zhou, Changqing Li, Wenhuan Huang, Weifei Yu, Duyi Wang, Chen Meng, Sheng Gui
TL;DR
LLM deployment on resource-limited hardware is challenged by memory and inference-performance demands. The paper proposes a oneCCL-based distributed inference optimization solution for CPUs, achieving 140 ms per output token for a 72B-parameter model. The authors position this result as enabling efficient CPU inference in constrained environments.
Problem
LLM deployment faces substantial memory consumption, stringent latency targets, and long sequence lengths, particularly in low-resource environments.
Source: §1 · p. 1Method
The paper proposes three distributed CPU inference optimizations using oneCCL: reduced synchronization, one-time decoder-layer synchronization, and zero-copy communication.
Source: §2 · p. 2, §2.1 · p. 2, §2.2 · p. 2Results
Takeaways & Limitations
Abstract
from arXiv · showhide
Large language models (LLMs) hold tremendous potential for addressing numerous real-world challenges, yet they typically demand significant computational resources and memory. Deploying LLMs onto a resource-limited hardware device with restricted memory capacity presents considerable challenges. Distributed computing emerges as a prevalent strategy to mitigate single-node memory constraints and expedite LLM inference performance. To reduce the hardware limitation burden, we proposed an efficient distributed inference optimization solution for LLMs on CPUs. We conduct experiments with the proposed solution on 5th Gen Intel Xeon Scalable Processors, and the result shows the time per output token for the LLM with 72B parameter is 140 ms/token, much faster than the average human reading speed about 200ms per token.
1 INTRODUCTION
LLM inference faces memory, latency, and sequence-length challenges, especially in low-resource environments. The paper proposes an efficient distributed CPU solution and reports 140 ms per output token for a 72B-parameter model.
1 INTRODUCTION
- LLM deployment is constrained by substantial memory consumption, stringent latency targets, and long sequence lengths.These challenges hinder practical applications in low-resource environments.Source: §1 · p. 1
- The paper targets CPU-based distributed inference because existing optimization solutions are primarily designed for GPUs.The proposed direction is motivated by cost savings, efficient hardware usage, scalability, and low-latency inference.Source: §1 · p. 1
- 140 ms per output token was achieved for a 72B-parameter LLM using the proposed distributed CPU solution.The result was obtained on 5th Gen Intel Xeon Scalable Processors.Source: §1 · p. 1
2 APPROACH
The approach uses oneCCL and three communication optimizations: reducing synchronization payloads, limiting synchronization frequency, and eliminating communication-related memory copies.
2 APPROACH
- The solution leverages oneCCL to optimize communication cost for distributed LLM inference on CPUs.oneCCL provides a unified communication API compatible with various hardware accelerators.Source: §2 · p. 2
2.1 IMPROVE SCALABILITY BY MINIMIZING SYNCHRONIZATION
- Workers broadcast token IDs and compute top-k locally before the end-of-round reduction to reduce synchronization overhead.The implementation is presented in Figure 1.Source: §2.1 · p. 2, Figure 1 · p. 2
2.2 ONE-TIME SYNCHRONIZATION
- GPT-J and Falcon can perform one synchronization per decoder layer because their attention and feed-forward sections run in parallel.The one-time synchronization design is shown in Figure 2.Source: §2.2 · p. 2, Figure 2 · p. 2
2.3 MINIMIZE MEMORY COPY
- The computation module writes results directly into the communication module’s location during its last operation before communication.This produces a zero-copy implementation, illustrated in Figure 3.Source: §2.3 · p. 3, Figure 3 · p. 3
3 EXPERIMENT RESULTS
Experiments evaluate the proposed solution with Qwen-72B on four Intel Xeon Scalable Processors. With 512-token inputs and batch size 1, generation took 140 ms per output token.
3 EXPERIMENT RESULTS
- 140 ms/token was measured for next-token generation of Qwen-72B on 4 Intel Xeon Scalable Processors 8575C.Each device had one socket with 48 cores; the measurement used input size 512 tokens and batch size 1.Source: §3 · p. 3
4 CONCLUSION
The paper presents an efficient oneCCL-based distributed inference optimization solution for CPUs and reports 140 ms per-token generation latency.
4 CONCLUSION
- 140 ms per-token generation latency was reported for the proposed oneCCL-based distributed CPU inference solution.Future work aims to support more CPU varieties and contribute to open-source distributed LLM inference.Source: §4 · p. 3