Source-linked AI summary
PowerInfer-2: Fast Large Language Model Inference on a Smartphone
Zhenliang Xue, Yixin Song, Zeyu Mi, Xinrui Zheng, Yubin Xia, Haibo Chen
TL;DR
Smartphone resource limits restrict LLM deployment to SLMs, compromising their capabilities. PowerInfer-2 uses neuron clusters to schedule computation across mobile NPUs and CPUs and pipeline it with storage I/O. It reports up to 27.8× speedup over existing solutions and serves a 47B model at 11.68 tokens/s with minimal accuracy loss.
Problem
Smartphone computing and memory constraints limit deployment to SLMs, significantly compromising model capabilities.
Source: §1 · p. 1Method
PowerInfer-2 uses neuron clusters to assign dense work to NPUs and sparse work to CPUs while coordinating computation with storage I/O.
Source: §1 · p. 2Results
Up to 27.8× speedup over existing solutions; TurboSparse-Mixtral-47B runs on a smartphone at 11.68 tokens/s with minimal accuracy impact.
Source: §1 · p. 2, §9 · p. 13Takeaways & Limitations
PowerInfer-2 supports smartphone inference for models as large as 47B while maintaining model quality with minimal accuracy loss.
Source: §9 · p. 13
Abstract
from arXiv · showhide
Large language models (LLMs) on smartphones enable real-time AI assistance and privacy-preserving, offline operation. However, resource constraints of smartphones limit current deployments to small language models (SLMs), significantly compromising their capabilities. This paper introduces PowerInfer-2, a smartphone-based framework that enables fast inference for LLMs exceeding the memory capacity. The key insight is decomposing matrix operations into neuron clusters as the basic processing unit, which enables flexible scheduling and efficient I/O-computation pipelining. PowerInfer-2 leverages this neuron-cluster-based design in both computation and storage. For computation, neuron clusters with dense activations are processed on NPU, while sparse clusters use CPU. The storage engine provides a fine-grained pipeline mechanism that coordinates cluster-level computation and I/O operations, enhanced by a segmented neuron cache to reduce I/O activities. PowerInfer-2 achieves up to a 27.8x speed increase compared to state-of-the-art frameworks. PowerInfer-2 is the first system to serve a 47B LLM on a smartphone, achieving 11.68 tokens/s. Notably, these performance improvements preserve model quality with negligible accuracy degradation.
1 Introduction
Smartphone resource and hardware constraints limit capable LLM deployment, while PC-oriented sparse systems lose speed on mobile. PowerInfer-2 uses neuron clusters to coordinate heterogeneous computation and I/O for large-model inference on smartphones.
1 Introduction
- Smartphone LLM deployment offers instantaneous AI assistance, local-data privacy, and reliable operation regardless of network conditions.The growing interest is reflected in mobile LLM initiatives from Apple, Google, and smartphone manufacturers.Source: §1 · p. 1
- Smartphone computing and memory limits confine deployment to SLMs, compromising capabilities; the paper notes that larger models consistently perform better under the scaling law.Gemini Nano is given as an example: 3.25B parameters and less than 2GB of memory, versus tens to hundreds of billions of parameters for models on high-end PCs and servers.Source: §1 · p. 1
- PC-centric sparse systems decode 7B models approximately 70% slower on smartphones, where NPUs lack sparse-operation support and UFS has lower bandwidth than PC NVMe storage.UFS sequential reads are 4GB/s versus 7.5GB/s for NVMe; random reads are 100K versus 1,200K IOPS, contributing to computing-unit idling during I/O.Source: §1 · pp. 1–2
- PowerInfer-2 groups neurons with the same activation pattern into clusters whose size is determined dynamically at runtime based on target computing units.This cluster abstraction is used to coordinate heterogeneous computation and I/O.Source: §1 · p. 2
- Sparsity-Aware Adaptation assigns differently sized clusters to NPUs for dense operations and CPUs for sparse computation, while I/O-Aware Orchestration pipelines cluster computation with storage access.The NPU/CPU cluster ratio changes with sparsity patterns across batch sizes; cluster-level pipelining overlaps computation and I/O and permits specialized access patterns.Source: §1 · p. 2
- 24.6× (up to 27.8×) over llama.cpp and 3.84× average speedup (up to 4.63×) over LLMFlash on OnePlus 12; TurboSparse-Mixtral-47B runs at 11.68 tokens/s with minimal accuracy impact.The evaluation also reports that PowerInfer-2 uses 40% less memory for 7B models while matching llama.cpp and MLC-LLM speed.Source: §1 · p. 2
2 Background and Motivation
LLM inference combines parallel prompt processing with autoregressive decoding, while activation sparsity shifts with batch size. Smartphone compute and storage characteristics expose limitations in existing inference frameworks, especially when model weights are offloaded.
2.1 LLM Inference Background
- Prefill processes prompt tokens in parallel, then autoregressive decoding generates tokens sequentially until the desired length or an end-of-sequence token.A decoder-only transformer layer contains attention and FFN components; FFNs account for about 80% of parameters in several modern models.Source: §2.1 · p. 2
2.2 LLM Inference with Dynamic Sparsity
- Highly activated neurons rise from less than 1% at batch size 1 to approximately 75% at batch size 32, while single-batch activations are sparser and more dispersed.Batch size also fluctuates during decoding as sequences terminate at different times.Source: §2.2 · p. 3, Figure 2 · p. 3
2.3 Mobile Hardware Characteristics
- Smartphones differ from PCs in their heterogeneous compute units and storage architectures, both of which directly affect LLM inference.The paper examines these differences through mobile CPU, GPU, NPU, shared-memory, and UFS characteristics.Source: §2.3 · p. 3
2.3.1 Computing Unit Characteristics
- Mobile CPUs handle sparse matrix-vector computations well, while NPUs excel at large-batch dense work; mobile GPUs lag in the tested operations.On OnePlus 12, CPU-only and NPU-only matrix multiplication reached 43.9GB/s and 56GB/s memory bandwidth, while using both reached 59.6GB/s.Source: §2.3.1 · pp. 3–4, Figure 3 · p. 3
2.3.2 Storage I/O Characteristics
- UFS random-read throughput depends on block size, data range, and the CPU core handling I/O, while its single command queue limits concurrency.Using multiple cores for I/O can degrade performance by up to 40% due to command-queue contention.Source: §2.3.2 · p. 4, Table 1 · p. 4
2.4 Limitations of Existing Solutions
- 89% decoding speed reduction for PowerInfer and 82% decline for LLMFlash occurred when 50% of Mistral 7B FFN weights were offloaded; I/O consumed 82% and 77% of total latency, respectively.CPU utilization fell to 41% during weight loading.Source: Table 2 · p. 4, §2.4 · pp. 4–5
3 PowerInfer-2 Overview
PowerInfer-2 organizes inference around adaptive neuron clusters and two principles: matching work to mobile hardware and coordinating computation with storage I/O. Offline planning analyzes model activations and device capabilities to produce plans for online inference.
3.1 Neuron Cluster and Design Principles
- Neuron clusters group same-layer FFN neurons with similar activation patterns; cluster size adapts to hardware and activation frequency, assigning hot clusters to NPU dense computation and cold clusters to CPU sparse computation.Sparsity-Aware Adaptation dynamically distributes work across CPU and NPU, while I/O-Aware Orchestration uses fine-grained pipelining and differentiated weight I/O strategies.Source: §3.1 · p. 5
3.2 Overall Architecture
- Offline planning analyzes activation patterns across batch sizes and hardware capabilities to create neuron and hardware plans, which guide online workload distribution, caching, and cluster-level computation/I/O overlap.During online inference, the engine adjusts CPU-NPU allocations to dynamic sparsity and retrieves uncached weights using differentiated I/O strategies.Source: §3.2 · p. 5
4 Detailed Design of Online Inference
PowerInfer-2 adapts its compute allocation to inference phase and batch-dependent sparsity, and pairs a segmented neuron cache with cluster-level pipelining and differentiated weight-loading strategies. These mechanisms coordinate mobile compute and storage around neuron clusters.
4.1 Adaptive Neuron Engine
- The adaptive neuron engine selects computation strategies according to inference phase and dynamic sparsity to use mobile XPUs.Its prefill and decoding workflows differ, and decoding adjusts the CPU-NPU share as batch size changes.Source: §4.1 · p. 5, Figure 5 · p. 6
4.1.1 NPU-Centric Prefill
- During prefill, aggregate activations are less sparse, so PowerInfer-2 uses an NPU-centric dense computation workflow while a CPU core loads weights sequentially into shared memory.A big core asynchronously preloads the next layer’s weights into the neuron cache.Source: §4.1.1 · pp. 5–6
4.1.2 Hybrid CPU-NPU Decoding
- During decoding, CPU and NPU process assigned attention heads in parallel, while the NPU handles dense hot FFN neurons and the CPU handles sparse cold neurons.The FFN split shifts toward the NPU as batch size rises; the CPU uses a predictor to select cold neurons for computation.Source: §4.1.2 · p. 6, Figure 5 · p. 6
4.1.3 Dynamic CPU-NPU Adjustment
- As batch size changes, the engine shifts the CPU-NPU computation ratio: at large batches NPU handles about 70% of work, while CPU can handle up to 50% at batch size 1.It prepares multiple NPU graphs offline and loads a replacement asynchronously during attention computation when the allocation must change.Source: §4.1.3 · p. 6
4.2 In-Memory Neuron Cache
- The temperature-based cache separates retained attention weights and KV cache from hot and cold weights, managing hot neurons in clusters and cold neurons individually with LRU eviction.The hot-cold cache ratio changes with batch size, expanding the hot region for larger batches and the cold region for smaller ones.Source: §4.2 · pp. 6–7
4.3 Neuron-Cluster-Level Pipeline
- The neuron-cluster-level pipeline starts computing an in-memory cluster from the next matrix as soon as the current cluster finishes, overlapping I/O across matrices and reducing waiting bubbles.Matrix-level overlap can still leave CPU cores idle while waiting for all neurons in a matrix to become available.Source: Figure 6 · p. 7, §4.3 · p. 7
4.4 Flexible Neuron Loading
- PowerInfer-2 loads attention weights with sequential reads, preloads hot FFN neurons asynchronously, and loads cold neurons on demand with small random reads.Corresponding Gate, Up, and Down neurons have an 80% co-activation probability; for 4-bit models, Up/Down weights are loaded only if the Gate output is non-zero.Source: §4.4 · pp. 7–8
5 Execution Plan Generation
PowerInfer-2 uses offline planning to produce device-specific, batch-adaptive execution strategies. The planner accounts for neuron activations, hardware capabilities, and I/O characteristics when assigning work and preparing runtime configurations.
Neuron Classification
- The planner classifies neurons using activation frequencies across batch sizes and target-device I/O and compute characteristics, informed by runs on more than 10M tokens.It balances hot-neuron assignments against I/O bandwidth and attention-computation time so prefetching can be hidden within attention computation.Source: §5 · p. 8
Batch-Adaptive Planning
- Batch-adaptive planning prepares execution strategies and static NPU graphs for common batch sizes, allowing runtime changes without real-time analysis overhead.The planner classifies more neurons as hot for larger batches and more as cold for smaller batches.Source: §5 · p. 8
Hardware-Aware Optimization
- Hardware-aware optimization profiles NPU, CPU, memory-bandwidth, and storage-I/O performance to adapt execution plans to each mobile platform.The evaluation lists smartphone hardware specifications, including physical DRAM and the maximum memory available to an application.Source: §5 · p. 8, Table 3 · p. 8
6 Implementation
PowerInfer-2 builds on PowerInfer with additional C++ code and runs on rooted Android, using privileged APIs without modifying the kernel.
6 Implementation
7 Evaluation
PowerInfer-2 is evaluated on two OnePlus phones across five models and representative LLM tasks. Under offloading, it outpaces comparison frameworks in decoding and prefill, while its speed varies with memory capacity, task, token cache misses, and activation sparsity.
7.1 Experimental Setup
- The evaluation uses OnePlus 12 and OnePlus Ace 2 phones and five models, comparing against llama.cpp, QNN, MLC-LLM, and LLMFlash across representative LLM tasks.The models range from 7B to TurboSparse-Mixtral-47B; experiments include prefill, decoding, and Best-of-N generation.Source: §7.1 · pp. 8–9
7.2.1 Decoding Performance
- On OnePlus 12, PowerInfer-2 averages 3.84× speedup (up to 4.63×) over LLMFlash and 24.6× (up to 27.8×) over llama.cpp; on OnePlus Ace 2, it averages 2.93× and 14.1×, respectively.FFN weights are limited to 50% of DRAM, except TurboSparse-Mixtral-47B on OnePlus Ace 2, which requires 75% offloading.Source: §7.2.1 · p. 9, Figure 7 · p. 9
- PowerInfer-2 reduces I/O overhead to 13.7%, versus 76.7% for LLMFlash, using a neuron-cluster pipeline that overlaps I/O with computation and flexible bundle loading.Source: §7.2.1 · p. 9, Table 4 · p. 9
7.2.2 Prefill Performance
- At 512-token prompts, PowerInfer-2 is 48.97× faster than LLMFlash, 44.23× faster than llama.cpp, and 1.99× faster than QNN on OnePlus 12.The reported OnePlus Ace 2 improvements are 29.98× and 16.67×; 128-token prompts reach up to 22.47× over LLMFlash.Source: §7.2.2 · p. 9, Figure 8 · p. 10
7.2.3 Performance with Various Memory Capacities
- With 7GB available memory, TurboSparse-Mixtral-47B runs at 2.13 tokens/s; at 19GB, it reaches 11.68 tokens/s, 3.12× faster than LLMFlash and 21.2× faster than llama.cpp.At 7GB, a 400MB neuron cache stores 1.8% of FFN weights; decoding speed scales linearly with memory up to 19GB.Source: §7.2.3 · p. 10, Figure 10 · p. 10
7.2.4 Decoding Consistency Analysis
- Across role-play, multi-turn dialogue, math, and code-generation tasks, PowerInfer-2 maintains at least 11.4 tokens/s; task-level differences reflect activation sparsity.Source: §7.2.4 · p. 10, Figure 11 · p. 10
- TurboSparse-Mixtral-47B’s P99 latency is 40.9% slower than its mean, while its average cache-miss rate is 3.5% and P99 miss rate is 18.9%.The passage attributes latency variation to token activation patterns and neuron swapping between cache and flash storage.Source: §7.2.4 · pp. 10–11, Table 5 · p. 11
7.2.5 SiLU-based LLM Performance
- PowerInfer-2 achieves a 2.4× speedup on SiLU-based LLMs, with a lower gain than on ReLU-based models because more neurons remain active and increase neuron-loading demands.Source: §7.2.5 · p. 11, Table 6 · p. 11
7.3 In-Memory Performance
With sufficient memory, PowerInfer-2 accelerates Bamboo-7B prefill and decoding on smartphones; it also retains performance while offloading half the FFN weights.
7.3 In-Memory Performance
- For prefill, PowerInfer-2 and QNN both exceed 700 tokens/s, reaching a 32× speedup over other frameworks; PowerInfer-2 is slightly slower than QNN.The passage attributes the QNN difference to PowerInfer-2's mixed-precision quantization, a trade-off for accuracy preservation.Source: §7.3 · p. 11
- In decoding, PowerInfer-2 is 2.24× faster than llama.cpp, 2.48× faster than MLC-LLM, and 1.86× faster than QNN.The reported mechanisms are concurrent CPU-NPU computation, increasing bandwidth utilization from 43.9GB/s to 59.6GB/s, and reduced FFN computation from activation sparsity.Source: §7.3 · p. 11, Figure 12 · p. 11
- With 50% of FFN weights offloaded, PowerInfer-2 reduces memory usage by 1.5GB (40%) while maintaining performance comparable to llama.cpp and MLC-LLM, even for models already fitting in memory.Source: §7.3 · p. 11
7.4 Best-of-N Sampling
On Bamboo-7B Best-of-4 sampling, PowerInfer-2 outperforms QNN and its CPU-only variant as the number of parallel candidates falls, using dynamic CPU-NPU dispatch.
7.4 Best-of-N Sampling
- Best-of-N generates multiple response sequences for one prompt and selects the best; the comparison uses Bamboo-7B and three configurations.Source: §7.4 · pp. 11–12
- In Best-of-4 sampling, PowerInfer-2 is 1.84× faster than QNN and 1.28× faster than PowerInfer-2-CPUOnly initially, then retains a 1.42× advantage as candidates finish.Source: Figure 13 · p. 11, §7.4 · p. 12
- When one candidate remains, PowerInfer-2 is 1.1× faster than PowerInfer-2-CPUOnly and 1.77× faster than QNN.The passage attributes the advantage to hybrid CPU/NPU computation; QNN falls below the CPU-only variant as sparsity increases.Source: Figure 13 · p. 11, §7.4 · p. 12
7.5 Performance Breakdown
The evaluation breaks down incremental performance gains, reports PowerInfer-2’s accuracy relative to other frameworks, and compares its energy use.
7.5 Performance Breakdown
- 11.07 tokens/s follows XPU optimization’s additional 15% gain, after pipeline optimization reaches 9.60 tokens/s, Neuron Cache reaches 4.181 tokens/s, Bundle reaches 1.1 tokens/s, and the baseline achieves 0.4 tokens/s.Neuron Cache records a 95% hit rate; pipeline optimization overlaps computation and memory access. Figure 14 shows the decoding-phase breakdown for Bamboo-7B on OnePlus 12 with 50% of FFN weights offloaded to flash.Source: §7.5 · p. 12, Figure 14 · p. 12
7.6 Accuracy Evaluation
- PowerInfer-2 maintains comparable accuracy to llama.cpp and significantly outperforms QNN across models and tasks under INT4 quantization.Because NPUs do not support group-wise quantization, PowerInfer-2 uses INT8 for outlier weights and INT4 per-channel quantization for the remaining weights.Source: §7.6 · p. 12, Table 7 · p. 12
7.7 Energy Consumption
- 0.257 J/token represents 31.1% and 61.8% lower energy consumption than QNN and llama.cpp, respectively.Mean energy use was measured on 100 prompts sampled from lmsys/lmsys-chat-1m; Table 8 compares frameworks.Source: §7.7 · p. 12, Table 8 · p. 12
8 Related Work
Related systems leave constraints around model scale, memory fit, or mobile co-processing, while compression techniques and speculative decoding offer complementary approaches.
Resource-Efficient DNN/LLM Inference
- BAND, MLC-LLM, mllm-NPU [57], and STI are limited by LLM-scale resource demands, full-memory fit, absent CPU-NPU co-processing, or BERT-scale pipelines.STI’s reported scope is 0.33B-parameter BERT-scale models on embedded devices, below modern LLMs that typically exceed 1B parameters.Source: §8 · pp. 12–13
- Pruning, knowledge distillation [29], and quantization reduce model memory footprints and can complement PowerInfer-2.Source: §8 · p. 13
Speculative Decoding
- Speculative decoding [12] [33] uses a smaller model to propose tokens for batched validation by a larger model, but integration with sparse activation in memory-constrained XPU environments remains open.SpecInfer demonstrates that speculative decoding reduces decoding iterations.Source: §8 · p. 13
9 Conclusion
PowerInfer-2 runs large language models on memory-limited smartphones. The evaluation reports up to 27.8× speedup and support for models as large as 47B with minimal accuracy loss.
9 Conclusion
- Up to 27.8× speedup over existing solutions is reported for PowerInfer-2.Source: §9 · p. 13
- PowerInfer-2 supports models as large as 47B on smartphones with limited memory.Source: §9 · p. 13
- The reported speedup comes with minimal accuracy loss.Source: §9 · p. 13