PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast On-device LLM Inference with NPUs

Daliang Xu, Hao Zhang, Liming Yang, Ruiqi Liu, Gang Huang, Mengwei Xu, Xuanzhe Liu

arXiv:2407.05858v2cs.AI

TL;DR

On-device LLMs are gaining interest, but variable-length prompts conflict with mobile NPUs’ static-shape execution. llm.npu restructures prompts and model execution for NPU offloading, reporting faster prefill and end-to-end inference against baselines.

  • Problem

    Mobile NPUs support static shapes while LLM prompts vary in length, making per-prompt graph preparation costly or padding wasteful.

    Source: §1 · p. 2

  • Method

    llm.npu restructures prompt chunks, activation outliers, and Transformer-block scheduling to offload integer computation to mobile NPUs while retaining essential floating-point operations on CPU/GPU.

    Source: §1 · p. 2

  • Results

    7.3×–18.4× faster than baselines on CPU and 1.3×–43.6× on GPU with a 1024-token prompt; accuracy loss compared to FP16 is <1%.

    Source: §1 · p. 3

  • Takeaways & Limitations

    llm.npu is the first system reported to achieve more than 1,000 tokens/sec of prefill on COTS mobile devices for billion-sized LLMs.

    Source: §1 · p. 3

    llm.npu currently supports only CPU-based decoding, which increases end-to-end latency for datasets with more output tokens; using a mobile GPU as the decoding backend can mitigate this.

    Source: §4.3 · p. 12

Abstract

from arXiv · show

On-device inference for Large Language Models (LLMs), driven by increasing privacy concerns and advancements of mobile-sized models, has gained significant interest. However, even mobile-sized LLMs (e.g., Gemma-2B) encounter unacceptably high inference latency, often bottlenecked by the prefill stage in tasks like screen UI understanding. We present llm.npu, the first LLM inference system utilizing on-device Neural Processing Unit (NPU) offloading to reduce prefill latency. llm.npu enhances NPU offloading efficiency by re-constructing the prompt and model in three levels: (1) At prompt level, it divides variable-length prompts into multiple fixed-sized chunks while maintaining data dependencies; (2) At tensor level, it identifies and extracts significant outliers to run on the CPU/GPU in parallel with minimal overhead; (3) At block level, it schedules Transformer blocks in an out-of-order manner to the CPU/GPU and NPU based on their hardware affinity and sensitivity to accuracy. Compared to competitive baselines, llm.npu achieves 22.4x faster prefill speed and 30.7$\times$ energy savings on average, and up to 32.8x speedup in an end-to-end real-world application. For the first time, llm.npu achieves more than 1,000 tokens/sec prefilling for a billion-sized model.

1 Introduction

Privacy concerns and mobile-sized models are driving interest in on-device LLMs, but long-context prefill often dominates latency in tasks such as UI automation. llm.npu addresses this by restructuring prompts and model execution for mobile NPU offloading.

1 Introduction

  • Prefill often dominates end-to-end latency for long-context on-device tasks, while much existing acceleration work targets decoding.Prefill accounts for 94.4%–98.8% of latency in UI automation tasks.Source: §1 · p. 2
  • Mobile NPUs favor integer computation, but variable prompt lengths, per-group quantization, and essential floating-point operators complicate efficient offloading.Static-shape NPUs require costly graph preparation or wasteful padding for variable-length prompts; splitting group MatMuls incurs up to 10.7× overhead, while moving floating-point operators off-NPU can lengthen the critical path.Source: §1 · p. 2
  • llm.npu is presented as the first efficient on-device LLM NPU-offloading system, restructuring execution at prompt, tensor, and Transformer-block levels.It aims to accelerate integer computation on NPUs while keeping essential floating-point operations on CPU/GPU to maintain accuracy.Source: §1 · pp. 2–3
  • Chunk-sharing graphs execute variable-length prompts as fixed-size chunks with causal dependencies, while sharing prompt-size-independent operators to reduce graph memory overhead by up to 4×.The approach reduces graph preparation time; operators such as FFN can be shared across chunk executions.Source: §1 · p. 2
  • Shadow outlier execution runs sparse activation outlier channels on CPU/GPU in parallel with NPU MatMul, retaining only hot-channel weights in memory and pruning unimportant outliers.Outliers comprise 0.1%–0.3% of channels; layer-level pruning reduces synchronization overhead.Source: §1 · pp. 2–3
  • Out-of-order subgraph scheduling prioritizes work that reduces NPU stalls, aiming to maximize NPU utilization while limiting the impact of CPU/GPU workloads.The online scheduler addresses the NP-hard ordering problem at microsecond-level granularity.Source: §1 · p. 3
  • 7.3×–18.4× faster than baselines on CPU and 1.3×–43.6× on GPU at 1024-token prompts; end-to-end real-world inference latency is reduced by 1.4×–32.8×.Experiments also report <1% accuracy loss compared to FP16 and more than 1,000 tokens/sec of prefill speed on COTS mobile devices for billion-sized LLMs.Source: §1 · p. 3

2 Background

Mobile LLM inference is often dominated by prefill, while mobile NPUs offer strong INT8 MatMul performance. Variable-length prompts, quantization constraints, and floating-point operators complicate efficient NPU use.

2.1 On-device LLM Inference Analysis

  • Prefill is often the bottleneck in typical mobile applications, constituting 88.3%–98.8% of total latency on mobile CPUs for UI task automation, chat summaries, and context-aware generation, and 54.2%–91.7% on mobile GPUs.Its share rises as prompt length increases; chat-summary prompt and output lengths are relatively balanced.Source: §2.1 · p. 3

2.2 Opportunity: Mobile NPUs

  • INT8 MatMul on mobile NPUs is 4.5–5.8× faster than CPU INT8 and 1.8–3.5× faster than GPU FP16 across the tested sizes.By contrast, NPU FP16 MatMul is up to 159× slower than CPU INT8; the results align with NPUs’ INT8 SIMD design.Source: Table 3 · p. 4, §2.2 · p. 4

2.3 Gaps between LLMs and Mobile NPUs

  • On mobile NPUs, creating a compute graph typically takes 300–500 ms and optimizing it takes many seconds; variable-length LLM prefill prompts cause excessive time building and compiling the graph.Source: §2.2 · p. 4, §2.3 · p. 4
  • K-Quant and AWQ incur 8.1×–10.7× overhead versus per-tensor quantization, while SmoothQuant loses 3.9% accuracy on LlaMA-2-7B and 8.4% on Qwen1.5-1.8B on HelloSwag.Per-group quantization requires multiple sub-tensor MatMuls on NPUs; SmoothQuant uses per-tensor quantization.Source: Figure 4 · p. 5, §2.3 · p. 5, Figure 3 · p. 5
  • Quantized inference keeps linear-layer MatMuls in INT8/INT4, but Attention and LayerNorm remain floating-point operations that degrade NPU performance.The operator formats in current quantization approaches retain float Attention and normalization.Source: §2.3 · p. 5, Figure 5 · p. 5, Table 4 · p. 5

3 llm.npu Design

llm.npu targets lower prefill latency and energy for mobile-sized LLMs by maximizing mobile-NPU integer execution while retaining essential floating-point work on CPU/GPU. Its design combines fixed-length prompt chunks, activation-outlier handling, and out-of-order scheduling across the NPU and CPU/GPU.

3.2 Chunk-sharing graph execution

  • 120 of 144 subgraphs can be shared in Qwen1.5-1.8B, reducing memory consumption by up to 75% (7.2GB) for a 1024-token prompt and chunk length 256.The shared graph components are built and optimized once, while dynamic operators are constructed for individual chunks.Source: §3.2 · p. 6
  • On Xiaomi 14, llm.npu empirically selects chunk length 256 to use NPU capabilities while reducing intra-chunk padding; profiling must be repeated across NPUs.The experiments covered Qwen1.5-1.8B and Gemma-2B.Source: §3.2 · pp. 6–7

3.3 Shadow outlier execution

  • Shadow outlier execution extracts activation outlier channels into a compact tensor for CPU execution, then merges that result with the original operator’s NPU output.This enables per-tensor activation quantization intended to avoid compromising LLM accuracy.Source: §3.3 · p. 7
  • Shadow execution nearly doubles memory use through duplicated MatMul weights, while CPU-NPU reduced-sum synchronization adds 29.7% end-to-end latency and 20.1% energy on Qwen1.5-1.8B.Source: §3.3 · p. 7
  • Less than 3% of channels contribute over 80% of outliers; retaining hot-channel weights in CPU memory and retrieving rare others from disk reduces shadow memory overhead by 34.3% with negligible latency.The retrieval can overlap with execution of the original MatMul on the NPU.Source: §3.3 · pp. 7–8
  • Pruning outliers from the top 85% most unimportant layers, identified through offline profiling, eliminates CPU-NPU synchronization for those layers.The paper reports that outliers in most MatMul operators can be removed without accuracy impacts.Source: §3.3 · p. 8

3.4 Out-of-order subgraph execution

  • Simple overlap leaves a 37% critical-path bubble rate; out-of-order scheduling runs input-ready subgraphs beyond chunk order while preserving cross-chunk and intra-chunk dependencies.Attention uses prior-chunk dependencies; LayerNorm, Linear, and Quantize depend on earlier subgraphs within the same chunk.Source: §3.4 · p. 8
  • Because optimal ordering is NP-hard and prompt-dependent, an online heuristic with microsecond-level overhead selects the pending subgraph with maximum C to reduce NPU stalls.C reflects the execution time of newly enabled NPU work, with the score’s sign depending on whether the selected subgraph runs on CPU/GPU or NPU.Source: §3.4 · pp. 8–9

4 Implementation and Evaluation

llm.npu is implemented and evaluated as an on-device NPU inference system, with optimizations spanning prompt preparation, outlier handling, and scheduling. Across prefill, real-world workloads, accuracy, memory, and coordination tests, it generally improves performance while exposing trade-offs from prompt length, decoding, and implementation choices.

4.1 Experiment setups

  • The prototype targets Qualcomm Hexagon NPUs, was implemented on MLLM and QNN, and was evaluated across five models, four benchmarks, two phones, and five baselines.It profiles equivalent linear-layer shapes and prioritizes computationally intensive tasks such as FFN on the NPU under memory limits.Source: §4 · p. 9, §4.1 · p. 9

4.2 Prefill performance.

  • At 1024 tokens, prefill latency fell 18.17–38.4× vs llama.cpp-CPU and 32.5–43.6× vs MLC-GPU on Redmi K70 Pro; Redmi K60 Pro energy consumption fell 35.63–59.52× vs llama.cpp-CPU.Against PowerInfer-V2-NPU, prefill speed improved 3.28–5.32× and 3.4–5.6× on the two devices, respectively; shorter prompts saw smaller speedups, partly due to padding and reduced out-of-order scheduling efficiency.Source: Figure 14 · p. 10, Figure 15 · p. 10, §4.2 · p. 10

4.3 End-to-end performance

  • Across three real-world workloads, llm.npu had the lowest latency; speedups included 23.0–46.2× over llama.cpp-CPU on LongBench, 15.0–32.8× over MLC-LLM-GPU on DroidTask, and 10.5× over MLC-LLM-GPU on Persona-Chat.Gains were smaller on shorter-prompt DroidTask workloads, and CPU-based decoding increased latency on Persona-Chat’s larger-output workload.Source: §4.3 · pp. 10–12, Table 5 · p. 11

4.4 Inference accuracy

  • Accuracy averaged only 1% below FP16, with improvements up to 32.9% over SmoothQuant and 70.9% over K-Quant.Higher outlier pruning increased generation speed but could lower accuracy.Source: Table 6 · p. 12, §4.4 · pp. 12–13

4.5 Memory consumption

  • With a 512-token prompt on Redmi K60 Pro, llm.npu consumed up to 1.32× more memory than llama.cpp and TFLite.The additional memory for shadow outlier execution was only 0.6%–1% of total memory; the larger overhead came from framework activation buffers.Source: Figure 17 · p. 13, §4.5 · p. 13

4.6 GPU-NPU coordination analysis

  • GPU-NPU coordination matched CPU-NPU prefill speed and reduced end-to-end latency by 80–90 ms.The reported improvement was attributed to faster GPU decoding than CPU decoding.Source: Figure 18 · p. 13, §4.6 · p. 13

4.7 Ablation study.

  • Chunk-sharing improved prefill speed 1.46–5.09×, shadow outlier execution reduced latency 3.91–8.68×, and out-of-order execution reduced latency 18%–44%.Directly offloading prefilling to mobile NPUs instead caused 2.55–2.68× delays.Source: Figure 19 · p. 13, §4.7 · p. 13

5 Discussion and Future Work

The discussion contrasts mobile GPUs’ competing graphics workloads with NPUs’ dedicated deep-learning use, capacity, and energy efficiency. It also identifies hardware opportunities and notes that the current implementation relies on CPUs for floating-point operators and decoding.

NPU and GPU trade-offs

  • Because mobile GPUs also serve graphics rendering, LLM inference can contend for resources; NPUs are dedicated to deep learning and are described as more capable, energy-efficient, and available.The discussion says NPUs may have more available time for inference because their workloads are lighter.Source: §5 · pp. 13–14

Implementation boundary and future work

  • The current implementation uses CPUs for floating-point operators and decoding; GPU-NPU coordination is compatible with the design but remains planned.Proposed hardware improvements include dynamic shape-aware optimization, larger data caches, and mixed-precision operands.Source: §5 · p. 14

Technique stability

  • The authors expect integer operations to remain dominant in mobile NPU-side LLM inference, with CPUs or GPUs supplementing them to improve accuracy.They also state that llm.npu can support legacy devices and identify hardware changes that could further improve performance.Source: §5 · p. 14

6 Related work

The related work places llm.npu alongside algorithm-level on-device LLM optimizations and heterogeneous mobile inference systems. It distinguishes llm.npu by addressing LLM-specific NPU-prefill challenges and describes it as compatible with other optimization layers.

On-device LLM optimization

  • llm.npu is orthogonal to quantization, distillation, token pruning, and speculative decoding, and is compatible with those algorithm-level optimizations.The passage defines speculative decoding as accelerating token generation by offloading tasks to a smaller LLM.Source: §6 · p. 14

On-chip offloading for ML

  • Unlike prior mobile NPU methods that do not address LLM-specific features, llm.npu targets NPU offloading for LLM prefilling and reports up to 5.32× greater efficiency than PowerInfer-V2.The system is also described as orthogonal to mobile NPU execution optimizers, which can serve as operator-library backends.Source: §6 · p. 14

7 Conclusions

The conclusion presents llm.npu as an on-device NPU-offloading system for reducing LLM prefill latency and energy consumption. It attributes its reported performance benefits to three techniques for more efficient NPU offloading.

7 Conclusions

  • llm.npu combines chunk-sharing graphs, shadow outlier execution, and out-of-order subgraph execution to improve on-device NPU offloading efficiency.Source: §7 · p. 14
  • The conclusion reports up to 43.6× speedup and 59.5× energy savings.Source: §7 · p. 14

A.1 Abstract

The artifact evaluates prefill performance and inference accuracy, with instructions for reproducing both. The authors recommend validated hardware setups and use Qwen1.5-1.8B as the representative demonstration model.

A.1 Abstract

  • The artifact’s evaluations cover prefill performance in Section 4.2 and inference accuracy in Section 4.4.The repository provides instructions for installing and executing llm.npu and baseline evaluations.Source: §A.1 · p. 15
  • The authors recommend a Redmi K70 Pro 24G for on-device inference and an NVIDIA A100 40G server for testing.These setups are described as validated by the authors.Source: §A.1 · p. 15
  • Qwen1.5-1.8B is the representative model used for most experiments, while users may adapt the code for other models.The artifact instructions include environment setup, model and dataset downloads, compilation, and execution.Source: §A.1 · p. 15

A.2 Artifact check-list (meta-information)

The checklist records the artifact’s supported models and datasets, implementation materials, evaluation metrics, and approximate resource requirements.

A.2 Artifact check-list (meta-information)

  • The demonstration model is Qwen1.5-1.8B-Chat, and the artifact also supports Gemma-2B, Phi-2-2.7B, LLaMA-2-7B, and Mistral-7B.The checklist identifies inference accuracy and prefill speed as its metrics.Source: §A.2 · p. 15
  • HellaSwag and LAMBADA are demonstration datasets, with OpenBookQA [62], WinoGrande, and MMLU [43] also supported.The checklist states that the workflow uses C++, Python, and bash scripts and is licensed under MIT.Source: §A.2 · p. 15
  • The checklist estimates 200GB of disk space, five hours to prepare the workflow, and two hours to complete the experiments.It lists an NVIDIA A100 40G server and a Redmi K70 Pro 24G device as hardware.Source: §A.2 · p. 15

A.3 Description

The description begins by stating that the artificial evaluation is publicly accessible.

A.3 Description

  • The artificial evaluation is described as publicly accessible.The passage ends before specifying how to access it.Source: §A.3 · p. 15

A.3.2 Hardware dependencies.

The artifact instructions specify hardware and software prerequisites, supported models and datasets, installation steps, and an evaluation workflow. They also identify the expected evaluation outputs and include a methodology heading.

A.3.2 Hardware dependencies.

  • Accuracy evaluation requires an NVIDIA A100 40GB GPU server with x86 CPU cores and preferably over 200GB of CPU memory; performance evaluation requires a Redmi K70 Pro with 24GB memory.The accuracy server should run Ubuntu 20.04.Source: §A.3.2 · p. 15

A.3.3 Software dependencies.

  • Accuracy evaluation requires CUDA 12.5 and Conda with Python 3.8 or higher; its datasets include HellaSwag, LAMBADA, OpenBookQA [62], WinoGrande, and MMLU [43].The datasets can be automatically downloaded using code from Hugging Face.Source: §A.3.3 · p. 15, §A.3.4 · p. 15

A.3.4 Datasets.

  • The device demonstration uses Qwen1.5-1.8B, and users may integrate customized models including Gemma-2B, Phi-2-2.7B, LLaMA-2-7B, and Mistral-7B.Source: §A.3.5 · p. 15

A.4 Installation

  • Accuracy reproduction uses the environment specified in accuracy_results/environment.yml, while performance reproduction follows the README instructions in performance_results.Baseline evaluation, particularly SmoothQuant [84], requires its separately documented environment.Source: §A.4 · p. 15

A.5 Experiment workflow

  • The workflow first runs accuracy evaluation to generate the quantized model needed for on-device inference, then runs performance evaluation with that model.Source: §A.5 · p. 15

A.6 Evaluation and expected results

  • Expected results comprise accuracy evaluation for Section 4.4 and prefill performance evaluation for Section 4.2, contributing to Table 6 and Figure 14.Source: §A.6 · p. 15
Loading 2407.05858v2…