PaperlayerView exact revision on arXiv

Source-linked AI summary

On-Device Language Models: A Comprehensive Review

Jiajun Xu, Zhiyuan Li, Wei Chen, Qun Wang, Xin Gao, Qi Cai, Ziyuan Ling

arXiv:2409.00088v2cs.CL

TL;DR

Cloud-centric LLM deployment raises latency, data-security, and connectivity concerns, motivating on-device alternatives. This review synthesizes architectures, compression, hardware, and deployment strategies for edge LLMs, concluding that these advances facilitate sophisticated models on resource-constrained devices and support more personalized, responsive AI experiences.

  • Problem

    Cloud deployment presents latency, data-security, and continuous Internet-connectivity challenges, motivating interest in edge-device LLMs.

  • Method

    The review critically analyzes model architectures, compression techniques, energy-efficient computing, hardware strategies, and edge deployment approaches for resource-constrained devices.

  • Results

    Advances in model compression, efficient architectures, and hardware-software cooptimization facilitate sophisticated language models on resource-constrained edge devices.

  • Takeaways & Limitations

    On-device LLMs support personalized, context-aware, and instant AI experiences across mobile, IoT, healthcare, and autonomous systems.

  • Takeaways & Limitations

    Cloud-based LLMs remain powerful but can have latency and data concerns because they depend on networks.

Abstract

from arXiv · show

The advent of large language models (LLMs) revolutionized natural language processing applications, and running LLMs on edge devices has become increasingly attractive for reasons including reduced latency, data localization, and personalized user experiences. This comprehensive review examines the challenges of deploying computationally expensive LLMs on resource-constrained devices and explores innovative solutions across multiple domains. The paper investigates the development of on-device language models, their efficient architectures, including parameter sharing and modular designs, as well as state-of-the-art compression techniques like quantization, pruning, and knowledge distillation. Hardware acceleration strategies and collaborative edge-cloud deployment approaches are analyzed, highlighting the intricate balance between performance and resource utilization. Case studies of on-device language models from major mobile manufacturers demonstrate real-world applications and potential benefits. The review also addresses critical aspects such as adaptive learning, multi-modal capabilities, and personalization. By identifying key research directions and open challenges, this paper provides a roadmap for future advancements in on-device language models, emphasizing the need for interdisciplinary efforts to realize the full potential of ubiquitous, intelligent computing while ensuring responsible and ethical deployment. For a comprehensive review of research work and educational resources on on-device large language models (LLMs), please visit https://github.com/NexaAI/Awesome-LLMs-on-device. To download and run on-device LLMs, visit https://www.nexaai.com/models.

1 Introduction

On-device LLMs are motivated by cloud deployment’s latency, security, connectivity, bandwidth, and energy costs, but constrained devices require specialized architectures and efficient deployment strategies.

  • Motivation and challenges: Cloud-hosted LLMs face latency, security, and continuous-connectivity challenges, motivating local deployment on smartphones, vehicles, and wearables.On-device inference is presented as a path toward reduced response times and personalized user experiences.
  • Motivation and challenges: Edge devices’ limited computational power, memory, and energy make direct adoption of cloud-scale LLM architectures difficult.A 405-billion-parameter model is described as infeasible on a smartphone without substantial compromises.
  • Review scope: The review analyzes model compression, energy-efficient computing, lightweight architectures, and deployment strategies for adapting LLMs to edge constraints.It also considers industry applications and resulting benefits.
  • Review scope: The paper is organized around foundations, efficient architectures, compression, hardware-software strategies, deployment, applications, and open challenges.The stated structure begins with foundations and preliminaries before examining efficient architectures and subsequent deployment topics.

2 Foundations and Preliminaries

On-device LLM research has progressed from sub-10B text models toward multimodal systems and collaborative deployment, while resource limits continue to make efficiency essential.

  • Evolution of on-device LLMs: Since 2023, sub-10B language-model series have made it possible to run LLMs on edge devices.The review identifies this period as the beginning of earnest experimentation with on-device LLMs.
  • Evolution of on-device LLMs: Although small-parameter models may underperform traditional large-parameter models, they enable LLM operation on edge devices.The review also notes the use of mixture-of-experts and quantization technologies in this development.
  • Evolution of on-device LLMs: Multimodal on-device models, including LLaVa, QwenVL, Gemini Nano, and Yi VL, emerged from 2023 onward for more complex mobile scenarios.These systems represent attempts to extend edge deployment beyond text-only processing.
  • Evolution of on-device LLMs: The 2024 model landscape became denser and more diverse, with increased emphasis on models supporting both text and multimodal functionalities.The review links this diversification with maturing hardware and software technologies for deployment across terminal devices.
  • Deployment and efficiency: Collaborative and hierarchical approaches distribute computation across devices and remote infrastructure to address limited memory and computational power.The paper presents these approaches as solutions for resource-constrained deployment.
  • Deployment and efficiency: Quantization-aware scaling, sparse updates, and Tiny Training Engine techniques reduce training memory or computation for resource-constrained devices.The described mechanisms include gradient scaling, selective weight updates, and in-place operation reordering.
  • Deployment and efficiency: 88% of participants preferred edge-cloud collaboration, while 58.33% supported local deployment and 81.82% were dissatisfied with cloud-only solutions.Reported concerns included remote-service latency, personal-data transmission, and cloud-service costs.
  • Deployment and efficiency: On-device inference reduces latency and data transmission, can operate offline, and benefits from customized hardware accelerators.The review cites faster-than-200-ms token generation by common end-side models compared with GPT-4’s approximately 200 ms per token.

3 Efficient Architectures for On-Device LLMs

Efficient on-device LLM deployment combines compression, architectural redesign, collaborative inference, memory-centric hardware, and sparse expert computation. Across these strategies, the reviewed systems target lower memory use, latency, and computation while preserving model quality.

  • Model Compression and Parameter Sharing: AWQ protects 0.1%-1% of crucial weights and enables up to 3× speedup over FP16 inference with TinyChat on desktop and mobile GPUs.The weight-only method avoids backpropagation and reconstruction while preserving generalization across domains and modalities.
  • Model Compression and Parameter Sharing: MobileLLM uses deep-and-thin design, embedding sharing, grouped-query attention, and block-wise weight sharing to improve sub-billion-parameter model accuracy.It reports 2.7% and 4.3% accuracy boosts over 125M and 350M models, respectively.
  • Collaborative and Hierarchical Model Approaches: EdgeShard partitions large LLMs across edge devices and cloud servers, achieving up to 50% latency reduction and 2× throughput enhancement versus traditional cloud-based methods.Dynamic programming optimizes shard placement by balancing computation and communication overhead.
  • Collaborative and Hierarchical Model Approaches: LLMCad uses a smaller model to generate candidate tokens and a larger model to verify them, achieving up to 9.3× token-generation speedup without compromising accuracy.Token-tree generation, self-adaptive fallback, and speculative generation support real-time mobile inference.
  • Memory and Computational Efficiency: Processing-in-Memory and Processing-near-Memory solutions improve memory bandwidth and capacity, delivering up to 4.5× performance improvement and 71% energy reduction.The reviewed Samsung solutions embed or place computational logic near memory to address memory bottlenecks in LLM inference.
  • Mixture-of-Experts (MoE) Architectures: MoE-oriented systems reduce resource demands through expert quantization, communication-aware routing, sparse activation, and multi-precision execution.EdgeMoE achieves up to 2.78× inference speedup, while JetMoE activates 2B of 8B parameters per token and reduces inference computation by about 70% compared to Llama2-7B.

4 Model Compression and Optimization Techniques for On-Device LLMs

On-device LLM efficiency depends on compression methods that reduce model size, computation, and memory while preserving performance. The review covers quantization, pruning, knowledge distillation, and low-rank factorization.

  • Quantization: Quantization converts high-precision weights and activations into lower-bit representations to reduce model size, computation, and memory consumption.It can enable faster inference while preserving accuracy.
  • Quantization: Post-training quantization avoids retraining, while GPTQ uses second-order information to reduce weights to 3 or 4 bits with minimal perplexity increase.GPTQ supports efficient deployment of models such as OPT-175B on constrained hardware.
  • Quantization: Quantization-aware training incorporates reduced-precision constraints during training, generally improving postquantization accuracy by learning to compensate for quantization errors.Weight-only and weight-activation co-quantization offer different trade-offs for computational simplicity and hardware-efficient matrix multiplication.
  • Pruning: Pruning removes less important weights or neurons, while structured pruning removes parameter groups such as layers, channels, or filters for hardware-friendly computation.LLM-Pruner uses gradient data to remove non-essential groups and can use LoRA for performance recovery.
  • Knowledge Distillation: Knowledge distillation transfers knowledge from a large teacher model to a smaller student model, including through black-box learning from teacher outputs.Black-box distillation is useful when teacher parameters are proprietary or teacher and student architectures differ.
  • Low-Rank Factorization: Low-Rank Factorization approximates matrices with products of smaller factors, and Low-Rank Compensation combines it with post-training quantization to reduce size and preserve accuracy.LoRC is described as mitigating detrimental effects of activation quantization.

5 Hardware Acceleration and Deployment Strategies

Hardware accelerators enable on-device LLM inference, while deployment strategies distribute computation across cloud and edge environments. Choosing an accelerator requires balancing performance, power consumption, and cost.

  • Hardware Acceleration: GPUs, TPUs, FPGAs, and specialized AI chips support on-device LLM inference through substantial computational capacity and memory bandwidth.Accelerator selection involves trade-offs among performance, power consumption, and cost.
  • Deployment Strategies: Deployment can range from fully cloud-based solutions to edge-only systems depending on the use case and available infrastructure.This spectrum provides alternatives for distributing LLM computation between centralized and local resources.

1. Edge-only (a) Llama.cpp

Edge-only deployment frameworks target efficient LLM inference across mobile, desktop, and embedded hardware. They differ in platform coverage, acceleration support, training capabilities, and multimodal support.

  • Llama.cpp: Llama.cpp provides quantized inference, GPU acceleration, CPU+GPU hybrid execution, and on-device LoRA fine-tuning across ARM and x86 hardware.It is implemented as a C/C++ library for broad hardware support.
  • MNN: MNN supports efficient mobile LLM inference with dynamic inputs, multimodal interactions, on-device full-sized and LoRA fine-tuning, and CPU, CUDA, and OpenCL backends.It deploys models in ONNX and MNN formats.
  • PowerInfer: PowerInfer and PowerInfer2 accelerate LLM inference on consumer-grade GPUs through locality-centric designs and support x86-64 CPUs and Apple M Chips.They are optimized for Windows and Linux and provide no built-in training capabilities.
  • ExecuTorch and MediaPipe: ExecuTorch deploys PyTorch models on edge devices such as phones and wearables, leveraging CPUs, NPUs, and DSPs without built-in training.MediaPipe supports multimodal video, audio, and time-series pipelines across major desktop and mobile platforms.
  • Edge-cloud: MLC-LLM and vLLM support edge-cloud deployment, with MLC-LLM targeting universal CPU and GPU deployment and vLLM optimizing quantized key-value memory management.Their supported platforms include ARM and x86 architectures and multiple GPU APIs.

6 Examples and Applications

On-device LLMs are advancing through mobile-manufacturer models and deployment systems that support faster, offline, multimodal, and personalized applications. Examples span smartphones, accessibility, translation, healthcare, research, robotics, and autonomous vehicles.

  • Examples of On-Device Language Models: Mobile manufacturers have released on-device or device-cloud language models, reflecting advances in mobile hardware and new capabilities beyond traditional cloud deployment.Table 2 presents state-of-the-art models released by mobile phone manufacturers.
  • Examples of On-Device Language Models: Gemini Nano provides a smaller, faster, lower-inference-latency model through Google AI Edge SDK, with system-level memory management on mobile devices.Its deployment supports mobile applications through an operating-system AI infrastructure.
  • Examples of On-Device Language Models: A 2 billion parameter Octopus model surpasses GPT-4 in reported accuracy and latency, reduces context length by 95%, and completes typical function calls in 1.1 to 1.7 seconds.The model uses functional tokens to map software-function descriptions to specific tokens on Android devices.
  • Examples of On-Device Language Models: OpenELM uses layer-wise scaling to deploy 1.1 billion parameters, achieving a 2.36% accuracy increase over prior models while requiring half the pre-training tokens.Phi-3-mini has 3.8 billion parameters and achieves 69% on MMLU and 8.38 on MT-bench.
  • Examples of On-Device Language Models: MiniCPM-Llama3-V 2.5 is an 8.5 billion parameter multimodal model with a 65.1 average Open-Compass score, while Gemma2-9B combines sliding-window and global attention.Gemma2-9B is reported to outperform Llama 3-8B and similar open models in reasoning, math, and code.
  • Examples of On-Device Language Models: Qwen2-0.5B has 0.5 billion parameters and a 32K context length, while Qwen-Agent’s Agentic RAG can extend processing context to 1M.The model performs similarly to Gemma-2B and Phi-2 in multiple tests despite having fewer parameters.
  • Applications: On-device applications address translation, meeting summarization, healthcare, scientific research, companion robotics, disability support, and autonomous driving.Reported benefits include lower latency, offline operation, localized data, reduced hardware or subscription costs, and multimodal assistance.
  • Applications: On-device multimodal systems support image descriptions for blind or low-vision users, while DriveVLM Dual uses language to interpret urban driving scenes and develop waypoint plans.These examples extend on-device or edge language modeling beyond text-only interaction.

7 Future Directions and Open Challenges

Future on-device LLM research centers on strengthening privacy and security while addressing the interconnected challenges of deployment in dynamic, resource-constrained edge environments.

  • On-device LLM research prioritizes data security, low latency, and personalized AI experiences on edge devices.
  • Progress across compression, inference, security, energy efficiency, hardware integration, and adaptation remains interconnected, requiring a holistic deployment perspective.
  • Security and privacy: Privacy techniques must balance data-security guarantees, model utility, and computational constraints.
  • Security and privacy: Risk assessment should combine benchmarking, real-time monitoring, and detection or mitigation of PII leakage during inference.
  • Security and privacy: Efficient model sharding, security-enhancing architectures, and minimized data transmission are key architectural and communication priorities.
  • Security and privacy: Collaborative and distributed learning requires secure multi-party computation, long-conversation data protection, and broader support for LLM architectures and tasks.

7.2 Adaptive Edge-Cloud Collaboration

Adaptive edge-cloud collaboration and multimodal on-device deployment require coordinated advances in transmission, scheduling, compression, performance optimization, and cross-modal understanding.

  • Adaptive Edge-Cloud Collaboration: Edge-cloud collaboration research targets caching, request analysis, and uncertainty-guided token sampling to optimize data transmission.
  • Adaptive Edge-Cloud Collaboration: Intelligent scheduling should personalize inference, allocate resources across heterogeneous infrastructures, and distribute workloads with batch-size awareness.
  • Adaptive Edge-Cloud Collaboration: Efficient knowledge transfer depends on adapter-based distillation, dynamic quantization, and adaptive compression of weight updates.
  • Adaptive Edge-Cloud Collaboration: Collaborative systems need token-level adaptive control, real-time constraint satisfaction, and techniques that reduce latency and improve pipeline execution.
  • Multimodal Capabilities: Multimodal on-device research spans efficient processing and compression for audio, text, and video.
  • Multimodal Capabilities: Knowledge transfer from larger cloud models to smaller on-device models aims to improve few-shot and zero-shot capabilities across modalities.
  • Multimodal Capabilities: Multimodal understanding requires datasets and encoders supporting high-resolution images, long videos, complex audio, web pages, 3D vision, heat maps, and tables or figures.
  • Multimodal Capabilities: Future systems should extend context across previous interactions and process temporal information for tasks such as audio captioning and acoustic scene classification.

7.4 Resource-Efficient Solutions

Resource-efficient on-device LLMs depend on compression, sparsity, energy-aware execution, and hardware-software co-design across memory, compilers, runtimes, and edge systems.

  • Resource-Efficient Solutions: Advanced pruning, quantization, knowledge distillation, larger-than-memory execution, and adaptive inference are central resource-efficiency directions.
  • Resource-Efficient Solutions: Runtime activation sparsity can reduce inference time and memory footprint by activating only a small portion of the model for a given task.
  • Resource-Efficient Solutions: Energy-aware training, runtime optimization, and parameter-efficient fine-tuning should balance security, energy efficiency, and performance.
  • Hardware-Software Co-Design: Hardware-software integration should advance PIM/PNM architectures, including CXL-based optimizations and low-power edge solutions.
  • Hardware-Software Co-Design: Hardware-aware optimization includes pruning-aware quantization, contextual sparsity, and dynamic sparse attention.
  • Hardware-Software Co-Design: AI-specific compilers and runtimes should automatically optimize operations for PIM/PNM hardware at graph and hardware-specific levels.
  • Hardware-Software Co-Design: Edge and multi-device systems require dynamic sparse tree optimization, adaptive bit widths, and energy-aware co-design.

7.6 Robustness and Reliability

Robust and scalable on-device LLMs require safeguards against biased or hallucinated outputs, formal reliability assessment, uncertainty handling, and efficient distributed operation.

  • Robustness and Reliability: Safety-critical deployments need methods to detect and mitigate biases and hallucinations in on-device LLM outputs.
  • Robustness and Reliability: Formal verification and validation frameworks should assess on-device language-model reliability in real-world scenarios.
  • Robustness and Reliability: Ensemble methods and probabilistic inference can address variance, bias, and uncertainty throughout the LLM pipeline.
  • Scalability: Scaling inference across heterogeneous edge devices requires dynamic resource allocation and load balancing.
  • Scalability: Collaborative edge computing should reduce latency and improve throughput through model sharding and pipelined inference.
  • Scalability: Managing multiple LLM versions across diverse devices requires accounting for network constraints and device capabilities while improving model and dataset reproducibility.

7.8 Continual Learning and Personalization

On-device LLMs enable personalized AI experiences but must remain relevant as information and user preferences change. The review highlights controllable forgetting, autonomous skill acquisition, and history tracking as future research priorities.

  • On-device LLMs offer personalized AI experiences while creating challenges for maintaining relevance as information and user preferences evolve.
  • Controllable knowledge retention and forgetting can selectively manage information from new data streams to address misinformation and ongoing accuracy.
  • Future systems should autonomously learn new skills and improve existing capabilities from user interactions and local data.
  • Effective history-tracking mechanisms are identified as necessary for understanding how an LLM evolves through learning.
  • Progress on these pathways could support innovative frameworks for intelligent and tailored applications while addressing unresolved issues in edge computing.
  • The anticipated advancements aim to make sophisticated AI capabilities more accessible and efficient across devices and use cases.

8 Conclusion

The review surveys how compression, efficient architectures, and hardware-software cooptimization support sophisticated language models on resource-constrained devices. It presents on-device deployment as a route to personalized, context-aware, and instant AI, while emphasizing unresolved resource, robustness, learning, sustainability, and responsibility challenges.

  • Compression techniques, efficient architectures, and hardware-software cooptimization collectively facilitate sophisticated language models on resource-constrained edge devices.
  • On-device deployment brings advanced language processing directly to end-user devices, opening personalized, context-aware, and instant AI experiences.
  • The central research problem is balancing model performance with edge-device resource limitations while maintaining robustness across heterogeneous operating conditions.
  • Continual learning, energy efficiency, sustainability, and responsible deployment remain additional challenges requiring innovative solutions and ethical consideration.
  • Realizing on-device language models’ potential requires multidisciplinary progress in compression, efficient architectures, security, reliability, hardware-software co-design, and edge-cloud collaboration.
  • The convergence of efficient models, stronger edge hardware, and innovative deployment strategies is presented as promising for future human-AI interaction.
  • The review serves as a resource that analyzes the current state of on-device LLMs and identifies areas for future research and development.
Loading 2409.00088v2…