PaperlayerView exact revision on arXiv

Source-linked AI summary

A Review on Edge Large Language Models: Design, Execution, and Applications

Yue Zheng, Yuhao Chen, Bin Qian, Xiufang Shi, Yuanchao Shu, Jiming Chen

arXiv:2410.11845v2cs.DC

TL;DR

Deploying LLMs on resource-constrained, heterogeneous edge devices is limited by computational, memory, and practical deployment challenges. This survey synthesizes the field across pre-deployment design, runtime optimization, and on-device applications, concluding that their cohesive pipeline supports compact, efficient models and diverse edge use cases while real-world heterogeneous deployment and complex reasoning remain open challenges.

  • Problem

    Edge deployment must address computational and memory constraints, hardware heterogeneity, and practical application requirements that complicate efficient and reliable on-device LLM inference.

  • Method

    The survey systematically reviews offline pre-deployment techniques, online runtime inference optimizations, and on-device LLM applications as a cohesive optimization pipeline.

  • Results

    The reviewed techniques reduce model size and computational demands, facilitate edge integration, and cover applications spanning personal, enterprise, industrial, healthcare, and embodied systems.

  • Takeaways & Limitations

    Combining offline compression with runtime optimization provides a survey-supported route toward accessible and sustainable edge AI across diverse applications.

  • Takeaways & Limitations

    Many hardware-software co-design techniques remain simulation-based, while heterogeneous real-world deployment and multi-hop reasoning remain challenging.

Abstract

from arXiv · show

Large language models (LLMs) have revolutionized natural language processing with their exceptional understanding, synthesizing, and reasoning capabilities. However, deploying LLMs on resource-constrained edge devices presents significant challenges due to computational limitations, memory constraints, and edge hardware heterogeneity. This survey provides a comprehensive overview of recent advancements in edge LLMs, covering the entire lifecycle: from resource-efficient model design and pre-deployment strategies to runtime inference optimizations. It also explores on-device applications across various domains. By synthesizing state-of-the-art techniques and identifying future research directions, this survey bridges the gap between the immense potential of LLMs and the constraints of edge computing.

1 Introduction

This introduction frames edge LLM deployment as a lifecycle spanning resource-efficient model design, runtime optimization, and practical on-device applications. It motivates this scope through the benefits of local inference and the computational, memory, heterogeneity, and application challenges of edge devices.

  • Motivation: Local inference offers faster responses, operation without internet connectivity, protection for sensitive data, and adaptation to user preferences.These benefits motivate deployment in personal agents, office assistants, robotics, autonomous systems, and industrial IoT.
  • Challenges: A 7B-parameter LLaMA-2 model requires over 8GB of FP16 memory, illustrating why edge deployment faces severe loading and inference constraints.Without compression, latency spikes and memory overflow can occur during model loading.
  • Challenges: Heterogeneous edge hardware requires runtime optimization across software strategies, hardware enhancements, and hardware-software co-design.The survey distinguishes mobile CPUs, IoT chips, and edge GPUs as environments with different optimization needs.
  • Challenges: Practical applications must bridge centralized processing and distributed edge scenarios while maintaining responsiveness, accuracy, latency, and reliability.Personal, enterprise, and industrial systems require domain-specific task prioritization and dynamic resource allocation.
  • Survey scope: The survey organizes edge LLM research around pre-deployment compression, runtime optimizations, and on-device applications across personal, enterprise, and industrial domains.Pre-deployment methods include quantization, pruning, knowledge distillation, and low-rank approximation.
  • Survey scope: The unified pipeline combines offline compression with real-time optimization to support diverse applications on resource-constrained devices.The survey presents this lifecycle as a way to reduce model size and computational demands while preserving performance.

2 Background and Related Work

This section explains that LLM complexity is outpacing edge-device capability and that existing surveys leave edge-specific LLM deployment insufficiently covered. It positions the survey as a comprehensive synthesis of pre-deployment, runtime, and application research.

  • Motivation: The survey estimates LLM pre-training FLOPs with C ≈6ND, where N is parameter count and D is the number of pre-training tokens.This heuristic provides the basis for the complexity trend shown in Figure 3.
  • Motivation: LLM pre-training complexity has increased rapidly while edge-device AI performance has improved more slowly, creating a widening capability gap.Figure 3 compares estimated LLM pre-training FLOPs in TFLOPs with edge-device performance in TOPS.
  • Related work: Earlier efficient-learning surveys focus on CNNs and RNNs or address resource management and security without adequately treating LLM scale and attention mechanisms.These characteristics create challenges distinct from those of earlier DNN architectures.
  • Related work: Existing NLP surveys cover compression, low-resource settings, efficient LLMs, and constrained NLP, but do not specifically address deployment in edge environments.The section identifies a need for focused research on edge-specific LLM challenges.
  • Survey positioning: The survey contributes a comprehensive edge-focused analysis spanning offline model design, online runtime inference optimization, hardware-software co-design, hardware enhancements, and applications.Its scope is presented as broader than surveys focused primarily on collaborative resource management or application scenarios.
  • Research trends: On-device LLM research from 2019 to 2024 is categorized into pre-deployment techniques, runtime optimizations, and on-device applications.The trend overview groups quantization, pruning, distillation, and low-rank approximation under pre-deployment research.

3 Offline Pre-Deployment Model Design Techniques

Offline pre-deployment techniques reduce LLM resource demands before deployment, with quantization offering compression through lower-precision representations while introducing accuracy and outlier-management challenges.

  • Pre-deployment design reduces LLM computational and memory footprints before deployment, facilitating execution on resource-constrained edge devices.
  • Quantization: Quantization reduces numerical precision and is categorized into weight-only and weight-activation co-quantization methods.
  • Quantization: Weight-only quantization lowers weight precision to reduce memory usage and potentially speed inference, while outlier weights may require higher-precision storage.
  • Quantization: Weight-activation co-quantization compresses both weights and activations, but outlier handling remains a challenge requiring adaptive calibration or learnable transformations.
  • Quantization: Weight-only methods provide moderate compression for quick deployment, whereas co-quantization offers higher compression with greater complexity and potential accuracy loss.

3.2 Pruning

Pruning reduces LLM parameter counts for smaller models and faster inference, but specialized methods must address Transformer complexity and trade-offs between structured control and unstructured flexibility.

  • Pruning removes LLM parameters to reduce model size and accelerate inference, while attention-head importance and Transformer complexity complicate pruning decisions.
  • Structured pruning: Structured pruning removes complete components such as neurons, channels, layers, or attention heads, enabling controlled architectural reductions.
  • Unstructured pruning: Movement Pruning, oBERT, SparseGPT, Plug-and-Play, and Wanda use weight dynamics, second-order information, sparse regression, activations, or input-weight products to guide pruning.
  • Structured methods preserve architecture integrity, whereas unstructured methods offer flexibility but can produce irregular sparse models that are harder to optimize.

3.3 Knowledge Distillation

Knowledge distillation creates smaller computationally efficient students from complex teacher models, using either internal teacher information or outputs alone depending on model access.

  • Knowledge distillation transfers teacher knowledge to simpler student models, reducing model size, computational costs, and deployment requirements.
  • White-box distillation: White-box distillation uses teacher architecture and parameters, including internal features and logits, while MiniLM and MiniLMv2 extract knowledge from Transformer representations.
  • White-box distillation: Reverse Kullback-Leibler divergence, layer-wise alignment, entity-based transfer, token-scaled logits, and MoE models address teacher-student capacity mismatches.
  • Black-box distillation: Black-box distillation relies only on teacher outputs, supporting proprietary or API-based teachers and transferring reasoning or instruction-following abilities.
  • White-box methods suit settings with accessible model internals, whereas black-box methods suit industrial or proprietary contexts without such access.

3.4 Low-Rank Approximation

Low-rank approximation compresses high-dimensional LLM matrices into products of smaller matrices, reducing redundancy, storage, and computation while targeting large-scale embeddings and attention weights.

  • Traditional factorization methods face challenges on Transformer-based LLMs because of their scale, computational complexity, and structural elements.
  • Low-rank approximation represents W_m×n as U_m×r(V^T)_r×n, with r much smaller than m and n.
  • Examples include ALBERT’s vocabulary-embedding factorization, FWSVD’s Fisher-weighted decomposition, and DRONE’s compressed weight-matrix optimization.
  • Low-rank approximation reduces parameters and redundancy in embedding layers and attention weights, lowering storage and computation costs while maintaining performance.
  • Complementary methods: Complementary compact-model techniques include curated data preprocessing, grouped- or multi-query attention, RoPE, and layer-wise scaling.
  • Complementary methods: Development and optimization of these compact-model techniques predominantly depend on organizations with substantial computational resources and domain expertise.

3.6 Comparative Analysis and Selection

Selecting offline pre-deployment techniques requires matching model architecture, deployment environment, hardware constraints, performance requirements, and tolerance for accuracy loss. The review contrasts quantization, pruning, distillation, low-rank approximation, and efficient architectures as options for different edge settings.

  • Technique selection depends on model architecture, deployment constraints, performance requirements, and the target hardware environment.The comparison is intended to guide practitioners toward techniques suited to their use case.
  • Weight-only quantization and structured pruning suit strict memory constraints when computational resources remain moderate.Examples include LLM.int8() and CoFi.
  • Weight-activation co-quantization and aggressive unstructured pruning favor maximum compression when some accuracy loss is acceptable.Examples include SmoothQuant and Movement Pruning.
  • Efficient architectures such as LLaMA or Gemma can provide strong performance under strict resource constraints without extensive post-training compression.These models rely on architecture design and training strategies developed with high-quality data and substantial computing resources.

4 Online Runtime Inference Optimizations

Online runtime inference optimizations organize edge LLM execution across software strategies, hardware-software co-design, and hardware enhancements. The reviewed methods distribute computation, reduce inference work, improve memory use, and adapt execution to device capabilities.

  • Runtime optimization categories: Runtime optimizations span software-level strategies, hardware-software co-design, and hardware-level enhancements for efficient edge inference.Software methods include resource scheduling and framework optimizations independent of hardware.
  • Cloud and Multi-Edge Collaboration: Cloud-edge collaboration uses split inference or speculative decoding to distribute LLM computation across devices.Speculative decoding reduces communication by sending edge-generated drafts for cloud verification.
  • Single-Device Resource Scheduling: Token reduction, early exiting, and dynamic offloading reduce single-device inference work through selective computation and resource redistribution.Examples include prompt compression, confidence-based termination, and hybrid GPU-CPU-disk memory scheduling.
  • Framework-Level Optimizations: Framework-level optimizations improve portability and latency through lightweight execution systems, operator fusion, memory-layout selection, and heterogeneous neuron placement.ExecuTorch, DNNFusion, SmartMem, and PowerInfer illustrate these strategies.
  • Hardware-Software Co-Design: ASIC accelerators prioritize throughput through fine-grained sparsity, while in-memory accelerators reduce data movement using token-based designs.Low-bit formats improve energy efficiency and predictability but can sacrifice accuracy; adaptive encoding preserves precision flexibility at higher complexity.
  • Hardware-Software Co-Design: Much hardware-software co-design research remains simulation-based, with limited deployment on general-purpose heterogeneous platforms.The review presents co-design as a future-oriented route for exploiting hardware-specific features.

SoCs (CPU + iGPU +

Edge SoC platforms combine CPUs, GPUs, and NPUs to balance flexibility, throughput, and energy efficiency for on-device LLM inference. Their heterogeneity creates integration, power, operator-compatibility, and resource-management challenges.

  • SoC and Edge Hardware Landscape: Edge hardware spans mobile, laptop, desktop, and IoT platforms with substantially different compute capabilities and memory capacities.Examples range from Snapdragon 8 Gen 3 at up to 20 tokens/sec to RTX 4090 at 1321 TOPS with 24GB memory.
  • SoC Architecture: Specialized accelerators in heterogeneous SoCs balance flexibility and performance, but coordinating data flow and power consumption remains challenging.Examples include Apple, Google, and Qualcomm platforms integrating general-purpose and parallel processing units.
  • GPUs: GPUs accelerate LLM inference through massive parallelism and tensor-oriented hardware, but their power demands can require hybrid cloud-local execution.GPU integration with CPUs and NPUs further complicates scheduling and resource management.
  • NPUs: NPUs improve performance and energy efficiency through low-precision arithmetic and parallel architectures for real-time inference.Apple Neural Engines and Qualcomm AI engines are cited as examples.
  • NPUs: NPU specialization limits compatibility with many modern LLM architectures and can require CPU-NPU co-processing or broader task offloading.Rapidly changing model architectures increase adaptation demands and may prevent some models from executing entirely.
  • Integrated Optimization: Offline compression and runtime optimization together form a holistic approach to improving edge LLM performance under resource constraints.Offline methods reduce complexity and memory footprint, while runtime methods emphasize resource utilization, adaptation, and scalability.

5 On-Device LLM-Based Applications

On-device LLM applications span personal, enterprise, and industrial domains, combining compact models with runtime optimization for efficient, private, and responsive edge AI. The review covers assistants, healthcare, robots, enterprise workflows, autonomous systems, fault localization, and anomaly detection.

  • Personal Applications: Personal applications provide low-latency, privacy-preserving, and continuously available assistance for tasks such as scheduling, health tracking, and device automation.AutoDroid uses multi-granularity token pruning to operate within mobile computational budgets.
  • Healthcare Assistants: Healthcare applications support clinical decision-making, mental-health analysis, pathology education, and near-real-time searches over wearable data.BioMistral combines domain-specific fine-tuning with quantization for low-latency inference.
  • Companion Robots: Companion robots use on-device LLMs for rapid, context-aware verbal and non-verbal interaction, spatial reasoning, planning, and physical action.3D-LLM, E2WM, and Inner Monologue illustrate embodied reasoning and execution.
  • Enterprise Applications: Enterprise systems apply on-device LLMs to message completion, meeting summarization, computer operations, privacy, and cost-efficiency.Examples include Gboard, MobiVQA, WebAgent, SheetCopilot, and the RCI agent.
  • Industrial Applications: Industrial applications support real-time analysis and decision-making for autonomous driving, software fault localization, and anomaly detection while reducing network overhead.Applications combine linguistic, visual, and multimodal capabilities for local industrial workflows.
  • Cross-Domain Synthesis: Offline compression and runtime methods jointly enable efficient, private, and responsive applications across personal, enterprise, and industrial edge environments.The review highlights quantization, pruning, cloud-edge collaboration, early exiting, and hardware acceleration as complementary strategies.

6 Future Directions and Open Challenges

Future work targets compact architectures, edge-cloud coordination, heterogeneous hardware deployment, graph-based reasoning, multi-agent collaboration, and continual personalization. These directions address limitations involving resource demands, communication, generalization across hardware, complex reasoning, coordination, and adaptation.

  • Compact LLM Architecture Development: Mamba and Jamba offer compact-architecture directions, but their computational demands, limited parallelization, and difficult compression remain deployment challenges.The selective state space and recurrent components complicate hardware acceleration, quantization, and pruning.
  • Innovative Edge-Cloud Collaboration: Edge-cloud collaboration can offset individual devices’ memory and computing limits, but distributed decoding still requires solutions for fault tolerance and communication costs.Split inference, speculative decoding, and disaggregated prefill and decoding are identified as promising techniques.
  • Heterogeneous Deployment with Simulation Techniques: Hardware-software co-design needs validation and scalability across general-purpose CPUs, GPUs, and NPUs rather than remaining limited to simulations or specific ASIC prototypes.The stated research direction is improving performance in real-world heterogeneous scenarios.
  • Graph-based LLM Development: Graph-based LLM development is needed for graph-related inference involving complex relationships and multi-hop reasoning in domains such as social networks, biology, and transportation.GraphRAG, GraphGPT, and GraphWiz are cited as approaches targeting structured extraction, graph interpretation, and complex reasoning.
  • Multi-Agent Collaboration: Multi-agent collaboration and continual learning address limits in dynamic coordination and personalization, but edge deployment remains constrained by bandwidth, latency, energy, and catastrophic forgetting.Collaborative intelligence can improve task performance beyond single-model systems, while continual learning supports dynamic adaptation under resource constraints.

7 Conclusion

This survey reviews techniques for enabling on-device LLMs across offline model design, online runtime inference, and applications. Together, these components form an optimization pipeline addressing efficiency, scalability, adaptability, and practical edge deployment.

  • 7 Conclusion: The survey systematically covers offline pre-deployment model design, online runtime inference optimization, and on-device LLM applications.These components are presented as the main stages of enabling on-device LLMs.
  • 7 Conclusion: Pre-deployment techniques such as quantization and pruning create compact, efficient models for edge use.The survey connects these methods to the first stage of its cohesive optimization pipeline.
  • 7 Conclusion: Runtime techniques support adaptability and performance across heterogeneous edge environments.This stage complements compact model creation by addressing deployment conditions after pre-deployment.
  • 7 Conclusion: Diverse on-device applications demonstrate the practical impact of edge LLMs.The conclusion positions applications as the pipeline’s practical-facing component.
Loading 2410.11845v2…