PaperlayerView exact revision on arXiv

Source-linked AI summary

Mobile Edge Intelligence for Large Language Models: A Contemporary Survey

Guanqiao Qu, Qiyuan Chen, Wei Wei, Zheng Lin, Xianhao Chen, Kaibin Huang

arXiv:2407.18921v2cs.NIcs.AIcs.LG

TL;DR

Cloud LLMs face latency, bandwidth, privacy, and resource challenges, while on-device LLMs remain constrained by limited device resources. This survey examines MEI for LLM caching and delivery, training, and inference, identifying methods and future opportunities for efficient edge deployment.

  • Problem

    Cloud provisioning can incur transmission delays, high bandwidth costs, and privacy risks, while on-device LLM deployment is constrained by excessive computing and memory demands.

  • Method

    The survey develops a 6G MEI4LLM architecture and synthesizes resource-efficient techniques for edge LLM caching and delivery, training, and inference.

  • Results

    The survey identifies MEI as a way to address edge LLM resource constraints through nearby edge-server computation and methods including speculative decoding, parallel training, and split learning.

  • Takeaways & Limitations

    MEI provides a middle-ground deployment paradigm for supporting privacy- and delay-sensitive LLM applications at the network edge.

  • Takeaways & Limitations

    Existing resource-efficient techniques tend to focus on individual devices and do not fully exploit networked computing resources or device communications.

Abstract

from arXiv · show

On-device large language models (LLMs), referring to running LLMs on edge devices, have raised considerable interest since they are more cost-effective, latency-efficient, and privacy-preserving compared with the cloud paradigm. Nonetheless, the performance of on-device LLMs is intrinsically constrained by resource limitations on edge devices. Sitting between cloud and on-device AI, mobile edge intelligence (MEI) presents a viable solution by provisioning AI capabilities at the edge of mobile networks, enabling end users to offload heavy AI computation to capable edge servers nearby. This article provides a contemporary survey on harnessing MEI for LLMs. We begin by illustrating several killer applications to demonstrate the urgent need for deploying LLMs at the network edge. Next, we present the preliminaries of LLMs and MEI, followed by resource-efficient LLM techniques. We then present an architectural overview of MEI for LLMs (MEI4LLM), outlining its core components and how it supports the deployment of LLMs. Subsequently, we delve into various aspects of MEI4LLM, extensively covering edge LLM caching and delivery, edge LLM training, and edge LLM inference. Finally, we identify future research opportunities. We hope this article inspires researchers in the field to leverage mobile edge computing to facilitate LLM deployment, thereby unleashing the potential of LLMs across various privacy- and delay-sensitive applications.

I. INTRODUCTION

LLMs support diverse applications, but cloud and on-device deployment each face important constraints. The survey motivates mobile edge intelligence as an intermediate paradigm and reviews resource-efficient MEI-based deployment across applications and system functions.

  • Motivation: LLMs support general-purpose, multimodal, and domain-specific applications such as healthcare and robotics.Examples include multimodal content analysis, medical question answering, and robot control.
  • Motivation: Cloud LLM provisioning creates privacy risks, bandwidth costs, and communication latency because user data must be uploaded to cloud centers.Sensitive information may be exposed during transmission and storage.
  • Motivation: On-device LLM deployment is constrained by scarce computing, memory, and storage resources, limiting supported model scale and functionality.Current industrial efforts focus on sub-10B models, while larger models may be needed for more complex functions.
  • MEI motivation: Mobile edge intelligence places computing resources between devices and cloud centers, enabling nearby edge servers to support larger LLMs and continuous fine-tuning.The paradigm is intended to reduce latency and bandwidth costs while using edge-server memory, energy, and computing resources.
  • Survey scope: The survey covers edge applications, an MEI framework, resource-efficient caching and delivery, training, inference, and future directions.Its application analysis emphasizes latency, bandwidth, and privacy requirements; its technical coverage emphasizes storage, communication, and computing efficiency.

III. PRELIMINARIES I: AN OVERVIEW OF LLMS AND MEI

The paper introduces LLM and MEI preliminaries before reviewing resource-efficient techniques needed for edge deployment.

  • Section overview: The survey presents foundational concepts of LLMs and MEI before discussing resource-efficient LLM techniques for edge deployment.These preliminaries establish the basis for understanding how MEI can support LLM services.

A. Large Language Models

Transformers use self-attention within stacked encoder-decoder layers to model long-range dependencies and generate outputs autoregressively. Their tokenization, embeddings, positional information, and attention-based processing form the foundation of many LLMs.

  • Transformers use self-attention to capture long-range relationships between sequence elements, overcoming recurrent models’ short-term context limitation.
  • A typical Transformer contains stacked encoder-decoder layers with multi-head self-attention, layer normalization, and feed-forward networks.
  • Input text is tokenized, converted into embeddings with positional information, and processed through attention-based layers into contextual representations.
  • The decoder generates output sequences autoregressively, predicting each new token from the input and previously generated tokens.

2) Unimodal LLMs:

LLMs are billion-parameter Transformer-based models whose capabilities extend from language processing to multimodal understanding and generation. Unimodal models process text, while multimodal LLMs connect separately pretrained modality foundations through alignment and instruction tuning.

  • Unimodal LLMs: LLMs are advanced Transformer-based language models with billions of parameters, pretrained on massive datasets for comprehension, reasoning, and text generation.
  • Unimodal LLMs: LLM architectures comprise encoder-only, encoder-decoder, and decoder-only categories, each differing in how inputs are represented and outputs are generated.
  • Unimodal LLMs: Traditional unimodal LLMs rely mainly on textual inputs, limiting their ability to comprehend visual, auditory, and sensor-based information.
  • Multimodal LLMs: Multimodal LLMs integrate text, images, audio, and other modality foundations to support diverse tasks, including image description and speech recognition.
  • Multimodal LLMs: The main construction challenge is connecting separately pretrained modality models through multimodal pretraining and instruction-tuning for cross-modal alignment.
  • Practical applications: Generative/Interactive AI: Generative AI creates content across modalities, while interactive AI uses LLMs and user interaction to support adaptive conversations in applications such as chatbots and virtual assistants.
  • Practical applications: Generative/Interactive AI: Retrieval-augmented generation retrieves relevant external knowledge during response generation to improve content generation performance.

B. Mobile Edge Intelligence

Mobile edge intelligence places computing resources between devices and cloud centers, allowing mobile networks to support computation-intensive AI services near users. For LLMs, this convergence creates challenges in communication, computation, memory, storage, and energy efficiency.

  • Mobile Edge Intelligence: MEI provides computing services through nearby edge servers, reducing edge-device computing burdens and avoiding direct transfer of massive data to the cloud.
  • Mobile Edge Intelligence: Edge split inference partitions an AI model so computation-intensive and energy-intensive sub-models run on 5G edge servers, while devices upload intermediate data.
  • Mobile Edge Intelligence: LLM deployment across mobile edge networks must address communication overhead, computing demands, resource efficiency, and energy efficiency while maintaining performance.
  • Lessons Learned: On-device LLM deployment is constrained by excessive computation, storage and memory demands, and high energy costs.
  • Lessons Learned: A LLAMA2-7B model requires about 1700 giga floating point operations for one forward pass on 128 tokens, while backward propagation is typically more demanding.
  • Lessons Learned: Adam-based training usually requires 12 times the memory needed for inference, creating stringent memory requirements for mobile devices.
  • Lessons Learned: Running a 13-billion-parameter INT4 LLM on a Xiaomi 11 consumes about 56 J/token, allowing approximately 700 tokens from a 3000 mAh battery.

1) LLM compression:

LLM compression reduces model size and inference costs for resource-constrained edge devices, while fast decoding lowers latency and energy use during generation. The surveyed techniques include quantization, pruning, knowledge distillation, and decoding strategies such as speculative decoding.

  • LLM compression: LLM compression reduces model size, computing demands, storage usage, and inference energy while aiming to preserve accuracy.The survey covers compression strategies tailored to LLM architectures and properties.
  • Quantization: Quantization converts high-precision parameters such as FP16 into lower-precision formats such as INT4, with PTQ and QAT as the main categories.Quantization reduces storage usage, computing latency, and inference energy.
  • Quantization: Uneven weight quantization preserves approximately 1% of critical weights in FP16 while quantizing most weights to INT3/INT4.The approach exploits unequal contributions of weights to inference results.
  • Pruning: Pruning removes redundant parameters through structured or unstructured patterns, reducing computing workload and storage usage.Unstructured pruning can require specialized hardware or software platforms during deployment.
  • Pruning: 60% sparsity is reported for OPT-175B with SparseGPT before significant accuracy loss occurs.Sparse models produced by unstructured pruning may require specialized deployment support.
  • Knowledge distillation: Knowledge distillation transfers knowledge from a large teacher model to a smaller student model for deployment on resource-constrained edge devices.The survey distinguishes white-box and black-box distillation according to model accessibility.
  • Fast decoding: Fast decoding techniques target autoregressive inference overhead by reducing computation, parameter use, or intermediate data generation.The surveyed methods include speculative decoding, early exit, mixture-of-experts, contextual sparsity prediction, and parallel decoding.
  • Fast decoding: Speculative decoding keeps a lightweight model in memory while a powerful LLM verifies its generated sequence, reducing memory-loading operations.It can halve per-token generation latency and energy consumption while maintaining generated-content quality.

B. Resource-efficient Fine-tuning

Resource-efficient fine-tuning reduces the memory and computation required to adapt LLMs on constrained devices. The survey covers PEFT families, low-rank adaptation, and zeroth-order optimization.

  • Motivation: On-device LLM training requires substantially more memory and computation than inference, making fine-tuning a practical alternative to full-parameter training.Computing OPT-13B gradients consumes 12 times the memory needed for inference.
  • PEFT: PEFT updates only a small number of parameters and comprises additive, selective, and reparameterized categories.These categories organize the surveyed approaches by how trainable parameters are introduced or selected.
  • Additive PEFT: Additive PEFT inserts minimally sized trainable components while freezing pretrained parameters, including adapters, prompts, and prefixes.The three additive types are adapter tuning, prompt tuning, and prefix tuning.
  • Selective PEFT: Selective PEFT preserves the model architecture by updating only a small subset of parameters, but sparse selections may be poorly supported by frameworks and hardware.Additive methods can introduce inference latency through additional parameters, motivating selective approaches.
  • Reparameterized PEFT: Reparameterized PEFT uses low-rank matrices while freezing the pretrained weight matrix, with LoRA as a prominent example.QLoRA combines quantization with LoRA to reduce memory consumption.
  • Reparameterized PEFT: QLoRA fine-tuned a 65B-parameter language model on a 48 GB GPU within 24 hours and achieved 99.3% of ChatGPT’s performance on evaluated tasks.The result demonstrates the reported effectiveness of combining quantization with LoRA.
  • Zeroth-order optimization: Zeroth-order optimization estimates gradient updates through forward passes, avoiding intermediate backpropagation results and reducing training memory.MeZO used 1/12 of Adam’s running memory, remained competitive on 7 of 11 tasks, and caused less than 1% accuracy degradation.

C. Lessons Learned

Resource-efficient techniques reduce computation and memory for individual devices but do not fully exploit collaboration across networked devices and edge servers. MEI4LLM addresses this gap through an AI-native architecture supporting shared delivery, distributed training, and distributed inference.

  • Lessons Learned: Resource-efficient inference shrinks model sizes or accelerates data processing, while fine-tuning reduces updated parameters or uses forward passes only.These techniques target efficient LLM training and inference on resource-limited edge devices and servers.
  • Lessons Learned: Single-device resource-efficient methods underuse networked computing and communications, motivating collaboration among edge devices and edge servers.MEI focuses on exploiting connected computing resources for model inference and training.
  • MEI4LLM: MEI4LLM aims to improve communication, computing, and storage efficiency through collaboration among networked edge devices and servers.The framework also identifies security defenses for training, inference, and model or data storage as a future consideration.
  • MEI4LLM: MEI4LLM contains AI-native architecture, parameter-sharing LLM caching and delivery, distributed LLM training, and distributed LLM inference.These components support LLM deployment and operations within mobile edge networks.
  • AI-native architecture: AI-native 6G networks are expected to use task-oriented objectives and native model partitioning across edge servers and devices.The task-oriented design can optimize output quality rather than only throughput or latency.
  • AI-native architecture: A neural edge distributes network layers across edge nodes and requires coordinated model splitting, placement, and data routing.The architecture is intended to support collaborative computing across distributed edge devices and servers.
  • LLM caching and delivery: Information-centric networking can support parameter-block naming and name-based transmission for model, feature, and data delivery.Nearby edge caches can receive parameter requests for popular named parameter blocks.
  • LLM caching and delivery: Caching and delivery should exploit parameter blocks shared across downstream LLMs or reused within one LLM to reduce delivery costs.External knowledge for RAG may also be cached at the edge, trading storage and delivery latency against fine-tuning computation.

D. Distributed LLM Inference

Distributed LLM inference at the edge must coordinate edge servers and devices under communication, computation, and privacy constraints. MEI4LLM addresses these deployment challenges through model splitting, parallel processing, and resource-efficient techniques such as parameter sharing.

  • MEI4LLM requires model splitting and parallel training or inference across interconnected edge nodes to support large-scale LLM deployment.
  • Edge LLM caching reduces downloading latency by placing models on nearby wireless edge servers instead of relying on remote cloud data centers.
  • Parameter-sharing caching stores shared LLM parameters once, while replacement strategies must account for cooperative caching, user mobility, and inference accuracy.
  • LoRA-based fine-tuning freezes pretrained parameters and trains newly introduced parameters, typically less than 1% of the original model.
  • Edge LLM caching and delivery remain challenging because model popularity changes and LLMs have enormous size, requiring joint consideration of storage, bandwidth, and model capability.

B L E

Parameter sharing enables compact adaptation and more efficient edge caching for fine-tuned LLMs. Edge cache performance depends on server capacity, model popularity, cooperative replacement, and latency-constrained service coverage.

  • LoRA integrates fine-tuned parameters ranging from hundreds of kilobytes to tens of megabytes into a pretrained GPT-2 large model of around 3.02 GB.
  • TrimCaching stores shared parameters across LLMs only once on edge servers to improve storage efficiency.
  • A cache hit occurs when any edge server serves the requested GPT-2 model within end-to-end latency requirements.
  • Naive LRU and LFU replacement strategies fail to exploit cooperative caching and parameter shareability among LLMs.
  • High user mobility can make centralized proactive caching costly and complex, motivating distributed replacement algorithms.

2) Parameter-sharing wireless model downloading:

Parameter-sharing wireless model downloading reduces edge delivery costs by transmitting and multicasting only reusable or missing parameter blocks. Effective delivery must balance latency, model accuracy, routing, placement, and compression decisions.

  • 2) Parameter-sharing wireless model downloading:: Wireless model downloading can multicast reusable parameter blocks to reduce downloading latency.
  • 2) Parameter-sharing wireless model downloading:: More shared parameter blocks lower downloading latency but can degrade downstream model performance, creating an accuracy-latency trade-off.
  • 2) Parameter-sharing wireless model downloading:: Parameter-sharing caching and delivery are coupled because model placement affects network traffic while radio-resource allocation and routing affect placement decisions.
  • Edge LLM training: Edge LLM training includes centralized, federated, split, and hierarchical learning, with large model scale and wireless PEFT creating distinctive challenges.
  • Edge LLM training: Centralized edge training can require 112 GB of GPU memory for LLaMA-2 7B in FP32, exceeding the memory of an 80 GB H100 GPU.

2) Parallel LLM training at the network edge:

Parallel LLM training at the edge distributes model computation across heterogeneous nodes but introduces communication bottlenecks and placement challenges. Importance-aware data transmission and parameter-efficient federated methods address complementary training costs and privacy concerns.

  • 2) Parallel LLM training at the network edge:: Pipeline parallelism can significantly reduce end-to-end training latency by splitting models into sub-models placed across edge servers.
  • 2) Parallel LLM training at the network edge:: Model splitting and placement must account for heterogeneous communication and computing capabilities and high-dimensional intermediate data exchange.
  • 2) Parallel LLM training at the network edge:: Parallel training differs from multi-hop inference because training processes batches divided into micro-batches rather than one input at a time.
  • 3) Importance-aware token uploading:: Multimodal training data uploads can congest networks, while allocating more radio resources to important samples can improve convergence rate and model accuracy.
  • B. Federated Edge Learning: Federated LLM training faces excessive on-device workload, massive model-transfer traffic, and resource heterogeneity across clients.
  • B. Federated Edge Learning: Parameter-efficient federated fine-tuning reduces communication and computing overhead by updating and uploading only a small parameter subset.

2) Resource management in FL for LLMs:

FL-based LLM training remains difficult for lightweight edge devices because local parameter updates and wireless communication are resource-intensive. Split learning and privacy-preserving variants reduce device workload but introduce smashed-data, target-token, and resource-allocation challenges.

  • LLM fine-tuning over wireless FL remains nascent because the trainable-parameter proportion must balance training performance against communication and computing latency.Adaptive parameter freezing can reduce synchronization, but existing work does not consider dynamic wireless channel conditions.
  • Billions of parameters make local LLM updates challenging for smartphones and IoT devices even when parameter-efficient fine-tuning is used.Split learning addresses this by moving most training computation to an edge server.
  • Split learning places only a sub-model on edge devices, reducing their workload, but sequential server-device interactions create idle-device waiting time.The paradigm has also been applied in healthcare scenarios.
  • Split learning faces communication overhead from high-dimensional smashed data and privacy leakage from transmitting ground-truth target tokens.Importance-aware pruning can reduce communication costs by around 50% compared with a benchmark without smashed-data selection.
  • Effective edge LLM training requires jointly optimizing model splitting and radio-resource allocation under device, communication, and training-status constraints.Transformer-based LLMs offer repeated blocks that may make this optimization more tractable, while shallower split points can reduce latency when device computing is limiting.

D. Hierarchical Collaborative Learning

Hierarchical collaborative learning combines clouds, edge servers, and edge devices to match LLM training workloads with heterogeneous resources. The survey emphasizes splitting training and updating only a small portion of parameters while adapting model placement and fine-tuning to network conditions.

  • D. Hierarchical Collaborative Learning: Hierarchical collaborative learning combines clouds, edge servers, and edge devices to accommodate varied task complexity and resource availability.Computation-intensive tasks can be offloaded to clouds, while easier tasks remain at the network edge.
  • D. Hierarchical Collaborative Learning: DLoRA keeps personal PEFT modules on edge devices while frozen LLM parameters remain in the cloud, enabling collaborative training through exchanged activations and gradients.
  • D. Hierarchical Collaborative Learning: Cloud-edge collaboration can pre-train LLMs in the cloud and fine-tune them on context-aware edge data, or aggregate edge-trained updates in the cloud.
  • D. Hierarchical Collaborative Learning: Three-tier cloud-edge-end learning improves resource-management flexibility by aggregating models first at edge servers and then at the cloud.Model quantization can be used to reduce communication costs in hierarchical federated learning.
  • D. Hierarchical Collaborative Learning: Hierarchical LLM training must dynamically coordinate aggregation intervals, model splitting, and PEFT parameter proportions using training status, wireless conditions, and Internet delay.
  • E. Lessons Learned: The guiding principle for edge LLM training is to split the workload and update only a small portion of parameters through parallel training and PEFT.The survey calls for flexible splitting and theoretical characterization of accuracy versus trainable-parameter percentage.

1) LLM inference with cross-modal input token reduction:

Edge LLM inference with cross-modal input reduction seeks to lower communication, memory, and computing demands while preserving useful information. Split inference distributes models across devices and servers, with representation reduction, feature extraction, and channel-aware designs providing key mechanisms.

  • 1) LLM inference with cross-modal input token reduction: Input token reduction removes unimportant text or image tokens before inference, reducing offloaded data without significant reported accuracy loss.
  • 2) Parameter-sharing service placement/migration for LLM inference: Limited edge-server memory restricts how many large LLM inference services can be supported, making service placement and migration important.
  • 2) Parameter-sharing service placement/migration for LLM inference: Parameter-sharing placement can load multiple substantially shared LLMs concurrently, improving throughput while jointly accounting for memory, storage, computing, and spectrum constraints.
  • 3) Split inference for LLMs: Autoregressive decoding makes inference delay unpredictable because output length is initially unknown, while KV-cache memory usage grows with generated tokens.
  • 3) Split inference for LLMs: Split inference partitions models between devices and servers, reducing device computation while retaining raw data locally for privacy-sensitive applications.
  • 1) LLM inference with cross-modal input token reduction: Intermediate representations can be reduced through quantization, pruning, merging, or lower-dimensional information-bottleneck feature extraction.The cut layer must balance communication efficiency against device-side computing burden.
  • 1) LLM inference with cross-modal input token reduction: Cross-modal representation reduction and channel-aware joint source-channel coding can reduce multimodal communication and improve robustness to channel noise and interference.

2) Progressive split inference:

Progressive and other split-inference variants reduce transmitted intermediate features or protect sensitive inference outputs while distributing LLM computation across edge devices and servers. Their design must jointly address communication, latency, privacy, and resource constraints.

  • 2) Progressive split inference: Progressive feature offloading sends higher-importance representations first and terminates transmission once the desired inference accuracy is satisfied.
  • 2) Progressive split inference: Early-exit modules can skip later user- or server-side layers, reducing computation and avoiding unnecessary intermediate-feature uploads.
  • 4) Other variants of split inference: U-shaped or Λ-shaped split inference prevents edge servers from recovering raw data or inference results by returning high-dimensional, difficult-to-interpret outputs.
  • 4) Other variants of split inference: Multi-hop split inference partitions an LLM across multiple edge devices and servers according to their computing abilities and inter-device communication conditions.
  • 4) Other variants of split inference: Speculative decoding offers a device-server collaboration mode that exchanges smaller information than high-dimensional intermediate features.
  • 4) Other variants of split inference: Edge LLM inference requires joint optimization of KV-cache management, multimodal feature extraction, and autoregressive decoding because they affect memory, communication, and latency.

A. Green Edge LLM

Green edge LLM research targets energy efficiency, privacy, security, data quality, and resource constraints in distributed edge training and inference. The survey frames integrated communication-computing design and automated quality control as key directions for practical deployment.

  • Energy efficiency: GPT-4 training consumes energy comparable to 5–6 years of 1,000 average US households, highlighting the need for energy-efficient LLM techniques.The passage also notes that inference energy costs can be higher because of frequent worldwide service requests.
  • Energy efficiency: Pushing intelligence to the network edge can reduce energy used for data transportation and improve training or inference efficiency through integrated communication-computing design.The survey identifies reduced data communication as one example of joint optimization.
  • Energy efficiency: Green edge LLMs jointly optimize wireless communication and computing energy, targeting either device-level savings or lower overall system energy.Offloading training or inference to edge servers is beneficial when it reduces energy use for end users.
  • Security: Secure edge LLM research must address inference attacks, data poisoning, and backdoors because distributed training can expose private information or produce harmful outputs.The survey notes that edge LLM security has received less attention than general LLM security.
  • Training quality: Edge training requires automated quality control because uncontrolled, low-quality data and labels can harm model performance.Proposed directions include increasing dataset diversity and exploiting unlabeled or noisy edge data.
  • Research scope: The survey advocates 6G MEI for LLM deployment and covers edge caching, delivery, training, and inference under resource constraints.It identifies resource limitations at the network edge as the main challenges motivating these methods.
Loading 2407.18921v2…