PaperlayerView exact revision on arXiv

Source-linked AI summary

Pushing Large Language Models to the 6G Edge: Vision, Challenges, and Opportunities

Zheng Lin, Guanqiao Qu, Qiyuan Chen, Xianhao Chen, Zhe Chen, Kaibin Huang

arXiv:2309.16739v4cs.LGcs.AI

TL;DR

Cloud LLM deployment faces latency, bandwidth, and privacy challenges, while on-device deployment is limited by edge-device resources. The paper explores 6G MEC end-edge cooperation, proposes an architecture, and surveys training and inference techniques. It concludes that split learning and inference, parameter-efficient adaptation, quantization, parameter sharing, and related methods can support efficient edge deployment.

  • Problem

    Cloud LLM deployment incurs long latency, high bandwidth costs, and privacy concerns, while on-device LLMs face scarce computing resources.

  • Method

    The paper proposes a 6G MEC end-edge cooperation architecture and surveys techniques for collaborative LLM fine-tuning and inference.

  • Results

    The paper identifies enabling techniques including split learning/inference, speculative decoding, parameter sharing, and energy- and privacy-aware cooperation for efficient edge deployment.

  • Takeaways & Limitations

    6G MEC can provide a pathway for deploying LLMs near users by combining edge-device and MEC-server capabilities.

  • Takeaways & Limitations

    On-device deployment remains constrained by computational latency, including approximately 2.3 seconds per token for LLaMA-7B on an Apple M1 Max processor.

Abstract

from arXiv · show

Large language models (LLMs), which have shown remarkable capabilities, are revolutionizing AI development and potentially shaping our future. However, given their multimodality, the status quo cloud-based deployment faces some critical challenges: 1) long response time; 2) high bandwidth costs; and 3) the violation of data privacy. 6G mobile edge computing (MEC) systems may resolve these pressing issues. In this article, we explore the potential of deploying LLMs at the 6G edge. We start by introducing killer applications powered by multimodal LLMs, including robotics and healthcare, to highlight the need for deploying LLMs in the vicinity of end users. Then, we identify the critical challenges for LLM deployment at the edge and envision the 6G MEC architecture for LLMs. Furthermore, we delve into two design aspects, i.e., edge training and edge inference for LLMs. In both aspects, considering the inherent resource limitations at the edge, we discuss various cutting-edge techniques, including split learning/inference, parameter-efficient fine-tuning, quantization, and parameter-sharing inference, to facilitate the efficient deployment of LLMs. This article serves as a position paper for thoroughly identifying the motivation, challenges, and pathway for empowering LLMs at the 6G edge.

I. INTRODUCTION

LLMs offer broad capabilities but cloud deployment incurs latency, bandwidth, and privacy problems, while on-device deployment is constrained by limited resources. The paper proposes 6G MEC end-edge cooperation and surveys supporting architecture and techniques.

  • LLMs generalize across applications including chatbots, content generation, healthcare, and robotics.
  • Cloud-based LLMs suffer excessive latency, high bandwidth costs, and severe privacy concerns, especially for multimodal and sensitive-data applications.
  • On-device LLMs reduce transmission and privacy leakage but are limited by scarce computing capabilities and typically contain fewer than 10 billion parameters.
  • 6G MEC enables end-edge cooperation, combining localized computing with more resources than edge devices alone.
  • The paper presents a 6G MEC architecture and discusses communication-computing techniques for LLM training and inference.

II. KILLER APPLICATIONS: THE NEEDS FOR DEPLOYMENT AT THE EDGE

Healthcare and robotics illustrate how multimodal LLM applications can benefit from deployment near users. These use cases make bandwidth, latency, and privacy concerns especially relevant at the network edge.

  • Healthcare: Healthcare is a mission-critical LLM application requiring interactions with patients, caregivers, and medical professionals.
  • Healthcare: Med-PaLM 2 achieved 86.5% accuracy on the US Medical License Exam after fine-tuning on medical datasets.
  • Humanoid robots: LLMs support robotic understanding of human intentions and complex environments, followed by sequential manipulation planning.
  • Healthcare and robotics together underscore the need for edge deployment to address bandwidth, latency, and privacy concerns.

III. LIMITATIONS FOR ON-DEVICE LLM DEPLOYMENT

On-device LLM deployment is limited by computational, energy, memory, and storage demands. The paper therefore positions end-edge cooperation with capable MEC hardware as an alternative deployment option.

  • On-device LLM deployment faces critical limitations that motivate deployment at the network edge.
  • 2.3 seconds per token makes LLaMA-7B latency prohibitive for real-time or long-form generation on an Apple M1 Max processor.
  • 0.7 J per token means processing 700 tokens consumes around 1% of a fully charged iPhone’s approximately 50kJ battery.
  • 84GB of memory is required for full-parameter fine-tuning of a 7-billion-parameter LLM at 16-bit precision, exceeding smartphones’ 8 GB RAM.
  • End-edge cooperation using modern 5G/6G MEC servers with high-performance accelerators may provide a better option for pushing LLMs to the network edge.

IV. 6G MEC FOR LARGE LANGUAGE MODELS: AN OVERVIEW

The proposed 6G MEC architecture organizes cloud servers, MEC servers, and users into cooperative hierarchies. It also supports horizontal MEC cooperation and model partitioning for training and inference.

  • The 6G MEC architecture is designed to support LLM deployment through critical cooperating modules.
  • Cloud servers retain pretrained foundation models that can be downloaded to edge networks and fine-tuned for context-specific downstream tasks.
  • MEC servers can cooperate horizontally to reduce communication latency and costs when cloud-edge links become bottlenecks.
  • Models can be partitioned layer-wise or by tensors across MEC servers for inference or training.

3) Parameter-sharing edge model caching and delivery:

6G MEC must cache, place, and migrate LLM components efficiently across edge servers. Parameter-efficient fine-tuning enables downstream models to share most parameters, so only task-specific components need migration.

  • 6G MEC architecture must store, cache, and migrate LLMs across edge networks for rapid delivery to users and distributed learning.
  • Parameter-efficient methods such as LoRA let downstream LLMs share most parameters, supporting more efficient model placement and migration.
  • Cloud-pretrained LLMs can be fine-tuned at the edge for new environments and personalized user needs.

1) Split learning:

Split learning distributes LLM training between clients and servers while avoiding raw-data sharing, and PEFT reduces the parameter, memory, latency, and communication burdens of adaptation. SplitLoRA combines these approaches but exposes an accuracy–latency trade-off controlled by freezing ratios.

  • 1) Split learning: Split learning partitions a model between clients and a server, reducing local computation while preventing raw data sharing with the server.
  • 1) Split learning: Parallel split learning and split federated learning allow multiple devices to train with a server simultaneously, accelerating training.
  • 1) Split learning: Full-parameter fine-tuning is prohibitively expensive at the edge and creates considerable model-aggregation communication costs in SFL.
  • 1) Split learning: PEFT updates typically less than 1% of model parameters, reducing training latency, communication latency, and memory consumption.
  • 1) Split learning: SplitLoRA integrates split learning with LoRA, offloading substantial training workloads to servers while updating a small portion of parameters.
  • 1) Split learning: A higher freezing ratio decreases communication-computing latency but may compromise model performance by restricting parameter updates.

3) Towards efficient split PEFT:

Efficient split PEFT requires jointly choosing model partitions, freezing ratios, and parallelism strategies under edge resource constraints. Multi-hop and other forms of parallelism distribute workloads across servers, but heterogeneity can create bottlenecks and added communication latency.

  • 3) Towards efficient split PEFT: GPT-3 split training with 100 samples of 1024 tokens can produce approximately 420 MB of smashed data in one training round.
  • 3) Towards efficient split PEFT: Model split strategies and freezing ratios should be jointly selected because both affect communication, computation, and model convergence.
  • 3) Towards efficient split PEFT: Freezing client-side models allows one forward pass and limits smashed-data uploading to the first training epoch, saving communication bandwidth.
  • 3) Towards efficient split PEFT: Pipeline parallelism acts as multi-hop split learning by processing micro-batches across multiple MEC servers collaboratively.
  • 3) Towards efficient split PEFT: Tensor parallelism distributes layer operators across processors, while data parallelism replicates models across processors with training datasets.
  • 3) Towards efficient split PEFT: Edge LLM parallelism must account for heterogeneous computing, memory, and communication capabilities, unlike distributed training in typically homogeneous GPU clusters.
  • 3) Towards efficient split PEFT: Adding edge servers can exploit distributed resources but also increases communication overhead and potentially lengthens end-to-end latency.

VI. END-EDGE LARGE MODEL INFERENCE

End-edge co-inference addresses the tension between cloud communication latency and limited edge-device computing resources. Split inference partitions models so devices cooperate with MEC servers, preserving privacy and potentially reducing communication for multimodal inputs.

  • VI. END-EDGE LARGE MODEL INFERENCE: Cloud inference incurs communication latency, whereas edge devices may incur long computing latency because of limited resources.
  • VI. END-EDGE LARGE MODEL INFERENCE: Split inference offloads model computation from edge devices to servers through model partitioning.
  • VI. END-EDGE LARGE MODEL INFERENCE: Cooperation between edge devices and MEC servers can improve privacy preservation and reduce communication when cut-layer features are smaller than raw inputs.
  • VI. END-EDGE LARGE MODEL INFERENCE: The communication benefit is especially pronounced for multimodal LLMs processing high-definition images or videos.

1) Split inference with KV caching:

Split inference partitions LLM execution across edge devices and servers, but autoregressive generation creates communication and memory bottlenecks. KV caching reduces exchanged data by retaining prior token information, while MoE can reduce computational and memory overhead.

  • Split inference with KV caching: Autoregressive split inference can create a communication bottleneck because smashed-data exchange grows with input-token length.Generated tokens are incorporated into subsequent inputs, increasing cut-layer data exchange.
  • Split inference with KV caching: KV caching limits smashed-data exchange to newly added tokens rather than all preceding tokens.The approach stores prior keys and values, trading additional memory for lower communication and computing workload.
  • Split inference with KV caching: A single FP16 GPT 6.7B inference task with 512 input sequences and 32 output sequences requires approximately 41.84 GB of running memory at batch size 64.Increasing applications and sequence lengths further increases the running-memory burden.
  • Split inference with KV caching: Mixture-of-experts divides an LLM into specialized expert subnetworks selected dynamically by a gating mechanism.This design processes inputs through relevant experts while reducing computational and memory overheads.
  • Split inference with KV caching: Combining split inference with MoE partitions execution across devices and servers while assigning specialized experts to improve efficiency in resource-constrained deployments.The edge device can offload workload to selected edge-server experts.

B. Small-Large Language Model (SLM-LLM) Cooperation

SLM-LLM cooperation reduces communication by letting edge devices generate or execute with small models while servers use large models for verification and correction. Speculative decoding accelerates this process by parallelizing large-model verification, though adaptive resource-aware threshold control remains an open issue.

  • Small-Large Language Model (SLM-LLM) Cooperation: SLM-LLM co-inference lets edge devices execute small models while edge servers occasionally verify or correct their results with LLMs.Because SLM outputs are small, this arrangement can reduce communication overhead relative to transmitting high-dimensional intermediate features.
  • Small-Large Language Model (SLM-LLM) Cooperation: Speculative decoding uses a smaller model to draft multiple tokens and a larger model to verify and correct them in parallel.This removes server-side autoregressive generation for the drafted tokens and can significantly reduce inference latency.
  • Small-Large Language Model (SLM-LLM) Cooperation: Hybrid SLM-LLM cooperation achieved LLM-level performance with about 25.8% of computing cost on datasets such as GSM8K.The reported approach deploys SLMs on edge devices and uses prediction confidence and an acceptance threshold for server verification.
  • Small-Large Language Model (SLM-LLM) Cooperation: Adaptive control of speculative-decoding thresholds according to communication and computing resources remains an open research problem.The cited hybrid approach does not adapt its thresholds to wireless-resource conditions.

A. Green and Sustainable End-edge Cooperation

Green end-edge cooperation seeks to reduce the energy footprint of LLM training and inference through model selection, scheduling, data selection, uncertainty-aware decisions, and joint optimization of splitting with parameter-efficient fine-tuning. Privacy remains a separate challenge because intermediate data, parameters, and draft tokens can still expose sensitive information.

  • Green and Sustainable End-edge Cooperation: LLM training and inference are power-hungry because of their huge model size, making energy-aware MEC scheduling and model selection necessary.The paper proposes using smaller models for simpler tasks and larger edge-server models for challenging tasks.
  • Green and Sustainable End-edge Cooperation: Uncertainty-aware mechanisms can let SLMs decide when to rely on LLM corrections using confidence scores.This targets improved training and inference efficiency while limiting unnecessary large-model use.
  • Green and Sustainable End-edge Cooperation: Joint optimization of model splitting and PEFT can shift more energy demand toward fixed infrastructure through adaptive freezing ratios under the same accuracy constraints.The paper identifies reinforcement-learning and gradient-based strategies as possible ways to select freezing ratios while balancing energy, communication, and training trade-offs.
  • Green and Sustainable End-edge Cooperation: Smashed data, model parameters, and SLM draft tokens may still create privacy-breach risks despite split learning and split federated learning.The paper discusses differential privacy, homomorphic encryption, and wireless-channel-noise mechanisms as possible protections.
  • Green and Sustainable End-edge Cooperation: The paper proposes a 6G MEC architecture and enabling techniques for efficient end-edge LLM fine-tuning and inference.Its position is to move LLM deployment from cloud computing toward 6G MEC with end-edge cooperation.
Loading 2309.16739v4…