PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Resource-efficient LLM and Multimodal Foundation Models

Mengwei Xu, Wangsong Yin, Dongqi Cai, Rongjie Yi, Daliang Xu, Qipeng Wang, Bingyang Wu, Yihao Zhao, Chen Yang, Shihe Wang, Qiyang Zhang, Zhenyan Lu, Li Zhang, Shangguang Wang, Yuanchun Li, Yunxin Liu, Xin Jin, Xuanzhe Liu

arXiv:2401.08092v2cs.LGcs.AIcs.DC

TL;DR

Large foundation models demand substantial hardware resources across training and deployment. This survey examines algorithmic and systemic approaches to resource efficiency and provides a systematic overview of recent literature on resource-efficient foundation models.

  • Problem

    Foundation models require substantial computing, memory, energy, and network resources for training and deployment, motivating research into resource-efficient strategies.

    Source: §1 · p. 2

  • Method

    The survey examines approaches spanning algorithmic efficiency, system optimizations, data management, and less resource-intensive architectures, from cloud to edge and devices.

    Source: §1 · p. 2

  • Results

    The survey provides a holistic, systematic overview of recent literature on resource-efficient large foundation models.

    Source: §6 · p. 38

  • Takeaways & Limitations

    The authors conclude that research in this domain will remain crucial as scaling points toward more powerful, larger models.

    Source: §6 · p. 38

    The survey identifies scaling laws as a limitation for lightweight models, noting that small-scale models are unlikely to attain the intelligence levels of larger models.

    Source: §6 · p. 38

Abstract

from arXiv · show

Large foundation models, including large language models (LLMs), vision transformers (ViTs), diffusion, and LLM-based multimodal models, are revolutionizing the entire machine learning lifecycle, from training to deployment. However, the substantial advancements in versatility and performance these models offer come at a significant cost in terms of hardware resources. To support the growth of these large models in a scalable and environmentally sustainable way, there has been a considerable focus on developing resource-efficient strategies. This survey delves into the critical importance of such research, examining both algorithmic and systemic aspects. It offers a comprehensive analysis and valuable insights gleaned from existing literature, encompassing a broad array of topics from cutting-edge model architectures and training/serving algorithms to practical system designs and implementations. The goal of this survey is to provide an overarching understanding of how current approaches are tackling the resource challenges posed by large foundation models and to potentially inspire future breakthroughs in this field.

1 INTRODUCTION

Foundation models offer broad zero-shot capabilities and improve with scale, but their growing resource demands constrain sustainable development and wider access. This survey reviews algorithmic and system innovations for improving resource efficiency across cloud, edge, and device settings.

Introduction

  • Foundation models generalize across open-world tasks, while scaling model size and data can predictably improve transformer performance [177].Examples include LLMs, ViTs, diffusion models, and multimodal models that support tasks such as image retrieval and visual question answering.Source: §1 · pp. 1–2
  • Training and deployment demand substantial compute, memory, energy, and bandwidth, with resource needs escalating as foundation models grow in size and complexity.LLaMa-2-70B pretraining is reported to consume 2.5 × 1012 Joules and produce an estimated 291 tons of CO2 equivalent; data processing, experimentation, and inference can consume comparable or greater electricity.Source: §1 · p. 2
  • Large resource footprints limit democratization: only a few major players could train and deploy state-of-the-art models by the end of 2023, while demonstrated on-device models remained below 10B parameters and lacked real-world deployment.Cloud serving also makes privacy preservation almost impossible.Source: §1 · p. 2
  • The survey examines algorithmic, system, data-management, and architectural strategies to reduce foundation models’ resource footprint without compromising performance, spanning cloud, edge, and devices.It focuses on resource-efficient algorithms and systems and their current state and future directions.Source: §1 · p. 2
  • The survey covers algorithm and system innovations and mainly physical resources, excluding hardware design, training data labels, and privacy; it emphasizes top-tier CS conference papers and selected high-impact arXiv work published mainly after 2020 [192] [185].The authors explain that AI innovation moves quickly and older methods can be overturned.Source: §1 · pp. 2–3

2 FOUNDATION MODEL OVERVIEW · 2.1 Language Foundation Models

This section surveys language foundation models through their Transformer architecture, representative text and speech model families, and the computational and storage costs that shape their use. It highlights the costs of embeddings, feedforward networks, long-sequence attention, and speech-specific CNN blocks.

2.1.1 Model Architectures

  • The Transformer embeds tokenized inputs with positional information, applies attention and feedforward layers, then maps the final representation to predictions.Common tokenizers include wordpiece and byte-pair encoding; self-attention relates sequence positions, while multi-head attention attends across representation subspaces.Source: §2.1.1 · pp. 4–5
  • Transformer encoders process inputs in parallel, while autoregressive decoders generate tokens sequentially using self-attention and cross-attention to the input.The KV cache [76] stores intermediate attention states to avoid recomputation during decoding, trading compute for memory.Source: §2.1.1 · p. 5

2.1.2 Representative Models and Downstream Tasks

  • Encoder-only models such as BERT [88] use bidirectional masked-language-model pretraining for discriminative tasks; DistilBERT is 40% smaller and 60% faster while retaining 97% of BERT’s comprehension capabilities.RoBERTa [256] improves BERT through longer training, larger batches, and more data, while Sentence-BERT [332] produces comparable sentence embeddings using siamese and triplet networks.Source: §2.1.2 · p. 5
  • Encoder-decoder models support text generation and reconstruction: T5 frames diverse tasks as text-to-text, while BART [210] learns to reconstruct text corrupted during pretraining.T5 is self-supervised and pretrained on C4; BART uses an arbitrary text-noising function.Source: §2.1.2 · p. 5
  • Decoder-only GPT models demonstrate scaling and zero-shot learning, while instruction tuning improves instruction following and informs open models such as LLaMA [383].GPT-1 has 117 million parameters and GPT-3 has 175 billion; GLM [95] improves blank-filling pretraining, and PaLM [70] was trained on 6144 TPU v4 chips.Source: §2.1.2 · p. 5

2.1.3 Cost Analysis

  • Approximately 25% of total storage costs comes from embeddings, while FFNs dominate computation and embeddings incur minimal inference compute.The lm head varies by task: a softmax classification layer for discriminative models such as BERT, or a linear layer for generative models such as GPT/T5; its size scales with vocabulary size.Source: §2.1.3 · p. 6, Figure 4 · p. 7
  • Attention costs O(T^2D), FFNs O(TD^2), and longer sequences sharply increase computation; decoder KV caching reuses keys and values but requires additional memory.For batch size B, sequence length S, hidden dimension D, and L decoder layers, single-precision KV-cache storage is (B × S × D × L × 2 × 4) Bytes.Source: §2.1.3 · p. 7, Figure 5 · p. 7
  • Speech models learn representations from audio for recognition and understanding; wav2vec 2.0 [28] has 19× more parameters than its CNN block but only 1.8× its computational load.These models commonly combine CNN feature extraction with Transformer context modeling; HuBERT trains self-supervised on 960-hour LibriSpeech, and Whisper uses 680,000 hours of multilingual, multitask supervised web data.Source: §2.1.2 · p. 6, §2.1.3 · p. 7

2.2 Vision Foundation Models

Vision foundation models use transformer architectures to process images and support varied downstream tasks, while their resource costs reflect fixed-length inputs and computation concentrated in attention and feed-forward layers. The section surveys representative encoder-only and encoder-decoder models and their training and task adaptations.

2.2.1 Model Architecture

  • ViT splits images into patches, adds a [CLS] token and positional embeddings, and sends the sequence through a Transformer encoder before task-specific heads.For a 3×224×224 image, the passage gives 14×14 patches of 16×16 pixels and says embedding overhead is less than 5% of encoder computation.Source: §2.2.1 · pp. 7–8

2.2.2 Representative Models and Downstream Tasks

  • ViT [92] demonstrated ImageNet Transformer-encoder pretraining but required more training data than traditional CNNs; DeiT uses distillation to improve data efficiency.Source: §2.2.2 · p. 8
  • BEiT learns to recover visual tokens from corrupted patches, while MAE uses a lightweight decoder and masks up to 80% of patches for cost-effective encoder training.Source: §2.2.2 · p. 8
  • YOLOS [102] transfers pretrained ViT to COCO detection, ViTDet adapts plain ViT for detection without a hierarchical backbone, and Swin Transformer uses shifted-window attention.Source: §2.2.2 · p. 8
  • DINOv2 trains a 1-billion-parameter ViT and distills smaller models that outperform OpenCLIP on most image- and pixel-level benchmarks.Source: §2.2.2 · p. 8
  • DETR combines a CNN backbone with an encoder-decoder Transformer for detection and segmentation, with about 40M parameters, aligning in size with Faster-RCNN.The supported tasks include object detection, instance segmentation, and panoptic segmentation.Source: §2.2.2 · p. 9
  • SegFormer combines Transformers with lightweight MLP decoders for semantic segmentation, while LVM learns visual information through image-sequence modeling without linguistic data.Source: §2.2.2 · p. 9

2.2.3 Cost Analysis

  • For fixed-length images represented by 14×14 or 16×16 patches, ViT’s computational bottleneck lies in the FFN fully connected layers and attention.The passage relates ViT’s resource consumption to BERT’s architectural alignment while distinguishing visual models’ fixed-length inputs.Source: §2.2.3 · p. 9

2.3 Multimodal Foundation Models

Multimodal foundation models either align different modalities in a shared representation space or generate across modalities, with the approaches converging toward multimodal-to-multimodal and any-to-any systems. The section surveys their architectures, representative model families, and resource costs.

2.3.1 Key Architectures

  • Multimodal architectures align modality-specific encoder embeddings in a joint semantic space, then use LLM decoders for text or diffusion models for image generation.LLMs generate autoregressively from text-aligned inputs; diffusion models generate images by progressively denoising them.Source: §2.3.1 · p. 9
  • Diffusion systems use VAEs to compress and reconstruct images and U-Nets to predict noise, while fusion decoders jointly process image and text representations for task-specific outputs.VQ-VAE uses vector quantization to encode and decode images; DDPM and DDIM are sampling algorithms for denoising.Source: §2.3.1 · pp. 9–10

2.3.2 Representative Models and Downstream Tasks

  • CLIP, ALBEF, and ALIGN established cross-modal alignment, while ImageBind and LanguageBind [492] extend it through intermediate modalities.Alignment performance remains constrained by encoder capabilities and the quantity and quality of image-text pair data; UNITER [64], FLAVA, CoCa [459], and GLIP explore fusion decoders.Source: §2.3.2 · p. 11
  • Encoder-large models such as BLIP-2 [215] and MiniGPT-4 [494] use image encoders and Q-Former fusion for cross-modal generation; Stable diffusion supports text-to-image generation, inpainting, editing, and super-resolution, while DALL-E generates images from textual descriptions.Source: §2.3.2 · p. 11
  • Any-to-any systems broaden input-output modalities: CoDi [372] can generate multiple modalities concurrently, and NExT-GPT [419] enables cost-effective modality expansion through minimal-parameter fine-tuning.M4 unifies text, image, audio, and motion tasks with multimodal embeddings and a transformer backbone for mobile AI.Source: §2.3.2 · p. 11

2.3.3 Cost Analysis

  • The multi-encoder module averages 0.27B parameters, 1.1G memory, and 65.9 GFLOPs per sample; its image encoder uses 0.63B parameters, 2.4G memory, and 167.59 GFLOPs per sample.The image encoder is the most resource-intensive encoder component.Source: Figure 6 · p. 12, §2.3.3 · p. 12
  • The integrated Vicuna-7B large-FM module uses 7B parameters, 14G memory, and 312 GFLOPs per sample, substantially exceeding the resource requirements of the encoder module.The module receives modality-aligned inputs for autoregressive generation.Source: Figure 6 · p. 12, §2.3.3 · p. 12
  • Stable Diffusion 2.1’s U-Net uses 865M parameters and 759G FLOPs per sample; its VAE uses 83M parameters and 4T FLOPs per sample; CLIP uses 289M parameters and 289M FLOPs per token.The U-Net processes 4×96×96 image latents; the VAE encodes 3×768×768 images to 4×96×96; CLIP encodes text tokens.Source: Figure 7 · p. 12, §2.3.3 · p. 13

3 RESOURCE-EFFICIENT ARCHITECTURES

The section surveys architectural strategies for reducing the computational and memory costs of large foundation models, covering efficient attention, dynamic networks, diffusion optimizations, and ViT-specific designs.

3.1.1 Sparse Attention

  • Sparse attention reduces the number of attention inner products while seeking to retain the empirical benefits of full self-attention.Longformer, ETC, and BIGBIRD combine local-window and task-specific global attention to reduce self-attention complexity to linear; HEPOS uses head-wise positional strides, while ALBERT shares parameters across layers and reports an 89% parameter-count reduction while guaranteeing accuracy relative to traditional BERT.Source: §3.1.1 · pp. 13–14

3.1.2 Approximate Attention

  • Approximate attention avoids directly computing the N × N attention matrix through low-rank approximations or reformulations of self-attention.Linformer projects key and value length dimensions to a lower-dimensional space; Reformer uses locality-sensitive hashing, while kernel-based methods exploit associative matrix multiplication. Other variants use polynomial sketching, gated attention with exponential moving averages, data-aware deformable attention, or linear cross-attention for variable-sized ViT tokens.Source: §3.1.2 · p. 14

3.1.3 Attention-Free Approaches

  • Attention-free architectures replace attention with alternatives such as long convolutions, state space models, recurrent networks, or retention mechanisms to improve sequence-length efficiency.Hyena offers a subquadratic alternative, Mamba combines selective SSMs with a streamlined network and reports 5× speed over traditional transformers with linear sequence-length scaling, and RetNet reports constant inference cost, 25–50% memory savings, and 7× training acceleration versus standard Transformers.Source: §3.1.3 · pp. 14–15

3.2.1 Mixture-of-Experts

  • Sparse Mixture-of-Experts routes only selected parameters, enabling large model capacity with reduced computation or training cost.Switch Transformer scales to one trillion parameters and 2,048 experts; V-MoE scales to 15 billion parameters while matching dense-model performance with less training time. Sparse upcycling reports better benchmark performance using about 50% of dense pretraining costs, while FFF is up to 220× faster than a feed-forward layer with about 5% accuracy loss.Source: §3.2.1 · p. 15

3.2.2 Early Exiting

  • Early-exit methods reduce inference work by stopping computation when predictions are confident or stable, with LGViT reporting an approximate 1.8× speed increase.Approaches include confidence thresholds, per-token exits for batched decoding, and internal classifiers that stop when predictions stabilize.Source: §3.2.2 · pp. 15–16

3.3.1 Efficient Sampling; 3.3.2 Diffusion in Latent Space

  • Diffusion efficiency work reduces generation cost through faster sampling and by moving diffusion from pixel space into a latent space.DDIM, PNDM, and DPM-Solver accelerate sampling; DPM-Solver uses as few as 10–20 denoising steps. Latent diffusion models such as LDM reduce the computational and memory burden of high-resolution generation, while also supporting text-guided synthesis.Source: §3.3.1 · p. 16, §3.3.2 · p. 17

3.3.3 Diffusion Architecture Variants

  • Diffusion architecture variants optimize model structure and inference, with SnapFusion generating high-quality 512×512 images in under 2 seconds on mobile devices.SnapFusion also reports better FID and CLIP scores than Stable Diffusion using fewer denoising steps; ScaleCrafter targets ultra-high-resolution generation without additional training or optimization, and ERNIE-ViLG reports a zero-shot FID-30k score of 6.75 on MS-COCO.Source: §3.3.3 · p. 17

3.4 ViT-specific Optimizations

  • Up to 6.4× speedup over Restormer in super-resolution is reported for EfficientViT, alongside lightweight ViT designs targeting efficient on-device inference.LeViT reports 80% ImageNet top-1 accuracy and 5× the speed of EfficientNet on CPU; PoolFormer reports 82.1% ImageNet-1K top-1 accuracy. MobileViT reports 78.4% ImageNet-1K top-1 accuracy with about 6 million parameters, while EfficientFormer-L1 reports 79.2% top-1 accuracy and 1.6 ms latency on iPhone 12.Source: §3.4 · pp. 17–18

4 RESOURCE-EFFICIENT ALGORITHMS

This section surveys resource-efficient algorithms for large foundation models (FMs), whose huge parameter sets and autoregressive inference distinguish them from traditional DNNs.

4 RESOURCE-EFFICIENT ALGORITHMS

  • The algorithms are categorized by FM lifecycle stage: pre-training, fine-tuning, serving, and model compression.Figure 11 illustrates this categorization.Source: §4 · p. 18

4.1 Pre-training Algorithms

Pre-training large foundation models consumes substantial computation, motivating resource-efficient algorithms that reduce training data, optimize architecture search, reuse computation through progressive learning, and lower precision during training.

4.1.1 Training Data Reduction

  • Training-data reduction removes text and image-patch redundancy; deduplication reduces training steps without performance loss, while TRIPS reports 40% less computation than previous pre-training vision-language models.MixMAE replaces [MASK] symbols with visible patches from mixed images, COPA uses patch-text alignment, PatchDropout samples fewer patches, and TPS assigns pruned tokens to reserved tokens.Source: §4.1.1 · pp. 18, 20

4.1.2 Neural Architecture Search

  • Neural architecture search reduces search costs through zero-shot evaluation; ZICO avoids model training during search, while PASHA [39] dynamically allocates resources and ElasticViT [371] designs ViTs below 1 × 10^9 FLOPs.RankNAS reframes architecture ranking as binary classification, and PreNAS selects architectures with a zero-shot proxy before training them once.Source: §4.1.2 · p. 20

4.1.3 Progressive Learning

  • Progressive learning reuses computation as models grow: stackingBERT stacks attention layers, staged training transfers training state, and LiGO maps small-model parameters to initialize larger models.CompoundGrow grows depth, width, and input length; knowledge inheritance uses pre-trained language models as teachers to speed larger-model training.Source: §4.1.3 · p. 20

4.1.4 Mixed Precision Training

  • Half-precision training approximately halves storage for weights, activations, and gradients; Mesa [303] compresses activations, while GACT adjusts compression by gradient importance to reduce compute and memory use.Mesa quantifies activations using multi-head self-attention layer distributions to minimize approximation error; GACT’s reductions apply across convolutional, graph, and transformer-based models.Source: §4.1.4 · pp. 20–21

4.2 Finetuning Algorithms

Efficient fine-tuning methods reduce the cost of adapting pretrained foundation models through additive tuning, selective tuning, and re-parameter tuning.

4.2.1 Additive Tuning

  • Additive tuning learns adapters, task-specific input prompts, or layerwise prefixes while limiting updates to reduce fine-tuning costs.Adapters keep the backbone frozen and can support multitask or multimodal extensions [98]; mixed soft prompts transfer knowledge across tasks [23] [208] [384], and prefixes target domain-specific generation [400] [247] [486] [287] [389].Source: §4.2.1 · pp. 21–22

4.2.2 Selective Tuning

  • Selective tuning freezes most parameters and updates a small subset; GreenTrainer reduces fine-tuning FLOPs by up to 64% versus full fine-tuning.SAM uses a second-order approximation to tune fewer parameters, SmartFRZ adaptively freezes layers, and FiSH-DiP uses sample feedback for sparse tuning; other approaches target visual-language and visual tasks.Source: §4.2.2 · p. 22

4.2.3 Re-parameter Tuning

  • Re-parameter tuning adapts a smaller subspace, commonly with low-rank updates, to reduce training cost and memory while retaining model performance.LoRA freezes pretrained weights and trains low-rank matrices; its improvements address added parameters, initialization, and learning procedure, while related methods use quantization, gradient-memory reduction, or sparse updates. [497] adds weight updates, [278] uses SVD initialization, and [254, 137] modify the learning process.Source: §4.2.3 · pp. 22–23

4.3 Inference Algorithms

Section 4.3 surveys inference-efficiency methods that reduce serial decoding, input processing, KV-cache costs, and the computation required for long contexts.

4.3.1 Opportunistic Decoding

  • Speculative and lookup-based decoding replace serial generation with parallel draft verification or corpus reuse, preserving exact outputs in reported settings.Speculative decoding verifies tokens from a small model using a larger model, while Draft & Verify, Medusa, and Look-ahead decoding avoid relying on a separate draft model in different ways. Prompt Cache reuses stored attention states, and LLMA verifies potential outputs against external text; Skeleton-of-Thoughts enables parallel filling of answer outlines.Source: §4.3.1 · pp. 23–24

4.3.2 Input Filtering and Compression

  • Up to 20× prompt compression with minimal performance loss and inference-time token pruning reduce input-side work across language, vision, and multimodal models.Methods include learned or adaptive token selection, pruning redundant vision tokens, and condensing high-resolution image inputs into smaller token representations.Source: §4.3.2 · p. 24

4.3.3 Key-Value Cache

  • KV-cache efficiency is addressed through quantization, sparse eviction, block management, and cache compression or reuse.H2O reports up to 1.9× lower latency and 29× higher throughput; Scissorhands reports up to 5× lower KV-cache memory use while maintaining quality. vAttention improves end-to-end serving throughput by up to 1.29× compared with vLLM.Source: §4.3.3 · pp. 24–25

4.3.4 Long Context

  • Long-context efficiency combines recurrent memory with attention designs that reduce complexity or extend usable sequence length.Transformer-XL reports evaluation speeds up to 1,800+ times faster than vanilla Transformers; LM-Infinite uses O(n) time and space for sequences up to 128k tokens, while LongNet supports sequences up to 1B tokens.Source: §4.3.4 · pp. 25–26

4.4 Model Compression

Model compression reduces model size while aiming to preserve performance, and the survey covers pruning, knowledge distillation, quantization, and low-rank decomposition.

4.4.1 Pruning

  • Pruning removes redundant connections, neurons, or layers: structured pruning removes components, unstructured pruning removes weights, and contextual pruning dynamically selects sparse computation during inference.Structured and unstructured pruning target weight reduction without changing inference sparsity, whereas contextual pruning selects activated neurons or layers based on model sparsity.Source: §4.4.1 · pp. 26–27
  • Movement pruning during fine-tuning removed entire components such as attention heads, yielding a BERT model 2.4× faster and 74% smaller than the original; PowerInfer achieves up to 11× acceleration versus llama.cpp.Source: §4.4.1 · p. 27

4.4.2 Knowledge Distillation

  • Knowledge distillation transfers a heavy teacher’s capabilities to a smaller student through API-generated prompt-response pairs (black-box) or access to teacher internals (white-box); DIME-FM transfers vision-model knowledge using inexpensive unpaired images and sentences [365].DIME-FM supports knowledge transfer when paired data may be limited or unavailable.Source: §4.4.2 · pp. 28–29

4.4.3 Quantization

  • QAT adapts model parameters to lower precision to mitigate accuracy loss, but its training cost remains substantial; large-model QAT is often paired with distillation or quantization is used during fine-tuning.LLM-QAT can produce a 4-bit LLM, and LLM-QAT and QuantGPT use distillation to reduce QAT training cost.Source: §4.4.3 · pp. 29–30
  • K-quant inference is approximately 3-4× faster than original FP16; ATOM 4-bit weight-activation quantization improves end-to-end throughput by up to 7.73× vs FP16 and 2.53× vs INT8 at the same latency target.K-quant is used by llama.cpp, and ATOM is described in.Source: §4.4.3 · pp. 30–31

4.4.4 Low-Rank Decomposition

  • Low-rank decomposition factors weight matrices into smaller matrices; TensorGPT [438] compresses the embedding layer by up to 38.40×.LoRD reduces parameters and computational overhead, with reported compression potential and minimal performance impact [180].Source: §4.4.4 · p. 31

5 RESOURCE-EFFICIENT SYSTEMS · 5.1 Distributed Training

The survey frames resource-efficient systems around distributed training, federated learning, and cloud and edge serving. It organizes distributed-training methods across resilience, parallelism, communication, storage, heterogeneous GPUs, and MoE.

5 RESOURCE-EFFICIENT SYSTEMS

  • Resource-efficient systems research spans distributed training, federated learning, and cloud and edge serving; Figure 16 outlines its taxonomy and Table 5 summarizes open-source frameworks.The survey notes that most systems and techniques in this section are designed specifically for LLMs, given their success and large parameter sizes.Source: §5 · p. 32, Figure 16 · p. 32, Table 5 · p. 33

5.1 Distributed Training

  • Distributed training underpins large-FM pretraining and fine-tuning, and this subsection organizes efficiency techniques across six system challenges.Pretraining is described as especially demanding in computation, communication, and resources.Source: §5.1 · p. 32
  • Resilience methods use restart checkpoints (Varuna and Gemini [25]), redundant computation (Bamboo), activation recomputation, or partial-layer recovery (Oobleck).These approaches trade checkpoint or recovery overhead against extra computation; Oobleck restores an affected pipeline using partial layers from other replicas.Source: §5.1 · pp. 32–33
  • Distributed training combines data, model, and sequence parallelism; ZeRO splits model states, while Galvatron selects hybrid strategies automatically.Model parallelism includes tensor parallelism and pipeline approaches; sequence parallelism divides long sequences across workers.Source: §5.1 · p. 33
  • Communication optimization either reduces communication time or hides it by overlapping communication with computation, using compression, traffic reduction, abstraction, collective decomposition, or pipelining.Examples include Optimux-CC, MiCS, CoCoNet, Wang et al., and Zhuang et al..Source: §5.1 · pp. 33–34
  • Storage methods ease GPU-memory limits by offloading to CPU or SSD, while Behemoth uses NAND flash to enable data-parallel training.ZeRO-Offload targets single-GPU training; FlashNeuron selectively offloads data to SSD for higher throughput.Source: §5.1 · p. 34
  • Heterogeneous-GPU training uses synchronization and hardware-aware load balancing to accelerate training across GPUs with different performance.Hetpipe uses Wave Synchronous Parallel; Whale introduces hardware-aware load balancing.Source: §5.1 · p. 34
  • MoE training systems optimize dynamism, parallelism, and communication through sparse computation, expert management, adaptive execution, and data-centric communication.Examples include MegaBlocks [115] for dynamic routing, FlexMoE for expert management, Tutel for adaptive parallelism and pipelining, and Janus for data-centric communication.Source: §5.1 · p. 34

5.2 Federated Learning

The survey presents federated learning as a privacy-preserving way to train foundation models across data sources and groups federated LLM systems into four approaches. These approaches address resource constraints through toolkits, parameter-efficient tuning, model decomposition, and backpropagation-free optimization.

5.2 Federated Learning

  • Federated LLM systems are grouped into frameworks and benchmarks, PEFT-based approaches, model decomposition, and zeroth-order optimization.The survey describes federated learning as a predominant way to train foundation models using data from multiple sources while preserving privacy.Source: §5.2 · p. 34

5.2.1 Frameworks & Benchmarks

  • Open-source FL toolkits provide APIs and efficient algorithms; examples include Flower’s Whisper-on-Raspberry-Pi support [37] and Federatedscope-LLM’s PEFT and acceleration package [198].FedML [139] introduces FedLLM compatible with platforms such as DeepSpeed, while FATE [100] targets FedLLM training efficiency with PEFT.Source: §5.2.1 · p. 34

5.2.2 PEFT-based Approaches

  • PEFT freezes FM backbones to reduce training and communication costs; FL variants use adapter search, bias-only prompts with pseudo-labeling [44], soft-label LoRA [170], and adaptive prompts for non-IID data [52].FedPrompt explores prompt learning’s potential for larger differential-privacy budgets; Malaviya et al. analyze PEFT across non-IID scenarios and client fractions, while Petals [40] enables collaborative tuning over the Internet.Source: §5.2.2 · pp. 34–35

5.2.3 Model Decomposition

  • Decomposition supports practical FL through partial client-side layer training, selective quantized-block uploads [65], and split learning that assigns Transformer updates to the server.FedPerfix uses plugins for ViT personalization based on layer sensitivity, while SplitLoRA [245] provides a benchmark for split-learning LLM PEFT.Source: §5.2.3 · p. 35

5.2.4 Backprop-free Approaches

  • Zeroth-order FL avoids costly backpropagation on edge devices: FwdLLM uses PEFT-restricted forward gradients, while FedKSeed [320] reconstructs gradients locally from accumulated scalars; Fed-BBPT [246] tunes instructions black-box.FwdLLM supports billion-parameter LLM training on standard mobile devices; BBTv2 offers a gradient-free divide-and-conquer method in a lower-dimensional subspace.Source: §5.2.4 · p. 35

5.3 Serving on Cloud

Because large models face heavy request loads in real-world applications, serving systems aim to improve efficiency without sacrificing accuracy. LLM serving generates tokens autoregressively and is organized around prefill and decoding, motivating computation, memory, and deployment-platform optimizations.

5.3 Serving on Cloud

  • LightSeq [398] caches attention key-value pairs, separating serving into prefill, which processes input tokens, and decoding, which processes new tokens and updates the cache.This reduces redundant computation across autoregressive token-generation iterations.Source: §5.3 · p. 35

5.3.1 Inference Accelerating

  • Flash-Decoding targets decoding with a CUDA kernel, while FlashDecoding++ [146] optimizes softmax and flat GEMM and adds AMD support; serving systems also use parallelism and request scheduling.Orca uses selective batching and iteration-level scheduling to batch requests of different lengths, return completed requests without waiting, and run newly arrived requests between iterations.Source: §5.3.1 · p. 36

5.3.2 Memory Saving

  • DeepSpeed-Inference and FlexGen offload activations or model parameters to DRAM or NVMe when GPU memory is insufficient; vLLM allocates KV-cache memory on demand.vLLM uses Paged Attention with this allocation mechanism; S-LoRA [350] extends paging to multiple LoRA adapters, and SGLang [490] uses RadixAttention to support more complex KV-cache management across requests.Source: §5.3.2 · p. 36

5.3.3 Emerging Platforms

  • SpotServe [281] dynamically adjusts parallelism to accommodate preemption on low-cost, unreliable spot instances, while HexGen [172] searches for high-performance model placement and parallelism on heterogeneous GPUs.These systems address deployment platforms beyond data centers with homogeneous high-performance servers.Source: §5.3.3 · p. 36

5.4 Serving on Edge

Privacy concerns and strict response-latency requirements are driving interest in on-device foundation-model inference. The section surveys edge-cloud collaboration and edge-only methods that address mobile devices’ compute, memory, and I/O constraints.

Edge-cloud collaboration

  • EdgeFM [444] adapts large FMs to customized edge models, aiming for low latency while retaining accuracy close to the original FMs.Source: §5.4 · p. 37

On-device MoE models

  • On-device MoE methods reduce computation or loading costs through sparse execution, expert bit-width adaptation and preloading [454], or temporal-locality-based expert caching [194].PC-MoE [194] uses a Parameter Committee to maintain a subset of crucial experts.Source: §5.4 · p. 37

Memory optimization

  • Memory-focused systems use draft-model speculative decoding, hot/cold neuron placement across GPU and CPU, or clustered neuron caching for smartphones [442].These designs target generation speed, GPU memory demand, and memory or I/O bottlenecks, respectively.Source: §5.4 · p. 37

I/O optimization

  • I/O methods adapt weight bit-width during loading or manage flash storage to transfer less data in larger, more contiguous chunks.STI adapts bit-width according to parameter importance to minimize loading overhead while preserving maximum inference accuracy.Source: §5.4 · p. 37

Kernel optimization

  • Edge kernels include a first 32-bit integer ViT implementation with post-training quantization, while mllm-NPU [430] accelerates LLM prefilling on mobile NPUs.The ViT kernel uses range-constrained quantization for activation and normalization operators; mllm-NPU addresses semantic gaps between LLM inference and NPU design.Source: §5.4 · p. 37

LLM as a system service

  • LLMaaS deploys one shared LLM in the mobile OS, avoiding per-app model copies while enabling system-level scheduling and accelerator use.Android AICore is an industry example; LMS provides app-level APIs and stateful invocation, while ELMS elasticizes the model to meet diverse app SLOs.Source: §5.4 · p. 37

6 CONCLUSIONS AND FUTURE DIRECTIONS

The survey synthesizes cost analysis and model, algorithm, and system designs for resource-efficient large foundation models. It argues that this research will become more crucial as scaling drives larger models and calls for interdisciplinary work.

Cloud-edge hybrid deployment

  • Cloud-edge deployment could bring foundation models closer to users for ubiquitous, privacy-preserving, highly available intelligence.Preliminary efforts have brought LLaMA-7B to smartphones and PCs; personal assistants/agents [235] [411] and multimodal information retrieval [211] are potential applications.Source: §6 · p. 38

Exploiting the model sparsity

  • Runtime activation sparsity, even in densely trained non-MoE models, could reduce inference time and memory footprint as models scale.The survey points to exploiting model and activation sparsity and developing efficient sparse architectures beyond MoE.Source: §6 · p. 38

Large foundation model as a service

  • Large foundation models may become a universal service across clouds and devices, creating opportunities for hardware-algorithm co-design and new system challenges [461].Those challenges include scheduling, load balancing, and security and isolation.Source: §6 · p. 38

Agent as a holistic system to optimize

  • Agent efficiency must be optimized for the workflow, including coordination among multiple foundation models with logic dependencies [235] [411].The design space includes selecting models for tasks and scheduling them on available hardware to maximize agent performance.Source: §6 · p. 38

Practical privacy-preserving FM

  • Practical privacy-preserving foundation models need techniques that balance privacy with performance, since existing methods hinder large-scale, in-the-wild deployment.Federated learning, homomorphic encryption, and disentanglement learning are theoretically sound but face significant performance challenges.Source: §6 · p. 38

Understanding the scaling law

  • Understanding scaling-law mechanisms and designing architectures with better or optimal scaling performance could help address the intelligence limitations of lightweight models.The survey notes that small-scale models seem unlikely to attain the intelligence of larger models.Source: §6 · p. 38

Related papers

Loading 2401.08092v2…