PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Resource-efficient LLM and Multimodal Foundation Models

Mengwei Xu, Wangsong Yin, Dongqi Cai, Rongjie Yi, Daliang Xu, Qipeng Wang, Bingyang Wu, Yihao Zhao, Chen Yang, Shihe Wang, Qiyang Zhang, Zhenyan Lu, Li Zhang, Shangguang Wang, Yuanchun Li, Yunxin Liu, Xin Jin, Xuanzhe Liu

arXiv:2401.08092v2cs.LGcs.AIcs.DC

TL;DR

Large foundation models deliver broad capabilities but require substantial computing, memory, energy, and bandwidth, motivating resource-efficient methods. This survey synthesizes algorithmic and system approaches across architectures, training, serving, and deployment, and concludes that efficiency research remains important as scaling continues. It also identifies scaling-law understanding as a fundamental limitation and future direction for lightweight models.

  • Problem

    Large foundation models impose substantial resource demands across training and deployment, creating a need for resource-efficient strategies.

  • Method

    The survey systematically reviews resource-efficient model architectures, algorithms, data practices, and systems spanning cloud, edge, and device settings.

  • Results

    The survey synthesizes approaches including sparse Mixture-of-Experts, latent-space diffusion, lightweight vision architectures, and serving systems for reducing resource demands across the foundation-model lifecycle.

  • Takeaways & Limitations

    Resource efficiency requires coordinated algorithmic and system design across the model lifecycle rather than isolated optimization of a standalone model service.

  • Takeaways & Limitations

    Scaling laws remain a fundamental limitation for lightweight models because small-scale models are unlikely to achieve higher intelligence levels than larger models.

Abstract

from arXiv · show

Large foundation models, including large language models (LLMs), vision transformers (ViTs), diffusion, and LLM-based multimodal models, are revolutionizing the entire machine learning lifecycle, from training to deployment. However, the substantial advancements in versatility and performance these models offer come at a significant cost in terms of hardware resources. To support the growth of these large models in a scalable and environmentally sustainable way, there has been a considerable focus on developing resource-efficient strategies. This survey delves into the critical importance of such research, examining both algorithmic and systemic aspects. It offers a comprehensive analysis and valuable insights gleaned from existing literature, encompassing a broad array of topics from cutting-edge model architectures and training/serving algorithms to practical system designs and implementations. The goal of this survey is to provide an overarching understanding of how current approaches are tackling the resource challenges posed by large foundation models and to potentially inspire future breakthroughs in this field.

1 INTRODUCTION

Foundation models offer scalable, versatile zero-shot capabilities, but their expanding resource demands create environmental, access, and privacy challenges. This survey reviews algorithmic and system innovations for improving efficiency across cloud, edge, and device settings.

  • Foundation models generalize across open-world tasks through scaling with data and parameters, while retaining simple underlying architectures.Scaling laws describe predictable performance improvements as model size and data volume increase.
  • 1.7× millions of GPU hours and 2.5 × 10^12 Joules are required to pre-train LLaMa-2-70B, with estimated emissions of 291 tons of CO2 equivalent.Data processing, experimentation, and inference can consume comparable or greater electricity.
  • The resource footprint limits democratization because only a few major players can train and deploy state-of-the-art foundation models, which are commonly cloud-served rather than device-resident.Cloud serving also makes data privacy preservation difficult.
  • The survey examines algorithmic efficiency, system optimization, data management, and resource-efficient architectures across cloud, edge, and device deployments.Its scope emphasizes algorithm and system innovations while excluding hardware design and limiting resources mainly to physical resources such as computing, memory, storage, and bandwidth.

2.1 Language Foundation Models

Language foundation models use Transformer-based pipelines whose attention, feed-forward, and decoding components impose distinct computational and memory costs. The section surveys representative architectures and analyzes how sequence length, model dimensions, and speech-specific encoders affect efficiency.

  • Model Architectures: Transformer language models tokenize inputs, add learned embeddings and positional information, then process them with attention, normalization, and position-wise transformations.Encoder-decoder models use encoders to represent inputs and decoders to generate output sequences.
  • Model Architectures: Auto-regressive decoding generates each new token from the preceding sequence, while KV caching avoids recomputing earlier attention states at the cost of additional memory.This trade-off is particularly relevant to inference efficiency.
  • Representative Models and Downstream Tasks: BERT, T5, BART, GPT, and speech foundation models represent encoder-only, encoder-decoder, decoder-only, and audio-oriented model families with distinct downstream uses.Speech models commonly combine CNN feature extraction with Transformers for contextual representation learning.
  • Cost Analysis: 25% of total storage is attributed to embeddings, while FFN layers are the most computation-intensive components in large foundation models.Embedding inference is inexpensive because embeddings operate as lookup tables.
  • Cost Analysis: Attention complexity is O(T^2D), whereas FFN complexity is O(TD^2), so longer sequences substantially increase computational demand through quadratic attention scaling.Here, T denotes sequence length and D the hidden-state dimension.
  • Cost Analysis: Despite having 19× more parameters, wav2vec 2.0 incurs only 1.8× more computational load than its CNN block.The CNN encoder’s initial layers can require more computation than an individual Transformer layer.

2.2 Vision Foundation Models

Vision foundation models center on Vision Transformers, which split images into patches and process them with Transformer encoders. Their resource consumption resembles BERT, with attention and feed-forward layers forming key computational bottlenecks.

  • Model Architecture: Vision Transformers split images into fixed-size patches, add positional embeddings and a classification token, then process the sequence with a Transformer encoder.For a 3×224×224 image, the example uses 14×14 patches of 16×16 pixels; embedding costs less than 5% of subsequent computation.
  • Model Architecture: Most visual foundation models use encoder-only architectures, while encoder-decoder designs support tasks such as detection and segmentation.ViT and related encoder-only models are contrasted with DETR and SegFormer-style encoder-decoder approaches.
  • Representative Models and Downstream Tasks: Representative vision foundation models span classification, detection, segmentation, and self-supervised reconstruction tasks.The surveyed model families include ViT, DeiT, BEiT, MAE, YOLOS, ViTDet, Swin Transformer, DETR, and SegFormer.
  • Cost Analysis: ViT resource consumption resembles BERT, with fully connected layers in the FFN and attention forming the computational bottlenecks for standard image inputs.The discussion focuses on fixed-length visual inputs such as 14×14 or 16×16 patches.

2.3 Multimodal Foundation Models

Multimodal foundation models align multiple modalities through encoders and then generate outputs with language, diffusion, or fusion decoders. Their costs vary substantially across modules, with large language-model decoders and diffusion components requiring major resources.

  • Key Architectures: Multimodal foundation models follow two broad research lines: mapping modalities into shared latent spaces or generating different modalities with transformer decoders.These lines increasingly converge in multimodal-to-multimodal and any-to-any generation.
  • Key Architectures: Multimodal architectures use modality-specific encoders to produce normalized, fixed-dimensional embeddings that are aligned before generation.The encoders are generally trained from scratch using paired data involving the relevant modalities.
  • Key Architectures: After alignment, models commonly reuse text-trained LLMs for text generation or diffusion models for high-quality image generation.Diffusion modules include an image encoder/decoder and a denoising network; fusion decoders jointly process image and text representations.
  • Representative Models and Downstream Tasks: Multimodal model families include multi-encoder, encoder-large, encoder-diffusion, encoder-fusion-decoder, and any-to-any systems.Examples include CLIP, BLIP-2, MiniGPT-4, Stable Diffusion, UNITER, and CoDi.
  • Cost Analysis: 0.27B parameters, 1.1G memory, and 65.9 GFLOPs characterize the average multi-encoder module, while its image encoder reaches 0.63B parameters, 2.4G memory, and 167.59 GFLOPs per sample.The image encoder is identified as the most resource-intensive component within the multi-encoder module.
  • Cost Analysis: 7B parameters, 14G memory, and 312 GFLOPs per sample characterize the integrated Vicuna-7B large-FM module, substantially exceeding the encoder module's resource requirements.The module receives aligned multimodal inputs and performs autoregressive generation.
  • Cost Analysis: Stable Diffusion 2.1 uses an 865M-parameter U-Net with 759G FLOPs, an 83M-parameter VAE with 4T FLOPs, and a 289M-parameter CLIP with 289M FLOPs per token.The U-Net predicts latent-space noise, the VAE maps images to and from latent space, and CLIP encodes text.

3 RESOURCE-EFFICIENT ARCHITECTURES

Resource-efficient architectures reduce the computational and memory costs of large foundation models through efficient attention, dynamic networks, diffusion-specific methods, and ViT-specific designs. The surveyed approaches include sparse or approximate attention, alternative sequence architectures, sparse activation, early exiting, latent-space diffusion, and lightweight ViT variants.

  • Efficient Attention: Efficient attention methods address the quadratic sequence-length complexity of standard attention by reducing computation or replacing the N × N attention matrix.The survey highlights sparse attention, approximate attention, and attention alternatives as major approaches.
  • Efficient Attention: Sparse attention retains selected interactions, with Longformer, ETC, and BIGBIRD reducing self-attention complexity to linear through local and task-specific global attention.Other sparse designs include head-specific subsets, multi-view attention, and selective focus mechanisms.
  • Attention Alternatives: Alternative architectures such as Hyena and state space models replace traditional attention with subquadratic or selective sequence-processing mechanisms.Hyena interleaves implicitly parameterized long convolutions with data-controlled gating, while SSMs provide another attention substitute.
  • Dynamic Architectures: Mixture-of-Experts uses routed sparse parameters to train and deploy models with large parameter sets while reducing computation and communication costs.Switch Transformer scales to up to one trillion parameters with 2,048 experts, while V-MoE matches dense-model performance with less training time.
  • Dynamic Architectures: Early exiting terminates computation when predictions are sufficiently confident or resources are constrained, and LGViT achieves approximately 1.8× higher speed in ViTs.LGViT uses diverse local-perception and global-aggregation exit heads to balance efficiency and accuracy.
  • Diffusion-specific Optimization: Latent-space diffusion reduces memory and computation for high-resolution generation by performing diffusion on VAE-derived latent representations rather than pixels.LDM combines latent diffusion with cross-attention to text-encoder inputs for detailed, text-guided visuals.
  • ViT-specific Optimizations: ViT-specific designs improve efficiency through hybrid convolutional architectures, pyramid backbones, pooling, linear attention, and hardware-aware optimization.LeViT reports 80% ImageNet top-1 accuracy and 5× higher CPU speed than EfficientNet; EfficientFormer-L1 reports 1.6 ms iPhone 12 latency, while EfficientViT reaches up to 6.4× speedup over Restormer.

4 RESOURCE-EFFICIENT ALGORITHMS

Resource-efficient algorithms for large foundation models are organized around the model lifecycle: pre-training, fine-tuning, serving, and model compression.

  • Algorithmic Organization: The survey categorizes resource-efficient algorithms by the foundation-model lifecycle into pre-training, fine-tuning, serving, and model compression.This lifecycle framing reflects the large parameter sets and autoregressive inference characteristics of large foundation models.

4.1 Pre-training Algorithms

Pre-training efficiency methods reduce the costs of computation and large-scale data through data deduction, neural architecture search, progressive learning, and mixed precision training. The surveyed techniques remove redundant inputs, search architectures more cheaply, reuse smaller-model computation, or reduce numerical storage requirements.

  • Overview: Pre-training large foundation models requires substantial computation, motivating data deduction, neural architecture search, progressive learning, and mixed precision training.GPT-3-175B consumes 3.14 × 10^23 FLOPs, while LLaMA-70B takes 1.7 × 10^6 GPU hours.
  • Training Data Deduction: Training-data reduction targets trillion-scale datasets through text deduplication and image patch removal.GPT-3-175B uses 0.3 trillion tokens and LLaMA-2-70B uses 2 trillion tokens, making data reduction a major efficiency target.
  • Training Data Deduction: Text deduplication removes near-duplicate examples and repetitive substrings, enabling fewer training steps without compromising performance.The method addresses redundancy within training data rather than changing the model architecture.
  • Training Data Deduction: Image patch removal reduces visual inputs through patch selection or token reorganization, with TRIPS reporting a 40% computation-resource reduction over previous pre-training vision-language models.TRIPS computes attentive image tokens using text guidance.
  • Neural Architecture Search: Neural Architecture Search automatically explores model designs for efficiency and performance, with zero-shot proxies avoiding model training during the search phase.PASHA dynamically increases resource allocation as needed, while RankNAS reformulates architecture search as ranking and binary classification.
  • Progressive Learning: Progressive learning grows a model from a smaller starting point and reuses earlier computations, reducing resource consumption compared with training the large model from scratch.StackingBERT progressively adds attention layers, while CompoundGrow grows depth, width, and input length through compound operators.
  • Mixed Precision Training: Mixed precision training uses half-precision representations to approximately halve storage for weights, activations, and gradients.Mesa combines activation-compressed training with mixed precision to further reduce activation memory.

4.2 Finetuning Algorithms

Efficient fine-tuning adapts pre-trained foundation models with lower workload by adding parameters, selectively updating existing parameters, or tuning a smaller re-parameterized subspace. These approaches generally preserve most of the backbone while restricting the trainable component.

  • Overview: Efficient fine-tuning is organized into additive tuning, selective tuning, and re-parameter tuning.The categories differ in whether they add task-specific modules, update selected parameters, or optimize a smaller parameter subspace.
  • Additive Tuning: Additive tuning freezes the pre-trained model while introducing task-specific parameters through adapters, prompts, or prefixes.These methods reduce adaptation costs by limiting updates to newly introduced components.
  • Additive Tuning: Adapter tuning inserts trainable adapter modules into selected layers while the pre-trained backbone remains frozen.Adapters are used for task-specific knowledge and extensions such as multitask or multimodal capabilities.
  • Additive Tuning: Prompt tuning replaces parameter fine-tuning with task-specific input prompts, reducing the resources and time required for adaptation.Mixed soft prompts can transfer knowledge across tasks while reusing the frozen pre-trained model.
  • Additive Tuning: Prefix tuning adds a trainable task-specific prefix at each layer and limits updates to that prefix.The method targets lower tuning cost while supporting domain-specific improvements.
  • Selective Tuning: Selective tuning freezes most parameters and updates only a small portion, with SAM using a second-order approximation to tune fewer parameters.The approach aims to maintain performance on new tasks while lowering training costs.
  • Re-parameter Tuning: Re-parameter tuning reduces training cost by optimizing low-rank matrix parameters within a substantially smaller subspace than the original training space.LoRA freezes the pre-trained weight matrix and trains low-rank matrices A and B, with ranks as low as 2, but retains a performance gap relative to full fine-tuning.

4.3 Inference Algorithms

Inference algorithms address autoregressive decoding and long-context costs through speculative, lookup-based, prompt/input compression, token pruning, KV-cache optimization, and attention or recurrent alternatives.

  • Decoding acceleration: Speculative decoding uses a small model to draft tokens and a large model to verify them in parallel, reducing autoregressive decoding work.Look-ahead decoding avoids a draft model, while SpecInfer and LLMCad build systems around speculative decoding.
  • Input reduction: Prompt compression can reduce computation by compressing inputs, with LLMLingua achieving up to 20× compression with minimal performance loss.Token pruning instead removes less important input tokens during inference through learned or searched pruning configurations.
  • KV-cache efficiency: KV-cache optimization targets the memory overhead of autoregressive inference through quantization, stricter sparse attention, block-wise management, and cache reuse or compression.vLLM manages cache blocks with runtime reallocation while preserving standard-attention computation results.
  • Long-context inference: Long-context methods address quadratic attention costs using recurrent structures and efficient attention designs, including LM-Infinite’s O(n) time and space complexity.Transformer-XL reports evaluation speeds up to 1,800+ times faster than vanilla Transformers, while LM-Infinite supports sequences up to 128k tokens in the cited evaluations.

4.4 Model Compression

Model compression reduces large-model size and resource use through pruning, knowledge distillation, quantization, and low-rank decomposition, while related methods target inference speed and training cost.

  • Compression taxonomy: Model compression comprises pruning, knowledge distillation, quantization, and Low-Rank decomposition, aiming to reduce model size without significant performance degradation.Pruning removes redundant connections, neurons, or layers to reduce computational and storage costs while maintaining accuracy.
  • Pruning: Structured pruning removes model components such as channels or blocks, whereas unstructured pruning removes low-weight neurons and contextual pruning dynamically selects sparse computation during inference.Structured pruning is often combined with fine-tuning, and LLM-Pruner uses gradient information with a small amount of data to remove non-essential structures.
  • Pruning: 2.4× speedup and 74% smaller than the original BERT are reported for a movement-pruned model after removing components such as attention heads.The cited result follows pruning during fine-tuning within the movement pruning framework.
  • Knowledge distillation: Knowledge distillation transfers knowledge from a complex teacher to a simpler student through white-box or black-box procedures, including prompt-response training when the teacher is inaccessible.Black-box methods use API-generated prompt-response pairs, while GKD trains on self-generated sequences with teacher feedback.
  • Quantization: Quantization-aware training remains costly for models with tens or hundreds of billions of parameters, so it is combined with distillation or used during fine-tuning.Examples include LLM-QAT, QuantGPT, PEQA, LoftQ, and QLoRA.
  • Quantization: Quantization compresses model representations and can also accelerate inference; llama.cpp’s K-quant is approximately 3-4× faster than the original FP16 model.K-quant uses blocks of size 16 × 8 and an additional FP16 secondary quantization parameter.

5 RESOURCE-EFFICIENT SYSTEMS

Resource-efficient systems for large foundation models span distributed training, federated learning, cloud serving, and edge serving, with techniques addressing resilience, parallelism, communication, storage, heterogeneous hardware, and MoE execution.

  • System scope: The survey organizes resource-efficient system research into distributed training, federated learning, cloud serving, and edge serving.The taxonomy is accompanied by a summary of widely used open-source frameworks and tools.
  • Distributed training: Distributed training systems support pretraining and fine-tuning, with pretraining demanding substantial computation and communication resources.The section emphasizes resource-efficient designs for the scale and execution patterns of large foundation models.
  • Distributed training: Resilience methods recover from failures through checkpoints, redundant computation, and partial-layer recovery, with Oobleck restoring affected pipelines from replicas.Partial-layer recovery incurs less overhead than using an entire checkpoint.
  • Distributed training: Parallelism partitions data or model computation, while communication methods reduce or hide overhead by compressing traffic, exploiting heterogeneity, or overlapping communication with computation.Model parallelism includes tensor and pipeline parallelism, and communication schedules can decompose collectives or coordinate pipeline execution.
  • Distributed training: Storage optimization offloads data and computation from GPU memory to CPU memory or SSD, while heterogeneous-GPU methods balance work across hardware with different performance levels.ZeRO-Offload trains large models on a single GPU by offloading to the CPU; Hetpipe and Whale target heterogeneous GPUs.
  • Distributed training: MoE training systems optimize dynamic routing, load imbalance, parallelism, and communication to improve efficiency and scalability.MegaBlocks uses sparse primitives for dynamic routing and load-imbalanced computation.

5.2 Federated Learning

Federated learning supports foundation-model training across multiple data sources while preserving data privacy, and resource-efficient methods reduce client computation, communication, and model-update costs.

  • Federated learning motivation: Federated learning trains foundation models using data from multiple sources while ensuring data privacy.Recent work develops efficient federated-learning systems tailored to foundation models.
  • Frameworks: Open-source federated-learning frameworks provide APIs and benchmarks for efficient foundation-model training, including federated fine-tuning of Whisper on Raspberry Pi 5.Flower and FedML are examples of frameworks integrated with foundation models.
  • Efficient adaptation: Parameter-efficient fine-tuning freezes the foundation-model backbone and updates a small set of additional parameters to reduce training time and communication costs.FedAdapter, FP-FL, FedPrompt, and FedPepTAO explore adapter, LoRA, prompt, and partial-prompt tuning strategies.
  • Model decomposition: Model decomposition reduces federated-learning costs by training selected layers or uploading quantized semantic blocks for server aggregation.FedBFPT gradually increases the client-side layer count, while FedOBD decomposes models into semantic blocks.
  • Optimization: Zeroth-order optimization avoids explicit back-propagation gradients, addressing the computational and memory burden of training large models on edge devices.The cited motivation identifies back-propagation as challenging for federated learning because of edge-device computation and memory overhead.

5.3 Serving on Cloud

FM serving systems address the efficiency challenges of autoregressive generation through computation, memory, batching, parallelism, and deployment-platform optimizations. These approaches target high request volumes while preserving model accuracy.

  • Serving-system motivation: Serving systems optimize autoregressive token generation, where each iteration processes all preceding input and output tokens.This pattern creates redundant computation across iterations and motivates specialized serving techniques.
  • Computation optimizations: KV caching avoids recomputing earlier attention states by separating serving into prefill and decoding phases.The cache reduces computation but introduces additional memory requirements.
  • Optimization dimensions: Serving optimizations span computation, memory management, and deployment on emerging platforms.Examples include specialized kernels, offloading, request scheduling, and adaptation to unreliable or heterogeneous infrastructure.
  • Computation optimizations: Large-scale inference combines data, pipeline, tensor, and expert parallelism across multiple GPUs or TPUs.These strategies distribute inference workloads across modern accelerators.
  • Computation optimizations: Request batching and scheduling reduce wasted computation caused by different input and output lengths.Padding requests to a common length can make the system process unnecessary padded tokens.
  • Memory optimizations: Memory techniques offload parameters or activations and allocate KV-cache storage on demand.DeepSpeed-Inference and FlexGen use DRAM or NVMe offloading, while vLLM uses block-level allocation and Paged Attention.

5.4 Serving on Edge

Edge serving research responds to privacy, latency, and device-resource constraints by combining edge-cloud collaboration with direct on-device optimization. Techniques target computation, memory, parameter loading, and shared model services.

  • Motivation: On-device inference is motivated by growing data-privacy concerns and stringent response-latency requirements.Large models are therefore being adapted for local execution on mobile devices.
  • Edge-cloud collaboration: Edge-cloud collaboration adapts large foundation models into customized edge models that aim for low latency and accuracy close to the original.EdgeFM is an example of this strategy.
  • Edge-only optimization: On-device mixture-of-experts models reduce computation by executing only routed sparse parameters during inference.Expert-wise bit-width adaptation can reduce parameter size and loading time with acceptable accuracy loss.
  • Edge-only optimization: Memory optimization addresses the device memory wall through speculative decoding, token-tree verification, fallback strategies, and staged generation.LLMCad offloads most workloads to a smaller memory-resident draft model.
  • Edge-only optimization: I/O optimization adapts parameter bit-width during loading to reduce disk-to-memory overhead under an inference-accuracy constraint.STI targets cases where parameter-loading time exceeds computation time.
  • Edge-only optimization: LLM-as-a-Service places one shared foundation-model instance in the mobile operating system to avoid memory depletion by app-specific models.A unified model architecture and operator set supports this shared service paradigm.

6 CONCLUSIONS AND FUTURE DIRECTIONS

The survey organizes resource-efficient foundation-model research across model, algorithmic, and system designs, and identifies future directions spanning deployment, sparsity, services, agents, privacy, and scaling laws. These directions reflect the need to make increasingly large models more practical and sustainable.

  • Survey scope: The survey provides a holistic, systematic overview of literature on resource-efficient large foundation models.It covers background and cost analysis alongside model architecture, algorithm, and system designs across large, vision, and multimodal models.
  • Cloud-edge hybrid deployment: Cloud-edge hybrid deployment aims to bring foundation models closer to users for ubiquitous, privacy-preserving, and highly available intelligence.Future device capability in model size and speed is identified as a competitive concern for hardware vendors.
  • Exploiting model sparsity: Exploiting model and activation sparsity is proposed as a route toward sustainable scaling of model size.The survey notes that activated ratios may decrease with model size and that runtime sparsity can occur even in densely trained models.
  • Foundation model as a service: Treating large foundation models as services creates opportunities for hardware-algorithm co-design while introducing scheduling, load-balancing, and security-isolation challenges.The service abstraction is discussed for both cloud and device settings.
  • Agent as a holistic system: Agent-oriented efficiency must account for workflows in which multiple foundation models cooperate under inherent logical dependencies.The survey argues that standalone LLM-service efficiency is insufficient for such systems.
  • Practical privacy-preserving foundation models: Privacy-preserving foundation-model methods must balance privacy with performance to achieve large-scale deployment.Federated learning, homomorphic encryption, and disentanglement learning are identified as existing approaches with performance challenges.
  • Understanding the scaling law: Understanding scaling-law mechanisms may clarify the limitations of lightweight models and motivate architectures with better scaling performance.The survey presents scaling-law theory and architecture design as directions for future investigation.
Loading 2401.08092v2…