Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directions

A. Motivation

• Distributed approaches target computational scalability, resource constraints, and dataset privacy across training and deployment of language models.The survey emphasizes distributed training and inference across multiple datasets and diverse computational resources.Source: A. Motivation · p. 2

B. Brief History of LLMs

• Language-model development progressed from statistical models to neural and pretrained Transformer-based models, culminating in large models with broad task capabilities.GPT-3 is described as having 175 billion parameters and performing diverse tasks without task-specific fine-tuning.Source: B. Brief History of LLMs · p. 2

C. Related works and Contributions

• Existing surveys address selected aspects of distributed or federated LLMs, but this survey extends coverage to LLMs, VLMs, MLLMs, and SLMs across six decentralization areas.The six areas are distributed training; distributed inference and optimization; distributed computing infrastructures; federated learning and fine-tuning; edge computing and mobile intelligence; and communication efficiency.Source: C. Related works and Contributions · pp. 3–4, Table II · p. 3

D. Organization

• The paper is organized around model overviews, pipeline decentralization, related studies on LLMs and VLMs, SLMs, aggregated future directions, and a conclusion.The survey also provides a GitHub page intended to track rapidly evolving distributed and federated MLLM research.Source: §I · p. 1, I Introduction · p. 1, D. Organization · p. 4

A. Definition of Multimodal Large Language Models

• LLMs primarily process text, whereas MLLMs receive, process, and generate information across modalities including text, images, videos, and audio.MLLMs combine billion-parameter LLMs with multimodal instruction tuning and can support tasks such as image-to-code generation and mathematical problem solving.Source: A. Definition of Multimodal Large… · p. 4
• MLLMs remain limited by extended multimodal sequences and instruction-following capabilities relative to closed-source systems such as GPT-4V.These limitations constrain processing of long videos or complex multimodal datasets.Source: A. Definition of Multimodal Large… · p. 4

B. A Brief Overview of the LLM Pipeline

• The LLM pipeline begins with collecting, curating, cleaning, and preparing large, diverse datasets before preprocessing, with distributed synthetic-data generation remaining an open question.Synthetic data can be generated through prompt engineering and multi-step generation, but distributed devices for incrementally increasing data diversity and complexity are not resolved.Source: B. A Brief Overview of the LLM Pipeline · p. 6

2. Data Preprocessing:

• The standard LLM pipeline includes data preparation, self-supervised pre-training, supervised fine-tuning, alignment to human preferences, post-training optimization, deployment, and evaluation stages.Pre-training can use model and data parallelism, while fine-tuning can use PEFT methods such as LoRA to reduce computational burden.Source: §2 · p. 6, §3 · p. 6, §4 · pp. 7–8

C. Differences of LLM and MLLM pipelines

• MLLMs extend text-only LLM pipelines by collecting and synchronizing multimodal data, using cross-modal objectives, and supporting capabilities such as image captioning and visual question answering.Their pipelines require multimodal preprocessing, specialized encoders, alignment modules, and multimodal fine-tuning procedures.Source: C. Differences of LLM and MLLM pipelines · pp. 8–9
• MLLM optimization is more computationally expensive than LLM optimization because it must process additional modalities, motivating modality-specific compression, efficient cross-modal attention, and shared parameters.Deployment likewise faces increased infrastructure, computational-load, and latency requirements for multimodal inputs and outputs.Source: C. Differences of LLM and MLLM pipelines · pp. 8–9

A. Overview of FL

• Federated learning distributes local model updates across clients and aggregates them into a global model, enabling privacy-preserving and resource-efficient training with local data.Clients repeatedly fine-tune locally, send updates to a server, and receive the aggregated model for subsequent rounds.Source: A. Overview of FL · p. 9

Federated MLLM framework

• Federated MLLM training combines modality-specific encoders, feature alignment, a language-model cognitive module, and server-side aggregation of locally trained models.The framework processes images, video, audio, and text while preserving local data and iteratively producing a global model.Source: B. Decentralization at Different Steps… · p. 9

1) Data Collection:

• Decentralized data collection and preprocessing keep multimodal data or sensitive information near its source while distributing cleaning, normalization, augmentation, tokenization, and related workloads.Federated frameworks can share metadata or processed outputs, and MapReduce or Spark can parallelize preprocessing across nodes.Source: B. Decentralization at Different Steps… · pp. 10–11

3) Pre-training:

• Distributed pre-training uses data and model parallelism to spread computation across nodes, while gradient accumulation, asynchronous updates, and communication overlap reduce synchronization costs.Edge resources and federated learning can further move computation toward data sources without sharing local data.Source: B. Decentralization at Different Steps… · p. 12
• Mamba achieves linear sequence-length scaling and is reported as five times faster than Transformers, reducing memory requirements while supporting language, audio, and genomics workloads.The method selectively propagates or filters information through input-dependent state-space dynamics.Source: B. Decentralization at Different Steps… · p. 12

4) Supervised Fine-tuning:

• Federated fine-tuning shares local model updates instead of raw multimodal data, while LoRA, FLoRA, ZeRO-Offload, and PETALS reduce communication or memory demands across heterogeneous resources.These approaches support privacy-preserving adaptation, resource-constrained fine-tuning, and collaborative access to large models.Source: B. Decentralization at Different Steps… · pp. 12–13

5.1. Decentralizing Reward Model Training

• Federated RLHF trains reward models locally from multimodal user feedback and aggregates updates centrally, preserving privacy while incorporating diverse preferences.Gradient compression, PEFT, and personalized federated learning can reduce communication overhead and address client heterogeneity.Source: 5) RLHF · p. 13

5.2. Decentralizing Policy Optimization

• Distributed policy optimization lets clients update policies locally using shared reward models and aggregate them into a global policy.MIRA uses gradient-norm constraints to reduce policy-gradient variance, while distributed PPO and zeroth-order federated tuning support decentralized optimization.Source: 5) RLHF · p. 13

6) Model Optimization:

• Federated distillation, compression, quantization, and parameter isolation can collaboratively optimize models without sharing data and with reduced communication overhead.FewFedPIT is presented as a privacy-preserving few-shot FedIT approach using local synthetic data and aggregated updates.Source: 5) RLHF · p. 13

7) Deployment:

• Distributed deployment extends RLHF-related decentralization to inference across servers, edge devices, and peer-to-peer networks, supporting lower-latency operation under constrained resources.Examples include FastServe, CoLLM, 6G-based distributed foundation models, and OpenFedLLM.Source: 5) RLHF · pp. 13–14

Algorithmic Framework

• FastServe [2] reduces distributed LLM inference latency through skip-join scheduling, proactive GPU memory management, and a prototype that improves throughput under comparable latency constraints.Its challenges include variable job sizes and GPU memory overhead from key-value caches; future work targets dynamic scheduling, edge serving, and energy efficiency.Source: §IV · p. 14
• DISTFLASHATTN [3] improves long-context distributed training through load-balanced scheduling, communication overlap, and re-materialization-aware checkpointing, yielding up to 2× throughput, 1.32× end-to-end speedup, and 1.31× training speedup.The work addresses token imbalance, communication overhead, and redundant recomputation, while future directions include longer sequences, sparse attention, energy efficiency, and cloud-edge deployment.Source: §IV · p. 16
• ACCO [4] overlaps gradient computation and communication, shares optimizer states, compensates delays, and adapts workloads across heterogeneous hardware to improve memory efficiency and GPU utilization.Brakel et al. [5] categorize inter-operator, intra-operator, and hybrid model parallelism, while identifying communication, synchronization, bandwidth, and auto-parallelization challenges.Source: §IV · pp. 17–18

Hardware Infrastructure

• CPU-oriented distributed inference [6] reduces communication and memory transfers by broadcasting token IDs, synchronizing decoder layers once, and writing results directly into communication buffers.The approach targets constrained memory, bandwidth, latency, synchronization, and CPU heterogeneity, with future work on broader architectures, adaptive scheduling, compression, and open-source benchmarking.Source: §IV · pp. 18–19
• PETALS [7] enables fault-tolerant Internet-scale inference and fine-tuning across unreliable heterogeneous nodes, achieving up to 10× speedup over local offloading in geo-distributed deployments.Its challenges include disconnections, high-latency communication, load balancing, fault tolerance, throughput, and correctness; future work includes privacy protection, compression, and broader hardware support.Source: §IV · p. 19

Data Layer

• FedKSeed [8] enables federated full-parameter tuning of billion-parameter LLMs with communication costs under 18 kilobytes per round using finite seeds, scalar gradients, and differentiated seed sampling.It addresses communication, resource, and heterogeneous-data constraints, while future work targets privacy preservation, adaptive sampling, asynchronous federation, and hybrid optimization.Source: §IV · p. 19

Hardware Infrastructure

• MEI4LLM [9] integrates caching, training, inference, parameter sharing, and split learning for resource-efficient edge deployment in latency-sensitive and privacy-critical domains.Khoshsirat et al. [10] additionally use semi-Markov energy modeling and adaptive scheduling to improve decentralized inference under energy harvesting and heterogeneous edge conditions.Source: §IV · pp. 19–20
• Ren et al. optimize heterogeneous-GPU serving through heuristic scheduling and model pipelining, while ScaleLLM [12] combines end-to-end latency analysis, hybrid parallelism, quantization, and gateway optimization.ScaleLLM reports a 4.3× speedup over vLLM and 1.5× higher throughput than state-of-the-art solutions; FS-LLM [15] adds federated fine-tuning benchmarks, PEFT, and extensible resource-efficient operators.Source: §IV · pp. 20–21

FwdLLM

• FwdLLM [16] removes backpropagation through forward gradients and PEFT, using variance-controlled perturbations and discriminative sampling to reduce mobile-device overhead and accelerate convergence.Its scalability remains challenged by non-iid data, device constraints, perturbation accuracy, and large device populations; future work includes differential privacy, secure aggregation, adaptive optimization, multimodal extension, and edge-hardware compatibility.Source: §IV · p. 21

FedMeZO

• FedMeZO [17] combines two-point zeroth-order estimation, convergence guarantees for i.i.d. and non-i.i.d. settings, low-rank Hessian analysis, and personalized learning rates to reduce federated LLM-tuning memory.The method still must balance reliable convergence, personalization, computational efficiency, and privacy across heterogeneous clients; proposed extensions include more efficient ZOO, secure aggregation, and multimodal or cross-device FL.Source: §IV · pp. 21–22

Communication- and scale-efficient FL

• FedRDMA [19] improves WAN federated communication with chunked RDMA, buffering, reassembly, and a memory-pool FedRDMA-E variant, while Ferret [20] scales full-parameter tuning through low-dimensional projections and shared-randomness reconstruction.Both methods face network, representation, privacy, and aggregation trade-offs; future work targets heterogeneous data, larger systems, diverse WAN conditions, reconstruction accuracy, and adaptive optimization.Source: §IV · p. 22

OpenFedLLM and federated safety

• OpenFedLLM [21] integrates FedIT, federated value alignment, seven FL algorithms, eight datasets, and 30+ metrics, while studying safety attacks that exploit malicious clients and post-hoc defenses for FedIT.The studies highlight heterogeneous preferences, private-data aggregation, stealthy attacks, and safety risks; future work emphasizes robust detection, broader attack coverage, personalized FL, and stronger privacy mechanisms.Source: §IV · pp. 22–23

Benchmarks and domain coverage

• FedLLM-Bench [24] provides four realistic datasets, eight baseline methods, and six metrics capturing multilingualism, heterogeneity, quality, quantity, and user preferences, while FedDCA [25] improves domain coverage through client selection and retrieval-based instruction augmentation.Their challenges include fair evaluation, data heterogeneity, privacy threats, encoder alignment, and consistent performance; future work targets multilingual collaboration, personalization, safety, dynamic coverage, and synthetic augmentation.Source: §IV · pp. 23–24

Resource-efficient and few-shot tuning

• FedBiOT compresses LLMs into an emulator and adapter aligned by bi-level optimization, while FewFedPIT [27] combines synthetic data, parameter isolation, and secure local aggregation for privacy-preserving few-shot instruction tuning.FedBiOT must handle server-client distribution shifts and convergence under resource constraints; FewFedPIT faces data scarcity, privacy, aggregation overhead, and human-centered-design challenges.Source: §IV · p. 24

Secure distributed training

• Huang et al. secure distributed LLM training with TEE-based model slicing, split fine-tuning, lightweight encryption, and sparsification parameter fine-tuning, but TEEs, encrypted precision, and GPU-TEE communication constrain scalability.Future work includes TEE accelerators, adaptive encryption, and more efficient sparsification and fine-tuning without sacrificing accuracy.Source: §IV · p. 25

FedPEAT

• FedPEAT [30] combines PEFT, emulator-assisted tuning, FL, and SABPPO-based adaptive control for private, resource-efficient fine-tuning across heterogeneous devices and server-client settings.Remaining issues include scalability, privacy-performance balance, transmission overhead, heterogeneous hardware, and dynamic networks; future work targets adaptive control, emulator compression, and multimodal datasets.Source: §IV · p. 25

Federated pre-training

• Photon [31] enables scalable federated pre-training of billion-scale LLMs across organizations using private data and heterogeneous computational resources, achieving competitive centralized-training performance with lower communication overhead.Its design addresses statistical heterogeneity, partial participation, and scaling with model size, while larger models achieve better consensus and generalization.Source: §IV · pp. 25–26

Federated fine-tuning

• FedPFT [32] combines layer-wise sub-FM compression with layer- and neuron-level knowledge distillation to enable privacy-preserving federated fine-tuning across all foundation-model layers and diverse NLP tasks.The framework targets gradient errors, computational overhead, and scalability limitations caused by sub-model construction and alignment.Source: §IV · p. 26

Cross-modal federated forecasting

• TIME-FFM [33] converts time series into text tokens, uses dynamic domain-specific prompts and personalized federated training, and achieves state-of-the-art forecasting in few-shot and zero-shot settings.The framework addresses cross-domain heterogeneity, modality alignment, privacy, and communication and computation overhead through local prediction heads and domain adaptation.Source: §IV · p. 26

Resource-efficient federated adaptation

• FedCyBGD [35] enables full-parameter LLM tuning on resource-constrained edge devices through cyclic block updates and compression, achieving competitive centralized performance while reducing memory, computation, communication, and model-download costs.Related federated foundation-model methods further support privacy-preserving pre-training, fine-tuning, continual learning, retrieval-augmented generation, and prompt tuning, while FTL-FM [37] provides a taxonomy of privacy- and efficiency-oriented transfer-learning approaches.Source: §IV · pp. 26–27

Secure training and edge deployment

• FL-GLM [42] protects private data through client-side input and output layers, encrypted communication, and parallel client-server training, matching centralized ChatGLM performance on SuperGLUE and summarization tasks.Its remaining challenges include embedding-gradient attacks, client computation, sequential-training inefficiency, and non-IID data; edge deployment studies similarly examine model-size, PEFT, RAG, compression, personalization, energy, latency, and stability trade-offs [43] [44].Source: §IV · pp. 27–28

Collaborative inference and distributed optimization

• Collaborative methods address distributed LLM inference and training challenges through CoLLM’s tensor parallelism and adaptive load balancing, PAFT’s parallel SFT and preference alignment, and Co-S2P’s structured pruning with semi-asynchronous aggregation.CoLLM achieves up to 2.3x inference-latency speedup, while PAFT ranks first for 7B and 70B models on the Open LLM Leaderboard.Source: §IV · p. 29

Distributed training and scheduling

• Distributed optimization methods use quantized stochastic gradient descent, which has convergence guarantees in non-convex domains and was validated on GPT-family models up to 1.3 billion parameters, despite communication bottlenecks [48].Future research includes improving quantization, dynamically optimizing bit-width, exploring architectures beyond LLMs, and integrating advanced distributed frameworks for larger models and datasets.Source: §IV · p. 30

Distributed multimodal training

• DISTMM achieves up to 3.27× speedup over Megatron-LM by combining modality-aware partitioning, heterogeneity-aware placement, and pipeline scheduling for distributed multimodal training.These mechanisms address submodule imbalance, inter-device communication overhead, and pipeline limitations that can degrade multimodal model quality.Source: §IV · p. 32

Multi-task multimodal training

• Spindle improves multi-task multimodal training by pairing heterogeneity-aware workload parallelization with dependency-driven scheduling and a stage-based runtime engine [55].The approach targets heterogeneous data flows and dependencies between shared and task-specific components that otherwise waste computing resources; future work includes larger datasets, dynamic balancing, and communication or memory optimization.Source: §IV · p. 32

Edge inference

• TPI-LLM enables 70B-scale LLM inference on low-resource edge devices through sliding-window memory scheduling and star-based allreduce, addressing memory, latency, and privacy constraints [56].Future directions include broader device compatibility, adaptive resource scheduling, network-aware allreduce refinement, and further memory reduction for larger models.Source: §IV · p. 32

Decentralized inference

• MetaInf adaptively selects decentralized inference acceleration methods using historical performance data and dataset-, model-, and hardware-aware embeddings [57].Its main challenges are hardware and workload heterogeneity, changing system conditions, and generalization under latency, cost, scalability, and robustness constraints; future work considers speculative decoding, adaptive quantization, edge computing, and federated learning.Source: §IV · pp. 32–33

Parameter-efficient fine-tuning

• LLM-Adapters reduces computational and storage requirements through diverse adapter-based PEFT, while DLoRA distributes fine-tuning across cloud and edge devices to address scalability, privacy, and computational-efficiency challenges [59] [60].LLM-Adapters evaluates adapter placements, configurations, and performance across arithmetic and commonsense reasoning datasets and creates Math10K and Commonsense170K; DLoRA uses a "Kill and Revive" algorithm to dynamically adjust active parameters.Source: §IV · p. 33

Federated instruction tuning

• FedHDS [63] combines hierarchical clustering, multilayer feature fusion, and coreset selection to reduce federated instruction-tuning redundancy and computational cost while improving unseen-task Rouge-L scores.Its two-layer selection mechanism targets both intra-client and inter-client redundancy, but challenges remain in non-IID overfitting, data traversal overhead, privacy compatibility, and extending selection across modalities and heterogeneous clients.Source: §IV · p. 34

Federated retrieval

• FRAG [64] enables mutually distrustful parties to perform privacy-preserving collaborative encrypted ANN searches through SK-MHE and MC protocols.The framework targets key-management, homomorphic-encryption overhead, scalability, query and modality expansion, and adversarial robustness in federated RAG.Source: §IV · pp. 34–35

Personalized federated fine-tuning

• MIRA [65] and FDLoRA [66] personalize federated LLM fine-tuning across heterogeneous, non-IID clients using multi-task regularization, LoRA, dual global-personalized modules, and gradient-free adaptive fusion.They address task alignment, parameter synchronization, memory, computation, and communication constraints while reporting improved local, global, client-specific, and domain performance; future work targets larger task networks, adaptive personalization, compression, and broader applications.Source: §IV · p. 35

Gradient-subspace federated fine-tuning

• FedFTG [67] addresses LoRA rank inflation and heterogeneous-client learning through direct weight aggregation, GaLore optimization, and efficient gradient-subspace learning for LLMs and ViTs.The work targets quadratic risk bounds and PEFT inefficiency, with future directions including adaptive aggregation, memory-efficient optimization, multimodal extension, and privacy integration.Source: §IV · p. 36

Cloud, edge, and decentralized training

• Cross-cloud federated training [68], PAC, and FusionLLM [70] pool heterogeneous cloud or edge resources using dynamic partitioning, optimized communication, adaptive aggregation, hybrid parallelism, activation caching, DAG execution, and adaptive compression.These systems improve scalability, resource utilization, training speed, memory efficiency, or communication efficiency while confronting platform heterogeneity, bandwidth limits, synchronization, stragglers, device constraints, and inter-device latency.Source: §IV · pp. 36–37

RLHF systems

• HybridFlow [71] improves RLHF execution by combining single- and multi-controller dataflows, hierarchical APIs, 3D-HybridEngine transitions, and automated GPU placement to increase throughput and reduce memory redundancy.Its design addresses inflexible dataflow representations, redundant communication and memory, workload balancing, and training-generation transitions, with future work targeting larger models, emerging RLHF algorithms, and easier configuration.Source: §IV · p. 37

Efficient and private edge deployment

• EdgeQAT [72] uses entropy- and distribution-guided QAT with token-importance-aware bit allocation to mitigate activation-quantization distortion and achieve up to 2.37× speedup over FP16 counterparts.PrivateLoRA [73] complements edge deployment with low-rank residual transmission and local sensitive computation, reducing communication while preserving data locality and achieving improved throughput and task performance versus device-only solutions with comparable GPU benchmarks.Source: §IV · pp. 37–38

Multimodal wireless learning

• Distributed multimodal foundation-model training for 6G combines pipeline and data parallelism, compressed activations and gradients, FL with AirComp, and modality alignment for integrated NLP-CV learning over wireless networks.The approach must address non-IID multimodal data, unstable links, disconnections, resource heterogeneity, synchronization, interference, model partitioning, and communication management.Source: §IV · p. 38

Heterogeneous Training and Acceleration

• Chen et al [75], Huang et al [76], and Xu et al [77] improve heterogeneous distributed execution through pipeline partitioning, CPU-FPGA acceleration, unified communication, performance prediction, and automatic parallel planning.These methods target memory, computation, operator, communication, and hardware incompatibilities across mobile and heterogeneous accelerator environments.Source: §IV · p. 39

Federated Fine-Tuning and Efficiency

• Shuai et al [78], Fang et al [80], Fan et al [81], and Woisetschläger et al [82] combine federated learning with robust alignment, PEFT, quantization, privacy protection, and energy-aware optimization for heterogeneous clients.Their challenges include data and architecture heterogeneity, stragglers, model-update privacy, limited bandwidth and memory, energy constraints, and scalability.Source: §IV · pp. 40–41

Distributed Inference and Communication

• Fu et al [84] and Xin et al reduce distributed inference overhead through multi-tier checkpoint loading, live migration, locality-aware scheduling, immediate communication, and tile-wise computation-communication overlap.These systems address cold starts, resource contention, dependency-aware scheduling, communication-library adaptation, and limited compiler support.Source: §IV · pp. 41–42

Privacy-Preserving LLM Fine-Tuning

• Shabani et al combine federated learning with LoRA to keep data localized, reduce computation, and achieve comparable performance to centralized T5 summarization training.The approach remains challenged by non-iid convergence, communication overhead, update privacy, and client-participation incentives.Source: §IV · p. 42

Unsupervised Federated VLM Learning

• Yan et al [87] use CLIP zero-shot predictions, evolving pseudo-labels, and class-balanced data generation to enable lightweight unsupervised federated learning under unlabeled, heterogeneous client data.The method addresses annotation and resource limits but depends on improving pseudo-label quality, robustness to class imbalance, scalability, and synthetic-instance sampling.Source: §IV · p. 42

Fairness and Communication-Efficient Federated Learning

• Zeng et al introduce FF-DVP to debias pretrained VLMs using fairness-aware visual prompting, modality-fused heads, and client-specific constraints, while Raje et al use sparse LoRA updates to reduce FL communication by up to 10x.FF-DVP targets fairness and convergence under heterogeneous data, whereas FLoSS addresses communication budgets through sparsity and LoRA-rank heuristics.Source: §IV · p. 43

Communication-Efficient FL for LLMs

• Raje introduced FLoSS, applying communication-phase top-k sparsity to LoRA-based federated training to reduce communication costs by up to 10x while preserving model performance.The work also developed heuristics for selecting LoRA ranks and sparsity ratios under varying network conditions and communication budgets.Source: §IV · p. 43
• Communication-efficient federated LLM training must balance sparsity-induced communication savings with accuracy, heterogeneous bandwidth, client constraints, convergence, and fairness across non-identically distributed data.Future work includes automated sparsity and LoRA-rank selection, quantization or advanced pruning, and privacy-preserving extensions of FLoSS.Source: §IV · p. 43

Distributed LLM Training

• DisLLM combines split learning, federated learning, LoRA, and local differential privacy by keeping sensitive data on clients while distributing computation across client- and server-side model components.Evaluations reported comparable accuracy to centralized models alongside enhanced privacy protection and optimized resource utilization.Source: §IV · pp. 43–44
• Distributed LLM training faces computational constraints on clients, privacy-preservation difficulties, accuracy–efficiency trade-offs, and challenges in choosing model split points and privacy levels.These issues affect scalability and effectiveness in real-world applications.Source: §IV · p. 44
• Future DisLLM research should dynamically allocate resources and adjust model splits, adapt privacy mechanisms to data sensitivity, and improve client–server communication efficiency for larger networks.These directions target changing resource availability, privacy needs, and client populations.Source: §IV · p. 44

Popov et al [91]

• Popov et al combine random rehearsal and model averaging to adapt language models to private user data while preserving original-dataset quality and reducing communication costs.The framework addresses catastrophic forgetting through on-device and server-side updates.Source: A. Popov et al [91], Distributed… · p. 44

Wu et al [93]

• CG-FedLLM compresses and reconstructs federated gradients with an AutoEncoder, using TGAP and FAF to reduce communication costs while retaining performance and privacy.Its evaluations report improvements in performance and robustness over centralized and federated fine-tuning approaches.Source: A. Popov et al [91], Distributed… · p. 45

Sun et al [94]

• FFA-LoRA freezes one low-rank matrix to halve trainable parameters, improving stability and performance under privacy-preserving federated learning with heterogeneous data and noisy gradients.The method targets discordances involving differential privacy, data heterogeneity, and scaling-factor sensitivity.Source: A. Popov et al [91], Distributed… · p. 45

Bai et al [95]

• FlexLoRA uses SVD-based aggregation to combine client updates with varying resources and tasks, improving generalization and scalability in heterogeneous federated fine-tuning.The work studies how LoRA rank, client heterogeneity, and resource distributions affect cross-device federated learning.Source: A. Popov et al [91], Distributed… · pp. 45–46

Gao et al [96]

• PMG-FL adapts vision-language foundation models at the edge through personalized prompts, distance-based aggregation, and cross-granularity guidance for IID and non-IID data.These mechanisms address device diversity, limited resources, and knowledge fusion across fine- and coarse-granularity data.Source: A. Popov et al [91], Distributed… · p. 46

Wang et al [97]

• FLoRA enables noise-free aggregation of heterogeneous LoRA ranks through stacking, yielding improved performance and convergence efficiency in homogeneous and heterogeneous settings.The framework addresses aggregation noise, diverse client configurations, and communication or storage overhead.Source: A. Popov et al [91], Distributed… · p. 46

Li et al [98]

• MLLM-FL combines server-side multimodal pretraining, federated fine-tuning, and global alignment to improve privacy and efficiency on heterogeneous and long-tailed data.The approach reduces client-side computational overhead but relies on high-quality server-side resources for global alignment.Source: A. Popov et al [91], Distributed… · pp. 46–47

Zhang et al [99]

• Distributed foundation-model training for 6G uses pipeline and data parallelism, gradient compression, and adaptive resource allocation to distribute multimodal workloads across wireless devices.The approach targets energy consumption, bandwidth limits, device disparities, and communication instability.Source: A. Popov et al [91], Distributed… · p. 47

• The section presents core novelties from selected VLM and decentralized-VLM studies.Source: §V · p. 47
• For each study, the review discusses contributions and challenges.Source: §V · p. 47
• The review also outlines potential future directions and includes some relevant survey papers alongside its main focus on non-survey articles.Source: §V · p. 47

• VLMs support image captioning, visual question answering, and cross-modal retrieval by aligning visual perception with linguistic reasoning.Source: A. Introduction to VLMs · p. 47
• Vision encoders extract image features, language encoders process text, and linear projection maps both representations into a shared embedding space.Source: A. Introduction to VLMs · p. 47
• Fusion may occur early, intermediately through attention, or late at the decision level, producing different patterns of cross-modal interaction.Source: A. Introduction to VLMs · p. 48
• CLIP uses contrastive image-text alignment, BLIP-2 uses a Querying Transformer, and Flamingo's Perceiver Resampler reduces visual features to a fixed number of outputs.The Perceiver Resampler reduces computational complexity while retaining essential visual information.Source: A. Introduction to VLMs · p. 48
• A typical VLM pipeline processes multimodal inputs with separate encoders, followed by linear projection and cross-attention to align and integrate the modalities.Source: A. Introduction to VLMs · p. 48, Figure 10 · p. 48

VLM Training

• Image encoders such as ViT or CLIP produce visual embeddings, text decoders process language, and a multimodal projector aligns both into a shared space.Source: 1) VLM Training · p. 48
• Fine-tuning selects vision, language, projection, or cross-attention layers according to domain differences and task requirements.Source: 1) VLM Training · p. 48
• Full fine-tuning enables complete adaptation but is computationally intensive, whereas partial fine-tuning and LoRA reduce costs by freezing or lightly modifying selected layers.Source: 1) VLM Training · p. 48

CLIP

• CLIP aligns image and text embeddings using contrastive learning on 400 million image-text pairs, producing transferable multimodal representations.Its shared semantic space supports generalization across diverse tasks with limited task-specific fine-tuning.Source: B. Pretrained Foundation Models · pp. 48–49
• CLIP’s development required substantial computation, robust architectural and loss design, and curation of noisy, variable web-sourced data.These challenges affected scalability, robustness, and training-example quality.Source: B. Pretrained Foundation Models · p. 49

Future Directions

• Future work for these foundation models includes video and temporal understanding, adaptive data curation, domain-specific applications, fairness, bias mitigation, scalability, and adversarial robustness.The proposed directions extend multimodal interaction and improve reliability in real-world deployment.Source: B. Pretrained Foundation Models · p. 49

BLIP

• BLIP combines a vision-language encoder, a generative decoder, and CapFilt to support both understanding and generation while improving pretraining data quality.CapFilt generates pseudo-captions and filters noisy or irrelevant image-text pairs.Source: B. Pretrained Foundation Models · p. 49
• BLIP addresses the tension between retrieval-oriented encoding and caption generation while mitigating noisy and insufficient alt-text captions through pseudo-caption generation and filtering.The framework’s data curation improves the relevance and informativeness of pretraining examples.Source: B. Pretrained Foundation Models · p. 49

Flamingo

• Flamingo integrates pretrained vision and language models with a Perceiver Resampler and Gated Cross-Attention Dense layers for interleaved visual-text inputs.It reports state-of-the-art few-shot performance across 16 multimodal benchmarks while using minimal task-specific data.Source: C. Centralized VLM · pp. 49–50
• Flamingo must preserve pretrained knowledge while controlling cross-attention overhead and high-resolution scalability, and it trails contrastive methods on classification tasks.Its future directions include more efficient resampling and hybrid generative-contrastive approaches.Source: C. Centralized VLM · p. 50

LLaVA

• LLaVA combines CLIP’s vision encoder with a language decoder through GPT-assisted instruction-data generation and two-stage end-to-end training.LLaVA-Bench and open-source models, datasets, and tools support multimodal instruction following and visual reasoning.Source: C. Centralized VLM · p. 50
• LLaVA is constrained by scarce high-quality instruction-following data, difficult visual-language alignment, training-scale demands, and limited visual tokenization for complex image semantics.Future work emphasizes improved tokenization, fusion, and scalable synthetic or machine-annotated data generation.Source: C. Centralized VLM · p. 50

MiniGPT-4

• MiniGPT-4 aligns a pretrained vision encoder with Vicuna through one projection layer and uses two-stage training with curated detailed image descriptions.The model demonstrates image description, handwritten-draft website generation, humor, poetry, and meme interpretation with limited computational resources.Source: C. Centralized VLM · p. 50
• MiniGPT-4’s early short-caption training caused fragmented language, while longer descriptions exposed hallucination and spatial-relation limitations.The cited remedy is curated data and targeted fine-tuning, with future work including AI-feedback reinforcement learning and spatial-reasoning datasets.Source: C. Centralized VLM · pp. 50–51

FLoRA

• FLoRA applies LoRA to CLIP for parameter-efficient federated fine-tuning, reducing communication overhead while maintaining performance across datasets.Its design targets privacy, resource constraints, and efficient updates without full-model aggregation.Source: D. Summary, Contributions, Challenges,… · p. 51

CLIP2FL

• CLIP2FL transfers CLIP knowledge to clients through distillation and uses text-guided prototype contrastive learning to retrain a balanced server classifier.The framework reports state-of-the-art results on CIFAR-10/100-LT and ImageNet-LT.Source: D. Summary, Contributions, Challenges,… · p. 51
• CLIP2FL remains limited by client heterogeneity, long-tailed imbalance, scalability bottlenecks, pretrained-weight access, and evaluation dominated by classification accuracy.Broader efficiency, fairness, robustness, and energy metrics remain underexplored.Source: D. Summary, Contributions, Challenges,… · p. 51

TriplePlay

• TriplePlay combines CLIP adapters, GAN-based synthetic data, quantization, and QLoRA to address long-tailed client data and resource constraints.It reports faster convergence, lower GPU usage, and improved accuracy than FedCLIP across PACS and Office-Home.Source: D. Summary, Contributions, Challenges,… · pp. 51–52

FedCLIP

• FedCLIP uses attention-based adapters instead of full-model updates to reduce computation and communication while supporting generalization and personalization.The main design challenge is preserving pretrained capabilities under heterogeneous client distributions.Source: D. Summary, Contributions, Challenges,… · p. 52

pFedPrompt

• pFedPrompt combines global linguistic consensus with local visual adaptation to personalize VLMs under non-IID data.Its evaluations report better accuracy and robustness than state-of-the-art personalized federated methods.Source: D. Summary, Contributions, Challenges,… · p. 52

FedDAT

• FedDAT balances client-specific and client-agnostic knowledge through parallel adapters and mutual knowledge distillation in heterogeneous multimodal federated learning.Evaluations on four multimodal benchmarks report superior performance, scalability, and convergence versus existing PEFT methods.Source: D. Summary, Contributions, Challenges,… · p. 53

FedDCA

• FedDCA augments domain-specific federated instructions with client-private and server-public data, using heterogeneous encoders to improve domain coverage while maintaining privacy and efficiency.Its challenges include scaling, memory-extraction threats, and semantic alignment across heterogeneous encoder outputs.Source: D. Summary, Contributions, Challenges,… · p. 53

Overview of Small Language Models

• Small language models are designed to run efficiently on resource-constrained edge devices where local execution of full LLMs is difficult.They support broader deployment of language-model capabilities in distributed applications.Source: §VI · p. 53
• Table VII summarizes selected small language models, while accompanying repository links, model-hub links, and PyTorch snippets guide deployment.Additional model details are provided in Appendix A.Source: §VI · p. 53, Table VII · p. 54

Distributed Inference and Optimization

• Distributed MLLM inference should pursue memory-efficient alternatives, automated parallelism, and adaptive scheduling across heterogeneous edge-cloud resources.State Space Models such as Mamba, LSSL, and S4 offer linear-time inference, while dynamic scheduling can respond to workload, network, energy, and device-resource changes.Source: §VII · p. 55

Memory Management Optimization

• Future DLLM systems should optimize memory through adaptive allocation, secure RDMA, smart KV-cache management, federated memory techniques, compression, partitioning, and efficient attention or sparsification.These approaches target lower latency, communication overhead, memory consumption, and computational cost across constrained and heterogeneous devices.Source: §VII · pp. 55, 57

Edge Computing and Mobile Intelligence

• Edge intelligence research should reduce energy use and environmental impact through compression, gradient methods, power-aware scheduling, renewable energy, carbon-aware operation, multilingual adaptation, and personalized DLLMs.Personalized strategies must address non-IID local data while adapting model complexity to device constraints through methods such as LoRA, prefix-tuning, clustered FL, and meta-learning.Source: §VII · p. 57

Communication Efficient Algorithms

• Communication-efficient DLLMs should combine gradient compression and pruning with adaptive, hierarchical, clustering-based, and semantic aggregation to reduce bandwidth while preserving scalability and convergence.Task-specific updates, compact semantic representations, and mechanisms such as ACCO can avoid transmitting raw model data unnecessarily.Source: §VII · pp. 57–58

Privacy-Preserving Distributed LLMs

• Privacy-preserving DLLM research should tailor encryption, secure aggregation, differential privacy, multi-party computation, TEEs, and federated fine-tuning to resource-constrained and asynchronous settings.Open challenges include adaptive malicious-attack detection, secure federated transfer learning, and reducing the overhead of privacy protection.Source: §VII · p. 58

Addressing Heterogeneity in DLLMs

• Addressing data, task, model, and hardware heterogeneity requires adaptive client updates, synthetic augmentation, multimodal cross-domain alignment, temporal learning, mixture-of-experts, asynchronous processing, and resource-aware participation.These methods aim to improve convergence, fairness, robustness, personalization, and scalability across diverse devices, objectives, modalities, and data distributions.Source: §VII · pp. 58–59

Vision-Language Models

• VLM research should reduce hallucinations and improve fine-grained vision-language correlation, commonsense reasoning, cyberattack resilience, and mathematical reasoning for reliable real-world applications.Suggested mechanisms include broader and better-synthesized training data, fine-grained pretraining, structured and symbolic reasoning, Chain of Thought, reinforcement learning, and external computational engines.Source: §VII · pp. 59–60
• Further VLM directions include energy-efficient architectures for edge deployment and continual learning that preserves prior knowledge while adapting to new multimodal data.Potential solutions include pruning, quantization, distillation, structured memory, self-supervised learning, and cross-modal consistency regularization.Source: §VII · pp. 59–60

Explainability and Transparency

• VLM research should embed explainability into model architectures, strengthen causal interpretability, and verify consistency across visual and language inputs to improve trustworthiness.LVLM-Interpret demonstrates analysis through attention maps, relevancy scores, and causal explanations, while current multimodal fusion remains difficult to interpret.Source: §VII · p. 60

Advanced Multi-Modal Learning

• Distributed multimodal learning should develop modality-specific compression, quantization, sparsification, adaptive offloading, and efficient cross-modal representations for resource-constrained edge systems.Proposed techniques include cross-modal fusion, grouping, attention pruning, Layer Normalization fine-tuning, and entropy-aware quantization.Source: §VII · p. 60

Distributed Model Optimization for MLLMs

• Distributed MLLMs should dynamically adapt model structure, hardware allocation, scheduling, communication, and distillation to heterogeneous modalities and devices from edge nodes to cloud servers.Relevant methods include gradient compression, activation checkpointing, channel pruning, weight factorization, and cross-modal distillation for lightweight sub-models.Source: §VII · p. 60

Score Entropy Discrete Diffusion models

• SEDD [319] offers a direction for distributed LLMs and MLLMs by enabling parallelized inference, improved memory efficiency, and potentially more scalable, robust, and privacy-preserving edge-cloud deployment.Future work includes adapting discrete diffusion to hierarchical, federated, and edge-cloud architectures.Source: §VII · pp. 60–61

Large Concept Models

• Large Concept Models [320] could reduce distributed MLLM complexity and communication by reasoning over modality-agnostic high-level concepts rather than individual tokens.Future directions include optimizing SONAR embeddings for low-resource settings and extending hierarchical reasoning to paragraph- or section-level abstractions.Source: §VII · p. 61

DeepSeek-R1-Zero

• DeepSeek-R1-Zero [321] demonstrates a scalable reasoning pathway based on pure reinforcement learning, while DeepSeek-R1 combines cold-start data, reasoning-focused RL, rejection sampling, and distillation into smaller models.The approach targets stronger, more readable reasoning with reduced supervised-data reliance and supports iterative improvement through reinforcement learning and distillation.Source: §VII · p. 61

• The survey examines distributed training, inference, and deployment for LLMs and MLLMs while identifying implementation challenges.Source: §VIII · p. 61
• It highlights scalability and growing computational demands as emerging challenges for these models.Source: §VIII · p. 61
• Decentralized approaches offer solutions for improving scalability and deploying models across diverse computational resources and at the edge.Source: §VIII · p. 61
• The survey also discusses LCM, which processes high-level concepts in a universal space rather than individual words or tokens.Source: §VIII · p. 61
• LCM uses SONAR sentence embeddings and supports multilingual tasks across 200 languages with fewer computational resources than token-based models.Source: §VIII · p. 61

A. GPT-2 Small

• GPT-2 Small uses 124M parameters and reduces communication costs in federated learning by sharing relatively few parameters with the central server.It is presented as a practical option for resource-constrained devices and heterogeneous distributed environments.Source: A. GPT-2 Small · p. 69

B. DistilGPT-2

• DistilGPT-2 reduces computational requirements by approximately 40% while retaining 97% of GPT-2’s performance.Its average forward pass is approximately twice as fast, with reduced memory and bandwidth needs in distributed settings.Source: B. DistilGPT-2 · p. 69

C. GPT-Neo 125M

• GPT-Neo 125M supports heterogeneous devices and non-IID datasets through open-source customization and broad training on The Pile.Its training data may also produce socially unacceptable content, motivating content-filtering mechanisms.Source: C. GPT-Neo 125M · p. 70

D. LLaMA-2 7B

• LLaMA-2 7B delivers strong performance on multilingual NLP, code generation, and debugging but imposes substantial computational and communication overhead.These costs make it less suitable for decentralized environments with limited hardware.Source: D. LLaMA-2 7B · p. 70

E. ALBERT-Base

• ALBERT-Base reduces parameters through cross-layer sharing and factorized embeddings, supporting edge and federated learning while limiting complex generation and reasoning.Its parameter reduction lowers memory and computational requirements but restricts performance across some NLP domains.Source: E. ALBERT-Base · p. 70

F. TinyBERT

• TinyBERT has 14.5M parameters, offers 10X faster inference, and incurs a 4% GLUE performance reduction after transformer distillation.It is suited primarily to NLU tasks such as classification, question answering, and sentiment analysis rather than generation.Source: F. TinyBERT · p. 70

H. MobileBert

• MobileBERT is approximately 4.3 times smaller than BERT-Base, achieves a 5.5x speedup, and remains comparable on GLUE while outperforming BERT-Base on SQuAD.Its bottleneck transformer improves efficiency but makes fine-tuning more sensitive to hyperparameter selection.Source: H. MobileBert · p. 71

I. T5 Small

• T5-Small provides a unified sequence-to-sequence architecture for translation, summarization, and multi-task federated learning.Source: §I · p. 71

M. ByT5 Small

• ByT5-Small processes text at the byte level, improving robustness to noisy and informal data while creating longer sequences that require more computation.Source: M. ByT5 Small · p. 72

N. Flan-T5 Small

• Flan-T5 Small is instruction-tuned and generalized across multimodal datasets, but fully leveraging it requires pre-training data that may be unavailable in constrained environments.Source: N. Flan-T5 Small · p. 72

O. BLOOMZ 560M

• BLOOMZ 560M supports multilingual zero-shot tasks and transfers knowledge from high-resource to low-resource languages.Source: O. BLOOMZ 560M · p. 72

Q. Galactica 125M

• Galactica-125M is specialized for scientific question answering, mathematical reasoning, molecular prediction, and citation-aware generation but lacks broad conversational generalization.Source: Q. Galactica 125M · pp. 72–73

R. Pythia 160M

• Pythia-160M balances generalization and efficiency through open-source checkpoints and training on The Pile for distributed, resource-limited use.Source: R. Pythia 160M · p. 73

S. Cerebras-GPT 256M

• Cerebras-GPT improves efficiency per model size by following Chinchilla Scaling rather than traditional scaling laws.Source: S. Cerebras-GPT 256M · p. 73

T. TinyLlama 1.1B

• TinyLLaMA uses FlashAttention and LitGPT to reduce memory and computational overhead, although domain-specific pretraining remains resource-intensive.The model reportedly requires approximately 90 days on 16 A100-40G GPUs for such pretraining.Source: T. TinyLlama 1.1B · p. 73

W. Reformer

• Reformer reduces transformer complexity from O(n^2) to O(nlogn) and lowers memory use through LSH attention, reversible layers, and partitioned computation.It is especially suited to long sequences and long-document summarization on resource-constrained devices.Source: W. Reformer · p. 74