PaperlayerView exact revision on arXiv

Source-linked AI summary

Ten Challenging Problems in Federated Foundation Models

Tao Fan, Hanlin Gu, Xuemei Cao, Chee Seng Chan, Qian Chen, Yiqiang Chen, Yihui Feng, Yang Gu, Jiaxiang Geng, Bing Luo, Shuoling Liu, Win Kent Ong, Chao Ren, Jiaqi Shao, Chuan Sun, Xiaoli Tang, Hong Xi Tae, Yongxin Tong, Shuyue Wei, Fan Wu, Wei Xi, Mingcong Xu, He Yang, Xin Yang, Jiangpeng Yan, Hao Yu, Han Yu, Teng Zhang, Yifei Zhang, Xiaojin Zhang, Zhenzhe Zheng, Lixin Fan, Qiang Yang

arXiv:2502.12176v1cs.LGcs.AI

TL;DR

FedFMs seek to combine foundation models’ general capabilities with federated learning’s privacy preservation while enabling mutual learning with local domain models. This paper surveys ten challenges across theory, data, heterogeneity, security and privacy, and efficiency, formalizing objectives, reviewing methods, and identifying potential solutions. Its central conclusion is that unified multi-objective optimization can analyze the trade-offs among these challenges, although practical deployment still faces substantial data, continual-learning, privacy, watermarking, and resource constraints.

  • Problem

    FedFMs must balance privacy, utility, efficiency, heterogeneous knowledge, security, and resource constraints when combining foundation models with private domain models.

  • Method

    The paper defines and reviews ten FedFM problems, formulates their objective functions, analyzes existing methods, and unifies their trade-offs in a multi-objective framework.

  • Results

    The survey identifies ten challenges organized into five aspects and presents a unified optimization view spanning utility, efficiency, watermark, contribution, and privacy objectives.

  • Takeaways & Limitations

    FedFMs require coordinated progress in theory, private-data utilization, continual learning, unlearning, heterogeneity, security, incentives, watermarking, and efficiency.

Abstract

from arXiv · show

Federated Foundation Models (FedFMs) represent a distributed learning paradigm that fuses general competences of foundation models as well as privacy-preserving capabilities of federated learning. This combination allows the large foundation models and the small local domain models at the remote clients to learn from each other in a teacher-student learning setting. This paper provides a comprehensive summary of the ten challenging problems inherent in FedFMs, encompassing foundational theory, utilization of private data, continual learning, unlearning, Non-IID and graph data, bidirectional knowledge transfer, incentive mechanism design, game mechanism design, model watermarking, and efficiency. The ten challenging problems manifest in five pivotal aspects: ``Foundational Theory," which aims to establish a coherent and unifying theoretical framework for FedFMs. ``Data," addressing the difficulties in leveraging domain-specific knowledge from private data while maintaining privacy; ``Heterogeneity," examining variations in data, model, and computational resources across clients; ``Security and Privacy," focusing on defenses against malicious attacks and model theft; and ``Efficiency," highlighting the need for improvements in training, communication, and parameter efficiency. For each problem, we offer a clear mathematical definition on the objective function, analyze existing methods, and discuss the key challenges and potential solutions. This in-depth exploration aims to advance the theoretical foundations of FedFMs, guide practical implementations, and inspire future research to overcome these obstacles, thereby enabling the robust, efficient, and privacy-preserving FedFMs in various real-world applications.

I. INTRODUCTION

FedFMs combine pretrained foundation models with private, locally trained domain models through privacy-preserving federated learning. The paper organizes ten challenges across foundational theory, data, heterogeneity, security and privacy, and efficiency.

  • Motivation: FedFMs use federated learning to connect at least one pretrained foundation model with multiple private domain models.This framework supports distributed collaboration while respecting data privacy.
  • Scope: The paper examines ten problems spanning foundational theory, private-data use, continual learning, unlearning, heterogeneity, knowledge transfer, incentives, security, watermarking, and efficiency.These problems are organized into five aspects: Foundational Theory, Data, Heterogeneity, Security and Privacy, and Efficiency.
  • Foundational Theory: Foundational theory frames FedFMs as multi-objective optimization balancing privacy, utility, and efficiency while also considering personalization, model types, fairness, robustness, and aggregation.The framework treats these dimensions as interacting trade-offs rather than isolated objectives.
  • Data and Heterogeneity: Data and heterogeneity challenges arise because private data vary across clients and time, while large foundation models and small domain models require bidirectional knowledge transfer.Continual learning must integrate heterogeneous knowledge, avoid forgetting, and address conflicts across clients or time.
  • Security and Privacy: Security and privacy problems include privacy attacks, local data or model theft, and the need for model watermarking across the model life cycle.These concerns motivate defensive techniques and game-theoretic mechanisms for deterring attackers.
  • Efficiency: Efficiency remains a major concern because FedFMs require improvements in training, communication, and parameter efficiency under resource constraints.The paper reviews methods and challenges for reducing computational and communication costs in large-model federated systems.

B. Existing Methods

Existing FedFM methods address multi-objective trade-offs, privacy protection, aggregation, and system heterogeneity across open- and closed-source models. Pareto optimization frames balancing privacy, utility, and efficiency, while aggregation choices trade customization against accessibility.

  • Multi-objective trade-offs: Pareto optimization identifies solutions balancing privacy, utility, and efficiency without improving one objective at the expense of another.The privacy-utility trade-off is also linked to limits on simultaneously minimizing privacy leakage and utility loss.
  • Privacy protection: Privacy protection methods include homomorphic encryption and differential privacy for absolute protection, plus knowledge distillation for protecting selected information.
  • Efficiency and heterogeneity: Federated PEFT, key-parameter selection, client-subset selection, and global-local adapters target communication efficiency and heterogeneous personalization needs.
  • Aggregation: White-box FedFMs support parameter-based aggregation such as FedAvg and layer-specific customization, but require substantial expertise, computation, and infrastructure.
  • Aggregation: Black-box FedFMs use less customizable prompt-level aggregation through APIs, reducing operational overhead for organizations with limited resources.

C. Challenges and Potential Solutions

Private-data utilization in FedFMs must balance utility, privacy, data quality, efficiency, and personalization across isolated clients. Existing and proposed approaches span theoretical constraints, data and model adaptation, privacy defenses, and hybrid global-local learning.

  • Private-data formulation: Private-data utilization formulates global-model learning over isolated client datasets while constraining each client’s privacy leakage below a predefined threshold.The formulation uses an aggregation mechanism such as FedAvg and a client-specific data-utilization objective.
  • Existing methods: Data-centric methods improve distributed private-data utility, while model-centric methods adapt architectures and training through instruction tuning and model splitting.
  • Data quality: Non-IID private data creates imbalance and quality variation; synthetic data generation, scoring, and global thresholds are proposed to improve consistency.
  • Computational efficiency: Representative data selection and gradient-free black-box optimization reduce redundancy, communication rounds, or data requirements while preserving model quality or performance.
  • Privacy protection: Privacy defenses address training-data, parameter, and architecture leakage through targeted-attack protection and privacy-preserving applications in medical and campus settings.
  • Model adaptation: Global-local adapters, federated instruction tuning, and split learning support personalization, multimodal transfer, and task-specific processing under private-data and resource constraints.

C. Challenges and Potential Solutions

FedFMs face data-quality, computational, continual-learning, and spatial-temporal heterogeneity challenges. Proposed directions combine efficient data selection, adaptive split learning, contribution evaluation, continual updates, and knowledge-retention methods.

  • Challenges and potential solutions: Foundation models are more sensitive to private-data quality than traditional federated models, require large high-quality datasets, and cost substantially more to train.
  • Challenges and potential solutions: Proposed solutions combine hierarchical data selection, adaptive layer splitting, and unified quality metrics covering utility, relevance, diversity, and effects on existing knowledge.
  • Continual learning: Continual learning enables incremental updates without full retraining, but growing clients and samples can increase communication overhead and slow convergence.
  • Continual learning: Spatial-temporal catastrophic forgetting arises when clients learn new tasks over time while non-IID data causes global models to lose performance on local test sets.
  • Continual learning: Federated continual learning seeks to update local models, exchange knowledge across clients, and reuse fused knowledge for future learning while retaining old tasks.
  • Continual-learning methods: Existing retention approaches replay real or synthetic samples, regularize or decompose parameters, distill teacher knowledge, and transfer prompts across tasks and clients.

C. Challenges and Potential Solutions

FedFMs must support continual knowledge transfer and privacy-compliant unlearning while managing conflicts, malicious updates, and the high cost of removing private data. Proposed directions combine external knowledge, selective fusion, and active forgetting aligned with retraining.

  • Continual learning: Spatial-temporal heterogeneity can cause catastrophic forgetting, requiring knowledge transfer across clients and later knowledge to correct earlier errors.Knowledge conflicts also arise from complex real-world data.
  • Potential solutions: Potential solutions use external knowledge and selective knowledge fusion to enrich context, prevent conflicts, and identify malicious clients.These mechanisms are presented as ways to improve robustness and security during federated learning.
  • Machine unlearning: Federated unlearning is needed to satisfy data-removal requirements because trained models retain implicit information despite not directly accessing user data.The paper links this need to model inversion risks, unauthorized or biased content, and high retraining costs.
  • Machine unlearning: The unlearning pipeline identifies clients and records linked to deletion requests, then applies active forgetting while preserving model integrity.The stated objective is to remove specified data contributions without rebuilding the entire model from scratch.
  • Machine unlearning: Federated unlearning targets a subset of a client’s data and constructs an unlearned model that approximates a reference retrained model.The reference model is obtained using an aggregation mechanism, and model distance can be measured with Euclidean distance, cosine similarity, KL divergence, or Maximum Mean Discrepancy.

B. Existing Methods

Existing methods address unlearning through target selection, execution strategies, verification, and principles for matching a modified model to retraining. Related FedFM settings additionally require handling Non-IID and graph-structured data under privacy constraints.

  • Machine unlearning: Federated unlearning considers four aspects: targets, executors, verification, and principles.These aspects organize how data removal is requested, performed, evaluated, and aligned with retraining.
  • Machine unlearning: Requests may remove partial samples, features, classes, or entire clients, with client removal risking catastrophic forgetting because of large-scale data deletion.Partial-sample removal is fine-grained, while feature and class removal target specific semantic content.
  • Machine unlearning: Unlearning executors include server-side parameter adjustment and domain-model-based additional training, with full retraining remaining costly in federated settings.Recent approaches combine influence elimination with relearning and parameter adjustment.
  • Machine unlearning: Verification evaluates removal effectiveness, efficiency, and security using performance metrics, model discrepancy measures, and attack-based tests.Accuracy, loss, statistical errors, Euclidean distance, and KL divergence are among the stated evaluation tools.
  • Machine unlearning: FedFM unlearning is complicated by model complexity, knowledge coupling across modalities, and the need for cross-client synchronization and validation.These factors can threaten targeted updates, model integrity, and performance.
  • Machine unlearning: Modular unlearning, disentangled knowledge decoupling, and federated cross-client coordination are proposed to localize removal and maintain consistency.The proposed techniques include constrained fine-tuning, sensitivity analysis, cross-modal attention tracking, mutual-information minimization, gradient averaging, and replay buffers.
  • Non-IID and graph data: Non-IID data differs across clients in features, labels, and sample sizes, while concept drift and concept shift create additional dynamic distribution changes.The passages distinguish changes in P_k(x|y) from changes in P_k(y|x).
  • Non-IID and graph data: FedFMs with graphs must learn intra- and inter-graph relationships, whereas graphs of FedFMs represent clients as network nodes connected by relationships such as proximity or data similarity.The two perspectives are presented as distinct graph settings.

B. Existing Methods

Existing approaches mitigate heterogeneous FedFM environments through distillation, model pruning, distribution adaptation, topology-aware learning, and privacy-preserving fusion. Bidirectional transfer further organizes knowledge exchange from the server, from clients, or through joint co-optimization.

  • Non-IID and graph data: Knowledge distillation and model pruning can unify heterogeneous client models, while graph optimization can represent relationships through similarity, task relevance, or proximity.These strategies target optimization complexity and adaptability in Non-IID federated systems.
  • Non-IID and graph data: Distribution-adaptation methods use foundation-model priors, adapters, or dynamic weighting to improve performance across diverse client distributions.The cited examples include FedBaF, TriplePlay, and FedDPA.
  • Graph of models: Federated graph-of-models methods optimize client topology and neighbor information to support scalable learning across distributed graph-structured data.FedGTA is described as using topology-aware local smoothing and mixed neighbor features.
  • Graph of models: Graph-based model fusion must balance security, fairness, personalization, and privacy while adapting to dynamic environments.PrivFusion is cited as a privacy-preserving fusion architecture using graph structures and hybrid local differential privacy.
  • Graph of models: Heterogeneous data and dynamic topologies can produce slow or unstable convergence and complicate communication, aggregation, and information flow.Sustainable optimization also must balance utility, privacy, fairness, and personalization across nodes with different priorities.
  • Potential solutions: Proposed solutions combine personalized optimization, topology-aware aggregation, and multi-objective optimization with differential privacy, fair allocation, and reinforcement learning.These mechanisms target scalability, adaptability, and fairness in Non-IID and graph-based environments.
  • Bidirectional knowledge transfer: Bidirectional knowledge transfer exchanges knowledge between server FMs and client DMs in both directions while preserving privacy.The interaction is framed as teacher-student learning in which domain-specific client knowledge can enrich the server model.
  • Bidirectional knowledge transfer: The three transfer settings optimize client DMs from the server FM, the server FM from client knowledge, or both jointly.The formulations use local private data for clients and public server data where specified.

B. Existing Methods

Existing bidirectional-transfer methods operate at the data, representation, and model levels, while persistent heterogeneity and privacy challenges motivate unified architectures, adaptive transfer, synthetic data, and advanced privacy techniques. FedFM incentives additionally require accurate evaluation of client contributions.

  • Bidirectional knowledge transfer: Data-level transfer uses knowledge distillation and synthetic data to convey foundation-model capabilities to smaller models.Examples include stepwise Chain-of-Thought distillation, multitask explanation-based transfer, and adversarial distillation.
  • Bidirectional knowledge transfer: Representation-level methods use split learning, distillation, and aggregated representations to transfer knowledge while addressing privacy and model differences.SAP, FedGKT, MINILLM, and CreamFL illustrate these approaches.
  • Bidirectional knowledge transfer: Model-level methods use private-data-informed pretraining, synthetic-data initialization, adapters, and server aggregation to transfer knowledge efficiently.The cited approaches include SPT, Offsite-Tuning, and its federated extension.
  • Bidirectional knowledge transfer: Co-optimization methods enhance both FMs and DMs through selective distillation, token alignment, LoRA adapters, and mutual knowledge distillation.FedMKT, FedCoLLM, and CrossLM exchange knowledge without sharing private data.
  • Challenges and solutions: Bidirectional transfer remains challenged by data, representation, and model heterogeneity together with privacy concerns.These constraints motivate compatibility and privacy-preserving designs.
  • Challenges and solutions: Proposed responses include unified architectures, adaptive transfer, synthetic data, differential privacy, homomorphic encryption, and secure multi-party computation.The methods target compatibility, model-specific adaptation, data augmentation, and privacy-risk reduction.
  • Incentives and contribution evaluation: FedFM incentives seek to attract sustained contributions by measuring the importance of client datasets or computational resources and compensating high-quality participation.Contribution evaluation is motivated by foundation models’ dependence on substantial data and computation.
  • Incentives and contribution evaluation: Contribution evaluation formulates a mechanism C that estimates each client’s contribution from introducing private dataset D_k, with accuracy measured against an oracle contribution.The formulation includes a contribution-estimation loss for each client.

B. Existing Methods

Existing FedFM methods evaluate client contributions with multiple schemes and utility functions, while game mechanisms model participant actions, costs, privacy, reputation, and efficiency. These approaches seek fair contribution assessment and incentives for secure participation.

  • Contribution evaluation: Contribution evaluation begins by selecting an evaluation scheme and a utility function for assessing FedFM performance.
  • Contribution evaluation: Individual, interaction, leave-one-out, Shapley value, and least-core methods provide alternative contribution evaluation schemes.Individual and interaction methods have O(n) time complexity, whereas Shapley value and least core require O(2^n).
  • Utility functions: Task-specific utilities measure model performance or mutual cross entropy, while task-agnostic utilities use data size, diversity, model similarity, or mutual information.
  • Challenges: Existing contribution evaluation faces exponential computational overhead and difficulty mapping data contributions to task-specific value at billion-scale.Suggested directions include learned contribution evaluation and crowdsourced data-contribution maps.
  • Game mechanisms: Game-theoretic defenses raise technical and economic attack costs while aiming to make compliant participation and fair compensation rational choices.
  • Game mechanisms: Game mechanisms model participant actions and utilities while balancing rewards, action costs, reputation, privacy loss, and computational efficiency.The formulation includes reward for honest behavior and weighted reputation, privacy, and efficiency terms.

B. Existing Methods

Existing incentive mechanisms use game-theoretic structures to balance privacy, model efficacy, fairness, communication, accuracy, and resource costs. The literature includes leader–follower pricing, competition, contribution-based rewards, installment payments, and auctions.

  • Incentive mechanisms: Game-theoretic incentive methods seek equilibrium among sensitive-information protection, model efficacy, and cost-effectiveness through quantified payoff structures.
  • Stackelberg Game-based Methods: Stackelberg methods model servers as leaders and clients as sellers, optimizing utility across data, rewards, resources, fairness, communication, accuracy, and privacy.Reported limitations include uniform pricing and unrealistic assumptions.
  • Yardstick Competition-based Schemes: Yardstick competition rewards faster training and can reduce delays in synchronous federated learning, but it simplifies the factors affecting delay.
  • Shapley-Value Based Schemes: Shapley-value schemes allocate rewards according to contribution, although cited methods may ignore client input or remain limited to Horizontal FL.
  • Other Incentive Mechanisms: Other mechanisms use real-time installment payments or auction-based data trading to improve fairness, revenue, efficiency, and competition.

C. Challenges and Potential Solutions

FedFM security and privacy methods combine game-theoretic mechanisms with model watermarking. Watermarking supports ownership and traceability through white-box parameter verification or black-box trigger verification, but privacy-cost modeling remains difficult.

  • Security and privacy: Privacy-preserving mechanism design must account for encryption costs, breach losses, participant incentives, information asymmetry, and changing attack strategies.
  • Security and privacy: Potential solutions include adaptive reinforcement-learning games, resource-aware privacy mechanisms, incentive-compatible rewards and penalties, and AI-driven anomaly detection.
  • Model watermarking: FedFM watermarking embeds client and server identifiers into shared models to assert ownership, verify contributions, and support traceability against unauthorized use.
  • White-Box: White-box watermarking embeds binary strings in model parameters and verifies them using an extractor and a Hamming-distance threshold.
  • Black-Box: Black-box watermarking embeds trigger samples and confirms ownership when model outputs achieve trigger accuracy above a predefined threshold.
  • Watermark objectives: Client watermarking objectives combine the main task loss with black-box trigger and white-box feature watermark losses, weighted within the aggregation process.

C. Challenges and Potential Solutions

FedFM efficiency concerns computation, communication, and storage, and existing work organizes solutions around data, models, devices, and communication. Large-scale watermarking and efficiency optimization remain constrained by resource, model-size, and network demands.

  • Watermarking challenges: Large-scale watermarking is constrained by client computation, massive-model communication costs, compression or quantization distortion, modality diversity, and dynamic participation.
  • Watermarking solutions: Proposed watermarking solutions include lightweight embedding, federated pruning, resource-efficient designs, modality-agnostic representations, and adaptive self-reinforcing signatures.
  • Efficiency definition: FedFM efficiency is defined across computation, communication, and storage, with the objective of minimizing their weighted sum for resource-constrained deployment.
  • Efficiency methods: Existing efficiency methods are categorized as data-oriented, model-oriented, device-oriented, and communication-oriented approaches.
  • Data-oriented methods: Data-oriented methods improve efficiency through data selection that reduces training FLOPs and prompt compression that condenses prompts or learns compact representations.

2) Model-oriented Methods

FedFMs efficiency is addressed through model, device, and communication strategies while balancing utility, privacy, and computational costs. The paper unifies these competing objectives mathematically across ten challenges.

  • Model-oriented Methods: Quantization and pruning reduce foundation-model size and computation, with lower-precision formats such as 8-bit or 4-bit potentially minimizing payload with minimal accuracy loss.Pruning removes redundant or less important model weights.
  • Device-oriented Methods: Resource-aware scheduling and lightweight frameworks tailor FedFMs deployment to the capabilities and constraints of client devices.These device-oriented methods optimize local training and communication efficiency.
  • Communication-oriented Methods: Hierarchical aggregation, clustering, and decentralized device-to-device communication reduce parameter transmission and communication costs during federated learning.RFL-HA organizes edge nodes into clusters, while decentralized approaches remove the need for a central aggregator.
  • Challenges: FedFMs face synchronization bottlenecks, latency, repetitive computation, and an efficiency-performance-privacy trade-off involving techniques such as homomorphic encryption.These constraints make simultaneous optimization of efficiency, privacy, and model performance difficult.
  • Potential Solutions: Potential solutions include adaptive aggregation, efficient compression, hybrid privacy-preserving techniques, and task disentanglement across clients.The proposed directions target communication overhead, privacy protection, and redundant computation.
  • Unified Objective: A unified objective combines utility, efficiency, watermark, contribution, and privacy losses under a no-free-lunch constraint, with coefficients selecting different optimization priorities.The coefficients α1 through α5 lie in [0, 1] and sum to 1; the formulation is presented as a mathematical perspective on the ten problems.
Loading 2502.12176v1…