PaperlayerView exact revision on arXiv

Source-linked AI summary

Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions

Qinglin Yang, Chen Qiu, Hongyuan Zhang, Pengdeng Li, Yuan Liu, Zhihong Tian

arXiv:2608.13844v1cs.LGcs.AIcs.DC

TL;DR

Federated prompt learning addresses the privacy, computational, and deployment challenges of adapting large language models without centralizing raw data. This paper surveys FPL across the model lifecycle, system perspectives, applications, and security, and concludes that lightweight prompts and adapters reduce federated costs while methods retain important differences and unresolved risks.

  • Problem

    LLM training and deployment are hindered by high computational costs, data centralization, and privacy concerns, motivating federated prompt learning as a distinct research focus.

  • Method

    The paper presents a comprehensive, lifecycle-oriented survey of federated prompt learning spanning adaptation, personalization, inference, deployment, applications, and security.

  • Results

    Lightweight prompts and adapters can substantially reduce federated LLM communication, computation, and storage costs while keeping raw data local.

  • Takeaways & Limitations

    Federated PEFT should be evaluated across multiple objectives because approaches differ in non-IID handling, resource heterogeneity, personalization, scalability, and privacy.

  • Takeaways & Limitations

    Existing empirical evidence lacks a principled theory for some reported properties of federated prompt and PEFT optimization.

Abstract

from arXiv · show

Large language models (LLMs) have become core components of cloud-based intelligent services in academia and industry, yet their training and deployment are hindered by high computational costs, data centralization, and privacy concerns. Federated learning (FL) offers a decentralized training paradigm that enables clients to collaboratively train a learning model without sharing raw data, making it a promising solution for privacy-preserving LLM training and reasoning. This paper presents a comprehensive survey of federated prompt learning (FPL) to review recent advances in integrating the federated learning paradigm and large language models, answering the following research questions: RQ1: The fundamental motivations, characteristics, and enabling technologies of FPL, and how it differs from conventional FL and full-model federated fine-tuning; RQ2: The trade-offs FPL approaches exhibit in performance, communication efficiency, computational overhead, scalability, personalization, and heterogeneity handling; RQ3: The remaining security, privacy, robustness, and system challenges, along with key future research directions. To this end, we systematically examine existing FPL methods across the full model lifecycle: pre-training, fine-tuning, and practical applications, while discussing security, privacy, and robustness issues and summarizing existing defense mechanisms. Finally, we highlight open challenges and future directions, aiming to help readers understand how the insights drive research in FPL.

1 Introduction

Federated prompt learning addresses the privacy, scalability, and resource challenges of adapting large language models by federating lightweight prompt or adapter modules over a shared frozen foundation model. This survey unifies fragmented research across the LLM lifecycle, system deployment, applications, security, and future directions.

  • Motivations: Centralized LLM training faces privacy, regulatory, computational, and communication constraints because valuable data cannot always be aggregated.Examples include enterprise, medical, financial, legal, personal-device, and cross-institutional business data.
  • Motivations: LLM scale makes transmitting and updating only critical, minimal, and effective parameters a central federated-learning challenge.Full-model transmission creates extremely high communication costs and storage pressure.
  • Federated Prompt Learning: Prompt learning and PEFT freeze the pretrained backbone while optimizing small task-specific modules such as soft prompts, prefixes, or LoRA adapters.This reduces training cost while preserving the generalization power of large models.
  • Federated Prompt Learning: FPL uses lightweight prompt or adapter modules as the interface between LLMs and federated optimization, rather than federating the entire model.The design reduces communication costs, mitigates client-side resource constraints, and improves robustness to non-IID data while maintaining strong privacy guarantees.
  • Survey Scope: The survey addresses fragmented prior literature through a lifecycle-oriented framework covering adaptation, personalization, inference, deployment, applications, and protection.It compares algorithmic and cloud-system perspectives across performance, efficiency, scalability, personalization, privacy, robustness, and deployability.
  • Contributions: The paper synthesizes empirical findings, security and privacy defenses, and open challenges to guide subsequent research in federated prompt learning.Its stated goal is to provide a structured reference for scalable, efficient, and privacy-preserving collaboration with foundation models.

2 Related Work and Background

Prior work connects federated learning and large language models to data scarcity, privacy, scalability, and cost challenges, while this survey centers federated prompt learning as a distinct interface. It organizes FPL methods across the LLM lifecycle and examines algorithmic, system, application, security, and privacy dimensions.

  • Existing FedLLM Surveys: Earlier FedLLM surveys address data scarcity, privacy, computational cost, communication cost, and decentralized model training across the model lifecycle.They review pre-training, fine-tuning, deployment, parameter-efficient methods, prompt-based learning, split learning, personalization, and backpropagation-free techniques.
  • Existing FedLLM Surveys: Existing reviews often provide macro-level FL–LLM taxonomies or focus on security, IoT, applications, or federated fine-tuning rather than prompt- and adapter-based optimization.The cited surveys differ in emphasis, including architectural synergies, security taxonomies, application-driven analysis, and comparisons among FedLLMs, KD-FedLLMs, and Split-FedLLMs.
  • Survey Scope and Contribution: This survey places FPL at the center and systematically reviews prompt- and adapter-based collaboration for efficient, scalable, and privacy-preserving LLM adaptation.Its framework spans the full LLM lifecycle while jointly examining algorithmic, system-level, application-driven, and security-related dimensions.
  • FPL Background: FPL communicates compact prompt or adapter parameters instead of full model parameters, reducing communication and client-side optimization costs.The frozen backbone preserves transferable knowledge and reduces catastrophic forgetting during local adaptation.
  • FPL Background: FPL supports personalization through shared and local prompt components and avoids destructive weight averaging under heterogeneous data.Its modular design also reduces requirements to store or transmit full gradients and optimizer states and supports multiple tasks or domains without separate full-model copies.
  • Security and Privacy: Lightweight parameter exchange does not remove privacy and security risks, including gradient inversion, membership inference, representation reconstruction, poisoning, and backdoors.The survey also discusses formal defenses and their limitations, including attacks on prompt or adapter updates.
  • FPL Background: FPL reduces communication and computation by orders of magnitude compared with conventional FL while retaining broader pretrained knowledge.The survey identifies substantially fewer parameters and communication rounds than FedAvg in scarce-data scenarios.

3 Federated Learning for Large Language Models

Federated learning for LLMs applies decentralized collaboration to training, adaptation, and deployment on distributed private data, spanning the LLM lifecycle. Recent work targets parameter efficiency, communication, data quality, heterogeneity, personalization, and practical edge deployment.

  • Federated learning supports LLM training, adaptation, and deployment on distributed private data while keeping client data local.
  • Federated LLM research spans pretraining, fine-tuning, instruction tuning, alignment, prompt learning, model compression, and continual learning.
  • Reported results include up to 60.8% improvement over baselines, reduced communication and resource demands, and up to 60% lower memory-access and bandwidth usage.
  • FLoSS applies unstructured top-k sparsity only during communication phases while preserving dense local optimization for federated LoRA fine-tuning.
  • FedHDS uses hierarchical selection, cross-layer feature fusion, and density-based clustering for data-efficient instruction tuning under resource constraints and non-IID data.
  • FedDQC performs client-side data quality control and uses instruction-response alignment to progressively train from higher-quality to noisier samples.
  • Federated prompt and adapter methods address heterogeneity through generalized and personalized adaptation, adaptive LoRA allocation, influential-layer selection, and split learning.
  • Empirical and theoretical studies also examine generalization, personalization, convergence, IoT efficiency, and deployment, with higher local ranks and larger client populations linked to improved generalization bounds.

Trajectory

The trajectory emphasizes parameter-efficient approaches.

  • Parameter-efficient methods are identified as part of the research trajectory.
  • The supplied trajectory fragment does not specify a method, comparison, or result beyond parameter efficiency.

Driven Integrated

Large language model-enhanced federated learning broadens the design space beyond direct model federation toward reusable knowledge, lightweight adaptation, and joint optimization across system factors.

  • Research has shifted from reducing full-model federated optimization costs toward deciding what to train, communicate, and share.
  • Federated LLM optimization increasingly jointly considers model parameters, data, communication, computation, and privacy.
  • In LLM-enhanced FL, LLMs provide reusable knowledge priors and can assist lightweight collaborative adaptation rather than being directly federated.
  • LLMs can support data generation, annotation, pseudo-labeling, knowledge transfer, client selection, aggregation design, log analysis, anomaly detection, and privacy-risk interpretation.
  • These roles are reported to improve federated learning’s optimization efficiency, manageability, and intelligence level.
  • The optimization target shifts toward prompts, adapters, and low-rank modules, while clients may not optimize the same parameter space or possess the full foundation model.

Research

The supplied passages identify communication compression through gradient sparsification and quantization as a research direction.

  • FLM-TopK further compresses parameter-efficient fine-tuning updates through joint gradient sparsification.
  • The supplied passages do not report a quantitative comparison or downstream performance result for this direction.
  • The method addresses communication after trainable updates through compression and quantization.

Lightweight

The section points toward reducing the parameter space and transmitted information through sparsification.

  • The lightweight strategy focuses on limiting the communicated update representation.
  • Sparsification reduces the amount of transmitted information.

Adaptation

Federated adaptation methods freeze large backbones and train prompts, adapters, or other small modules, reducing communication and computation while retaining competitive performance across heterogeneous settings.

  • Lightweight adaptation: Prompt- and adapter-based methods reduce communication from full-model scale to module scale.Clients exchange only lightweight adaptation parameters while retaining most backbone parameters locally.
  • Prompt tuning: FedPrompt freezes the PLM backbone and trains soft prompts, reducing communication to approximately 0.01% of the original model size.It supports prompt tuning, P-tuning, and prefix-tuning with minimal accuracy degradation.
  • Federated LLM systems: FS-LLM provides benchmarking, PEFT methods, offsite tuning, and resource-efficient operators for federated LLM fine-tuning.The framework achieves orders-of-magnitude communication reductions while maintaining competitive performance.
  • Off-site and split adaptation: FedBiOT combines server-side emulator distillation with locally fine-tuned LoRA adapters to avoid full-model exposure.Across LLaMA-2 tasks, it outperforms offsite-tuning and FedOT while cutting client computation by up to 50%.
  • Application-oriented adaptation: FIT fine-tunes lightweight adapter parameters of a frozen multimodal LLM while preserving sensitive local data.Its feature-diversity strategy augments instruction data across time, weather, and occlusion, improving reported driving-task outcomes.

5 The Synergistic Integration of FL and LLMs

FPL extends federation from shared parameters toward personalized prompts, semantic knowledge, and black-box optimization, creating systematic trade-offs across efficiency, accuracy, personalization, and heterogeneity handling.

  • Research trajectory: FPL relaxes traditional assumptions about identical models, shared tasks, and white-box foundation-model access.Collaboration can occur through adaptation modules, semantic concepts, multimodal knowledge, or prompt-evaluation information.
  • Research trajectory: FL–LLM collaboration evolves from parameter-level federation toward knowledge-level federation.Federated objects include personalized prompts, prompt mixtures, cross-task adapters, semantic representations, and optimization statistics.
  • Personalization: Personalized prompt methods improve adaptability through local prompts, mixtures, gating, and transfer-aware aggregation.Richer personalization mechanisms increase communication, storage, and computation as expert counts grow.
  • Heterogeneity handling: Concept-level alignment addresses the problem that position-wise averaging can combine semantically unrelated prompts.PFPT therefore aligns prompts by concepts rather than prompt positions.
  • Heterogeneity handling: FPL remains stable across several configurations, but aggregation preferences depend on heterogeneity type.Equal aggregation performs slightly better under label skew, whereas sample-weighted aggregation is preferable under domain shift.
  • Trade-offs: Unified experiments find no single FPL method dominates every dimension: LoRA is accurate and stable but costly, while prompt tuning minimizes communication with more variable accuracy.P-tuning’s larger parameter count does not ensure greater stability, indicating that parameterization structure matters.

6 Security and Privacy Issues in FPL

FPL introduces distinct security and privacy risks, including prompt backdoors, leakage, and safety degradation, while defenses combine privacy mechanisms, encryption, split architectures, pruning, and post-hoc recovery.

  • Backdoor defense: Post-hoc server-side defense decouples safety recovery from client training by generating aligned and normal instruction–response data.The defense restores safety performance by up to 69% without significantly degrading utility.
  • Secure architectures: FL-GLM combines split learning, public-key encryption, and parallelism to protect intermediate representations and improve efficiency.Client-batch and server-hierarchical parallelism cuts training time by over 48% versus serial execution.
  • Limitations: Security evaluations remain bounded by assumptions and limited settings, while stronger privacy constraints reduce predictive accuracy.Examples include few benchmark datasets, limited clients, full participation, one local step, semi-honest threats, or model-scale restrictions.
  • Trusted execution and encryption: Model-slicing defenses use TEEs, lightweight encryption, and selective parameter updates to address leakage without relying solely on differential privacy.The schemes protect fine-tuned components and intermediate activations across client and server execution environments.
  • Encrypted aggregation: FedShield-LLM combines LoRA, pruning, and fully homomorphic encryption so servers aggregate encrypted updates without plaintext access.It provides convergence and security guarantees and outperforms Vanilla FL and DP-LoRA in reported benchmarks.
  • Threats: Communication-efficient prompt updates remain vulnerable to targeted backdoor attacks despite a frozen vision-language backbone.Reported attacks can reduce safety metrics by up to 70%, while classical robust aggregation provides only marginal protection.

7 Experimental Benchmarking and Empirical Insights

The unified benchmark addresses unreliable cross-study comparisons by evaluating representative federated PEFT methods under a consistent protocol. Results show a multi-objective trade-off: LoRA is more stable, while prompt-based methods reduce communication but can be sensitive to heterogeneous optimization.

  • Benchmark Design: The benchmark uses a consistent protocol to compare representative federated PEFT strategies across methods originally evaluated in heterogeneous settings.It uses four clients, 100 communication rounds, a common held-out test set, and mean ± standard deviation over three random seeds.
  • Efficiency and Stability: Prompt tuning achieves the lowest trainable parameter ratio and smallest communication volume, but its performance is more sensitive to random seeds and data partitions.The benchmark therefore identifies a communication-efficiency advantage alongside weaker optimization stability than LoRA.
  • Efficiency and Stability: P-tuning does not consistently outperform prompt tuning despite using substantially more trainable parameters and shows large variance under LDA α=0.1 and α=0.5.Increasing prompt-encoder parameter scale does not automatically produce more stable federated adaptation.
  • Empirical Implications: Federated PEFT methods should be evaluated jointly by accuracy, stability, trainable scale, communication cost, and robustness under heterogeneity.Comparable accuracy can conceal sharp differences in communication cost and seed-to-seed variance.

8 Insights and Future Research

The survey finds that federated prompt learning remains constrained by limited realism, incomplete theory, and unresolved trustworthiness issues. It proposes standardized evaluation, realistic-scale optimization, stronger privacy and safety protections, trajectory-level verification, and real-world validation.

  • Current Limitations: Surveyed studies commonly rely on simulated heterogeneity, small or moderate client populations, single backbones, informal privacy claims, and fragmented protocols.These limitations indicate that integration of federated learning and LLMs remains at an early stage of maturity.
  • Evaluation: Shared benchmark suites with multi-seed reporting and standardized non-IID generators are needed to treat communication, robustness, and parameter footprint as co-equal objectives.Published results often omit these dimensions jointly because they use disparate backbones, partitions, and protocols.
  • Theory: A principled theory connecting prompt rank, capacity, and domain shift to generalization under non-IID nonlinear transformer dynamics is still missing.Existing convergence guarantees remain largely confined to simplified regimes, while several reported phenomena are supported only empirically.
  • Systems and Scale: Resource- and heterogeneity-aware methods require validation on thousands of genuinely heterogeneous clients, larger LLM backbones, and authentic edge testbeds.Current mechanisms are primarily assessed under simulated conditions, moderate model sizes, and limited client counts.
  • Future Directions: Future research should combine adaptive federated control, formal privacy analysis, safety auditing, incentive design, energy-aware scheduling, and real-world case studies.The survey frames these directions as necessary for translating methodological progress into deployable, accountable federated LLM systems.
  • Trustworthiness: Trustworthy federated agentic systems require trajectory-level verification, leakage protection beyond training, continuous safety auditing, and accountability mechanisms.Agentic deployments introduce risks through tool calls, retrieved documents, persistent memory, and inter-agent messages that can affect downstream reasoning.

9 Conclusion

The survey concludes that prompts and lightweight adapters reduce federated LLM adaptation costs while preserving local data, but methods differ substantially in robustness, scalability, personalization, and privacy. Its benchmark supports treating federated PEFT as a multi-objective problem and motivates resource-aware optimization, standardized evaluation, and formal protection.

  • Conclusion: Prompts and lightweight adapters can reduce communication, computation, and storage costs of federated LLM fine-tuning while keeping raw data local.
  • Conclusion: Federated prompt learning methods differ considerably in handling non-IID data, resource heterogeneity, personalization, scalability, and privacy.
  • Conclusion: LoRA generally provides more stable performance, whereas prompt-based methods reduce communication costs but may be more sensitive to heterogeneous optimization.
  • Conclusion: Federated PEFT should be evaluated as a multi-objective problem rather than by predictive accuracy alone.
  • Conclusion: Future work should pursue resource-aware optimization, stronger personalization and generalization, standardized evaluation, and formal defenses against privacy and integrity attacks.
Loading 2608.13844v1…