Source-linked AI summary
PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Model
Tao Guo, Song Guo, Junxiao Wang, Wenchao Xu
TL;DR
Federated learning can be hindered by limited bandwidth and insufficient local data, motivating a more efficient way to adapt shared models. PromptFL instead distributes an off-the-shelf CLIP and cooperatively trains shared soft prompts. The paper reports feasibility, privacy preservation, and competitive performance against strong FL baselines.
Problem
Limited bandwidth and insufficient user data can make federated aggregation inefficient, training slow, and model accuracy reduced.
Method
PromptFL ships an off-the-shelf CLIP to clients, which collaboratively train shared continuous soft prompts instead of a shared model.
Results
PromptFL demonstrates system feasibility, keeps data private through gradient communication, and is competitive with strong FL baselines across image classification tasks.
Takeaways & Limitations
PromptFL uses foundation-model adaptation to reduce aggregation overhead and exploit very few local data samples in distributed tasks.
Takeaways & Limitations
PromptFL’s inference cost on modern cell phones is identified as a key bottleneck, with potential mitigation from offloading, increased RAM, tiny CLIPs, and quantization.
Abstract
from arXiv · showhide
Quick global aggregation of effective distributed parameters is crucial to federated learning (FL), which requires adequate bandwidth for parameters communication and sufficient user data for local training. Otherwise, FL may cost excessive training time for convergence and produce inaccurate models. In this paper, we propose a brand-new FL framework, PromptFL, that replaces the federated model training with the federated prompt training, i.e., let federated participants train prompts instead of a shared model, to simultaneously achieve the efficient global aggregation and local training on insufficient data by exploiting the power of foundation models (FM) in a distributed way. PromptFL ships an off-the-shelf FM, i.e., CLIP, to distributed clients who would cooperatively train shared soft prompts based on very few local data. Since PromptFL only needs to update the prompts instead of the whole model, both the local training and the global aggregation can be significantly accelerated. And FM trained over large scale data can provide strong adaptation capability to distributed users tasks with the trained soft prompts. We empirically analyze the PromptFL via extensive experiments, and show its superiority in terms of system feasibility, user privacy, and performance.
Introduction
Edge users generate valuable distributed data, but limited bandwidth and insufficient local data make conventional federated training slow, communication-intensive, and less accurate. PromptFL addresses these constraints by having participants cooperatively learn prompts with a foundation model instead of training a shared model.
- Traditional centralized AI gathers raw edge data in the cloud, creating communication overhead and potential privacy leakage.
- Federated learning keeps training data local while coordinating local training and global aggregation across edge users.
- Limited bandwidth and insufficient edge data can cause inefficient aggregation, excessive training time, and reduced model accuracy.
- PromptFL replaces federated model training with cooperative prompt learning using foundation models, accelerating local training and global aggregation.
- PromptFL is evaluated for system feasibility, performance against strong FL baselines, and privacy preservation across image classification tasks.
Preliminaries
Foundation models provide broad pretrained capabilities that can be adapted to downstream tasks, while federated learning coordinates local updates without exchanging client data. The preliminaries describe CLIP-based classification and the iterative client-selection, local-training, and aggregation process.
- Foundation Model: Foundation models are large models trained on broad data that can adapt to a wide range of downstream tasks.
- CLIP: CLIP is trained on hundreds of millions of image-caption pairs and aligns visual and textual representations for semantic classification.
- Image Classification with CLIP: CLIP-based image classification converts class labels into textual captions and selects the caption with the strongest image pairing.
- Federated Learning: Federated learning addresses users’ limited labeled data by learning across multiple privacy spheres while retaining data locally.
- Federated Learning: Each round selects eligible clients, performs local updates using downloaded global parameters, and aggregates the resulting updates at the server.
- Federated Learning: FedSGD shares gradients, whereas FedAvg performs several local epochs before sharing updated models that practically convey equivalent update information.
Prompt-Based Federated Learning
PromptFL adapts an out-of-the-box CLIP model for federated learning by training and aggregating only small continuous prompt parameters. This design targets limited local data and communication while retaining compatibility with multiple vision tasks and privacy-preserving updates.
- Prompt Engineering: PROMPTFL uses prompt engineering to adapt CLIP, choosing continuous soft prompts over discrete, human-interpretable text templates.Continuous prompts operate directly in the model’s embedding space and can be tuned from user training data.
- Framework to Learn Prompts in FL: Each client combines a prompt learner with a frozen, out-of-the-box CLIP backbone, while the server aggregates only prompt-learner updates.The CLIP image and text encoders remain frozen during local training; only prompt parameters are updated and transmitted.
- Framework to Learn Prompts in FL: PROMPTFL changes federated learning from model training to prompt-learner training using p learnable d-dimensional vectors prepended to class labels.The prompt vectors are continuous embeddings whose dimension matches the text encoder’s word-embedding dimension, 512 by default.
- Framework to Learn Prompts in FL: CLIP computes class probabilities by comparing image embeddings with text embeddings generated from prompt vectors and class names.The image and text encoders produce aligned representations, and cosine similarity supplies the classification logits.
- Framework to Learn Prompts in FL: Prompt learning is designed to converge quickly and require less data per user, while being less affected by non-IID data, limited bandwidth, and unreliable devices.The paper presents prompting as particularly useful for federated learning under these adverse conditions.
- System Feasibility: 600MB takes 1.4 minutes to transfer with PROMPTFL, compared with 9 hours for FL transferring 40GB under the stated mobile-network assumptions.The comparison assumes 100 communication rounds, while PROMPTFL downloads the backbone locally and repeatedly communicates only the small updates.
- System Feasibility: Modern cell-phone RAM is the key feasibility bottleneck, despite acceptable computational and storage costs.The paper points to offloading, increasing RAM, tiny CLIPs, and quantization as developments that may reduce this constraint.
- Privacy Concerns: PROMPTFL is compatible with object detection, video understanding, and visual question answering, and uploads prompt updates rather than raw images.Prompt parameters describe class–text correlations and are static across training inputs, supporting the paper’s privacy rationale.
Experiments
The experiments evaluate PROMPTFL across image-classification tasks, data distributions, client counts, shots, and system costs. PROMPTFL maintains competitive performance while substantially reducing communication and computation relative to federated baselines.
- Experimental Setup: PROMPTFL is evaluated on Caltech101, Flowers102, OxfordPets, and Food101 using ResNet50 and ViT-B/16 backbones.The experiments compare PROMPTFL with training from scratch and fine-tuning pretrained models.
- Impact of Shots and Clients: Increasing the number of shots improves overall performance, while different client counts reach similar optima once each client has sufficient classes.With 16 shots on Caltech101, all tested client-count settings achieve around 89%.
- Overall Performance: PROMPTFL achieves similar or better accuracy and F1 with only 0.01%–0.1% learnable parameters in the IID setting, while remaining competitive in extreme non-IID settings.The reported comparison covers both performance and efficiency against existing frameworks.
- Data Distribution Analysis: With sufficient classes per client, class distribution has little impact on PROMPTFL performance, with only a small improvement at 50% class overlap.Existing frameworks show poor stability under shifted class distributions beyond the unified setting.
- Computation and Communication Cost: PROMPTFL reduces uploaded communication cost by up to 110 times and uses about half as many rounds to converge compared with fine-tuning FL.The comparison measures uploaded data per round and total transmitted rounds.
- Computation and Communication Cost: PROMPTFL requires roughly three times less training GPU time than existing frameworks and consumes substantially less GPU memory.GPU time is measured for the same training steps, while memory usage reflects system burden during training.
Conclusion
The paper demonstrates PROMPTFL’s feasibility for federated image classification, privacy-preserving gradient-based prompt updates, and competitive performance against strong FL baselines.
- PROMPTFL is feasible on modern hardware across communication, training, and inference overhead dimensions.
- PROMPTFL keeps data on each device private by communicating prompt gradients rather than raw data.
- PROMPTFL is competitive with strong federated-learning baselines across popular image-classification tasks.