PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Prompt Based Personalized Federated Learning

Guanghao Li, Wansen Wu, Yan Sun, Li Shen, Baoyuan Wu, Dacheng Tao

arXiv:2303.08678v1cs.LGcs.CVcs.DC

TL;DR

PFL must combine global knowledge with client-specific information, while model-centric approaches often leave client data characteristics unused. pFedPT adds locally generated visual prompts to images so the aggregated backbone can learn client-distribution information; it improves test accuracy over SOTA PFL baselines on CIFAR10 and CIFAR100.

  • Problem

    PFL must extract useful global features from heterogeneous clients and combine them with personalized features, while existing model-level approaches leave data-level personalization underused.

    Source: §1 · p. 1

  • Method

    pFedPT generates client-specific visual prompts from local data, adds them to images, and trains an aggregating backbone to use the prompts' distribution information.

    Source: §1 · p. 2

  • Results

    pFedPT improves test accuracy by 1%–3% over SOTA PFL baselines on CIFAR10 and CIFAR100.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The visual prompt can also be added to existing FL methods to personalize client models or boost PFL performance.

    Source: §1 · p. 3

Abstract

from arXiv · show

As a popular paradigm of distributed learning, personalized federated learning (PFL) allows personalized models to improve generalization ability and robustness by utilizing knowledge from all distributed clients. Most existing PFL algorithms tackle personalization in a model-centric way, such as personalized layer partition, model regularization, and model interpolation, which all fail to take into account the data characteristics of distributed clients. In this paper, we propose a novel PFL framework for image classification tasks, dubbed pFedPT, that leverages personalized visual prompts to implicitly represent local data distribution information of clients and provides that information to the aggregation model to help with classification tasks. Specifically, in each round of pFedPT training, each client generates a local personalized prompt related to local data distribution. Then, the local model is trained on the input composed of raw data and a visual prompt to learn the distribution information contained in the prompt. During model testing, the aggregated model obtains prior knowledge of the data distributions based on the prompts, which can be seen as an adaptive fine-tuning of the aggregation model to improve model performances on different clients. Furthermore, the visual prompt can be added as an orthogonal method to implement personalization on the client for existing FL methods to boost their performance. Experiments on the CIFAR10 and CIFAR100 datasets show that pFedPT outperforms several state-of-the-art (SOTA) PFL algorithms by a large margin in various settings.

1. Introduction

Personalized federated learning seeks to address client heterogeneity, but existing model-level approaches may struggle with highly heterogeneous data and leave data-level personalization unused. pFedPT uses locally generated visual prompts to represent client data distributions, and experiments report improved accuracy on CIFAR10 and CIFAR100.

1. Introduction

  • Existing PFL methods can struggle with highly heterogeneous data and typically personalize at the model level, leaving data-level adaptation underused.The introduction frames PFL's challenges as extracting useful global features from heterogeneous clients and combining them with personalized features.Source: §1 · p. 1
  • pFedPT generates client-specific visual prompts from local data and adds them to images so the aggregating backbone can learn information about client distributions.The prompts also increase inter-class variation and decrease intra-class variation in the illustrated client representation.Source: §1 · p. 2, Figure 1 · p. 2
  • pFedPT improves test accuracy by 1%–3% over SOTA PFL baselines on CIFAR10 and CIFAR100.Experiments also indicate that prompts learn personalized knowledge related to each client.Source: §1 · pp. 2–3
  • The paper presents pFedPT as the first approach to use client-specific prompts to improve local adaptation and generalization of aggregated models.It proposes the framework for federated image classification and describes prompts as a way to fine-tune the aggregated model with local-distribution information.Source: §1 · p. 3
  • Visual prompts can also be added to existing FL methods to personalize client models or boost PFL performance.The paper reports integration with several existing FL and decoupled PFL methods.Source: §1 · pp. 2–3

2. Related Work

Prior PFL approaches share or cluster model components, use related learning techniques, or adapt aggregated models, but the reviewed methods focus on model-level personalization. The paper distinguishes pFedPT by incorporating client characteristics through visual prompts at the data level.

2. Related Work

  • Decoupled PFL shares feature-extraction layers while keeping personalized layers local; clustered FL groups similar clients and trains a model for each group.These approaches seek to transfer common knowledge while retaining client- or group-specific adaptation.Source: §2 · p. 3
  • Other PFL work applies clustering, multitask learning, transfer learning, regularization, or meta-learning to improve client models.Fine-tuning a FedAvg model is also cited as a way to improve personalized local models.Source: §2 · p. 3
  • The paper characterizes prior approaches as model-focused and pFedPT as using visual prompts to represent client data distributions during training.The prompts incorporate client characteristics at the data level, a dimension the reviewed model-level methods leave unused.Source: §2 · p. 3
  • PROMPTFL and FedPrompt fine-tune pretrained models and share prompt information, whereas pFedPT trains from scratch with alternating backbone and prompt-generator updates and client-specific prompts.The related-work comparison also notes that PROMPTFL and FedPrompt evaluate on NLP datasets, while pFedPT focuses on computer vision.Source: §2 · p. 3

3. Methodology

pFedPT trains client-specific visual prompts alongside an aggregated backbone, using prompts to represent local data distributions and guide client-level prediction. Its workflow alternates prompt-generator and backbone training, then aggregates and redistributes backbones across clients.

3.1. Problem setup

  • pFedPT models personalized federated learning as client-specific models, rather than the single aggregated model targeted by traditional federated learning.The clients' personalized models are denoted w_i.Source: §3.1 · p. 4

3.2. Workflow of pFedPT

  • Each client adds its generated prompt to input images and alternates prompt-generator and backbone updates as aggregated backbones are exchanged through the server.The prompt generator is updated while the backbone is frozen, then frozen while the backbone is trained; clients send their trained backbones for aggregation and receive the aggregated backbone.Source: §3.2 · p. 4
  • Padding prompts of size p = 4 perform best among the tested templates and sizes on CIFAR10 with a Dirichlet (0.3) partition, so the paper uses this setting by default.The alternatives include random-location patches, fixed-location patches, and padding.Source: §3.2 · p. 5

3.3. Modeling for pFedPT

  • The shared backbone extracts common client knowledge, while each locally trained prompt encodes client-specific information and guides the backbone's adaptation to that client's data distribution.The prompt is added to the input data rather than incorporated as a personalized inference-model layer.Source: §3.3 · p. 5

3.4. Optimization for pFedPT

  • Training alternates between locally updating the prompt generator with the aggregation model fixed and training the backbone with the prompt generator frozen; locally trained backbones are then aggregated.The paper states that the backbone can learn the client data distribution corresponding to its prompt, with prompt knowledge passed between clients through server aggregation.Source: §3.4 · p. 5
  • Unlike FedRep, pFedPT keeps personalization at the client data level through a prompt generator, leaving the backbone's inference process unchanged.The paper describes the visual prompt as orthogonal to FedRep-type methods and says the approaches can be combined to further boost performance.Source: §3.4 · p. 6

4. Experiments

Experiments evaluate pFedPT on CIFAR10 and CIFAR100 across non-IID settings, CNN and ViT backbones, and comparisons with existing methods. The reported results favor pFedPT overall, while exploratory studies examine prompt behavior and design.

4.1. Experimental Setup

  • Experiments use CIFAR10 and CIFAR100, Dirichlet and pathological non-IID partitions, and CNN and ViT models; clients participate probabilistically in communication rounds.The setup specifies 50 clients, a 20% participation chance per round, and padding prompts of size 4.Source: §4.1 · p. 6

4.2. Main Results

  • 80.83% test accuracy for pFedPT versus 61.92% for FedAvg and 77.98% for FedPer on Dirichlet CIFAR10 with CNN; pFedPT also leads on CIFAR10 and generally on CIFAR100.The comparison covers image datasets and different non-IID settings; CIFAR100 results are comparable in the Dirichlet setting.Source: §4.2 · pp. 6–7, Table 1 · p. 7
  • pFedPT performs with both ViT and CNN backbones, and the paper reports stronger performance as tasks become more difficult, while personalization methods degrade in IID settings.The authors attribute IID degradation to limited use of client personalization; pFedPT outputs become similar across clients and degenerate toward FedAvg.Source: §4.2 · p. 7
  • Adding prompts raises Dirichlet CIFAR10 FedProx accuracy from 61.95% to 80.47% and MOON accuracy from 62.53% to 77.84%.The reported prompt benefit also extends to decoupled methods FedRep and FedPer; Table 2 covers prompted baselines using CNN on non-IID image datasets.Source: §4.2 · p. 7, Table 2 · p. 7

4.3. Exploratory Study

  • Grad-CAM visualizations show pFedPT with ViT and CNN shifting some attention toward added prompts, which the authors say provide prior knowledge for inference.The comparison uses FedAvg and pFedPT under Dirichlet CIFAR10.Source: §4.3 · pp. 7–8, Figure 4 · p. 8
  • With pure-color inputs, pFedPT outputs for different clients become distinguishable when prompts are added, indicating that prompts carry client-model prior knowledge.The study uses learned prompts from different clients to examine embeddings while excluding image-content variation.Source: §4.3 · p. 8, Figure 5 · p. 8
  • Padding prompts achieve their highest reported accuracy at size 4, while their accuracy decreases at larger sizes; fixed- and random-location prompt accuracy grows with size but remains slightly lower.The authors suggest padding may cover more original-image pixels and block classification information; other visual tasks may need different prompts.Source: §4.3 · p. 8, Figure 6 · p. 8

5. Conclusion

The conclusion presents pFedPT as a visual-prompt-based personalized federated learning framework. It describes locally learned prompts as client-distribution guidance for backbone fine-tuning and reports that prompt generators keep that distribution information from being revealed to other clients or the server.

5. Conclusion

  • pFedPT uses a prompt generator to distill local-data information into visual prompts that guide backbone fine-tuning for personalized downstream tasks.The backbone and prompt generator are trained in the pFedPT process.Source: §5 · p. 8
  • Because the prompt generator is trained locally, client data-distribution information is not revealed to other clients or the server.Source: §5 · p. 9
  • The paper presents pFedPT as a plugin for boosting existing FL methods and reports experiments across heterogeneous settings and dataset partitions.Source: §5 · p. 9

A. Appendix: More Experiment Results

The appendix experiment description identifies the image-classification datasets, heterogeneity scenarios, and methods used for comparison.

A. Appendix: More Experiment Results

A.1. Setups

The setup describes datasets, model backbones, prompt templates, data partitions, and comparison methods. It also specifies the partition heterogeneity and model configurations used in the experiments.

A.1. Setups

  • CIFAR10, CIFAR100, and Tiny ImageNet are image-classification datasets; Table 3 is titled as comparing predicted and real data distributions.CIFAR10 and CIFAR100 have 50,000 training and 10,000 test images each; Tiny ImageNet has 100,000 training and 10,000 test images.Source: §A.1 · p. 11, Table 3 · p. 11
  • The experiments use ViT and CNN backbones, with the ViT processing flattened image patches through transformer encoder blocks.The described ViT uses 8 blocks, 8 attention layers per block, 8 heads, patch size 4, and embedding dimension 128; the CNN uses convolutional and fully connected layers.Source: §A.1 · p. 11
  • Visual prompts are tested as random-location patches, fixed-location patches, or padding added to the input image.The prompt templates are illustrated in Figure 7, with their comparison results referred to Figure 6.Source: Figure 7 · p. 11, §A.1 · p. 12
  • Dataset partitions use Dirichlet or Pathological label distributions, with a heterogeneity parameter controlling the overall partition level.The heat maps encode sample counts by class and client; the text describes category dominance at specified heterogeneity weights.Source: §A.1 · p. 12, Figure 8 · p. 12
  • The comparison methods include FedAvg, FedProx, MOON, FedPer, FedRep, FedMTL, FedBABU, and local-only training.Source: §A.1 · p. 12
  • The setup fixes the local-training learning rate at 0.005, prompt-generator rate at 1.0, batch size at 16, and local-training epochs at 5.It also lists method-specific settings for FedProx, MOON, FedRep, FedMTL, and FedBABU.Source: §A.1 · p. 12

A.2. Other Experimental Results

On TinyImageNet, pFedPT achieves 25.95% test accuracy with ViT on Pathological distribution, but is not highest on Dirichlet distribution. Prompt changes decline toward stability, and pFedPT performs significantly better than FedAvg when adapting to new clients.

A.2. Other Experimental Results

  • On TinyImageNet Dirichlet distribution, pFedPT does not achieve the highest test accuracy; the authors attribute this to difficulty stabilizing prompts within 500 rounds.Source: §A.2 · p. 12
  • 25.95% test accuracy for pFedPT, versus 9.37% for FedAvg and 19.23% for FedPer, on TinyImageNet Pathological distribution with ViT.Table 4 reports results across different non-IID settings and backbones.Source: §A.2 · p. 12, Table 4 · p. 13
  • Prompt changes first rise and then decline during training, with round-to-round embedding change eventually approaching 0 as prompts stabilize for each client's data distribution.Source: §A.2 · p. 13, Figure 10 · p. 13
  • With 400 samples for a new CIFAR10 client, pFedPT's prompt-generator fine-tuning performs significantly better than FedAvg's last-layer fine-tuning.The backbone is fixed during pFedPT fine-tuning; FedAvg keeps all but its last layer fixed.Source: §A.2 · pp. 13–14, Figure 11 · p. 14
Loading 2303.08678v1…