PaperlayerView exact revision on arXiv

Source-linked AI summary

FedCP: Separating Feature Information for Personalized Federated Learning via Conditional Policy

Jianqing Zhang, Yang Hua, Hao Wang, Tao Song, Zhengui Xue, Ruhui Ma, Haibing Guan

arXiv:2307.01217v2cs.LGcs.AI

TL;DR

Non-IID client data can make a single global model fit local data poorly, while existing personalized federated learning methods focus on model parameters rather than the data-derived information they encode. FedCP generates a sample-specific policy to separate global and personalized feature information, and it outperforms eleven state-of-the-art methods by up to 6.69%.

  • Problem

    Non-IID, unbalanced client data can make a single global model fit local data poorly, and existing pFL methods focus on model parameters rather than sample-specific feature information.

    Source: §1 · pp. 1–2

  • Method

    FedCP uses a Conditional Policy Network to generate a policy for each sample, separating global and personalized feature information for processing by a global head and a personalized head.

    Source: §1 · p. 2

  • Results

    FedCP outperforms eleven state-of-the-art methods by up to 6.69% and maintains performance when some clients accidentally drop out.

    Source: §9 · p. 9

  • Takeaways & Limitations

    FedCP considers personalization at the sample-specific feature level, which the paper describes as more fine-grained than client-level model-parameter approaches.

    Source: §1 · p. 2

    The setting models clients’ private datasets as samples from distinct, non-overlapping distributions.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Recently, personalized federated learning (pFL) has attracted increasing attention in privacy protection, collaborative learning, and tackling statistical heterogeneity among clients, e.g., hospitals, mobile smartphones, etc. Most existing pFL methods focus on exploiting the global information and personalized information in the client-level model parameters while neglecting that data is the source of these two kinds of information. To address this, we propose the Federated Conditional Policy (FedCP) method, which generates a conditional policy for each sample to separate the global information and personalized information in its features and then processes them by a global head and a personalized head, respectively. FedCP is more fine-grained to consider personalization in a sample-specific manner than existing pFL methods. Extensive experiments in computer vision and natural language processing domains show that FedCP outperforms eleven state-of-the-art methods by up to 6.69%. Furthermore, FedCP maintains its superiority when some clients accidentally drop out, which frequently happens in mobile settings. Our code is public at https://github.com/TsingZ0/FedCP.

1 INTRODUCTION

Federated learning supports collaboration without sharing private data, but non-IID, unbalanced client data can make a single global model fit local data poorly. FedCP addresses a gap in personalized federated learning by separating global and personalized information in each sample’s features, and the paper reports gains across two scenarios.

1 INTRODUCTION

  • Client data can be sparse, while privacy concerns and regulations challenge centralized AI; federated learning uses local data without sharing it.Non-IID, unbalanced client data can make a single global model fit each client poorly.Source: §1 · p. 1
  • Existing pFL methods focus on global and personalized information in client-level model parameters, although heterogeneous client data contains both kinds of information.The paper identifies sample-specific feature information as a more fine-grained target for personalization.Source: §1 · p. 2
  • FedCP uses a Conditional Policy Network to generate a policy for each sample, separating global and personalized feature information for processing by separate heads.The global head is frozen, while the personalized head stores personalized information.Source: §1 · p. 2
  • FedCP outperforms eleven state-of-the-art methods in both pathological and practical settings.The experiments cover computer vision and natural language processing, and FedCP also maintains strong performance when some clients drop out.Source: §1 · p. 2

2 RELATED WORK

Prior personalized federated learning methods generally exploit global and personalized information through model parameters, while conditional computing offers input-dependent model behavior. FedCP adapts conditional computing to separate feature information for personalized federated learning across tasks.

2.1 Personalized Federated Learning

  • Existing pFL methods span meta-learning (Per-FedAvg), regularization (pFedMe [42], FedAMP [15], Ditto [24]), personalized heads (FedPer [2], FedRep [6], FedRoD [4]), and aggregation (FedFomo [56], FedPHP), but focus on model parameters.Traditional FL methods such as FedAvg [32] and FedProx also face the challenge of fitting heterogeneous local data with one global model.Source: §2.1 · p. 2

2.2 Conditional Computing

  • Conditional computing uses task-dependent inputs and an auxiliary module to generate signals that dynamically affect a model, including routing or feature adaptation.Input-conditioned routing and feature adaptation are illustrated by ConvNet-AIG [44], SpotTune [11], TADAM [35], and TMA.Source: §2.2 · p. 3
  • Earlier conditional-computing methods target centralized AI scenarios and specific tasks; FedCP combines dynamic routing and feature adaptation for pFL across tasks.Its Conditional Policy Network separates global and personalized feature information for processing along different routes.Source: §2.2 · p. 3

3 METHOD

FedCP learns personalized models for clients with distinct data distributions by separating global and personalized information in sample features. It routes the separated features through global and personalized heads, with feature alignment during local learning.

3.1 Overview

  • Each client uses a global feature extractor and head alongside a personalized feature extractor, personalized head, and CPN; the global extractor and head are frozen during local learning.The global feature extractor is used for local learning but is not part of the personalized inference model.Source: §3.1 · p. 3

3.2 Federated Conditional Policy (FedCP)

  • FedCP separates global and personalized information in each sample’s feature vector, making personalization more fine-grained than methods focused on client-level model parameters.A Conditional Policy Network (CPN) generates the sample-specific policy.Source: §3.2 · p. 4, §3.2.1 · p. 4

3.2.1 Separating feature information.

  • The CPN uses sample features and a client-specific vector derived from personalized-head parameters to generate the policy for separating feature information.The client-specific vector is reused at inference.Source: §3.2.1 · p. 4

3.2.2 Processing feature information.

  • The policy weights the feature vector into global and personalized features, which are processed by the global and personalized heads, respectively.The two heads’ outputs are averaged to form the upload head.Source: §3.2.1 · p. 4, §3.2.2 · pp. 4–5

3.2.3 Aligning features.

  • A maximum mean discrepancy (MMD) loss aligns features from the personalized and global feature extractors, and the local loss combines this alignment loss with the task loss.The alignment uses a kernel feature map, such as one induced by an RBF kernel.Source: §3.2.3 · p. 5

3.3 Privacy Analysis

  • FedCP’s privacy analysis describes the uploaded head as incorporating global parameters and argues that keeping the dynamically generated sample policy from the server makes policy recovery difficult.The paper also says this separation breaks the connection between the feature extractor and head for attacks based on shared model parameters.Source: §3.3 · p. 5

4 EXPERIMENTAL SETUP

Experiments span image and text classification under pathological and practical client heterogeneity. The evaluation reports personalized-model test accuracy across repeated runs.

Datasets and models

  • The image tasks use MNIST, Cifar10, Cifar100, and Tiny-ImageNet with a 4-layer CNN, plus ResNet-18 on Tiny-ImageNet; AG News uses fastText.The local learning rates are 0.005 for the 4-layer CNN and 0.1 for ResNet-18 and fastText.Source: §4 · p. 5

Heterogeneity settings

  • Pathological partitions assign clients disjoint classes, while practical partitions allocate class samples by Dirichlet proportions with β=0.1 by default; client data is split 75%/25% for training and testing.The pathological setup samples 2/2/10 classes per client for MNIST/Cifar10/Cifar100, respectively.Source: §4 · p. 5

Training and evaluation protocol

  • Runs use 20 clients, full participation, batch size 10, one local epoch, and up to 2000 iterations; results are means and standard deviations over five runs.Traditional FL reports the best global model's test accuracy, while pFL reports the average test accuracy of the best personalized models.Source: §4 · p. 5

5 ABLATION STUDY

The ablations examine separated feature attention, CPN inputs, and FedCP modules. Results indicate that sample-specific input and the CPN/global-head combination are especially consequential.

5.1 Feature Information Visualization

  • Grad-CAM shows the global head attending to background context and the personalized head to foreground objects or distinctive features in Tiny-ImageNet samples.For one sample, the rarely used pink color is treated as more personalized than the widely used blue color.Source: §5.1 · p. 5

5.2 Effectiveness of CPN input

  • Removing either CPN input vector lowers accuracy, and the sample-specific vector matters more than the client-specific vector; using a random vector still beats all baselines.The random-vector variant can still learn feature separation through end-to-end training.Source: §5.2 · p. 6

5.3 Effectiveness of FedCP modules

  • Removing CPN and the frozen global head drops accuracy by 8.74%; removing CPN alone drops it by 3.01%, versus 1.31% without GFM.Removing CPN and GFM together is worse than removing either alone, while adding GFM to the variant without CPN, GFM, and global head causes degeneration.Source: §5.3 · p. 6, Table 1 · p. 6

6 EVALUATION AND ANALYSIS

Across classification benchmarks and varied training conditions, FedCP outperforms the evaluated baselines. The experiments also examine runtime, heterogeneity, client scale, local epochs, and random client dropout.

6.1 Main Experiments

  • FedAvg and FedProx struggle when global models do not fit local data, while FedCP separates global and personalized feature information; its sample-specific conditioning also addresses client-specific needs.The paper attributes weaker results for several other methods to loss of global information or competing objectives.Source: §6.1 · pp. 6–7
  • FedCP beats all baselines with either CNN or ResNet-18; on practical Cifar100, it exceeds Ditto by 6.69%.The CPN adds 0.527M parameters per client, 9.25% of the 4-layer CNN or 4.67% of ResNet-18.Source: §6.1 · p. 6, Table 2 · p. 7

6.2 Computing and Communication Overhead

  • FedCP takes less time per iteration than most baselines and adds 4.67% communication overhead per iteration versus FedAvg with ResNet-18.Ditto and pFedMe require more time per iteration than most methods.Source: §6.2 · p. 7, Table 3 · p. 7

6.3 Different Heterogeneity Degrees

  • FedCP outperforms baselines across heterogeneity degrees; as β grows, most pFL methods lose their advantage over FedAvg on Tiny-ImageNet, unlike methods using global information during local learning.pFedMe and FedAMP fall below traditional FL methods when β=1.Source: §6.3 · p. 7

6.4 Scalability with Different Client Amounts

  • FedCP remains ahead of baselines as client counts vary; increasing the client count with more total data also raises accuracy by facilitating information transfer.At N=500, clients average 90 training samples; the paper notes this is insufficient for FedFomo's weight calculation, while FedAMP diverges.Source: §6.4 · pp. 7–8, Table 5 · p. 8, Table 4 · p. 8

6.5 Large Local Epochs

  • FedCP maintains superiority with larger local epochs, while FedRoD accuracy drops by 2.16% when local epochs rise from 5 to 40.The paper notes that larger local epochs increase per-iteration computing overhead for most methods.Source: §6.5 · p. 8, Table 6 · p. 8

6.6 Clients Accidentally Dropping Out

  • FedCP maintains superior and stable performance under randomly varying client participation, whereas pFedMe and FedPHP lose up to 6.65% and 9.80% accuracy, respectively.The standard deviations of Per-FedAvg, pFedMe, Ditto, and FedRoD exceed 1% when ρ∈[0.1,1].Source: Table 7 · p. 8, §6.6 · p. 9

7 EFFECT OF THE HYPER-PARAMETER 𝜆

The study varies λ, which weights the MMD loss aligning personalized and frozen-global feature outputs. Accuracy first rises and then falls as λ increases, with smaller values favored under greater heterogeneity.

Effect of λ

  • Accuracy first increases and then decreases with λ; overly large λ=50 limits learning from local data, while greater heterogeneity calls for relatively smaller λ.A proper λ balances learning local information with fitting personalized features to the frozen global head.Source: §7 · p. 9, Table 8 · p. 9

8 POLICY STUDY

The policy study examines sample-level policy behavior through the personalization identification ratio (PIR) and policy distributions. The observed policy patterns vary with backbone and client, while test-time policy values indicate more personalized than global feature information in the described scenarios.

8 POLICY STUDY

  • PIR is computed by averaging a client’s sample-specific policies and then averaging the elements of that result into one scalar.The study calls this scalar the personalization identification ratio.Source: §8 · p. 9
  • With the 4-layer CNN, client #0’s PIR rises from 0.50 to about 0.58 in the first 20 iterations; with ResNet-18, it first falls, then rises to about 0.61.The authors interpret the ResNet-18 pattern as reflecting more global feature information early on, which the CPN captures as it changes over FL iterations.Source: §8 · p. 9, Figure 5 · p. 9
  • Policy-value ranges differ across clients, but mean policy values are similar across clients for a given backbone and exceed 0.5 during inference in these scenarios.Client #10 has the largest policy-value range.Source: §8 · p. 9, Figure 5 · p. 9

9 CONCLUSION

FedCP uses a policy for each sample to separate global and personalized feature information, which are processed by global and personalized heads. It outperforms eleven state-of-the-art methods by up to 6.69% and maintains performance when some clients drop out.

9 CONCLUSION

  • FedCP generates a policy for each sample to separate global and personalized feature information, then processes them with global and personalized heads, respectively.The method is reported to outperform eleven state-of-the-art methods by up to 6.69% and to maintain performance when some clients accidentally drop out.Source: §9 · p. 9
  • FedCP outperforms eleven state-of-the-art methods by up to 6.69%.The conclusion also reports that it maintains performance when some clients accidentally drop out.Source: §9 · p. 9
  • FedCP maintains its performance when some clients accidentally drop out.The paper notes this occurs in mobile settings.Source: §9 · p. 9

A CONVERGENCE ANALYSIS

The appendix evaluates FedCP’s convergence, privacy-preserving ability, conditional policy network designs, and hyperparameter settings. It reports convergence after around 74 iterations in one setting, compares privacy results with representative baselines, and describes accuracy and cost trade-offs across CPN variants.

A CONVERGENCE ANALYSIS

  • On Tiny-ImageNet with ResNet-18 in the default practical setting, FedCP converges after around 74 iterations; loss and accuracy curves fluctuate before iteration 56 due to policy updates.The loss after local learning approaches the loss before local learning as both reach their minimum; test accuracy increases as training loss decreases.Source: A CONVERGENCE ANALYSIS · p. 12, Figure 6 · p. 12

B PRIVACY-PRESERVING ABILITY

  • Under the semi-honest server setting, the authors report that Table 9’s PSNR results show FedCP’s privacy-preserving ability is superior to representative FedAvg and FedPer baselines.The passage states that lower PSNR indicates better privacy-preserving ability; the comparison is on Cifar100 in the default practical setting.Source: B PRIVACY-PRESERVING ABILITY · p. 12, Table 9 · p. 12

C CONDITIONAL POLICY NETWORK DESIGN

  • Adding CPN fully connected layers improves ResNet-18 accuracy but slightly decreases 4-layer CNN accuracy, while their added computation is not worth the small gain; sigmoid improves both backbones over ReLU.Using batch normalization instead of layer normalization improves 4-layer CNN accuracy by 0.64% but decreases ResNet-18 accuracy by around 0.48%; group normalization decreases accuracy for both.Source: C CONDITIONAL POLICY NETWORK DESIGN · pp. 12–13, Table 10 · p. 12

D HYPERPARAMETER SETTINGS

  • The reported grid-search choices are λ = 5 for the 4-layer CNN and λ = 1 for ResNet-18 and fastText.The search space was λ ∈ {0, 0.1, 1, 5, 10}.Source: D HYPERPARAMETER SETTINGS · p. 13

E DATA DISTRIBUTION VISUALIZATION

The appendix visualizes training and test data distributions across image and text tasks. The figures cover varying client counts, heterogeneity settings, and—in Tiny-ImageNet—varying β.

E DATA DISTRIBUTION VISUALIZATION

Related papers

Loading 2307.01217v2…