PaperlayerView exact revision on arXiv

Source-linked AI summary

FedFA: Federated Learning with Feature Anchors to Align Features and Classifiers for Heterogeneous Data

Tailin Zhou, Jun Zhang, Danny H. K. Tsang

arXiv:2211.09299v4cs.LG

TL;DR

Heterogeneous client data can degrade federated learning, and common approaches address weight divergence or feature inconsistency separately. FedFA uses feature anchors to align features and calibrate classifiers across clients; experiments report significant gains over baselines under data heterogeneity.

Abstract

from arXiv · show

Federated learning allows multiple clients to collaboratively train a model without exchanging their data, thus preserving data privacy. Unfortunately, it suffers significant performance degradation due to heterogeneous data at clients. Common solutions involve designing an auxiliary loss to regularize weight divergence or feature inconsistency during local training. However, we discover that these approaches fall short of the expected performance because they ignore the existence of a vicious cycle between feature inconsistency and classifier divergence across clients. This vicious cycle causes client models to be updated in inconsistent feature spaces with more diverged classifiers. To break the vicious cycle, we propose a novel framework named Federated learning with Feature Anchors (FedFA). FedFA utilizes feature anchors to align features and calibrate classifiers across clients simultaneously. This enables client models to be updated in a shared feature space with consistent classifiers during local training. Theoretically, we analyze the non-convex convergence rate of FedFA. We also demonstrate that the integration of feature alignment and classifier calibration in FedFA brings a virtuous cycle between feature and classifier updates, which breaks the vicious cycle existing in current approaches. Extensive experiments show that FedFA significantly outperforms existing approaches on various classification datasets under label distribution skew and feature distribution skew.

I. INTRODUCTION

Federated learning preserves client-data privacy, but heterogeneous data can slow convergence and degrade generalization. The paper attributes this to coupled feature inconsistency and classifier divergence, and proposes FedFA to align both across clients.

I. INTRODUCTION

  • Data heterogeneity drives a vicious cycle in which feature inconsistency and classifier divergence reinforce each other, degrading federated training.Existing approaches that regularize weight divergence or feature inconsistency did not show clear advantages over FedAvg on various classification tasks.Source: NOMENCLATURE LIST · pp. 1–2

NOMENCLATURE LIST

  • FedFA uses feature anchors to align client features and classifiers, and significantly outperforms existing methods under label, feature, and combined distribution skews.The paper also analyzes FedFA’s non-convex convergence rate.Source: NOMENCLATURE LIST · p. 2

1) Tackle data heterogeneity by controlling weight divergence:

  • FedProx, FedDyn, and SCAFFOLD regularize or adjust local updates to address weight divergence.Other classifier-focused work calibrates classifiers using virtual features or addresses long-tail overfitting.Source: 1) Tackle data heterogeneity by… · p. 2

2) Tackle data heterogeneity by controlling feature inconsistency:

3) Tackle data heterogeneity by improving aggregation schemes:

  • Server-side alternatives include normalizing local updates, adaptive server momentum, and neuron-wise averaging; these can complement FedFA’s client-side auxiliary loss.The cited examples are FedNova, adaptive momentum, and FedMA.Source: 3) Tackle data heterogeneity by… · pp. 2–3

4) Tackle data heterogeneity in wireless networks:

  • Wireless factors and client availability further affect federated learning, motivating work on resource allocation, client selection, and communication reliability.The paper positions FedFA as addressing both label and feature skew, unlike methods that tackle only label skew.Source: 4) Tackle data heterogeneity in… · p. 3

A. Terminology; B. Federated learning

  • The setup treats each client as holding a local dataset and decomposes its model into a feature extractor and a linear classifier.Clients optimize local objectives before weighted model aggregation.Source: A. Terminology · p. 3, B. Federated learning · p. 3

C. Data heterogeneity in federated learning

Experiments and analysis link heterogeneous client data to coupled feature inconsistency and classifier divergence. The section examines this relationship empirically and theoretically, motivating simultaneous treatment of both issues.

C. Data heterogeneity in federated learning

  • The paper distinguishes label skew (varying P_i(y)), feature skew (varying P_i(x)), and their co-occurrence across clients.Label skew varies label marginals, while feature skew varies input-feature marginals.Source: C. Data heterogeneity in federated… · p. 3

IV. MOTIVATION: INCONSISTENT FEATURES AND DIVERGED CLASSIFIERS ACROSS CLIENTS

  • Heterogeneous client data can slow convergence and degrade generalization, while prior work had not examined its effects across different model architectures.The section studies heterogeneity through the relationship between feature and classifier updates.Source: C. Data heterogeneity in federated… · pp. 3–4, §IV · p. 4

A. Experimental demonstration

  • Across both skews, FedFA alleviates feature inconsistency and classifier divergence, while MOON and other existing methods retain inconsistent features.The figure compares label skew in FMNIST with feature skew in Mixed Digit.Source: Figure 2 · p. 4, A. Experimental demonstration · p. 4

B. Theoretical demonstration

  • Data heterogeneity first diverges classifier updates, then inconsistent features worsen classifier divergence; this vicious cycle persists after global-model convergence.The analysis links classifier divergence to inconsistent feature updates across clients.Source: B. Theoretical demonstration · p. 5

V. FEDERATED LEARNING WITH FEATURE ANCHORS

FedFA uses shared feature anchors to align client features and calibrate classifiers in a consistent feature space, addressing the coupled feature–classifier problem.

V. FEDERATED LEARNING WITH FEATURE ANCHORS

  • FedFA is designed to train client models in a consistent feature space with classifiers corresponding to that space, breaking the identified vicious cycle.Source: §V · p. 5

A. Propose Method: FedFA

  • The server initializes one feature anchor per class, and an anchor loss measures average distance between each sample feature and its corresponding anchor.Source: A. Propose Method: FedFA · p. 5
  • Because the anchors are shared across clients, minimizing the anchor loss reduces intra-class feature distances within and across clients.The paper points to Figures 1(b) and 5(c) for this effect.Source: A. Propose Method: FedFA · p. 6

1) Minimizing local objective with feature anchor loss:

FedFA combines local feature alignment and classifier calibration around shared anchors, while keeping anchors fixed during local training and updating them through feature momentum.

1) Minimizing local objective with feature anchor loss:

  • Clients add a weighted feature-anchor loss to supervised loss in each local objective, reducing feature inconsistency through mini-batch updates.The balance parameter μ weights the anchor loss against the supervised loss.Source: 1) Minimizing local objective with… · p. 6

2) Calibrating local classifiers with feature anchors:

  • At each mini-batch, clients use shared anchors and their class labels as classifier inputs; calibration reduces proxy-to-anchor distance and classifier divergence.The paper links classifier calibration to subsequent mitigation of feature inconsistency.Source: 2) Calibrating local classifiers with… · p. 6

3) Fixing feature anchors in local training but computing their momentum:

  • Anchors remain fixed during local training, while clients accumulate class-feature momentum for server aggregation; this avoids recomputing final class features from the training data.Source: 3) Fixing feature anchors in local… · p. 6

4) Feature anchor and model aggregation at server:

FedFA aggregates client models and feature anchors at the server, and its convergence analysis gives a non-convex rate under stated smoothness, stochastic-gradient, and feature-extractor assumptions.

4) Feature anchor and model aggregation at server:

  • The server aggregates class feature anchors using weighted averaging and aggregates the global model as in FedAvg; the process repeats across rounds.Source: 4) Feature anchor and model… · p. 6

B. Non-convex Convergence analysis of FedFA

C. Loss smoothness analysis of FedFA.

FedFA’s feature alignment and classifier calibration smooth classifier loss updates and promote a virtuous cycle between classifier harmony and feature consistency. Across label-skew settings, FedFA outperforms baselines, with gains increasing under more severe skew.

C. Loss smoothness analysis of FedFA.

  • FedFA improves classifier-weight loss Lipschitzness over FedAvg; together, feature alignment and classifier calibration promote classifier harmony and feature consistency.The analysis assumes orthogonal feature-anchor initialization; FedFA breaks the vicious cycle under both label and feature skew.Source: C. Loss smoothness analysis of FedFA · p. 7

D. Computational Overhead of FedFA

E. Advantages of FedFA

  • Feature anchors create a shared feature space, keep classifiers consistent, and promote same-class feature polymerization across clients.The reported consequences include reduced intra-class distance, increased inter-class distance, improved classification accuracy, smoother loss, and better convergence.Source: E. Advantages of FedFA · p. 7

1) Datasets and data heterogeneity setups:

  • Experiments cover image classification under label and feature skew, using EMNIST, FMNIST, CIFAR-10, CIFAR-100, and Mixed Digits.Setups include class-based and Dirichlet label splits, plus real-world EMNIST and artificial mixed-digit feature skew; Mixed Digits reports average top-1 accuracy across five digit datasets.Source: 1) Datasets and data heterogeneity… · pp. 7–8, Table II · p. 8

2) Baselines:

  • Baselines include FedAvg, weight-regularization methods FedProx and FedDyn, and feature-alignment methods MOON and FedProc.FedFA modifies client-side updates; server-side approaches are treated as complementary rather than baselines.Source: 2) Baselines · p. 8

3) Models:

  • Models use two-layer CNNs for EMNIST, FMNIST, and CIFAR-10, ResNet-18 for CIFAR-100, and a three-layer CNN for Mixed Digits.FedFA uses λ = 0.5 for anchor momentum, µ = 0.1 for the local loss, and pairwise orthogonal feature-anchor initialization.Source: 3) Models · p. 8, 4) FedFA setup · p. 8

5) Federated simulation setups:

  • The main simulations use 100 clients, 10 participants per round, five local epochs, batch size 64, and 200 communication rounds.The setup also specifies SGD at learning rate 0.01 and weight decay 0.001; Figure 3 varies client sample rate and local epochs under a separate setup.Source: 5) Federated simulation setups · p. 8, Figure 3 · p. 9

1) Performance under label distribution skew:

  • 52.95% vs 34.89%: FedFA achieves higher CIFAR-10 top-1 accuracy than MOON under α = 0.1 label skew.Across datasets, FedFA’s decline as label skew increases is smaller than the baselines’; its CIFAR-100 accuracy advantage is about 3%.Source: Table I · p. 8, 1) Performance under label… · p. 9

2) Performance under feature distribution skew:

FedFA is evaluated across feature skew, combined label and feature skew, more homogeneous data, and varied federated settings. The reported results favor FedFA across these settings, while ablations and calibration-timing experiments examine the roles of its components.

2) Performance under feature distribution skew:

  • FedFA reaches 99.28% accuracy on EMNIST, 0.77% above MOON (98.51%), and gains 7.14% over FedDyn (83.59%) on Mixed Digits under feature skew.Mixed Digits assigns MNIST or SVHN subsets to clients, creating feature distribution skew.Source: 2) Performance under feature… · p. 9

3) Performance under both label and feature skews:

  • Under combined label and feature skews, FedFA maintains at least a 10% performance advantage over all baselines, with the largest gap reaching 31.8% against FedDyn under #C = 2.FedFA decreases from 90.73% to 83.46%, while the largest baseline drop is 31.93% for FedDyn under #C = 2.Source: 3) Performance under both label and… · p. 9

4) Performance without label and feature distribution skew:

  • FedFA is 8.64% more accurate than FedAvg on CIFAR-100 and reaches 88.40% on FMNIST with mild heterogeneity, versus FedAvg's 85.90% without skew.The comparison uses the same learning rate for the more homogeneous and heterogeneous settings.Source: 4) Performance without label and… · p. 9

5) Performance on different client sample rate and local epoch:

  • FedFA performs better than all baselines across the tested client setups; larger sample rates improve accuracy, while more local epochs hurt it.Increasing sample rate from 0.1 to 0.3 yields about 10% accuracy gain, exceeding the gain from 0.3 to 0.5; FedProx and MOON degrade more with larger local epochs than FedAvg and FedFA.Source: 5) Performance on different client… · p. 9

6) Performance of Lipschitzness of loss:

  • With momentum, all methods perform better than without it, yet FedFA without momentum exceeds momentum-based baselines under the stated heterogeneous setting.At α = 0.1, FedFA without momentum reaches 52.95%, versus FedProx with momentum at 49.87%; the authors report that smoother FedFA loss achieves better generalization.Source: 6) Performance of Lipschitzness of loss · pp. 9–10

1) Ablation on anchor updates and classifier calibration of FedFA:

  • Both anchor updating and classifier calibration contribute to FedFA's performance, with calibration boosting it by 22.37% under combined label and feature skew.Anchor updating adds at least around 2%; mini-batch calibration outperforms calibration at other tested training phases, while post-training calibration improves little over no calibration.Source: 1) Ablation on anchor updates and… · p. 10, 4) Timing to calibrate classifiers · p. 10

2) The momentum update of feature anchors:

VII. CONCLUSION AND FUTURE WORKS

  • The reported experiments focus on classification; extending the analysis to encoder-decoder tasks and testing shallow-layer alignment for deep models remain future directions.The authors propose investigating whether an encoder-decoder update cycle exists and whether shallow-layer alignment could improve deep-model training.Source: §VII · pp. 10–11

APPENDIX

The appendix proof derives a non-convex convergence bound for FedFA by bounding local-update deviations and averaging expected loss changes across rounds.

A. Proof of Non-convex Convergence rate of FedFA

  • The proof treats T1 and T2 as separate terms, using Jensen’s inequality and update assumptions to bound them.It also derives terms involving d(t) and the distance between the global and client models.Source: A. Proof of Non-convex Convergence… · p. 13
  • The analysis bounds one-round loss deviation, then averages expected changes across rounds to obtain the convergence result.It combines intermediate bounds before taking the round average.Source: A. Proof of Non-convex Convergence… · p. 14
  • The proof derives a sufficient-round bound for FedFA under a stated upper bound on the learning rate.The bound uses the initial-to-final loss gap and update-related terms.Source: A. Proof of Non-convex Convergence… · p. 14

B. Proof of Lemma 1

The proof analyzes classifier updates through a cross-entropy gradient decomposition, while allowing same-class features to differ across clients.

B. Proof of Lemma 1

  • The cross-entropy gradient is decomposed into deviations involving positive features and mean negative features.The passage labels these components as the mean deviation by positive features and deviation by mean negative features.Source: B. Proof of Lemma 1 · p. 14
  • One gradient term is approximated using Property 1 in, based on statistics reported in Figures 4 and 5 of.Source: B. Proof of Lemma 1 · p. 14
  • The analysis does not assume same-class features extracted across clients are similar, unlike.Source: B. Proof of Lemma 1 · p. 14

C. Proof of Theorem 1

The proof connects skewed client data to classifier-update divergence, including when clients share classes or have differing class sets.

C. Proof of Theorem 1

  • Classifier updates diverge for classes absent from both clients or present in one client but not the other.Source: C. Proof of Theorem 1 · p. 15
  • With feature-distribution skew, clients share an extractor at each round’s start, but skewed inputs induce inconsistent mapped features.Source: C. Proof of Theorem 1 · p. 15
  • Both label and feature skews diverge classifier updates; avoiding divergence for shared classes would require a perfect feature extractor, deemed impractical [6].Source: C. Proof of Theorem 1 · pp. 14–15

D. Proof of Lemma 2 and Theorem 2

The appendix proofs state a simplifying probability assumption and an anchor initialization condition, then provide supplementary material and terminology definitions.

D. Proof of Lemma 2 and Theorem 2

E. Proof of Theorem 4

  • The anchor orthogonality condition ac · a¯c = 0 provides an initialization method for feature anchors.The passage points to Figure 4 for discussion of this initialization.Source: E. Proof of Theorem 4 · p. 15

SUPPLEMENTARY MATERIALS

  • Supplementary materials cover privacy issues from feature anchors, additional visualizations and experiments, and experiment setup details.Source: SUPPLEMENTARY MATERIALS · p. 16

A. Terminologies

  • The terminology section distinguishes client-local models from aggregated global models and defines the vicious and virtuous cycles through feature and classifier updates.Source: A. Terminologies · p. 16
  • Positive pairs share a label; positive features belong to the proxy’s class, while positive proxies are the corresponding class proxy.Source: A. Terminologies · p. 16

B. Discussion of privacy issues introduced by feature anchors

Across the tested client sample rates, FedFA retains an accuracy advantage over baselines, and it performs better across client counts. The discussion also identifies a privacy–generalization trade-off for feature-anchor updates and describes the validation settings and models.

B. Discussion of privacy issues introduced by feature anchors

  • FedFA without feature-anchor updates performs about 2% worse than FedFA with updates, exposing a privacy–generalization trade-off.The no-update variant still outperforms the best baseline under label and feature skew.Source: B. Discussion of privacy issues… · p. 16

1) Feature Visualization and Similarity Histogram for All Methods under Label Distribution Skew

  • The feature-visualization figures show t-SNE maps and cosine-similarity histograms under label skew for FedAvg and FedFA, alongside FedAvg under homogeneous data.The comparisons use 10 clients.Source: Figure 5 · p. 17, Figure 6 · p. 17

2) Performance under different batch size:

3) Performance under different client numbers:

1) Specific Models:

  • The validation and test experiments replace batch-normalization layers with GroupNorm layers to ablate the effect of batch normalization.The passage says this replacement is used in all experiments.Source: 1) Specific Models · p. 17

2) Feature Visualization and Similarity Histogram for All Methods under Feature Distribution Skew

  • The feature-visualization figures show t-SNE maps and cosine-similarity histograms for baselines under feature-distribution skew, with FedAvg and FedFA also shown under homogeneous data.The comparisons use 10 clients.Source: Figure 7 · p. 18, Figure 8 · p. 18

2) Validation Experiment Setup:

  • For visualization, the same validation samples are input to client models, but only classes held by each client are displayed.Label- and feature-skew visualizations use the labels owned by the corresponding client.Source: 2) Validation Experiment Setup · p. 18

1) Performance under different client sample rate with 400 rounds:

  • For CIFAR-10, performance is reported under different client sample rates, batch sizes, and local epochs; the cited passage gives no numerical accuracy advantage or communication-round persistence result.Source: Figure 9 · p. 19

3) Test Experiment Setup:

The test setup compares FedFA with common federated-learning and local-regularization methods on image-classification benchmarks with label, feature, and combined distribution skew. It defines distinct data-partition procedures for label skew and includes both natural and artificial feature imbalance.

3) Test Experiment Setup:

  • The comparison includes FedAvg, FedProx, FedDyn, MOON, and FedProc.These are presented as common federated-learning algorithms and local-regularization methods.Source: 3) Test Experiment Setup · p. 19
  • The benchmarks are EMNIST, FMNIST [35], CIFAR-10, CIFAR-100, and Mixed Digits, evaluated under label, feature, and combined skew.The paper says these datasets use heterogeneity settings from prior work,,.Source: 3) Test Experiment Setup · p. 20
  • Label skew uses either class-based client splits or Dirichlet-distributed class proportions; α values of 0.1 and 0.5 set the heterogeneity level.At α = 0.1, label and local-dataset-size skew occur together.Source: 3) Test Experiment Setup · p. 20
  • Feature-imbalance tests include natural EMNIST imbalance and an artificial Mixed Digits dataset built from five digit datasets.The EMNIST subset has 10 classes and 341873 samples; Mixed Digits comprises MNIST, SVHN, USPS, SynthDigits, and MNIST-M.Source: 3) Test Experiment Setup · p. 20

Related papers

Loading 2211.09299v4…