PaperlayerView exact revision on arXiv

Source-linked AI summary

Federated Learning with Label Distribution Skew via Logits Calibration

Jie Zhang, Zhiqi Li, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, Chao Wu

arXiv:2209.00189v2cs.LGcs.AI

TL;DR

Label distribution skew makes federated local models biased toward overrepresented, minority, and missing classes, reducing global-model accuracy. The paper proposes FedLC, which calibrates logits with pairwise margins before cross-entropy; experiments report improved performance across skewed federated and real-world datasets.

  • Problem

    Label distribution skew causes local models to overfit minority and missing classes, making existing softmax cross-entropy-based FL updates unsuitable.

  • Method

    FedLC calibrates each class logit according to occurrence probability and applies a fine-grained cross-entropy loss with pairwise label margins.

  • Results

    FedLC consistently outperforms competing methods across skewed datasets, including 54.55% CIFAR-10 accuracy at β = 0.05 and 62.43% and 54.43% on highly skewed ImageNet-subset settings.

  • Takeaways & Limitations

    Reducing bias in local updates improves the global model, and combining FedLC with methods addressing inter-client inconsistency can further improve server performance.

  • Takeaways & Limitations

    The paper leaves combining intra-client label-skew and inter-client objective-inconsistency approaches as future work.

Abstract

from arXiv · show

Traditional federated optimization methods perform poorly with heterogeneous data (ie, accuracy reduction), especially for highly skewed data. In this paper, we investigate the label distribution skew in FL, where the distribution of labels varies across clients. First, we investigate the label distribution skew from a statistical view. We demonstrate both theoretically and empirically that previous methods based on softmax cross-entropy are not suitable, which can result in local models heavily overfitting to minority classes and missing classes. Additionally, we theoretically introduce a deviation bound to measure the deviation of the gradient after local update. At last, we propose FedLC (\textbf {Fed} erated learning via\textbf {L} ogits\textbf {C} alibration), which calibrates the logits before softmax cross-entropy according to the probability of occurrence of each class. FedLC applies a fine-grained calibrated cross-entropy loss to local update by adding a pairwise label margin. Extensive experiments on federated datasets and real-world datasets demonstrate that FedLC leads to a more accurate global model and much improved performance. Furthermore, integrating other FL methods into our approach can further enhance the performance of the global model.

1. Introduction

The paper focuses on label distribution skew in federated learning, where heterogeneous client data can degrade global-model accuracy. It attributes this degradation to biased local updates and proposes FedLC, a calibrated-loss approach targeting intra-client skew.

  • Motivation: Existing federated methods use rigid partitioning strategies and do not consistently outperform others across non-IID settings.This motivates algorithms specialized for particular types of heterogeneity.
  • Observed impact: 26.07% and 13.97% accuracy decreases occur under α = 2 and β = 0.2, respectively, compared with IID FedAvg.Lower α and β indicate more skewed distributions.
  • Motivation: Label distribution skew varies across clients and is a challenging, practically common non-IID setting.The paper studies quantity-based and distribution-based label-skew scenarios.
  • Problem: Intra-client skew biases local models toward minority and missing classes, worsening objective inconsistency and global-model performance.The proposed motivation is to reduce local-update bias so aggregation receives higher-quality local models.
  • Approach: FedLC calibrates class logits before softmax cross-entropy using occurrence probabilities and pairwise label margins.The method aims to encourage larger margins for underrepresented classes.

2. Related Works

Related work addresses general federated heterogeneity, imbalanced learning, and label-skew-specific problems. The paper positions label distribution skew as requiring more specialized treatment than existing general-purpose methods.

  • Federated learning: General FL methods address client drift and objective inconsistency through proximal terms, normalized averaging, adaptive optimization, or control variates.These methods treat non-IID data as a general problem.
  • Imbalanced data: Imbalanced-learning approaches commonly use re-sampling, re-weighting, over-sampling, or under-sampling to emphasize minority categories.These techniques are presented as related tools for skewed data.
  • Label distribution skew: Prior FL methods for label skew include monitoring schemes and FedRS, but auxiliary data, privacy, computation, or limited correction remain concerns.The supplied passage truncates the specific limitation of FedRS.
  • Empirical context: Skewed CIFAR10 accuracy falls heavily on minority classes and reaches zero for missing classes.The figure combines a class-sample histogram with per-class accuracy.
  • Empirical context: A two-client toy example illustrates how intra-client skew biases local optima and how averaging biased models can miss the global optimum.This connects local skew to aggregation failure.

3. FL with Label Distribution Skew

The paper defines label distribution skew and analyzes how it disrupts federated optimization. Its deviation-bound analysis links class imbalance to biased local updates and explains the resulting aggregation problem.

  • Definitions: Label distribution skew means clients have different label marginals while sharing the same conditional label distribution given inputs.Formally, Pi(y) varies across clients, while Pi(y | x) is shared.
  • Federated objective: The global FL objective aggregates client objectives according to relative client sample sizes.Clients perform local updates before their models are aggregated.
  • Optimization effects: Label-skewed local updates learn biased decision boundaries, so aggregation produces poor global performance.Client objectives may differ and lack shared minimizers, creating client drift.
  • Empirical evidence: After local training, majority-class accuracy can increase while minority-class accuracy drops and missing-class accuracy approaches zero.The empirical test compares models before and after local updates.
  • Deviation analysis: When nj/nr ≫ Djr > 0, the update is more likely to deviate from its expected direction, with minority-class and competing-class weight changes having opposite signs.The bound quantifies how majority scores overwhelm minority scores during local training.
  • Deviation analysis: The deviation bound exposes why standard softmax cross-entropy can produce locally misdirected updates under label distribution skew.Missing classes are treated as a limiting special case of minority classes.

4. Federated Learning via Logits Calibration

The section argues that standard softmax cross-entropy is unsuitable for highly skewed local data and introduces FedLC, which calibrates logits with pairwise label margins before cross-entropy. The calibration is designed to reduce local-update bias and gradient deviation, thereby benefiting the global model.

  • Motivation: Label distribution skew makes standard softmax cross-entropy unsuitable because minimizing error weighted by skewed class priors does not target balanced per-class performance.The paper instead formulates a calibrated error based on averaging per-class error rates.
  • Logits Calibration: FedLC calibrates each class logit before softmax cross-entropy using the estimated probability of class occurrence.The method encourages minority-class logits to be reduced by a relatively larger amount through class-prior-based calibration.
  • Fine-grained Calibrated Cross-Entropy: The calibrated loss applies a pairwise label margin Δ(y,i) to each logit and seeks margins that minimize test error under label skew.Δ(y,i) represents the desired score gap between the target class y and comparison class i.
  • Fine-grained Calibrated Cross-Entropy: The fine-grained calibrated loss jointly minimizes classification errors and directs learning toward minority-class margins to balance class margins during local training.This is presented as the intended trade-off of LCal under label distribution skew.
  • Deviation Bound: FedLC theoretically introduces a deviation bound to quantify how local gradient updates depart from their expected direction.The analysis considers majority and minority classes and derives the bound before and after adding pairwise margins.
  • Deviation Bound: Adding pairwise margins enlarges the minority-class term in the deviation bound, making gradient deviation harder and reducing bias in local model updates.The paper links less biased local updates to improved global-model performance.

5. Experiments

The experiments evaluate FedLC across multiple label-skew partitioning strategies, datasets, client counts, and analyses of local updates. FedLC consistently improves performance under skew, including highly heterogeneous settings, while improving minority and missing-class behavior.

  • Experimental settings: FedLC evaluates quantity-based and distribution-based label skew, where clients may contain only a few labels or label proportions sampled from a Dirichlet distribution.Quantity-based skew uses Q(α), while distribution-based skew uses D(β); smaller β indicates greater skewness.
  • Federated datasets: 75.45% prediction accuracy on Synthetic(1,1) exceeds the best baseline FedRS by 9.07%, and FedLC consistently outperforms the other baselines across federated datasets.Larger λ and µ produce poorer test accuracy, indicating that stronger heterogeneity and local skew make the task more difficult.
  • Real-world datasets: 54.55% test accuracy on CIFAR-10 with β = 0.05 exceeds FedRS by 10.16% under highly skewed distribution-based label skew.Across SVHN, CIFAR10, and CIFAR100 scenarios, FedLC achieves higher accuracy than the other compared methods.
  • ImageNet-subset: 62.43% and 54.43% mean accuracy on ImageNet-subset with α = 2 and β = 0.1 exceed FedRS by 7.2% and 6.01%, respectively.These results are reported for quantity-based and distribution-based skew on the complex ImageNet-subset dataset.
  • Method analysis: After one local epoch, FedLC improves average per-class accuracy over FedAvg before and after aggregation and separates minority from missing-class features more effectively.The t-SNE analysis uses 0, 448, and 2608 training samples for missing, minority, and majority classes, respectively.
  • Client-count analysis: FedLC achieves the best performance for m = 10, 30, 50, and 100 clients, reaching 43.81% accuracy with 100 clients.Performance decreases for all methods as the number of clients increases.

6. Conclusion

The paper proposes a fine-grained calibrated loss to improve the global model under label distribution skew and reports that experiments demonstrate reduced bias in local updates.

  • FedLC uses a fine-grained calibrated loss to improve global-model performance under label distribution skew.
Loading 2209.00189v2…