PaperlayerView exact revision on arXiv

Source-linked AI summary

FedCorr: Multi-Stage Federated Learning for Label Noise Correction

Jingyi Xu, Zihan Chen, Tony Q. S. Quek, Kai Fong Ernest Chong

arXiv:2204.04677v1cs.LGcs.AIcs.CV

TL;DR

Heterogeneous label noise and non-IID client data make privacy-preserving federated learning difficult, while centralized noise-handling methods are inadequate for small local datasets. FedCorr introduces a multi-stage framework that identifies noisy clients and labels, corrects them, and resumes training, achieving robustness and outperforming state-of-the-art FL methods across multiple noise levels and data settings. Its current formulation does not handle dynamic client participation.

  • Problem

    Federated learning lacks a unified privacy-preserving approach for simultaneously handling heterogeneous local label quality and data statistics, while centralized label-noise methods are inadequate on small client datasets.

  • Method

    FedCorr uses LID scores to identify noisy clients, per-sample losses to identify noisy labels, adaptive noise-level-based proximal regularization, clean-client finetuning, and subsequent training on all clients.

  • Results

    FedCorr is robust to label noise and outperforms state-of-the-art FL methods across multiple noise levels and diverse IID and non-IID data settings.

  • Takeaways & Limitations

    Privacy-preserving label identification and correction can be integrated into a multi-stage FL framework that jointly addresses heterogeneous label quality and data statistics.

  • Takeaways & Limitations

    FedCorr does not currently handle dynamic participation, so late-joining noisy clients may be incorrectly categorized as clean.

Abstract

from arXiv · show

Federated learning (FL) is a privacy-preserving distributed learning paradigm that enables clients to jointly train a global model. In real-world FL implementations, client data could have label noise, and different clients could have vastly different label noise levels. Although there exist methods in centralized learning for tackling label noise, such methods do not perform well on heterogeneous label noise in FL settings, due to the typically smaller sizes of client datasets and data privacy requirements in FL. In this paper, we propose $\texttt{FedCorr}$, a general multi-stage framework to tackle heterogeneous label noise in FL, without making any assumptions on the noise models of local clients, while still maintaining client data privacy. In particular, (1) $\texttt{FedCorr}$ dynamically identifies noisy clients by exploiting the dimensionalities of the model prediction subspaces independently measured on all clients, and then identifies incorrect labels on noisy clients based on per-sample losses. To deal with data heterogeneity and to increase training stability, we propose an adaptive local proximal regularization term that is based on estimated local noise levels. (2) We further finetune the global model on identified clean clients and correct the noisy labels for the remaining noisy clients after finetuning. (3) Finally, we apply the usual training on all clients to make full use of all local data. Experiments conducted on CIFAR-10/100 with federated synthetic label noise, and on a real-world noisy dataset, Clothing1M, demonstrate that $\texttt{FedCorr}$ is robust to label noise and substantially outperforms the state-of-the-art methods at multiple noise levels.

1. Introduction

Real-world federated learning combines heterogeneous client data statistics with heterogeneous label quality, creating a privacy-constrained need for unified label correction and robust training. FedCorr addresses these challenges through a multi-stage framework that identifies noisy clients and labels, applies adaptive regularization, and outperforms existing FL methods across diverse settings.

  • Motivation: Heterogeneous local data statistics and label quality jointly challenge federated learning, while privacy requirements constrain cross-client label correction.Clients may be non-IID, imbalanced, and affected by different noise levels; existing approaches generally address only one discrepancy or treat dissimilar clients as malicious.
  • FedCorr framework: FedCorr identifies noisy clients using LID scores, detects noisy labels with per-sample losses, and uses an adaptive proximal term based on estimated local noise.The framework sends each client’s LID score as a scalar, relabels identified high-loss samples using global-model predictions, and regularizes local training according to estimated noise levels.
  • FedCorr framework: FedCorr is a general multi-stage framework that jointly addresses local label-quality and data-statistics discrepancies while preserving client data privacy.The framework is organized around privacy-preserving noisy-client identification, label correction, and training procedures for heterogeneous FL.
  • Results: FedCorr outperforms state-of-the-art FL methods across multiple datasets, noise levels, and both IID and non-IID data partitions.The reported evaluation covers federated synthetic label noise and diverse client data settings, including IID and non-IID partitions.

2. Related work

Prior FL work addresses non-IID data, robust aggregation, and related client discrepancies, but does not provide a unified privacy-preserving approach for identifying and correcting noisy labels. FedCorr combines diverse federated data generation with noisy-label identification and correction, while using LID as a measure of local intrinsic dimensionality.

  • Federated optimization: Federated optimization methods such as FedProx, FedDyn, and SCAFFOLD primarily address discrepancies in local data statistics and client drift.Their mechanisms include proximal terms, dynamic proximal terms, and control variates, respectively.
  • Robust federated learning: Existing robust FL methods are not designed to identify noisy labels and often lack label correction at noisy clients or require auxiliary data.These methods include robust aggregation, reputation mechanisms, credibility reweighting, distillation, and personalized learning.
  • FedCorr: FedCorr differs by combining diverse local-data generation with synthetic label noise, noisy-label identification, and correction without privacy leakage.This positions the framework across non-IID data generation, robust FL, and label-noise correction rather than addressing only one component.
  • Local intrinsic dimension: LID measures the intrinsic dimensionality of a data manifold by relating neighborhood growth to radius, without assuming a specific data distribution beyond continuity.Its estimation can use nearest-neighbor distances and maximum-likelihood estimation.

3. Proposed Method

FedCorr is a multi-stage federated learning framework that addresses heterogeneous client data statistics and label quality while preserving privacy. It uses LID scores, per-sample losses, adaptive proximal regularization, selective finetuning, label correction, and final training across all clients.

  • Multi-stage framework: FedCorr runs pre-processing, finetuning, and usual training stages to identify noisy clients and samples, correct labels, and then use all local data.Pre-processing estimates client noise and relabels high-confidence noisy samples; finetuning uses relatively clean clients before final training over all clients.
  • Client scheduling: Clients participate once per pre-processing iteration through small-fraction sampling without replacement, reducing divergence between aggregated and local models.Each iteration ends after every client has participated exactly once; the scheduling scheme combines small- and large-fraction advantages.
  • Optimization: An adaptive local proximal regularization term uses estimated local noise levels, alongside mixup augmentation, during pre-processing.The proximal term is added to the local loss and depends on the estimated noise level from the preceding iteration.
  • Privacy: The framework preserves client-data privacy because the server receives only an additional scalar LID score derived from model predictions.The LID score is computed from output-layer predictions and does not reveal raw input data.
  • Noisy-client identification: LID scores measure prediction-subspace dimensionality, with cumulative LID scores used to distinguish noisy clients from clean clients during training.Noisy-data models tend to have larger LID scores, while cumulative scores become more reliable as individual scores overlap over time.

4. Experiments

FedCorr is evaluated across IID and non-IID settings, multiple datasets and noise levels, against centralized label-noise methods and federated baselines. It achieves strong accuracy and remains versatile when combined with other FL methods, while ablations identify fraction scheduling as especially influential.

  • IID settings: FedCorr achieves the best test accuracies across all tested IID noise settings on CIFAR-10 and CIFAR-100, with particularly large gains at high noise.The comparison uses Tables 2 and 3 and the same configurations across baselines.
  • Non-IID settings: FedCorr consistently outperforms all baselines by at least 7% on non-IID CIFAR-10 and achieves the highest federated accuracy on Clothing1M.On Clothing1M, its federated accuracy also exceeds the reported centralized JointOpt accuracy.
  • Combination with other FL methods: Applying FedCorr’s first two stages to FedDyn, Median, and PoC consistently improves CIFAR-10/100 accuracy across different noisy-client ratios.These methods represent local optimization, aggregation, and client-selection approaches.
  • Ablation study: Ablations show that every component improves accuracy, with fraction scheduling having the largest effect.The small preprocessing fraction helps capture local data characteristics by limiting information loss from aggregating multiple models.
  • Ablation study: The highest accuracy across noise levels is primarily attained at ρ = 0.4 rather than zero noise because label correction can introduce additional noise.This observation is reported as an ablation insight on CIFAR-10.

5. Conclusion

The paper concludes that FedCorr jointly addresses discrepancies in local label quality and data statistics while providing privacy-preserving label correction. Experiments show robustness and strong performance across multiple noise levels and data settings, although dynamic client participation remains outside its formulation.

  • FedCorr jointly tackles local label-quality and data-statistics discrepancies while performing privacy-preserving label correction for identified noisy clients.
  • Experiments demonstrate FedCorr’s robustness and outperformance across multiple noise levels and diverse data settings.
  • FedCorr does not currently handle dynamic participation, and late-joining noisy clients may be incorrectly categorized as clean.The paper identifies handling dynamic participation as future work.

Supplementary Material for FedCorr: Multi-Stage Federated Learning for Label Noise Correction

The supplementary material provides implementation details, additional experiments, architectural comparisons, communication analyses, and further explanations of FedCorr’s correction and ablation results. It also discusses potential negative societal impacts.

  • Appendix B documents implementation details for FedCorr and the baselines, while Appendix C provides additional experimental results.
  • Supplementary experiments cover non-IID CIFAR-100 partitions, model-agnostic behavior, communication efficiency, cumulative-LID motivation, label correction, and ablation results.
  • Appendix D discusses the potential negative societal impact of FedCorr.

B. Implementation details

The implementation uses standardized optimization settings and matched communication budgets across methods, with dataset-specific hyperparameters and published or reimplemented baselines. FedCorr fixes key parameters for LID estimation, mixup, and proximal regularization.

  • Experiments reimplement several baselines, use official implementations for FedDyn and ARFL, and unofficial implementations for RoFL and Median.
  • Most methods use SGD with momentum 0.5, no weight decay, dataset-specific batch sizes, and matched communication costs.The communication cost is defined as the cumulative number of participating clients.
  • FedCorr: FedCorr fixes k = 20 for LID estimation, α = 1 for mixup, and β = 5 for proximal regularization, while matching baseline communication cost.
  • Label-noise baselines: JointOpt and DivideMix receive centralized and federated configurations, including grouped noise settings and a federated warm-up process.
  • FL baselines: FedAvg, FedProx, and related baselines use specified learning rates and fixed or method-specific proximal mechanisms.

C.1. CIFAR-100 with non-IID data partition

On CIFAR-100 with non-IID data, FedCorr improves best test accuracy over the evaluated baselines under the reported noisy setting.

  • The comparison reports average and standard deviation of best test accuracies across methods on non-IID CIFAR-100.Table 9 covers five trials and the caption specifies noise setting (ρ, τ) = (0.4, 0).
  • At least 7% improvement in best test accuracy over the baselines is achieved by FedCorr on non-IID CIFAR-100.The experiment uses noise setting (ρ, τ) = (0.4, 0.5) and non-IID setting (p, αDir) = (0.7, 10).

C.2. Comparison of different architectures

FedCorr is evaluated across multiple architectures, while its communication efficiency is compared at target accuracies on CIFAR-10 and CIFAR-100.

  • Comparison of different architectures: FedCorr’s model-agnostic evaluation uses ResNet-18, VGG-11, and LeNet-5 on CIFAR-10 with IID data and varied synthetic noise.The comparison reports best test accuracies for each architecture.
  • Comparison of communication efficiency: Targeted communication cost is defined as the lowest total communication required to reach a desired test accuracy.Lower targeted communication cost indicates higher communication efficiency.
  • Comparison of communication efficiency: At least 1.9× communication-efficiency improvement is reported on CIFAR-10 and at least 1.3× on CIFAR-100.The comparisons use targeted communication costs at test accuracies of 80% and 65% on CIFAR-10, and 50% on CIFAR-100.

C.5. Evaluation of label noise identification and label correction

FedCorr evaluates noisy-client identification and label correction through LID distributions, estimated noise levels, sample separation, and confusion matrices.

  • Cumulative LID scores remain separated between clean and noisy clients after four iterations, whereas ordinary LID scores begin overlapping.Both metrics separate the client groups earlier, but cumulative LID remains more robust later in preprocessing.
  • Estimated noise levels closely match post-correction ground truth, while the gap between pre-training and post-stage-1 noise levels reflects label-correction effectiveness.Clean clients are assigned exactly zero estimated noise in most cases.
  • Small numbers of false positives across identified noisy clients indicate effective noisy-sample identification in the pre-processing stage.The evaluation uses a Gaussian Mixture Model and reports true/false positives/negatives.
  • After label correction, ground-truth noise levels are close to zero for all five illustrated clients.For the initially clean second client, only a minimal amount of label noise is introduced.
  • The ablation study reports that fraction scheduling has the largest effect, while label correction significantly improves training stability during usual training.Without finetuning, the total communication cost ends at 5000 rather than reaching the later 5500-round marker.

C.7. Illustration of non-IID data partitions on CIFAR-10

The CIFAR-10 non-IID evaluation uses three local data settings across 100 clients and illustrates their class distributions and dataset sizes.

  • Three non-IID settings, (p, αDir) = (0.7, 10), (0.7, 1), and (0.3, 10), are illustrated across 100 CIFAR-10 clients.The illustration covers local class distributions and local dataset sizes.

D. Potential negative impact: the issue of freeloaders

FedCorr assumes collaborative participation, but a client with completely random labels could exploit it as an annotation service. This creates an unfairness concern for clients that annotated their data beforehand.

  • D. Potential negative impact: the issue of freeloaders: A “freeloader” with completely random labels could use FedCorr to obtain corrected annotations through participation.The paper describes this as using FedCorr as the actual annotation process.
  • D. Potential negative impact: the issue of freeloaders: FedCorr’s robustness to noisy clients therefore creates a fairness concern for clients that performed annotation before joining.The concern arises because prior annotators bear annotation effort that the freeloader avoids.
  • D. Potential negative impact: the issue of freeloaders: The issue relies on the paper’s implicit assumption that participating clients are collaborative and jointly train the global model.
Loading 2204.04677v1…