PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Federated Learning with Noisy Labels

Seunghan Yang, Hyoungseob Park, Junyoung Byun, Changick Kim

arXiv:2012.01700v1cs.LGcs.CV

TL;DR

Client labels can be corrupted, and differing noise distributions across clients make federated learning with noisy labels challenging. The paper aligns local models using server-aggregated class-wise centroids and adds confident-sample selection and global-guided pseudo-labeling; it reports outperforming existing state-of-the-art methods on CIFAR-10 and Clothing1M.

  • Problem

    Client labels can be corrupted, and differing noise distributions produce inconsistent local decision boundaries and weight divergence in federated learning.

    Source: Introduction · p. 1

  • Method

    The server aggregates and broadcasts class-wise centroids to align local models, while local updates select confident samples and use the global model to correct unconfident samples’ labels.

    Source: Introduction · p. 2

  • Results

    The approach outperforms existing state-of-the-art methods on CIFAR-10 and Clothing1M.

    Source: Conclusion · p. 8

  • Takeaways & Limitations

    The approach is reported as effective in the noisy federated setting by reducing weight divergence.

    Source: Conclusion · p. 7

    The centroid approach relies on the expectation that early learning prioritizes simple patterns, helping form global centroids less susceptible to noisy samples.

    Source: Local updates · p. 4

Abstract

from arXiv · show

Federated learning is a paradigm that enables local devices to jointly train a server model while keeping the data decentralized and private. In federated learning, since local data are collected by clients, it is hardly guaranteed that the data are correctly annotated. Although a lot of studies have been conducted to train the networks robust to these noisy data in a centralized setting, these algorithms still suffer from noisy labels in federated learning. Compared to the centralized setting, clients' data can have different noise distributions due to variations in their labeling systems or background knowledge of users. As a result, local models form inconsistent decision boundaries and their weights severely diverge from each other, which are serious problems in federated learning. To solve these problems, we introduce a novel federated learning scheme that the server cooperates with local models to maintain consistent decision boundaries by interchanging class-wise centroids. These centroids are central features of local data on each device, which are aligned by the server every communication round. Updating local models with the aligned centroids helps to form consistent decision boundaries among local models, although the noise distributions in clients' data are different from each other. To improve local model performance, we introduce a novel approach to select confident samples that are used for updating the model with given labels. Furthermore, we propose a global-guided pseudo-labeling method to update labels of unconfident samples by exploiting the global model. Our experimental results on the noisy CIFAR-10 dataset and the Clothing1M dataset show that our approach is noticeably effective in federated learning with noisy labels.

Introduction

Client labels may be corrupted, and differing noise distributions across clients make federated learning difficult. The paper addresses this with server-aligned class-wise centroids and local sample-selection and relabeling updates.

Introduction

  • Different client noise distributions produce inconsistent decision boundaries and weight divergence, while centralized noisy-label methods degrade when combined with FedAvg in federated learning.The resulting weight divergence makes client-model aggregation difficult.Source: Figure 1 · p. 1, Introduction · p. 2
  • The server aggregates and broadcasts class-wise centroids, which clients use to align local models’ decision boundaries despite differing noise distributions.The centroids are central features of local data on each device.Source: Introduction · p. 2
  • Local updates derive centroids from small-loss samples, adjust them by global-centroid similarity, select confident samples, and use the global model to correct unconfident samples’ labels.The authors use small-loss samples to reduce noisy-data effects and global-guided relabeling to alleviate local overfitting to noisy samples.Source: Introduction · p. 2
  • The authors describe this as the first federated learning algorithm for noisy labels and report high performance across various noise ratios.The introduction presents the scheme as interchanging centroids and reducing the effect of noisy data.Source: Introduction · p. 2

Related work

The paper addresses noisy labels in federated learning, where clients’ noise distributions differ, and describes centroid exchange, local sample selection, global-guided pseudo-labeling, and centroid aggregation as parts of its approach.

Federated learning

  • Federated learning transmits network parameters rather than raw data, while noisy-label methods described here include sample selection, label correction, and robust-loss approaches.The paper notes that prior federated studies assume clean client datasets, although correct annotation is not guaranteed for locally created data.Source: Federated learning · p. 2, Learning on noisy data · p. 2

Learning on noisy data

  • The proposed federated approach uses class-wise centroids to maintain similar class representations, alongside confident-sample selection and global-guided self-training.It is framed against inconsistent decision boundaries and weight divergence when clients have different noise distributions.Source: Learning on noisy data · p. 3

Problem definition and notations

  • Local centroids are average feature vectors from a client’s global-average-pooling layer, while global centroids reflect selected clients’ local centroids.The server cannot access training data.Source: Problem definition and notations · p. 3

Local updates

  • Clients use global-guided pseudo-labels and a binary mask to choose between given and pseudo-labels when training local models.The method also constrains local features using global centroids; centroid communication adds 0.01% to 0.03% of model-parameter communication in the experiments.Source: Local updates · p. 3

Global updates

  • The server aggregates client model parameters with FedAvg and updates global centroids by similarity-based aggregation of uploaded local centroids.The authors state that class-wise centroid summation is less affected by different noise ratios across classes.Source: Global updates · p. 5

Experiments

Experiments evaluate the method on CIFAR-10 with synthetic label noise and Clothing1M with real-world noisy labels. The reported results cover benchmark comparisons and ablations of noise variation, confident-sample selection, pseudo-labeling, and centroid exchange.

Datasets

  • CIFAR-10 experiments vary symmetric- and pair-flipping noise ratios, while Clothing1M experiments use one million noisy-labeled clothing images divided among 100 clients.The CIFAR-10 data are randomly distributed to 100 clients; Clothing1M has 14 categories and the clean training set is not used.Source: Datasets · p. 5

Analysis

  • On Clothing1M, the method outperforms others by a large margin in federated learning; on CIFAR-10, it achieves better overall test accuracy across noise ratios.The CIFAR-10 comparison covers symmetric and pair-flipping noise; the Clothing1M results concern real-world noise with unknown structure.Source: Table 2 · p. 6, Analysis · pp. 6–7, Table 1 · p. 6

Implementation details

  • CIFAR-10 uses a 9-layer CNN, while Clothing1M uses ImageNet-pretrained ResNet-50 with resized, normalized, and cropped training images.Deep self-learning is not reported on CIFAR-10 because the CNN is not pretrained and lacks enough representation power to extract valid prototypes.Source: Implementation details · p. 6

Analysis

  • Co-teaching’s fixed loss-based selection is vulnerable to client-specific noise, while local self-training can become self-biased; the proposed method uses centroids and global-guided labels to address these issues.The method constrains client class representations to remain close to global class representations, encouraging similar decision boundaries.Source: Analysis · p. 6

Ablation studies

  • The method is reported robust to variation in client noise ratios and class noise ratios, including the case where each client has one entirely mislabeled class.Table 3 evaluates client noise variation at a fixed ratio of 0.4 and separately tests class-level variation.Source: Table 3 · p. 6, Ablation studies · p. 7
  • With a 0.4 noise ratio set by symmetric flipping, training only on selected samples performs better than training on all samples; pseudo-labeling unconfident samples considerably increases performance, and sampling achieves high precision in noisy-label detection.Source: Ablation studies · p. 7
  • Global-guided pseudo-labeling outperforms naive pseudo-labeling and is reported to prevent local models from becoming self-biased.The comparison removes global guidance or replaces the proposed method with naive pseudo-labeling.Source: Ablation studies · p. 7
  • Global centroids make client feature representations more similar, reducing weight divergence.Figure 4 visualizes client features and global class-wise centroids.Source: Ablation studies · p. 7, Figure 4 · p. 7

Conclusion

The proposed approach exchanges class feature centroids and combines them with local sample selection and global-guided pseudo-labeling. The authors report reduced weight divergence and better performance than existing state-of-the-art methods on CIFAR-10 and Clothing1M.

Conclusion

  • The method exchanges class feature centroids as additional information between clients and the server.The conclusion identifies global and local feature centroids as the exchanged information.Source: Conclusion · p. 7
  • The approach outperforms existing state-of-the-art methods on CIFAR-10 and Clothing1M.The authors also report effectiveness in noisy federated learning by reducing weight divergence.Source: Conclusion · pp. 7–8
  • Local updates use similarity-based confident-example sampling and global-guided pseudo-labeling.The authors describe these algorithms as reducing the effect of noisy data.Source: Conclusion · p. 8

Ethics statement

The authors suggest the approach could support learning on edge devices with noisy labels, including in fields such as healthcare. Their experiments focus on i.i.d. data, and they identify non-i.i.d. noisy federated learning as an area needing more work.

Ethics statement

  • The authors suggest the scheme could help prevent medical accidents associated with erroneous data in federated healthcare.This is presented as a potential application, not an evaluated outcome.Source: Ethics statement · p. 8
  • The scheme is presented as enabling client-side learning without precise data or expert annotation for each device.The authors frame this as supporting a shift from central servers toward edge devices.Source: Ethics statement · p. 8
  • The study focuses on noisy labels in i.i.d. data; noisy labels with non-i.i.d. data remain future work.The authors explicitly identify this as the scope boundary of their setting.Source: Ethics statement · p. 8

Algorithm details

The algorithm initializes global centroids with average features, then updates centroids and training choices over time. It delays pseudo-label use and increases the centroid-loss weight gradually.

Algorithm details

  • Average features initialize global centroids in the first round; later rounds aggregate clients’ local centroids.Random global-centroid initialization could hinder accurate local centroids because local centroids are updated using global-centroid similarities.Source: Algorithm details · p. 10
  • The method starts with a large R(t) to keep more instances, then gradually reduces it to limit fitting to noisy samples.The authors set T to 10 and τ to ϵ in their experiments and report robustness to these parameters.Source: Algorithm details · p. 10
  • Unconfident samples use ground-truth labels initially; after Tpl rounds, training uses both ground-truth and pseudo-labels.The authors cite insufficient early training as the reason pseudo-labels are initially replaced.Source: Algorithm details · p. 10
  • The centroid-loss weight λcen starts at 0 and gradually rises to a predefined value to avoid an early noisy mask.Source: Algorithm details · p. 10
  • For mini-batch SGD, local centroids are updated at each iteration by blending the previous local centroid with the current feature estimate.The blend weights are determined by the squared similarity; the first local epoch uses global centroids as its starting centroids.Source: Algorithm details · p. 10

Two types of noisy labels

The experiments manually introduce label noise into clean CIFAR-10 and MNIST data using transition matrices. They consider symmetric and pair flipping noise.

Two types of noisy labels

  • Clean CIFAR-10 and MNIST datasets are manually corrupted using a label transition matrix Q.Q is defined as the conditional probability of a noisy label given the clean label.Source: Two types of noisy labels · p. 10
  • Symmetric label noise is introduced using a transition matrix parameterized by the noise ratio ϵ and number of classes n.Source: Two types of noisy labels · p. 10
  • Pair flipping is described as a noise-generation method for fine-grained classification, with its own transition matrix.Source: Two types of noisy labels · p. 10

Implementation details

Experiments used a shared federated setup and method-specific configurations. The proposed method, Co-teaching, Joint Optimization, and Deep self-learning used distinct hyperparameters.

Shared setup

  • Experiments used 100 clients, selected 10 per round, batch size 50, and five local epochs, with training rounds varying by dataset.Training ran for 100, 1,000, and 40 rounds on MNIST, CIFAR-10, and Clothing1M, respectively; all algorithms used SGD with momentum 0.5 and weight decay 10^-4.Source: Implementation details · p. 10

Ours

  • The proposed method used λcen=1.0 and λe=0.8 across datasets, with Tpl set to 30, 100, and 5 for MNIST, CIFAR-10, and Clothing1M.It used T=10 and τ=ϵ for loss-based centroids; the authors report robustness to these parameters in ablation studies.Source: Implementation details · p. 10

Co-teaching

  • Co-teaching used two same-architecture networks with different initializations and a 0.15 initial learning rate for MNIST and CIFAR-10.For Clothing1M, its initial learning rate was 0.001 and decreased by a factor of 10 every 10 rounds.Source: Implementation details · p. 10

Joint Optimization

  • Joint Optimization used α=1.2, β=0.8, and start epoch 100 on MNIST and CIFAR-10, while Clothing1M used α=2.4 and β=0.8.The Clothing1M learning rate was 8 × 10^-4; MNIST and CIFAR-10 used noise-ratio-specific learning rates following Tanaka et al. 2018.Source: Implementation details · p. 10

Deep self-learning

  • Deep self-learning used a 0.002 initial learning rate, decreased by a factor of 10 every five epochs, and sampled 128 images per class for label correction.Each class used three prototypes; the authors adjusted the selected-sample and prototype counts because each federated client held only a small dataset portion.Source: Implementation details · p. 10

More experimental results

Additional experiments test label and feature noise, client participation, computational cost, and hyperparameter sensitivity. Reported results include strong performance across settings, with Tpl sensitivity as an exception.

Algorithm 1

  • Algorithm 1 combines client-side sample selection and local updates with server aggregation of global weights and centroids.Clients obtain global-guided pseudo-labels, form small-loss sets and confident masks, update weights and centroids, then return them for aggregation.Source: More experimental results · p. 11

Experimental results on MNIST

  • On MNIST, the proposed approach maintained high performance across symmetric and pair-flipping label-noise ratios.The experiment used a 10-category dataset and reports average test accuracy over the last 10 rounds in Table 6.Source: More experimental results · pp. 10, 12, Table 6 · p. 11

Image variances in different domains

  • 88.4% was achieved on noisy CIFAR-10 with both label noise and client-specific light-intensity variation, versus 88.7% in the original setting.The light-intensity factor ranged from 0.5 to 1.5; the accompanying figure depicts image variances in intensity.Source: Figure 5 · p. 11, More experimental results · p. 12

Real federated learning scenario

  • 93.5% was achieved on COVIDx distributed across 100 medical centers, compared with 93.6% in the centralized setting.The dataset contained 15,282 chest x-ray images; the experiment assumes each medical center holds its own data.Source: More experimental results · p. 12

Number of participating clients

  • With a 100-client population, performance remained comparable when only two clients participated in each communication round.The experiment used noisy CIFAR-10 with a noise ratio of 0.4.Source: More experimental results · p. 12

Computational cost

  • The proposed method’s speed was similar to Joint Optimization, with a marginal computational-cost increase attributed to similarity-based updates and confident samples.Co-teaching took longer than methods using one network because it uses two networks.Source: Table 8 · p. 11, More experimental results · p. 12

Performance dependency of hyper-parameters

  • Prediction accuracy was robust to the tested hyperparameters except Tpl, with lower performance when pseudo-labels were introduced before they were accurate.The CIFAR-10 experiment used symmetric label noise at ratio 0.4; Figure 6 presents the hyperparameter dependency.Source: More experimental results · p. 12, Figure 6 · p. 12
Loading 2012.01700v1…