PaperlayerView exact revision on arXiv

Source-linked AI summary

Different Corruptions, Different Signals: Uncertainty and Loss in Federated Data Quality

Bradley Scott, Zeqi Luo, Edmond S. L. Ho

arXiv:2609.31454v1cs.LGcs.AIcs.CV

TL;DR

Federated-learning corruption can affect inputs or labels, but it is unclear whether uncertainty and prediction-label loss detect these modes equally. The paper compares both signals across image noise and persistent random label flips, finding that their detection strength differs by corruption type.

  • Problem

    It is unclear whether input-conditional uncertainty and prediction-label loss expose input corruption and label corruption equally in federated learning.

    Source: §I · p. 1

  • Method

    The paper compares input-conditional uncertainty and prediction-label loss for persistent random label flips and additive image noise, using per-client and within-client per-sample detection tests.

    Source: §I · p. 1

  • Results

    For label flips, loss AUC is 0.85 on CIFAR-10 and 0.95 on SVHN while uncertainty is 0.49–0.50; for image noise, expected-entropy AUC is 0.67 on CIFAR-10 and 0.66 on SVHN versus loss at 0.64 on both.

    Source: §I · p. 1, C. Within-client per-sample detection · p. 5

  • Takeaways & Limitations

    When corruption type is unknown, the results motivate monitoring both entropy-based uncertainty and prediction-label loss rather than selecting either signal alone.

    Source: §I · p. 2, §V · p. 6

    The tested setting uses ResNet-20, 10 clients, FedAvg, two datasets, random flips, and one Gaussian-noise level, so generalisability across other architectures, datasets, client counts, strategies, and corruption behaviours remains to be tested.

    Source: §V · p. 6

Abstract

from arXiv · show

Federated learning (FL) data corruption can affect either inputs or labels, but it remains unclear whether input-conditional uncertainty and prediction-label loss expose these corruption modes equally. This paper compares two corruption-detection signals in FL: input-conditional uncertainty and prediction-label loss. The uncertainty signal is characterised using a learned aleatoric variance estimate together with Monte Carlo (MC) dropout variance and entropy measures, while the loss is computed against the supplied label. We test these signals against additive image noise and persistent random label flips. On ResNet-20 with CIFAR-10 and SVHN under Dirichlet partitions with data that are not independent and identically distributed (non-IID), the two corruption types behave differently. For persistent random label flips, the within-client per-sample area under the receiver operating characteristic curve (AUC) is 0.85 on CIFAR-10 and 0.95 on SVHN for prediction-label loss, while every uncertainty estimator stays at chance (0.49--0.50). This pattern is consistent with the model remaining confident in the underlying image despite the supplied label being wrong. For image noise, expected-entropy uncertainty rises above chance (0.67 on CIFAR-10 and 0.66 on SVHN), while loss responds comparably (0.64 on both). Each signal is therefore the stronger detector for a different corruption: the prediction-label loss for persistent label flips, and expected-entropy uncertainty for image noise, with its advantage becoming apparent as federation-wide corruption prevalence increases. Robust FL data-quality assessment should match the signal to the corruption rather than rely on uncertainty alone across corruption types.

I. INTRODUCTION

The introduction asks whether input-conditional uncertainty and prediction-label loss detect different corruption modes in federated learning. It argues that the signals respond differently to input noise and label flips, and compares them at client and within-client sample levels.

I. INTRODUCTION

  • Because image noise changes x while persistent flips change the supplied y, the two corruption modes need not leave the same evidence in model outputs.A persistent flip reuses the same wrong label whenever that sample is seen.Source: §I · p. 1

A. Predictive uncertainty and prediction-label signals

  • Input-conditional uncertainty asks whether the model is unsure about an image, whereas prediction-label loss asks whether it believes the supplied label attached to it.The uncertainty measures include learned aleatoric variance, expected entropy, and MC dropout disagreement; Confident Learning is cited as an example of the prediction-label signal.Source: §I · p. 1

I. INTRODUCTION

  • The comparison reports per-client and within-client per-sample AUC, with the latter holding client partition effects fixed by comparing samples inside one client under the same global model.The per-client view can be confounded by between-client heterogeneity, including class composition.Source: §I · p. 1
  • For persistent label flips, loss separates corrupted samples (AUC 0.85 on CIFAR-10 and 0.95 on SVHN), while uncertainty stays at chance (0.49–0.50); for image noise, expected-entropy AUC is 0.67/0.66 versus loss at 0.64 on both.The evaluation uses ResNet-20 on CIFAR-10 and SVHN under 10-client Dirichlet non-IID FL, sweeping corruption rate and corrupt-client count.Source: §I · p. 1

A. Predictive uncertainty and prediction-label signals

  • The paper keeps input-conditional uncertainty and prediction-label mismatch distinct, representing uncertainty through aleatoric and epistemic variance and entropy measures.The variance pair is learned aleatoric variance and MC dropout probability variance; the entropy pair is expected entropy and mutual information,.Source: A. Predictive uncertainty and… · p. 2

B. Label noise and corruption in FL

Prior FL corruption work uses varied signals and settings, leaving open what input-conditional uncertainty reveals about label noise. The method probes client data with a common global model and evaluates both client-level and within-client detection.

B. Label noise and corruption in FL

  • Prior FL studies report feature-corruption detection as easier than label-corruption detection using Wasserstein distances, while this paper directly compares uncertainty with prediction-label loss across both modes.The cited work reports an asymmetry using optimal-transport distances.Source: B. Label noise and corruption in FL · p. 2

C. Established uses of uncertainty in FL

  • Established FL uses of uncertainty address personalization, calibration, model selection, misclassification detection, fairness weighting, or poisoned-update inspection rather than label-quality assessment.These are described as input-side or model-side judgments, not label-quality assessment.Source: C. Established uses of uncertainty in FL · p. 2

D. Uncertainty for noisy-client detection

  • FedNed and FedDPSO target roughly 90%-noise clients using server-side auxiliary data, whereas this study tests 20–60% persistent random flips with a common global model on local probes.The paper frames this as a test of which signal reveals moderate within-client corruption.Source: D. Uncertainty for noisy-client… · p. 2

III. METHOD

  • Each round, the global model probes client-local data and records four input-conditional uncertainty signals plus prediction-label loss, then scores separation across clients and within a corrupt client.The scores are diagnostic only; uncertainty is not used to weight or filter clients.Source: §III · p. 2

A. Signals

The paper records prediction-label loss alongside four input-conditional uncertainty signals. It separates variance-based and entropy-based estimators, with variance signals derived from a learned aleatoric head or MC dropout predictions.

A. Signals

  • Each client probes its local validation split with the current global model and records prediction-label loss alongside four input-conditional uncertainty signals.The signals form variance-based AV/EV and entropy-based AE/EE pairs.Source: A. Signals · p. 2

1) Variance signals:

  • AV is a learned per-sample variance estimate, while EV measures per-class probability variation across repeated MC dropout predictions.AV comes from a dropout-free aleatoric head; AV and EV are recorded separately as diagnostic signals at probe time.Source: 1) Variance signals · pp. 2–3
  • AE is expected entropy across stochastic predictions, whereas EE is mutual information measuring disagreement between those predictions.EE is the BALD acquisition score used here solely as an epistemic-uncertainty measure; no active learning is performed.Source: 2) Entropy signals · p. 3

2) Entropy signals:

The section defines prediction-label loss as a score based on the supplied label and describes learned variance and entropy-based uncertainty estimators. It distinguishes which signals use label information and how variance enters the training objective.

3) Prediction-label loss:

  • ℓ = −log ¯p_y(x) scores the supplied label using its probability under the mean of T MC-dropout predictive distributions.Ranking by high loss is equivalent to ranking by low self-confidence.Source: 3) Prediction-label loss · p. 3

B. Estimators and the loss-attenuation mechanism

  • The learned aleatoric head predicts a non-negative, image-dependent scalar, while a learned softplus fusion head combines aleatoric and epistemic variance signals.The image-dependent estimate is learned through a label-dependent objective.Source: B. Estimators and the loss-attenuation… · p. 3
  • Training uses K MC-dropout passes for mean logits and independent Gaussian draws to approximate the attenuated predictive probability.Source: B. Estimators and the loss-attenuation… · p. 3
  • Training combines plain per-pass cross-entropy with an attenuated term; its variance penalty discourages large variance but does not guarantee aleatoric-variance calibration.β balances the two loss terms, while λ weights the variance penalty.Source: B. Estimators and the loss-attenuation… · p. 3
  • AE measures within-prediction entropy averaged across passes, whereas EE measures disagreement between stochastic predictions; neither reads the supplied label.Source: B. Estimators and the loss-attenuation… · p. 3

C. Why flips should evade input-conditional uncertainty

The proposed asymmetry between flip detection signals is motivated by how random incorrect labels interact with the image’s underlying class signal. Image noise, by contrast, perturbs the input itself.

  • Randomly assigning each flipped sample to an incorrect class spreads wrong labels across alternatives, allowing the underlying class to remain the dominant label signal.Source: C. Why flips should evade… · p. 4
  • This pattern can leave input-conditional uncertainty unable to rank flipped samples while prediction-label loss remains high.Source: C. Why flips should evade… · p. 4
  • Class-consistent flips or sufficiently long training may change this relationship, while image noise directly perturbs the input.Sufficiently long training may fit individual corrupted labels.Source: C. Why flips should evade… · p. 4

D. Per-condition corruption mask

The corruption conditions use fixed sample assignments within corrupt clients, with flips and additive Gaussian image noise studied separately. This creates clean and corrupted samples in the same client for sample-level detection.

E. Detection metrics

The section defines a cross-group ranking AUC and contrasts federation-wide client ranking with within-client sample ranking. The latter holds client identity and model state fixed, while per-client comparisons can reflect pre-existing client differences.

E. Detection metrics

  • AUC is the probability that a randomly chosen corrupt item scores as more suspicious than a randomly chosen clean item; 0.5 is chance and values below it indicate inverse ranking.The measure counts only corrupt-versus-clean pairs, not within-group ordering.Source: E. Detection metrics · p. 4

1) Per-client:

  • Per-client AUC ranks clients by their mean signal and can conflate corruption with non-IID class composition and other between-client differences.Source: 1) Per-client · p. 4
  • In matched clean runs, separation between designated client groups reflects pre-existing between-client differences rather than corruption.Source: 1) Per-client · p. 4

E. Detection metrics

  • Within-client AUC compares corrupted and clean samples from one client under the same global model, controlling for client identity and model state.Residual differences in class composition or intrinsic sample difficulty may remain.Source: 2) Per-sample · p. 4

2) Per-sample:

Client-level corruption rankings are mixed and can reflect pre-existing client heterogeneity. The study uses a non-IID FedAvg setup and keeps per-sample scores local.

F. Measure-only protocol

  • Probe scores and within-client AUCs are computed locally; the server receives scalar client metrics, while uncertainty is not used to weight or gate clients.Source: F. Measure-only protocol · p. 4

A. Setup

  • The evaluation uses ResNet-20 on CIFAR-10 and SVHN with 10 non-IID clients, FedAvg, and separate persistent-flip and Gaussian-noise conditions.The setup sweeps corruption rate and corrupt-client count, Dirichlet skew, and three random seeds.Source: A. Setup · p. 4, Figure 2 · p. 5

B. Per-client detection is confounded by client heterogeneity

  • Client-level rankings are mixed: aleatoric variance is below chance on flips, while entropy can rank corrupt clients higher, especially on SVHN.Label-dependent error ranks flip clients cleanly, but clean-run controls also produce AUCs of 0.36–0.67 on CIFAR-10 and 0.45–0.66 on SVHN.Source: B. Per-client detection is confounded… · p. 5
  • Table II reports within-client per-sample corruption-detection AUCs across the full M × q × α × seed grid for variance, entropy, and prediction-label loss.Source: Table II · p. 5

C. Within-client per-sample detection

Within-client results show that the stronger detection signal depends on the corruption type. Persistent random flips sharply reduce clean-test accuracy even though uncertainty remains at chance, while expected entropy detects image noise.

C. Within-client per-sample detection

  • For persistent flips, within-client loss AUC is 0.85 on CIFAR-10 and 0.95 on SVHN, while every uncertainty signal stays at 0.49–0.50.For image noise, expected-entropy AUC is 0.67/0.66 versus loss at 0.64 on both datasets; expected entropy exceeds loss in every noise cell once M ≥4.Source: C. Within-client per-sample detection · p. 5

D. The corruption missed by uncertainty is more harmful

E. The loss erodes as corruption prevalence increases

Prediction-label loss detects persistent random flips less well as corruption prevalence rises but remains above chance throughout the tested grid. The discussion limits the findings to the evaluated data-level corruptions and frames combined signal use as future work.

E. The loss erodes as corruption prevalence increases

  • 0.93/0.98 at the mildest setting falls to 0.70/0.86 at M=8, q=0.6 (CIFAR-10/SVHN), though loss stays above chance throughout the tested grid.The decline is consistent with training assigning increasing probability to corrupted labels, reducing the prediction-label mismatch; the tested random flips remain dispersed across alternative classes.Source: E. The loss erodes as corruption… · p. 6

V. DISCUSSION AND LIMITATIONS

  • The evaluation covers ResNet-20, 10 clients, FedAvg, two datasets, persistent random flips, and one Gaussian-noise level—not adversarial model poisoning.Class-consistent flips could change the observed relationship; generalisability across other architectures, datasets, client counts, objectives, aggregation strategies, and corruption models remains to be tested.Source: §V · p. 6
  • The study isolates diagnostic signals rather than testing sample selection, aggregation, or complete defence pipelines that act on them.Because corruption type may be unknown beforehand, the discussion motivates monitoring both entropy-based uncertainty and prediction-label loss; validating a combined decision rule remains future work.Source: §V · p. 6

VI. CONCLUSION

  • For the studied persistent random flips, loss separates samples while uncertainty stays at chance; for image noise, expected entropy is modestly more informative than loss.The conclusion notes that label corruption reduces clean-test accuracy by up to 39 points, whereas Gaussian image noise has little adverse effect; it calls for testing harder datasets and integrating sample-level information into federated detection and aggregation.Source: §VI · p. 6

APPENDIX A FULL DETECTION GRIDS

The appendix tables report detection AUC across the corruption sweep and per-client robustness checks. Their captions specify different aggregation and stratification dimensions for within-client and per-client results.

APPENDIX A FULL DETECTION GRIDS

  • Table IV reports within-client per-sample AUC by corruption type, M, and q, with each cell averaging α×seed runs.Compare signal AUC across corruption types and settings within this grid.Source: Table IV · p. 8, Appendix A · p. 8
  • Table V reports per-client AUC by Dirichlet α, averaged over the M×q×seed grid.Compare per-client AUC across the listed α values.Source: Table V · p. 8, Appendix A · p. 8
  • Table VI reports per-client AUC by corruption type, M, and q, with each cell averaging α×seed runs.Compare per-client detection AUC across corruption types and corruption settings.Source: Table VI · p. 8, Appendix A · p. 8
  • Table VII reports seed-specific grid averages of within-client image-noise AUC, with each row averaging 36 M×q×α configurations.Differences are computed before rounding.Source: Table VII · p. 8
Loading 2609.31454v1…