PaperlayerView exact revision on arXiv

Source-linked AI summary

FedCC: Towards Addressing Label Distribution Skews in Distillation-Based Federated Learning

Wenxuan Ye, Onur Ayan, Xueli An, Georg Carle

arXiv:2608.23031v1cs.LGcs.AI

TL;DR

Label skew biases client predictions in distillation-based FL, and unlabeled public data prevent reliable server calibration. FedCC adds an unknown class and calibrated pseudo-labeling to represent uncertainty, achieving 67.3% accuracy in the harshest ten-class setting while baselines fall near random guessing.

  • Problem

    Label skew biases client models toward majority classes, while the absence of public-data ground truth prevents server calibration and distorts supervision.

  • Method

    FedCC frames skew as an open-set problem, adds an unknown class for minority and missing classes, and uses calibrated pseudo-labels with adaptive uncertainty weighting.

  • Results

    67.3% accuracy is achieved when each CIFAR-10 client holds only one class, while baselines collapse to near-random guessing of at most 12.7%.

  • Takeaways & Limitations

    FedCC consistently outperforms current FL methods, with its advantage widening as label skew becomes more severe.

Abstract

from arXiv · show

Federated Learning (FL) enables distributed clients to collaboratively train models without sharing raw data, making it promising for leveraging massive devices in communication networks. In distillation-based FL, each client applies its local model on an unlabeled public dataset, and shares only prediction results with the server. While heterogeneous local data introduces label distribution skew, thus biasing client models toward majority classes and leading to potentially inaccurate predictions. The lack of ground-truth labels in the public dataset hampers the server's ability to calibrate predictions, which ultimately degrades overall performance. To address this, we propose FedCC, a simple and effective algorithm for mitigating client misclassification. Instead of being forced to classify and risking error propagation, clients are allowed to tag ambiguous samples as 'unknown'. This additional class, together with calibrated pseudo-labels on the public data, balances confidence in majority classes against uncertainty in under-represented ones. Extensive experiments demonstrate that FedCC significantly outperforms existing methods, especially under severe label skew. In the extreme scenario where each client holds samples from only one of ten classes, FedCC achieves 67.3% accuracy, while baselines collapse to near-random results.

I. INTRODUCTION

Distillation-based FL reduces communication overhead and supports heterogeneous models, but unlabeled public data and skewed client distributions bias predictions. FedCC addresses this by letting clients mark ambiguous samples as unknown and using calibrated pseudo-labels, improving performance under severe skew.

  • I. INTRODUCTION: Distillation-based FL exchanges prediction vectors from public data, reducing communication overhead and supporting heterogeneous client architectures.Clients run local models on public samples, while the server aggregates their prediction vectors.
  • I. INTRODUCTION: Label distribution skew overfits local models to majority classes, while unlabeled public data prevents server-side logit calibration and degrades global performance.Existing parameter-based remedies assume access to client model weights, which are unavailable when only logits are exchanged.
  • I. INTRODUCTION: FedCC frames skew as an open-set problem and adds an unknown class to absorb minority and missing classes.Clients can acknowledge ambiguous inputs instead of issuing potentially unreliable class predictions.
  • I. INTRODUCTION: FedCC incorporates unlabeled data into the local objective and generates calibrated pseudo-labels that balance majority-class confidence with uncertainty.The method uses an adaptive weight for each client and sample, and can be interpreted as adding a regularizer that mitigates overfitting.
  • I. INTRODUCTION: 67.3% accuracy is achieved in the extreme ten-class scenario, while baselines fail to generate meaningful predictions.Across CIFAR-10, CIFAR-100, and TinyImageNet, the performance gap grows as label skew becomes more severe.

II. RELATED WORKS

Related work addresses skew through parameter aggregation, regularization, calibration, client weighting, clustering, or hard-label communication. These approaches face limitations when public data are unlabeled and only sparse prediction signals are exchanged.

  • II. RELATED WORKS: Parameter-based FL methods use regularization, variance reduction, gradient filtering, logit rescaling, or prediction calibration to address label skew.Examples include FedProx, FedSAM, FedRS, FedLC, FedMR, and FedVLS.
  • II. RELATED WORKS: FedConcat, FedDF, and Co-Boosting pursue alternatives through encoder concatenation, client reweighting, synthetic data, or ensemble updates.These methods depart from simple model averaging or combine synthetic-data generation with aggregation.
  • II. RELATED WORKS: Labeled public datasets or large-scale synthesis introduce annotation and privacy concerns, while unlabeled public data complicate bias correction.The absence of ground-truth labels limits direct calibration of client predictions.
  • II. RELATED WORKS: Unlabeled-data methods retain high-confidence predictions, cluster clients, or communicate hard labels, but may struggle to filter misinformation from local biases.These methods emphasize reliable local knowledge but operate with limited supervision.
  • II. RELATED WORKS: The distillation protocol converts client logits into probabilities, aggregates them into an ensemble teacher, and synchronizes models through KL divergence.Clients then refine their models on private data before sharing predictions in the next round.

IV. APPROACH

FedCC targets client-side misclassification as a route to improving global generalization. Its approach is organized around client-level learning and the overall federated training pipeline.

  • IV. APPROACH: FedCC targets client-side misclassification and seeks to improve global generalization by lowering local errors.The approach focuses on a single client when presenting local notation.
  • IV. APPROACH: Sections IV-A and IV-B detail the client-level algorithm, while Section IV-C outlines the overall federated training pipeline.Client indices are omitted in the single-client formulation for notational brevity.
  • IV. APPROACH: The method uses simplified notation such as f, θ, and σ instead of client-indexed forms during the single-client presentation.This notation applies when the discussion focuses on one client.

A. Local learning objective

FedCC augments the local learning objective with unlabeled-data supervision and an auxiliary unknown class. Its pseudo-label construction assigns uncertainty to samples that are unreliable for a client’s represented classes.

  • A. Local learning objective: Classical self-training expands labeled data with model-generated pseudo-labels and re-optimizes on the combined dataset.The model assigns each unlabeled sample the class with maximum posterior probability.
  • A. Local learning objective: FedCC combines labeled and unlabeled cross-entropy losses, with λ balancing the two terms.Ll(θ) and Lu(θ) denote the labeled and unlabeled losses, respectively.
  • A. Local learning objective: Severe minority under-representation produces poorly calibrated posteriors that drift toward majority classes, while missing classes are unrecognized.The initial model is fitted on the client’s labeled data and lacks training exposure to absent classes.
  • A. Local learning objective: FedCC adds an auxiliary unknown class as a catch-all for samples with unreliable predictions.The augmented label space is C ∪ {u}, with probability mass allocated across known classes and the unknown class.
  • A. Local learning objective: Samples outside a client’s sufficiently represented classes are intended to receive the unknown label during pseudo-label assignment.The ideal assignment distinguishes samples from majority classes from all others.

B. Weight design

FedCC derives an adaptive weighting factor from entropy-based measures of majority-class bias, unknown-class uncertainty, and sample-level confidence. This factor preserves probability for the unknown class during pseudo-labeling, limiting bias-driven errors.

  • B. Weight design: Entropy-only weights quantify majority-class concentration and uncertainty over under-represented classes without using class probabilities.The method uses entropy because it reflects uncertainty and reduces privacy-leakage risk.
  • B. Weight design: After pseudo-label generation, each client optimizes its local objective to obtain parameters used for server-shared predictions.The pipeline’s local update follows pseudo-label generation.
  • B. Weight design: αbase combines majority-class concentration and unknown-class uncertainty through their ratio, producing a client-specific trade-off in [0, 1].The two weights vary across clients according to their local class distributions.
  • B. Weight design: Sample-specific entropy reduces the base weight when an unlabeled instance has low confidence under the pretrained model.The final factor is α(xu) := wl − wxu.
  • B. Weight design: α reserves non-zero probability for the unknown class instead of forcing ambiguous inputs into majority classes.This deferral is intended to curb early minority-sample mislabeling and prevent cascading errors during later rounds.

C. Federated training pipeline

FedCC adds a local pseudo-label refinement step before clients upload temperature-scaled predictions. The server confidence-weights and aggregates these predictions, removes the unknown-class component, and redistributes the resulting soft targets for the next round.

  • C. Federated training pipeline: Each communication round begins with clients downloading prior server soft targets and synchronizing their local models.The downloaded targets are used before adaptation to each client’s private labeled data.
  • C. Federated training pipeline: FedCC generates pseudo-labels and optimizes round-specific client parameters before evaluating the public dataset.This is an additional refinement step compared with schemes that share predictions immediately.
  • C. Federated training pipeline: Clients upload temperature-scaled prediction vectors whose last entry represents the unknown class.The vectors are produced for all public samples.
  • C. Federated training pipeline: The server normalizes client contributions using known-class confidence and aggregates the received prediction vectors.The preliminary client weight is 1 − ŷu(k)(xu).
  • C. Federated training pipeline: The server zeroes the unknown-class component, renormalizes the remaining classes, and broadcasts the soft targets to start the next round.After T rounds, an ensemble selects the most probable label among the original classes.

A. Experimental details

Experiments evaluate FedCC on CIFAR-10, CIFAR-100, and TinyImageNet under multiple label-skew partitions against parameter-based and distillation-based baselines. The evaluation uses shared unlabeled public data and reports repeated-seed accuracy.

  • A. Experimental details: CIFAR-10, CIFAR-100, and TinyImageNet are split into public and private datasets to evaluate classification under label skew.The public portion contains 10% of training samples and has its labels removed.
  • A. Experimental details: FedCC is compared with nine parameter-based and six distillation-based FL methods under a heterogeneous-model setup.Clients are split evenly between ResNet14 and ResNet20 for distillation-based FL.
  • A. Experimental details: Table I reports performance across various label-skew scenarios, with the comparison focused on accuracy.The experiments use multiple skew partitions, including Dirichlet and fixed-class settings.
  • A. Experimental details: 10 clients, 15 global epochs, and 20 local epochs per client-training step define the main training configuration.Adam uses batch size 64 and learning rate 0.001; temperature τ is 1, λ is 0.2, and δ is 10^-3.
  • A. Experimental details: Class bias labels a class as majority when p_c(k) > 2%, minority when 0 < p_c(k) ≤ 2%, and absent when p_c(k) = 0.Reported accuracies are based on three experiments with different random seeds.

B. An overall comparison

FedCC outperforms existing methods across severe label-skew scenarios, with especially strong performance when each client holds only one class. Its local-model evaluation also shows improved performance on global and local test data.

  • 67.3% accuracy is achieved by FedCC when each CIFAR-10 client holds samples from only one class, while baselines reach at most 12.7%.The results describe baseline performance as near-random in this pathological partition.
  • At least 3% points separate FedCC from current FL methods under heavy skew, with advantages often exceeding 10%.The performance gap widens as label skew intensifies.
  • FedCC consistently outperforms all baselines on local test data and significantly improves performance on global test data.The global-test evaluation treats the unknown class as valid when the true label is outside the majority classes.

C. Performance visualization

FedCC’s visualizations indicate improved decision margins and better-separated representations under severe label skew. Its client embeddings isolate familiar classes from unknown regions, while aggregation recovers clusters for all classes.

  • FedCC shifts the decision-margin distribution rightward to a positive mean, while the best competing method remains negative.Figure 2 uses margin histograms with a logarithmic y-axis under two challenging partitions.
  • FedCC client embeddings isolate the client’s own class from the unknown region, and aggregated embeddings recover distinct clusters for all classes.The representations are visualized with T-SNE after PCA whitening in the single-class-per-client setting.
  • Baseline local representations collapse along class-specific axes, while averaging their misaligned feature spaces yields weak global performance.FedCC instead ensembles soft predictions after training with its unknown-class objective.

VI. CONCLUSION

FedCC addresses client misclassification in distillation-based FL by adding an auxiliary unknown class for ambiguous predictions. The paper reports consistent gains in local and global performance and emphasizes its suitability for heterogeneous communication networks.

  • FedCC adds an auxiliary unknown class so clients can defer ambiguous predictions, reducing overconfidence on majority classes and suppressing noisy updates.The method remains based on lightweight logit exchanges.
  • FedCC consistently outperforms state-of-the-art baselines in both local and global performance.
  • FedCC’s logit-based communication and robustness make it suited to networks with heterogeneous compute power, bandwidth, and data distributions.
Loading 2608.23031v1…