PaperlayerView exact revision on arXiv

Source-linked AI summary

FedDisco: Federated Learning with Discrepancy-Aware Collaboration

Rui Ye, Mingkai Xu, Jianyu Wang, Chenxin Xu, Siheng Chen, Yanfeng Wang

arXiv:2305.19229v1cs.LG

TL;DR

Category distribution heterogeneity makes dataset-size-based aggregation insufficiently informative in federated learning. FedDisco combines dataset size with local-to-global category-distribution discrepancy to determine aggregation weights. It consistently achieves state-of-the-art performance across diverse settings while preserving efficiency, privacy, and modularity.

  • Problem

    Category distribution heterogeneity creates divergent local objectives, while existing methods often use dataset-size-based aggregation without optimizing server-side weights.

  • Method

    FedDisco assigns larger aggregation weights to clients with larger datasets and smaller discrepancy from the global category distribution.

  • Results

    FedDisco consistently achieves state-of-the-art performance across diverse settings, datasets, and participation regimes.

  • Takeaways & Limitations

    FedDisco is privacy-preserving, adds negligible communication and computation cost, and can be incorporated into many existing federated learning methods.

  • Takeaways & Limitations

    The paper mainly studies category-level heterogeneity and suggests extending discrepancy-aware collaboration to feature-level and mask-level heterogeneity.

Abstract

from arXiv · show

This work considers the category distribution heterogeneity in federated learning. This issue is due to biased labeling preferences at multiple clients and is a typical setting of data heterogeneity. To alleviate this issue, most previous works consider either regularizing local models or fine-tuning the global model, while they ignore the adjustment of aggregation weights and simply assign weights based on the dataset size. However, based on our empirical observations and theoretical analysis, we find that the dataset size is not optimal and the discrepancy between local and global category distributions could be a beneficial and complementary indicator for determining aggregation weights. We thus propose a novel aggregation method, Federated Learning with Discrepancy-aware Collaboration (FedDisco), whose aggregation weights not only involve both the dataset size and the discrepancy value, but also contribute to a tighter theoretical upper bound of the optimization error. FedDisco also promotes privacy-preservation, communication and computation efficiency, as well as modularity. Extensive experiments show that our FedDisco outperforms several state-of-the-art methods and can be easily incorporated with many existing methods to further enhance the performance. Our code will be available at https://github.com/MediaBrain-SJTU/FedDisco.

1. Introduction

Category distribution heterogeneity causes clients to optimize toward different local objectives, while prior methods largely adjust local or global models rather than aggregation weights. FedDisco combines dataset size and category-distribution discrepancy in aggregation weights and achieves consistent gains across diverse settings.

  • Category distribution heterogeneity gives clients drastically different label distributions and divergent local optimization directions.
  • Most prior methods adjust local or global models, while few optimize server-side aggregation weights and commonly rely only on dataset size.
  • FedDisco preserves privacy and adds negligible communication and computation costs while remaining compatible with existing methods.
  • FedDisco consistently improves performance over existing federated learning algorithms across balanced and imbalanced distributions, partial and full client participation, and four datasets.
  • Empirical evidence indicates dataset size is not always optimal, whereas discrepancy between local and global category distributions can complement it when setting aggregation weights.
  • FedDisco derives aggregation weights from the theoretical error-bound optimization, using both dataset size and discrepancy.

2. Background of Federated Learning

Federated learning aggregates locally trained client models into a shared global model, but category heterogeneity can make dataset-size-based aggregation suboptimal. The background formalizes the standard training and aggregation process that motivates adjusting aggregation weights.

  • Each federated round broadcasts the global model, performs local SGD training, uploads client models, and aggregates them into the next global model.
  • Category distribution heterogeneity creates different local objectives, motivating attention to aggregation weights alongside local training adjustments.
  • The standard aggregation assigns client weights from local dataset relative sizes, despite neglecting category information.

3. Empirical Observations

Empirical studies show that dataset-size-based aggregation can be far from optimal and that discrepancy better reflects local model performance under category heterogeneity. These observations motivate assigning greater aggregation weight to clients with smaller discrepancy.

  • The experiments use ten CIFAR-10 clients with heterogeneous local category distributions generated under a commonly used Dirichlet setting.
  • At r = 0.5, representing dataset-size-based aggregation, performance can be far from optimal; higher weight for the better local model performs best in the trials.
  • Discrepancy better indicates local model performance than dataset size: similarly sized clients differ substantially in accuracy, while smaller discrepancy aligns with better performance.
  • The observations motivate using discrepancy in aggregation weights because lower discrepancy may indicate a better-performing local model deserving greater weight.

4. Theoretical Analysis

The analysis derives an optimization error bound showing that aggregation weights should reflect both dataset size and local discrepancy. Minimizing this bound yields a concise weight expression that motivates discrepancy-aware aggregation.

  • Optimization error upper bound: Theorem 4.5 bounds the global optimization error under standard assumptions, including bounded objective, unbiased gradients, bounded variance, and discrepancy-aware gradient dissimilarity.The analysis modifies the standard dissimilarity assumption to relate gradient dissimilarity to distribution discrepancy.
  • Effect of aggregation weights: Larger differences between aggregation weights and dataset-size weights loosen the bound, while weights negatively correlated with discrepancy reduce a discrepancy-dependent term.The bound contains terms whose behavior favors weights aligned with dataset size and lower discrepancy.
  • Effect of aggregation weights: The tightest bound can arise from weights correlated with both dataset size and discrepancy, showing dataset size alone need not be optimal.Discrepancy is therefore a complementary indicator for selecting aggregation weights.
  • Upper bound minimization: The proposed analytical weight is obtained by replacing direct upper-bound minimization with a simplified objective involving T1 through T5 and a λT0 term.This transformation avoids the complicated expression produced by directly minimizing the original bound.
  • Upper bound minimization: The resulting Disco weight assigns larger weight to clients with larger datasets and smaller local discrepancy, mitigating standard dataset-size-only aggregation.The expression depends on both n_k and d_k through two constants and is designed to produce a tighter upper bound.

5. FL with Discrepancy-Aware Collaboration

FedDisco integrates local category-distribution discrepancy into federated aggregation weights while retaining standard local training and global aggregation. It computes discrepancy locally, emphasizes larger datasets with lower discrepancy, and supports privacy, efficiency, and modularity.

  • 5. FL with Discrepancy-Aware Collaboration: FedDisco combines standard local training and global aggregation with local discrepancy computation and discrepancy-aware aggregation-weight determination.The method has four steps: local model training, discrepancy computation, weight determination, and global model aggregation.
  • 5.2. Properties: Discrepancy communication occurs only in the first round, and its numerical computation and transmission are negligible relative to model training and communication.FedDisco also avoids the server computation burden and additional fine-tuning required by some competing methods.
  • 5.1. Framework: Each client compares its category distribution with a uniform global target using a metric such as L2 difference or KL divergence, without sharing additional data.The client computes a scalar discrepancy locally, and the server collects only the resulting discrepancy levels.
  • 5.1. Framework: The aggregation rule gives larger weights to clients with larger relative datasets and smaller local discrepancy levels.A ReLU-based formulation uses hyper-parameters to balance dataset size and discrepancy and adjust the resulting weight.
  • 5.2. Properties: The method does not reveal clients’ exact category distributions because discrepancy is computed locally and only scalar discrepancy levels are transmitted.The paper contrasts this with methods that transmit exact category distributions.
  • 5.2. Properties: Because FedDisco focuses on model aggregation, it can combine with existing methods that modify local updates, communication, or aggregation.This aggregation-focused design supports plug-and-play integration with other federated-learning components.

6. Related Works

Related federated-learning methods mainly address heterogeneity through local-model correction, server-side adjustment, or re-weighting for system effects. FedDisco instead uses category-distribution discrepancy with dataset size to refine aggregation and can extend existing methods.

  • 6. Related Works: Federated-learning methods for distribution heterogeneity primarily adjust local training or the global model, reflecting two major directions in prior work.Local methods reduce differences among local models, while global methods seek a better aggregated model.
  • 6. Related Works: Existing methods generally assign aggregation weights from local dataset size, which does not sufficiently distinguish clients’ contributions.The paper identifies this limitation across the related methods discussed.
  • 6. Related Works: FedDisco combines dataset size with local-global category-distribution discrepancy to determine more distinguishing aggregation weights.Its re-weighting targets data heterogeneity rather than the system heterogeneity targeted by FedNova.
  • 6. Related Works: FedDisco has plug-and-play capability and can be incorporated with existing methods to further enhance overall performance.This complements prior approaches that correct local training, fine-tune global models, or compress communication.

7. Experiments

Experiments evaluate FedDisco across diverse datasets, heterogeneity settings, participation regimes, modalities, and hyper-parameter choices. FedDisco consistently improves accuracy over baselines, while also improving modularity and training efficiency.

  • Performance: state-of-the-art accuracy: FedDisco consistently outperforms existing methods across heterogeneous settings and datasets, including the difficult HAM10000 NIID-1 task.The evaluation reports state-of-the-art accuracy across settings, with significantly better performance on HAM10000 under β = 5.
  • Modularity: improvements over baselines: FedDisco acts as a plug-and-play module, consistently improving state-of-the-art baselines and achieving 19.8% relative accuracy improvement over FedAvg on HAM10000.The modularity experiments combine Disco with existing methods and report the largest improvement on the most difficult task.
  • Applicability to partial client participation scenarios: Under partial client participation, Disco improves accuracy by up to 7.32% and reduces communication cost to reach 55% accuracy by up to 65.52%.These results indicate faster training alongside accuracy gains in the partial-participation scenario.
  • Applicability to text-modality scenarios: Disco improves baselines on AG News under full and partial participation, extending its reported benefits to text classification.The text-modality experiments report consistent improvements, with significant gains under partial participation.
  • Applicability to globally imbalanced scenarios: FedDisco remains effective under globally imbalanced distributions and across client numbers, heterogeneity levels, local epochs, and discrepancy metrics.The experiments report improvements across imbalance scenarios and FL argument choices, while four discrepancy metrics achieve similar performance.
  • Ablation Study: Across tested baseline methods, heterogeneity types, and datasets, Disco improves performance for a wide hyper-parameter range, with a = 0.4 ∼ 0.6 and b = 0.1 described as a stable choice.The reported robust range is a ∈[0.2, 0.7] and b ∈[0.05, 0.4].

8. Conclusions

FedDisco addresses category distribution heterogeneity by combining dataset size with discrepancy-aware aggregation weights. It adds negligible communication and computation cost, remains modular, and consistently achieves state-of-the-art performance, while broader heterogeneity extensions remain future work.

  • FedDisco assigns larger aggregation weights to clients with larger datasets and smaller category-distribution discrepancies.The discrepancy value complements conventional dataset-size-based aggregation.
  • FedDisco introduces negligible computation and communication cost and can be incorporated with many existing federated-learning methods.
  • Experiments show that FedDisco consistently achieves state-of-the-art performance.
  • The paper mainly explores category-level heterogeneity, with extensions to feature-level and mask-level heterogeneity left for future work.

A. Methodology

The experiments evaluate FedDisco across globally imbalanced data, multiple baselines, datasets, and category-level performance. Results show consistent baseline improvements, including under severe imbalance, while improving tail-class accuracy without severely harming head classes.

  • Experimental Settings: The evaluation spans four standard datasets and compares FedDisco across two heterogeneity types, with additional globally imbalanced-data experiments.
  • Globally Imbalanced Category Distribution: FedDisco consistently improves baselines under globally imbalanced data, demonstrating applicability when the aggregated global distribution is non-uniform.Table 9 evaluates the setting at imbalance level ρ = 10.
  • Per-Category Performance: FedDisco improves tail-class accuracy by nearly 8% while keeping head and middle performance within 0.75% of FedAvg.

B.3. Partial Client Participation Scenario: Accuracy, Stability and Training Speed

Under partial client participation, Disco-enhanced methods achieve higher and more stable accuracy while reaching the target accuracy in fewer rounds. Additional evaluations examine fairness and compare against equal aggregation weights.

  • Accuracy, Stability and Training Speed: Disco-enhanced methods achieve higher mean accuracy, lower standard deviation, and fewer rounds to reach 55% accuracy under partial participation.These outcomes indicate more stable performance and faster training in the evaluated setting.
  • Additional Dataset Evaluation: FedDisco consistently improves performance on CIFAR-100 under NIID-1, supporting modular gains beyond the primary evaluation setting.
  • Client-Level Fairness: The paper evaluates client-level fairness using variance in test accuracy across clients, where lower variance denotes higher fairness.
  • Equal-Weight Comparison: FedDisco performs significantly better than FedAvg with equal aggregation weights across datasets and heterogeneity types.

B.7. Experiments on Device Heterogeneity

FedDisco is evaluated alongside device heterogeneity and related extensions. It improves accuracy under stragglers and also shows gains in regression and feature-level heterogeneity experiments, while device and distribution heterogeneity can interact.

  • Device Heterogeneity: FedDisco achieves 63.20% accuracy versus FedAvg’s 60.21% under CIFAR-10 straggler heterogeneity.Clients perform randomly sampled numbers of local iterations between 50 and 500 per round.
  • Interaction with Distribution Heterogeneity: Device and distribution heterogeneity can occur concurrently, and stragglers with different discrepancy levels can bias the global model.
  • Broader Applications: A feature-level heterogeneity experiment on FEMNIST reports 78.36% accuracy for FedDisco versus 76.40% for FedAvg.
  • Theoretical Analysis: The theoretical analysis derives an optimization-error bound involving aggregation weights, dataset relative sizes, discrepancy levels, and training constants.

C.4. Further Analysis with Proper Learning Rate

The analysis distinguishes finite-round and asymptotic regimes, showing that the dominant optimization-bound term depends on the learning rate and communication rounds. It then motivates discrepancy-aware aggregation weights that jointly account for dataset size and discrepancy to tighten the bound.

  • For finite T, a larger learning rate makes T5 dominant, whereas a smaller learning rate makes T2 dominant in the optimization bound.The analysis links these regimes to the relative sizes of A, T2, and T5.
  • As T →∞, the optimization upper bound approaches 0 and reaches a stationary point, matching previous convergence results.
  • Because communication rounds should remain finite in practice, the paper explores aggregation weights that produce a tighter optimization upper bound.The paper notes that excessively large T creates communication burden, despite asymptotic convergence.
  • FedDisco derives the aggregation weights by reformulating bound minimization into an objective that maximizes T0 while minimizing T1 + T2 + T3 + T4 + T5.The constrained optimization yields a stationarity equation and a concise expression for p_k containing constants a and b.
  • Theoretical analysis indicates that aggregation weights should depend on both dataset size n_k and local discrepancy d_k, not dataset size alone.Larger dataset size and smaller discrepancy are associated with larger aggregation weight in the proposed adjustment.
Loading 2305.19229v1…