Source-linked AI summary
Collaborative Fairness in Federated Learning
Lingjuan Lyu, Xinyi Xu, Qian Wang
TL;DR
Federated learning improves on standalone training but conventional FL can treat participants equally despite unequal contributions, creating a collaborative-fairness gap. CFFL uses reputation to allocate different models according to contribution quality. Experiments report high fairness, comparable accuracy to Distributed frameworks, and better accuracy than Standalone learning.
Problem
Conventional FL gives participants the same or similar models regardless of contribution, overlooking collaborative fairness despite unequal data and update quality.
Method
CFFL updates participant reputations from validation performance and allocates aggregated updates according to those reputations, producing contribution-based model differences.
Results
CFFL achieves high fairness, comparable accuracy to Distributed frameworks, and higher accuracy than the Standalone framework across benchmark experiments.
Takeaways & Limitations
CFFL demonstrates collaborative fairness and utility by rewarding higher-contribution participants with better-performing models without compromising reported predictive performance.
Takeaways & Limitations
Selecting the reputation threshold is challenging because fairness and accuracy may be inversely affected, and unsuitable values can admit or isolate too many participants.
Abstract
from arXiv · showhide
In current deep learning paradigms, local training or the Standalone framework tends to result in overfitting and thus poor generalizability. This problem can be addressed by Distributed or Federated Learning (FL) that leverages a parameter server to aggregate model updates from individual participants. However, most existing Distributed or FL frameworks have overlooked an important aspect of participation: collaborative fairness. In particular, all participants can receive the same or similar models, regardless of their contributions. To address this issue, we investigate the collaborative fairness in FL, and propose a novel Collaborative Fair Federated Learning (CFFL) framework which utilizes reputation to enforce participants to converge to different models, thus achieving fairness without compromising the predictive performance. Extensive experiments on benchmark datasets demonstrate that CFFL achieves high fairness, delivers comparable accuracy to the Distributed framework, and outperforms the Standalone framework.
1 Introduction
Federated learning addresses the computational and generalization limits of standalone local training, but conventional FL overlooks fairness when participants contribute unequally. CFFL uses reputation to allocate performance-commensurate models while preserving utility.
- Federated learning lets participants jointly train a model while keeping their datasets local.It seeks a global model with better generalizability by leveraging multiple local datasets.
- Conventional FL gives participants the same federated model despite differences in data quality, quantity, and contribution.Unequal update quality can improve or impair model performance, making equal model allocation unfair.
- Contribution-insensitive FL can discourage high-quality data sharing and allow free-riders to benefit without contributing.This may hinder the formation and progress of a healthy FL ecosystem.
- CFFL uses a reputation mechanism to allocate models whose performance is commensurate with participants’ contributions rather than distributing one common model.Reputations evaluate participant contributions and are updated iteratively during learning.
- CFFL achieves the highest fairness while giving its most contributive participant accuracy comparable to Distributed learning and higher than Standalone learning.The framework is designed to achieve fairness without sacrificing model performance.
2 Related Work
Prior FL fairness research includes incentive and profit-sharing schemes that distribute rewards according to participant contributions. Other approaches reward model improvements or jointly account for contributions, costs, and waiting time, while some schemes are difficult to scale.
- Existing FL fairness approaches commonly use incentive schemes to distribute payoffs according to participant contributions.Equal division and individual profit-sharing represent different payoff-allocation rules.
- The Labour Union game bases payoff shares on marginal contributions to collectives formed by predecessors.Its contribution estimate depends on participant sequence.
- The Fair-value game uses marginal loss, while the Shapley game removes participant-order effects when estimating marginal contributions.The Shapley game’s complexity is exponential in the number of participants, limiting practicality for large-scale FL.
- For gradient-based FL, model accuracy is the most relevant basis for rewards because output-agreement rewards require a usually unavailable multi-task setting.Gradient information can serve as a useful data source, but it does not make output agreement directly applicable.
- One proposed scheme pays for marginal model improvements, corrects contribution overestimation, and rewards early model-update submissions.The approach makes the budget for reaching a target model quality level predictable.
- Another approach jointly optimizes participant contributions, costs, and waiting time to support additional fairness notions when distributing payoffs.
3 The CFFL Framework
CFFL allocates different FL model versions according to participant contributions, using reputation to adjust rewards while preserving a contribution–performance relationship. It quantifies fairness as the correlation between participant contributions and final model accuracies.
- Collaborative Fairness: CFFL allocates different versions of the federated model, with performance commensurate with each participant’s contribution.
- Fairness via Reputation: The server updates participant reputations from validation accuracy of uploaded gradients and uses them to allocate aggregated gradients in later rounds.Higher validation informativeness produces higher reputation, while historical reputation is also incorporated.
- Fairness via Reputation: Higher-reputation participants receive more aggregated gradients as a reward in each communication round.
- Quantification of Fairness: Collaborative fairness is measured as the correlation between standalone model accuracies representing contributions and final model accuracies representing rewards.Standalone accuracy is used because participants with higher standalone accuracies empirically contribute more.
- Quantification of Fairness: The fairness coefficient ranges from -1 to 1, with higher values indicating better fairness and negative values indicating poor fairness.
4 Experimental Evaluation
Experiments evaluate CFFL on MNIST and Adult under heterogeneous data, comparing it with Standalone and Distributed baselines across fairness, accuracy, and convergence.
- Datasets: Experiments use MNIST and Adult, with 10% of training examples reserved as validation data.MNIST contains 60,000 training and 10,000 test examples; the balanced Adult dataset contains 23,374 records after an 80–20 train-test split.
- Experimental scenarios: The evaluation simulates heterogeneity through imbalanced data sizes and imbalanced class numbers across participants.Data sizes vary among 5, 10, or 20 participants; class-number imbalance is evaluated on MNIST by assigning progressively more classes to later participants.
- Fairness results: CFFL achieves the highest fairness, while pretraining generally provides slightly higher fairness than no pretraining.DSSGD and FedAvg yield significantly lower fairness because their communication protocols and learning algorithms do not incorporate fairness.
- Accuracy results: 93.25 (93.34)% test accuracy is achieved by CFFL (CFFL*) on MNIST with 20 participants, versus 90.64% for Standalone and 96.26% for FedAvg.Across the reported comparisons, CFFL obtains accuracy comparable to Distributed frameworks and higher than Standalone.
- Individual convergence: CFFL produces different final models for participants while improving individual accuracy over Standalone in both data-size and class-number imbalance scenarios.More obvious accuracy gaps under imbalanced class numbers result in higher fairness; more participants also require longer convergence.
5 Discussions
CFFL addresses free-riding by using validation-based reputation to identify and isolate participants sending random or noisy updates. Its effectiveness depends on selecting a reputation threshold that balances fairness and accuracy.
- Robustness to Free-riders: CFFL identifies and isolates free-riders whose random or noisy gradients produce low empirical utility.Their reputations decline during collaboration until falling below the reputation threshold.
- Robustness to Free-riders: Additional experiments with one random-gradient free-rider found early isolation without affecting accuracy or convergence.
- Choice of Reputation Threshold: The reputation threshold cth creates a lower reputation bound for detecting and isolating free-riders.
- Choice of Reputation Threshold: Threshold selection involves a fairness–accuracy trade-off: values that are too small admit low-contribution participants, whereas values that are too large isolate too many participants.Suitable values were found empirically for different scenarios.
6 Conclusion and Future Work
CFFL introduces reputation-based collaborative fairness by allocating models according to participant contributions and encouraging convergence to different models. Experiments report comparable Distributed accuracy, better Standalone accuracy, and fairness improvements across benchmark settings.
- 6 Conclusion and Future Work: CFFL uses validation-set model performance to update participant reputations and mediate rewards across communication rounds.
- 6 Conclusion and Future Work: CFFL achieves comparable accuracy to two Distributed frameworks and better accuracy than the Standalone framework while addressing fairness and utility.
- 6 Conclusion and Future Work: Figure 1 compares MNIST individual convergence across 5, 10, and 20 participants and five Standalone or CFFL configurations.
- 6 Conclusion and Future Work: Figure 2 compares Adult individual convergence across 5, 10, and 20 participants and the same five framework configurations.