Source-linked AI summary
Towards Fairness-Aware Federated Learning
Yuxin Shi, Han Yu, Cyril Leung
TL;DR
Most FL work centers the controller’s interests, while unfair treatment can discourage client participation and threaten ecosystem sustainability; no comprehensive FAFL survey had addressed the field. This paper reviews assumptions and fairness notions, organizes approaches by FL stage, and discusses evaluation metrics and future directions.
Problem
Most FL work focuses on the central controller, and the literature lacked a comprehensive survey of FAFL despite fairness concerns affecting client participation and ecosystem sustainability.
Source: §I · p. 2Method
The paper reviews FAFL assumptions and fairness notions, organizes approaches around major FL steps, and discusses evaluation metrics and future directions.
Source: §I · p. 2Results
The review proposes a taxonomy covering client selection, optimization, contribution evaluation, and incentive mechanisms, and summarizes challenges in each step.
Source: §I · p. 2Takeaways & Limitations
The paper identifies promising future research directions and says progress in interdisciplinary FAFL requires collaboration among researchers and industry practitioners.
Source: §VII · p. 14
Abstract
from arXiv · showhide
Recent advances in Federated Learning (FL) have brought large-scale collaborative machine learning opportunities for massively distributed clients with performance and data privacy guarantees. However, most current works focus on the interest of the central controller in FL,and overlook the interests of the FL clients. This may result in unfair treatment of clients that discourages them from actively participating in the learning process and damages the sustainability of the FL ecosystem. Therefore, the topic of ensuring fairness in FL is attracting a great deal of research interest. In recent years, diverse Fairness-Aware FL (FAFL) approaches have been proposed in an effort to achieve fairness in FL from different perspectives. However, there is no comprehensive survey that helps readers gain insight into this interdisciplinary field. This paper aims to provide such a survey. By examining the fundamental and simplifying assumptions, as well as the notions of fairness adopted by existing literature in this field, we propose a taxonomy of FAFL approaches covering major steps in FL, including client selection, optimization, contribution evaluation and incentive distribution. In addition, we discuss the main metrics for experimentally evaluating the performance of FAFL approaches, and suggest promising future research directions towards FAFL.
I. INTRODUCTION
Fairness in federated learning must account for both server and client interests: client treatment can affect participation and model generalization. The paper surveys FAFL assumptions, fairness notions, pipeline approaches, evaluation metrics, and future directions.
I. INTRODUCTION
- Server-focused selection can exclude weaker clients, limiting their access to suitable models and incentives while potentially reducing model generalization.Existing methods often prioritize convergence speed or final model performance.Source: §I · p. 1
- Optimization can bias predictions on individual clients when the server’s loss distribution fails to capture the diversity of the distributed training set.Source: §I · p. 1
- Equal aggregated-model rewards overlook inconsistent client-update quality, potentially discouraging high-quality clients through the free-rider issue.Rewards may compensate clients for computation, battery power, and communication resources.Source: §I · p. 1
- Contribution evaluation estimates each client’s value without exposing private data and can inform both client selection and incentive schemes.It may use self-reported data quantity or quality, or observed marginal model improvement.Source: §I · p. 2
- The survey examines common FL assumptions and diverse fairness notions, then proposes a taxonomy organized around major FL steps and their challenges.The authors describe it as the first such FAFL taxonomy to their knowledge.Source: §I · p. 2
- The paper reviews experimental evaluation metrics and outlines future research directions, analyzing current limitations and potential ways forward.Source: §I · p. 2
II. COMMON ASSUMPTIONS
The paper distinguishes widely adopted FL assumptions from scenario-specific simplifying assumptions, then surveys fairness notions by what they measure and whose interests they serve.
A. Fundamental Assumptions: These assumptions are es-
- FL assumes local data are sensitive and actors are self-interested and rational, while simplifying assumptions enable particular approaches rather than applying field-wide.Examples include trusting clients’ self-reports, assuming invitations are accepted, and assuming client data remain unchanged during training.Source: A. Fundamental Assumptions: These… · pp. 2–3
III. NOTIONS OF FAIRNESS IN FAFL
- Performance distribution fairness seeks uniform model performance across clients, while good-intent fairness minimizes the worst group’s loss to avoid sacrificing it for others.Performance uniformity is measured using mean and variance of accuracy; good-intent fairness aims to reduce accuracy variance across groups.Source: §III · p. 3
- Group fairness targets disparities in decisions, whereas selection fairness increases participation chances for under- or never-represented clients.Group disparities may be measured by demographic parity or equal opportunity; selection fairness can use sampling constraints or probabilities tied to client cost and potential contribution.Source: §III · p. 3
- Contribution fairness aligns client payoff with model contribution; regret and expectation fairness address inequity in incentive payouts over time.Regret fairness accounts for waiting time before full payoff, while expectation fairness concerns inequity as rewards are gradually paid.Source: §III · p. 3
- Good-intent and group fairness protect eventual model users; selection and performance distribution fairness primarily serve clients while also considering server and user interests.Contribution fairness commonly guides monetary and non-monetary incentives; regret and expectation fairness suit rewards derived from future earnings.Source: §III · p. 3
- Fairness notions can also be classified by training step—performance or cooperation—and target entity—individual or group.In FL, groups may be defined by sensitive attributes, class labels, or newly added clients.Source: §III · pp. 3–4
IV. THE PROPOSED TAXONOMY OF FAFL
The taxonomy organizes FAFL work around client selection, optimization, contribution evaluation, and incentives. For selection, approaches balance clients’ participation opportunities against system goals.
IV. THE PROPOSED TAXONOMY OF FAFL
- The taxonomy covers client selection, model optimization, contribution evaluation, and incentive mechanisms, with some approaches spanning multiple aspects.Contribution evaluation also informs whom to invite and how to distribute rewards.Source: §IV · p. 4, Figure 2 · p. 4
A. Fairness in FL Client Selection
- Threshold-based selection can favor high-capability clients, while excluding weaker clients and risking oversampling from particular groups.Such oversampling can bias the global model toward those clients’ data and deteriorate performance; exclusion can also deny weaker clients participation and rewards.Source: A. Fairness in FL Client Selection · p. 4
- Fair selection is not equal-probability selection: it should balance server and client interests while accounting for client heterogeneity.The reviewed approaches either ensure each client a reasonable selection probability or customize models and training procedures.Source: A. Fairness in FL Client Selection · p. 4
1) FL Client Selection Probability Control:
- Long-term participation constraints and an Exp3-based method reserve selection chances for clients, improving final accuracy at the cost of training efficiency.The former uses queuing dynamics to optimize participation rates; the latter calculates selection probabilities each round and can vary its fairness parameter.Source: 1) FL Client Selection Probability… · p. 4
- A combinatorial bandit approach rewards client class distributions while increasing estimated rewards to give less-frequently engaged clients opportunities to participate.Frequently selected clients are treated as more trustworthy and receive higher payoff.Source: 1) FL Client Selection Probability… · p. 4
- A reputation-based policy adds historical behavior and successful transmissions to fairness-constrained selection, limiting highly reputable clients’ selection to make room for others.This addresses earlier methods’ omission of clients’ real-time contributions when setting fairness factors.Source: 1) FL Client Selection Probability… · pp. 4–5
2) Client Customization:
Client customization addresses heterogeneity by adapting models, workloads, or communication handling so clients with limited resources can participate, with distinct trade-offs for each approach.
Customizing Client Models
- Federated Dropout (FD) assigns resource-sized sub-models, then reconstructs client updates for aggregation; compression and smaller models reduce communication and local computation costs.FD drops activations or filters, allowing lower-capability clients to train a pruned sub-model.Source: 2) Client Customization · p. 5, Figure 3 · p. 5
- Adaptive Federated Dropout selects important activation nodes using activation scores, aiming to fit sub-models to clients while speeding convergence and reducing accuracy loss.Unlike FD’s random dropout, AFD uses an activation score map to select nodes.Source: 2) Client Customization · p. 5
- FD and AFD share the same pruned model with all clients in a round, restricting stronger clients; FD also lacks computational savings at inference when the original model is deployed.Ordered Dropout instead distributes capability-specific sub-models and can drop less important units at inference, but this extension is described separately.Source: 2) Client Customization · p. 5
Customizing Client Workloads
- FedProx varies clients’ local epochs to accommodate resources, then uses a proximal term to constrain updates and stabilize training with heterogeneous data.The proximal term keeps local updates closer to the global model and supports aggregation across differing amounts of work.Source: 2) Client Customization · p. 5
Customizing based on Client Network Connectivity
- Poor network conditions can delay or exclude client updates, biasing the global model; ThrowRightAway handles this by ignoring some lost packets and recalculating aggregation.It requests retransmission only for clients classified as having sufficient network conditions; effectiveness depends on accurate classification and honest client reports.Source: 2) Client Customization · p. 6
B. Fairness in Model Optimisation
The survey groups optimization approaches into objective-based and gradient-based methods for addressing uneven client performance. It describes fairness mechanisms alongside robustness and estimation limitations.
1) Objective Function-based Approaches:
- Objective-based methods target fairness constraints in global or local objectives, including uniform accuracy across clients while maintaining similar average accuracy.These approaches primarily address performance distribution fairness by reducing accuracy variance across clients.Source: 1) Objective Function-based Approaches · p. 6
- AFL optimizes for a target distribution formed from client mixtures to protect the worst-performing client, but its guarantees may not hold at large client counts.AFL’s generalization guarantees are reported as achievable only for a small number of clients.Source: 1) Objective Function-based Approaches · p. 6
- q-FFL reweights aggregate loss toward higher-loss devices, with q tuning fairness; it achieves lower accuracy variance and faster convergence than AFL.q-FFL behaves similarly to AFL at large q, and q-FedAvg is a more communication-efficient aggregation approach based on q-FFL.Source: 1) Objective Function-based Approaches · p. 6
- FedMGDA+ uses separate client objectives and Pareto-stationary descent to preserve selected clients’ performance, adding gradient normalization and robustness against inflated-loss attacks.Accurately identifying malicious clients remains open; Ditto instead balances global and local learning through personalized fine-tuning regularized toward the global model.Source: 1) Objective Function-based Approaches · p. 6
- AgnosticFair uses kernel reweighting and an agnostic demographic-parity constraint for data shift, but requires prior knowledge to choose the reweighting function.FCFL combines worst-client loss minimization with client-level group-fairness constraints, but does not guarantee reasonable global disparity.Source: 1) Objective Function-based Approaches · p. 7
2) Gradient-based Approaches:
- Gradient-based approaches seek fairer client-update averages; FedFV projects gradients to reduce conflicts among selected clients and between selected and unselected clients.FedFV balances fairness, accuracy, and efficiency, but its prior-round external-gradient estimates may become outdated and incompatible with current updates.Source: 2) Gradient-based Approaches · p. 7
- Applying estimated gradients directly to current global updates may cause model divergence, motivating more reliable estimation coupled with client selection.Source: 2) Gradient-based Approaches · p. 7
C. Fairness in Contribution Evaluation
Contribution evaluation estimates each client’s importance without accessing local data, and its values can inform client selection and reward distribution. The survey groups approaches into self-reported information and individual evaluation, noting assumptions that can limit both.
C. Fairness in Contribution Evaluation
- Contribution evaluation estimates each client’s importance without accessing its local data, and its values can inform client selection and reward distribution.The survey identifies five categories: self-reported information, individual evaluation, utility game, Shapley value, and empirical methods.Source: C. Fairness in Contribution Evaluation · p. 7
1) Self-reported Information:
- Self-reported approaches use clients’ reports on dataset quality, quantity, collection costs, and committed computational or communication resources to assess contributions or set incentives.Examples include server-assigned ratings, contract items, truthful price reports, and auction bids used to estimate potential contribution and select winners.Source: 1) Self-reported Information · p. 7
- These methods assume clients can reliably assess their circumstances and report truthfully, an assumption the survey says may not hold in practice.Source: 1) Self-reported Information · p. 7
2) Individual Evaluation:
- Individual evaluation measures client performance on specific tasks rather than the overall FL model, including reputation schemes that track historical reliability and contribution.Centralized schemes may use validation accuracy, update similarity, or loss-based behavior; they require an accurate, balanced validation dataset, while decentralized schemes may rely on attack detection, computation time, or gradients.Source: 2) Individual Evaluation · pp. 7–8
- Individual evaluations also use bids, resource quality, and pairwise mutual assessments, but commonly assume trustworthy parties and that model similarity signals contribution.Under non-i.i.d. data, dissimilar updates can provide valuable complementary knowledge, so similarity-based judgments may undermine perceived fairness; some approaches also face cheating or historical-storage burdens.Source: 2) Individual Evaluation · p. 8
3) Utility Game:
Utility-game approaches adapt profit-sharing rules to evaluate client contributions, while Shapley value averages marginal contribution across coalitions. The survey describes efforts to improve Shapley-based evaluation efficiency, alongside remaining computational and estimation limits.
3) Utility Game:
- Utility-game approaches map participant-generated utility to rewards through egalitarian, marginal-gain, or marginal-loss schemes, with marginal schemes making payoff depend on joining order.Source: 3) Utility Game · p. 8
- Marginal loss is the most commonly used FL scheme, with cited approaches applying it to hierarchical FL or single-training contributions to reduce communication and computation overhead.These approaches target contribution fairness, but simple marginal loss is relative to the participating cohort and does not express the actual value of a client’s local data.Source: 3) Utility Game · p. 8
4) Shapley Value:
- Shapley value averages a client’s marginal contribution across subsets, making its contribution score independent of joining order; its exact computational complexity is O(2^n).The survey describes this as addressing the limitation of relative marginal-loss evaluation, while noting the exponential cost.Source: 4) Shapley Value · pp. 8–9
- Client-based Shapley estimators reconstruct models from gradient updates, with OR calculating once from the final round and MR aggregating round-wise values; TMR weights more accurate rounds and skips the last few rounds.Source: 4) Shapley Value · p. 9
- Permutation sampling, group testing, and guided truncation reduce within-round or between-round Shapley computation, while feature-based VFL evaluation groups features to limit sensitive-feature exposure but remains expensive.The VFL feature-based cost grows exponentially with training data size.Source: 4) Shapley Value · p. 9
5) Empirical Methods:
The survey contrasts empirical contribution estimation with theory-based methods and classifies fair incentive mechanisms as monetary or non-monetary. It describes monetary designs for contribution-based rewards while identifying assumptions and scope boundaries that constrain several approaches.
5) Empirical Methods:
- Empirical contribution methods are presented as alternatives to costly, potentially imprecise data-based counterfactual approaches, but FedCCEA’s simple neural-network architecture limits it to simple FL tasks.FedCCEA uses sampled data size and round-wise accuracy to learn client data quality and contribution, and permits partial participation through desired local-data sizes.Source: 5) Empirical Methods · p. 9
D. Fairness in Incentive Mechanisms
- Fair incentive mechanisms compensate clients for contributions and training costs, and the survey divides them into monetary payoffs and non-monetary model-performance rewards.The cited motivation includes clients’ resource costs, security risks, and possible opportunity costs from sharing a model with competitors.Source: D. Fairness in Incentive Mechanisms · p. 9
1) Monetary Incentive Mechanisms:
- FMore uses bids and resource quality in an auction-style scoring rule, while ψ-FMore adjusts winning probabilities to favor low-score clients; the latter assumes truthful bids and raises privacy concerns in score calculation.FMore’s scoring rule also enables clients to assess whether they are treated fairly, and is presented as potentially encouraging participation.Source: 1) Monetary Incentive Mechanisms · pp. 9–10
- Contribution-fair payment schemes include equal unit pricing for reported data quality and reputation-based reverse auctions, but self-reporting may be untruthful and reputation systems need historical records and bootstrapping.Source: 1) Monetary Incentive Mechanisms · p. 10
- Contract-based incentives reward higher data quality or resource contributions, but surveyed designs assume a single task publisher and do not adequately account for competing publishers.For delayed revenue, FLI addresses regret-distribution and expectation fairness through dynamic payoff sharing that accounts for contribution quality and waiting time.Source: 1) Monetary Incentive Mechanisms · p. 10
2) Non-Monetary Incentive Mechanisms:
Non-monetary incentive mechanisms reward clients with models that vary by contribution, while the evaluation section surveys metrics for measuring model performance and different notions of fairness.
2) Non-Monetary Incentive Mechanisms:
- Model-based, non-monetary rewards suit settings without an incentive budget, feasible future-revenue payouts, or where clients compete over equal rewards.The paper identifies the last case with the free-rider problem.Source: 2) Non-Monetary Incentive Mechanisms · p. 11
- HFFL trains separate models for client levels, but clients within a level may differ in data amount and self-reported characteristics can be falsified.Higher-level clients receive better-performing models.Source: 2) Non-Monetary Incentive Mechanisms · p. 11
- FPPDL gives participants model variants according to contribution, but current approaches lack precise control over the resulting model’s performance.Participants earn transaction points through local credibility and commitment, and use them to download others’ gradients.Source: 2) Non-Monetary Incentive Mechanisms · p. 11
V. FAFL EVALUATION METRICS
- Average validation accuracy and accuracy variance are common model-performance measures, while test accuracy can assess performance on each client.Training time and training-round counts measure framework efficiency rather than model performance.Source: §V · p. 11
- Average variance and standard deviation measure cross-client performance variability, but average variance is outlier-sensitive and provides only relative fairness comparisons.Lower average variance indicates greater fairness under this measure.Source: §V · p. 11
- Distance metrics compare estimated with ground-truth contribution vectors, while Pearson correlation assesses their relationship and is sensitive to outliers and nonlinearity.The vectors are normalized to remove numerical differences and reflect contribution ratios.Source: §V · pp. 11–12
- Jain’s Fairness Index measures uniformity in local performance and can also represent selection or contribution fairness using other client characteristics.Its values range from 1/n, described as most unfair, to 1, described as most fair.Source: §V · p. 12
- Risk difference metrics include demographic parity and equal opportunity; lower risk difference indicates greater fairness.They measure disparity in positive predictions between sensitive and non-sensitive groups, either marginally or conditionally on the true outcome.Source: §V · p. 12
VI. PROMISING FUTURE RESEARCH DIRECTIONS
The survey finds that FAFL research remains nascent and that existing approaches generally address fairness notions separately. It identifies broader use of Jain’s Fairness Index and six research directions as opportunities for further work.
VI. PROMISING FUTURE RESEARCH DIRECTIONS
- FAFL research remains in its infancy, and the survey identifies six promising future research directions.It finds fundamental assumptions A1–A2 common, while simplifying assumptions are relaxed in some specific research areas.Source: §VI · p. 13
- Existing approaches commonly align performance-distribution fairness with optimization, selection fairness with client selection, and contribution fairness with evaluation and incentives.The survey finds that no one approach jointly addresses these fairness notions.Source: §VI · p. 13
- Jain’s Fairness Index can evaluate diverse fairness notions, although most existing use focuses on performance-distribution fairness.The survey suggests applying it to other fairness notions in future work.Source: §VI · p. 13
A. Temporal consideration in contribution evaluation
Contribution evaluation should account for the time clients wait for full payoff when incentives depend on future model revenues.
A. Temporal consideration in contribution evaluation
- When rewards depend on future model revenues, contribution evaluation should account for clients’ waiting time for full payoff as a temporal cost.The paper notes a temporary mismatch between contributions and rewards.Source: A. Temporal consideration in… · p. 13
B. Incentive mechanism for non-monopoly FL settings
Most incentive-mechanism research assumes a single task publisher, but the survey argues that settings with competing publishers warrant further study.
B. Incentive mechanism for non-monopoly FL settings
- A monopoly-publisher assumption excludes client alternatives, and the paper argues that competing publishers merit study because they face pressure to offer reasonable rewards.The survey describes the single-publisher setting as unrealistic and says a non-monopoly setting may inspire fairer incentive schemes.Source: B. Incentive mechanism for… · pp. 13–14
C. Deterrence mechanism design
The survey’s future directions address robustness, incentive control, client–server relations, and trust, while its conclusions organize FAFL approaches, assumptions, fairness notions, evaluation metrics, and research directions.
C. Deterrence mechanism design
- A penalty system could deter malicious clients by making the expected utility gain from an attack negative, with Stackelberg game-based inspection offered as one approach.Existing FAFL approaches mostly assume trustworthy clients, while malicious updates can degrade model performance and defenses may filter rare but informative minority-class updates.Source: C. Deterrence mechanism design · p. 14
D. Model performance control for non-monetary incentive
- Existing non-monetary incentives cannot precisely control the performance of different model variants allocated according to client contributions.The paper identifies this control challenge as a barrier to practical adoption.Source: D. Model performance control for… · p. 14
E. Social norm-based federation formation
- The monopoly assumption limits client choice among servers; social norms could help clients collectively press for equitable treatment and encourage servers to improve their reputations.Source: E. Social norm-based federation… · p. 14
F. Trust building through explainability
- Explainability could help clients understand server decisions and their effects on client interests, but must preserve privacy.The paper connects uncertainty about fair treatment with clients’ future participation decisions.Source: F. Trust building through explainability · p. 14
VII. CONCLUSIONS
- The survey reviews FAFL assumptions and fairness notions, proposes a taxonomy across FL steps, discusses evaluation metrics, and identifies future research directions.It presents the work as a roadmap and calls for collaboration between researchers and industry practitioners.Source: §VII · p. 14