Source-linked AI summary
FOCUS: Dealing with Label Quality Disparity in Federated Learning
Yiqiang Chen, Xiaodong Yang, Xin Qin, Han Yu, Biao Chen, Zhiqi Shen
TL;DR
FL enables privacy-preserving collaboration, but existing approaches generally do not account for differences in local label quality. FOCUS estimates client-data credibility using server-held benchmark samples and weights client contributions accordingly; experiments report improved handling of noisy-label clients, while concept drift remains unaddressed.
Problem
Existing FL approaches generally do not account for local label-quality disparities, which can arise from annotator skill differences, bias, or tampering.
Source: §1 · p. 1Method
Results
FOCUS detects noisy-label clients and reduces their impact more effectively than existing FL approaches in synthetic and real-world evaluations; on the real-world dataset, it outperformed FedAvg in accuracy by 7.24%.
Source: §1 · p. 2, §5.3 · p. 6Takeaways & Limitations
FOCUS provides a credibility-weighted alternative to equal client aggregation for addressing label-quality disparity in cross-silo FL.
Source: §1 · p. 1The label-quality inference can be incorrect under potential concept drift, which the paper does not address.
Source: §4.1 · p. 3
Abstract
from arXiv · showhide
Ubiquitous systems with End-Edge-Cloud architecture are increasingly being used in healthcare applications. Federated Learning (FL) is highly useful for such applications, due to silo effect and privacy preserving. Existing FL approaches generally do not account for disparities in the quality of local data labels. However, the clients in ubiquitous systems tend to suffer from label noise due to varying skill-levels, biases or malicious tampering of the annotators. In this paper, we propose Federated Opportunistic Computing for Ubiquitous Systems (FOCUS) to address this challenge. It maintains a small set of benchmark samples on the FL server and quantifies the credibility of the client local data without directly observing them by computing the mutual cross-entropy between performance of the FL model on the local datasets and that of the client local FL model on the benchmark dataset. Then, a credit weighted orchestration is performed to adjust the weight assigned to clients in the FL model based on their credibility values. FOCUS has been experimentally evaluated on both synthetic data and real-world data. The results show that it effectively identifies clients with noisy labels and reduces their impact on the model performance, thereby significantly outperforming existing FL approaches.
1 Introduction
Federated learning supports privacy-preserving collaboration in ubiquitous systems, but label-quality disparities remain a challenge, especially in healthcare. FOCUS weights client contributions by credibility, and experiments report better handling of noisy-label clients than existing FL approaches.
1 Introduction
- End-Edge-Cloud systems collect data through end devices, perform distributed computing at the edge, and coordinate cloud decision support.The growth of these systems enables collection and processing of personal data, raising privacy concerns.Source: §1 · p. 1
- Label-quality disparity remains an open FL challenge, as existing approaches assume similar label quality and may treat client models equally.Annotator skill differences, bias, or tampering can introduce noise; a noisy client can negatively affect the learned model.Source: §1 · p. 1
- FOCUS uses client-data credibility to weight participation in FL, rather than simply averaging client models as in FedAvg.It maintains benchmark samples on the server and uses a Twin Network contrastive loss to measure credibility.Source: §1 · p. 1
- FOCUS was evaluated on a synthetic activity-recognition dataset with tampering in some clients and a real-world hospital dataset for Parkinson’s Disease symptoms.The introduction describes the synthetic evaluation but does not specify its detailed setup.Source: §1 · pp. 1–2
- Experiments report that FOCUS detects noisy-label clients and reduces their impact on FL model performance more effectively than existing approaches.This result is reported for both synthetic and real-world evaluations.Source: §1 · p. 2
2 Related Work
Prior work addresses label noise through data sanitization or noise-tolerant learning, but the described methods have limitations for federated settings. The paper identifies label-noise mitigation under FL as an open research gap.
2 Related Work
- Label noise is common in machine learning, especially deep learning on large datasets, and methods to address it fall into two categories: data-level and algorithm-level approaches.Source: §2 · p. 2
- Data-level methods sanitize noisy labels, but federated sanitization was reported vulnerable to data-poisoning attacks.Examples include provisional-label generation and Byzantine-robust aggregation against label-flipping attacks [Xie et al., 2019]; federated sanitization vulnerability was reported by [Koh et al., 2018].Source: §2 · p. 2
- Algorithm-level methods train noise-tolerant models, but the described approaches require raw data and therefore do not directly apply in FL.Examples include a weighted surrogate loss and meta-learning with synthetic noisy labels [Zhang et al., 2019].Source: §2 · p. 2
- A data-sharing strategy for non-IID clients can require more communication and risk diluting client information, while EMD calculation requires raw client data.The cited work links non-IID client data to poor global models [Zhao et al., 2018].Source: §2 · p. 2
- The paper states that no published work had addressed mitigating label-noise impact under FL, to the authors’ knowledge.Source: §2 · p. 2
3 Traditional FL Model Training
Traditional FL distributes training data across clients, which train local models on their own data. The server collects client updates and aggregates them into a global model; FedAvg is presented as a widely used aggregation method.
3 Traditional FL Model Training
- Clients train local models by minimizing a loss on their own subsets of the distributed training data.The data for client k are denoted Dk = (Xk, Yk).Source: §3 · p. 2
- The paper uses a CNN architecture for exposition and identifies cross-entropy as the classification model’s objective function.Source: §3 · p. 2
- The server collects client model updates and aggregates them to form the new global FL model.Source: §3 · p. 2
- FedAvg is identified as the most widely used FL aggregation method and is introduced with an aggregation equation.Source: §3 · p. 2
4 The Proposed FOCUS Approach
FOCUS measures client-label credibility by comparing global-model performance on local data with local-model performance on server benchmarks, then uses credibility to weight aggregation. Its design relies on accurately labeled benchmark samples and does not address concept drift.
4 The Proposed FOCUS Approach
- FOCUS evaluates the global model on each client’s local data and each local model on server benchmark data, then uses their mutual cross-entropy to estimate label credibility.The server uses the resulting credibility values in weighted FedAvg.Source: §4 · p. 2, §4.1 · p. 3
4.1 Client Label Noise Measurement
- Accurately labeled benchmark samples are required; the paper does not address cases where concept drift makes the benchmark-based inference unreliable.The server maintains a small benchmark set, and adopters may need field specialists to ensure its labels are accurate.Source: §4.1 · pp. 2–3
- A small mutual cross-entropy suggests similar client and benchmark distributions, while a large value may indicate distribution differences and noisy client labels.A medium value is inconclusive: poor local-model performance may reflect insufficient client data, while poor global-model performance may reflect noisy labels from other clients.Source: §4.1 · p. 3
4.2 Opportunistic FL Model Update
- The credibility measure is used to adjust FedAvg aggregation weights according to client credibility.The method combines credibility with the amount of client data when weighting updates.Source: §4.1 · p. 3, §4.2 · p. 4
- FOCUS uses the latest credibility values from the preceding round for opportunistic updates, and its convergence is guaranteed if FedAvg converges.The credibility calculation depends on both local and global models at the current round.Source: §4.2 · p. 4
Communication Cost
- Compared with FedAvg, FOCUS additionally transmits each client’s global-model performance value on its local dataset.The described process uses two communications per round: global-model broadcast and client model-update submission.Source: Communication Cost · p. 4
5 Experimental Evaluation
FOCUS was evaluated against FedAvg on synthetic USC-HAD data and real-world PD-Tremor data, using accuracy, client weights, and training loss to assess its performance under differing label quality.
5.2 Evaluation on the Synthetic Dataset
- USC-HAD was split into five parts, with one randomly selected as the server benchmark and the other four assigned to clients; one client’s labels were randomized for the noisy scenario.The Normal scenario used correctly labeled data at all four clients; the Noisy scenario used one noisy client.Source: §5.2 · p. 4
5.3 Evaluation on the Real-world Dataset
- FOCUS outperformed FedAvg by 5.82% in accuracy on noisy USC-HAD data and by 7.24% on PD-Tremor, significantly outperforming FedAvg across both datasets.The reported comparisons are shown for USC-HAD in Figure 2 and PD-Tremor in Figure 5.Source: §5.2 · p. 5, Figure 2 · p. 5, §5.3 · p. 6
5.2 Evaluation on the Synthetic Dataset
- On USC-HAD, FOCUS and FedAvg had equal accuracy in the Normal scenario, while FOCUS outperformed FedAvg by 5.82% in accuracy in the Noisy scenario.FOCUS assigned the noisy client a significantly reduced weight while assigning almost equal weights to normal clients with identical data distributions.Source: §5.2 · p. 5, Figure 2 · p. 5, Figure 3 · p. 5
- FOCUS converged faster than FedAvg in both scenarios, although both methods took longer to converge with noisy data than with normal data.The authors attribute FOCUS’s reduced sensitivity to noisy data to the noisy client receiving less weight during server aggregation.Source: §5.2 · p. 5, Figure 4 · p. 5
5.3 Evaluation on the Real-world Dataset
- On PD-Tremor, FOCUS outperformed FedAvg by 7.24% in accuracy, and the lower-tier hospital received a smaller weight, indicating lower-quality data with noisy labels.The three hospitals were treated as FL clients; two were top-tier and one was lower-tier.Source: §5.3 · pp. 5–6, Figure 5 · p. 6, Figure 6 · p. 6
6 Conclusions and Future Work
FOCUS addresses label-quality disparity in federated learning by estimating client data quality without accessing raw data and weighting aggregation by credibility. The authors identify distinguishing malicious label attacks from genuine annotation difficulties as future work.
6 Conclusions and Future Work
- Label quality disparity remains an open federated-learning challenge, with noisy client labels able to corrupt the learned model.Source: §6 · p. 6
- FOCUS estimates client label quality without accessing raw data, then uses credibility to modify FedAvg aggregation.The authors report that experiments on synthetic and real-world data demonstrate an advantage over FedAvg.Source: §6 · p. 6
- Distinguishing clients who fake labels from those facing genuine difficulty remains an open issue affecting how each should be treated.Source: §6 · p. 6