PaperlayerView exact revision on arXiv

Source-linked AI summary

Data Poisoning Attacks Against Federated Learning Systems

Vale Tolpegin, Stacey Truex, Mehmet Emre Gursoy, Ling Liu

arXiv:2007.08432v2cs.LGcs.CRstat.ML

TL;DR

Federated learning keeps data local but lacks centralized validation, leaving global models vulnerable to malicious participants. The paper studies targeted label-flipping attacks and proposes an update-based defense, finding substantial targeted degradation and clear separation of malicious and honest updates across 2-20% malicious participation.

  • Problem

    Federated learning must preserve local data while maintaining trustworthy global models despite potentially malicious participants and no centralized data validation.

  • Method

    The paper evaluates targeted label-flipping attacks under constrained local-data manipulation and develops a PCA-based defense using model updates to identify malicious participants.

  • Results

    The attacks substantially degrade global performance, can target selected classes, and the defense separates malicious from honest updates across malicious-participant rates of 2-20%.

  • Takeaways & Limitations

    Poisoning impact increases with malicious-participant proportion and later-round availability, while the proposed defense helps the aggregator identify and block malicious participants.

  • Takeaways & Limitations

    The defense was designed and tested against label-flipping attacks, while broader FL scenarios and more sophisticated poisoning strategies remain future work.

Abstract

from arXiv · show

Federated learning (FL) is an emerging paradigm for distributed training of large-scale deep neural networks in which participants' data remains on their own devices with only model updates being shared with a central server. However, the distributed nature of FL gives rise to new threats caused by potentially malicious participants. In this paper, we study targeted data poisoning attacks against FL systems in which a malicious subset of the participants aim to poison the global model by sending model updates derived from mislabeled data. We first demonstrate that such data poisoning attacks can cause substantial drops in classification accuracy and recall, even with a small percentage of malicious participants. We additionally show that the attacks can be targeted, i.e., they have a large negative impact only on classes that are under attack. We also study attack longevity in early/late round training, the impact of malicious participant availability, and the relationships between the two. Finally, we propose a defense strategy that can help identify malicious participants in FL to circumvent poisoning attacks, and demonstrate its effectiveness.

1 Introduction

Federated learning preserves data locality by sharing model updates, but malicious participants can poison the global model through mislabeled local data. The paper demonstrates targeted attacks and evaluates their timing, availability, and detectability, while proposing an update-based defense.

  • Federated learning motivation: Federated learning keeps participant data local while aggregating model updates into a global model.Participants train locally and send parameters to a central aggregator without sharing raw data.
  • Threat: Malicious participants can exploit the absence of centralized data validation by introducing mislabeled samples into local training.The resulting updates can poison the globally trained model.
  • Attack findings: Small percentages of malicious users can still reduce model utility, and attacks can concentrate harm on selected classes while minimally affecting others.The study evaluates attack effectiveness across malicious-user percentage, targeted classes, timing, and participant availability.
  • Attack findings: Later-round participation combined with high malicious-participant availability produces the largest poisoning impact.The paper reports that early-round poisoning may stop while the global model still converges accurately.
  • Defense: The proposed defense extracts relevant update parameters and applies PCA to separate malicious from honest participants.Across malicious-participant rates of 2-20%, the aggregator obtains clear separation and can identify and block malicious participants.

2 Preliminaries and Attack Formulation

The paper formalizes federated training, its threat model, and targeted label-flipping attacks, then defines metrics for evaluating their effects. FL participants train locally and share parameters, while adversaries alter local labels to induce errors on chosen classes.

  • 2.1 Federated Machine Learning: Deep neural networks map feature inputs through layered parameters to final-layer class probabilities.The training objective minimizes a loss over labeled examples.
  • 2.1 Federated Machine Learning: Stochastic gradient descent repeatedly computes batch gradients and updates model parameters in the negative-gradient direction.Iterating over the batches constitutes an epoch.
  • 2.1 Federated Machine Learning: In federated learning, N participants execute training locally and upload updated parameters to a centralized aggregator, which broadcasts global parameters across rounds.After R rounds, the finalized model uses parameters θR.
  • 2.2 Threat and Adversary Model: The threat model assumes a subset of participants is malicious while the aggregator remains honest.Malicious devices may be added, compromised, or induced to poison for selected rounds.
  • 2.2 Threat and Adversary Model: The adversary targets high errors for particular classes rather than indiscriminate errors across all classes.Targeted effects can reduce influence on non-targeted classes and thereby decrease detection likelihood.
  • 2.2 Threat and Adversary Model: Each malicious participant can manipulate only its own local training data, without changing other data, the learning process, or server aggregation.The attack is not tied to a specific DNN architecture, loss function, or optimizer.
  • 2.3 Label Flipping Attacks in Federated Learning: A label-flipping attack changes every source-class label csrc to a chosen target class ctarget on each malicious participant’s device.The notation csrc →ctarget identifies the source-to-target transformation.
  • 2.3 Label Flipping Attacks in Federated Learning: Label flipping is evaluated with global accuracy, class recall, and baseline misclassification counts relative to a non-poisoned FL model.These metrics quantify overall performance, target-class errors, and pre-existing source-to-target confusion.

3 Analysis of Label Flipping Attacks in FL

Label-flipping attacks substantially degrade FL models even with few malicious participants, can selectively damage attacked classes, and are especially effective when malicious participation persists in late rounds. Early attacks are often recoverable, while late attacks and high availability produce larger, more persistent utility losses.

  • 3.2 Label Flipping Attack Feasibility: At m = 40%, CIFAR-10 source class recall falls to 0% and global model accuracy declines from 78.3% to 74.4%.On Fashion-MNIST, source class recall drops by approximately 10% at m = 4% and approximately 20% at m = 10%.
  • 3.2 Label Flipping Attack Feasibility: At m = 30%, CIFAR-10 source class recall reaches 19.7%, compared with 58.2% for Fashion-MNIST under the same settings.The datasets therefore exhibit different degrees of vulnerability to label flipping.
  • 3.2 Label Flipping Attack Feasibility: The most effective source→target pair is not reliably predicted by baseline misclassifications: CIFAR-10 0→2 is strongest in four of six scenarios, while Fashion-MNIST 4→6 outperforms 6→0.Baseline misclassification counts range from 16 for CIFAR-10 0→2 to 200 for 5→3, yet attack effectiveness does not track that baseline.
  • 3.2 Label Flipping Attack Feasibility: Label flipping is targeted: source and target recalls change substantially, while non-source and non-target recalls change by at most 0.34% on CIFAR-10 and 0.2% on Fashion-MNIST.The resulting misclassification is concentrated from the source class toward the target class.
  • 3.3 Attack Timing in Label Flipping Attacks: Early poisoning before round 75 is usually followed by recovery to non-poisoned accuracy and recall, although some CIFAR-10 runs converge 50–100 rounds later.Late poisoning instead produces rapid recall changes and can leave CIFAR-10 source recall approximately 10% below the non-poisoned model.
  • 3.3 Attack Timing in Label Flipping Attacks: When at least one malicious participant is selected late in training, final source recall is on average 12.08% lower for CIFAR-10 and 24.46% lower for Fashion-MNIST.The attack also increases round-to-round recall variance, tying utility impact predominantly to malicious selections in the last training rounds.
  • 3.4 Malicious Participant Availability: With m ≥10%, source-class recall loss exceeds 20% when malicious availability is manipulated, and impact is highest at α = 0.9.Availability effects are ordered α = 0.9, 0.8, 0.7, 0.6; when m < 10%, changing α produces little variation.
  • 3.4 Malicious Participant Availability: Source-class recall is often much smaller at α = 0.9 than at α = 0.6 across both CIFAR-10 and Fashion-MNIST.Observed recall also varies substantially from round to round because participant selection is probabilistic.

4 Defending Against Label Flipping Attacks

The paper proposes identifying malicious FL participants by extracting attack-relevant update parameters, reducing them with PCA, and clustering malicious and honest updates. This approach remains effective across participation rates and long training periods, enabling aggregators to restrict malicious updates and support model recovery.

  • Defense strategy: The defense extracts source-class parameters from participant update deltas, standardizes them, and applies two-component PCA to identify update patterns.For an unknown source class, the aggregator can repeat the procedure for every possible source class and target class.
  • Empirical evaluation: PCA plots form visibly different clusters for malicious and honest participants across CIFAR-10 and Fashion-MNIST participation rates.The separation remains visible even when the malicious participation rate is low.
  • Empirical evaluation: The defense remains capable of separating malicious and honest updates across 190 training rounds, indicating robustness to gradient drift.The evaluation uses vulnerable rounds R = [10, 200].
  • Practical defense: Aggregators can cluster gradients before aggregation, restrict identified participants, and do so without requiring a public validation dataset.After malicious participants are removed, high-utility model convergence can eventually be achieved following early-round attacks.

5 Related Work

Related work situates FL poisoning among broader attacks on machine-learning systems and distinguishes data poisoning from model poisoning. It also connects FL poisoning to Byzantine threats and robust aggregation methods.

  • Poisoning attacks: Poisoning attacks have been studied across application domains including spam filtering, malware detection, disease diagnosis, computer vision, and recommender systems.Prior work covers models such as SVMs, regression, linear classifiers, unsupervised learners, and neural networks.
  • Attacks in FL: FL security research includes backdoor, gradient leakage, membership inference, data poisoning, and model poisoning attacks.Data poisoning changes local training data while model poisoning modifies the learning process to produce adversarial updates.
  • Attacks in FL: Data poisoning may be preferable in some scenarios because it does not require adversarial manipulation of the local learning process.The local learning process remains otherwise unmodified in data poisoning.
  • Byzantine threats: FL poisoning is connected to Byzantine threats, motivating Byzantine-resilient aggregation methods such as Krum, Bulyan, trimmed mean, and coordinate-wise median.Byzantine behavior in FL has been associated with sub-optimal models or non-convergence.

6 Conclusion

The paper concludes that label-flipping data poisoning can substantially damage FL models, with attack impact increasing with malicious participation and becoming targetable by class. It also proposes a defense that separates malicious from honest participants and remains robust to gradient drift, while identifying broader settings for future study.

  • Conclusion: Label-flipping attacks significantly negatively impact global FL models, with stronger effects as the proportion of malicious participants increases.The attacks can also target selected classes while leaving remaining classes little affected.
  • Conclusion: Higher malicious-participant availability in later training rounds enhances attack effectiveness.The paper introduces malicious participant availability to study adversarial control over participation during training.
  • Conclusion: The proposed defense separates malicious from honest participants and is robust to gradient drift.The aggregator can use this identification to restrict malicious participation.
  • Future work: Future work will evaluate diverse FL data settings, more subtle poisoning strategies, and the defense against model poisoning and other anomalous-gradient attacks.The defense’s applicability beyond label flipping is presented as a hypothesis for future study.

A DNN Architectures and Configuration

The experiments train neural networks with PyTorch 1.2.0, random weight initialization, and an NVIDIA 980 Ti GPU; Fashion-MNIST receives no data preprocessing.

  • Implementation: All neural networks use PyTorch 1.2.0 with random weight initialization and are trained using an NVIDIA 980 Ti GPU accelerator.CUDA tensors are mapped to CPU tensors when necessary before export to NumPy arrays.
  • Implementation: Fashion-MNIST experiments use no data preprocessing.
Loading 2007.08432v2…