PaperlayerView exact revision on arXiv

Source-linked AI summary

Privacy and Robustness in Federated Learning: Attacks and Defenses

Lingjuan Lyu, Han Yu, Xingjun Ma, Chen Chen, Lichao Sun, Jun Zhao, Qiang Yang, Philip S. Yu

arXiv:2012.06337v3cs.CRcs.AI

TL;DR

Federated learning addresses centralized training’s efficiency and privacy challenges, but its protocols remain vulnerable to adversaries targeting privacy and robustness. This paper presents a comprehensive survey and taxonomy of threat models, poisoning and inference attacks, and their defenses. It highlights that privacy-preserving and robust FL must account for attack diversity, deployment constraints, and defense trade-offs.

  • Problem

    Existing FL protocol designs are vulnerable to adversaries that compromise data privacy and system robustness, creating a need to understand these threats and defenses.

  • Method

    The paper conducts a comprehensive survey organized around threat models, poisoning attacks and robustness defenses, and inference attacks and privacy defenses.

  • Results

    The survey synthesizes the intuitions, key techniques, and fundamental assumptions underlying representative FL attacks and defenses.

  • Takeaways & Limitations

    The survey provides an accessible overview intended to help FL system designers recognize vulnerabilities and address deployment roadblocks.

Abstract

from arXiv · show

As data are increasingly being stored in different silos and societies becoming more aware of data privacy issues, the traditional centralized training of artificial intelligence (AI) models is facing efficiency and privacy challenges. Recently, federated learning (FL) has emerged as an alternative solution and continue to thrive in this new reality. Existing FL protocol design has been shown to be vulnerable to adversaries within or outside of the system, compromising data privacy and system robustness. Besides training powerful global models, it is of paramount importance to design FL systems that have privacy guarantees and are resistant to different types of adversaries. In this paper, we conduct the first comprehensive survey on this topic. Through a concise introduction to the concept of FL, and a unique taxonomy covering: 1) threat models; 2) poisoning attacks and defenses against robustness; 3) inference attacks and defenses against privacy, we provide an accessible review of this important topic. We highlight the intuitions, key techniques as well as fundamental assumptions adopted by various attacks and defenses. Finally, we discuss promising future research directions towards robust and privacy-preserving federated learning.

I. INTRODUCTION

Federated learning distributes training to data-originating devices, addressing centralized learning’s infrastructure and privacy challenges without directly sharing private training data. The introduction categorizes FL by data distribution and model architecture while noting that common sharing methods remain vulnerable to privacy and poisoning attacks.

  • Motivation: Centralized ML struggles with bandwidth, connectivity, delay, cost, and privacy constraints in ubiquitous data deployments.Legal restrictions such as GDPR further make direct data aggregation less feasible.
  • Federated learning: Federated learning trains models on data-originating devices, enabling joint model construction without exposing private training data.It can also accommodate unbalanced and non-I.I.D. data.
  • Data distribution: HFL participants share similar features but hold data about different users, whereas VFL participants share records but hold different features and FTL has little overlap in both spaces.HFL is further divided into business-oriented H2B and consumer-oriented H2C settings with different participation and capability profiles.
  • Model architectures: Homogeneous FL uses FedSGD or communication-efficient FedAvg with mean aggregation, but one compromised participant can arbitrarily manipulate the global model mean.The local objective is defined over each participant’s training dataset.
  • Model architectures: Heterogeneous FL shares model predictions rather than parameters, allowing model-architecture collaboration and eliminating white-box inference risk in conventional FL.FedMD is described as black-box, model agnostic, and without a single forced global model.
  • Security gap: Existing sharing methods do not defend against privacy and poisoning attacks, which are two main sources of threats to FL.

C. Threats to FL

FL faces privacy threats from inference over communicated updates and robustness threats from data or model poisoning by malicious actors. The survey organizes these threats and reviews privacy-preserving and robustness defenses, whose applicability is constrained by FL’s limited server-side visibility and attack diversity.

  • Threat model: FL protocol designs can expose participants to malicious servers that infer information, tamper with training, or control participants’ views.
  • Privacy threats: Communicated gradients can reveal sensitive information, including substantial portions of local data and even training data itself.
  • Robustness threats: Data and model poisoning can disrupt global-model convergence or implant backdoors while preserving main-task performance.
  • Privacy threats: Privacy attacks include passive observation of model updates and active protocol tampering, including gradient ascent membership inference.
  • Privacy defenses: HE and SMC can impose substantial computation and communication overhead, while DP requires calibrated noise in aggregated values.
  • Robustness defenses: Robustness defenses are difficult because the server sees only local gradients and must address both data and model poisoning.Many existing aggregation methods were developed for untargeted Byzantine attacks and had not been tested on targeted backdoors.

E. Motivation of this Survey and Our Contribution

The survey addresses the lack of systematic organization in research on federated-learning privacy and robustness threats, attacks, defenses, and future directions.

  • Prior FL surveys largely emphasize system or protocol design, leaving privacy and robustness threats insufficiently categorized and systematically explored.
  • The survey systematically categorizes FL and summarizes threats with their corresponding protections.
  • It examines attacks and defenses through their assumptions, principles, reasons, and differences.
  • The survey identifies conflicts between privacy and robustness, multiple design goals, and gaps between existing work and real FL scenarios.
  • Future directions aim to improve the practicality and impact of robust, privacy-preserving FL while integrating multidisciplinary design goals.
  • Its coverage is organized around threat models, privacy attacks and protections, poisoning attacks and countermeasures, and future research gaps.

A. Insider v.s. Outsider

FL attacks are categorized by whether adversaries are insiders or outsiders and whether attacks occur during training or inference. The survey focuses primarily on insider threats, which can target privacy or model robustness.

  • Insider attacks originate from the FL server or participants, whereas outsider attacks involve communication eavesdroppers or users of deployed FL models.
  • Insider attacks are generally more dangerous because they enhance the adversary’s capabilities, motivating the survey’s primary focus on them.
  • Training Phase: Training-phase attacks can learn, influence, or corrupt the FL model through data poisoning, model poisoning, or inference on participant or aggregated updates.
  • Inference Phase: Inference-phase attacks generally leave the model unchanged but can induce wrong predictions or collect evidence about model characteristics.
  • Semi-honest adversaries observe global-model parameters without deviating from the protocol, while malicious adversaries may modify, replay, or remove messages.
  • Untargeted poisoning seeks arbitrary model-integrity compromise, whereas targeted poisoning induces specified labels for particular examples.
  • Gradients and successive model-parameter snapshots can leak unintended features of participants’ private training data.

A. Inferring Class Representatives

Federated-learning inference attacks can recover or infer sensitive properties of participants’ data from updates, often using active or passive adversarial strategies. Their practicality depends on strong attacker capabilities and repeated participation.

  • GAN attacks exploit FL’s real-time learning process to generate prototypical samples resembling targeted participants’ private training data.
  • Membership inference determines whether a specific sample belongs to a participant’s private training data, with active attacks enabling stronger protocol manipulation.
  • Property inference uses auxiliary labeled data to infer properties of other participants’ training data from observed gradients or manipulated multi-task learning.
  • DLG optimizes over shared gradients to recover training inputs and labels, including raw images and texts, while iDLG analytically extracts labels in common classification settings.
  • These attacks generally assume sophisticated technical capabilities, unlimited computation, and repeated adversarial selection across many FL training rounds.
  • Privacy-preserving FL remains challenging because sporadic resources, network connectivity, and statistical heterogeneity complicate established protection techniques.

A. Privacy Preservation through Homomorphic Encryption

Privacy-preserving FL methods use cryptographic computation or randomized perturbation to protect training information, but their privacy, utility, scalability, and trust assumptions involve important trade-offs.

  • Homomorphic Encryption: Homomorphic encryption performs arithmetic directly on ciphertexts and includes fully, somewhat, and partially homomorphic variants.Fully homomorphic encryption supports arbitrary computation but is less efficient.
  • Homomorphic Encryption: Homomorphic encryption protects learning processes by computing on encrypted data, but encrypted arithmetic increases memory and processing costs.With Paillier encryption, encoded floating-point ciphertexts are 2m bits long, and encrypted addition is 2∼3 orders of magnitude slower than its unencrypted equivalent.
  • Homomorphic Encryption: Aono et al.’s additively homomorphic encryption protocol protects individual gradients but incurs communication and computational overhead, causes utility loss, and does not withstand server–participant collusion.The passage presents these as limitations of the protocol.
  • Secure Multiparty Computation: Secure multiparty computation enables participants with private inputs to jointly compute without revealing those inputs to one another.SecureML requires data owners to process, encrypt, and/or secret-share data among two non-colluding servers during setup.
  • Secure Multiparty Computation: SMC provides high privacy and accuracy at the expense of high computation and communication overhead and requires simultaneous coordination throughout training.This interaction model may be undesirable in practical FL settings, especially under participant-server architectures.
  • Privacy and Scalability Trade-offs: SMC can still permit information leakage from computation outcomes, so additional differential privacy techniques may be needed.The survey also notes that cryptography-based protocols can hinder auditing of participant updates and may not scale to large FL scenarios.
  • Differential Privacy: Differential privacy bounds the effect of one record on output likelihood and trades privacy against accuracy through computationally efficient randomization.Approximate (ϵ, δ)-DP relaxes pure ϵ-DP with an additive δ term.
  • Differential Privacy: The standard recommendation is δ ≪ 1/|D|, although other forms of information leakage remain.This choice avoids the worst-case scenario of always violating privacy for a δ fraction.

V. POISONING ATTACKS

Poisoning attacks target FL robustness and can be either untargeted or targeted, with poisoned updates introduced through data collection or local model training.

  • Attack Taxonomy: Poisoning attacks compromise system robustness and are classified as untargeted or targeted according to the attacker’s objective.Both categories can be mounted during the training phase.
  • Attack Sources: Poisoned updates arise from data poisoning during local data collection or model poisoning during local model training.Both attack types attempt to modify the target model’s behavior in an undesirable way.

A. Untargeted Attacks

Untargeted poisoning attacks seek to arbitrarily compromise the global model, and Byzantine participants can manipulate aggregation to cause failure. The survey describes attack constructions that bypass or perturb robust aggregation and convergence.

  • Attack objective: Untargeted attacks aim to destroy global-model convergence or performance rather than induce a specific prediction on selected examples.This distinguishes them from targeted poisoning attacks.
  • Attack definition: Byzantine attacks let dishonest participants upload arbitrary values instead of their local gradients.An honest participant uploads its local objective gradient, whereas a dishonest participant may submit arbitrary values.
  • Attack impact: A single Byzantine participant can completely control FL aggregation when no defense is deployed.With n−1 benign participants, the Byzantine participant can choose an update that makes the aggregate any desired vector u.
  • Attack constructions: Camouflage attacks can hide malicious updates while attacking Adam-based FL and other optimizers.The same attack was also reported to work against AdaGrad and RMSProp.
  • Attack constructions: Inner product manipulation sends negative average benign gradients to make the robust aggregate oppose the ground-truth gradient.The attack was reported to bypass Coordinate-wise Median and Krum.

B. Targeted Attacks

Targeted poisoning attacks make the model produce an adversary-chosen label for selected inputs while preserving performance on other examples. The survey covers label flipping, backdoors, clean- versus dirty-label settings, and participant or collusion-based attacks.

  • Attack objective: Targeted attacks force adversary-specified labels on particular test examples while leaving testing error on other examples unaffected.They are generally harder to conduct than untargeted attacks because the adversary has a specific goal.
  • Label flipping: Label-flipping attacks change one class’s labels to another while preserving features, causing systematic misclassification of the original class.An example flips all labels 1 to 7, making the model predict 7 for inputs belonging to class 1.
  • Backdoor attacks: Backdoor poisoning modifies individual features or small data regions so a trigger causes a target prediction while clean-data behavior remains normal.The passage gives image stamps as an example of a trigger and reports 100% accuracy on a backdoor task in one case.
  • Backdoor variants: Clean-label attacks preserve certified labels and require imperceptible poisoning, whereas dirty-label attacks allow adversarial label changes.The distinction concerns whether the attacker can alter training labels.
  • Adversary organization: Targeted poisoning can use data or gradient manipulation by individual participants, colluding participants, or Sybil clones.A single non-colluding participant can induce high-confidence errors on chosen inputs, and single-shot updates may inject a backdoor.

VI. DEFENSES AGAINST POISONING ATTACKS

FL poisoning defenses primarily use robust aggregation or participant-update filtering because the server observes model updates rather than training data. The survey covers Byzantine-resilient methods, their assumptions, and evidence that targeted attacks can circumvent some guarantees.

  • Defense motivation: Centralized robust losses and anomaly detection do not directly transfer to FL because the server lacks control of participants and explicit access to training data.This motivates defenses operating on model parameters or updates.
  • Byzantine-resilient aggregation: Krum removes contributions farthest from the mean using Euclidean distance and theoretically tolerates up to 33% Byzantine participants.Its condition is n ≥ 2f + 3 when f of n participants are Byzantine.
  • Byzantine-resilient aggregation: Krum selects the update with the lowest summed distance to its n−f−2 closest updates and uses it to update the global parameter.The selection is based on the L2 distances among received updates.
  • Byzantine-resilient aggregation: Krum resists opposite-vector and Gaussian random-vector attacks, while Multi-Krum interpolates between Krum and averaging.Krum filters outliers using entire update vectors rather than coordinate-wise values.
  • Alternative defenses: Coordinate-wise median, trimmed mean, Bulyan, DRACO, SIGNSGD, and AUROR represent alternative filtering, redundancy, voting, or clustering defenses.DRACO assumes participants can access other participants’ data, limiting its practicality for FL; median-based rules can be computationally expensive at scale.
  • Defense limitations: Targeted model poisoning can circumvent Byzantine-robust rules such as Krum and coordinate-wise median despite their robustness guarantees.Prior work also found that limiting Byzantine influence in one round may not prevent cumulative convergence perturbation.

B. Defenses against Targeted Attacks

Defenses against targeted poisoning include detecting backdoors or erasing their effects, alongside Sybil-specific similarity weighting. Their effectiveness and guarantees depend strongly on the FL setting and participation pattern.

  • Backdoor defenses: Backdoor defenses comprise detection methods that identify compromised models or examples and erasing methods that purify adverse trigger effects.Detection may use activation statistics, model properties, or latent-representation differences between clean and backdoor inputs.
  • Backdoor defenses: Anti-Backdoor Learning exploits faster backdoor-task learning and the target-class specificity of backdoors to train clean models from poisoned data.Experiments on multiple benchmark datasets against 10 state-of-the-art attacks found automatic prevention without degrading main performance.
  • Certified robustness: FL backdoor mitigations include update-norm clipping, Gaussian noise, and per-round validation, but these methods do not provide certified robustness guarantees.CRFL is described as the first general framework for certifiably robust FL models against backdoors.
  • Sybil defenses: FoolsGold adapts participant learning rates from contribution similarity because Sybil updates are more similar to one another than honest-client updates.It does not bound the expected number of attackers and assumes attackers may spawn many Sybils.
  • Defense boundaries: Robust aggregation defenses have breaking points beyond which robustness guarantees cannot be provided when the malicious-participant fraction is too large.Some defenses have no theoretical breaking point, as noted in the accompanying table.
  • Operational setting: Infrequent participation reduces poisoning effectiveness, while production cross-device FL showed no impact from poisoning on existing robust algorithms even with impractically high compromise rates.For cross-silo FL, data poisoning was completely ineffective and model poisoning was considered unlikely under contractual and professionally maintained settings, except in exceptional collusion scenarios.

VII. DISCUSSIONS AND PROMISING DIRECTIONS

The survey identifies unresolved vulnerabilities and research directions spanning privacy, robustness, fairness, efficiency, decentralization, heterogeneous architectures, and simultaneous design objectives in federated learning. It emphasizes that practical trustworthy FL requires addressing both training- and test-phase threats while balancing competing goals.

  • General directions: FL research must address vulnerabilities beyond current privacy and robustness defenses, while considering multiple design goals simultaneously.The paper frames these as promising directions for improving FL systems.
  • Robustness and security: High-dimensional models remain susceptible to small, damaging poisoning changes, and existing Byzantine-robust aggregators do not fully resolve this curse of dimensionality.The vulnerability arises because local models are overwritten with the global model and small changes can evade detection.
  • Privacy attacks: Current privacy attacks often rely on restrictive assumptions, including single-participant class data, small batches, limited local training, or untrained models.The survey calls for attacks evaluated under more practical federated settings.
  • Defense mechanisms: Secure aggregation limits update inspection, while adversarial training remains insufficiently understood for non-IID federated data.These limitations complicate simultaneous privacy protection and robustness against poisoning.
  • Defense deployment: Defense deployment requires optimizing computational cost, attack-specific effectiveness, timing, and deterrence measures.The survey suggests game-theoretic research as a promising way to study this deployment problem.
  • Future directions: Future work includes test-phase privacy and robustness, heterogeneous architectures, decentralized FL, free-rider prevention, and efficient one-shot communication.One-shot FL methods face limitations involving auxiliary data, homogeneous models, extra information, and performance; FedSyn addresses several of these constraints.
  • Trustworthy FL: The paper argues that trustworthy FL requires defenses against both training-phase and test-phase attacks.This connects the survey’s discussion of deployment vulnerabilities with its broader robustness objective.
  • Multiple objectives: No existing work simultaneously satisfies fast convergence, generalization, communication efficiency, fault tolerance, privacy, and robustness to poisoning attacks and free-riders.The paper highlights an inherent conflict between privacy and robustness and identifies this combination as largely unexplored.

VIII. CONCLUSIONS

The survey provides a concise, accessible overview of privacy and robustness attacks and defenses in federated learning. It aims to help system designers understand current vulnerabilities and identifies broad interdisciplinary work as necessary for defenses that resist varied attacks without degrading model performance.

  • Conclusion: The survey maps the privacy and robustness attack-and-defense landscape to support future federated learning system design.It presents the overview as a way to help designers recognize vulnerabilities and address deployment roadblocks.
  • Conclusion: A general-purpose FL defense should withstand various attacks without degrading model performance, requiring interdisciplinary effort.The conclusion presents this as the ultimate goal for future research.
Loading 2012.06337v3…