Source-linked AI summary
Threats to Federated Learning: A Survey
Lingjuan Lyu, Han Yu, Qiang Yang
TL;DR
Centralized training faces challenges from costly data collection and privacy concerns, while federated learning can still expose sensitive information through communicated model updates. This survey provides a taxonomy of federated-learning threat models, poisoning attacks, and inference attacks, summarizes their techniques and assumptions, and identifies future research directions for stronger privacy preservation.
Problem
Centralized machine learning faces costly data collection and privacy concerns, while federated learning does not always provide sufficient privacy because model updates can reveal sensitive information.
Method
The paper develops a concise survey and taxonomy of federated-learning threat models, poisoning attacks, and inference attacks, highlighting their intuitions, techniques, and assumptions.
Results
The survey provides an accessible overview of federated-learning vulnerabilities, including data and model poisoning, gradient-based information leakage, membership inference, and inference-phase attack settings.
Takeaways & Limitations
Federated-learning system designers should account for poisoning and inference vulnerabilities when designing protocols and pursue defenses robust to varied attacks without degrading model performance.
Takeaways & Limitations
Adversarial training remains difficult to adapt to federated learning because its performance in non-IID settings is unclear and it typically requires many epochs.
Abstract
from arXiv · showhide
With the emergence of data silos and popular privacy awareness, the traditional centralized approach of training artificial intelligence (AI) models is facing strong challenges. Federated learning (FL) has recently emerged as a promising solution under this new reality. Existing FL protocol design has been shown to exhibit vulnerabilities which can be exploited by adversaries both within and without the system to compromise data privacy. It is thus of paramount importance to make FL system designers to be aware of the implications of future FL algorithm design on privacy-preservation. Currently, there is no survey on this topic. In this paper, we bridge this important gap in FL literature. By providing a concise introduction to the concept of FL, and a unique taxonomy covering threat models and two major attacks on FL: 1) poisoning attacks and 2) inference attacks, this paper provides an accessible review of this important topic. We highlight the intuitions, key techniques as well as fundamental assumptions adopted by various attacks, and discuss promising future research directions towards more robust privacy preservation in FL.
1 Introduction
Federated learning addresses the cost, infrastructure, and privacy constraints of centralized machine learning by training models across data-holding devices without directly sharing private data. The survey highlights that FL still exposes vulnerabilities through model updates and malicious servers or participants.
- Centralized storage of ubiquitous user data is costly and time consuming, while such data can contain sensitive information creating privacy risks.
- Traditional centralized machine learning struggles with limited bandwidth, intermittent connectivity, and strict delay constraints in ubiquitous deployments.
- Federated learning trains models on originating devices, enabling participants to build a joint model without exposing private training data.
- Federated learning includes horizontal, vertical, and transfer settings, distinguished by how participants’ data features and samples overlap.
- Despite avoiding direct data sharing, communicating model updates can leak sensitive information to third parties, servers, or participants.
- The survey focuses on poisoning attacks that compromise learning and inference attacks that target participant privacy, addressing a gap in prior FL surveys.
2 Threat Models
The survey organizes FL threats by attacker position, behavior, and attack phase, focusing primarily on stronger insider attacks. It distinguishes single, Byzantine, and Sybil attacks and concentrates on training-phase threats rather than inference-phase attacks.
- FL attacks may come from insiders such as servers and participants or outsiders such as eavesdroppers and final-model users.
- Because insider attacks strengthen adversary capabilities, the survey focuses primarily on insider threats.
- Insider attack forms: Single attacks involve one non-colluding participant targeting chosen inputs for high-confidence misclassification.
- Insider attack forms: Byzantine attackers behave arbitrarily while imitating correct-update distributions, whereas Sybil attackers create or exploit multiple participant accounts.
- Adversary behavior: Semi-honest adversaries passively observe permitted aggregated gradients, while malicious adversaries actively deviate from the protocol to learn private states.
- Attack phases: Training-phase attacks can poison data or models and infer information from individual or aggregate updates.
- Attack phases: The survey omits inference-phase evasion attacks and mainly reviews training-phase attacks.
3 Poisoning Attacks
Poisoning attacks manipulate training data or model updates to degrade FL performance, induce targeted errors, or implant backdoors. The survey distinguishes attack types, assumptions, effectiveness, and deployment constraints.
- Attack Taxonomy: Poisoning attacks can be random or targeted, and can manipulate either training data or model updates.Random attacks reduce model accuracy, whereas targeted attacks seek an adversary-specified output.
- Data Poisoning: Clean-label poisoning preserves certified labels and requires imperceptible poisoned samples, whereas dirty-label poisoning permits adversarial target labels.Dirty-label attacks can introduce samples intended to be misclassified with a desired target label.
- Data Poisoning: Label flipping changes class labels while preserving features, whereas backdoor poisoning modifies features so triggered inputs produce adversary-chosen behavior.Backdoor-poisoned models can retain performance on clean inputs, making the attack harder to detect.
- Data Poisoning: Data poisoning can be conducted by any participant, but its impact depends on attacker participation and the amount of poisoned training data.The passage reports that data poisoning is less effective in settings with fewer participants such as H2C.
- Model Poisoning: Model poisoning alters local updates or inserts hidden backdoors into the global model, with a single-shot attack potentially sufficient to introduce a backdoor.These attacks manipulate the training process rather than modifying test inputs.
- Model Poisoning: A targeted model-poisoning attack by one non-colluding participant was much more effective than data poisoning and could remain undetected.The attack used alternating minimization and parameter estimation to increase stealth and evade detection.
- Model Poisoning: Model poisoning subsumes data poisoning in FL because data poisoning ultimately changes a subset of updates sent to the model.Such attacks require sophisticated capabilities and substantial computational resources, making them more likely in H2B than H2C settings.
4 Inference Attacks
Inference attacks exploit exchanged model updates and gradients to recover private information from participants’ training data. The survey covers property, membership, active, passive, and gradient-leakage attacks, along with their assumptions and practical constraints.
- Privacy Leakage from Updates: Shared model updates can reveal unintended features, class representatives, membership, and properties of participants’ training data.Deep models may internally recognize features unrelated to the main learning task, enabling privacy leakage through updates.
- Gradient Leakage: Gradients depend on private-data features and upper-layer errors, allowing update observations to expose labels and recover original training samples.The passage reports label inference and sample recovery without prior knowledge of the training set.
- Active Inference: GAN attacks exploit FL’s real-time training process to generate prototypical samples from targeted private training data.A malicious participant can intentionally compromise another participant in this active inference setting.
- Membership Inference: Membership inference determines whether a specific data point was used for training, with the target being one participant’s data or an aggregate across participants.For text models, non-zero embedding gradients can reveal which words appeared in honest participants’ training batches.
- Membership Inference: An adversary can infer whether a given text appeared in the training dataset from non-zero embedding gradients.This links shared gradient patterns to membership in private training data.
- Attack Modes: Membership and property inference attacks may be passive or active, depending on whether the adversary only observes updates or tampers with training.Property inference uses auxiliary labeled data to train a classifier for properties independent of the FL model’s class-defining features.
- Gradient-Based Reconstruction: DLG recovers training inputs and labels in a few iterations, while iDLG extracts labels from gradient signs for differentiable cross-entropy classifiers.DLG can recover pixel-wise accurate images and token-wise matching texts; iDLG applies to one-hot-label classification models.
- Assumptions and Constraints: Inference attacks generally require sophisticated capabilities, large computational resources, and selection for many FL training rounds.The survey identifies gradient protection mechanisms such as homomorphic encryption as a possible response.
5 Discussions and Promising Directions
The survey identifies unresolved vulnerabilities across federated-learning settings and defenses, then outlines research directions for stronger privacy and security. Key directions include reducing exposed model information, studying underexplored architectures and deployment settings, and balancing defense effectiveness against cost and accuracy.
- Model Representation: Sharing high-dimensional model parameters exposes internal state to inference attacks and increases susceptibility to poisoning and backdoor attacks.The paper suggests exploring model updates, signs, or black-box predictions as potentially less sensitive alternatives.
- Federated-Learning Settings: Threats to vertically federated learning remain underexplored, including whether attacks developed for horizontally federated learning transfer to this setting.This gap is especially relevant because vertically federated learning is important to businesses.
- Federated-Learning Settings: Existing attack and privacy techniques are largely framed around homogeneous architectures, leaving heterogeneous federated-learning models as an open research direction.Future work should examine whether current attacks and defenses adapt to collaborative training across different model architectures.
- Federated-Learning Settings: Decentralized federated learning may introduce new attack surfaces, including model contamination by late-round servers and training-data theft by malicious neighbors.These risks arise in round-robin server selection and ring-all-reduce training, respectively.
- Defense Mechanisms: Secure aggregation can increase poisoning exposure because individual updates cannot be inspected, while adversarial training faces non-IID-data and computational challenges.Adversarial training may require many epochs, complicating deployment in H2C settings.
- Defense Mechanisms: Defense deployment requires optimizing timing and deterrence because mechanisms differ in attack effectiveness and computational cost.The paper identifies game-theoretic research as a promising way to study this trade-off.