Source-linked AI summary
Can You Really Backdoor Federated Learning?
Ziteng Sun, Peter Kairouz, Ananda Theertha Suresh, H. Brendan McMahan
TL;DR
Federated learning’s decentralized structure makes backdoor attacks difficult to detect and defend against because attackers can alter targeted tasks while preserving main-task performance. The paper studies these attacks and defenses on realistic EMNIST federated learning, allowing benign clients to retain correctly labeled targeted examples. It finds that attack success depends on adversary prevalence and targeted-task complexity, while norm clipping and small Gaussian noise mitigate attacks without substantially harming overall performance.
Problem
Federated learning lacks well-established evidence on defending against targeted model-update poisoning when secure aggregation and benign targeted-task examples are present.
Method
The paper evaluates backdoor attacks and defenses on the user-partitioned, non-iid EMNIST dataset using TensorFlow Federated.
Results
Attack success largely depends on the fraction of adversaries and targeted-task complexity, while norm clipping plus small Gaussian noise mitigates attacks without much overall-performance loss.
Takeaways & Limitations
Known backdoor attacks can be considerably limited by norm clipping, with weak differential privacy providing additional mitigation.
Takeaways & Limitations
The interaction between model capacity and backdoor attacks remains an open theoretical and practical question.
Abstract
from arXiv · showhide
The decentralized nature of federated learning makes detecting and defending against adversarial attacks a challenging task. This paper focuses on backdoor attacks in the federated learning setting, where the goal of the adversary is to reduce the performance of the model on targeted tasks while maintaining good performance on the main task. Unlike existing works, we allow non-malicious clients to have correctly labeled samples from the targeted tasks. We conduct a comprehensive study of backdoor attacks and defenses for the EMNIST dataset, a real-life, user-partitioned, and non-iid dataset. We observe that in the absence of defenses, the performance of the attack largely depends on the fraction of adversaries present and the "complexity'' of the targeted task. Moreover, we show that norm clipping and "weak'' differential privacy mitigate the attacks without hurting the overall performance. We have implemented the attacks and defenses in TensorFlow Federated (TFF), a TensorFlow framework for federated learning. In open-sourcing our code, our goal is to encourage researchers to contribute new attacks and defenses and evaluate them on standard federated datasets.
1 Introduction
Federated learning makes detecting and correcting failures difficult, especially under secure aggregation. The paper focuses on targeted model-update poisoning, or backdoor attacks, which preserve main-task performance while altering targeted subtasks.
- Federated learning’s distributed structure, particularly with secure aggregation, makes detecting and correcting failures and attacks challenging.
- Targeted attacks seek different behavior on selected subtasks while maintaining good overall performance, unlike untargeted attacks that corrupt the primary task.
- Untargeted attacks are easier to detect because they reduce overall primary-task performance.
- The paper evaluates recent backdoor attacks and defenses on a realistic federated-learning task using TensorFlow Federated.
2 Backdoor Attack Scenario
The attack scenario models federated learning with sampled clients, targeted tasks containing correctly labeled benign examples, and task complexity controlled by grouping target clients. More target clients create a richer feature space that is harder to backdoor without harming the main task.
- Each round randomly selects C·K clients from K total users, and the server aggregates their local model updates.
- Sampling of adversaries: Random-sampling attacks produce a hypergeometric number of adversaries per round, while fixed-frequency attacks place one adversary every f rounds.
- Sampling of adversaries: Fixed-frequency comparisons set f = 1/(ϵ·C·K), making attack frequency inversely proportional to the total number of attackers.
- Backdoor tasks: Unlike prior settings, benign clients may possess correctly labeled samples from the targeted backdoor tasks.
- Backdoor tasks: The number of backdoor tasks equals the number of selected target clients whose examples are grouped together.
- Backdoor tasks: More backdoor tasks create a richer feature space that is harder to break without damaging main-task performance.
3 Model Update Poisoning Attacks
The paper studies model-replacement attacks in which compromised clients send updates intended to move the global model toward a backdoored model. It examines unconstrained and norm-bounded attack constructions under realistic federated-learning assumptions.
- The attacker uses model replacement to move the global model toward a backdoored model w∗ through a malicious client update.
- A realistic setting may involve K = 10^7 or more users while selecting only 100 to 1000 clients per round.
- The attacker’s update is scaled using β, the total number of selected-client samples, so the aggregate approaches w∗ when benign updates are small.
- Multiple attackers are assumed to coordinate and divide the malicious update evenly when they appear in the same round.
- Obtaining a backdoored model: Attackers construct w∗ from backdoor-task data and samples from the true training distribution, although obtaining such data may be difficult in practice.
- Unconstrained boosted attacks train on true and malicious data without update constraints, typically producing large-norm updates.
- Norm-bounded attacks constrain the pre-boost update to M/β so the boosted update has norm at most M.
4 Defenses
The defenses limit malicious update influence through norm-based filtering or clipping, with an additional weak differential-privacy option that clips updates and adds Gaussian noise. The adversary is given knowledge of the norm threshold in the evaluation setup.
- Norm thresholding of updates: The server can ignore updates whose norm exceeds threshold M because boosted attacks tend to produce large-norm updates.
- Norm thresholding of updates: Assuming the adversary knows M makes norm-bounding equivalent to norm clipping rather than relying on hidden thresholds.
- Norm thresholding of updates: Norm clipping keeps each model update’s norm small, reducing its susceptibility to the server’s aggregation process.
- (Weak) differential privacy: Weak differential privacy first clips federated updates and then adds Gaussian noise, using less noise than traditionally required for privacy.
5 Experiments
Experiments on user-partitioned EMNIST examine how attack frequency, adversary prevalence, targeted-task complexity, update norms, and Gaussian noise affect federated backdoor attacks. The results identify norm bounding and weak differential privacy as defenses that mitigate attacks while largely preserving main-task performance.
- Experimental setup: EMNIST experiments use 3383 users with roughly 100 handwritten-digit images per user and a five-layer convolutional network in TensorFlow Federated.Each client trains locally for 5 epochs with batch size 20 and client learning rate 0.1; 30 clients are selected per round.
- Attack models: Fixed-frequency and random-sampling attacks behave similarly, although fixed-frequency attacks are slightly more effective and easier to interpret.The analysis therefore focuses on fixed-frequency attacks for the remainder of the experiments.
- Adversary prevalence: Attack success depends strongly on the adversary fraction, degrading when fully compromised users fall below 1%.The experiments use 3.3% and 0.33% malicious-user fractions with 30 backdoor tasks, including attacks every round or once every ten rounds.
- Target-task complexity: More backdoor tasks make it harder to fit a malicious model while maintaining main-task performance.Norm-bounded experiments compare 10, 20, 30, and 50 backdoor tasks under norm bound 10; benign users’ correctly labeled samples can also correct the attacked model.
- Defenses: A norm bound of 3 mitigates the attack with almost no effect on main-task performance, while Gaussian noise further helps beyond clipping without hurting overall performance much.The weak differential privacy experiment uses norm bound 5 and independent Gaussian noise with variance 0.025 per coordinate.
6 Discussion
The paper finds that norm clipping and added Gaussian noise can mitigate federated backdoor attacks, while several attack, capacity, and secure-aggregation questions remain open.
- Norm clipping considerably limits known backdoor attacks, and adding a small amount of Gaussian noise can further mitigate adversarial effects.
- Figure 3 examines how backdoor size affects a constrained attack with norm bound 10.
- Figure 4 examines norm bounding and Gaussian noise separately for the main task and backdoor task.
- Better attacks and defenses: Better attacks may directly craft worst-case norm-bounded updates or exploit repeated attack opportunities across multiple rounds.
- Effect of model capacity: The interaction between model capacity and backdoor attacks remains an open theoretical and practical question.
- Interaction of defenses with SecAgg: Efficient implementations of multiparty range proofs compatible with secure aggregation remain an active research direction.
A Additional figures for experiments
The appendix figures compare unconstrained and constrained attacks across fixed-frequency and random-sampling settings, varying the fraction of attackers.
- Figure 5 shows unconstrained attacks, with fixed-frequency attacks in the left column and random-sampling attacks in the right column.
- Figure 6 shows constrained attacks with norm bound 10, using the same fixed-frequency versus random-sampling column layout.