How To Backdoor Federated Learning

Eugene Bagdasaryan; Andreas Veit; Yiqing Hua; Deborah Estrin; Vitaly Shmatikov

Source-linked AI summary

How To Backdoor Federated Learning

Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, Vitaly Shmatikov

arXiv:1807.00459v3Jul 2, 2018cs.CRcs.LG

TL;DR

Federated learning protects private training data, but its aggregation design leaves the joint model exposed to malicious participants. The paper introduces model replacement and shows that a single-round attacker can implant accurate backdoors, outperform data poisoning, and evade anomaly detection.

Problem
Federated learning cannot generally inspect participants’ private data or model updates, leaving it vulnerable to model-poisoning attacks.
Method
The paper trains a malicious model on backdoor data and uses model replacement with constrain-and-scale to influence the joint model while evading defenses.
Results
100% backdoor-task accuracy is achieved after a single attacker participates in a single round, while 8 of 80,000 participants reach 50% accuracy versus 400 for data poisoning.
Takeaways & Limitations
Federated learning can preserve main-task accuracy while an attacker controls behavior on an attacker-chosen backdoor subtask.
Takeaways & Limitations
Robust federated-learning defenses remain an open problem because Byzantine-tolerant methods do not apply when participants’ data are non-i.i.d.

Abstract

from arXiv · show

Federated learning enables thousands of participants to construct a deep learning model without sharing their private training data with each other. For example, multiple smartphones can jointly train a next-word predictor for keyboards without revealing what individual users type. We demonstrate that any participant in federated learning can introduce hidden backdoor functionality into the joint global model, e.g., to ensure that an image classifier assigns an attacker-chosen label to images with certain features, or that a word predictor completes certain sentences with an attacker-chosen word. We design and evaluate a new model-poisoning methodology based on model replacement. An attacker selected in a single round of federated learning can cause the global model to immediately reach 100% accuracy on the backdoor task. We evaluate the attack under different assumptions for the standard federated-learning tasks and show that it greatly outperforms data poisoning. Our generic constrain-and-scale technique also evades anomaly detection-based defenses by incorporating the evasion into the attacker's loss function during training.

1 Introduction

The paper identifies model poisoning as a generic vulnerability in federated learning and introduces model replacement to implant backdoors while preserving main-task behavior. It evaluates the attack on image classification and word prediction and develops techniques to evade anomaly detection.

Attack: Model replacement lets a malicious participant directly influence the joint model and introduce attacker-controlled backdoor behavior.The attacker can alter image-classification or word-prediction behavior on selected inputs while retaining the federated task.
Evaluation: 100% accuracy on the backdoor task is achieved by a single attacker selected in a single training round.The result is demonstrated on CIFAR-10 image classification and Reddit word prediction.
Evaluation: 8 malicious participants achieve 50% backdoor accuracy in a word-prediction task with 80,000 participants, versus 400 for data poisoning.This comparison illustrates the advantage of model replacement over traditional training-data poisoning.
Problem: Federated learning is vulnerable because secure aggregation hides how participants generate their model updates from the aggregator.This prevents direct inspection of participants’ data or updates without compromising privacy.
Defense evasion: Constrain-and-scale incorporates anomaly-detector evasion into the attacker’s loss function during training.The technique is designed to evade detectors despite secure aggregation or possible future anomaly-detection deployments.

2 Related Work

Prior work addresses data poisoning, backdoors, privacy, and Byzantine behavior, but these defenses generally require inspecting training data or model updates. The paper emphasizes that such requirements conflict with federated learning’s confidentiality and non-i.i.d. setting.

Poisoning attacks: Traditional poisoning attacks target training data, whereas model-poisoning attacks alter submitted models to change behavior at test time.The paper positions model poisoning as distinct from earlier data-poisoning and backdoor approaches.
Defenses: Most poisoning and backdoor defenses require inspecting training data or the resulting model, which conflicts with federated confidentiality.Secure aggregation keeps users’ training data and local models confidential.
Defenses: Existing backdoor defenses are limited by domain or attack type, including pixel-pattern assumptions and privacy-invasive model inversion.The paper contrasts these limitations with semantic backdoors in text tasks involving thousands of labels.
Privacy: Participant-level differential privacy partially mitigates the attack but reduces the joint model’s main-task accuracy.This presents a privacy–utility trade-off for the defense.
Distributed learning: Byzantine-tolerant methods assume training data are i.i.d., unmodified, or equally distributed, assumptions explicitly false in federated learning.Alternative aggregation mechanisms can also reduce accuracy on non-i.i.d. data or be incompatible with secure aggregation.

3 Federated Learning

Federated learning trains a shared model by repeatedly aggregating local models or updates from selected participants while keeping their data local. Its design supports large-scale, non-i.i.d. participation but allows local models to diverge and malicious participants to influence the joint model.

Motivation: Federated learning keeps local training data on participants’ machines, enabling training on sensitive data from potentially millions of users.The framework is motivated by both efficiency and privacy.
Training process: Each round selects a subset of participants, sends them the current joint model, and receives locally trained model updates.The server averages the received updates to form the next joint model.
Aggregation: The global learning rate controls how much the joint model changes each round, with η = n/m fully replacing it by the average of local models.Lower η supports convergence for tasks such as CIFAR-10, while larger η gives local models more influence with very large user populations.
Heterogeneity: Because local datasets are small and drawn from different distributions, local models often overfit, diverge from the global model, and have low accuracy.Averaging balances these heterogeneous contributions into an accurate joint model.
Threat surface: Continuous updates give a malicious participant repeated opportunities to be selected and influence the model.The framework does not verify that submitted training was performed correctly.

4 Adversarial Model Replacement

Federated learning lets a compromised participant inject a backdoor while performing the assigned task, and model replacement makes that attack effective after a single aggregation round. The attack can preserve main-task accuracy, evade anomaly detection through constrained training, and outperform naive poisoning that aggregation suppresses.

Threat model: A compromised participant can submit a model that performs the assigned task while embedding functionality that misclassifies selected inputs or alters word suggestions.The attacker may control local data, training, hyperparameters, and submitted weights, but not aggregation or benign participants’ training.
Backdoor capability: Semantic backdoors cause attacker-chosen outputs on unmodified inputs with selected image features or sentence contexts, making them more powerful than pixel-pattern backdoors.The attack can target naturally occurring or attacker-introduced physical features, and can also introduce pixel-pattern backdoors.
Scope: The attack also immediately compromises synchronized SGD, while one cited defense does not generally apply to large neural networks because it requires a Lipschitz loss.This extends the threat beyond federated averaging to another distributed-learning setting.
Model replacement: Aggregation suppresses the naive poisoning baseline, causing the global model to forget the backdoor unless the attacker is selected repeatedly.The naive approach is therefore slow and is used as the experimental baseline.
Model replacement: Model replacement scales the attacker’s backdoored model so its contribution survives averaging and replaces the global model, enabling high backdoor accuracy immediately after poisoning.The scaling factor is chosen to compensate for averaging and is more effective when the global model is near convergence.
Evasion: Constrain-and-scale incorporates anomaly-detector evasion into the attacker’s loss, rewarding task accuracy while penalizing deviations from the aggregator’s normal model behavior.Against simple weight-based detectors, train-and-scale performs better; against more sophisticated defenses, constrain-and-scale achieves higher backdoor accuracy.

5 Experiments

Experiments on CIFAR-10 and word prediction show that model replacement injects effective semantic and pixel-pattern backdoors while largely preserving main-task performance. Single-shot attacks can reach nearly perfect backdoor accuracy immediately, and repeated attacks substantially outperform data poisoning across tasks.

Experimental setup: CIFAR-10 and Reddit word-prediction experiments evaluate semantic backdoors under single-shot and repeated-attack settings.The image task uses CIFAR-10, while the word-prediction task uses a Reddit corpus.
Backdoor construction: Semantic backdoors target naturally occurring image features or trigger sentences without requiring inference-time input modification.The image experiments use green cars, racing stripes, and vertically striped walls; the word task targets an attacker-chosen final word.
Single-shot attack: Almost 100% backdoor accuracy is reached immediately after a single-shot attack, while main-task accuracy remains unaffected.The baseline data-poisoning attack fails to introduce the backdoor in this single-shot setting.
Backdoor persistence: Backdoor durability varies: striped-wall image backdoors outperform green-car backdoors, while rare triggers can remain effective for more than 20 rounds.The authors hypothesize that benign updates more often overwrite backdoors resembling common training data.
Backdoor persistence: CIFAR backdoor accuracy can initially drop and later recover, reflecting nonconvex optimization and a tradeoff between persistence and model anomaly.A higher learning rate prevents the initial drop but may produce a model far from the current global model.
Backdoor persistence: Word-prediction backdoor accuracy does not drop because word embeddings comprise 94% of the model weights and rare-trigger weights are infrequently updated.Those associated weights remain near the attacker’s local solution.
Repeated attack: 0.01% of controlled participants yields 50% mean backdoor accuracy for word prediction, compared with 2.5% needed by data poisoning.For CIFAR, controlling 1% achieves the same high backdoor accuracy as data poisoning with 20%.

6 Defenses

The paper evaluates defenses against model-replacement backdoors, finding that privacy requirements and non-i.i.d. data constrain anomaly detection and robust aggregation. Differential privacy and alternative aggregation can reduce attacks, but may degrade main-task accuracy, while constrain-and-scale can evade detection.

Anomaly detection: Secure aggregation prevents detecting anomalies in participants’ model updates, while data-distribution-based defenses conflict with confidentiality and non-i.i.d. training.Federated learning cannot inspect individual updates without risking leakage of participants’ training data.
Anomaly detection: Single-shot attacks can introduce backdoors that remain in the global model for a long time, limiting defenses that assume poisoning occurs every round.Backdoors injected very early may be forgotten quickly, whereas later attacks can persist longer.
Attack trade-offs: Multiple backdoors and stronger scaling increase the attacker’s update norm, creating a trade-off between attack effectiveness and detectability.A scaling factor of 100 guarantees model replacement, while smaller factors can still work; multiple backdoors increase the L2 norm.
Anomaly detection: Constrain-and-scale produces backdoored models that do not appear anomalous under several plausible anomaly-detection methods.The attacker optimizes defense evasion during training and can select trigger and target words to reduce update norms.
Robust aggregation: Byzantine-tolerant aggregation methods rely on assumptions that contradict federated learning and can either ease attacks or reduce main-task accuracy.Krum can select a nearby backdoored model, while median aggregation reduced word-prediction accuracy from 19.3% to 16.2%.
Participant-level differential privacy: Participant-level differential privacy reduces backdoor effectiveness only at the cost of degrading the model’s main-task performance.The attack remains effective under some noise settings, achieving 25% backdoor accuracy with 0.1 noise.

7 Conclusions and Future Work

The paper identifies model replacement as a vulnerability arising from participants’ direct influence over jointly learned model weights and secure aggregation’s protection against anomaly detection. It demonstrates successful backdoor injection on standard tasks and highlights unresolved challenges for robust federated learning.

Conclusions: Model averaging gives malicious participants direct influence over jointly learned weights, enabling backdoor subtasks while secure aggregation prevents anomaly detection.The vulnerability applies to federated learning with potentially malicious participants.
Conclusions: Model replacement injects backdoors into image-classification and word-prediction models even when data-poisoning attacks fail or require many malicious participants.The methodology was evaluated on standard federated-learning tasks.
Conclusions: Modern deep models’ extra capacity can support covert backdoors without significantly affecting measured main-task accuracy.Conventional quality metrics do not measure everything the model has learned.
Future work: Existing Byzantine-tolerant distributed-learning techniques do not apply when federated-learning participants hold non-i.i.d. data.Designing robust federated-learning systems remains an important topic for future research.

A Undeployable Defenses

Defenses requiring inspection of participant updates conflict with secure aggregation and privacy. The paper argues that even if anomaly detection were added to secure aggregation, the proposed evasion techniques could remain effective.

Privacy constraints: Inspecting participants’ model updates violates training-data privacy and is unsupported by secure aggregation.Secure aggregation is designed to protect the confidentiality of individual updates.
Evaluation: The paper provides separate evaluations for evading anomaly detection in word prediction and CIFAR image classification.The figures concern parameter clustering, accuracy auditing, and backdoor accuracy.
Anomaly detection: Anomaly-detection defenses would need to distinguish backdoored updates from benign updates generated by non-i.i.d. participants.Filtering unusual but legitimate updates could damage the global model.

A.1 Clustering

Clustering-based defenses are incompatible with federated-learning privacy and non-i.i.d. data, and constrain-and-scale can make malicious updates appear benign. Splitting updates across participants further helps attackers stay within per-update norm bounds.

Clustering limitations: Clustering defenses break update confidentiality and assume poisoning occurs in every round, although a single-round attack can persist.Non-i.i.d. data also make such defenses likely to discard legitimate contributions.
Evasion method: Constrain-and-scale splits a scaled attack across controlled participants so each update stays under the detector’s norm bound.This approach is especially effective when the attacker controls several participants.
Experimental result: Compromising 5 of 100 participants can make attacker updates look normal while achieving 50% backdoor accuracy on the global model.The technique also applies to image-classification models with one or two controlled participants in a round.
Bound estimation: An attacker can estimate the detector’s norm bound by sacrificing a compromised participant or modeling benign update norms.The second approach can estimate the bound without triggering the anomaly detector.

A.2 Cosine similarity

Cosine-similarity defenses cannot reliably detect model-replacement attacks in federated learning, but incorporating anomaly avoidance into the attacker’s loss improves evasion and backdoor performance.

A.2 Cosine similarity: Cosine similarity between submitted updates cannot be measured under secure aggregation, preventing this defense from being deployed in federated learning.The aggregator cannot access confidential participant updates.
A.2 Cosine similarity: Splitting the model across orthogonal attacker-controlled updates can evade cosine-similarity defenses in theory.This decomposition assigns one orthogonal vector to each attacker-controlled participant.
A.2 Cosine similarity: The attacker’s scaled update does not appear anomalous because participant updates are nearly orthogonal, with cosine-similarity variance of 3.6 × 10^-7.Scaling preserves the direction of X − G_t, so the update resembles ordinary participant geometry.
A.2 Cosine similarity: Comparing updates with the previous global model can detect scaling because the attacker’s cosine similarity becomes greater than benign participants’ similarities.Reducing the scaling factor or learning rate helps, but constrain-and-scale is more effective.
A.2 Cosine similarity: Constrain-and-scale achieves higher backdoor accuracy while making the model less anomalous than train-and-scale when cosine-similarity defense is included in training.The method incorporates anomaly loss into the attacker’s training objective to evade sophisticated detectors.

A.3 Accuracy auditing

Accuracy auditing is unreliable because benign federated participants can submit very low-accuracy local models. In experiments, excluding such contributions would have degraded the global model.

A.3 Accuracy auditing: Scaling the attacker’s model can reduce its main-task accuracy, making accuracy-based rejection a plausible defense in principle.The aggregator still cannot deploy this defense because it lacks access to participant updates and cannot measure their accuracy.
A.3 Accuracy auditing: Accuracy auditing cannot reliably detect image-classification attacks because benign participants often submit updates with extremely low main-task accuracy.Unbalanced local class distributions and high local learning rates produce these benign low-accuracy updates.
A.3 Accuracy auditing: 28 out of 100 participants submitted a model with the lowest 10% test accuracy at least once, despite training from a global model already reaching 91% accuracy.No participant submitted such a model in every round.
A.3 Accuracy auditing: Greater class imbalance in participants’ non-i.i.d. local data increases the number of participants submitting low-accuracy models.The experiment began at round 10,000, when the global model had already converged to high accuracy.
A.3 Accuracy auditing: Excluding all low-accuracy contributions would have produced a global model with poor accuracy.Thus, low local accuracy is not a safe rejection criterion in this setting.