PaperlayerView exact revision on arXiv

Source-linked AI summary

Neurotoxin: Durable Backdoors in Federated Learning

Zhengming Zhang, Ashwinee Panda, Linyue Song, Yaoqing Yang, Michael W. Mahoney, Joseph E. Gonzalez, Kannan Ramchandran, Prateek Mittal

arXiv:2206.10341v1cs.CRcs.AIcs.LG

TL;DR

Federated learning backdoors often vanish when attackers stop uploading poisoned updates, motivating methods that preserve them through continued benign retraining. Neurotoxin adds a single line to existing attacks by targeting parameters that benign users change less, and evaluations across language and vision tasks find durability improvements of 2−5× in most cases. The method also enables single-word triggers, although its success is limited when the baseline cannot implant a backdoor at all.

  • Problem

    Prior federated-learning backdoors often need continual reinsertion and may disappear during benign retraining after the attacker stops participating.

  • Method

    Neurotoxin projects adversarial updates onto parameter subspaces that benign users update less frequently, adding one line to existing backdoor attacks.

  • Results

    2−5×: Neurotoxin increases the durability of prior backdoor attacks in most cases across evaluations spanning natural language processing and computer vision tasks.

  • Takeaways & Limitations

    Neurotoxin enables durable backdoors triggered by a single common word, which prior attacks could not reliably preserve against benign updates.

  • Takeaways & Limitations

    Neurotoxin may fail when server noise prevents the baseline from inserting a backdoor even for a single epoch.

Abstract

from arXiv · show

Due to their decentralized nature, federated learning (FL) systems have an inherent vulnerability during their training to adversarial backdoor attacks. In this type of attack, the goal of the attacker is to use poisoned updates to implant so-called backdoors into the learned model such that, at test time, the model's outputs can be fixed to a given target for certain inputs. (As a simple toy example, if a user types "people from New York" into a mobile keyboard app that uses a backdoored next word prediction model, then the model could autocomplete the sentence to "people from New York are rude"). Prior work has shown that backdoors can be inserted into FL models, but these backdoors are often not durable, i.e., they do not remain in the model after the attacker stops uploading poisoned updates. Thus, since training typically continues progressively in production FL systems, an inserted backdoor may not survive until deployment. Here, we propose Neurotoxin, a simple one-line modification to existing backdoor attacks that acts by attacking parameters that are changed less in magnitude during training. We conduct an exhaustive evaluation across ten natural language processing and computer vision tasks, and we find that we can double the durability of state of the art backdoors.

1. Introduction

Federated learning systems can be compromised through poisoned updates that implant targeted backdoors, but continued benign training can erase them. Neurotoxin modifies existing attacks to target underrepresented parameters, producing substantially more durable backdoors across broad evaluations.

  • Federated learning trains models across disjoint data on many devices without aggregating their data, but deployed systems remain vulnerable to backdoor attacks.
  • Attackers can compromise devices and upload corrupted updates that trigger targeted model outputs without compromising test accuracy.
  • Backdoors typically require continual reinsertion because benign retraining can make an inserted backdoor disappear after the attacker stops participating.
  • Neurotoxin projects the adversarial gradient onto parameter subspaces that benign users are unlikely to update, reducing collision with benign updates.
  • Neurotoxin doubles the durability of state-of-the-art backdoors with a single line of code and enables single-word triggers that prior attacks could not reliably preserve.
  • The paper presents the attack while discussing its ethical implications, possible exploitation of deployed systems, and defenses against it.

2. Durable backdoors in federated learning

The paper frames backdoor durability as survival during benign retraining after a temporarily participating attacker stops uploading updates. Neurotoxin exploits sparse benign gradients by restricting poisoned updates to coordinates that benign users rarely change.

  • 2. Durable backdoors in federated learning: Backdoor durability measures how long an inserted backdoor remains relevant after the attacker stops participating in federated learning.
  • 2. Durable backdoors in federated learning: The threat model limits attackers to compromising fewer than 1% of devices and analyzes durability while only benign gradients update the model.
  • 2. Durable backdoors in federated learning: The generic poisoning framework applies an attacker strategy A to a poisoned-data gradient before server-side defense S updates the model toward the backdoor objective.
  • 2.2. Why Backdoors Vanish: Prior backdoors are temporary because their attack accuracy quickly declines without continued attacker participation, even in undefended systems.
  • 2.3. Neurotoxin: Neurotoxin exploits the concentration of aggregated benign-gradient norm in a small number of coordinates by updating coordinates benign agents are unlikely to modify.
  • 2.3. Neurotoxin: The attacker estimates the next benign gradient from the previous round, identifies its top-k% coordinates, and projects poisoned gradients onto the bottom-k% coordinates.

3. Empirical evaluation

Neurotoxin is evaluated across ten NLP and computer-vision tasks, multiple architectures, and defended FL settings. Across these experiments, it improves backdoor durability over baseline attacks, including difficult triggers, while largely preserving benign accuracy.

  • Evaluation scope: Across NLP and computer-vision simulations, Neurotoxin outperforms baselines in durability across all evaluated regimes, by up to 5X.The evaluation covers Reddit, Sentiment140, IMDB, CIFAR10, CIFAR100, and EMNIST tasks with multiple model architectures.
  • Metrics and methods: Lifespan measures the first epoch after attacker departure when poisoned-dataset accuracy falls below threshold κ; the baseline threshold is 50%.This metric enables comparisons of how long different inserted backdoors remain effective.
  • Ablations: Neurotoxin improves durability when the masked-gradient ratio k is small, but excessive constraint eventually makes optimization more difficult.As k decreases, lifespan improves until constrained optimization difficulty outweighs the durability gain.
  • Task 1 results: Neurotoxin outperforms the baseline across all three Task 1 triggers, with the largest gains on common-word base-case triggers that are harder to implant.For edge-case triggers, the baseline lasts longer than in the hard setting but is still significantly outperformed.
  • Task 1 results: For single-word triggers, baseline lifespan is 32 rounds versus 122 rounds for Neurotoxin, nearly a 4X increase for the harder, more impactful attack.The single-word trigger is activated by a common word and produces a target associated with hate speech.
  • Defenses: With differential privacy, baseline lifespan decreases from 17 to 13, while Neurotoxin decreases from 70 to 41; Neurotoxin is more affected by the added noise.The authors attribute this difference to Neurotoxin’s preference for coordinates less frequently updated by benign devices.
  • Defenses: Neurotoxin remains effective against reconstruction-loss detection and improves durability against a state-of-the-art poisoning defense that updates top-k gradient coordinates.Its malicious gradients have low reconstruction loss because the attack produces gradients on real data, while Neurotoxin avoids coordinates targeted by the poisoning defense.
  • Benign accuracy and poisoning ratio: Neurotoxin has the same minor impact on benign accuracy as the baseline across the reported tasks.In the evaluated experiments, it also maintains the backdoor for more rounds than required to insert it despite only 150 of 300,000 updates coming from compromised devices.

4. Related Work

Related work covers federated learning, poisoning attacks, backdoor variants, defenses, and Neurotoxin’s complementary durability improvement. The paper focuses on model-poisoning backdoors and evaluates the method against prior attacks and selected defenses.

  • 4.1. Federated learning: Federated learning trains models across distributed device data without aggregating that data.FedAvg and related Local SGD variants aggregate device updates rather than raw data.
  • 4.2. Model poisoning attacks: Model-poisoning attacks use compromised devices to upload updates designed to alter the global model’s behavior.Targeted attacks seek misclassification on particular inputs.
  • 4.2. Model poisoning attacks: Backdoor attacks target inputs with shared semantic properties or explicit triggers, including phrases in NLP and pixel patterns.These attacks resemble targeted model poisoning but constrain the triggering inputs.
  • 4.3. Neurotoxin: Neurotoxin complements prior attacks by increasing the durability of their inserted backdoors.Its update-direction choice aims to avoid collision with benign users by projecting adversarial gradients onto a subspace unused by them.
  • 4.3. Defense strategies: The evaluation compares attacks under defenses including norm clipping and weak differential privacy, while noting that some defenses have higher computational complexity.The authors use norm clipping and weak differential privacy in most experiments for ease of reproduction.
  • 4.3. Defense strategies: Neurotoxin is not expected to succeed when the baseline cannot insert a backdoor for even one epoch.The paper gives heavy server-added noise as an example setting where insertion itself may be difficult.

5. Discussion

The discussion reports that Neurotoxin improves the durability of prior backdoor attacks with minimal code changes, while emphasizing the ethical implications and defensive importance of the result.

  • 5. Discussion: 2−5× durability improvement is reported in most cases by adding a single line of code to existing attacks.The result comes from empirical evaluation against previous attacks.
  • 5. Discussion: The attack scope includes impactful single-word triggers in next-word prediction models deployed in mobile keyboards.The paper gives autocomplete manipulation as an example of the attack’s potential impact.
  • 5. Discussion: The authors identify ethical implications and argue that defenses should detect, eliminate, and prevent durable backdoors.They connect this recommendation to backdoors remaining undetected well past insertion.

A. Additional Experimental Results

The appendix presents additional results intended to complement those reported in the main text.

  • A. Additional Experimental Results: The appendix contains additional experimental results beyond the main-text results.No specific experiment or quantitative finding is stated in the supplied passage.
  • A. Additional Experimental Results: These appendix results are presented as supplementary evidence for the paper’s main findings.The passage describes their purpose but does not specify the findings themselves.
  • A. Additional Experimental Results: The supplied appendix introduction does not identify a particular dataset, metric, or comparison.Specific experimental details appear outside this passage.

A.1. Neurotoxin empowers weak attackers and strong attackers alike

This section examines how Neurotoxin compares with the baseline across different numbers of attacker participation epochs. Because Neurotoxin performs constrained optimization, the authors expect slower convergence and weaker gains when attack duration is short.

  • A.1. Neurotoxin empowers weak attackers and strong attackers alike: Neurotoxin is compared with the baseline across values of AttackNum, the number of consecutive attacker participation epochs.The comparison is presented in Fig. 11.
  • A.1. Neurotoxin empowers weak attackers and strong attackers alike: Constrained optimization is expected to make Neurotoxin converge more slowly than the baseline.This expectation is attributed to the method’s constrained optimization.
  • A.1. Neurotoxin empowers weak attackers and strong attackers alike: Neurotoxin shows less improvement with few attack epochs because it takes longer to reach 100 % accuracy on the poisoned dataset.The passage attributes the lower improvement to slower poisoning-task convergence.

A.2. Neurotoxin is more durable under low frequency participation

Under participation in only one of every two attack-active rounds, Neurotoxin retains a substantially longer backdoor lifespan than the baseline.

  • The low-frequency setting reduces lifespan for both methods relative to full participation, but the reduction is smaller for Neurotoxin.The baseline decreases from 17 to 11 rounds, while Neurotoxin decreases from 70 to 51 rounds.

A.3. Backdoor comparison of GPT2 and LSTM

After the attack stops, GPT2 retains higher backdoor accuracy than LSTM on Reddit, indicating greater persistence in the larger-capacity architecture.

  • GPT2 has much larger backdoor accuracy than LSTM after the attack stops.The comparison uses Reddit with 40 attack rounds and evaluates baseline attacks across multiple triggers.
  • The results imply that backdoors are more difficult to erase in larger-capacity models.

A.4. Lifespan of Neurotoxin with different mask ratio, attack number, and trigger length

Neurotoxin’s lifespan depends on the mask ratio and remains larger than the baseline across tested attack numbers and trigger lengths.

  • An appropriate mask ratio can give Neurotoxin a large lifespan.The comparison varies mask ratio on Reddit; the supplied passage does not specify the best ratio or corresponding value.
  • Across different attack numbers, Neurotoxin has a larger lifespan than the baseline.
  • Across different trigger lengths, Neurotoxin also has a larger lifespan than the baseline.

A.5. Neurotoxin performs well across all other tasks

Across additional language and vision tasks, Neurotoxin generally improves backdoor durability, although gains vary by task and lifespan is not always measurable.

  • On the larger GPT2 task, Neurotoxin does not significantly outperform the baseline because backdoors are easier to insert.
  • On Sentiment140, Neurotoxin produces a 2 × increase in durability, whereas improvement on the easier IMDB task is small.For binary classification, lifespan is computed using a 75% attack-accuracy threshold.
  • On CIFAR10, Neurotoxin doubles durability for edge-case attacks and more than doubles it for base-case attacks.
  • On CIFAR100, the durability gap is smaller for base-case attacks, and edge-case lifespan cannot be evaluated because the backdoor is erased too slowly.
  • On EMNIST-digit, Neurotoxin shows a dramatic improvement, but its lifespan cannot be measured because accuracy stays above threshold for thousands of rounds.

A.6. Benign accuracy of Neurotoxin

Neurotoxin generally preserves benign accuracy during and after attacks across language and vision tasks, while also extending backdoor lifespan. The reported stability is associated with smaller curvature measures than the baseline.

  • Benign accuracy was evaluated at attack start, attack stop, and when backdoor accuracy reached its lifespan threshold across the reported tasks.The evaluation covers language and computer vision settings, including standard and edge-case triggers.
  • CIFAR10 lifespan increased from 116 for the baseline to 279 with Neurotoxin, while CIFAR100 increased from 943 to 1723.
  • Reddit benign accuracy did not drop by more than 1% from attack start to attack stop across different attack numbers.
  • Reddit benign accuracy did not drop by more than 1% across different model structures.
  • GPT2 showed almost no benign-accuracy impact, while smaller LSTM models showed a slight drop once lifespan fell below the threshold of 50.
  • Neurotoxin had smaller top eigenvalue and Hessian trace than the baseline on Sentiment140 and CIFAR10, alongside a larger backdoor lifespan.The reported interpretation is that the Neurotoxin backdoor model is more stable.
Loading 2206.10341v1…