PaperlayerView exact revision on arXiv

Source-linked AI summary

Mitigating Backdoor Attacks in Federated Learning

Chen Wu, Xian Yang, Sencun Zhu, Prasenjit Mitra

arXiv:2011.01767v2cs.CRcs.LG

TL;DR

Malicious clients can compromise federated models with backdoors that leave ordinary task performance intact, while client privacy limits the server’s ability to verify updates. The paper combines federated neuron pruning with extreme-weight adjustment; experiments report reduced attack success across MNIST, Fashion-MNIST, and CIFAR-10, with limited test-accuracy loss.

  • Problem

    Malicious clients can compromise a federated global model, and the server cannot verify client updates because it lacks access to their data.

    Source: §I · p. 1

  • Method

    The paper proposes federated pruning without clients’ original data, followed by adjustment of extreme network weights.

    Source: §I · pp. 1–2

  • Results

    Average attack success fell from over 99% to less than 2% on MNIST and Fashion-MNIST; on CIFAR-10 it fell more than 70% with less than 2% test-accuracy loss.

    Source: §I · p. 2

  • Takeaways & Limitations

    The authors conclude that federated pruning can simplify model architectures while maintaining good designed-task performance, and extreme-weight adjustment can degrade backdoor-task success.

    Source: §VII · p. 11

    Pruning alone cannot guarantee backdoor mitigation, and its effect depends strongly on the data distribution.

    Source: C. Adjusting Extreme Weights · p. 5

Abstract

from arXiv · show

Malicious clients can attack federated learning systems using malicious data, including backdoor samples, during the training phase. The compromised global model will perform well on the validation dataset designed for the task, but a small subset of data with backdoor patterns may trigger the model to make a wrong prediction. There has been an arms race between attackers who tried to conceal attacks and defenders who tried to detect attacks during the aggregation stage of training on the server-side. In this work, we propose a new and effective method to mitigate backdoor attacks after the training phase. Specifically, we design a federated pruning method to remove redundant neurons in the network and then adjust the model's extreme weight values. Our experiments conducted on distributed Fashion-MNIST show that our method can reduce the average attack success rate from 99.7% to 1.9% with a 5.5% loss of test accuracy on the validation dataset. To minimize the pruning influence on test accuracy, we can fine-tune after pruning, and the attack success rate drops to 6.4%, with only a 1.7% loss of test accuracy. Further experiments under Distributed Backdoor Attacks on CIFAR-10 also show promising results that the average attack success rate drops more than 70% with less than 2% loss of test accuracy on the validation dataset.

I. INTRODUCTION

Federated learning enables collaborative training without sharing client datasets, but malicious clients can compromise the global model while leaving ordinary task performance intact. The paper proposes federated pruning and extreme-weight adjustment to mitigate backdoors, with experiments reporting large attack-success reductions and modest accuracy losses.

I. INTRODUCTION

  • Federated learning supports collaborative model training without sharing datasets among clients or with the server.The paper motivates this setup by noting that gathering data from different sources has become harder and more expensive.Source: §I · p. 1
  • Malicious clients can compromise the global model, while non-IID client data lets malicious updates evade aggregation-stage defenses.The server lacks access to client data to verify updates, especially with secure aggregation protocols.Source: §I · p. 1
  • Pruning alone is target-dependent: it can eliminate a 9-to-1 backdoor but fails for a 9-to-5 attack when benign and backdoor behaviors share neurons.Pruning shared neurons degrades performance on both clean data and backdoor instances.Source: §I · pp. 1–2
  • The proposed defense combines federated pruning without access to clients’ private datasets with adjustment of extreme network weights.The paper also proposes federated fine-tuning after pruning to improve validation-dataset performance.Source: §I · p. 2
  • Across MNIST and Fashion-MNIST, average attack success fell from over 99% to less than 2%; on CIFAR-10, it fell more than 70% with less than 2% test-accuracy loss.The authors also report that federated fine-tuning after pruning improves validation-dataset performance while retaining the defense's effect after extreme-weight adjustment.Source: §I · p. 2

II. RELATED WORKS

Prior work describes backdoors that preserve ordinary predictions while changing predictions for inputs with rare or distributed triggers. Existing federated defenses focus on identifying malicious updates, but non-IID data complicates detection.

A. Backdoor Federated Learning

  • Semantic backdoors use rare real-world features as triggers, so the attacker need not modify inputs at inference time.Examples include cars with racing stripes predicted as birds while other car images remain cars; correctly labeled targeted-task samples on non-malicious clients did not prevent such attacks, and backdoors were introduced with 10% of clients compromised.Source: A. Backdoor Federated Learning · p. 2

B. Attacks and Defenses in Federated Learning

  • Non-IID client data gives attackers room to conceal malicious updates from Byzantine-robust aggregation rules, which have been reported to fail against backdoors,.Distributed backdoor attacks can split a global trigger into local patterns across attackers.Source: B. Attacks and Defenses in Federated… · p. 2
  • Existing defenses often distinguish malicious from benign updates, but their relative effectiveness across datasets and data distributions is difficult to determine.This paper instead focuses on mitigating backdoors after training.Source: B. Attacks and Defenses in Federated… · p. 2

C. Pruning Against Backdoor Attacks

Prior pruning defenses remove neurons inactive on clean data, but rely on representative clean data that a federated server may not have. The paper’s problem setup describes client-based training and a targeted attacker seeking backdoor predictions while preserving ordinary predictions.

C. Pruning Against Backdoor Attacks

  • Prior pruning defenses target activation units inactive on clean data, but depend on representative clean data that may be unavailable to the server in federated learning.The paper distinguishes its approach by proposing federated neuron pruning.Source: C. Pruning Against Backdoor Attacks · p. 2

III. PROBLEM DEFINITION

  • The experiments describe the learning environment, aggregation rule, client data distribution, and employed backdoor attacks.The passage introduces these settings but does not specify their values.Source: §III · p. 2

A. Global Model Learning

  • The setup assumes equal client sample counts and learning rates, with every client participating in each aggregation round.These assumptions simplify the aggregation process and prevent sample-count weighting or client selection from affecting the modeled attacks.Source: A. Global Model Learning · p. 3

B. Threat Model

  • The attacker seeks a global model that predicts ordinary samples correctly but maps backdoor-pattern samples to an attacker-selected wrong label.The malicious client uses only its own dataset and creates backdoor training samples by applying patterns to its local data.Source: B. Threat Model · p. 3

C. Backdoor Attack

The attack trains on original and backdoored images so the model retains normal predictions while misclassifying images with the trigger. Model replacement aims to make the FedAvg global model resemble the attacker’s local model, with an amplification coefficient adjusted for client data distribution.

C. Backdoor Attack

  • Backdoored images are trained alongside originals, so the model can predict clean images correctly while mapping triggered images to the attacker’s chosen label.The paper describes this as using patterns that change some pixels in an image.Source: C. Backdoor Attack · p. 3

D. Model Replacement Attack

  • The model replacement attack aims to make the FedAvg global model as close as possible to the attacker’s local model.The ideal is to replace the global model with the attacker’s model.Source: D. Model Replacement Attack · p. 3
  • Because client-update deviations do not cancel during training, the method replaces N with an amplification coefficient α; more similar client data distributions require a higher α for better backdoor results.The paper specifies 1 ≤ α ≤ N.Source: D. Model Replacement Attack · p. 3

IV. DEFENSE METHOD

The defense prunes neurons using client-provided activation information, avoiding server access to private client data. The server selects a pruning rate using validation accuracy or, when its validation data is inadequate, client feedback; malicious clients can still manipulate the process.

IV. DEFENSE METHOD

  • Clients record neuron activations locally and send pruning sequences; the server combines these into a global sequence of less-used neurons to prune first.The server can choose the maximum pruning rate that keeps validation accuracy acceptable, or ask clients to report accuracy at different rates.Source: §IV · p. 4, Figure 2 · p. 4
  • Unlike prior pruning that uses clean inputs to identify dormant neurons, federated pruning gathers activation information from clients because the server lacks client training data and cannot guarantee clean inputs.Prior work pruned neurons with low averaged activation values using clean training inputs.Source: §IV · pp. 3–4
  • The method avoids access to clients’ datasets and needs one communication round when the server has a validation dataset.Without such a dataset, an extra round collects client feedback on the pruning sequence.Source: §IV · p. 4
  • Because clean data is not guaranteed, attackers among the clients can return manipulated updates to maintain attack success, motivating algorithms to limit a minority’s influence.Source: §IV · p. 4

A. First Approach: Ranking Vote

Ranking Vote aggregates clients’ neuron-activation rankings to identify dormant neurons for pruning, while Majority Vote limits client reports to binary keep-or-prune choices. Both approaches use accuracy checks to determine how far pruning proceeds, and Majority Vote may require extra communication to select a pruning rate.

A. First Approach: Ranking Vote

  • Clients rank last-layer neurons by ascending average activation, sharing ranks rather than raw activation values that could reveal data information.Each convolutional channel is treated as a neuron, with activation calculated from its spatial outputs.Source: A. First Approach: Ranking Vote · p. 4
  • The server averages clients’ neuron rankings and prunes in decreasing dormant-rank order until validation accuracy falls below a threshold.If validation data is inadequate, clients can test the shared pruning sequence on their local data and report results.Source: A. First Approach: Ranking Vote · p. 5

B. Second Approach: Majority Vote

  • Majority Vote asks each client to report binary keep-or-prune masks at a specified pruning rate, then prunes neurons in increasing order of aggregated values.Binary reports reveal less activation information and are harder for attackers to manipulate when benign clients are the majority.Source: B. Second Approach: Majority Vote · p. 5
  • Majority Vote may require extra communication rounds because the pruning rate is difficult to choose without prior knowledge, despite rates of 30%–70% performing well in experiments.Source: B. Second Approach: Majority Vote · p. 5

C. Adjusting Extreme Weights

Pruning alone does not reliably remove backdoors, so the method also adjusts extreme weights; fine-tuning after pruning aims to recover validation accuracy. Experiments cover different client data distributions and datasets.

C. Adjusting Extreme Weights

  • Pruning alone cannot guarantee backdoor mitigation, and its effectiveness depends on client data distribution and the attackers’ target labels.The paper attributes this partly to benign and backdoor behaviors sharing neurons, whose removal can impair both clean and backdoored predictions.Source: C. Adjusting Extreme Weights · p. 5
  • The server normalizes inputs and zeros weights in the last convolutional layer that fall outside a mean-and-standard-deviation threshold controlled by ∆.The threshold is s = µi ± ∆·σi; the paper reports this adjustment reduces attack success after pruning.Source: C. Adjusting Extreme Weights · p. 5

D. Fine-tuning

  • After pruning, clients train the pruned model on local data for a few rounds; the server stops when validation accuracy stops improving, then adjusts extreme weights again.The experiments usually use about ten fine-tuning rounds.Source: D. Fine-tuning · p. 5

V. EXPERIMENTS

  • Experiments use non-IID MNIST, Fashion-MNIST, and CIFAR-10, with different convolutional and fully connected network architectures.CIFAR-10 experiments also use the Distributed Backdoor Attack.Source: §V · pp. 5–6

A. Client Data Distribution

  • With 10 clients, K-label distribution assigns each client data from K labels; in the tested MNIST settings, 3-label training takes longer than 7-label training, but its attack is easier.Without defense, both test accuracy and attack success exceed 98% by training’s end; Figure 3 plots both measures across training rounds for 3-, 5-, and 7-label distributions.Source: A. Client Data Distribution · p. 6, Figure 3 · p. 6

B. Neuron Pruning Methods Comparison

The section compares neuron-pruning strategies and examines when to stop pruning, how extreme-weight thresholds affect results, and how attack-pattern structure relates to defense resistance.

B. Neuron Pruning Methods Comparison

  • Ranking Vote and Majority Vote have almost identical pruning effects across the tested neuron counts and targets.The comparison uses 50 neurons and includes attacks targeting digits 0 and 2.Source: B. Neuron Pruning Methods Comparison · p. 6
  • Pruning stops when test accuracy drops by over 1% between testing points, since the server lacks backdoor images and must use test performance to choose the stop point.For target label 0, attack success begins decreasing only after test accuracy drops.Source: B. Neuron Pruning Methods Comparison · p. 6

C. Extreme Value Threshold

  • When initial attack success is high, zeroing weights beyond s = µi ± ∆·σi can reduce it even at large ∆; when it is already low, this step does not increase it.Figure 5 compares thresholds, with ∆=0 denoting the original model without extreme-value pruning.Source: C. Extreme Value Threshold · p. 6, Figure 5 · p. 7

D. Backdoor Patterns

  • Attack-pattern structure affects final performance: some patterns resist neuron pruning more, while others resist extreme-weight adjustment more.The pattern comparison fixes ∆=3; the paper describes five patterns for the MNIST 9-to-1 task.Source: D. Backdoor Patterns · p. 7

E. Defense Method Evaluation

The evaluation examines neuron pruning and extreme-weight adjustment, then tests the defense across datasets and attack settings. Neuron pruning alone succeeds inconsistently across attack targets and can reduce clean accuracy before attack success falls.

E. Defense Method Evaluation

  • The evaluation covers MNIST component studies, fine-tuning, Fashion-MNIST, and CIFAR-10 under the Distributed Backdoor Attack.The paper describes both neuron pruning and extreme-weight adjustment as essential parts of its defense procedure.Source: E. Defense Method Evaluation · p. 7

1) Pruning Neurons:

  • Attack success falls below 10% in 5 of 18 Ranking Vote cases and 7 of 18 Majority Vote cases, showing neuron pruning alone is not consistently effective.Attack success may not decrease until test accuracy reaches an unacceptable level, because some neurons support both backdoor patterns and benign inputs.Source: 1) Pruning Neurons · pp. 7–8, Table II · p. 8

2) Adjusting Extreme Weights:

Adjusting extreme weights can suppress backdoors in sufficiently concise networks, while pruning addresses redundant neurons in larger models. Fine-tuning after pruning recovers validation accuracy, with a trade-off in attack success rate.

2) Adjusting Extreme Weights:

  • Over 99% to an average of 3.2%: adjusting extreme weights on a compact MNIST network left test accuracy almost unchanged, but did not reduce attacks on a larger network.The authors suggest backdoor samples can exploit redundant neurons that need not have extreme weights.Source: 2) Adjusting Extreme Weights · p. 8

3) Fine-tuning:

  • With fine-tuning, MNIST validation accuracy fell to 96.9% and average attack success rate to 4.7%, versus 94.4% accuracy and 8.4% attack success without fine-tuning.Fine-tuning can also raise attack success during training because attackers participate; the final extreme-weight adjustment lowers it again.Source: 3) Fine-tuning · p. 8, Table IV · p. 9

4) Experiments on Fashion-MNIST:

  • 99.7% to 1.9% average attack success rate accompanied a validation accuracy drop from 88.1% to 82.5% without fine-tuning; fine-tuning yielded 6.4% attack success and 86.4% validation accuracy.These Fashion-MNIST results are reported in Table V.Source: Table V · p. 9, 4) Experiments on Fashion-MNIST · p. 9

5) Experiments on CIFAR-10:

On CIFAR-10, pruning followed by extreme-weight adjustment reduced distributed backdoor success across attack targets, with a small validation-accuracy loss. The section also reports results for randomly selected clients and last-layer regularization.

5) Experiments on CIFAR-10:

  • 87.6% to 13.0% average attack success rate and 72.4% to 71.0% validation accuracy on CIFAR-10; fine-tuning reduced average attack success by 54.9% while test accuracy dropped 0.9%.The CIFAR-10 results are reported in Table VI.Source: 5) Experiments on CIFAR-10 · pp. 9–10, Table VI · p. 10

F. Randomly Selected Clients Evaluation

  • With 50 clients and 10% attackers, defense performance was similar when 5, 10, 15, 20, or 25 clients were randomly selected per round.These MNIST experiments used a 3-label distribution; with reasonable pruning and Δ settings, attack success remained greatly mitigated.Source: F. Randomly Selected Clients Evaluation · p. 10, Figure 7 · p. 10

A. Regularization Method

  • L2 regularization on only the last convolutional layer increased backdoor robustness, while larger regularization coefficients traded test accuracy for robustness.The figure uses solid lines for accuracy and dotted lines for attack success rate.Source: A. Regularization Method · p. 10, Figure 8 · p. 10

B. Model Architectures

The paper uses federated neuron pruning to simplify architectures while preserving designed-task performance, then adjusts extreme weights to reduce backdoor success. It also discusses pruning attacks and the method’s evaluation scope.

B. Model Architectures

  • Pruning aims to remove unnecessary neurons while preserving designed-task performance, balancing the bias risk of simple architectures against greater backdoor vulnerability in complex ones.The authors say an unseen dataset makes it nearly impossible to design an architecture at exactly the minimum required complexity.Source: B. Model Architectures · pp. 10–11

C. Possible Attacks

  • The discussed pruning attacks had little effect on defense in the tested setting, where attackers were a 10% minority and the attacker’s access to the final pruning mask was described as nearly impossible in real cases.The authors also report that pruning extreme values further mitigated pruning-aware attacks.Source: C. Possible Attacks · p. 11

VII. CONCLUSION

  • Adjusting extreme weights in the simplified model degrades backdoor success; the method was evaluated across attack targets, patterns, client data distributions, and datasets.The authors report that federated fine-tuning after pruning can improve designed-task performance without impairing defense after the final extreme-weight adjustment.Source: §VII · p. 11
Loading 2011.01767v2…