PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepSight: Mitigating Backdoor Attacks in Federated Learning Through Deep Model Inspection

Phillip Rieger, Thien Duc Nguyen, Markus Miettinen, Ahmad-Reza Sadeghi

arXiv:2201.00763v1cs.CRcs.LG

TL;DR

Backdoor attacks threaten Federated Learning, while existing defenses can exclude benign models with deviating data distributions and degrade performance for those clients. DeepSight deeply inspects model structure, outputs, and training-data signals, combines filtering with clustering and clipping, and mitigates state-of-the-art attacks with negligible impact on benign-data performance.

  • Problem

    Existing Federated Learning backdoor defenses can filter benign models with deviating data distributions, degrading the aggregated model for their clients.

  • Method

    DeepSight uses deep model inspection, including DDifs, NEUPs, Threshold Exceedings, voting-based filtering, clustering, and clipping, to identify and mitigate poisoned updates.

  • Results

    DeepSight mitigates state-of-the-art backdoor attacks without affecting the resulting model's performance on benign data.

  • Takeaways & Limitations

    DeepSight preserves benign model updates, including those from clients with deviating data distributions, while excluding poisoned model clusters with high attack impact.

  • Takeaways & Limitations

    Individual DeepSight components are weakened in some corner cases, including multiple injected backdoors, low learning rates, and low poison-data ratios.

Abstract

from arXiv · show

Federated Learning (FL) allows multiple clients to collaboratively train a Neural Network (NN) model on their private data without revealing the data. Recently, several targeted poisoning attacks against FL have been introduced. These attacks inject a backdoor into the resulting model that allows adversary-controlled inputs to be misclassified. Existing countermeasures against backdoor attacks are inefficient and often merely aim to exclude deviating models from the aggregation. However, this approach also removes benign models of clients with deviating data distributions, causing the aggregated model to perform poorly for such clients. To address this problem, we propose DeepSight, a novel model filtering approach for mitigating backdoor attacks. It is based on three novel techniques that allow to characterize the distribution of data used to train model updates and seek to measure fine-grained differences in the internal structure and outputs of NNs. Using these techniques, DeepSight can identify suspicious model updates. We also develop a scheme that can accurately cluster model updates. Combining the results of both components, DeepSight is able to identify and eliminate model clusters containing poisoned models with high attack impact. We also show that the backdoor contributions of possibly undetected poisoned models can be effectively mitigated with existing weight clipping-based defenses. We evaluate the performance and effectiveness of DeepSight and show that it can mitigate state-of-the-art backdoor attacks with a negligible impact on the model's performance on benign data.

I. INTRODUCTION

Federated learning preserves client-data privacy but exposes aggregation to targeted poisoning attacks. DeepSight addresses the resulting defense trade-off by inspecting model updates rather than simply excluding all deviations.

  • Motivation: Federated Learning trains a shared neural network through private local updates sent to a central aggregator.Clients train locally, submit only model updates, and receive the aggregated global model.
  • Threat: Compromised clients can inject backdoors that make adversary-controlled inputs receive targeted labels while preserving the model’s main-task performance.The attacks manipulate local models or training data, depending on the adversary’s capabilities.
  • Problem: Existing outlier-detection defenses may wrongly filter benign models trained on deviating data distributions, degrading performance for those clients.The defense problem is distinguishing poisoned updates from benign non-IID updates.
  • Approach: DeepSight combines deep model inspection, classifier-based filtering, similarity-aware clustering, and clipping to identify malicious updates while retaining benign deviations.Its techniques analyze model structure, predictions, neurons, and training-data characteristics.
  • Evaluation: The paper evaluates DeepSight against several state-of-the-art backdoor attacks and reports no resulting-model performance impact on benign data.It also demonstrates an attack against a purportedly secure defense and reports that DeepSight mitigates such attacks.

2) Model Poisoning:

Model poisoning lets an adversary directly manipulate client model updates, including their scale and training objective. The paper studies a constrain-and-scale strategy that combines scaling with anomaly evasion.

  • Model Poisoning: A compromised client can arbitrarily modify its model update and scale it to increase its influence during aggregation.The scaling factor can reach N/N_A, while constraining the update norm can make the attack less suspicious.
  • Anomaly Evasion: Anomaly evasion reduces detectability by lowering the learning rate and adding a loss term measuring similarity to the global model.The attack balances classification performance against similarity to the original model.
  • Attack Strategy: The evaluated threat model combines anomaly evasion and scaling as the constrain-and-scale attack.The paper uses this combined strategy for its strong adversary model.
  • Adversarial Trade-off: Well-trained poisoned models are easier to detect because they differ substantially from benign models, whereas weakly trained models may have lower attack impact.The adversary can choose between stronger but more conspicuous models and weaker but less impactful ones.

III. SYSTEM AND PROBLEM SETTING

The system assumes private, heterogeneous client data and an aggregator that receives only model updates. The defense must mitigate backdoors without disrupting main-task performance or excluding benign non-IID clients.

  • System Setting: Clients keep their data secret, and the aggregator combines locally trained updates even though client data distributions may differ.The system uses FedAvg and does not rely on server-side training or testing data.
  • Adversary Model: The adversary seeks a target label C_A on trigger-set inputs while preserving predictions on non-trigger inputs.The formal setting distinguishes the trigger set I from the broader input domain D.
  • Adversary Model: The threat model allows the adversary to fully control fewer than half of the clients and to know the server’s defense without tampering with it.The adversary lacks control over benign clients and their data or updates.
  • Defense Requirements: An effective defense must preserve poisoning mitigation and main-task accuracy while avoiding disruption of the training process.Backdoor accuracy and main-task accuracy are required to remain at their no-attack or no-defense levels.
  • Design Challenges: Existing black-box metrics motivate techniques that distinguish poisoned models from benign models trained on different data without requiring exact data knowledge.The stated challenges include entangling detection with backdoor performance and maintaining high precision.
  • Proposed Techniques: DeepSight analyzes model updates using their predictions, individual neurons, and estimated training-data homogeneity before clustering them.The proposed techniques make groups of clients with similar training data visible from submitted models.

E. Our Defense Approach

DeepSight inspects model structure, predictions, neurons, and update-derived training-data signals to distinguish malicious updates from benign updates with differing data distributions. It combines these measurements with clustering and clipping to mitigate backdoor impact.

  • DeepSight measures fine-grained differences in model structure and outputs to identify updates trained on data focused on a specific backdoor task.Its inspection includes predictions, individual neurons, and training-data homogeneity.
  • DeepSight combines deep inspection with clustering and weight clipping so suspicious high-impact models are filtered while potentially remaining weakly trained poisoned models are mitigated later.The filtering scheme uses similarity information to support classification rather than simply rejecting outliers.
  • Division Differences: Division Differences compare local-model prediction ratios with global-model predictions to identify clients whose training data produce similar model adaptations.Clients starting from the same global model and using similar data tend to adapt their parameters similarly.
  • NormalizEd UPdate energy: NormalizEd UPdate energy analyzes output-layer parameter updates to extract information about label distributions in the underlying training data.Higher update magnitudes for output neurons indicate more frequent corresponding classes.
  • NormalizEd UPdate energy: Normalizing output-neuron update energies makes frequency distributions across models comparable and makes the technique more robust against adversarial obfuscation.Similar normalized energies indicate similar proportions of training labels across clients.

C. Threshold Exceedings

Threshold Exceedings use normalized output-layer update energies to estimate training-data homogeneity and identify suspiciously homogeneous poisoned updates. The method defines a model-specific threshold and compares the resulting count against a classification boundary.

  • Poisoned models are expected to have less heterogeneous training data than benign models because backdoors focus training on a few attack samples.In the NLP example, the backdoor uses a few sentences whereas the benign task uses many different sentences.
  • NEUPs provide a rough estimate of client label distributions, enabling homogeneous training data to be detected through Threshold Exceedings.Experiments confirmed a strong correlation between NEUPs and label-distribution homogeneity.
  • Threshold Exceedings count how many normalized update energies exceed a threshold derived from the model’s maximal NEUP.The threshold is defined separately for each local model.
  • The threshold is set to 1% of the maximal NEUP, with the factor increased according to the number of output classes when necessary.This adjustment prevents all NEUPs from exceeding the threshold in scenarios with very few output labels.
  • In the NIDS scenario, benign updates had significantly more Threshold Exceedings than poisoned updates across 70 benign and 30 poisoned updates.A model was labeled poisoned when its count fell below half the median number of Threshold Exceedings.

V. MITIGATING BACKDOOR ATTACKS ON FL BY DEEP MODEL INSPECTION

DeepSight addresses the inability of outlier-based defenses to distinguish poisoned updates from benign updates arising from non-IID data. Its filtering, clipping, and aggregation layers jointly target both strongly and weakly trained backdoors.

  • Outlier-based defenses may wrongly reject benign clients with deviating data distributions because they cannot distinguish non-IID data from poisoned data.This can degrade the aggregated model’s performance for those clients.
  • The clipping layer enforces a maximal L2-norm on updates, and the aggregation layer applies FedAvg to the remaining clipped updates.Clipping targets poisoned updates that use high scaling factors to compensate for weakly trained backdoors.
  • The layered defense creates an adversarial dilemma: homogeneous data expose poisoned models to filtering, whereas heterogeneous data weaken backdoor impact for later mitigation.This relationship is presented as the rationale for combining filtering with clipping and aggregation.
  • Filtering Layer: DeepSight’s filtering layer uses a Threshold-Exceedings classifier and clustering to detect well-trained backdoors while considering labels of similar models.The clustering component is intended to reduce mislabelings.
  • Filtering Layer: The filtering layer computes DDifs, NEUPs, and pairwise cosine distances for clustering, while deriving Threshold Exceedings from NEUPs for model labeling.DDifs are calculated three times using differently seeded random inputs.
  • Filtering Layer: The clustering procedure averages cluster distances from DDifs, NEUPs, and cosine-based clusterings before applying HDBSCAN.The resulting clusters inform which model groups are accepted or rejected.

3) Clustering:

DeepSight clusters model updates using training-data and model-structure signals, then combines cluster similarity with suspiciousness labels to remove poisoned clusters while retaining benign diversity.

  • Clustering: DeepSight clusters updates using DDifs, NEUPs, and cosine distances to group models trained on similar, IID data.HDBSCAN dynamically determines the number of clusters for each feature type.
  • Clustering: The clustering procedure converts each feature-specific clustering into pairwise distances and averages the resulting distance matrices.This dynamically combines the different feature views.
  • Poisoned Cluster Identification: For each cluster, DeepSight computes the share of suspiciously labeled updates and removes the entire cluster when that share reaches τ = 1/3.Clusters remain when fewer than one-third of their models are labeled suspicious.
  • Poisoned Cluster Identification: DeepSight combines prediction and neuron-parameter inspection with clustering to distinguish poisoned models from benign models trained on deviating data.Its features include DDifs, NEUPs, cosine similarity, and Threshold Exceedings.
  • Aggregation: Remaining models are clipped and aggregated with FedAvg, while the final round can aggregate models clusterwise so clients receive models for their respective data groups.Dynamic clipping uses a boundary based on the median update norm, under the assumption that most clients are benign.

A. Experimental Setup

The evaluation covers NLP and IoT network-intrusion scenarios, using realistic client datasets and comparisons with existing defenses across backdoor and benign-training conditions.

  • Experimental scope: DeepSight is evaluated in three FL applications, including an NLP scenario and image-dataset experiments, with an IoT NIDS scenario also examined.The NLP setup enables direct comparison with Bagdasaryan et al., while the NIDS setup follows Nguyen et al.
  • Text Prediction: The NLP experiment uses Reddit posts, LSTM models, 5,000 training rounds, 100 clients per round, and 10 malicious clients injecting advertisements.Main Task Accuracy measures the accuracy of suggested words.
  • Network Intrusion Detection System: The NIDS model uses merged real-world IoT traffic datasets and two GRU layers with 128 units each plus a linear output layer.The datasets include benign traffic from homes and offices and Mirai-infected-device attack traffic.
  • Network Intrusion Detection System: The NIDS data are organized into 44 device-type groups, with 22 sufficiently represented types distributed across at least 15 simulated clients each.Examples include printers, smart lights, smart plugs, and sensors.
  • Metrics and comparisons: The evaluation compares BA, MA, precision, and NPV across DeepSight and state-of-the-art defenses on NIDS and NLP datasets.The table’s no-defense condition measures the constrain-and-scale attack under plain FedAvg.
  • Evaluation findings: DeepSight mitigates attacks in both scenarios, whereas other defenses fail under non-IID or partly IID data, and FLGuard reduces MA by excluding benign clients.DeepSight is reported as the only approach effective in both scenarios.

2) Evaluation of Individual Components:

Component evaluations show complementary strengths between filtering and clipping: filtering detects imbalanced-data backdoors, clipping handles complex or weak attacks, and their combination preserves benign training performance.

  • Individual components: Filtering detects backdoors with highly imbalanced training data, whereas clipping is effective against complex backdoors with high L2-norms.Clipping is not effective against simple backdoors, while filtering fails for complex backdoors.
  • Varying attack parameters: 0% BA: combining DeepSight’s filtering and clipping layers defeats the evaluated weak attacks across attack-parameter settings.The evaluation varied alpha, learning rate, epochs, attack start time, PMR, and PDR.
  • Varying attack parameters: At PDR 5%, DeepSight detected no poisoned updates; at PDRs 8% or 10%, it missed 6 of 25 poisoned models.These results concern weakly trained backdoors, while benign models were not misclassified.
  • Impact on benign training: Without attack, DeepSight slows training slightly but soon performs similarly to the baseline, while Krum stops at 60% Main Task Accuracy.The comparison concerns benign training with no attack deployed.
  • Clipping invariance: DeepSight’s NEUPs and Threshold Exceedings remain unchanged when model updates are scaled or clipped.Theorem 2 follows because Threshold Exceedings use NEUPs as their only input.
  • Assumptions: DeepSight’s dynamic classification and clipping boundaries use medians and rely on a majority of benign clients to remain within benign-value ranges.The paper states this prevents the adversary from manipulating those boundaries.

VIII. RELATED WORK

Existing FL backdoor defenses often detect anomalous updates or rely on validation, but these strategies struggle with non-IID benign data and stealthy attacks. DeepSight instead combines deep model inspection, clustering, and clipping to identify poisoned updates while preserving benign diversity.

  • Outlier-based defenses assume similar benign client data, so non-IID distributions can cause benign models to be wrongly excluded and degrade performance.
  • Validation-client defenses require trigger samples or attacks that visibly affect the main task, limiting their ability to detect stealthy backdoors.
  • Similarity-based weighting can harm benign clients with similar data and may rely on update history rather than the current training round.
  • DeepSight inspects model structure, predictions, and neurons to identify poisoned updates, clustering similar models and combining filtering with clipping.
  • Secure aggregation prevents the server from accessing individual updates, so a privacy-preserving version of DeepSight remains future work.

APPENDIX

The appendix examines how attack design, clustering, and model filtering affect DeepSight’s robustness. Across the reported settings, the defense mitigates attacks while preserving benign-model performance, including cases where poisoned models are not directly filtered.

  • Increasing backdoor complexity reduces attack impact, while filtering can remain effective because poisoned training data remain sufficiently homogeneous.
  • At least 20% poisoned data may be needed for high attack impact under clipping, but this concentration makes poisoned updates easier for filtering to detect.
  • Reducing update heterogeneity through lower poisoned data rates or more complex backdoors also makes attacks easier to mitigate with clipping.
  • DeepSight’s clustering avoids requiring a preset cluster count, unlike k-means, which can be exploited by submitting random distracting updates.
  • DeepSight mitigates all reported NIDS backdoor attacks even when some poisoned models are not filtered; in one case, the backdoor accuracy is 0%.
  • The appendix also evaluates standard CIFAR-10 and MNIST federated-learning benchmarks alongside device-specific NIDS settings.

E. Scenarios with a Single Source Label

DeepSight is evaluated in settings where clients may have single-label local datasets and against varied backdoor or noise-based attacks. Later defense layers can mitigate attacks even when the classifier does not detect malicious models without harming main-task accuracy.

  • Single-label client datasets can challenge DeepSight’s label-distribution classifier, motivating alternative classifiers such as binary or siamese networks.
  • In the single-label CIFAR-10 experiment, later defense layers mitigated the attack despite DeepSight failing to detect malicious models.
  • The dynamic threshold produced no false positives in that experiment, so the resulting model’s main-task accuracy was not negatively affected.
  • DeepSight effectively mitigated the evaluated NIDS and word-prediction backdoors across multiple attack patterns.
  • Noise-based attempts to distract DeepSight failed, with backdoor accuracy remaining 0% even at the highest tested noise standard deviation.
  • Scaling updates does not affect the cosine or NEUP measures used in the defense.

I. Impact of Threshold Factor

The Threshold Factor controls the Threshold Exceedings boundary and therefore changes the balance between detecting malicious models and falsely flagging benign ones. The reported experiment shows that threshold selection can shift both false-positive and true-positive rates.

  • The boundary ξ_t,k is based on the model’s highest NEUP and a threshold factor, capped by the inverse number of labels.
  • Lowering the Threshold Factor increases Threshold Exceedings, especially for poisoned models, which can increase the false-negative rate.
  • Increasing the Threshold Factor reduces Threshold Exceedings, particularly for benign models, increasing false positives before the classification boundary shifts.
  • For TF > 0.05, the shifted classification boundary first reduces FPR and then reduces TPR.

J. Overhead and Complexity of DeepSight

DeepSight combines multiple component-based defenses to remain effective across attack corner cases, while its overhead is reported for models of varying sizes and client counts.

  • Overhead and Complexity: DeepSight requires 1.15 minutes for IoT-Traffic, 1.06 minutes for CIFAR-10, and 6.02 minutes for Reddit in the reported filtering and aggregation settings.The corresponding configurations use (N=100, M=300k), (N=10, M=9M), and (N=100, M=20M), respectively.
  • Overhead and Complexity: Pairwise distance matrices grow with complexity O(n^2), contributing to DeepSight’s computational overhead as model or client complexity increases.The passage identifies the quadratic growth of pairwise distance matrices as part of the implementation cost.
  • Effectiveness: DeepSight always detects all poisoned models by combining individual techniques whose weaknesses compensate for one another across corner cases.Cosine and DDif clustering and the Threshold Exceedings classifier are each weakened under specific attack settings, but their combination maintains detection.
  • Effectiveness: The NLP outlier-detection defense based on DDifs is completely circumvented because the scenario is highly non-IID.This result illustrates a corner case where an individual outlier-detection component fails, whereas DeepSight combines multiple techniques.
Loading 2201.00763v1…