PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing Fine-Tuning Based Backdoor Defense with Sharpness-Aware Minimization

Mingli Zhu, Shaokui Wei, Li Shen, Yanbo Fan, Baoyuan Wu

arXiv:2304.11823v1cs.AI

TL;DR

With limited benign data, vanilla fine-tuning can perform poorly against advanced backdoor attacks. The paper studies fine-tuning at the neuron level and proposes combining it with sharpness-aware minimization; experiments report state-of-the-art defense performance while preserving high accuracy.

  • Problem

    With limited benign samples, vanilla fine-tuning can fail to defend against several advanced backdoor attacks.

    Source: §3.2 · p. 3

  • Method

    FT-SAM combines sharpness-aware minimization with fine-tuning to perturb backdoor-related neurons and reduce their weight norms.

    Source: §1 · p. 2, §5 · p. 8

  • Results

    Experiments report that FT-SAM reliably eliminates injected backdoors and offers the highest robustness against various cutting-edge attacks while preserving high accuracy.

    Source: §5 · p. 8

  • Takeaways & Limitations

    FT-SAM can be integrated with other defense methods, including to make up for accuracy drop.

    Source: §1 · p. 2, §5 · p. 8

    The defender setting assumes access to only a few benign samples for fine-tuning.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Backdoor defense, which aims to detect or mitigate the effect of malicious triggers introduced by attackers, is becoming increasingly critical for machine learning security and integrity. Fine-tuning based on benign data is a natural defense to erase the backdoor effect in a backdoored model. However, recent studies show that, given limited benign data, vanilla fine-tuning has poor defense performance. In this work, we provide a deep study of fine-tuning the backdoored model from the neuron perspective and find that backdoorrelated neurons fail to escape the local minimum in the fine-tuning process. Inspired by observing that the backdoorrelated neurons often have larger norms, we propose FTSAM, a novel backdoor defense paradigm that aims to shrink the norms of backdoor-related neurons by incorporating sharpness-aware minimization with fine-tuning. We demonstrate the effectiveness of our method on several benchmark datasets and network architectures, where it achieves state-of-the-art defense performance. Overall, our work provides a promising avenue for improving the robustness of machine learning models against backdoor attacks.

1. Introduction

Fine-tuning is a natural backdoor defense, but limited benign data can leave the backdoor effect largely intact. The paper motivates FT-SAM by neuron weight norms and reports strong benchmark performance.

1. Introduction

  • Fine-tuning is more general than pruning and can moderately restore model utility, while complex unlearning and pruning techniques may cost accuracy.Pruning effectiveness also depends on network structure.Source: §1 · p. 1
  • With limited benign data, vanilla fine-tuning may change weights only slightly because the backdoored model already fits benign samples well.The paper identifies this as a possible reason that fine-tuning fails against strong attacks such as Blended and LF.Source: §1 · p. 1
  • The authors observe a positive correlation between neuron weight norms and backdoor-related neurons, motivating adaptive SAM perturbations during fine-tuning.They propose that larger-norm neurons receive greater perturbations, helping alter backdoor-related weights.Source: §1 · p. 2
  • FT-SAM achieves state-of-the-art defense performance and can boost existing fine-tuning-based defenses.The evaluation covers three benchmark datasets, two networks, and seven state-of-the-art defenses.Source: §1 · p. 2

2. Related work

Related work spans data-poisoning and controllable backdoor attacks, post-training defenses, and methods for seeking flatter loss minima. The paper positions its minimax fine-tuning approach as a post-processing defense that does not alter network architecture.

2. Related work

  • Data-poisoning attacks include BadNets, Blended [6], and more concealed triggers such as LF [45], WaNet [30], and Input-aware [29].Training-controllable attacks let attackers design triggers or choose images to attack [30] [22], while clean-label attacks link triggers to target labels [32, 2].Source: §2 · p. 2
  • Post-training defenses use a small benign sample set and include fine-tuning-based methods such as NC, NAD [24], and i-BAU [44], alongside pruning-based methods FP [25] and ANP [41].The cited methods use trigger optimization, teacher guidance, implicit unlearning, or neuron pruning and masking.Source: §2 · p. 2
  • SAM minimizes loss and loss sharpness to seek flatter minima [9], with variants including ASAM [19], GSAM [49], and PGN [43].Earlier work linked flat, wide minima to generalization, and entropy-SGD searches for wider minima.Source: §2 · pp. 2–3
  • Unlike existing post-processing defenses, the approach uses a minimax formulation for fine-tuning without modifying network architecture and aims to preserve model utility.The paper also examines SAM for backdoor-defense fine-tuning.Source: §2 · p. 3

3. Methodology

The methodology formulates a defender’s goal under limited benign data, investigates why vanilla fine-tuning can leave backdoors intact, and develops adaptive SAM perturbations for fine-tuning. Its analysis links small FT updates to persistent backdoor behavior and targets larger-norm neurons for stronger perturbation.

3.1. Problem Formulation

  • The threat model has an attacker poison training data so triggered inputs receive target labels while clean inputs are classified normally; the defender has a backdoored model and a few benign samples.The defender seeks to maintain benign-data performance while reducing the rate of poisoned samples classified as the target label.Source: §3.1 · p. 3

3.2. Investigating the Vanilla Fine-Tuning

  • Vanilla fine-tuning can struggle because the backdoored model already fits the benign samples, leaving limited data insufficient to move it far from its current solution.The paper frames this as a hypothesis for why the backdoor effect remains under fine-tuning.Source: §3.2 · p. 3
  • With 5% benign data, 100 epochs, and Blended poisoning on CIFAR-10 with PreAct-ResNet18, FT changed benign accuracy from 93.44% to 92.48% and ASR from 97.71% to 82.22%.The experiment used a 10% poisoning ratio.Source: §3.2 · p. 3
  • After FT, poisoned test features remained clustered and most last-convolution-layer neuron norms changed only slightly, consistent with limited movement from the backdoored model.Figure 2 compares FT and FT-SAM feature visualizations, neuron-norm changes, and gradient norms.Source: §3.2 · p. 3, Figure 2 · p. 3
  • FT gradients were small and showed no significant difference between backdoor-related and other neurons, indicating similar weight changes across the two groups.The analysis sorts 512 neurons by TAC, a measure of correlation between neuron activation differences on benign and corresponding poisoned samples.Source: §3.2 · p. 4

3.3. Proposed Method

  • FT-SAM uses a minimax objective with an adaptive perturbation budget that encourages larger perturbations for larger-magnitude weights, which are more likely to be backdoor-related.The adaptive scaling is defined by T_w = diag(|w_1|, |w_2|, ..., |w_d|).Source: §3.3 · p. 4
  • The optimization alternates perturbation updates with weight updates, then minimizes benign-data loss using a gradient evaluated at the perturbed weights.The paper describes the perturbation update as an inner maximization and the weight update as an outer minimization.Source: §3.3 · p. 4

4. Experiment

Across CIFAR-10 and Tiny ImageNet, FT-SAM performs robustly against the evaluated attacks, while analyses examine its sensitivity, compatibility with existing defenses, and effects on neuron norms and attention. Experiments use limited benign data and compare FT-SAM with existing defenses and configurations.

4.2. Experimental Results

  • FT-SAM achieves high DER in most CIFAR-10 attacks, decreases ASR by an average of 2.47%, and shows robust Tiny ImageNet results with a slight ACC drop and high DER.On Tiny ImageNet, the compared defenses fail to maintain both ACC and ASR on complex attacks; FT-SAM performs robustly against all attacks.Source: Table 1 · p. 5, Table 2 · p. 5, §4.2 · p. 6

4.1. Experimental Setup

  • Experiments evaluate 10 attacks across CIFAR-10, Tiny ImageNet, and GTSRB, using PreAct-ResNet18 and VGG19-BN except for two clean-label attacks.The defense comparisons include seven state-of-the-art methods; most methods use 5% benign training data, while AC and ABL use the entire poisoned dataset.Source: §4.1 · p. 5
  • The evaluation reports benign accuracy (ACC), attack success rate (ASR), and defense effectiveness rating (DER), which combines ACC and ASR changes.A superior defense has lower ASR, higher ACC, and higher DER.Source: §4.1 · pp. 5–6

4.3. Ablation Studies

  • With different benign-sample ratios, FT-SAM maintains robust defense and consistently high DER, with only a modest performance decrease at 1% benign data.FT has attack-dependent trends and poor DER at low benign-sample ratios.Source: Table 3 · p. 6, §4.3 · pp. 6–7
  • FT-SAM’s performance is not very sensitive to ρ across the tested range, although smaller values may leave backdoors from complex attacks incompletely removed.The passage reports that larger ρ increases weight perturbation, while smaller ρ can help maintain performance with limited training data.Source: §4.3 · p. 6, Figure 3 · p. 7

4.4. Further Analysis

  • Replacing fine-tuning with FT-SAM in FP and ANP improves both defense strategies on average, with high DER.The experiments compare the combinations with the original defenses, pure pruning, and ANP plus fine-tuning.Source: Table 4 · p. 7, §4.4 · p. 7
  • Under the evaluated attacks, defended models’ Grad-CAM maps focus on the deer’s head rather than the triggers.The authors interpret this as evidence that the backdoor was eliminated.Source: §4.4 · pp. 7–8
  • FT-SAM reduces overall neuron weight norms, with larger changes among higher-norm neurons, whereas FT produces only minor changes.The analysis reports a strong correlation between neuron weight norm and TAC; Figure 5 shows norms depressed more uniformly across neurons.Source: §4.4 · p. 7, Figure 4 · p. 8, Figure 5 · p. 8

5. Conclusion

FT-SAM applies sharpness-aware minimization to perturb backdoor-related neurons and reduce their weight norms. The paper reports strong robustness across attacks and high accuracy, and illustrates model attention with Grad-CAM.

5. Conclusion

  • The authors find that larger-norm neurons contribute more to backdoor attacks and propose FT-SAM to perturb backdoor-related neurons and reduce their norms.The conclusion describes its analysis of fine-tuning from a neuron-level perspective.Source: §5 · p. 8
  • Figure 6 compares Grad-CAM visualizations for FT-SAM and backdoored models across attacks on CIFAR-10 at a 5% poisoning ratio using PreAct-ResNet18.Source: Figure 6 · p. 8
  • FT-SAM reliably eliminates injected backdoors, offers the highest robustness against various cutting-edge attacks while preserving high accuracy, and is promising alongside other defenses.The conclusion also investigates gradients of neuron weights computed by FT-SAM.Source: §5 · p. 8

A. More Algorithmic Details on The Proposed Method

The algorithm alternates between finding a loss-increasing weight perturbation and updating model weights. Its objective uses benign-data cross-entropy, a perturbation budget, and a diagonal scaling matrix.

Optimization

B. More Implementation Details

The experiments use three benchmark datasets, two network architectures, and multiple backdoor attack settings. The implementation details specify comparison conditions and FT-SAM's perturbation and training parameters.

Datasets

  • CIFAR-10, Tiny ImageNet, and GTSRB are evaluated, with dataset sizes, class counts, and image dimensions specified.CIFAR-10 has 60,000 images and 10 classes; Tiny ImageNet has 100,000 training and 10,000 testing samples across 200 classes; GTSRB has 43 classes.Source: B. More Implementation Details · p. 11

Models

  • PreAct-ResNet18 and VGG19-BN are tested; comparisons use 10% poisoning and 5% clean defense samples, with additional CIFAR-10 tests at 5% and 1% poisoning.Source: B. More Implementation Details · p. 11

Attack Details

Defense Details

  • FT-SAM's perturbation radius ρ varies by dataset and network; experiments use SGD with momentum 0.9 and weight decay 1e−4.The radius is 2 for CIFAR-10, 8 for Tiny ImageNet and GTSRB on PreAct-ResNet18, and 6 for all datasets on VGG19-BN; without adaptive T, it is 0.5.Source: B. More Implementation Details · p. 11

C. Defense Results in Comparison to SOTA Defenses

Across the reported settings, FT-SAM is the only defense described as balancing clean accuracy and attack success rate, and it ranks first in average DER and ASR.

C. Defense Results in Comparison to SOTA Defenses

D. Defense Results under Different Poisoning Ratios

FT-SAM is evaluated at 5% and 1% poisoning ratios; its advantage over other defenses is especially pronounced at 1%.

D. Defense Results under Different Poisoning Ratios

E. Ablation Study of The Effectiveness of Tw

The ablations compare adaptive perturbation with an identity constraint and FT-SAM with direct l2 weight regularization, finding advantages for the adaptive method across attacks.

E. Ablation Study of The Effectiveness of Tw

F. Visualization Analysis

Visualizations compare backdoored and defended models’ attention and feature distributions, with the reported results showing attention shifting toward image subjects and poisoned features dispersing among normal features.

F. Visualization Analysis

Related papers

Loading 2304.11823v1…