PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Adversarial Robustness via Channel-wise Activation Suppressing

Yang Bai, Yuyuan Zeng, Yong Jiang, Shu-Tao Xia, Xingjun Ma, Yisen Wang

arXiv:2103.08307v2cs.LG

TL;DR

Adversarial examples differ from natural examples in both channel-wise activation magnitude and activation frequency, while existing adversarial training addresses only the magnitude issue. The paper introduces CAS to suppress redundant channel activation and reports consistent robustness improvements when combining CAS with existing adversarial defenses.

  • Problem

    Existing adversarial training reduces excessive adversarial activation magnitude, but uniform channel activation and its connection to robustness remain insufficiently addressed.

  • Method

    CAS uses an auxiliary classifier to learn channel importance and dynamically suppress redundant channel activation during adversarial training.

  • Results

    CAS consistently improves the robustness of existing state-of-the-art adversarial-training methods.

  • Takeaways & Limitations

    CAS is a generic intermediate-layer robustification strategy that can be combined with multiple existing defense methods.

  • Takeaways & Limitations

    CAS suppresses channel activation according to labels during training or predictions during testing, raising robustness concerns about the module's behavior.

Abstract

from arXiv · show

The study of adversarial examples and their activation has attracted significant attention for secure and robust learning with deep neural networks (DNNs). Different from existing works, in this paper, we highlight two new characteristics of adversarial examples from the channel-wise activation perspective: 1) the activation magnitudes of adversarial examples are higher than that of natural examples; and 2) the channels are activated more uniformly by adversarial examples than natural examples. We find that the state-of-the-art defense adversarial training has addressed the first issue of high activation magnitudes via training on adversarial examples, while the second issue of uniform activation remains. This motivates us to suppress redundant activation from being activated by adversarial perturbations via a Channel-wise Activation Suppressing (CAS) strategy. We show that CAS can train a model that inherently suppresses adversarial activation, and can be easily applied to existing defense methods to further improve their robustness. Our work provides a simple but generic training strategy for robustifying the intermediate layer activation of DNNs.

1 INTRODUCTION

The paper examines adversarial robustness through intermediate channel-wise activation, identifying high activation magnitude and unusually uniform channel activation as two relevant characteristics. It proposes Channel-wise Activation Suppressing (CAS) to adaptively reduce redundant channel activation and improve existing adversarial training methods.

  • Intermediate-layer activation is important for understanding how input perturbations accumulate across layers to subvert network outputs and how adversarial training mitigates this accumulation.
  • The paper analyzes activation channel-wise because different convolutional filters learn distinct patterns whose combination represents object types.
  • CAS adaptively learns channel importance with an auxiliary classifier and dynamically adjusts channels during training.
  • Applying CAS to existing state-of-the-art adversarial training methods consistently improves their robustness.
  • Adversarial examples exhibit higher activation magnitudes and more uniform channel activation than natural examples, while adversarial training addresses only the magnitude difference.

2 RELATED WORK

Prior adversarial-defense research includes input, model, gradient, and training-based approaches, with adversarial training providing the most reliable robustness. Related work also studies adversarial robustness through architecture and intermediate activation, but the paper positions channel-wise activation as an additional perspective.

  • Adversarial training has demonstrated the most reliable robustness among established adversarial-defense techniques.
  • Standard adversarial training optimizes model parameters against worst-case perturbations generated within an Lp-norm neighborhood.
  • Prior activation-focused studies examine adversarial subspace dimensionality and robustness certification under different activation functions.

3 CHANNEL-WISE ACTIVATION AND ADVERSARIAL ROBUSTNESS

The paper empirically studies channel-wise intermediate activation in standard and adversarially trained ResNet-18 and VGG16 models on CIFAR-10. It examines both activation magnitude and activation frequency to connect channel behavior with adversarial robustness.

  • ResNet-18 and VGG16 models trained on CIFAR-10 provide the basis for comparing channel-wise activation under standard and adversarial training.
  • Channel-wise Activation Magnitude: Adversarial examples generally produce higher channel-wise activation magnitudes than natural examples in standard models.
  • Channel-wise Activation Magnitude: Figure 1 compares averaged penultimate-layer activation magnitudes across 512 channels for natural and adversarial examples under standard and adversarial training.
  • Channel-wise Activation Frequency: Figure 2 compares channel activation frequencies for natural and PGD-20 adversarial examples in standard, adversarially trained, and CAS-based adversarially trained ResNet-18 models.
  • Channel-wise Activation Frequency: The analysis motivates distinguishing robust channels that produce generic class patterns from non-robust channels that respond differently to examples.

4 PROPOSED CHANNEL-WISE ACTIVATION SUPPRESSING

CAS dynamically learns channel importance for class prediction and uses it to reweight intermediate activations during training. It combines auxiliary channel classification with the network objective and can be integrated with multiple adversarial-training defenses.

  • 4.1 CAS MODULE: CAS uses global average pooling and an auxiliary classifier to learn channel importance, then multiplies that importance into the original activation before forwarding it onward.The adjusted activation is used for subsequent layers, while the network and auxiliary classifier are trained simultaneously.
  • 4.1 CAS MODULE: The auxiliary classifier selects class-specific channel weights using the ground-truth label during training and the predicted class during testing.This provides the channel importance vector used for activation reweighting in the two phases.
  • 4.2 MODEL TRAINING: CAS modules can be inserted into multiple intermediate layers and trained as auxiliary components with standard training or different adversarial-training methods.The paper illustrates the approach with original adversarial training and extends it to multiple CAS modules.
  • 4.2 MODEL TRAINING: The combined objective adds the CAS auxiliary classification loss to the network’s cross-entropy loss, with β controlling the CAS strength.The strategy can also be combined with TRADES and MART, and the paper summarizes these combined loss functions.

5 EXPERIMENTS

Experiments evaluate CAS as a channel-suppressing strategy across activation analyses, ablations, layers, attacks, datasets, and existing adversarial-training defenses. The results support explicit suppression, especially at deeper layers, and show consistent robustness gains when CAS is added to baseline methods.

  • Experimental Settings: The experimental protocol trains ResNet-18 with PGD-10 adversarial training and evaluates robustness using FGSM, PGD-20, and CW∞ attacks.The main CIFAR-10 setup uses 200 epochs, momentum 0.9, weight decay 2e-4, and ϵ = 8/255.
  • Empirical Understanding of CAS: CAS reduces activation-frequency gaps between natural and adversarial examples and remains robust against Avg-PGD and EOT attacks.Unlike kWTA and SAP, CAS does not rely on randomness; PCL does not directly enforce channel suppression.
  • Empirical Understanding of CAS: Removing explicit channel suppressing prevents CAS from improving robustness, showing that suppression is essential beyond the auxiliary classifier.Table 2 compares CAS with and without the channel-suppressing operation at Block4 of ResNet-18.
  • Empirical Understanding of CAS: Applying CAS at Block4 yields the largest robustness improvement, while Block3 or Block3+4 also improve robustness less substantially.The authors associate deeper-layer benefits with stronger correlation to class prediction and note that shallow layers may have less accurate channel-importance estimates.
  • Empirical Understanding of CAS: CAS modules remain robust under attacks and can provide higher natural accuracy and adversarial robustness when evaluated separately.For CAS models, attacks target the CAS module using its CAS loss; baseline defenses are attacked at the final layer.
  • Robustness Evaluation: CAS improves both natural accuracy and robustness across AT, TRADES, and MART, with especially larger gains against CW∞ attacks.The evaluation covers CIFAR-10 and SVHN using adaptive white-box attacks; the reported results are from the last checkpoint.

6 CONCLUSION

The paper identifies higher magnitude and more uniform channel activation as adversarial characteristics, then introduces CAS to suppress redundant channels during training. Combined with adversarial training, CAS improves robustness and can be applied across defense methods and intermediate layers.

  • 6 CONCLUSION: CAS dynamically learns channel importance and suppresses redundant channel activation during training to improve adversarial robustness.The strategy uses an auxiliary classifier and can be plugged into existing defense methods.
  • 6 CONCLUSION: Adversarial examples exhibit higher activation magnitudes and more uniform activation frequencies than natural examples.Standard adversarial training addresses the magnitude difference but not the uniform activation-frequency difference.
  • 6 CONCLUSION: CAS can be combined with adversarial training to produce models that inherently suppress redundant channels activated by adversarial examples.The paper describes CAS as a simple, generic strategy for robustifying intermediate DNN layers.

A ALGORITHM OF CAS TRAINING

CAS training combines adversarial training with a channel-wise suppression objective implemented through auxiliary classifiers and adjustable channel weights. The algorithm generates adversarial examples, computes CAS and classification losses, and jointly optimizes network and CAS parameters.

  • The algorithm generates adversarial examples with PGD and computes the CAS loss from channel-wise activations.
  • The adjusted intermediate activation is forwarded to the output layer to compute cross-entropy loss.
  • Network and CAS-module parameters are jointly optimized by gradient descent over minibatches and training epochs.
  • CAS-enhanced training combines the original adversarial loss with a CAS loss for AT, TRADES, and MART variants.
  • Each CAS module uses an auxiliary classifier to produce channel weights that adjust intermediate activations dynamically.

C CHANNEL-WISE ACTIVATION SUPPRESSING ON MORE DATASETS AND DEFENSE MODELS

CAS consistently suppresses redundant channel activation across TRADES and MART, narrowing the activation-frequency gap between natural and adversarial examples. The effect is also examined on SVHN and ImageNet and under varying white-box attacks.

  • CAS suppresses redundant channels in both TRADES and MART, making natural and adversarial activation frequencies more similar.The resulting channel-wise activation is described as more class-correlated.
  • The study extends channel-activation comparisons to SVHN with PGD-20 examples and ImageNet with PGD-30 examples.SVHN uses ResNet-18, while ImageNet uses ResNet-152 and 5,900 dog validation images.
  • Figure 5 shows that CAS narrows the frequency-distribution gap between natural and adversarial examples and suppresses channels #200–#512.
  • For AT+CAS, increasing β improves white-box robustness against FGSM, PGD-20, and CW∞, especially against CW∞.

D SENSITIVITY OF CAS TO PARAMETER β

The CAS loss coefficient β controls suppressing strength, and larger values improve AT+CAS robustness across the tested white-box attacks.

  • Larger β values improve AT+CAS robustness against FGSM, PGD-20, and CW∞, with the strongest improvement against CW∞.β = 0 corresponds to standard adversarial training.

E CAS IMPROVES REPRESENTATION LEARNING

CAS improves penultimate-layer representations under both natural and adversarial training. The paper attributes this improvement largely to CAS's channel-suppressing capability.

  • CAS improves representation learning for both natural training and adversarial training on CIFAR-10.The comparison uses t-SNE embeddings of penultimate-layer ResNet-18 features.

F.1 WIDERESNET RESULTS ON CIFAR-10

WideResNet-34-10 experiments evaluate CAS-augmented adversarial training on CIFAR-10, while feature embeddings assess class separation and compactness.

  • F.1 WIDERESNET RESULTS ON CIFAR-10: WideResNet-34-10 models are evaluated for white-box robustness on CIFAR-10 with AT, AT+CAS, TRADES, TRADES+CAS, MART, and MART+CAS.The table reports accuracy under various white-box attacks, with best results boldfaced.
  • F.1 WIDERESNET RESULTS ON CIFAR-10: The reported comparison includes existing adversarial-training methods both with and without the CAS strategy.The supplied passages identify the compared methods and evaluation scope but do not provide the table’s cell values.
  • F.1 WIDERESNET RESULTS ON CIFAR-10: CAS training is reported to improve inter-class separation and intra-class compactness for natural and adversarial examples in t-SNE feature embeddings.The embeddings come from the penultimate layer of ResNet-18 models trained with natural or adversarial training on CIFAR-10.

F.2 VGG16 RESULTS ON CIFAR-10

On CIFAR-10, CAS improves VGG16 robustness and natural accuracy, and supplementary evaluations report benefits across checkpoints and stronger attack assessments.

  • F.2 VGG16 RESULTS ON CIFAR-10: 7%-10%: CAS enhances the robustness of VGG16 ‘best’ models on CIFAR-10, while also improving natural-training accuracy.The passage reports that CAS is especially beneficial for the smaller-capacity VGG16 model compared with WideResNet.
  • F.2 VGG16 RESULTS ON CIFAR-10: CAS consistently improves robustness and natural accuracy for best-checkpoint ResNet-18 models on CIFAR-10 and SVHN.These results supplement last-checkpoint results and are described as holding throughout the training process.
  • F.2 VGG16 RESULTS ON CIFAR-10: CAS still noticeably improves AT, TRADES, and MART under AutoAttack, although the gains are smaller than under regular PGD and CW attacks.The evaluation is presented as evidence that the gains are substantial rather than caused by obfuscated gradients or improper evaluation.
  • F.2 VGG16 RESULTS ON CIFAR-10: The CAS module is tested against Margin Decomposition attacks because channel suppression may create imbalanced margins and gradients.The supplied table caption identifies the experiment as ResNet-18 on CIFAR-10 but provides no cell values.
  • F.2 VGG16 RESULTS ON CIFAR-10: Across thresholds of 0.5%, 1%, and 5%, activation-frequency distributions vary with noise, but their general patterns remain unchanged.Higher thresholds truncate more low-count activations in the channel-frequency analysis.
Loading 2103.08307v2…