PaperlayerView exact revision on arXiv

Source-linked AI summary

Defensive Quantization: When Efficiency Meets Robustness

Ji Lin, Chuang Gan, Song Han

arXiv:1904.08444v1cs.LGcs.CVstat.ML

TL;DR

Conventional quantization improves deployment efficiency but can make neural networks more vulnerable to adversarial attacks. The paper introduces Defensive Quantization (DQ), which combines low-bit quantization with Lipschitz control to suppress error amplification. Experiments report improved robustness over full-precision counterparts while retaining quantization efficiency, along with clean-image accuracy gains.

  • Problem

    Conventional quantized models can be more vulnerable to adversarial attacks despite similar clean-image accuracy, creating a security concern for efficiently deployed networks.

  • Method

    Defensive Quantization combines low-bit feature-map quantization with network Lipschitz-constant control to keep adversarial noise non-expansive during inference.

  • Results

    DQ quantized models achieve better adversarial robustness than full-precision counterparts while maintaining the efficiency of vanilla quantization, and can improve clean-image accuracy.

  • Takeaways & Limitations

    DQ provides a quantization-based defense that preserves efficient deployment while addressing the robustness drop associated with conventional quantization.

Abstract

from arXiv · show

Neural network quantization is becoming an industry standard to efficiently deploy deep learning models on hardware platforms, such as CPU, GPU, TPU, and FPGAs. However, we observe that the conventional quantization approaches are vulnerable to adversarial attacks. This paper aims to raise people's awareness about the security of the quantized models, and we designed a novel quantization methodology to jointly optimize the efficiency and robustness of deep learning models. We first conduct an empirical study to show that vanilla quantization suffers more from adversarial attacks. We observe that the inferior robustness comes from the error amplification effect, where the quantization operation further enlarges the distance caused by amplified noise. Then we propose a novel Defensive Quantization (DQ) method by controlling the Lipschitz constant of the network during quantization, such that the magnitude of the adversarial noise remains non-expansive during inference. Extensive experiments on CIFAR-10 and SVHN datasets demonstrate that our new quantization method can defend neural networks against adversarial examples, and even achieves superior robustness than their full-precision counterparts while maintaining the same hardware efficiency as vanilla quantization approaches. As a by-product, DQ can also improve the accuracy of quantized models without adversarial attack.

1 INTRODUCTION

The paper identifies a security problem in efficient quantized networks: conventional quantization can worsen adversarial vulnerability despite preserving clean accuracy. It proposes Defensive Quantization (DQ), which combines low-bit representations with Lipschitz control to improve robustness while retaining efficiency.

  • Motivation: Quantization reduces neural-network computation and memory costs, but conventional quantized models can be more vulnerable to adversarial attacks.This matters because quantized networks are widely deployed, including in safety-critical applications where efficiency and latency remain important.
  • Analysis: The paper attributes inferior robustness to error amplification, in which adversarial perturbations become large enough during inference for quantization to enlarge their distance.This explains why quantization may denoise small perturbations initially yet worsen amplified perturbations later.
  • Motivation: With at least 5 bits, quantized models show no clean-image accuracy loss but significant adversarial-accuracy loss relative to full-precision models.Figure 1 evaluates VGG-16 and Wide ResNet on CIFAR-10 using FGSM.
  • Proposed method: Defensive Quantization (DQ) jointly quantizes feature maps and controls the network Lipschitz constant so adversarial noise remains small during inference.The method is motivated by non-expansive networks, whose outputs vary no more than their inputs.
  • Reported advantages: Experiments report that DQ boosts robustness while maintaining efficiency, can combine with other defenses, and makes quantization easier through a constrained dynamic range.These advantages are presented as outcomes of the experimental study.

2 BACKGROUND AND RELATED WORK

This section reviews efficient low-bit inference, adversarial attacks, and defenses, while motivating DQ as a deterministic bridge between quantization and robustness. It describes attacks ranging from random perturbations and FGSM to iterative PGD, alongside preprocessing and adversarial-training defenses.

  • Model quantization: Low-bit quantization enables efficient inference by reducing computation and memory while causing little clean-image accuracy degradation.The section frames quantization as widely adopted for efficient deployment.
  • Related work: Prior work includes randomized or input-space quantization defenses, but the paper distinguishes its deployment-oriented DQ approach from stochastic quantization and gradient-masking concerns.The related-work discussion notes that randomized quantization requires extra hardware random-number generation and can create stochastic gradients.
  • Adversarial attacks: Adversarial attacks seek a small perturbation ∆ constrained by ||∆|| ≤ ϵ that changes the network output from X to X_adv = X + ∆.The experiments study attacks under the infinity norm, where each image element changes by at most ϵ.
  • Attack methods: FGSM follows the loss-gradient direction, R+FGSM prepends a random step, and iterative BIM or PGD applies repeated updates with a small step size.The paper uses PGD in experiments as a strong first-order attack and also evaluates other described attacks.
  • Defense methods: Defense methods either preprocess adversarial samples or make the network robust, including feature squeezing and adversarial training.The paper reports that these defenses can be combined with DQ to further improve robustness.

3 CONVENTIONAL NN QUANTIZATION IS NOT ROBUST

Conventional quantization can preserve clean-image accuracy while substantially worsening adversarial robustness. This vulnerability is linked to error amplification: activation quantization may enlarge perturbation distances once adversarial noise grows.

  • Empirical vulnerability: 25.3% and 9.2% accuracy drops occurred under attack with 5-bit quantization, despite limited clean-image degradation.The experiments used VGG-16 and Wide ResNet models on CIFAR-10 with FGSM perturbations.
  • Empirical vulnerability: Quantized models can be easier to fool than full-precision models even when clean-image accuracy is unchanged.This makes clean-image accuracy alone insufficient for evaluating quantized-model security.
  • Error amplification: Input image quantization can defend adversarial examples, whereas applying quantization to hidden layers can instead worsen robustness.The study varied adversarial strength from ϵ = 1 to 8 and examined activations after the first convolutional layer.
  • Error amplification: Error amplification increases relative perturbation distances as adversarial samples propagate through the network, explaining the inferior robustness of quantized models.Small input perturbations may initially be denoised, but amplified activation differences can place them in a regime where quantization enlarges the distance.

4 DEFENSIVE QUANTIZATION

Defensive Quantization suppresses adversarial-noise amplification by constraining the network’s Lipschitz behavior during quantization. It uses layer regularization and preserves efficiency while improving robustness over conventional quantization.

  • Motivation and approach: DQ suppresses noise amplification, keeping adversarial perturbations small enough for quantization to improve rather than reduce robustness.The method aims to keep perturbations within a range where they do not propagate and to turn activation quantization into a defense.
  • Experimental results: On CIFAR-10, DQ eliminates the robustness gap between quantized and full-precision Wide ResNet models and improves robustness beyond full precision.The table caption also reports that quantize gain improves as β increases.
  • Lipschitz control: A network with layer Lipschitz constants Lip(φi) ≤1 is non-expansive, preventing perturbations from being amplified across layers.The network Lipschitz constant is the product of individual layer constants and can grow exponentially when layer constants exceed 1.
  • Lipschitz control: DQ regularizes weight matrices toward WT W ≈ I, making their singular values close to 1 and reducing the linear layers’ Lipschitz constants.For convolutional layers, the weight tensor is reshaped into a matrix before applying the same regularization.
  • Optimization: The optimization combines cross-entropy loss with a β-weighted orthogonality regularizer, with ResNet aggregation layers modified as convex combinations.LCE is the original cross-entropy loss, W includes all network weight matrices, and β controls the regularizer’s relative importance.
  • Experimental results: White-box and black-box robustness trends are consistent: vanilla quantization drops robustness, whereas DQ improves it, especially at bit=1.The comparison uses Wide ResNet models with vanilla and defensive quantization.

5 EXPERIMENTS

Experiments show that Defensive Quantization retains or improves adversarial robustness while preserving quantization efficiency, and also improves clean-image training of quantized models. DQ further boosts robustness when combined with other defenses.

  • Robustness and combined defenses: DQ retains model robustness with low-bit quantization and can further improve robustness when combined with other defensive techniques.The experiments identify DQ as a general defense method across multiple settings.
  • Fixing robustness drop: The best vanilla quantized model has 9.1% lower robustness than its full-precision counterpart, whereas DQ quantized models achieve better robustness.Robustness generally improves with larger β under Lipschitz regularization, while smaller bit widths can denoise larger perturbations.
  • Robustness and combined defenses: Black-box and white-box attack trends are consistent: vanilla quantization reduces robustness, while DQ further improves it.Black-box samples are generated using a separately trained VGG-16 substitute model.
  • Robustness and combined defenses: DQ improves robustness under normal training, feature squeezing, and adversarial training settings, while adversarial training provides the best performance across attacks.The experiments include SVHN and CIFAR-10 evaluations with white-box and black-box attacks.
  • Training quantized models: DQ limits activation dynamic range, making quantized-model training easier and improving clean-image accuracy without adversarial attacks.It removes the accuracy gap between ReLU1 and ReLU6 quantized models observed with vanilla quantization.

6 CONCLUSION

The paper connects quantization efficiency with adversarial robustness through Defensive Quantization. Experiments on two datasets support using DQ to deploy more secure quantized models while maintaining efficiency.

  • Conclusion: DQ bridges model efficiency and adversarial defense by protecting quantized neural networks while maintaining their deployment efficiency.The paper targets quantized networks deployed on GPU, TPU, FPGA, and mobile-device platforms.

A.1 TANH-BASED ACTIVATION QUANTIZATION LEADS TO FAKE SECURITY

The appendix shows that Tanh-based activation quantization can create misleading white-box robustness. Its weaker black-box performance indicates gradient masking rather than dependable security.

  • Tanh-based activation quantization: Tanh-based quantization with PGD training achieves much higher white-box accuracy than the full-precision model, but black-box robustness decreases.The comparison uses PGD-adversarially trained ResNet-18 models on CIFAR-10 with 2-bit quantization.
  • Tanh-based activation quantization: Higher white-box than black-box accuracy is abnormal because black-box attacks are generally weaker, indicating severe gradient masking.This result motivates evaluating defenses under black-box attacks rather than relying only on white-box performance.

A.2 ADVERSARIAL FGSM TRAINING CAUSES GRADIENT MASKING

The appendix finds that standard FGSM adversarial training causes gradient masking, while randomized-start R+FGSM avoids this issue and is therefore used in the experiments.

  • Gradient masking: FGSM adversarial training produces higher white-box robustness but much worse black-box robustness than R+FGSM training.The models are evaluated with FGSM at ϵ = 8 under white-box and black-box settings using a VGG-16 substitute.
  • Gradient masking: R+FGSM training is used to avoid the gradient masking observed with standard FGSM adversarial training.The appendix treats the mismatch between white-box and black-box performance as evidence of masking.

B HYPER-PARAMETERS: β STUDY

For 4-bit quantized Wide ResNet, adversarial accuracy depends non-monotonically on β, peaking at β = 0.002 while clean accuracy remains relatively stable.

  • B HYPER-PARAMETERS: β STUDY: β = 0.002 delivers peak adversarial accuracy for the 4-bit quantized Wide ResNet.Adversarial accuracy rises initially with β, then declines under stronger regularization.
  • B HYPER-PARAMETERS: β STUDY: Adversarial robustness generally increases as β becomes larger before reaching its optimum.
  • B HYPER-PARAMETERS: β STUDY: Clean accuracy is more stable across the tested β values than adversarial accuracy.

C VISUALIZE SAMPLES AND PREDICTIONS

The visualization compares clean and adversarial predictions for full-precision, vanilla 4-bit, and Defensive Quantization models. Vanilla quantization misclassifies more adversarial samples, whereas DQ can outperform full precision in robustness and confidence.

  • C VISUALIZE SAMPLES AND PREDICTIONS: The visualization compares predictions from full-precision, vanilla 4-bit, and 4-bit Defensive Quantization models.It shows predicted classes and probabilities for clean and adversarial CIFAR-10 test samples.
  • C VISUALIZE SAMPLES AND PREDICTIONS: Vanilla 4-bit quantization misclassifies more adversarial samples than the full-precision and Defensive Quantization models.The cited examples are samples 1, 2, 4, and 6.
  • C VISUALIZE SAMPLES AND PREDICTIONS: Defensive Quantization succeeds on some attacks that the full-precision model fails to defend.The visualization identifies samples 5 and 7 as examples.
  • C VISUALIZE SAMPLES AND PREDICTIONS: When both models correctly defend an attack, DQ assigns higher confidence to the true label than full precision.This occurs in the cited examples samples 1, 4, and 6.
Loading 1904.08444v1…