Source-linked AI summary
Model Compression with Adversarial Robustness: A Unified Optimization Framework
Shupeng Gui, Haotao Wang, Chen Yu, Haichuan Yang, Zhangyang Wang, Ji Liu
TL;DR
The paper asks whether CNNs can be compressed without sacrificing adversarial robustness alongside accuracy. It proposes ATMC, which unifies adversarial training with structured compression constraints, and reports favorable trade-offs among model size, accuracy, and robustness across settings.
Problem
Few CNN compression studies examine adversarial robustness, leaving whether compressed models can remain accurate and robust an open question.
Method
ATMC formulates compression and adversarial training as a unified constrained min-max optimization problem integrating pruning, factorization, and quantization.
Results
Experiments show ATMC can achieve both high robustness and compactness, with properly compressed models sometimes maintaining nearly the original model’s accuracy and robustness.
Takeaways & Limitations
Robustness should be explicitly considered when evaluating and designing CNN compression methods because naive compression may significantly compromise it.
Takeaways & Limitations
The paper focuses on reducing model size and uses element-level ℓ0 sparsity, while acceleration-oriented adaptations may require group-sparsity constraints.
Abstract
from arXiv · showhide
Deep model compression has been extensively studied, and state-of-the-art methods can now achieve high compression ratios with minimal accuracy loss. This paper studies model compression through a different lens: could we compress models without hurting their robustness to adversarial attacks, in addition to maintaining accuracy? Previous literature suggested that the goals of robustness and compactness might sometimes contradict. We propose a novel Adversarially Trained Model Compression (ATMC) framework. ATMC constructs a unified constrained optimization formulation, where existing compression means (pruning, factorization, quantization) are all integrated into the constraints. An efficient algorithm is then developed. An extensive group of experiments are presented, demonstrating that ATMC obtains remarkably more favorable trade-off among model size, accuracy and robustness, over currently available alternatives in various settings. The codes are publicly available at: https://github.com/shupenggui/ATMC.
1 Introduction
CNN compression is important for resource-constrained IoT deployment, but compressed models may lose adversarial robustness even when clean accuracy remains strong. ATMC addresses this gap by jointly optimizing adversarial robustness with pruning, factorization, and quantization, while experiments report favorable robustness–size trade-offs.
- Background: CNN Model Compression: CNN deployment on IoT devices is constrained by model size, inference latency, and energy cost despite strong predictive performance.These deployment constraints motivate model compression for sensor-driven applications.
- Background: CNN Model Compression: Compression methods include pruning, factorization, and quantization, which reduce parameters through sparsity, smaller factors, or lower numerical precision.Pruning can include structured channel sparsity, while quantization can reduce 32-bit values to 8-bit or less.
- 1.1 Adversarial Robustness: Connecting to Model Compression?: CNNs remain vulnerable to adversarial input perturbations, creating security and trust concerns for applications such as mobile biometric verification.The paper notes that related tasks including image segmentation and graph classification are also vulnerable.
- 1.1 Adversarial Robustness: Connecting to Model Compression?: Few prior compression studies evaluated adversarial robustness, motivating the question of whether compressed models can remain both accurate and robust.Existing work primarily assessed accuracy on clean test sets, while robustness and compactness may vary by compression method.
- 1.2 Our Contribution: ATMC jointly incorporates adversarial training and pruning, factorization, and quantization in a unified constrained optimization framework for compact robust CNNs.An efficient algorithm is derived for the resulting constrained problem.
- 1.2 Our Contribution: ATMC can be extended beyond model size to inference speedup or energy efficiency by replacing its constraint with an appropriate measure such as FLOPs.The paper focuses on model size while identifying these extensions as possible.
- 1.2 Our Contribution: Experiments show ATMC achieves competitive robustness–model-size trade-offs, including high compression ratios with appealing robustness.The study compares ATMC with baselines and off-the-shelf solutions across extensive experiments.
2 Adversarially Trained Model Compression
ATMC formulates adversarially robust compression as a constrained min-max problem, integrating pruning, factorization, and quantization into a unified weight structure. It solves the formulation with an ADMM-based algorithm that alternates adversarial training, sparse optimization, quantization projection, and dual updates.
- 2 Adversarially Trained Model Compression: ATMC combines adversarial training with a structured compression constraint in a unified constrained optimization framework.The min-max objective models adversarial robustness, while the constraint enforces compression-related weight structures.
- 2.1 Formulating the ATMC Objective: Adversarial Robustness: Adversarial robustness is defined by maximizing the target loss over perturbations x′ within the bounded set B_∆^∞(x).The model then suppresses this worst-case risk through adversarial training.
- 2.2 Integrating Pruning, Factorization and Quantization for the ATMC Constraint: The framework integrates pruning, low-rank factorization, and quantization through a compound structure W = UV + C with bounded sparsity.The sparse factors U and V capture multiplicative structure, while C provides a sparse additive error term.
- 2.2 Integrating Pruning, Factorization and Quantization for the ATMC Constraint: Nonuniform quantization restricts nonzero parameters to a learned set of values, with the number of allowed values controlled by representation bits b.The quantization thresholds are learned rather than fixed in advance.
- 2.3 ATMC: Formulation: The global hyperparameters k and b control overall sparsity and per-element quantization precision, respectively, across the entire model.Different combinations can yield similar compression ratios but potentially different accuracy and robustness.
3 Experiments
Experiments evaluate ATMC against compression, defense, and sequential mixture baselines across datasets, models, compression settings, and attack conditions. ATMC delivers stronger robustness–accuracy–size trade-offs, including under varied attacks and perturbation levels.
- 3.2 Comparison to Pure Compression, Pure Defense, and Their Mixtures: NAP preserves strong benign accuracy but becomes substantially worse in robustness under adversarial attacks, exposing a hidden cost of naive compression.The results motivate evaluating compressed models on both benign and attacked testing sets.
- 3.2 Comparison to Pure Compression, Pure Defense, and Their Mixtures: Joint optimization by Aℓ0 and ATMC-32 bits achieves better benign-accuracy, robustness, and compression trade-offs than sequential AP and ALR.ATMC-32 bits is especially advantageous over Aℓ0 at low compression ratios, attributed to its new decomposition structure.
- 3.2 Comparison to Pure Compression, Pure Defense, and Their Mixtures: ATMC attains test accuracy and robustness comparable to dense adversarial training with only minimal parameters after compression.Its benign accuracy–compression trade-off is close to, and sometimes better than, NAP while providing much stronger robustness.
- 3.2 Comparison to Pure Compression, Pure Defense, and Their Mixtures: ATMC-8bit achieves roughly fourfold further compression at the same sparsity level while retaining competitive performance relative to ATMC-32bit.The framework jointly incorporates quantization with pruning and weight decomposition, allowing different combinations to be explored.
- 3.3 Generalized Robustness Against Other Attackers: ATMC-32bit outperforms AP across the full compression spectrum under PGD, FGSM, and WRM attacks with varied perturbation settings.ATMC-8bit enables more aggressive compression while maintaining similar or better robustness than ATMC-32bit at low compression ratios.
4 Conclusion
The paper addresses simultaneous robustness and compactness in CNNs through ATMC, a unified framework that incorporates both goals. Experiments indicate that naive compression can harm robustness, whereas joint optimization can preserve nearly the original model’s accuracy and robustness.
- 4 Conclusion: ATMC jointly optimizes CNN compactness and adversarial robustness through a unified constrained optimization framework.The experiments support the need to account explicitly for robustness during compression.
A ATMC combined with different adversarial training methods
ATMC remains advantageous when paired with WRM adversarial training and when PGD training uses a different perturbation magnitude.
- A ATMC combined with different adversarial training methods: The WRM experiment compares methods at matched pruning ratios, with Table 2 reporting the resulting model-size comparisons.For methods without quantization, pruning ratios equal standard compression ratios; ATMC-8bit therefore uses four times less model size than matched 32-bit ATMC under the stated comparison.
- A ATMC combined with different adversarial training methods: ATMC’s advantage persists when PGD adversarial training uses ϵ = 8 on CIFAR-10, with training and testing settings changed accordingly.Other settings remain untouched, and the comparison is reported in Table 3.
B ATMC quantization vs traditional quantization
ATMC jointly learns pruning and non-uniform quantization, and compares this approach with uniform quantization applied afterward.
- B ATMC quantization vs traditional quantization: ATMC-8bits is compared with a baseline that first applies ATMC-32bits and then uses standard uniform 8-bit quantization as post-processing.The comparison is evaluated on SVHN in Table 4.
C Experiments for larger Neural Networks
The paper evaluates ATMC on a larger ResNet101 network using CIFAR-10, with the other experimental settings kept identical to Section 3.1.
- C Experiments for larger Neural Networks: CIFAR-10 experiments evaluate ATMC on ResNet101, and Table 5 reports the results.The remaining experimental settings are identical to those in Section 3.1.