When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attacks

Minghao Guo; Yuzhe Yang; Rui Xu; Ziwei Liu; Dahua Lin

Source-linked AI summary

When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attacks

Minghao Guo, Yuzhe Yang, Rui Xu, Ziwei Liu, Dahua Lin

arXiv:1911.10695v3Nov 25, 2019cs.LGcs.CRcs.CVstat.ML

TL;DR

Deep neural networks are vulnerable to adversarial attacks, while prior robustness efforts have focused mainly on learning algorithms, losses, and related defenses rather than architecture. This paper uses one-shot NAS to study robust architectural patterns and develops RobNets, which outperform widely used architectures across multiple datasets and yield gains on additional benchmarks.

Problem
The paper addresses the limited comprehensive study of how intrinsic network architecture affects robustness to adversarial perturbations.
Method
The authors train a supernet once with one-shot NAS, sample sub-networks, fine-tune them, and analyze their robust accuracies to identify architectural patterns.
Results
RobNets consistently outperform other baselines under strong white-box attacks, with about 2% gains on CIFAR-100 and Tiny-ImageNet and approximately 10% on SVHN.
Takeaways & Limitations
Dense connectivity, convolution operations on direct edges under budget, and FSP-matrix distance provide design observations for robust architectures.

Abstract

from arXiv · show

Recent advances in adversarial attacks uncover the intrinsic vulnerability of modern deep neural networks. Since then, extensive efforts have been devoted to enhancing the robustness of deep networks via specialized learning algorithms and loss functions. In this work, we take an architectural perspective and investigate the patterns of network architectures that are resilient to adversarial attacks. To obtain the large number of networks needed for this study, we adopt one-shot neural architecture search, training a large network for once and then finetuning the sub-networks sampled therefrom. The sampled architectures together with the accuracies they achieve provide a rich basis for our study. Our "robust architecture Odyssey" reveals several valuable observations: 1) densely connected patterns result in improved robustness; 2) under computational budget, adding convolution operations to direct connection edge is effective; 3) flow of solution procedure (FSP) matrix is a good indicator of network robustness. Based on these observations, we discover a family of robust architectures (RobNets). On various datasets, including CIFAR, SVHN, Tiny-ImageNet, and ImageNet, RobNets exhibit superior robustness performance to other widely used architectures. Notably, RobNets substantially improve the robust accuracy (~5% absolute gains) under both white-box and black-box attacks, even with fewer parameter numbers. Code is available at https://github.com/gmh14/RobNets.

1. Introduction

The paper studies adversarial robustness from an architectural perspective, addressing which patterns, budget allocations, and indicators distinguish robust networks. Using one-shot NAS, it analyzes many architectures and develops RobNets from the resulting observations.

1. Introduction: The study targets the underexplored influence of intrinsic network architecture on resilience to adversarial perturbations.It frames architecture as an orthogonal dimension to adversarial learning, loss design, and preprocessing.
1. Introduction: The authors ask which architecture patterns matter, how to allocate parameters under a fixed capacity budget, and which statistical indicator predicts robustness.These questions organize the architectural analysis.
1. Introduction: One-shot NAS enables large-scale evaluation by training a supernet once, sampling sub-networks, and fine-tuning candidates to obtain robust accuracy.This reduces the burden of separately training many architectures.
1. Introduction: Analysis of 1,000 architectures finds a strong correlation between architectural density and adversarial accuracy, indicating that dense connectivity can improve robustness.The analysis uses a cell-based search space.
1. Introduction: Adding convolution operations to direct edges is more effective for robustness under fixed computational budgets, especially for small budgets.The comparison considers small, medium, and large budgets.
1. Introduction: The distance between clean-data and adversarial-data FSP matrices serves as an indicator of network robustness in a cell-free search space.The paper uses this observation alongside the architectural findings to design RobNets.
1. Introduction: Across CIFAR, SVHN, Tiny-ImageNet, and ImageNet, RobNets outperform widely used architectures, motivating greater attention to intrinsic architectural effects.The introduction presents this as evidence supporting the RobNet family.

2. Related Work

The related work covers adversarial defense methods and neural architecture search, contrasting established robustness techniques with computationally expensive conventional NAS approaches.

2. Related Work: Adversarial defense research includes attacks, preprocessing, feature denoising, regularization, unlabeled data, ensembles, and adversarial training.The passage identifies adversarial training as the most effective and standard method for improving robustness.
2. Related Work: Neural architecture search automatically designs architectures through reinforcement learning, evolution, or surrogate models instead of relying only on handcrafted networks.These approaches have been widely adopted in search processes.
2. Related Work: Conventional NAS methods often incur high computational costs, motivating weight-sharing approaches that reduce candidate-evaluation expense.The passage situates weight sharing as a response to NAS scalability concerns.

3. Robust Neural Architecture Search

The framework uses one-shot NAS to sample and finetune many architectures, enabling systematic analysis of architectural factors associated with adversarial robustness. The study finds that architecture density, convolution placement under a fixed budget, and deeper-cell FSP behavior are informative for robustness.

3.2. Robust Architecture Search Framework: One-shot NAS trains a supernet once, then samples and finetunes candidate subnetworks to evaluate robust accuracy across many architectures.The method avoids retraining a separate network for every architecture and is used to study architectural effects rather than select only one model.
3.1. Preliminary: The search space uses 3 × 3 separable convolution, identity, and zero, while permitting multiple operations between two intermediate nodes.This expands the topology space to include variants resembling human-designed architectures such as ResNet and DenseNet.
3.2. Robust Architecture Search Framework: Among 1,000 sampled architectures, robust accuracy varies substantially, and t-SNE embeddings separate the selected robust and non-robust groups.The results indicate that architecture parameters influence network robustness and motivate searching for shared architectural features.
3.3. Analysis of Cell-Based Architectures: Densely connected patterns show a strong correlation with adversarial accuracy, indicating that greater architecture density can benefit robustness.The study defines density as connected edges divided by all possible edges and reports predominantly positive classifier weights for denser wiring.
3.4. Architecture Strategy under Budget: As convolution count increases, adversarial accuracy improves steadily, while convolutions on direct edges contribute more than those on skip edges.Under small and medium computational budgets, a higher proportion of direct convolutions correlates positively with adversarial accuracy.
3.5. Towards a Larger Search Space: In deeper network cells, FSP distance positively correlates with the clean–adversarial accuracy gap, while robust networks have lower deeper-cell FSP matrix loss.The authors use this relationship as an indicator of robustness in the larger cell-free search space.

4. Experiments

Experiments evaluate RobNet architectures against white-box and black-box attacks across CIFAR-10 and additional datasets. RobNets consistently outperform comparison architectures, with robustness linked to architecture patterns, parameter budgets, and FSP-guided search.

RobNets consistently achieve higher robust accuracy than widely used architectures under different white-box attacks on CIFAR-10.
Architecture changes alone improve white-box robust accuracy by 5.1%, from 47.5% to 52.6%.
The Effect of Dense Connections: Densely connected patterns improve robustness, with DenseNet outperforming WideResNet despite using fewer parameters.
The Effect of Parameter Numbers: Larger computational budgets consistently produce higher network robustness across RobNet-small, RobNet-medium, and RobNet-large.
The Effect of Feature Flow Guided Search: RobNet-free achieves higher robust accuracy with 6× fewer parameters than RobNet-large-v2 through FSP-guided search without cell-based constraints.
Black-box Attacks: RobNet models are also more robust under transfer-based black-box attacks, and denoising modules improve both their clean and robust accuracy.
Transferability to More Datasets: Across SVHN, CIFAR-100, and Tiny-ImageNet, RobNets outperform baselines under strong white-box attacks, with gains of about 2% on CIFAR-100 and Tiny-ImageNet and approximately 10% on SVHN.

5. Conclusion

The paper proposes one-shot NAS for studying how network architecture affects adversarial robustness and uses the resulting observations to design RobNet architectures. Experiments validate the significance of architecture for network resilience to adversarial attacks.

The proposed robust architecture search framework uses one-shot NAS to understand architectural influence on adversarial robustness.
RobNet is a family of robust architectures discovered from observations about network design and validated through extensive experiments.
The study reports an intrinsic effect of network architectures on resilience to adversarial attacks.

A. Details of Robust Architecture Search

The robust architecture search algorithm trains sampled subnetworks from a supernet using PGD-based adversarial training. It then updates network parameters through stochastic optimization over sampled architectures and adversarial examples.

Algorithm 1 takes a supernet, architecture parameters, total iterations, and PGD attack iterations as inputs.
Each training iteration samples a batch and randomly zeros architecture-parameter elements to obtain a candidate network.
For each training example, the procedure initializes an adversarial example from the input and iteratively applies PGD updates.
The generated adversarial examples are used for the subsequent parameter update through SGD.
CIFAR, SVHN, and Tiny-ImageNet training uses standard augmentation with padding, random cropping, horizontal flipping, and normalization to [0, 1].

C. Additional Results on ImageNet

On ImageNet, RobNet-large achieves higher robust accuracy than ResNet variants against white-box PGD attacks while maintaining similar clean accuracy and using fewer parameters.

C. Additional Results on ImageNet: ImageNet adversarial training uses the “free” training scheme because standard adversarial training requires hundreds of GPUs for several days.The same hyper-parameters are retained across models.
C. Additional Results on ImageNet: RobNet-large consistently achieves higher robust accuracy than different ResNet models against white-box PGD attacks while maintaining similar clean accuracy.The comparison uses ImageNet models trained with the “free” adversarial training scheme.
C. Additional Results on ImageNet: RobNet-large has a substantially smaller model size than the baseline ResNet models.

D. Comparisons to More Architectures

The study broadens its CIFAR-10 comparison beyond ResNet and DenseNet to include VGG, MobileNetV2, and ResNeXt under identical adversarial-training settings.

D. Comparisons to More Architectures: The comparison includes VGG, MobileNetV2, and ResNeXt alongside ResNet and DenseNet as human-designed architecture baselines.
D. Comparisons to More Architectures: All models are evaluated on CIFAR-10 under an l∞-bounded white-box attack and identical training settings.

E. Complete Results of FSP Matrix Loss

The FSP analysis links deeper-cell FSP distance to the clean–adversarial performance gap and describes how RobNets are selected using density, direct convolutions, and FSP-based filtering.

E. Complete Results of FSP Matrix Loss: In deeper network cells, FSP distance positively correlates with the gap between clean accuracy and adversarial accuracy.The analysis indicates that robust networks have lower FSP matrix loss in deeper cells.
F. Visualization of RobNets: RobNet cell-based candidates are filtered for architecture density D ≥ 0.5 and a direct-convolution portion above 0.5.
F. Visualization of RobNets: For each computational budget, 50 filtered architectures are finetuned for 3 epochs, and the best-performing candidates define RobNet-small, RobNet-medium, and RobNet-large.
F. Visualization of RobNets: In the cell-free setting, architectures with average FSP distance above 0.006 across the last 10 cells are rejected before finetuning.Randomly sampled architectures are filtered using this robustness-indicator threshold.
F. Visualization of RobNets: Representative RobNet architectures are visualized to show the resulting family of selected designs.

G. Additional Black-box Attack Results

Under transfer-based black-box PGD attacks on CIFAR-10, RobNet-free has the highest robust accuracy across source networks and shows asymmetric transfer behavior relative to other architectures.

G. Additional Black-box Attack Results: The evaluation uses 100-iteration PGD-based black-box attacks generated on independently trained source networks, with all models adversarially trained using PGD with 7 steps.
G. Additional Black-box Attack Results: RobNet-free achieves the highest robust accuracy under transfer-based attacks from different source networks.
G. Additional Black-box Attack Results: For each target network, the strongest black-box adversarial examples come from a source network with the same architecture.The empirical lower bound is defined as the lowest accuracy in each target-network row.
G. Additional Black-box Attack Results: RobNet models are more robust to black-box attacks transferred from other models and generate stronger transferable adversarial examples than widely used models.

H. Additional White-box Attack Results

Under increasingly strong white-box PGD attacks on CIFAR-10, the RobNet family remains more robust than other networks. RobNet-large and RobNet-free achieve approximately 2% and 5% gains over state-of-the-art models, respectively.

∼2% and ∼5% improvements are achieved by RobNet-large and RobNet-free, respectively, compared to state-of-the-art models under strong white-box attacks.Attack iterations vary from 7 to 1000.
RobNet models outperform other architectures as white-box PGD attack iterations increase from 7 to 1000.All models are adversarially trained with PGD using 7 steps.
Attacker performance diminishes with 500∼1000 attack iterations.