Source-linked AI summary
Blockwisely Supervised Neural Architecture Search with Knowledge Distillation
Changlin Li, Jiefeng Peng, Liuchun Yuan, Guangrun Wang, Xiaodan Liang, Liang Lin, Xiaojun Chang
TL;DR
Existing NAS methods can misrank architectures when large search spaces are evaluated with undertrained shared parameters, making efficient search difficult. The paper introduces block-wise search with DNA architecture distillation to fully train candidates within blocks and guide the search, achieving 78.4% top-1 ImageNet accuracy and architectures that can exceed their teacher.
Problem
Existing NAS solutions have unclear effectiveness and efficiency because shared-parameter under-training can produce incorrect architecture ratings in large search spaces.
Method
The method modularizes the NAS search space into blocks, evaluates candidates within each block, and distills architecture knowledge from a teacher to supervise the search.
Results
78.4% top-1 accuracy on ImageNet is achieved in a mobile setting, about 2.1% higher than EfficientNet-B0, while searched architectures can exceed the teacher model’s capacity.
Takeaways & Limitations
Block-wise architecture supervision yields searched models that surpass their teacher and achieve state-of-the-art accuracy on ImageNet and transfer-learning datasets.
Abstract
from arXiv · showhide
Neural Architecture Search (NAS), aiming at automatically designing network architectures by machines, is hoped and expected to bring about a new revolution in machine learning. Despite these high expectation, the effectiveness and efficiency of existing NAS solutions are unclear, with some recent works going so far as to suggest that many existing NAS solutions are no better than random architecture selection. The inefficiency of NAS solutions may be attributed to inaccurate architecture evaluation. Specifically, to speed up NAS, recent works have proposed under-training different candidate architectures in a large search space concurrently by using shared network parameters; however, this has resulted in incorrect architecture ratings and furthered the ineffectiveness of NAS. In this work, we propose to modularize the large search space of NAS into blocks to ensure that the potential candidate architectures are fully trained; this reduces the representation shift caused by the shared parameters and leads to the correct rating of the candidates. Thanks to the block-wise search, we can also evaluate all of the candidate architectures within a block. Moreover, we find that the knowledge of a network model lies not only in the network parameters but also in the network architecture. Therefore, we propose to distill the neural architecture (DNA) knowledge from a teacher model as the supervision to guide our block-wise architecture search, which significantly improves the effectiveness of NAS. Remarkably, the capacity of our searched architecture has exceeded the teacher model, demonstrating the practicability and scalability of our method. Finally, our method achieves a state-of-the-art 78.4\% top-1 accuracy on ImageNet in a mobile setting, which is about a 2.1\% gain over EfficientNet-B0. All of our searched models along with the evaluation code are available online.
1. Introduction
NAS seeks to automate architecture design, but existing methods face unclear effectiveness because shared-parameter under-training can misrank candidates. The paper addresses this with block-wise search and architecture knowledge distillation, reporting stronger ImageNet performance and architectures that can exceed their teacher.
- Motivation: Existing NAS methods can be no better than random selection, leaving efficient and effective architecture search unresolved.The paper identifies unclear effectiveness and efficiency as a central problem.
- Motivation: Shared-parameter training accelerates NAS but may produce incorrect candidate ratings because architectures are undertrained in a large search space.The paper contrasts this with fully training every candidate, which is computationally impractical.
- Block-wise search: Block-wise search divides the large search space into smaller blocks so candidate architectures can be fully and fairly trained.The method reduces the number of candidates evaluated at each stage and also searches for architecture depth.
- Architecture distillation: DNA distills architecture knowledge from an existing model to supervise block-wise architecture search, treating architecture as a source of knowledge alongside parameters.The supervision reflects different representational roles across network blocks.
- Results: DNA can search architectures whose capacity is not bounded by the supervising model, including models with fewer parameters that significantly outperform it.The paper presents this as evidence of the method’s practicability and scalability.
- Results: 78.4% top-1 accuracy on ImageNet is achieved with 6.4M parameters, about 2.1% higher than EfficientNet-B0 with a similar parameter number.The paper presents this as state-of-the-art performance in a mobile setting.
2. Related Work
NAS research spans expensive full-training methods, weight-sharing approaches, and knowledge-distillation techniques. Related work highlights computational cost, optimization bias, and accuracy gaps between shared-weight proxies and independently trained models.
- Neural Architecture Search: Early NAS methods sample architectures and evaluate them through complete training, making them computationally expensive on large datasets.These methods use agents such as RNNs or evolutionary algorithms.
- Weight-sharing NAS: Gradient-based weight-sharing methods can bias sub-models through jointly optimized architecture choices and supernet weights.Poor early performers may receive less training and remain behind others.
- Weight-sharing NAS: One-shot NAS trains a shared supernet using path dropout or sampling, then evaluates sampled sub-models with inherited weights.The proxy evaluation is intended to avoid fully retraining every candidate.
- Knowledge Distillation: Knowledge distillation transfers information from a trained teacher to a smaller student through soft labels or internal representations.Block-wise distillation uses intermediate teacher feature maps to ease joint optimization, though it can increase the teacher–student gap.
3. Methodology
The method addresses inaccurate NAS evaluation by partitioning a large supernet into independently trained blocks, then uses teacher feature maps and architecture distillation to guide the search.
- Challenge of NAS: Shared-parameter supernets can misrank candidate architectures because sub-networks are not fairly or fully trained.The problem is especially acute in large search spaces, where fully training every candidate is impractical.
- Block-wise NAS: Block-wise NAS divides the supernet by depth into smaller search spaces whose candidates can be trained more fully and fairly.The approach is motivated by the finding that evaluation can be accurate when the search space is small and candidates are fully trained.
- Block-wise NAS: For block i with depth d and C candidate operations per layer, the search space contains C^d_i candidate architectures.The block-wise decomposition makes each block’s candidate set exponentially smaller than the whole search space.
- Block-wise Supervision: DNA supervises each student block with the corresponding teacher block’s output feature map, using L2 distance to mimic its behavior.The teacher’s previous feature map feeds both teacher and student blocks, while cells independently learn to reproduce the teacher block’s output.
- Block-wise Supervision: The block-wise search can be parallelized by feeding each student block the previous teacher block’s output feature map.This replaces sequential dependence during search with independently supervised block inputs.
- Constraint-Aware Search: Relative L1 loss normalizes feature-map discrepancies by teacher-feature standard deviation to support fair complexity allocation across blocks.The paper notes that MSE depends on feature-map size and variance, motivating the normalized criterion.
4. Experiments
Experiments evaluate DNA on ImageNet and transfer-learning datasets, comparing accuracy, efficiency, ranking effectiveness, training behavior, distillation, and teacher dependence. The results show strong performance under computational constraints, improved model ranking, progressive convergence, effective feature imitation, and robustness to teacher choice.
- ImageNet performance: DNA models achieve state-of-the-art ImageNet results under parameter and FLOPs constraints.DNA-a surpasses SCARLET-A with 1.8M fewer parameters under a 350M-FLOPs constraint.
- ImageNet performance: 78.4% top-1 accuracy is achieved by DNA-d with 6.4M parameters and 611M FLOPs, while DNA-b and DNA-c outperform EfficientNet-B0 by 1.2% and 1.5%.At 240 × 240 input size, DNA-d reaches 78.8% top-1 accuracy and is 1.4M smaller than EfficientNet-B1.
- Transfer learning: DNA models maintain superiority when transferred to CIFAR-10 and CIFAR-100.Transfer performance is reported in Table 3.
- Ranking effectiveness: DNA ranks candidate models more effectively than SPOS because its block-wise procedure trains candidates more fairly and fully.The ranking comparison samples 16 models and evaluates predictive versus stand-alone performance.
- Training progress: Searched-model accuracy increases progressively with supernet training and converges between the 16th and 20th epochs.The accuracy trend follows the decrease in supernet training loss.
- Distillation: The student supernet closely imitates teacher feature maps in blocks 2 and 4, including highly abstracted 14 × 14 maps.The feature-map comparison is presented at epoch 16.
- Ablation study: The proposed distillation strategy outperforms progressive alternatives S1 and S2, while additional cells improve searched-model performance.Table 4 evaluates the strategies and the effect of increasing channel and layer variability.
- Teacher dependency: DNA performance does not necessarily depend on a high-performing teacher: EfficientNet-B0-based search performs almost the same as EfficientNet-B7-based search.DNA-B0 outperforms its teacher by 1.5% at the same model size, and a scaled student outperforms its heavy teacher by 2.1%.
5. Conclusion
DNA modularizes NAS search spaces into blocks and adds block-wise knowledge distillation supervision. The searched architectures surpass the teacher model and achieve state-of-the-art accuracy across ImageNet and two transfer-learning datasets.
- DNA modularizes large NAS search spaces into blocks to improve one-shot architecture search effectiveness.
- Block-wise knowledge distillation supervises architecture search, supported by a multi-cell supernet and efficient evaluation and search algorithms.
- The searched architectures surpass the teacher model and achieve state-of-the-art accuracy on ImageNet and two transfer-learning datasets when trained from scratch.
A.1. Model Architectures
DNA architectures are organized into modules described by inverted bottleneck expand rates and kernel sizes.
- DNA-a,b,c,d are presented as distinct neural architectures.
- “MB x y × y” denotes an inverted bottleneck convolution module with expand rate x and kernel size y.