Source-linked AI summary
Path-Level Network Transformation for Efficient Architecture Search
Han Cai, Jiacheng Yang, Weinan Zhang, Song Han, Yong Yu
TL;DR
The paper addresses the limitation that existing function-preserving network transformations modify layers but not connection-path topology. It introduces path-level transformations and a tree-structured reinforcement-learning search space, achieving improved parameter efficiency and test accuracy on CIFAR-10 and ImageNet with about 200 GPU-hours.
Problem
Existing function-preserving transformations support layer-level modifications but cannot change connection-path topology, limiting efficient search over complex multi-path architectures.
Method
The method combines path-level function-preserving transformations, a tree-structured architecture space, and a bidirectional tree-structured reinforcement-learning meta-controller.
Results
The discovered architectures improve parameter efficiency and test performance, including 2.30% test error on CIFAR-10 with 14.3M parameters and 25.4% top-1 error on ImageNet with TreeCell-B.
Takeaways & Limitations
Path-level transformations support efficient architecture search over more complex path topologies while retaining weight reuse and transferring learned cells across architectures.
Takeaways & Limitations
The explored tree-structured architecture space is not the full architecture space enabled by the proposed path-level transformations.
Abstract
from arXiv · showhide
We introduce a new function-preserving transformation for efficient neural architecture search. This network transformation allows reusing previously trained networks and existing successful architectures that improves sample efficiency. We aim to address the limitation of current network transformation operations that can only perform layer-level architecture modifications, such as adding (pruning) filters or inserting (removing) a layer, which fails to change the topology of connection paths. Our proposed path-level transformation operations enable the meta-controller to modify the path topology of the given network while keeping the merits of reusing weights, and thus allow efficiently designing effective structures with complex path topologies like Inception models. We further propose a bidirectional tree-structured reinforcement learning meta-controller to explore a simple yet highly expressive tree-structured architecture space that can be viewed as a generalization of multi-branch architectures. We experimented on the image classification datasets with limited computational resources (about 200 GPU-hours), where we observed improved parameter efficiency and better test results (97.70% test accuracy on CIFAR-10 with 14.3M parameters and 74.6% top-1 accuracy on ImageNet in the mobile setting), demonstrating the effectiveness and transferability of our designed architectures.
1. Introduction
The paper targets efficient neural architecture search by extending weight-reusing network transformations from layer-level edits to path-topology changes. It combines these operations with a tree-structured reinforcement-learning controller and reports strong CIFAR-10 and ImageNet results under limited compute.
- Manual architecture engineering is expensive and may be suboptimal, motivating automatic neural architecture design.
- Conventional architecture search trains each candidate from random initialization, requiring substantial computation and often failing to surpass the best human-designed architectures.One cited search used 500 P100 GPUs for four days.
- Existing network transformations reuse weights but only modify layers, preserving the starting network’s connection-path topology.These operations include adding or pruning filters and inserting or removing layers.
- Path-level transformations modify path topology while reusing weights and preserving the network’s function, enabling searches beyond chain-structured networks.
- The method uses a tree-structured architecture space and a bidirectional tree-structured reinforcement-learning meta-controller to explore expressive multi-branch designs.
- 2.30% test error was achieved on CIFAR-10 with 14.3M parameters and additional regularization, compared with 2.40% for NASNet-A with 27.6M parameters.The search used about 200 GPU-hours, versus 48,000 GPU-hours for NASNet-A.
2. Related Work and Background
Prior architecture-search methods include several optimization paradigms and often evaluate architectures trained from scratch. This paper instead extends function-preserving transformations to path-level edits, using tree-structured cells and a tree-aware controller to explore complex multi-branch topologies.
- Architecture search has been pursued through neuro-evolution, Bayesian optimization, Monte Carlo Tree Search, and reinforcement learning.
- Searching CNN cells on smaller datasets reduces the cost of evaluating architectures intended for later transfer to large datasets.
- Network-transformation methods explore architectures from trained networks and reuse weights, unlike approaches that construct and evaluate candidates from scratch.
- Multi-Branch Neural Networks: Modern CNNs use multi-branch motifs with branch allocation, primitive operations, and a merge scheme; ResNets and DenseNets can be viewed as two-branch architectures.
- Tree-Structured Architecture Space: The proposed tree-structured space generalizes multi-branch architectures and embeds many paths within each CNN cell.
- Function-Preserving Network Transformation: Function-preserving transformations initialize a student network to preserve a trained teacher’s functionality, including widening, deepening, and compression operations.
- Path-Level Transformation: The paper extends function-preserving transformations from layer-level to path-level architecture modifications.
3. Method
The method expands function-preserving network transformations from individual layers to connection paths, enabling tree-structured architectures while reusing weights. A bidirectional tree-structured reinforcement-learning controller explores this space through node and edge transformations.
- 3.1. Path-Level Network Transformation: Path-level operations replace a single layer with a multi-branch motif using add or concatenation while preserving the original network function.Identity and convolution layers are presented as examples, with analogous transformations applicable to other layer types.
- 3.1. Path-Level Network Transformation: Combining these transformations with Net2Net operations inserts different layers into individual branches, producing substantially different paths and complex topologies.The operations can be repeatedly applied to applicable layers, including layers inside branches.
- 3.2. Tree-Structured Architecture Space: A tree architecture allocates an input feature map across child branches, applies edge operations, and merges child outputs recursively at each node.Feature maps move from the root toward leaves and are aggregated from leaves back toward the root.
- 3.2. Tree-Structured Architecture Space: The explored space uses replication or split allocation, add or concatenation merging, and seven primitive operations including convolutions, identity, and pooling layers.The listed primitive set includes 1 × 1 convolution, three depthwise-separable convolutions, two pooling layers, and identity.
- 3.2. Tree-Structured Architecture Space: Pooling layers are handled by reconstructing student-network weights to mimic teacher logits through knowledge distillation, with negligible reconstruction cost reported.This reconstruction addresses pooling layers that cannot be initialized as identity mappings.
- 3.3. Architecture Search with Path-Level Operations: The meta-controller expands leaf nodes, predicts branch structures and merge schemes, and replaces identity edges with primitive operations.It uses a tree-structured LSTM rather than a chain LSTM; Child-Sum units handle unordered add branches, N-ary units ordered concatenation branches, and edge LSTMs encode transitions.
4. Experiments and Results
Experiments show that path-level search discovers effective, transferable CNN cells using limited computation, improving parameter efficiency on CIFAR-10 and performance in the ImageNet mobile setting.
- Architecture search on CIFAR-10: RL validation accuracy gradually increases with sampled architectures, whereas random search fluctuates, indicating more focused exploration.The comparison uses random search as a black-box optimization baseline under the same condition.
- CIFAR-10 results: 3.64% test error is achieved with 3.2M parameters, compared with 3.46% from a 25.6M-parameter DenseNet-BC.Increasing the model to 13.1M parameters lowers test error to 3.35%.
- CIFAR-10 results: 3.14% test error with 5.7M parameters outperforms the best PyramidNet result of 3.31% with 26.0M parameters.The learned cells were trained from a DenseNet start point and transferred to PyramidNets.
- CIFAR-10 results: 2.30% test error with 14.3M parameters is obtained with Cutout, outperforming compared human-designed and automatically designed CIFAR-10 architectures.With DropPath, TreeCell-A reaches 2.99% test error with 5.7M parameters; with Cutout, it reaches 2.49% with 5.7M parameters.
- ImageNet results: 25.4% top-1 error is achieved by TreeCell-B in CondenseNet with 588M multiply-add operations, while TreeCell-A achieves 25.5% top-1 and 8.0% top-5 error.These results outperform MobileNet, ShuffleNet, and a similarly efficient CondenseNet, and are slightly better than NASNet-A.
5. Conclusion
The paper extends function-preserving network transformation from layer-level changes to path-topology modifications. It uses a tree-structured architecture space and bidirectional tree-structured reinforcement learning, achieving efficient and transferable architectures.
- Conclusion: Path-level transformations modify neural-network connection topology while preserving functionality through weight reuse.The operations extend existing function-preserving transformations beyond layer-level architecture modifications.
- Conclusion: A generalized tree-structured architecture space and bidirectional tree-structured RL meta-controller explore cells containing plentiful paths.The learned cells improve parameter efficiency and test accuracy on CIFAR-10 when combined with DenseNets and PyramidNets.
- Conclusion: The learned cell structures transfer to ImageNet in the Mobile setting after being combined with efficient architectures.The paper identifies transferability as a demonstrated outcome and proposes combining the method with compression operations as future work.
A. Architecture Search Starting from Scratch
A separate search begins from a chain of identity mappings rather than a successful human-designed architecture. RL still outperforms random search, but the resulting cell is weaker than TreeCell-A.
- Architecture search starting from scratch: The from-scratch search replaces identity mappings with sampled cells while keeping other configurations unchanged.The start point is a chain of identity mappings on CIFAR-10.
- Search comparison: RL achieves 1.54% better validation accuracy than random search in the from-scratch setting.The progress trends resemble the earlier search, with a larger RL advantage over random search.
- Results: 3.93% test error with 11.5M parameters is achieved by the best RL cell, versus 4.44% with 10.0M parameters for the best random cell.TreeCell-A reaches 3.14% test error with 5.7M parameters under the corresponding comparison.
B. Details of Architecture Space
The paper identifies two effective tricks for obtaining good performance in its tree-structured architecture space.
- Architecture-space details: Two experimental tricks are reported as effective for reaching good performance in the tree-structured architecture space.The supplied passage introduces the tricks but does not specify them.
B.1. Group Convolution
The experiments use standard 3 × 3 group convolution in the base networks, with the group count selected from the sampled tree-structured cell’s merge scheme and branch count.
- Base networks use standard 3 × 3 group convolution instead of normal 3 × 3 convolution.
- The number of groups G is selected from {1, 2, 4} according to the sampled tree-structured cell.
- Concatenation uses G = 1, while addition uses G = 2 for two branches and G = 4 for three branches.
B.2. Skip Node Connection and BN layer
Each tree-structured cell adds a skip connection for every root-child output and merges it with the child output after batch normalization, preserving a fallback to the original unit.
- A skip connection is inserted for each child of the root node in a tree-structured cell.
- The child output passes through a BN layer before being merged with its corresponding skip connection via addition.
- Each tree-structured unit can at least return to the original unit when the cell is not helpful.
C. Detailed Structure of TreeCell-B
Figure 9 presents the detailed structure of TreeCell-B.
- Figure 9 shows the detailed structure of TreeCell-B.
- TreeCell-B is the cell structure depicted in Figure 9.
- The figure is dedicated to TreeCell-B’s detailed structure.
D. Meta-Controller Training Procedure
The training procedure takes a base network and datasets, then repeatedly samples tree-structured cells, trains transformed networks, evaluates validation accuracy, and records results.
- The procedure takes a base network, training set, validation set, batch size B, and maximum number of networks M as inputs.
- A randomly initialized meta-controller samples a tree-structured cell during each training iteration.
- For an untested cell, the procedure trains a transformed base network on the training set and evaluates its validation accuracy.