Source-linked AI summary
SMASH: One-Shot Model Architecture Search through HyperNetworks
Andrew Brock, Theodore Lim, J. M. Ritchie, Nick Weston
TL;DR
Neural-network architecture search is costly because candidate models require substantial engineering and validation. SMASH trains a HyperNet conditioned on architecture encodings to generate candidate weights, rank architectures in one run, and achieves competitive performance across several datasets, though its search space and correlation evidence are limited.
Problem
Choosing effective neural-network architectures requires substantial expertise and experimental time, while existing search methods may require many training runs.
Method
SMASH uses a HyperNet to generate weights conditioned on variable architectures, compares their validation performance, and selects an architecture for normal training.
Results
SMASH achieves competitive, though not state-of-the-art, performance across several datasets and similarly-sized hand-designed networks.
Takeaways & Limitations
SMASH can efficiently explore a broad, predefined architecture space through a proxy based on HyperNet-generated weights.
Takeaways & Limitations
SMASH does not address regularization, learning-rate schedules, initialization, or data augmentation, and its architecture space is predefined.
Abstract
from arXiv · showhide
Designing architectures for deep neural networks requires expert knowledge and substantial computation time. We propose a technique to accelerate architecture selection by learning an auxiliary HyperNet that generates the weights of a main model conditioned on that model's architecture. By comparing the relative validation performance of networks with HyperNet-generated weights, we can effectively search over a wide range of architectures at the cost of a single training run. To facilitate this search, we develop a flexible mechanism based on memory read-writes that allows us to define a wide range of network connectivity patterns, with ResNet, DenseNet, and FractalNet blocks as special cases. We validate our method (SMASH) on CIFAR-10 and CIFAR-100, STL-10, ModelNet10, and Imagenet32x32, achieving competitive performance with similarly-sized hand-designed networks. Our code is available at https://github.com/ajbrock/SMASH
1 Introduction
SMASH addresses the expense and expertise required to identify effective neural-network architectures by using a HyperNet to compare many candidates after one training run. Its flexible search mechanism covers diverse connectivity patterns and achieves competitive performance across several datasets.
- Architecture design requires substantial expertise and experimental time because depth, layer width, and connectivity are difficult to choose.
- SMASH trains a HyperNet to generate weights for variable-architecture models instead of fully training every candidate.
- Relative validation performance with HyperNet-generated weights can approximately rank many architectures at the cost of a single training run.
- The memory read-write scheme supports diverse architectures, including ResNet, DenseNet, and FractalNet connectivity patterns.
- SMASH achieves competitive performance with similarly-sized hand-designed networks across CIFAR-10, CIFAR-100, Imagenet32x32, ModelNet10, and STL-10.
2 Related Work
Prior architecture and hyperparameter-search methods improve exploration efficiency but often require many training runs or struggle with large, variable neural-network design spaces. SMASH instead targets a predefined, flexible architecture space while leaving several training hyperparameters outside its scope.
- Random search, Bayesian optimization, and bandit methods treat performance as a black box, but evaluation generally requires multiple training runs.
- Bayesian optimization is not typically used for variable-length settings such as model connectivity and structure.
- Evolutionary methods flexibly discover varied models but often struggle to scale to deep networks with vast search spaces.
- Reinforcement-learning architecture search can require twelve to fifteen thousand full training runs to find a solution.
- SMASH differs from random-weight and stochastic-connectivity approaches by generating weights dynamically through a HyperNet.
- SMASH searches a rich but predefined architecture space and does not address regularization, learning-rate schedules, initialization, or data augmentation.
3 One-Shot Model Architecture Search through HyperNetworks
SMASH samples variable network architectures, generates their weights with a HyperNet, and ranks them by validation performance before normally training the selected architecture. Its memory-bank representation encodes branching connectivity and architecture choices as inputs to the weight generator.
- SMASH procedure: SMASH samples a random architecture at each training step, generates its weights with a HyperNet, and updates the system end-to-end by backpropagation.
- SMASH procedure: After HyperNet training, SMASH evaluates sampled architectures on validation data, selects the best estimated architecture, and trains its weights normally.
- Core components: SMASH combines architecture sampling through memory-bank connectivity with a HyperNet mapping binary architecture encodings to weight space.
- Core assumption: The method hypothesizes that validation errors from generated weights correlate with performance after normal training, while warning that this correlation may not generalize.
- Memory-bank representation: A memory-bank network represents each layer as reading, modifying, and writing selected tensor banks, supporting single-branch and branching patterns.
- Memory-bank representation: The base network uses blocks with memory banks at successive spatial resolutions, with downsampling performed through a 1x1 convolution followed by average pooling.
- Architecture sampling: Architecture sampling varies bank counts, channels, read-write patterns, and operation definitions, including filter size, dilation, groups, and output units.
- Architecture-to-weight mapping: The HyperNet maps a tensor encoding of the architecture to weights, using a fully convolutional design whose output dimensions vary with the encoded architecture.
4 Experiments
The experiments test whether SMASH scores rank architectures reliably, examine conditions that weaken this correlation, and evaluate transfer and benchmark performance across several datasets. Results support rapid architecture comparison in some settings, while revealing sensitivity to HyperNet capacity and learned-weight ratios.
- Testing the SMASH correlation: SMASH scores correlated with true validation performance for 50 random CIFAR-100 architectures, but the authors caution that this was a single, non-generalizable test.The experiment’s expense prevented a satisfactory number of repeat trials.
- Testing the SMASH correlation: A lower-capacity HyperNet showed a breakdown in the correlation between SMASH scores and true performance.The authors expected reduced HyperNet capacity to impair weight generation across the full architecture range.
- Testing the SMASH correlation: Increasing the proportion of normally learned parameters produced lower SMASH validation errors but eliminated correlation with true performance and reduced the quality of top models.The authors identify insufficient HyperNet responsibility and cross-network score comparisons as two potential pitfalls.
- Architectural Gradient Descent by Proxy: Correct architecture encodings consistently produced the highest SMASH validation performance, suggesting that the HyperNet learned an architecture-to-weights mapping rather than ignoring the encoding.The test corrupted c by changing architecture information such as dilation values.
- Architectural Gradient Descent by Proxy: Gradient-based perturbations of the discrete architecture encoding did not improve SMASH scores over random perturbations in preliminary tests.The authors attribute this partly to the lack of an intuitively satisfying update rule for the discrete architecture space.
- Transfer Learning: The best CIFAR-100 architecture transferred to STL-10 with 17.54% error, outperforming the STL-10-discovered architecture at 20.275% but not WRN baselines at 15.43% and 16.06%.The result favors the hypothesis that larger training sets may help distinguish general architectures more than domain-specific search does.
- Transfer Learning: An 8M-parameter transferred architecture achieved 93.28% accuracy on ModelNet10 versus 93.61% for an 18M-parameter hand-designed Inception-ResNet trained on ModelNet40.The comparison uses different parameter counts and training datasets.
- Benchmarking: SMASHv2 used an expanded search space with variable filter sizes, groups, and operation structure, and the two highest-scoring CIFAR architectures were fully trained and evaluated.The resulting 16M-parameter networks achieved 20.60% error on CIFAR-100 and 4.03% on CIFAR-10, comparing favorably with other automatic methods.
5 Conclusion
SMASH accelerates architecture selection by learning a model over network parameters conditioned on architectural form. The method correlates proxy performance from generated weights with fully trained performance and achieves competitive, though not state-of-the-art, results across several datasets.
- SMASH learns a model over network parameters conditioned on the network’s parametric form to accelerate architecture selection.
- The method generates weights for variable architectures and uses their validation performance as a proxy for comparing architectural designs.
- SMASH introduces a flexible scheme for defining network connectivity patterns and generating weights for highly variable architectures.
- Generated-weight performance correlates with performance from fully trained weights, supporting efficient exploration of the architectural design space.
- The method achieves competitive, though not state-of-the-art, performance on several datasets.
Appendix A: Hyperparameters
The appendix specifies the sampled architecture and HyperNet settings for SMASHv1 and SMASHv2. SMASHv2 varies memory-bank sizes, filter sizes, dilation, convolutional groups, and operation configurations, while the HyperNet uses an ad-hoc DenseNet design.
- SMASHv1 uses memory banks with N = 6 channels, up to 240 banks per block, and depth compression ratio D = 3.
- SMASHv1 samples each layer’s units uniformly from 6 to 42 in even multiples of 6 and dilation factors uniformly from 1 to 3.
- SMASHv2 uses variable memory-bank sizes that are multiples of N = 8 up to Nmax = 64.
- SMASHv2 samples filter sizes from [3] [5] [7], dilation values with maximum spatial extent 9, and convolutional groups from [1].
- The HyperNet is an ad-hoc DenseNet with three dense blocks containing 8, 10, and 4 layers, each using growth rate 10.
Appendix B: Encoding Scheme Details
The encoding scheme represents variable architectures as conditional tensors that encode memory-bank reads, writes, and dilation. Constraints on channel divisibility reduce the embedding size, while the HyperNet generates all main-network weights in one pass.
- Layer output units must be divisible by memory-bank size N, input units must be divisible by D, and output units must be less than Nmax.
- These constraints reduce the embedding vector size by DN^2.
- The embedding tensor has shape c ∈ R1×(2M+dmax)×(Nmax/N)^2×nch/D.
- The conditional embedding is a one-hot encoding whose channels identify read banks, write banks, and the following convolution’s dilation factor.
- The HyperNet generates the entire main-network weight tensor in a single pass, producing W = H(c) before reshaping it into convolutional weights.
Appendix C: Experiment Details
The experiments define sampling, training, and evaluation procedures for SMASHv1 and SMASHv2. They use constrained architecture spaces, staged perturbation and Markov-chain selection, and separately train the resulting networks with freely learnable parameters.
- SMASHv1 uses N = 6-channel memory banks, D = 3 incoming-bank constraints, and output-unit counts divisible by 6 with Nmax = 42.
- The HyperNet is a 26-layer DenseNet using Leaky ReLU, 3x3 convolutions, simplified WeightNorm, and no biases.
- SMASHv2 evaluation samples 500 architectures, evaluates 100 perturbations around the best, then performs 100 better-score-only Markov-chain perturbations.
- Standard initialization outperformed SMASH-generated initialization for resulting networks, likely because SMASH uses WeightNorm while resulting networks use BatchNorm.
- The SMASH design-space exploration was intentionally limited to three settings for N and D and an untuned ad-hoc DenseNet HyperNet.
- ModelNet-10 uses fixed 3x3x3 filters to fit memory and manage compute, resulting in 8M rather than 16M parameters.
Appendix D: Future Directions
The authors identify several extensions to SMASH, including more intelligent architecture sampling, training initialization, broader architecture exploration, and richer memory operations.
- Sampling and optimization: SMASH samples configuration elements independently and uniformly, motivating Bayesian Optimization, HyperBand, reinforcement learning, or parallel validation workers.These alternatives could trade off exploration of rarely sampled architectures against architectures already performing well.
- Training acceleration: HyperNet-generated weights could initialize the resulting network to accelerate subsequent training.The authors relate this possibility to Net2Net.
- Broader architecture spaces: Current architecture exploration mainly varies layer sizes and skip connections, leaving activations, operation order, convolution counts, and block types unexplored.The authors propose multiscale SMASH and alternative mixtures of generated, learned, and fixed parameters.
- Memory mechanisms: Memory-bank operations could use learned softmax attention to choose read and write locations dynamically at inference time.The memory could also persist across training steps instead of being zeroed each time.
Appendix E: Visualizations of Discovered Architectures
The visualizations show simplified and expanded views of the best-found SMASHv1 and SMASHv2 architectures, including their blocks and connectivity. The captions clarify that floating paths are graph-generation artifacts attached to nearby rectangular nodes.
- SMASHv2: Figure 6 presents a simplified view of the best-found SMASHv2 architecture from CIFAR-100 with the highest SMASH score.N denotes the number of output units.
- SMASHv1: Figure 5 presents a simplified view of the best-found SMASHv1 architecture, including output units and dilation factors.N represents output units, while d represents the dilation factor for the 3x3 filter.
- SMASHv2: Figures 8 and 10 provide expanded, partially simplified views of the first and final blocks of the best SMASHv2 network.The figure captions identify these as block-level visualizations of the discovered network.
- SMASHv1: Figures 9, 12, 13, and 11 show expanded, partially simplified views of the blocks of the best SMASHv1 network.The visualizations cover its second, second, final, and first blocks, respectively, as labeled in the captions.
- Diagram interpretation: Floating paths in the expanded diagrams are graph-generation artifacts and attach to the nearest rectangular node.This clarification applies to the expanded visualizations of both SMASHv1 and SMASHv2.