Source-linked AI summary
Neural Architecture Search without Training
Joseph Mellor, Jack Turner, Amos Storkey, Elliot J. Crowley
TL;DR
Manual neural architecture design is costly, and NAS remains expensive because candidate networks are typically trained during search. The paper uses activation overlap in untrained networks to score architectures, enabling training-free search that reaches 92.81% CIFAR-10 accuracy in 30 seconds on NAS-Bench-201 and can integrate with existing NAS methods.
Problem
NAS automates architecture design, but its reliance on training many candidate networks makes search slow and resource-intensive.
Method
The paper derives a KH-based score from activation patterns in untrained networks and uses it to rank randomly proposed architectures without training.
Results
Across multiple NAS benchmarks and design spaces, the score correlates with final accuracy; NASWOT reaches 92.81% CIFAR-10 accuracy in 30 seconds on NAS-Bench-201.
Takeaways & Limitations
NAS can be navigated in seconds on a single GPU, and the training-free score can be combined with more expensive search strategies.
Takeaways & Limitations
The demonstrated scope is restricted to convolutional architectures for image classification.
Abstract
from arXiv · showhide
The time and effort involved in hand-designing deep neural networks is immense. This has prompted the development of Neural Architecture Search (NAS) techniques to automate this design. However, NAS algorithms tend to be slow and expensive; they need to train vast numbers of candidate networks to inform the search process. This could be alleviated if we could partially predict a network's trained accuracy from its initial state. In this work, we examine the overlap of activations between datapoints in untrained networks and motivate how this can give a measure which is usefully indicative of a network's trained performance. We incorporate this measure into a simple algorithm that allows us to search for powerful networks without any training in a matter of seconds on a single GPU, and verify its effectiveness on NAS-Bench-101, NAS-Bench-201, NATS-Bench, and Network Design Spaces. Our approach can be readily combined with more expensive search methods; we examine a simple adaptation of regularised evolutionary search. Code for reproducing our experiments is available at https://github.com/BayesWatch/nas-without-training.
1. Introduction
Manual architecture design is costly, motivating NAS, but conventional search remains expensive because candidate networks must be trained. The paper proposes using activation-overlap structure in untrained networks to rank architectures and search without training.
- NAS automates architecture discovery, but conventional methods train candidate networks to guide successive proposals.One early approach used 800 GPUs for 28 days, illustrating the resource burden of training-driven search.
- The Hamming-distance kernel KH, computed from activation patterns in an untrained network, is distinctive for architectures that later perform well.Across NAS-Bench-201 and NDS-DARTS, good architectures show less similarity between different images.
- NASWOT scores randomly proposed architectures without training and returns the highest-scoring network after N proposals.The score is based on KH and can be integrated into existing search methods.
- 92.81% accuracy was achieved on CIFAR-10 in 30 seconds within NAS-Bench-201.The authors describe this as several orders of magnitude faster than traditional NAS methods, with a modest change in final accuracy.
- The approach uses a single GPU and is presented as a proof-of-concept that can also be combined with regularised evolutionary search.This supports both very rapid standalone search and integration with more expensive NAS techniques.
2. Background
NAS addresses the difficulty of manually designing neural architectures, while successive methods reduce but do not eliminate the computational cost of searching. Benchmarks make search algorithms tractable to evaluate, and related work explores weight sharing and training-free accuracy heuristics.
- 2. Background: Hand-designing architectures is challenging because it is difficult to choose connection placements and operations as networks grow more complex.This difficulty motivated NAS as an automated network-design process.
- NAS methods: Early NAS trained candidate networks and used reinforcement learning to update a controller, requiring 800 GPUs for 28 days on CIFAR-10.Searching over cells reduced cost, but a later cell-based method still used 500 GPUs across 4 days.
- Activation regions: Figure 2 illustrates that ReLU activation patterns partition input space into linear regions, with each region identified by the preceding units’ binary activation code.This establishes the geometric interpretation used by activation-based network scoring.
- NAS methods: Weight sharing enabled simultaneous candidate training and reduced CIFAR-10 search time to half a day on one GPU, but evidence suggests it can inhibit optimal-architecture search.This motivated interest in alternatives to training-based evaluation and shared weights.
- Training-free evaluation: Prior accuracy-estimation methods used predictive models, whereas this work investigates more fundamental architectural properties before training.Other heuristics include Gaussian-process approximations, neural tangent kernel conditioning, and counts of unique linear regions.
- NAS Benchmarks: NAS benchmarks provide tractable search spaces and training metadata, enabling direct evaluation of whether an algorithm finds good networks.The paper uses NAS-Bench-101, NAS-Bench-201, NATS-Bench, and Network Design Spaces.
3. Scoring Networks at Initialisation
The paper scores untrained architectures using overlap among activation patterns, aiming to predict trained accuracy without the expensive training step. Across multiple search spaces, the score correlates positively with validation accuracy and remains reasonably robust to data batches and initialisations.
- Scoring objective: The score is designed to estimate final trained accuracy from an architecture’s initial state, replacing or guiding NAS training evaluations.This targets the expensive inner-loop training step in NAS.
- Activation-based score: ReLU activation indicators form binary codes for inputs, with each code identifying a locally linear region of the network.The code records whether each rectified linear unit is active or inactive.
- Activation-based score: More similar binary codes make inputs harder to separate, while Hamming distance measures dissimilarity between their activation codes.Inputs sharing a code lie in the same linear region and are particularly difficult to disentangle.
- Score construction: The resulting kernel matrix compares activation-code similarity across a mini-batch, and the score is the logarithm of its determinant.The kernel uses NA−dH(ci, cj), where NA is the number of rectified linear units.
- Score interpretation: High-performing networks have fewer highly similar off-diagonal kernel elements, and a higher initial score implies improved final accuracy after training.For equal-trace kernels, the score is higher when the kernel is closer to diagonal.
- Empirical evaluation: The score positively correlates with trained validation accuracy across NAS-Bench-101, NAS-Bench-201, NATS-Bench SSS, and NDS, especially in NAS-Bench-201 and NDS-DARTS.Across NDS spaces, it correlates with accuracy while grad norm and synflow fluctuate substantially.
- Robustness: The ranking is reasonably robust to random mini-batches and noisy across initialisations, while the best-performing networks remain distinctive across mini-batch sizes.The score depends on mini-batch size, so the ablation normalises scores within each sampled size.
- Score dynamics: During training, the score rises sharply in the first few epochs and then flattens, while the relative ranking between networks remains stable.This observation was made for 10 NAS-Bench-201 networks evaluated on CIFAR-10.
4. Neural Architecture Search without Training — NASWOT
NASWOT ranks randomly sampled architectures using an untrained-state score instead of training candidates, achieving useful accuracy in seconds and integrating with evolutionary search.
- NASWOT randomly proposes N architectures, scores each while untrained, and outputs the highest-scoring network.
- NAS-Bench-101: NASWOT finds NAS-Bench-101 networks with final accuracy roughly midway between REA and random selection in under one minute on a single GPU.
- NAS-Bench-201: Non-weight-sharing methods outperform NASWOT, but NASWOT approaches their performance in seconds while incurring much lower search cost.
- Assisted Regularised EA — AREA: AREA uses the score to select REA’s initial population and outperforms REA on NAS-Bench-201 CIFAR-100 and ImageNet-16-120.AREA is very similar to REA on NAS-Bench-101.
5. Conclusion
The paper concludes that NASWOT can navigate benchmark search spaces in seconds using observations from initialized networks, while also supporting integration with existing NAS methods.
- NASWOT navigates NAS benchmark search spaces in a matter of seconds using observations from initialized neural networks.The authors present this as a proof-of-concept challenging expensive training-based black-box methods.
- Assisted Regularised EA — AREA: AREA combines the proposed score with regularised evolutionary search by selecting an initial population before training proceeds.
- Figure 7 compares search time against final accuracy for NAS-Bench-201 strategies on a 1080Ti.
- Limitations: The stated scope is restricted to convolutional architectures for image classification.
A.1. NAS-Bench-101
NAS-Bench-101 searches for a cell repeatedly stacked into a predefined network skeleton, representing each cell as a constrained directed acyclic graph.
- NAS-Bench-101 restricts search to an individual cell repeatedly stacked into a predefined skeleton.
- Each cell is a directed acyclic graph with up to 9 nodes and 7 edges.
- Nodes represent operations selected from 3 × 3 convolution, 1 × 1 convolution, and 3 × 3 max pool.
A.2. NAS-Bench-201
NAS-Bench-201 uses a common skeleton containing repeated unique cells and fixed residual downsampling blocks, with operations assigned to a densely connected four-node cell.
- NAS-Bench-201 networks share a common skeleton of unique-cell stacks interleaved with fixed residual downsampling blocks.
- Each cell is a densely connected DAG with four ordered nodes: A input, B and C intermediate, and D output.
- The cell has six edges, and each edge can perform one of five possible operations.
A.3. NATS-Bench
NATS-Bench provides topology and size search spaces built on a common skeleton, while the accompanying figures contrast cell representations across benchmark spaces.
- NATS-Bench contains topology and size search spaces that share a common skeleton based on NAS-Bench-201.The topology space varies cell operations, whereas the size space varies layer channels across five blocks.
- The NATS-Bench topology search space varies the operations comprising each network cell, matching NAS-Bench-201.
- The size search space selects each block's layer channel count from eight possible sizes, producing 32768 networks.
- NAS-Bench-101 cells use nodes for operations and edges for connections, whereas NAS-Bench-201 cells use intermediate states and edges as operations.
B. Additional Plots
Additional plots compare the untrained architecture score with validation accuracy after training across NATS-Bench and Network Design Spaces search spaces.
- Figure 9 plots the score from Equation 2 for 1000 randomly sampled untrained NATS-Bench SSS architectures against trained validation accuracy.The comparisons cover CIFAR-10, CIFAR-100, and ImageNet16-120.
- Figure 10 plots the score from Equation 2 for around 1000 randomly sampled untrained architectures in NDS-DARTS, NDS-ENAS, and NDS-PNAS against trained validation accuracy.The top row shows fixed-width and fixed-depth variants, while the bottom row shows variable-width and variable-depth spaces.