Source-linked AI summary
Hierarchical Representations for Efficient Architecture Search
Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, Koray Kavukcuoglu
TL;DR
Handcrafted neural architecture design is costly in a combinatorially expanding space. The paper combines hierarchical genetic representations with evolutionary or random search, achieving competitive image-classification results including 3.6% CIFAR-10 top-1 error and 20.3% ImageNet top-1 error.
Problem
Handcrafted neural architecture design requires extensive expert trial and error, while the combinatorial design space makes architectures expensive to obtain and potentially suboptimal.
Method
The paper uses hierarchical representations that compose lower-level motifs into larger architectures and searches them with evolutionary or random algorithms.
Results
3.6% top-1 error on CIFAR-10 and 20.3% top-1 error on ImageNet demonstrate competitive image-classification performance.
Takeaways & Limitations
Strong image-classification architectures can be obtained with simplistic search algorithms when the architecture representation and search space are well designed.
Takeaways & Limitations
The search learns a convolutional cell rather than the entire model, using fewer cells for fitness computation and more cells for full-model evaluation.
Abstract
from arXiv · showhide
We explore efficient neural architecture search methods and show that a simple yet powerful evolutionary algorithm can discover new architectures with excellent performance. Our approach combines a novel hierarchical genetic representation scheme that imitates the modularized design pattern commonly adopted by human experts, and an expressive search space that supports complex topologies. Our algorithm efficiently discovers architectures that outperform a large number of manually designed models for image classification, obtaining top-1 error of 3.6% on CIFAR-10 and 20.3% when transferred to ImageNet, which is competitive with the best existing neural architecture search approaches. We also present results using random search, achieving 0.3% less top-1 accuracy on CIFAR-10 and 0.1% less on ImageNet whilst reducing the search time from 36 hours down to 1 hour.
1 INTRODUCTION
The paper motivates automated architecture search because handcrafted neural networks are costly to design in an explosively large space. It introduces hierarchical representations and shows that evolutionary and random search can achieve competitive image-classification architectures.
- Handcrafted architecture design is expensive and may be suboptimal because neural-network design spaces grow combinatorially as topologies become more complex.
- Automated architecture search seeks architectures that maximize validation accuracy within a specified search space.
- Hierarchical representations build higher-level computation graphs from lower-level motifs and stack top-level motifs into the final network.Changes to motifs propagate across the whole network, reflecting modular patterns used in VGGNet, ResNet, and Inception.
- The method argues that random and evolutionary search can scale to ImageNet and achieve competitive performance with substantially fewer computational resources when paired with a powerful representation.
- The paper contributes hierarchical architecture representations, competitive random-search results, and a scalable evolutionary method that improves performance.
2 ARCHITECTURE REPRESENTATIONS
The architecture representation encodes neural networks as graphs of operations, then extends this encoding hierarchically so lower-level motifs become reusable building blocks for larger computation graphs.
- 2.1 FLAT ARCHITECTURE REPRESENTATION: Flat architectures are represented as single-source, single-sink computation graphs whose nodes are feature maps and edges carry primitive operations.
- 2.1 FLAT ARCHITECTURE REPRESENTATION: An architecture consists of available operations and an adjacency matrix that assigns an operation to each directed edge.
- 2.1 FLAT ARCHITECTURE REPRESENTATION: The architecture is assembled by placing operations according to the adjacency matrix.
- 2.2 HIERARCHICAL ARCHITECTURE REPRESENTATION: The hierarchy assembles primitive operations into level-2 motifs and then level-2 motifs into a level-3 motif.
- 2.1 FLAT ARCHITECTURE REPRESENTATION: Merging combines multiple feature maps, implemented here as depthwise concatenation rather than element-wise addition.
- 2.2 HIERARCHICAL ARCHITECTURE REPRESENTATION: Hierarchical representations use lower-level motifs as operations when constructing higher-level motifs, with the top-level motif defining the full architecture.
- 2.3 PRIMITIVE OPERATIONS: The representation supports six bottom-level primitives, stride-one operations with padded convolutions, and a none operation for absent edges.
3 EVOLUTIONARY ARCHITECTURE SEARCH
The search treats hierarchical architecture representations as genotypes, mutates them through randomized graph edits, and also supports flat genotypes as a special case.
- The search framework defines mutation actions, diversification-based initialization, tournament selection, random search, and distributed execution for hierarchical genotypes.
- 3.1 MUTATION: A hierarchical genotype mutation samples a target level, motif, successor node, predecessor node, and replacement operation.
- 3.1 MUTATION: For flat genotypes, mutation omits level sampling and fixes the target level to 2.
- 3.1 MUTATION: Randomly sampled mutation choices can add a new edge when the current operation is none.
3. Remove an existing edge: if o(ℓ−1)
The paper initializes diverse genotypes and evaluates them through asynchronous evolutionary or random search, refining or selecting architectures by validation fitness.
- Initialization: Initial genotypes begin as identity-mapping motifs and receive many random mutations to diversify the population.This provides non-trivial initial coverage while avoiding bias from handcrafted initialization.
- Evolutionary search: Asynchronous evolution selects promising genotypes by tournament, mutates them, and queues their offspring for evaluation.The controller uses shared memory and a data queue, while workers evaluate unevaluated genotypes.
- Fitness evaluation: Each evolution step trains a model from scratch for fixed iterations and records validation accuracy as fitness.The highest-fitness genotype in the population is selected after a fixed search duration.
- Evolutionary search: Tournament selection uses a sample of the current population, with tournament size set to 5% of the population and no genotype removals.Retaining genotypes allows the population to grow while maintaining architecture diversity.
- Random search: Random search generates genotypes independently, evaluates them in the same way as evolution, and selects the highest-fitness architecture.Its population-wide evaluations can run in parallel, substantially reducing search time.
4 EXPERIMENTS AND RESULTS
The experiments search for convolutional cells on CIFAR-10, compare flat and hierarchical genotypes, and transfer learned cells to larger CIFAR-10 and ImageNet models. Hierarchical evolution and random search produce competitive classification results while using substantially less search time than prior approaches.
- 4.1 EXPERIMENTAL SETUP: The framework searches for a convolutional cell in a small CIFAR-10 model, then transfers the selected cell to larger CIFAR-10 and ImageNet models.Fitness uses fewer cells for rapid evaluation, while final evaluation uses larger models with more cells.
- 4.2 ARCHITECTURE SEARCH ON CIFAR-10: The best-fitness curve rises quickly and then plateaus; the first 200 randomly initialized genotypes provide the random-search baseline before mutation begins.Thus, the best architecture at evolution step 200 corresponds to random search over 200 architectures.
- 4.2 ARCHITECTURE SEARCH ON CIFAR-10: Flat genotypes achieve higher fitness with more parameters, while parameter-constrained flat and hierarchical genotypes achieve similar fitness.The parameter threshold is selected to give the flat representation a parameter count comparable to the hierarchical representation.
- 4.2 ARCHITECTURE SEARCH ON CIFAR-10: 1 hour is required for random search over 200 architectures using 200 GPUs, while evolutionary search over 7000 steps takes 1.5 days.These search times are reported as faster than prior results using 250 and 450 GPUs.
- 4.3 ARCHITECTURE EVALUATION ON CIFAR-10 AND IMAGENET: 3.63% ± 0.10% CIFAR-10 test classification error and 20.3% ImageNet top-1 classification error are achieved by the best evolutionary architectures.The CIFAR-10 result uses c0 = 128 channels; the ImageNet model also achieves 5.2% top-5 error.
5 CONCLUSION
The paper presents an efficient evolutionary method built on hierarchical representations and shows that even simple search algorithms can find strong architectures. The best architecture performs strongly on CIFAR-10 and scales to ImageNet.
- Hierarchical representations use smaller operations as building blocks for larger neural-network components.
- Simple evolution or random search can obtain strong results when paired with a well-designed architecture representation.
- The best architecture achieves the state-of-the-art result on CIFAR-10 among evolutionary methods.
- The approach scales successfully to ImageNet with highly competitive performance.
A ARCHITECTURE VISUALIZATION
The visualization presents the learned cell and motifs of the best-performing hierarchical architecture. Only motifs 1, 3, 4, and 5 construct the cell, with motifs 3 and 5 dominating.
- The figure set includes a learned cell and six labeled motifs.
- Only motifs 1, 3, 4, and 5 are used to construct the cell.
- Motifs 3 and 5 are the dominating motifs in the cell visualization.