Hierarchical Representations for Efficient Architecture Search

1 INTRODUCTION

• The growing complexity of image-classification networks makes handcrafted architecture design expensive and potentially suboptimal, motivating automated search for architectures that maximize validation accuracy.Evaluating candidate architectures typically requires training a neural network, so search-space constraints are commonly used to limit computational cost.Source: §1 · p. 1
• The search space builds higher-level motifs from lower-level primitives, permits flexible directed acyclic graphs at each level, and stacks top-level motifs into the final network.Motif changes propagate across the network, following a modular design pattern found in handcrafted architectures; architectures are discovered through evolutionary or random search.Source: §1 · pp. 1–2
• Although architecture evolution and indirect encodings had prior study, large-scale evolutionary or random search on image-classification benchmarks was only investigated more recently.The paper argues that well-designed search spaces can substantially enhance simple search methods.Source: §1 · p. 2
• Random or evolutionary search with a powerful architecture representation can scale to competitive ImageNet performance with significantly fewer computational resources, and evolutionary search achieves the best published results among evolutionary architecture-search techniques.The introduction also reports competitive image-classification architectures from simplistic random search, highlighting the importance of search-space construction.Source: §1 · p. 2

2.1 FLAT ARCHITECTURE REPRESENTATION

• A flat architecture is a single-source, single-sink computation graph whose nodes are feature maps and whose edges apply operations such as convolution or pooling.It is specified by an available operation set and an adjacency matrix that assigns an operation to each edge.Source: §2.1 · p. 2

2.2 HIERARCHICAL ARCHITECTURE REPRESENTATION

• Hierarchical representations build higher-level motifs by composing lower-level motifs, with the single highest-level motif corresponding to the full architecture.The lowest level consists of primitive operations; the representation is determined by the structures of motifs at every level and the bottom-level primitives.Source: §2.2 · p. 3

2.3 PRIMITIVE OPERATIONS

• The bottom hierarchy level contains six primitives, and a special none operation marks the absence of an edge between nodes.Applicable primitives use stride one and preserve spatial resolution through padding; convolutions are followed by batch normalization and ReLU, with channel count fixed to C.Source: §2.3 · pp. 3–4

3 EVOLUTIONARY ARCHITECTURE SEARCH

• The search treats hierarchical representations as genotypes and covers mutation, diversification-based population initialization, tournament selection, random search, and distributed implementation.Source: §3 · p. 4

3.1 MUTATION

• A mutation samples a target motif, successor node, and predecessor node, then replaces the operation between those nodes with a randomly sampled operation.Source: §3.1 · p. 4
• For flat genotypes with two levels, mutation omits the first step and sets ℓ to 2.Source: §3.1 · p. 4

3.2 INITIALIZATION

• Initialization starts with identity-mapping chains in every motif, then applies many random mutations, such as 1000, to diversify the genotype.The authors contrast this with trivial-network initialization used by Stanley & Miikkulainen (2002) and Real et al. (2017); diversification aims to cover the search space with non-trivial architectures, avoid handcrafted bias, and produce reasonably performing initial architectures without search.Source: §3.2 · pp. 4–5

3.3 SEARCH ALGORITHMS

• Evolution repeatedly selects promising genotypes, mutates offspring, and returns them to the population; the highest-fitness genotype after a fixed time is the final output.Each evolution step trains a model from scratch for a fixed number of iterations, with validation accuracy as fitness.Source: §3.3 · p. 5
• Tournament selection samples genotypes from the effective population, selects the highest-fitness individual, and uses a tournament size of 5% of the population.Genotypes are not removed, allowing the population to grow and maintain architecture diversity; the method resembles binary tournament selection in Real et al. (2017).Source: §3.3 · p. 5
• Random search evaluates randomly generated genotypes and returns the highest-fitness one; its population can be evaluated in parallel to reduce search time.Fitness is computed as in evolution.Source: §3.3 · p. 5

3.4 IMPLEMENTATION

• The distributed system separates an evolution controller from evaluation workers, with shared memory for genotypes and fitness and a queue for unevaluated genotypes.The controller selects and mutates a genotype when a worker becomes available, then queues it for evaluation.Source: §3.4 · pp. 5–6
• Workers assemble queued genotypes into architectures, train from scratch with random weight initialization for fixed steps, validate them, and record accuracy as fitness.The system does not use weight inheritance as in Real et al. (2017), though the authors say it could be incorporated.Source: §3.4 · p. 6
• Architecture evolution requires no synchronization, and all workers remain fully occupied.Workers evaluate queued genotypes while the controller continues selection and mutation as workers become available.Source: §3.4 · p. 6

4.1 EXPERIMENTAL SETUP

• Search optimized a convolutional cell using 40K CIFAR-10 training images and 10K validation images, then transferred the selected cell to a larger model; the test set was reserved for final evaluation.Fitness averaged four training-evaluation runs; larger CIFAR-10 evaluation reported the mean and standard deviation over five runs.Source: §4.1 · pp. 6–7, Figure 2 · p. 6
• CIFAR-10 evaluation used three groups of learned cells, while the ImageNet model used four groups and 299 × 299 inputs.CIFAR-10 group-end separable convolutions downsample spatial resolution; ImageNet training omitted auxiliary losses, weight averaging, label smoothing, and path dropout.Source: §4.1 · pp. 6–7, Figure 2 · p. 6

4.2 ARCHITECTURE SEARCH ON CIFAR-10

• Flat genotypes achieved higher fitness at greater parameter cost, while parameter-constrained flat genotypes and hierarchical genotypes achieved similar fitness.Fitness grew quickly and then plateaued during evolution; the first 200 steps were random initialization before mutation began.Source: §4.2 · pp. 7–8, Figure 3 · p. 7
• 1 hour on 200 GPUs enabled random search over 200 architectures; evolutionary search took 1.5 days, versus 11 days in (Real et al., 2017) and 4 days in (Zoph et al., 2017).Randomly sampled architectures performed surprisingly well, and random search over 200 architectures achieved very competitive results on CIFAR-10 and ImageNet.Source: §4.2 · p. 8, §4.3 · p. 8

4.3 ARCHITECTURE EVALUATION ON CIFAR-10 AND IMAGENET

• 3.63% ± 0.10% CIFAR-10 test classification error was achieved with c0 = 128; ImageNet validation errors were 20.3% top-1 and 5.2% top-5.The best results used evolution over hierarchical representations; the paper reports best published CIFAR-10 results using evolutionary architecture search and competitive performance on both datasets.Source: Table 2 · p. 9, Table 3 · p. 9, §4.3 · p. 9
• The evolved hierarchical cell contained direct input-to-output and internal skip connections through Motif 4 and Motif 5.Motifs 4 and 5 each contain an input-to-output skip connection.Source: §4.3 · p. 9

• The method uses a hierarchical representation in which smaller operations serve as building blocks for larger ones.Source: §5 · p. 9
• Strong results can be obtained with simple search algorithms, including evolution or random search, when paired with a well-designed architecture representation.Source: §5 · p. 9
• The best architecture achieves a state-of-the-art result on CIFAR-10 among evolutionary methods and scales to ImageNet with highly competitive performance.Source: §5 · p. 10

A ARCHITECTURE VISUALIZATION

• Only motifs 1, 3, 4, and 5 construct the best-performing hierarchical architecture’s cell, with motifs 3 and 5 dominating.The section visualizes the learned cell and motifs.Source: A ARCHITECTURE VISUALIZATION · p. 12, Figure 5 · p. 12, Figure 6 · p. 12