PaperlayerView exact revision on arXiv

Source-linked AI summary

Graph HyperNetworks for Neural Architecture Search

Chris Zhang, Mengye Ren, Raquel Urtasun

arXiv:1810.05749v3cs.LGcs.CVstat.ML

TL;DR

Neural architecture search is costly because evaluating candidates requires repeatedly training their weights. The paper introduces Graph HyperNetworks, which operate on computation graphs to generate weights and use generated-weight validation accuracy for search, achieving nearly 10× faster search and extending to anytime prediction with stronger speed-accuracy trade-offs than manual designs.

  • Problem

    NAS requires expensive repeated optimization of candidate-network weights, motivating a faster way to evaluate architectures.

  • Method

    GHNs combine graph neural networks and hypernetworks to generate all parameters of a candidate architecture directly from its computation graph.

  • Results

    GHN-based search achieves competitive results on CIFAR-10 and ImageNet-mobile with nearly 10× faster speed than other random search methods.

  • Takeaways & Limitations

    GHNs also extend NAS to anytime prediction, where the paper reports outperforming the best human-designed architectures while reducing computation cost.

Abstract

from arXiv · show

Neural architecture search (NAS) automatically finds the best task-specific neural network topology, outperforming many manual architecture designs. However, it can be prohibitively expensive as the search requires training thousands of different networks, while each can last for hours. In this work, we propose the Graph HyperNetwork (GHN) to amortize the search cost: given an architecture, it directly generates the weights by running inference on a graph neural network. GHNs model the topology of an architecture and therefore can predict network performance more accurately than regular hypernetworks and premature early stopping. To perform NAS, we randomly sample architectures and use the validation accuracy of networks with GHN generated weights as the surrogate search signal. GHNs are fast -- they can search nearly 10 times faster than other random search methods on CIFAR-10 and ImageNet. GHNs can be further extended to the anytime prediction setting, where they have found networks with better speed-accuracy tradeoff than the state-of-the-art manual designs.

1 INTRODUCTION

Neural architecture search automates architecture design but traditionally incurs expensive repeated network training. The paper proposes Graph HyperNetworks, which model architectures as computation graphs to generate weights and accelerate search, including resource-aware anytime prediction.

  • Motivation: Traditional NAS treats architecture design as optimization but requires expensive inner-loop training for each candidate.The inner loop optimizes weights for a fixed architecture before the outer loop searches architectures.
  • Motivation: Hypernetworks amortize repeated optimization by learning to generate network weights directly.A trained hypernetwork can substitute for lengthy SGD-based optimization during architecture evaluation.
  • Graph HyperNetworks: GHNs represent architectures as computation graphs, explicitly modeling topology and accommodating varying graph sizes.This representation is presented as intuitive and extensible compared with tensor or sequence encodings.
  • Graph HyperNetworks: GHNs generate all weights of arbitrary CNNs, strengthening the correlation between generated-weight performance and fully trained performance.The paper positions complete weight generation as an advantage over methods that generate only a subset of weights.
  • Anytime Prediction: The method extends to anytime prediction, targeting settings where computational budgets vary and predictions must progressively update.The paper reports outperforming existing manually designed state-of-the-art models in this setting.
  • Results: GHN-based search achieves competitive NAS results on CIFAR-10 and ImageNet-mobile while running 10× faster than other random search methods.The contribution list reports this as a main empirical result.

2 RELATED WORK

Related NAS work improves architecture sampling or predicts performance, but these approaches generally do not eliminate the costly optimization needed to obtain training-ground-truth signals. This paper instead focuses on inferring parameters for a given architecture and uses computation-graph representations for hypernetworks.

  • NAS Search Methods: NAS research has used reinforcement learning, evolutionary methods, and gradient-based methods to optimize the outer architecture-search loop.These methods learn to sample architectures likely to achieve higher accuracy.
  • Performance Prediction: Performance predictors can use learning curves or architectures directly, but training them still requires expensive ground-truth optimization.Thus, these predictors do not avoid computing the inner optimization.
  • Hypernetworks: HyperNetworks generate another network’s weights, and SMASH applied them to NAS using a 3D tensor memory-channel encoding.The paper contrasts this encoding with its computation-graph representation.
  • Resource-Aware NAS: Recent NAS work also searches for computationally efficient architectures, including models optimized for anytime prediction’s speed-accuracy trade-off curve.The paper applies GHN-based search to this resource-aware setting.

3 BACKGROUND

The paper’s background combines graph neural networks, which propagate information over nodes and edges, with hypernetworks, which generate another network’s parameters. Figure 1 summarizes how graph propagation produces weights used to evaluate sampled architectures.

  • Graph Neural Networks: A graph neural network consists of nodes and edges, with recurrent node units sending and receiving messages across the graph.Each node stores an embedding that is updated through message passing.
  • GHN System Overview: Figure 1 depicts sampled architectures being graph-propagated so each node generates associated weights, after which the GHN is trained using the sampled network’s loss.Random architectures are ranked using validation performance with GHN-generated weights.
  • Graph Neural Networks: Node updates use a recurrent cell and messages aggregated from neighbors with incoming edges.The message function operates over the incoming-neighbor set, and recurrent cells may use LSTMs or GRUs.
  • Graph Neural Networks: Repeated propagation lets node embeddings aggregate graph-level information for node- or graph-level prediction.The embeddings can be further aggregated for graph prediction tasks.
  • Hypernetworks: A hypernetwork generates the parameters of another network through a learned function applied to layer embeddings.The same hypernetwork is shared across layers and can accommodate varying kernel or spatial sizes through concatenation and slicing.

4 GRAPH HYPERNETWORKS FOR NEURAL ARCHITECTURAL SEARCH

GHN combines graph neural networks with hypernetworks to generate all parameters of architectures represented as computation graphs, enabling architecture search using one trained GHN. It scales to repeated modules and uses efficient graph propagation and differentiable joint learning.

  • Graph HyperNetwork: GHN combines a graph neural network and hypernetwork to generate all free parameters in an architecture's computation graph.During evaluation, generated parameters score random architectures, and the best performer on a separate validation set is selected.
  • Graphical Representation: Architectures are directed acyclic graphs whose nodes apply parameterized operators to incoming activations and sum the resulting outputs.Nodes carry computational operators and edges represent activation flow between nodes.
  • Graph HyperNetwork: GHN uses graph message passing over a topology-matched graph, then applies a hypernetwork to node embeddings to generate each node's parameters.Node embeddings begin as one-hot operator encodings, and the hypernetwork is shared across nodes.
  • Architectural Motifs and Stacked GNNs: Repeated-module architectures are handled by stacking the same graph hypernetwork along depth and passing graph-level embeddings between modules.Each module embedding averages its node embeddings after propagation, while the GNN parameters are shared across repeated modules.
  • Propagation: Forward-backward propagation reduces graph messaging from O(|V|2) under synchronous propagation to O(|V|).A full pass uses 2|V|−1 propagation steps.
  • Learning: GHN parameters are learned by differentiating the training loss through generated network weights, with GNN and hypernetwork contributions jointly optimized.The gradient decomposes into the generated-weight loss gradient and the derivatives of the GNN and hypernetwork modules.

5 EXPERIMENTS

The experiments evaluate GHN-based architecture search on CIFAR-10, ImageNet-Mobile, and anytime prediction, alongside correlation and ablation studies. GHN achieves competitive search results with substantially lower search cost and benefits from graph-topology modeling and forward-backward propagation.

  • Experimental setup: GHN searches CNN architectures on CIFAR-10 and ImageNet benchmarks, then extends the search to anytime prediction and performance-correlation evaluation.The experiments also examine node counts, propagation schemes, and stacked GHN design choices.
  • CIFAR-10 and ImageNet search: GHN evaluates 1000 sampled architectures using generated weights and selects the top 10, while the reported search cost includes GHN training and evaluation.The random baseline trains 10 sampled architectures until convergence for comparison.
  • Search results: Nearly 10× lower search cost accompanies competitive GHN results against random-search, reinforcement-learning, and evolutionary NAS methods.The advanced-search comparison uses GHN to select top architectures after sampling 1000 candidates.
  • Anytime prediction: The best searched architectures can outperform current state-of-the-art human-designed networks on CIFAR-10.The result is reported for a strong search space using random search with GHN.
  • Predicted-performance correlation: GHN-generated parameters provide a stronger performance signal than a one-shot model, while remaining over 1000 times cheaper than running 1000 SGD steps per architecture.A six-hour GHN pretraining stage enables subsequent evaluation of many architectures of different sizes.
  • Ablation studies: Forward-backward propagation consistently outperforms synchronous propagation, and shared parameters with message passing improve stacked-GHN performance.More propagation steps help with diminishing returns, while fewer training nodes generally produce better performance and can generalize to full-size graphs.

6 CONCLUSION

The paper introduces Graph HyperNetworks, which generate weights directly from computation-graph representations and support faster, competitive architecture search across classification and anytime prediction.

  • GHNs generate the weights of any architecture by operating directly on its computation graph representation.The method combines graph neural networks with hypernetworks to predict parameters for unseen networks.
  • GHNs achieve competitive results on CIFAR-10 and ImageNet mobile with nearly 10× faster search than other random search methods.
  • The approach extends to anytime prediction and outperforms manually designed state-of-the-art models.This extension also reduces the computation cost of real-time neural networks.

7 APPENDIX

The appendix specifies the classification and anytime search spaces, implementation settings, and figures illustrating sampled-architecture comparisons and the best discovered blocks.

  • SEARCH SPACE: Classification blocks concatenate all leaf nodes and use two input nodes receiving outputs from the two preceding blocks.Input bottleneck layers can reduce spatial size with stride 2.
  • SEARCH SPACE: GHN requires O(N) + O(K) memory, whereas one-shot methods require O(KN) memory for K operations and N candidate nodes.This difference allows GHNs to search more operators than ENAS, which supports only five under memory constraints.
  • ANYTIME PREDICTION: Anytime blocks concatenate inputs, remove identity operations, and constrain node scales progressively from three scales in Block 1 to one scale in Block 3.Block 1 supports 32 × 32, 16 × 16, and 8 × 8; Block 2 supports 16 × 16 and 8 × 8; Block 3 supports 8 × 8.
  • IMPLEMENTATION: The GHN classification training setup uses 200 epochs, batch size 64, ADAM, and learning-rate halvings at epochs 100 and 150.Node operators are initialized as one-hot vectors and passed through a learned embedding matrix before graph propagation.
  • ANYTIME PREDICTION: Anytime GHNs additionally encode node scale and early-exit presence, while training with variable node counts requires bottleneck parameters.
  • IMPLEMENTATION: Final candidates use 600 training epochs for CIFAR-10, 250 for ImageNet Mobile, and 300 for anytime prediction.These settings use SGD-based training and distributed acceleration, with anytime models also matched to comparable FLOPS.
  • RESULTS: Figure 6 compares 50-epoch trained accuracy with accuracy from GHN-generated weights for the same 100 randomly sampled architectures.
  • RESULTS: Figure 7 presents the best block found for CIFAR-10 classification.
Loading 1810.05749v3…