PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepArchitect: Automatically Designing and Training Deep Architectures

Renato Negrinho, Geoff Gordon

arXiv:1704.08792v1stat.MLcs.LG

TL;DR

DeepArchitect targets the largely unexplored problem of automatically searching complex deep architectures, which are otherwise chosen through manual trial and error. It introduces a modular language and structure-aware search framework that represents and compiles architectures automatically. Experiments report that structure-aware algorithms outperform random search, while the framework supports expressive model discovery with limited expert effort.

  • Problem

    Complex deep-architecture search spaces remain largely unexplored, leaving experts to choose architectures through slow, intuition-guided trial and error.

  • Method

    A modular, compositional, extensible language represents expressive architecture search spaces, which search algorithms traverse and compile into computational graphs.

  • Results

    Search algorithms that leverage search-space structure outperform random search by generalizing more effectively across different models.

  • Takeaways & Limitations

    The framework enables experts to specify expressive architecture spaces and automatically discover competitive models without much effort.

  • Takeaways & Limitations

    The SMBO experiments use a simple ridge-regression surrogate with n-gram module features and leave more complex features, surrogates, and losses for future work.

Abstract

from arXiv · show

In deep learning, performance is strongly affected by the choice of architecture and hyperparameters. While there has been extensive work on automatic hyperparameter optimization for simple spaces, complex spaces such as the space of deep architectures remain largely unexplored. As a result, the choice of architecture is done manually by the human expert through a slow trial and error process guided mainly by intuition. In this paper we describe a framework for automatically designing and training deep models. We propose an extensible and modular language that allows the human expert to compactly represent complex search spaces over architectures and their hyperparameters. The resulting search spaces are tree-structured and therefore easy to traverse. Models can be automatically compiled to computational graphs once values for all hyperparameters have been chosen. We can leverage the structure of the search space to introduce different model search algorithms, such as random search, Monte Carlo tree search (MCTS), and sequential model-based optimization (SMBO). We present experiments comparing the different algorithms on CIFAR-10 and show that MCTS and SMBO outperform random search. In addition, these experiments show that our framework can be used effectively for model discovery, as it is possible to describe expressive search spaces and discover competitive models without much effort from the human expert. Code for our framework and experiments has been made publicly available.

1. Introduction

DeepArchitect addresses the difficulty of manually searching complex deep architectures by providing a modular language for expressing model search spaces and algorithms for exploring them. The framework automatically compiles selected models to computational graphs, while experiments show structure-aware search outperforms random search.

  • Motivation: Manual architecture selection requires supervising many interacting choices through slow trial-and-error guided largely by intuition.The expert typically specifies, trains, and evaluates one model at a time before proposing variations.
  • Contributions: The framework provides a modular, compositional, and extensible language for compactly representing expressive search spaces over models.The language gives the expert control over which model variations are considered.
  • Contributions: The language makes it easier to search automatically for performant models in expressive architectural search spaces.It combines structural model specification with structured hyperparameter search capabilities.
  • Contributions: Fully specified models can be directly compiled to computational graphs without requiring the human expert to write additional code.Compilation is part of the framework’s automated model-construction workflow.
  • Model search: The framework introduces search algorithms that exploit tree-structured spaces to allocate search effort across candidate models.The paper considers random search and algorithms that use information from prior evaluations.
  • Results: Random search can be effective, but algorithms leveraging search-space structure outperform it by generalizing more effectively across models.The comparison is reported as an experimental result among the model-search algorithms studied.

2. Related Work

Prior architecture-search and hyperparameter-optimization methods provide useful search strategies but often constrain how model spaces are represented. DeepArchitect instead combines structured search with a modular language that lets experts define, traverse, and compile expressive architecture spaces.

  • Conventional hyperparameter-optimization methods are primarily designed for Euclidean spaces and are ill suited to discrete architectural choices.
  • Evolutionary and reinforcement-learning architecture-search approaches typically use fixed, hard-coded spaces that limit expert control over task-specific inductive biases.
  • TPE requires experts to distill search-space hyperparameters, express them in Hyperopt, and write model-compilation code.
  • DeepArchitect uses a modular, compositional, and extensible language whose search spaces are built compositionally and whose composite hyperparameters are derived automatically.
  • Once hyperparameter values are chosen, models can be mapped to computational graphs without additional code from the human expert.

3. Roadmap to the DeepArchitect Framework

DeepArchitect separates model-space specification, model search, and model evaluation into modular components. These components communicate through defined interfaces, allowing search methods and evaluation procedures to be changed or extended independently.

  • The framework reduces model search to specification-language, search-algorithm, and evaluation-algorithm components.
  • Its specification language uses computational modules to express choices such as including modules, selecting module types, and setting structural hyperparameters.
  • The search algorithm allocates effort across the search space using performance from previously evaluated models and selects subsequent models to try.
  • Fully specified models are evaluated according to an expert-defined criterion, typically by training on a training set and evaluating on a validation set.
  • Evaluation hyperparameters can themselves be introduced into the specification language when the expert lacks one reasonable training procedure for every model.
  • Well-defined interfaces allow each framework component to be changed, improved, or extended while the others remain fixed.

4. Model Search Space Specification Language

The specification language represents architectures as compositions of computational modules, producing a tree whose paths correspond to fully specified models. Modules support automatic traversal and recursive compilation into computational graphs.

  • A computational module is the fundamental unit of the model search-space specification language.
  • The module definition assumes discrete hyperparameter values, with input dimensionality n, valid hyperparameter set H, parameter count p, and output dimensionality m.
  • Parameter count and output dimensionality may depend on input dimensionality and selected hyperparameters, as in affine modules with m = h and p = (n + 1)h.
  • The framework focuses on single-input, single-output architectures without output sharing, while noting that these ideas extend to multiple paths and sharing.
  • Basic modules implement transformations, whereas composite modules combine other modules and may introduce or inherit hyperparameters.
  • The example composition uses three composite and five basic modules to encode 24 models, represented by root-to-leaf paths in the search tree.
  • Each internal tree node represents a partial hyperparameter assignment, and each leaf represents a fully specified model.
  • In the illustrated path, choices include 64 filters, 3 × 3 filters, batch normalization before ReLU, and no dropout.

5. Model Search Algorithms

The framework supports random search, MCTS, tree restructuring, and SMBO over tree-structured architecture spaces. Tree restructuring increases information sharing for MCTS, while SMBO uses a surrogate to relate models beyond shared ancestors.

  • Random Search: Random search samples outgoing edges uniformly until reaching a model, making expressive structural search spaces searchable without considerable expert effort.
  • Monte Carlo Tree Search: MCTS uses accumulated evaluation information to steer search toward better-performing regions through tree and rollout policies.It expands the search tree incrementally and evaluates models at reached leaves.
  • Monte Carlo Tree Search: MCTS can be sample-inefficient when many related hyperparameter values are siblings, because it expands all children before deeper exploration.Similar numeric values may have similar performance, such as choosing 64 versus 80 convolutional filters.
  • MCTS with Tree Restructuring: Bisection restructures ordered hyperparameter choices into sequential binary decisions, trading greater depth for more sharing among values.The restructuring can use branching factors other than two and has no effect when the branching factor is already sufficiently large.
  • Sequential Model Based Optimization: SMBO fits a surrogate over evaluated models, approximately optimizes it through random rollouts, and balances surrogate-guided selection with random exploration.The surrogate is updated after each evaluation and can use architecture-pattern features.
  • Sequential Model Based Optimization: The experiments use a simple ridge-regression surrogate based on module-sequence n-grams, while richer features and objectives remain future work.

6. Model Evaluation Algorithms

The evaluation algorithm assigns a score after a model is fully specified, typically by training it and measuring validation performance. Training and evaluation hyperparameters may also be included in the searchable space.

  • A fully specified model is evaluated by computing a score for its root-to-leaf path, typically through training-set fitting and validation-set evaluation.
  • The searchable evaluation process can include optimizer choice, learning-rate schedules, early stopping, data augmentation, and their associated hyperparameters.The expert defines how these hyperparameters affect evaluation and implements the corresponding compilation functionality.

7. Experiments

Experiments search architecture and training hyperparameters for deep convolutional models on CIFAR-10 under a fixed evaluation budget. MCTS with bisection and SMBO eventually outperform random search, while the space also contains many poorly training models.

  • Experimental Setup: The CIFAR-10 search space varies deep convolutional architectures alongside optimizer, learning-rate, reduction, and patience hyperparameters.The architecture design reflects high-level hypotheses about depth, batch normalization, and dropout.
  • Experimental Setup: Five repetitions of each algorithm use 64 model evaluations, with every model trained for 30 minutes on GeForce GTX 970 GPUs.
  • Results: Around 89% accuracy is reached by all algorithms after 64 evaluations, but sophisticated methods do not outperform random search within the first six evaluations.
  • Results: MCTS with bisection begins outperforming random search around 32 evaluations, while SMBO eventually also outperforms random search.
  • Results: MCTS with bisection and SMBO achieve higher top accuracy and evaluate a larger fraction of high-performance models than random search.
  • Results: A significant number of models fail to exceed random validation performance, and models show large performance variability after 30 minutes of training.This variability indicates that the language represents expressive spaces containing substantially different model performances.

8. Conclusion

The paper presents a modular framework for specifying, evaluating, compiling, and searching expressive deep-model spaces. On CIFAR-10, MCTS with tree restructuring and SMBO outperform random search.

  • The framework combines a composable search-space language, a model evaluation algorithm, and search algorithms for automatically designing and training deep models.
  • The language defines expressive architecture spaces, models compile automatically to computational graphs, and random search can explore these spaces with limited expert effort.
  • On CIFAR-10, MCTS with tree restructuring and SMBO outperform random search.

A. Detailed Experimental Setup

The Section 7 search space represents architectural, module, and training hyperparameters in both LISP-like pseudocode and runnable Python. It uses modular components to define convolutional models and compile the selected configurations.

  • The search space varies network depth, normalization ordering, dropout, convolutional filter counts and sizes, and learning-rate schedules.
  • The training configuration includes optimizer choice, logarithmic initial learning rates, rate multipliers, patience values, and minimum learning rate.
  • Training hyperparameters are included by concatenating the module assigned to MH with the remaining model-hyperparameter modules.
  • The Python specification closely matches the LISP-like pseudocode in both semantics and length.
  • The implementation uses reusable module construction through Module_fn, which is instantiated within the complete search-space declaration.

B. List of Modules

The framework distinguishes basic modules with local hyperparameters from composite modules whose behavior depends on submodules. This module vocabulary supports configurable sequential, branching, repetition, optional, and residual architectures.

  • Basic modules operate without submodules and expose local hyperparameters and parameters.
  • Basic modules include affine, ReLU, dropout, two-dimensional convolution, max pooling, batch normalization, user-defined hyperparameters, and identity.
  • Composite modules accept submodules, and their specified hyperparameters depend on both the composite and its selected or contained submodules.
  • Concat connects submodules in series, while Or selects one submodule and Repeat applies a submodule multiple times with independently chosen repetition hyperparameters.
  • RepeatTied shares one hyperparameter assignment across repetitions; Optional conditionally includes a submodule, and Residual adds a skip connection with dimension padding when needed.
  • MaybeSwap connects two submodules in series while choosing which one comes first.

C. Module Interface

Every module follows a common interface for initialization, specification, choice selection, output-dimension computation, and computational-graph compilation. Composite modules implement this interface recursively through their submodules.

  • Implementing a new module type requires implementing the framework’s module interface.
  • initialize provides input dimensionality, while get outdim computes output dimensionality from the input and selected hyperparameters.
  • is specified checks completion before outdim and compile can be called.
  • get choices returns possible values for the current hyperparameter, and choose assigns one while advancing the module’s internal search state.
  • compile creates the model’s computational graph, recursively invoking compilation on submodules for composite modules.
  • Composite modules use submodule interfaces to determine dimensions, specify choices, and implement their own interfaces while keeping specification information local.

D. Beyond Single-Input Single-Output Modules

The framework supports complex modules with multiple signal paths when those paths remain encapsulated between a single input and single output. Such modules can be specified and compiled through their submodules and internal hyperparameters.

  • Complex signal paths are supported when they fork from one input and merge into one output within an encapsulated module.
  • NewModule can contain arbitrary single-input single-output submodules M1–M5 and transformations g1 and g2 with potentially additional hyperparameters.
  • The module interface can manage internal transformation hyperparameters while delegating submodule operations through recursive interface calls.
  • A NewModule instance becomes fully specified only after choosing hyperparameters for all constituent submodules and transformations.
  • Its output dimensionality is computed from g2’s hyperparameters and the output dimensions of M4 and M5.
Loading 1704.08792v1…