PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards modular and programmable architecture search

Renato Negrinho, Darshan Patil, Nghia Le, Daniel Ferreira, Matthew Gormley, Geoffrey Gordon

arXiv:1909.13404v1cs.LGcs.AIstat.ML

TL;DR

Architecture search systems are specialized, while general-purpose hyperparameter tools leave architecture-specific encoding and mapping to experts. This paper introduces a modular language for general computational-graph search spaces and a consistent interface between spaces and algorithms. Its experiments demonstrate comparing combinations without implementing each pair from scratch, supporting more reusable and comparable research.

  • Problem

    Architecture-search systems are tied to specific use-cases, while hyperparameter optimization tools lack constructs for efficiently encoding architecture spaces and require experts to map hyperparameter values to architectures.

    Source: §1 · pp. 1–2

  • Method

    The paper defines a modular language for general computational-graph search spaces and exposes them to search algorithms through a consistent interface.

    Source: §1 · p. 2, §3 · p. 3, §6.3 · p. 8

  • Results

    The authors reimplement literature search spaces and algorithms and compare them under the same conditions, demonstrating mix-and-match experiments without implementing each pair from scratch.

    Source: §1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    The authors conclude that decomposing experiments through this language may support more reusable and comparable architecture-search research.

    Source: §8 · p. 9

Abstract

from arXiv · show

Neural architecture search methods are able to find high performance deep learning architectures with minimal effort from an expert. However, current systems focus on specific use-cases (e.g. convolutional image classifiers and recurrent language models), making them unsuitable for general use-cases that an expert might wish to write. Hyperparameter optimization systems are general-purpose but lack the constructs needed for easy application to architecture search. In this work, we propose a formal language for encoding search spaces over general computational graphs. The language constructs allow us to write modular, composable, and reusable search space encodings and to reason about search space design. We use our language to encode search spaces from the architecture search literature. The language allows us to decouple the implementations of the search space and the search algorithm, allowing us to expose search spaces to search algorithms through a consistent interface. Our experiments show the ease with which we can experiment with different combinations of search spaces and search algorithms without having to implement each combination from scratch. We release an implementation of our language with this paper.

1 Introduction

Architecture search has improved on handcrafted architectures, but existing tools do not readily support general search workflows. The paper proposes a modular language that decouples search spaces from algorithms and makes their combinations easier to compare.

1 Introduction

  • Current architecture-search systems are tied to particular use-cases, while general-purpose hyperparameter tools lack constructs for efficiently encoding architecture search spaces.Using hyperparameter optimization tools also requires experts to encode the search space as conditional hyperparameters and manually map values to architectures.Source: §1 · p. 1
  • Ad-hoc encodings often intertwine search spaces and algorithms, making new combinations costly to implement.This limits extensibility and programmability for researchers building on prior work.Source: §1 · p. 2
  • The proposed language encodes search spaces over general computational graphs and exposes them through a consistent interface, decoupling their implementations from search algorithms.The authors use it to encode literature search spaces and compare combinations without implementing each pair from scratch.Source: §1 · p. 2

2 Related work

Prior work includes general-purpose hyperparameter optimization and architecture-search systems, but neither supplies a consistent, extensible approach to encoding general architecture search spaces.

2 Related work

  • Hyperparameter optimization systems are domain-independent, but experts must translate problems into hyperparameter spaces and map values to implementations [3] [4].Methods include Gaussian-process Bayesian optimization, sequential model-based optimization, and random search [10].Source: §2 · p. 2
  • Architecture-search research spans algorithms, evaluation strategies, and search spaces, but most spaces target recurrent or convolutional models for image classification or language modeling [13] [14] [15].Systems such as Auto-Sklearn and TPOT target featurized vector data, while Auto-Keras focuses on image classification [20] [21].Source: §2 · p. 2
  • The literature lacks a consistent method for designing and encoding architecture search spaces, and existing systems’ limited extensibility makes them unsuitable for general research workflows.Search spaces specify graph connectivity, operation counts and types, and operation values.Source: §2 · p. 2

3 Proposed approach: modular and programmable search spaces

The proposed language treats search spaces as composable structures over computational graphs. Its modularity, laziness, and compilation support reusable encodings and flexible search workflows.

3 Proposed approach: modular and programmable search spaces

  • Search-space nodes may themselves be search spaces, and assigning all their hyperparameters yields a single computational graph.This makes writing a search space similar to specifying a fixed graph in a deep-learning framework.Source: §3 · p. 2
  • Modules and hyperparameters compose search spaces, allowing local module implementations, reuse across spaces, and applications beyond deep-learning architectures, including Scikit-Learn pipelines [22].New spaces can combine existing ones.Source: §3 · p. 3
  • Substitution modules delay creating a subsearch space until their hyperparameters are assigned, allowing implicit conditional constructions and spaces that can expand infinitely.This avoids explicitly writing every conditional branch, unlike Hyperopt’s conditional spaces.Source: §3 · p. 3
  • After choices are made, the terminal search space can be mapped to a runnable computational graph, rather than requiring experts to write that mapping manually.The paper presents this as automatic compilation to runnable graphs.Source: §3 · p. 3

4 Components of the search space specification language

The language represents a search space as a graph of hyperparameters and modules. Independent and dependent choices, computation-bearing basic modules, structural substitutions, and auxiliary functions support progressively more complex encodings.

4 Components of the search space specification language

  • A search space is a graph of independent or dependent hyperparameters and basic or substitution modules.The section uses code and graph examples to illustrate the components.Source: §4 · p. 3
  • Independent hyperparameters take values from a specified set, while dependent hyperparameters compute values from other hyperparameters.For example, a dependent value can encode a relation such as increasing filter count after spatial reduction.Source: §4 · p. 3
  • Basic modules implement computation using their property values; once all hyperparameters are assigned, the resulting fixed graph can be mapped to an implementation.A fully connected layer can be wrapped as a module with a hyperparameter for its number of units.Source: §4 · pp. 3–4
  • Substitution modules replace themselves with subsearch spaces after their hyperparameters are assigned, potentially creating new hyperparameters or an identity module.Figure 2’s search space uses substitutions to choose ReLU or tanh and the number of hidden layers.Source: §4 · p. 4, Figure 2 · p. 4
  • Auxiliary functions combine search-space constructors; an RNN-cell example captures dependencies between hidden states and outputs.The example instantiates the cell using constructors for sequential and multilayer networks.Source: §4 · p. 4

5 Example search space

The example encodes a convolutional search space with optional dropout and two parallel chains whose lengths depend on one another. Assigning hyperparameter values progressively resolves the space into one architecture; it contains 25,008 distinct models.

5 Example search space

  • 25,008 distinct models are represented by a space with optional dropout and parallel convolutional chains of linked lengths.The first chain has length 1, 2, or 4; the second has double its length, and the chains' outputs are concatenated.Source: §5 · p. 4
  • Each convolutional layer independently chooses 64 or 128 filters, while optional dropout uses rate 0.25 or 0.5.Source: §5 · p. 4
  • Assigning IH-3 triggers a dependent-hyperparameter assignment and resolves both repeat substitutions; assigning IH-2 to 1 creates dropout and its rate hyperparameter.After the remaining independent hyperparameters are assigned, the graph represents one architecture that can be mapped to a deep-learning framework.Source: §5 · p. 5, Figure 5 · p. 5

6 Semantics and mechanics of the search space specification language

The language represents a search space as a graph of modules and hyperparameters, then uses value assignments and graph traversal to reach runnable architectures. Its ordered iteration and assignment interface lets search algorithms use different search spaces without implementing each pairing from scratch.

6.1 Semantics

  • A search space consists of modules and hyperparameters connected in a graph, with independent and dependent hyperparameters and basic and substitution modules as distinct types.Source: §6.1 · pp. 5–6
  • Dependent hyperparameter values are computed from their dependencies, while independent values are assigned directly from their possible-value sets.Modules have inputs, outputs, and associated hyperparameters; directed edges connect module outputs to inputs.Source: §6.1 · p. 6

6.2 Mechanics

  • Assigning an independent hyperparameter value resolves resulting dependent assignments and substitutions, progressively shrinking the reachable search spaces until no unassigned independent hyperparameters remain.Terminal spaces contain only basic modules, with no unresolved substitutions.Source: §6.2 · pp. 6–7, Figure 6 · p. 7
  • Ordered traversal exposes unassigned independent hyperparameters to search algorithms, and terminal spaces map to implementations by traversing modules in dependency order.Search algorithms assign values until a terminal space is reached; the resulting graph can then be mapped to a domain implementation.Source: §6.2 · pp. 7–8

6.3 Supporting search algorithms

  • Search algorithms use ordered iteration and value assignment through a shared interface, enabling search spaces and algorithms to be mixed without implementing each pair from scratch.Source: §6.3 · p. 8

7 Experiments

The experiments separately vary search spaces and search algorithms through a shared evaluation setup. The reported comparison finds that search-space choice had a much larger impact on test accuracy than search-algorithm choice.

7 Experiments

  • The shared search-space interface supports experiments that reuse search-space and search-algorithm implementations without implementing every pairing from scratch.The experiments demonstrate this modularity by varying the search space while fixing the algorithm and evaluation method, then varying algorithms on one search space.Source: §7 · p. 8, §7.1 · p. 8, §7.2 · p. 8

7.1 Search space experiments

  • The search-space experiments sampled 128 architectures from each of Nasbench, Nasnet, Flat, and Genetic, training each for 25 epochs before evaluating the best-validation-accuracy architecture.They characterize parameter counts, training times, and validation performance at 25 epochs.Source: §7.1 · p. 8, Figure 8 · p. 9

7.2 Search algorithm experiments

  • Search-space choice had a much larger impact on test accuracies than search-algorithm choice in the reported comparison.The comparison draws on the test results for both search-space and search-algorithm experiments.Source: Table 2 · p. 8, Table 1 · p. 8, §7.2 · p. 9
  • The search-algorithm experiments used the Genetic search space, ran random search, regularized evolution, SMBO, and MCTS for three trials each, and trained 256 sampled architectures per run for 25 epochs.The reported algorithm comparison includes best-architecture performance by sample count and the distribution of encountered validation accuracies.Source: §7.2 · pp. 8–9, Figure 9 · p. 9

8 Conclusions

The paper presents a language for encoding architecture search spaces over computational graphs and decoupling search-space implementations from search algorithms. It reimplements literature search spaces and algorithms under shared conditions, and argues this decomposition may support more reusable and comparable research.

8 Conclusions

  • The language encodes architecture search spaces and decouples their implementations from search algorithms, allowing combinations without writing each pair from scratch.The paper reimplements search spaces and algorithms from the literature and compares them under the same conditions.Source: §8 · p. 9

A Additional details about language components

The appendix illustrates how the language represents shared and dependent hyperparameters, wraps operations as modules, and composes reusable search-space functions. It also encodes a recurrent-cell search space and describes ordered graph traversal and architecture instantiation.

A Additional details about language components

B Search space example

  • The recurrent-cell search space from [23] is encoded using reusable functions for the cell, input nodes, intermediate nodes, and overall search space.A substitution module determines each node's input connection, while subsearch spaces determine the operations performed by nodes.Source: B Search space example · p. 14, Figure 16 · p. 15

C Additional details about language mechanics

  • Architecture instantiation traverses modules in topological order, applying each module's local computation from its inputs and hyperparameters until unconnected outputs are produced.The ordered traversal begins at output modules and works backward through connections to produce a unique module ordering.Source: C Additional details about language… · pp. 14–15

D.1 Infinite search spaces

Lazy substitution modules can encode infinite search spaces through recursion, stopping when a hyperparameter selects the terminating branch. The language also composes and transforms search spaces, including changing repetition structure and sharing hyperparameters across repetitions.

D.1 Infinite search spaces

  • A substitution module can recursively create another substitution module and hyperparameter, producing an infinite search space until the terminating choice stops recursion.The resulting construction can be composed into other search spaces, and the same principle applies to more complex recursive spaces.Source: §D.1 · p. 15, Figure 18 · p. 15

D.2 Search space transformation and combination

  • Changing the nesting of repetition and choice changes the search space, while instantiating a hyperparameter outside repetitions lets those repetitions share its value.The resulting search spaces can be encapsulated and composed into a new search space.Source: §D.2 · p. 16, Figure 19 · p. 17

E Implementation details

The implementation extends a shared Module class to support basic and substitution modules, then adds domain-specific wrappers for basic modules. Search algorithms use a common sample/update interface, while graph traversal instantiates terminal architectures.

E Implementation details

  • The implementation details section describes extending Module for new domains and implementing search algorithms in Python.It points readers to the implementation's code and documentation.Source: E Implementation details · p. 17, §E.1 · p. 17

E.1 Supporting new domains

  • Module provides shared registration and access functions, while basic modules use computation and value-transfer methods and substitution modules use _update._update checks hyperparameter assignments and invokes a substitution function to replace the module with a search space.Source: §E.1 · pp. 17, 19, Figure 20 · p. 18
  • Basic modules map architectures to framework implementations through _compile and _forward, evaluated in topological order to produce graph outputs.The implementation iterates over the graph and evaluates each module's forward call in turn.Source: §E.1 · p. 17, Figure 22 · p. 20
  • get_io retrieves dictionaries mapping local names to inputs and outputs; single ports typically use in/out and multiple ports use indexed names.A module's scope and name prefix are used to generate unique module and hyperparameter names.Source: §E.1 · p. 19
  • KerasModule and helper functions wrap Keras operations, with single-input/single-output and function-based helpers reducing setup effort.The general keras_module accepts input and output names, hyperparameters, and a compilation function.Source: §E.1 · p. 19, Figure 21 · p. 20

E.2 Implementing a search algorithm

  • Searchers implement sample and update: sample assigns independent hyperparameters to produce a terminal architecture, and update receives its evaluation results.For random search, update is a no-op; the interface can also support other algorithms.Source: §E.2 · p. 19

F Additional experimental results

The additional results report validation and test outcomes for search-space and search-algorithm experiments. The selected architectures were evaluated after grid searches over learning rate and L2 penalty.

F Additional experimental results

  • The grid search tested six learning rates and three L2 penalties, training each evaluation for 600 epochs with momentum SGD and a cosine learning-rate schedule.The learning rates were 0.1, 0.05, 0.025, 0.01, 0.005, and 0.001; L2 penalties were 0.0001, 0.0003, and 0.0005.Source: F Additional experimental results · p. 21
  • Table 3 reports results for the search-space experiments after grid search on each search phase's best architecture.Each grid-search evaluation was trained for 600 epochs.Source: Table 3 · p. 21, F Additional experimental results · p. 21
  • Table 4 reports results for the search-algorithm experiments after grid search on each search phase's best architecture.Each grid-search evaluation was trained for 600 epochs.Source: F Additional experimental results · p. 21, Table 4 · p. 22
Loading 1909.13404v1…