PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoML: A Survey of the State-of-the-Art

Xin He, Kaiyong Zhao, Xiaowen Chu

arXiv:1908.00709v6cs.LGcs.CVstat.ML

TL;DR

Building high-quality deep-learning systems depends heavily on human expertise, motivating automated machine learning. This paper reviews AutoML across the full pipeline, emphasizes NAS, compares representative NAS algorithms, and discusses important NAS directions and open problems. Its synthesis presents AutoML as a broad field whose methods and interpretability, evaluation, and optimization challenges remain active research topics.

  • Problem

    Building high-quality deep-learning systems requires substantial expert time and resources, while AutoML methods and their broader pipeline coverage need systematic review.

  • Method

    The paper systematically reviews AutoML methods from data preparation through model evaluation, with focused discussion of NAS algorithms, research directions, and open problems.

  • Results

    The review compares NAS performance and efficiency on CIFAR-10 and ImageNet and synthesizes one/two-stage NAS, one-shot NAS, and joint hyperparameter–architecture optimization.

  • Takeaways & Limitations

    The survey provides a comprehensive introduction to AutoML while identifying evaluation-efficiency trade-offs, joint optimization, reproducibility, and interpretability as continuing concerns.

  • Takeaways & Limitations

    NAS comparisons are not fully fair because methods use different training-related hyperparameters and data-augmentation settings.

Abstract

from arXiv · show

Deep learning (DL) techniques have penetrated all aspects of our lives and brought us great convenience. However, building a high-quality DL system for a specific task highly relies on human expertise, hindering the applications of DL to more areas. Automated machine learning (AutoML) becomes a promising solution to build a DL system without human assistance, and a growing number of researchers focus on AutoML. In this paper, we provide a comprehensive and up-to-date review of the state-of-the-art (SOTA) in AutoML. First, we introduce AutoML methods according to the pipeline, covering data preparation, feature engineering, hyperparameter optimization, and neural architecture search (NAS). We focus more on NAS, as it is currently very hot sub-topic of AutoML. We summarize the performance of the representative NAS algorithms on the CIFAR-10 and ImageNet datasets and further discuss several worthy studying directions of NAS methods: one/two-stage NAS, one-shot NAS, and joint hyperparameter and architecture optimization. Finally, we discuss some open problems of the existing AutoML methods for future research.

1. Introduction

The paper motivates AutoML as a way to reduce the expertise, time, and resources required to build high-quality deep-learning systems. It surveys representative methods across the complete AutoML pipeline while emphasizing NAS and related research directions.

  • Motivation: Expert-designed deep-learning models require substantial resources and time because they are created through trial and error.VGG-16 is cited as an example with more than 130 million parameters, nearly 500 MB of memory, and 15.3 billion floating-point operations per 224 × 224 image.
  • Motivation: AutoML aims to automate machine-learning pipeline construction and reduce the demand for specialized data-science and machine-learning expertise.The paper describes AutoML as automated construction of an ML pipeline under a limited computational budget.
  • AutoML scope: The AutoML pipeline covers data preparation, feature engineering, model generation, and model evaluation.Model generation includes search-space design and optimization methods for hyperparameters and architectures.
  • NAS: NAS searches for robust, well-performing neural architectures by selecting and combining operations from a predefined search space.The survey notes that NAS research has grown rapidly and classifies commonly used search spaces and optimization approaches.
  • Contribution: Unlike earlier surveys focused mainly on NAS or covering little NAS, this paper reviews representative methods across the complete AutoML pipeline.The authors prioritize breadth over exhaustive treatment of every AutoML sub-topic.

2. Data Preparation

Data preparation is presented as the first AutoML stage, covering collection, cleaning, and augmentation. The survey describes challenges in obtaining suitable data and methods for filtering, labeling, synthesizing, and automatically augmenting datasets.

  • Overview: Data preparation comprises data collection, data cleaning, and data augmentation.Collection builds or extends datasets, cleaning filters noisy data, and augmentation can improve robustness and performance.
  • Data collection: Finding suitable datasets is difficult for specialized or privacy-sensitive tasks, motivating data-searching and data-synthesis methods.The paper gives medical-care tasks as an example where standard dataset-search approaches may be inadequate.
  • Data searching: Web-data collection requires filtering irrelevant results and addressing incorrect or missing labels.Active learning, self-labeling, and multilabel assignment are described as approaches for improving Web-data usability.
  • Data cleaning: Data cleaning removes noise that can negatively affect model training, but continuous cleaning remains an open problem for continuously generated data.The survey describes automation and workflow orchestration as responses to specialist scarcity and repeated cleaning demands.
  • Data augmentation: Data augmentation generates new examples, acts as a regularizer, and includes image, audio, and text transformations.The survey covers affine, elastic, advanced, neural-based, textual, and automatic policy-search techniques.
  • Data augmentation: Automatic augmentation-policy search reduces manual operation selection but can be computationally expensive.AutoAugment is cited as requiring almost 500 GPU hours for one augmentation search.

3. Feature Engineering

Feature engineering transforms raw or basic features to improve their usefulness for machine-learning models. The survey organizes it into feature selection, extraction, and construction, emphasizing automated search and evaluation of feature transformations.

  • Overview: Feature engineering consists of feature selection, feature extraction, and feature construction.Extraction and construction create new feature sets, whereas selection reduces irrelevant or redundant original features.
  • Feature selection: Feature selection builds a subset of original features to simplify models, avoid overfitting, and improve performance.The process repeatedly searches for and evaluates subsets before validation determines whether a subset is acceptable.
  • Feature selection: Feature-selection search strategies include complete, heuristic, and random search.Examples include breadth-first, branch-and-bound, beam, best-first, sequential forward, sequential backward, and bidirectional search.
  • Feature selection: Feature-subset evaluation methods include filter, wrapper, and embedded methods.Filter methods score features using criteria such as variance, correlation, chi-square, or mutual information.
  • Feature construction: Feature construction creates new features from raw or basic features to increase representational ability, and automatic methods search and evaluate operation combinations.Decision-tree-based methods and genetic algorithms are examples of automated construction approaches.
  • Feature extraction: Feature extraction reduces dimensionality through mapping functions that produce informative and non-redundant features.The survey discusses PCA, ICA, isomap, nonlinear dimensionality reduction, and autoencoder-tree methods.

4. Model Generation

Model generation is divided into defining a model search space and optimizing within it. For NAS, the survey distinguishes search space, architecture optimization, and model evaluation, with evaluation requiring a balance between fidelity and computational cost.

  • Model generation: Model generation comprises search-space design and optimization methods.The search space specifies model structures that can be designed, while optimization methods determine how parameters are selected.
  • Search space: The model search space includes traditional machine-learning models such as SVM and KNN and deep neural networks.The survey treats these as the two broad model categories considered in model generation.
  • Optimization: Optimization methods distinguish training-related hyperparameters from model-related parameters.Learning rate and batch size illustrate hyperparameters, while neural-network layer count and KNN neighbor count illustrate model parameters.
  • Neural architecture search: NAS is organized around search space, architecture optimization, and model evaluation.The architecture-optimization method guides the search toward high-performing architectures after the search space is defined.
  • Model evaluation: Model evaluation can train a generated model to convergence and estimate performance on a validation set, but this is time-consuming and resource-intensive.Faster evaluation methods may improve efficiency at the cost of fidelity, making the efficiency-effectiveness trade-off an open issue.

4.1. Search Space

NAS search spaces define how neural architectures are represented and explored. The survey contrasts entire-structured and cell-based spaces, highlighting transferability, search complexity, hierarchical design, and the two-stage gap.

  • General Representation: A neural architecture is represented as a DAG whose nodes are feature tensors and whose edges specify operations selected from a candidate set.Candidate operations include convolution, pooling, activation functions, skip connections, concatenation, and addition.
  • Entire-structured Search Space: Entire-structured spaces stack ordered nodes as layers, optionally allowing skip connections, but deep-network search is computationally expensive and architectures transfer poorly across dataset scales.A model generated on a small dataset may require regeneration for a larger dataset.
  • Cell-based Search Space: Cell-based spaces assemble a fixed number of repeating normal and reduction cells, simplifying full-network search and enabling transferability across model variants.Normal cells preserve spatial dimensions, whereas reduction cells halve feature-map width and height.
  • Cell-based Search Space: Cell internals are built from ordered blocks whose nodes choose inputs and operations, then combine their outputs through addition or concatenation.Inputs may come from the previous two cells and earlier blocks within the same cell; 1×1 convolutions can align channel counts.
  • Search-space Complexity: With M = 5, L = 10, and B = 3, the entire-structured space contains 3.44 × 10^20 architectures versus 5.06 × 10^16 for the cell-based space.Entire-structure search complexity grows exponentially with the number of layers.
  • Two-stage Gap: Cell-based NAS has a two-stage gap because search often uses shallower networks than evaluation, and simply adding cells can deteriorate performance.DARTS searches with eight cells but evaluates with 20; P-DARTS progressively increases depth during search to bridge this gap.

4.2. Architecture Optimization

Architecture optimization automates the search for neural network designs that would otherwise require costly expert trial and error. The section surveys evolutionary, reinforcement-learning, gradient-based, and search-based approaches, along with their representations, procedures, efficiency gains, and limitations.

  • Architecture Optimization: Architecture optimization searches for high-performing neural architectures automatically, reducing reliance on expert-driven trial and error.The process treats architecture design as a search problem over model-related parameters.
  • Evolutionary Algorithm: Evolutionary algorithms represent networks through direct or indirect encodings and iteratively apply selection, crossover, mutation, and population updates.Direct encodings specify network structures explicitly, whereas indirect encodings specify generation rules.
  • Gradient Descent: DARTS relaxed discrete architecture choices into a continuous differentiable space, then retained the maximum-weight operation on each edge.Its architecture and weights are optimized jointly through a bilevel formulation, although that formulation is difficult to solve directly because both parameter sets are high dimensional.
  • Comparative Findings: Mixed-level optimization was reported to address single-level overfitting while avoiding bilevel gradient error, whereas random search remained competitive with leading NAS methods.Early-stopped random search performed as well as ENAS in the cited comparison, and another study found SOTA NAS techniques not significantly better than random search.
  • Limitations: The surveyed methods retain important trade-offs: evolutionary methods require substantial computation, while reinforcement-learning controllers may lack guaranteed search efficiency and stability.The evolutionary operators are also described as random, and RL controllers may require several iterations to learn effective architectural patterns.

4.3. Hyperparameter Optimization

Hyperparameter optimization methods search training-related parameters under different efficiency and resource trade-offs. The section contrasts grid, random, Bayesian, and gradient-based approaches, emphasizing surrogate modeling and budget-aware search.

  • Grid and Random Search: Random search explores more distinct values along important dimensions than grid search in the same number of trials.In the illustrated nine-trial example, grid search covers three important parameter values, whereas random search explores nine.
  • Grid and Random Search: Grid search is simple and parallelizable but becomes computationally expensive as hyperparameter dimensionality increases.The number of trials grows exponentially with the dimensionality of the hyperparameter space.
  • Grid and Random Search: Random search is more practical and efficient than grid search but does not guarantee the optimum and consumes more resources as search length increases.Hyperband addresses this trade-off by allocating limited resources across configurations.
  • Bayesian Optimization: Bayesian optimization builds a probabilistic mapping from hyperparameters to validation metrics and balances exploration with exploitation.It is presented as an efficient method for global optimization of expensive black-box functions.
  • Bayesian Optimization: Sequential model-based optimization fits a surrogate, selects the next configuration with an acquisition function, evaluates it, and updates the record dataset.The loop repeats for a resource-dependent number of iterations T.
  • Gradient-Based Optimization: Gradient-based optimization uses gradient information to improve hyperparameter-search efficiency, although optimizing many hyperparameters remains computationally challenging.A reversible-dynamics memory-tape approach is described for handling thousands of hyperparameters.

5. Model Evaluation

Model evaluation is a major computational bottleneck in neural architecture search. The surveyed methods reduce this cost through lower-fidelity evaluation, weight reuse, surrogate prediction, and early stopping.

  • Evaluation Bottleneck: Training each candidate network to convergence can require extensive time and computing resources.One cited NAS search used 800 K40 GPUs for 28 days, while NASNet and AmoebaNet used hundreds of GPUs.
  • Low Fidelity: Lower-fidelity evaluation accelerates model assessment by reducing image quantity, image resolution, or dataset size.FABOLAS trains models on a subset of the training set.
  • Low Fidelity: Transfer Series Expansion combines multiple low-fidelity estimators to avoid bias from relying on a single low-fidelity estimator.The method linearly combines a series of basic estimators.
  • Weight Sharing: Weight sharing and inherited weights accelerate NAS by reusing information across candidate architectures.ENAS shares parameters among child networks and is reported as a thousand-fold faster than the cited earlier method.
  • Surrogate Models: Surrogate models reduce evaluations by approximating the expensive black-box objective and guiding the search toward promising configurations.PNAS is reported as eight times faster in total computational speed than ENAS, while SemiNAS uses unlabeled architectures to train its surrogate.
  • Early Stopping: Early stopping accelerates evaluation by terminating candidates predicted to perform poorly on validation data or using local gradient statistics.The latter approach no longer relies on the validation set and can use all training data.

6. NAS Discussion

The survey compares NAS methods globally using accuracy and algorithm efficiency, while recognizing that architecture variants target different performance, size, and deployment scenarios.

  • Global Comparison: The survey compares NAS development using representative algorithm results rather than exhaustively reporting every architecture variant.The comparison is intended to provide a clearer global view of NAS methods.
  • Global Comparison: Accuracy and algorithm efficiency are the comparison indices used to support valid comparisons across NAS studies.The surveyed variants may differ in performance, model size, and suitability for mobile devices.

GPU Days #GPUs AO

The supplied passages identify a CIFAR-10 NAS performance table and explain its architecture-optimization labels, but they do not provide enough table content to summarize GPU days or compare algorithms quantitatively.

  • Table Scope: The provided table excerpt does not state GPU days, total GPU counts, or a quantitative AO comparison.No numerical comparison beyond the table’s described scope is supported by the supplied passages.
  • Table Scope: Table 3 reports representative NAS performance on CIFAR-10 and identifies the architecture-optimization method in its AO column.The supplied table passage explains the abbreviations RL, EA, GD, RS, and SMBO.

Days #GPUs AO

The survey compares NAS methods by performance and search efficiency, then explains two-stage, one-stage, and one-shot search flows. It also highlights reproducibility concerns caused by inconsistent evaluation settings.

  • Days #GPUs AO: Tables 3 and 4 compare NAS performance on CIFAR-10 and ImageNet, using GPU Days to approximate search efficiency.GPU Days account for differing numbers of GPUs and search durations.
  • Days #GPUs AO: 3,150 GPU days and 450 GPUs were required by AmoebaNet, illustrating the resource costs of early high-performance NAS.ENAS reduced its GPU budget to 1 and shortened searching to less than one day through parameter sharing.
  • Reproducibility: Comparisons across NAS methods are not fully fair because training hyperparameters and data augmentation settings differ between studies.NAS-Bench datasets address this by fixing search spaces and providing tabulated evaluation results.
  • One-stage NAS: One-stage NAS performs architecture optimization and parameter training simultaneously, exporting a trained architecture without extra retraining.This improves efficiency but is difficult because architecture and weight parameters are highly coupled.
  • One-shot NAS: One-shot NAS embeds the search space in an overparameterized supernet whose candidate architectures share weights, distinguishing it from the one-stage classification.Most one-stage NAS methods are based on the one-shot paradigm, but one-shot and one-stage are not equivalent.
  • One-shot NAS: Multimodel forgetting occurs when continuously sampled architectures negatively affect previously trained architectures, degrading their performance.A constrained optimization method applied to RandomNAS and GDAS reduced multimodel forgetting and improved the supernet’s predictive ability as an evaluator.

6.4. Joint Hyperparameter and Architecture Optimization

The survey motivates joint hyperparameter and architecture optimization because fixed training settings can unfairly rank architectures. It reviews methods that combine architecture and training-hyperparameter search.

  • Motivation: Fixed training hyperparameters may produce suboptimal results because different architectures fit different hyperparameters, making model rankings unfair.Joint hyperparameter and architecture optimization is proposed as a solution.
  • Methods: BOHB jointly optimizes NAS and standard hyperparameters by casting NAS as a hyperparameter optimization problem with a combined search space.The architecture and hyperparameter spaces are combined before optimization.
  • Methods: AutoHAS uses a differentiable method that constructs a Cartesian product of architecture and hyperparameter search spaces.The survey presents it as another approach to joint optimization.
  • Methods: FBNetv3 jointly searches architectures and their training recipes, representing architectures with categorical and normalized integral variables.An encoder produces architecture embeddings that are combined with training hyperparameters to predict accuracy.
  • Resource-aware optimization: Resource-aware NAS adds computational cost to the loss function as a resource constraint to trade off model performance against budget.Latency-aware methods use differentiable or lookup-table-based cost estimates, with manually set weighting hyperparameters in some formulations.

7. Open Problems and Future Directions

The survey identifies open problems spanning human-biased search spaces, limited NLP performance, interpretability, reproducibility, robustness, complete pipelines, and lifelong learning. It presents these as directions for extending AutoML beyond fixed benchmark settings.

  • Flexible Search Space: Current NAS search spaces rely on human knowledge and experience, introducing human bias into the design of searched architectures.AutoML-Zero explores simple mathematical operations to reduce this bias and discover alternative model designs.
  • Exploring More Areas: NAS models achieve comparable results to manually designed models on CIFAR-10 and ImageNet, but NLP evidence remains concentrated on PTB.On PTB, NAS-designed models still show a large performance gap from human-designed models.
  • Exploring More Areas: AutoML has also been applied to network compression, federated learning, image captioning, recommendation systems, and loss or activation-function search.These applications indicate potential beyond computer vision and language modeling.
  • Interpretability: AutoML often finds promising configurations without explaining scientifically why those configurations perform better, motivating mathematical interpretability.The survey notes that existing explanations can be hindsight-based and lack rigorous mathematical proof.
  • Reproducibility: NAS reproducibility is limited by manually set implementation parameters, omitted random seeds, and substantial resource consumption.NAS-Bench datasets reduce repeated evaluation costs by providing fixed benchmark results.
  • Robustness: NAS also faces real-world data challenges because deployed data can be noisy or adversarial unlike the well-labeled benchmark datasets commonly used in research.Only a few studies have attempted to improve NAS robustness against adversarial data.
  • Complete AutoML Pipeline: Existing AutoML libraries often cover only parts of the pipeline, motivating easy-to-use systems that integrate data augmentation, HPO, NAS, compression, and training.Vega is cited as a tool constructing a more complete pipeline from decoupled functions.
  • Lifelong Learning: Most AutoML algorithms target specific tasks and fixed datasets, whereas lifelong learning requires learning new data while retaining prior knowledge.Meta-learning, few-shot learning, and incremental learning are discussed as approaches to these challenges.

8. Conclusions

The paper reviews AutoML across the deep-learning pipeline, compares NAS methods on CIFAR-10 and ImageNet, and discusses NAS directions and open problems. It aims to provide newcomers with a comprehensive understanding of the field.

  • Conclusions: The survey systematically reviews AutoML studies from data preparation through model evaluation.It organizes methods according to the deep-learning pipeline.
  • Conclusions: It compares NAS performance and efficiency on CIFAR-10 and ImageNet and discusses one/two-stage NAS, one-shot NAS, and joint HAO.The survey also examines open problems and future research directions.
  • Conclusions: The review is intended to provide beginners with a comprehensive and clear understanding of AutoML.The authors position it as support for future research in the area.
Loading 1908.00709v6…