PaperlayerView exact revision on arXiv

Source-linked AI summary

Automated Machine Learning: From Principles to Practices

Zhenqian Shen, Yongqi Zhang, Lanning Wei, Huan Zhao, Quanming Yao

arXiv:1810.13306v5cs.AIcs.LGstat.ML

TL;DR

Configuring increasingly complex machine-learning methods requires substantial human effort, motivating AutoML. This survey formalizes AutoML, analyzes its principles, and organizes its practices around search space, search algorithm, and evaluation strategy. It concludes with applications and emerging directions while noting unresolved theoretical and computational challenges.

  • Problem

    Selecting hyper-parameters and model structures for desirable performance is increasingly difficult and requires substantial human effort and domain expertise.

  • Method

    The survey defines AutoML, formulates its bi-level objective, presents a general learning strategy and theoretical interpretation, and taxonomizes methods by three main factors.

  • Results

    The survey synthesizes representative AutoML methods across search spaces, search algorithms, and evaluation strategies, alongside applications and emerging directions.

  • Takeaways & Limitations

    AutoML offers a systematic framework for automating learning-configuration selection and understanding the trade-offs among search-space size, optimization, and computation.

  • Takeaways & Limitations

    The convergence speed of AutoML optimizers and how quickly they identify good configurations remain unclear, while configuration evaluations are expensive.

Abstract

from arXiv · show

Machine learning (ML) methods have been developing rapidly, but configuring and selecting proper methods to achieve a desired performance is increasingly difficult and tedious. To address this challenge, automated machine learning (AutoML) has emerged, which aims to generate satisfactory ML configurations for given tasks in a data-driven way. In this paper, we provide a comprehensive survey on this topic. We begin with the formal definition of AutoML and then introduce its principles, including the bi-level learning objective, the learning strategy, and the theoretical interpretation. Then, we summarize the AutoML practices by setting up the taxonomy of existing works based on three main factors: the search space, the search algorithm, and the evaluation strategy. Each category is also explained with the representative methods. Then, we illustrate the principles and practices with exemplary applications from configuring ML pipeline, one-shot neural architecture search, and integration with foundation models. Finally, we highlight the emerging directions of AutoML and conclude the survey.

1 INTRODUCTION

Machine-learning methods have become increasingly capable, but configuring models and hyper-parameters to meet task constraints remains laborious and expertise-intensive. AutoML addresses this by atomizing and recombining learning configurations, and this survey organizes its principles, practices, applications, and emerging directions.

  • Motivation: Growing model complexity makes selecting hyper-parameters and structures necessary for desirable performance increasingly difficult and labor-intensive.The challenge requires substantial human effort and domain expertise.
  • Motivation: AutoML atomizes candidate learning configurations into a search space and recombines them to generate satisfactory solutions under task constraints.The constraints include inference time and computational cost limitations.
  • Survey scope: The survey formalizes AutoML and develops its principles through a bi-level objective, learning strategy, and theoretical interpretation.Its theoretical analysis addresses performance improvement and associated trade-offs.
  • Survey scope: The survey also discusses exemplary applications and emerging directions, while emphasizing principle analysis beyond surveys focused on NAS or HPO.The comparison with related surveys is framed around formalization, bi-level optimization, and theoretical analysis.
  • Survey scope: It summarizes existing methods using a taxonomy based on search space, search algorithm, and evaluation strategy.Representative methods are introduced within each category.

2 PRINCIPLES

The paper structures AutoML principles around the problem definition, its universal learning objective and strategy, theoretical interpretation, and taxonomy of existing approaches. It also situates related AutoML topics within this framework.

  • Section organization: The principles section defines AutoML, presents its learning objective and strategy, interprets it theoretically, and organizes existing work by search space, algorithm, and evaluator.Relevant AutoML topics are discussed after the core principles and taxonomy.

2.1 Definition: from ML to AutoML

Classical machine learning requires human choices across complex design spaces, whereas AutoML formalizes atomization and optimization-based recombination of learning configurations. This expands available choices while reducing manual exploration.

  • From ML to AutoML: Machine learning is defined as improving performance through experience on a task, measured by a performance criterion.The formal definition uses experience E, task T, and performance measure P.
  • Comparison: The paper compares classical machine learning and AutoML through application examples to clarify their differing solution-design processes.The comparison is presented in Table 1.
  • From ML to AutoML: Classical machine-learning design involves complex choices across hyper-parameters, features, and architectures, making the optimum difficult to obtain manually.Human-designed methods cover only scattered points in the potential design space.
  • AutoML definition: AutoML atomizes all or part of learning configurations and recombines them by optimizing performance on experience for specified tasks.This is the paper’s formal definition of AutoML.
  • AutoML definition: Atomization provides a broader design space, while optimization-based recombination reduces human labor in exploring domain-specific prior knowledge.The paper presents these as consequences of the AutoML definition.

2.2 Learning Objective and Strategy

AutoML represents learning solutions through atomized configurations, constructs candidate methods from a search space, and optimizes their evaluated performance in a bi-level process. Its design therefore depends on search space, search algorithm, and evaluation strategy.

  • Learning strategy: AutoML samples a configuration α from search space A to construct a recombined learning method C_α.The configuration consists of atomized components of classical machine-learning solutions.
  • Learning objective: The bi-level objective separates optimization of model parameters from optimization of the learning configuration.Training and validation datasets, loss, performance metric, and constraints define the associated objectives.
  • Learning strategy: The general strategy trains each constructed method for optimized parameters w*(α) and evaluates it on validation data.Figure 2 represents atomized configurations as LEGO blocks and complete solutions as assembled shapes.
  • Three factors: Search space determines configuration ranges, search algorithm optimizes α, and evaluation obtains the validation metric for the trained method.These are the three main factors used to organize AutoML practices.

2.3 Theoretical Interpretation

AutoML’s total error decomposes into approximation, optimization, and estimation components, linking performance to search-space design, search effort, and evaluation cost. Compared with classical machine learning, AutoML can reduce approximation error through a larger hypothesis space but introduces optimization error.

  • Error decomposition: AutoML total error comprises approximation, optimization, and estimation errors.Approximation concerns representational closeness, optimization concerns finding the best function in the searched space, and estimation concerns empirical versus expected risk.
  • Search-space effects: Larger search spaces lower approximation error but increase optimization difficulty and computation time.Expanding the hypothesis space makes the optimum harder to find.
  • Search and evaluation effects: More search iterations or higher per-iteration evaluation cost lower optimization error while increasing computation time.These changes improve optimization but impose additional computational expense.
  • Classical versus automated learning: Classical machine learning is a special case of AutoML’s decomposition, with a smaller hypothesis space and relatively larger approximation error.AutoML adds optimization error by searching for a good learning configuration; estimation error is similar in both settings.
  • Design trade-off: AutoML design therefore balances better performance against computation cost.The trade-off follows from enlarging the search space, increasing search iterations, or using more expensive evaluation.

2.4 Taxonomies of AutoML Approaches

The survey organizes AutoML approaches around search space, search algorithm, and evaluation strategy. These factors respectively determine candidate configurations, recombination and optimization, and the cost–accuracy trade-off of evaluating candidates.

  • Three factors: AutoML methods are categorized by search space, search algorithm, and evaluation strategy.The search algorithm selects configurations based on evaluation results from the defined search space.
  • Search space: Search-space design balances sufficient breadth for lower approximation error with compactness for lower computation and optimization error.The survey further divides spaces into general, structured, and transformed forms.
  • Search algorithm: Search algorithms differ in search cost and result quality, with taxonomies based on candidate-generalization methods.The listed categories include grid and random search, evolutionary algorithms, gradient-based methods, and Bayesian optimization.
  • Evaluation strategy: Evaluation strategies reduce costly lower-level optimization while balancing computational efficiency against evaluation accuracy.Categories include reduced data, learning-curve monitoring, parameter reuse, performance prediction, and bandit methods.

2.5 Relevant Topics

The survey situates AutoML alongside related machine learning topics and clarifies their relationships and differences. It identifies bi-level optimization as a broader formulation that can encompass AutoML and other learning problems.

  • Scope and relationships: The section clarifies the relatedness and differences between AutoML and several machine learning topics.It frames these topics as neighboring concepts rather than identical formulations.
  • Bi-level optimization: Bi-level optimization embeds one optimization problem within another and can formulate meta-learning, adversarial learning, deep reinforcement learning, and AutoML.AutoML problems can generally be regarded as bi-level optimization problems.

3 SEARCH SPACE

AutoML begins by atomizing machine-learning solutions into search spaces that may remain general, become structurally compact, or be transformed for easier optimization. The survey describes structured representations and transformations that trade expressiveness, search efficiency, and reliance on prior knowledge.

  • Search-space design: A useful AutoML search space must be expressive enough for generalization while compact enough to ease optimization.The space contains candidate learning configurations for the target solutions.
  • Search-space processing: Search spaces progress from general space to structured space and transformed space.Structuring reduces size, while transformation can provide properties such as continuity or differentiability.
  • General space: General spaces commonly represent pipeline configuration choices or neural-network operations and connections.They may use discrete or continuous Cartesian-product choices such as learning rates, batch sizes, and optimizers.
  • Structured space: Structured spaces include directed acyclic graphs, cells, 2/3D meshes, and hierarchies.DAGs represent node contents and connections; cells share repeated structures; meshes encode state transitions; hierarchies encode dependent choices.
  • Softmax transformation: Softmax transformation maps discrete operation choices to continuous weights that can be trained jointly in supernet methods.DARTS uses operation weights α_o to form a weighted combination of candidate operations.
  • Sparse coding: Sparse coding compresses configurations into representations based on key components identified using prior knowledge.NASP removes redundant connections and imposes sparsity based on the assumption that one operation per module is selected in the optimized model.
  • Autoencoder transformation: Autoencoders transform complex search spaces into continuous spaces to make optimization easier.The framework uses an encoder for transformation and a decoder for reconstruction.

4 SEARCH ALGORITHM

AutoML search algorithms generate and evaluate candidate learning configurations within a search space, balancing search efficiency against solution quality. The survey covers grid and random search, evolutionary, gradient-based, Bayesian optimization, and reinforcement-learning approaches.

  • Search algorithm overview: Search algorithms seek high-performing learning configurations while trading off search efficiency against convergence quality.Efficient methods may not guarantee search-result quality, whereas methods with good convergence properties are often time-consuming.
  • Grid and random search: Grid search exhaustively evaluates grid-like configurations, whereas random search samples configurations randomly and can explore important dimensions more effectively.The survey reports that random search has been empirically shown to outperform grid search in such settings.
  • Evolutionary algorithms: Evolutionary algorithms maintain a population of candidate configurations and update it through selection, crossover, and related evolutionary operations.These methods draw inspiration from natural evolution and repeatedly modify the candidate set.
  • Gradient-based methods: Gradient-based methods generate configurations by following the gradient of validation performance with respect to learning-configuration parameters.Practical strategies include iterative differentiation and approximate implicit differentiation.
  • Bayesian optimization: Bayesian optimization models configuration performance with a probabilistic surrogate and uses an acquisition function to select the next configuration.The surrogate and acquisition function are adapted as new evaluation results become available, balancing exploration and exploitation.
  • Reinforcement learning: AutoML search can be formulated as reinforcement learning because configurations are explored through trial and error using evaluation results as feedback.A controller typically samples candidate learning configurations in this framework.

5 EVALUATION STRATEGY

AutoML evaluation strategies reduce the cost of measuring configuration performance while managing the trade-off between speed and accuracy. The survey organizes these strategies around reducing data, monitoring learning curves, reusing parameters, predicting performance, and allocating evaluation resources.

  • Evaluation overview: Evaluation strategies trade off the accuracy of training configurations to convergence against the speed of returning performance estimates.Accurate evaluation is often too time-consuming, while faster methods can be inaccurate.
  • Reducing data quantity: Reducing dataset quantity accelerates training by decreasing either the number of data samples or the sample size and feature dimension.Methods include sampling dataset subsets and generating synthetic data.
  • Monitoring learning curves: Learning-curve monitoring uses early stopping or partial-curve prediction to reduce evaluation cost.Early stopping terminates underperforming training processes, while learning-curve exploration predicts performance from partial curves.
  • Parameter reusing: Parameter reusing accelerates evaluation by initializing models from previously trained parameters or by reusing parameters from a trained supernet.Parameter warming-up can accelerate convergence, while supernet training supports evaluation of multiple neural architectures.
  • Performance prediction: Performance predictors replace full performance measurement with surrogate models that preprocess configuration features and predict their performance.The prediction pipeline avoids conducting the full training procedure for every configuration.
  • Resource allocation: Bandit-based methods allocate limited evaluation resources across configurations and progressively eliminate less promising candidates.SuccessiveHalving initially distributes resources broadly, then repeatedly removes half of the less promising training processes.

6 EXEMPLARY APPLICATIONS

AutoML applications automate ML pipeline configuration and neural architecture search by exploring structured configuration spaces. These approaches reduce search time or human intervention, but efficiency can involve performance trade-offs.

  • 6.1 Automated Configuring Machine Learning (ML) Pipeline: AutoML treats ML pipeline configuration as a search problem spanning feature engineering, model selection, hyper-parameter optimization, and evaluation.The CASH formulation jointly selects models and tunes their hyper-parameters to minimize validation loss.
  • 6.1 Automated Configuring Machine Learning (ML) Pipeline: The CASH search space combines candidate models with their corresponding hyper-parameters, creating a difficult optimization problem involving discrete variables and costly model training.Validation-loss evaluation may require repeatedly training models and updating their parameters.
  • 6.1 Automated Configuring Machine Learning (ML) Pipeline: Auto-WEKA, Auto-Sklearn, and TPOT represent complementary pipeline-configuration strategies using Bayesian optimization, meta-learning and bandit methods, and evolutionary search.Their methods also include reduced-data evaluation, warm starts, ensemble selection, and budget allocation.
  • 6.1 Automated Configuring Machine Learning (ML) Pipeline: Under similar time constraints, Auto-Sklearn and TPOT noticeably outperform Random Search on several benchmark datasets, especially Digits and Madeline.Random Search selects 100 randomly sampled models and hyper-parameter settings from DSM, retaining the best result.
  • 6.2 One-shot NAS: One-shot NAS trains a supernet once, then evaluates sampled sub-networks by inheriting shared weights instead of independently training every architecture.The supernet is represented as a directed acyclic graph whose nodes denote latent representations and whose edges denote operations.
  • 6.2 One-shot NAS: DARTS makes architecture choices differentiable through softmax relaxation and jointly optimizes architecture parameters and supernet weights.Its procedure alternates outer validation-loss updates for architecture parameters with inner training-loss updates for model weights.
  • 6.2 One-shot NAS: DARTS reduces vanilla NAS time, while few-shot DARTS achieves the lowest error with time cost closest to DARTS; zero-shot methods reduce time further but sacrifice performance.Weight sharing assumes one-shot rankings correlate with independently trained rankings, but coupling can cause rank disorder and operational biases.
  • 6.3 Foundation Models: AutoML can be applied across all stages of the foundation-model lifecycle, optimizing each stage’s subject and associated hyper-parameters for its objective.The survey also presents a technical roadmap for NAS and illustrates AutoML’s broader pipeline involvement.

7 EMERGING DIRECTIONS

Emerging AutoML research extends beyond classical supervised learning toward new problem settings, more efficient search techniques, stronger theory, and scientific applications. These directions remain constrained by search-space coverage, data quality, computational cost, and incomplete generalization theory.

  • 7.1 Problem setup: For few-shot, positive-unlabelled, and foundation-model problems, AutoML must automate task-specific algorithm or objective selection beyond classical supervised-learning settings.Examples include selecting meta-learning algorithms, positive-prior and label-propagation methods, and pre-training self-learning tasks.
  • 7.2 Techniques: Efficiently finding strong learning configurations in complex search spaces is a central AutoML challenge, motivating search-space reduction and other emerging techniques.Meta-learning can transfer information to reduce search complexity, but these approaches introduce new research questions.
  • 7.2 Techniques: Reducing the search space may overlook potential models, while performance predictors depend on the quality and representativeness of their training data.These issues require further investigation and experimentation.
  • 7.3 Theory: AutoML theory still needs convergence-rate analysis for identifying good configurations and more comprehensive quantitative analysis of generalization ability.Configuration evaluation is expensive because it usually requires model training.
  • 7.4 Applications: AutoML is also being applied to scientific research, including biomedical tasks such as molecular activity prediction and drug-formulation optimization.The survey frames these applications as ways to expedite research, enhance precision, and uncover insights beyond human analytical capacity.

8 CONCLUSION

AutoML is presented as a systematic effort to automate learning-configuration decisions and survey the field from principles through practices and future directions.

  • AutoML aims to substitute human involvement in machine learning by automatically determining configurations for given tasks.
  • The survey formulates AutoML as a bi-level optimization problem, proposes a general learning strategy, and provides a theoretical interpretation.
  • Existing AutoML approaches are categorized by search space, search algorithm, and evaluation strategy, with representative approaches under each category.
  • The survey concludes by discussing exemplary applications and emerging directions of AutoML.
Loading 1810.13306v5…