PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoML to Date and Beyond: Challenges and Opportunities

Shubhra Kanti Karmaker Santu, Md. Mahadi Hassan, Micah J. Smith, Lei Xu, ChengXiang Zhai, Kalyan Veeramachaneni

arXiv:2010.10777v4cs.LGcs.AI

TL;DR

AutoML seeks to make machine learning more accessible and efficient, yet key stages still require manual collaboration between domain experts and data scientists. This review maps the end-to-end pipeline, proposes a level-based taxonomy of automation, and uses a case study to expose challenges on the path toward more autonomous data science.

  • Problem

    Current AutoML systems still require substantial human involvement in understanding data, defining prediction tasks, preparing training data, and coordinating expertise.

  • Method

    The paper reviews the end-to-end machine-learning process, classifies AutoML systems by automation level, and develops a roadmap through an exploratory real-world case study.

  • Results

    The review identifies task formulation and result summary and recommendation as major bottlenecks, with prediction engineering also requiring prolonged interaction.

  • Takeaways & Limitations

    Further automation requires addressing task formulation, prediction engineering, useful-task recommendation, and fair model evaluation.

  • Takeaways & Limitations

    Identifying promising prediction tasks remains an open problem because promise has no single right definition and depends on multiple criteria.

Abstract

from arXiv · show

As big data becomes ubiquitous across domains, and more and more stakeholders aspire to make the most of their data, demand for machine learning tools has spurred researchers to explore the possibilities of automated machine learning (AutoML). AutoML tools aim to make machine learning accessible for non-machine learning experts (domain experts), to improve the efficiency of machine learning, and to accelerate machine learning research. But although automation and efficiency are among AutoML's main selling points, the process still requires human involvement at a number of vital steps, including understanding the attributes of domain-specific data, defining prediction problems, creating a suitable training data set, and selecting a promising machine learning technique. These steps often require a prolonged back-and-forth that makes this process inefficient for domain experts and data scientists alike, and keeps so-called AutoML systems from being truly automatic. In this review article, we introduce a new classification system for AutoML systems, using a seven-tiered schematic to distinguish these systems based on their level of autonomy. We begin by describing what an end-to-end machine learning pipeline actually looks like, and which subtasks of the machine learning pipeline have been automated so far. We highlight those subtasks which are still done manually - generally by a data scientist - and explain how this limits domain experts' access to machine learning. Next, we introduce our novel level-based taxonomy for AutoML systems and define each level according to the scope of automation support provided. Finally, we lay out a roadmap for the future, pinpointing the research required to further automate the end-to-end machine learning pipeline and discussing important challenges that stand in the way of this ambitious goal.

1 INTRODUCTION

AutoML emerged to help stakeholders exploit growing volumes of data, but current systems still depend on substantial human work. The paper reviews this end-to-end process and proposes a level-based framework for expanding automation.

  • Motivation: Growing big data and demand for machine learning have increased interest in AutoML for applying ML to real-world problems.Businesses are hiring more data scientists and ML engineers as organizations seek to use data more effectively.
  • Human involvement: Domain experts and data scientists must collaborate because they possess complementary domain and machine-learning expertise.The data scientist translates domain goals into prediction tasks and selects models, while the domain expert supplies contextual knowledge.
  • Human involvement: Understanding domain-specific data, defining prediction problems, and creating training data remain largely manual, ad-hoc steps.Existing AutoML systems generally begin only after these datasets and prediction tasks have been prepared by a data scientist.
  • Motivation: Current systems fall short of the minimal-intervention meaning commonly associated with AutoML and leave domain experts dependent on data scientists for much of the process.The paper identifies efficiency for data scientists and direct ML access for domain experts as northstar goals.
  • Paper scope: The review describes the end-to-end process, identifies automated and manual subtasks, introduces a level-based classification, and proposes a future research roadmap.Its classification distinguishes current and future AutoML solutions according to the subtasks they automate.

2 WHAT DOES A DATA SCIENTIST DO?

The paper formalizes the data scientist’s end-to-end workflow and surveys automation across its subtasks. Existing work covers areas such as data curation, feature engineering, model exploration, tuning, evaluation, and integrated systems, while some steps remain manual.

  • Task Formulation: Task formulation begins with domain interaction, data examination, and conversion of a practical problem into a machine-learning task.The formalized workflow starts from understanding the problem and available data.
  • Prediction Engineering: Prediction engineering constructs labels and meaningful training and testing sets, and is usually performed manually by data scientists or annotators.The resulting examples are collections of <data_point, label> tuples.
  • Feature Engineering: Feature engineering transforms raw data into features that better represent the underlying problem for predictive models.The paper notes that automated systems and libraries increasingly extract features and generate feature matrices.
  • Alternative Models Exploration, Testing and Validation: Automation research has addressed alternative model exploration, hyperparameter tuning, and evaluation, including comparative benchmarking efforts.The surveyed work spans research contributions, open-source systems, and commercial systems.
  • Systems: Integrated machine-learning systems combine practical functions such as distributed training, tuning, model storage, serving, and deployment.The survey also organizes tools by subtask and catalogs related research, open-source, and commercial systems.

3 THE CURRENT FLOW OF END-TO-END MACHINE LEARNING PIPELINES

The current end-to-end pipeline combines automated progress with substantial manual work and communication. Task formulation and result recommendation are the largest communication bottlenecks, while prediction engineering can also require prolonged interaction.

  • Current pipeline: AutoML’s primary goal is to reduce manual effort in machine-learning development and accelerate deployment.The paper uses the end-to-end process to locate where that effort remains concentrated.
  • Current pipeline: Existing systems have advanced automation for data visualization, cleaning, curation, machine learning, feature engineering, and model exploration, testing, and validation.These advances cover several pipeline subtasks but do not eliminate human involvement across the process.
  • Bottlenecks: Task formulation and result summary and recommendation are the biggest communication bottlenecks because both require significant manual work.Prediction engineering can also produce prolonged back-and-forth between domain experts and data scientists.
  • Bottlenecks: The difficult subtasks remain largely unstructured, manually performed through trial and error, and subject to subjective evaluation.The paper links this challenge especially to human interaction and identifying interesting prediction tasks.
  • Access and scope: Current AutoML solutions primarily take a software-systems perspective, while domain experts have minimal access to feature engineering, machine learning, and model exploration.Deployment, maintenance, and monitoring are also handled by data scientists and software engineers.

4 CLASSIFYING AUTOML SYSTEMS BASED ON LEVELS OF AUTOMATION

The taxonomy organizes AutoML systems into seven levels according to which end-to-end machine-learning subtasks they automate, with higher levels requiring less manual work and enabling greater domain-expert access. Levels 5 and 6 extend automation beyond model construction toward prediction engineering, task formulation, and recommendation, but introduce substantial open research challenges.

  • The seven-level taxonomy classifies systems by their degree of automation, with higher levels corresponding to less manual work and greater domain-expert access.The levels organize solutions by automated data-science subtasks rather than assigning absolute capability scores.
  • Level 0 performs all machine-learning tasks manually, whereas Level 1 supplies only basic implementations of core algorithms.At Level 1, training-set construction, labeling, feature construction, and hyperparameter tuning remain manual.
  • Levels 2 and 3 automate learning and hyperparameter tuning when some inputs are prepared, but still leave substantial work to data scientists and provide little or no direct domain-expert access.Level 2 assumes an available training set and defined features; Level 3 jointly automates learning and tuning.
  • 4.6 Level 5 - ML + ATV + FE + PE automated: Level 5 adds automatic prediction engineering, enabling systems to help create high-quality training sets while domain experts interact comfortably and data-scientist workload is minimal.Prediction engineering distinguishes Level 5 from lower levels, but requires resolving missing information in natural-language problem descriptions.
  • 4.7 Level 6 - ML + ATV + FE + PE + TF + RSR automated: Level 6 automates the pipeline from task formulation through result summary and recommendation, allowing domain experts to use the system independently with minimal manual work.Its defining functionalities are automatic prediction-task formulation and recommendation, both of which raise open evaluation and design questions.
  • Most existing AutoML solutions focus on hyperparameter tuning and feature engineering, while the broader taxonomy emphasizes theoretical problems across multiple automation levels.The proposed classification uses binary indicators for whether each data-science subtask is automated.

5 TOWARDS LEVEL 6 AUTONOMY FOR AUTOML

The paper argues that current AutoML efforts are concentrated below the levels needed for substantial domain-expert participation and proposes Level 6 as the target for end-to-end autonomy. Reaching that target requires automating task formulation and prediction engineering while addressing limitations in existing function- or data-specific approaches.

  • Automation efforts have primarily produced Level 1–4 tools, leaving Levels 5 and 6—the levels that most reduce data-scientist workload and increase domain-expert participation—largely absent.The paper identifies this gap as a central motivation for advancing AutoML toward higher autonomy.
  • Level 6 is proposed as an end-to-end automated machine-learning pipeline that could let domain experts engage with big data without detailed machine-learning knowledge.The paper connects this goal with more accessible analytics and increased productivity through reduced manual labor.
  • Task formulation and prediction engineering are especially important because they strongly affect how useful and valuable the resulting machine-learning model is.The paper describes task formulation as not currently automated and prediction engineering as rarely automated.
  • Prior systems such as Trane and MLFriend provide foundations for automating prediction-task definition, but remain limited to particular functions or data types and lack a user-centric task recommender.MLFriend targets event-driven time-series data, while Trane lets data scientists specify relational-dataset prediction problems.
  • The roadmap frames Level 6 development as an open research program involving an intelligent data-science agent and challenges associated with achieving end-to-end automation.The paper presents this program as a set of future research opportunities rather than a completed system.

6 CHALLENGES AND OPPORTUNITIES

Level 6 AutoML research focuses on translating human goals into prediction tasks, ranking promising tasks, and presenting results while addressing scalability, interpretability, validity, and user-dependent evaluation challenges.

  • Case Study: A Level 6 AutoML agent is envisioned to help domain experts perform predictive analysis directly on event-driven time-series data.The Flight Delay dataset is used as an exploratory case study for this setting.
  • 6.1 Challenges in Task Formulation: Prediction tasks are characterized by an outcome function, prediction window, and prediction horizon, with outcome-function definition forming the central translation challenge.The paper illustrates this by quantifying website engagement as visits during a week.
  • 6.1 Challenges in Task Formulation: PeTEL should encode outcome functions and search parameters, while human-readable translations are needed for users to comprehend the resulting tasks.The proposed representation uses operators such as filtering, aggregation, and smoothing.
  • 6.2 Challenges in Prediction Engineering for Identifying “Promising” Tasks: The exponentially growing task space makes exhaustive pipeline execution infeasible for interactive agents, requiring filtering and prioritization of promising tasks.Promising-task assessment involves task validity, user preference, potential business value, and sufficient training examples.
  • 6.3 Challenges in Result Summary and Recommendation (RSR): Task utility combines automatically computed measures with human-dependent judgments, but the appropriate feedback format and task representation remain unresolved.Candidate feedback formats include ratings, pairwise comparisons, and partial rankings; representing PeTEL expressions as machine-learning features also requires further investigation.
  • 6.3 Challenges in Result Summary and Recommendation (RSR): Recommendation diversity depends on user preference, and no clear general standard currently evaluates the required degree of diversity.The paper contrasts task sets sharing attributes with task sets spanning different attributes.

7 HUMAN-AI INTERACTION IN LEVEL 6 AUTOML AGENT

Level 6 AutoML requires human-AI interaction that helps analysts understand the space of possible prediction tasks and assess whether recommendations are useful and interesting.

  • Human-AI Interaction: Human analysts and decision-makers must be convinced that Level 6 AutoML recommendations are useful and interesting.This requirement motivates research into human-AI interaction.
  • Human-AI Interaction: Problem-space understanding requires domain experts to grasp possible tasks, raw data, useful tables and attributes, and data-collection frequency.This understanding provides context for the data-science process.

8 THE LONG ROAD AHEAD

The paper argues that current AutoML remains inefficient and inaccessible because key formulation and data-science decisions still require human expertise. It proposes a Level 6 agent and identifies interactive task recommendation, automated task formulation, prediction engineering, useful-task recommendation, and fair evaluation as major research challenges.

  • Current AutoML still leaves prediction-problem definition and alternative navigation to data scientists, limiting direct access for domain experts and business leaders.
  • A Level 6 agent would automate prediction-task formulation and recommendations to increase data-scientist productivity and enable domain experts to use predictive machine learning directly.
  • Designing an interactive prediction-task recommender that actively engages users is identified as the key technical challenge for Level 6 AutoML.
  • An exploratory real-life case study demonstrates the feasibility of a currently hypothetical Level 6 agent while exposing challenges in task formulation, prediction engineering, and useful-task recommendation.
  • Fair model evaluation requires unbiased training, testing, benchmarking, and labeling choices throughout the data-science process.

A AUTOMATION IN DATA VISUALIZATION, CLEANING AND CURATION (DCC)

Data visualization, cleaning, and curation automation spans framework-based, category-based, indicator-based, active, end-to-end, and commercial-system approaches. These methods automate different parts of data-quality assessment, repair, transformation, and workflow recommendation.

  • Framework-based systems support data-quality assessment and cleaning for domain-specific settings such as electronic health records.
  • ClearView categorizes online-review noise as syntactic, semantic, or structural and applies corresponding cleaning methods.
  • Indicator-based cleaning assesses wireless-sensor data using amount, correctness, completeness, and time-correlation measures while reducing redundant cleaning operations.
  • ActiveClean prioritizes records by model value and dirtiness likelihood, interleaving cleaning with model training; its optimizations improve accuracy by up to 2.5x for the same amount of data cleaned.
  • End-to-end systems such as KATARA and Wisteria combine user interaction with knowledge-based repair or cost-aware iterative cleaning-plan recommendations.
  • Available tools include browser-based cleaning, cloud-based transformation suggestions, and configurable or scheduled duplicate-removal filters.

B.1 Research Contributions

Research on automated feature engineering follows transformation-, generation-, and learning-based strategies, with systems producing candidate features from existing attributes, relational structures, linked data, or learned exploration policies. These approaches are represented in open-source tools for relational, temporal, and production feature workflows.

  • Transformation-based methods synthesize candidate features by applying operators to original features and selecting additions according to empirical performance.Evaluation across 25 datasets and three classification algorithms reported an overall 20% classification-error reduction.
  • Van den Bosch uses unary, binary, and group-by operators plus a background classifier, separating feature generation from feature selection.
  • Deep Feature Synthesis generates relational features by following data relationships and sequentially applying mathematical functions along entity paths.
  • Linked-data methods discover datasets and entity URIs, then generate and select potential features for machine-learning problems.
  • Learning-based approaches frame feature generation as model learning, including reinforcement learning under a budget and regression-based feature augmentation.
  • ExploreKit and FeatureTools operationalize transformation-based and Deep Feature Synthesis approaches, while Feast maintains consistency between training and production feature serving.

C.1 Automation in Hyperparameter Tuning

Hyperparameter-tuning research progresses from grid and random search toward sequential, Bayesian, gradient-based, and resource-adaptive methods. The literature addresses efficiency, transfer across tasks, broad parameter spaces, early stopping, and reusable experimental histories.

  • Random search can find good or better models than grid search and manual search with a fraction of grid search’s computation time.The authors also report that only a handful of hyperparameters matter for most datasets, although the important hyperparameters vary across datasets.
  • Sequential model-based optimization can outperform random search when more trials are enabled by computer clusters and GPUs.
  • SMBO’s limitations include numerical-only parameters, optimization for single instances, and no mechanism for terminating poorly performing configurations early.
  • A meta-modeling framework automates tuning through configuration distributions, a loss function, an optimization algorithm, and a database of prior experiments.
  • Bayesian optimization selects evaluations using a probabilistic surrogate model and acquisition function, while multi-task variants transfer knowledge across related domains.
  • Gradient-based tuning computes hyperparameter gradients through reversed stochastic gradient descent, enabling optimization over thousands of hyperparameters.
  • Practical tools implement Bayesian optimization, adaptive resource allocation with early stopping, genetic algorithms, and parallelized search across CPUs and GPUs.

C.2 Automation in Alternative Models Exploration

Alternative-model exploration in AutoML has used sequential model-based, gradient-based, evolutionary, and structured search approaches, alongside systems that automate model construction and configuration. These methods vary in how they represent architectures, select configurations, and reduce search costs.

  • Sequential Model-Based Approach: Sequential model-based optimization frames automatic algorithm selection and hyperparameter tuning as the CASH problem.SMBO builds a model to choose candidate hyperparameter configurations for evaluation.
  • Sequential Model-Based Approach: Auto-sklearn warm-starts Bayesian optimization using performance on similar datasets and constructs ensembles from evaluated models.Its configuration space focuses on scikit-learn base classifiers.
  • Architecture Search: Neural architecture search methods explore structured spaces using recurrent controllers, policy gradients, evolutionary mutations, or hierarchical motifs.Examples include NAS, NASNet, ENAS, hierarchical search spaces, and evolutionary search over NASNet architectures.
  • Architecture Search: ENAS searches for an optimal subgraph within a large computational graph while forcing child models to share parameters.The controller is trained with policy gradient descent to maximize validation-set reward.
  • Other Approaches: Other approaches emulate expert decisions, automate model discovery in staged settings, or provide open-source tools for architecture and pipeline search.Baker et al. predict final validation performance from early learning-curve information, while D3M separates complete-data and unspecified-problem phases.

D AUTOMATION IN EVALUATION

Automation in evaluation includes benchmark suites, surrogate-based hyperparameter studies, and distributed systems for tuning and inference. The reviewed tools cover diverse datasets, algorithms, platforms, and optimization strategies rather than a single evaluation setting.

  • Benchmarking Studies: Benchmarking efforts evaluate GPU-accelerated deep learning tools, reinforcement-learning algorithms, and supervised-classification methods across public task suites.The cited studies include two CPU platforms, three GPU platforms, 31 continuous-control tasks, and the PMLB dataset suite.
  • Benchmarking Studies: No single GPU-accelerated deep learning tool consistently outperforms the others across the benchmarked platforms.The study therefore identifies further optimization opportunity.
  • Surrogate Evaluation: Surrogate models can evaluate hyperparameter-optimization methods by predicting algorithm performance across hyperparameter settings instead of repeatedly running the real algorithm.Eggensperger et al. train regression models on observed performance data with similar hyperparameter spaces and response surfaces.
  • Automated Tuning Systems: Distributed systems support automated tuning through Bayesian, bandit, black-box, grid, or random-search strategies and can reuse information from strong trials.ATM uses conditional parameter trees, Vizier uses Gaussian Process Bandits, and Rafiki supports collaborative tuning and online ensembles.
  • Open Source and Commercial Systems: Tools such as DotData, TPOT, and IBM AutoAI automate different portions of data preparation, feature engineering, pipeline optimization, model evaluation, and algorithm ranking.IBM AutoAI progressively evaluates promising algorithms on increasingly larger data subsets.
Loading 2010.10777v4…