Source-linked AI summary
Automated Machine Learning: State-of-The-Art and Open Challenges
Radwa Elshawi, Mohamed Maher, Sherif Sakr
TL;DR
Rapid data growth and the limited scalability of data-scientist expertise motivate automation of machine-learning model construction. This survey reviews CASH and broader AutoML methods, tools, frameworks, and open challenges, concluding that integration, decision-making, and optimization remain unresolved.
Problem
Growing data volumes and the complexity of selecting algorithms and tuning hyper-parameters create a need to automate machine-learning model construction.
Method
The paper comprehensively surveys CASH techniques, broader AutoML pipeline automation, meta-learning, optimization methods, tools, frameworks, and distributed systems.
Results
The survey covers state-of-the-art AutoML research across model construction and the wider pipeline, while identifying open challenges in current approaches.
Takeaways & Limitations
AutoML remains a broad research area spanning data understanding through deployment, with continued work needed to achieve its intended automation goals.
Takeaways & Limitations
Current AutoML approaches lack a clear one-size-fits-all optimization technique across datasets and still provide limited end-to-end decision-making and integration.
Abstract
from arXiv · showhide
With the continuous and vast increase in the amount of data in our digital world, it has been acknowledged that the number of knowledgeable data scientists can not scale to address these challenges. Thus, there was a crucial need for automating the process of building good machine learning models. In the last few years, several techniques and frameworks have been introduced to tackle the challenge of automating the process of Combined Algorithm Selection and Hyper-parameter tuning (CASH) in the machine learning domain. The main aim of these techniques is to reduce the role of the human in the loop and fill the gap for non-expert machine learning users by playing the role of the domain expert. In this paper, we present a comprehensive survey for the state-of-the-art efforts in tackling the CASH problem. In addition, we highlight the research work of automating the other steps of the full complex machine learning pipeline (AutoML) from data understanding till model deployment. Furthermore, we provide comprehensive coverage for the various tools and frameworks that have been introduced in this domain. Finally, we discuss some of the research directions and open challenges that need to be addressed in order to achieve the vision and goals of the AutoML process.
1. INTRODUCTION
Building high-quality machine-learning models is iterative, complex, and time-consuming because practitioners must choose algorithms and tune many hyper-parameters. AutoML surveys methods that automate CASH and other pipeline stages to reduce reliance on human expertise.
- Machine-learning model development requires iterative decisions across algorithms and numerous hyper-parameters.
- Different parameter configurations and learning algorithms can substantially change performance on the same dataset.The passage reports roughly 5% accuracy variation across Random Forest configurations and around 20% variation across classifier algorithms on yeast data.
- CASH formulates joint algorithm selection and hyper-parameter tuning as an optimization problem.
- The optimization target is near-optimal user-defined evaluation performance within a user-defined search-time budget.Examples of evaluation metrics include accuracy, sensitivity, specificity, and F1-score.
- The paper surveys CASH methods, broader AutoML pipeline automation from data understanding through deployment, and associated tools, frameworks, and open challenges.
2. META-LEARNING
Meta-learning supports warm-starting AutoML by transferring information from prior tasks, evaluations, or pretrained models to new tasks. The surveyed approaches use task similarity, meta-features, learned meta-models, and transfer learning to recommend configurations or initialize adaptation.
- Meta-learning uses prior experience from applying learning algorithms to different data to inform new machine-learning tasks.
- Warm-start techniques group into task-property learning, previous model-evaluation learning, and pretrained-model transfer.
- Task-property methods characterize datasets with meta-features and transfer information between similar tasks.Similarity can be computed from distances between task feature vectors; features include dataset size, statistics, landmarks, and information-theoretic measures.
- Meta-rules can learn pairwise configuration preferences, predicting whether one configuration outperforms another on a new task.
- Meta-models learn relationships among prior-task meta-features and evaluations to recommend or rank configurations for new tasks.
- Transfer learning adapts pretrained models, architectures, and parameters from similar prior tasks through initialization and fine-tuning.
3. NEURAL ARCHITECTURE SEARCH FOR DEEP LEARNING
Neural Architecture Search automates neural-network architecture design for deep-learning tasks. The survey organizes NAS methods around random search, reinforcement learning, gradient-based optimization, evolutionary algorithms, and Bayesian optimization.
- NAS automates neural-network architecture design and has been applied to image and language tasks.
- Random search: Random-search methods explore architectures directly, including approaches using shared weights or early stopping.
- Reinforcement learning: Reinforcement-learning methods train recurrent controllers with gradient-based search to generate architectures maximizing expected accuracy.
- Gradient-based optimization: Gradient-based NAS uses continuous relaxation of architectures so gradient descent can guide the search.The cited experiments reported high-performance convolutional architectures on CIFAR-10 and ImageNet.
- Evolutionary algorithms: Evolutionary approaches optimize neural-network architectures and, in some cases, their weights using genetic or hierarchical evolution.
- Bayesian optimization: Bayesian optimization uses Gaussian processes, random forests, or tree-based models to optimize architectures and their hyper-parameters.The passage notes that Bayesian optimization may outperform evolutionary algorithms on some problems.
4. HYPER-PARAMETER OPTIMIZATION
Hyper-parameter optimization extends model selection by searching configuration spaces with black-box and multi-fidelity methods. The survey covers Bayesian, simulated-annealing, genetic, learning-curve, and bandit-based approaches that trade search quality against computational cost.
- Model selection can itself be treated as a categorical hyper-parameter before tuning the selected model’s parameters.
- Black-box optimization: Black-box optimization methods search configurations without assuming an explicit objective-function form.
- Black-box optimization: Bayesian optimization combines a surrogate objective model with an acquisition function to select promising evaluations for expensive objectives.Gaussian processes are standard surrogates, while random forests are another option; Gaussian processes have cubic complexity in the number of data points.
- Black-box optimization: Simulated annealing iteratively evaluates neighboring hyper-parameter states and accepts or rejects updates according to a criterion.
- Multi-fidelity optimization: Multi-fidelity optimization reduces evaluation cost by combining many cheap low-fidelity evaluations with fewer expensive high-fidelity evaluations.Low-fidelity evaluations use data subsets, whereas high-fidelity evaluations use the whole dataset; speed gains may exceed approximation error.
- Multi-fidelity optimization: Learning-curve methods decide whether to allocate more resources or stop a configuration, and predictive early termination can accelerate Bayesian optimization.The cited passage reports a 2× speed-up and state-of-the-art neural-network performance on CIFAR-10.
- Multi-fidelity optimization: Bandit methods such as successive halving and HyperBand allocate budgets across configurations by eliminating weaker candidates or combining resource schedules.The passage states that successive halving reduces computation time and required iterations relative to uniform allocation, while HyperBand performs better than random search and Bayesian optimization.
5. TOOLS AND FRAMEWORKS
AutoML tools and frameworks automate algorithm selection, hyper-parameter tuning, pipeline construction, and neural architecture search across centralized, distributed, and specialized settings.
- AutoML frameworks are commonly classified as centralized, distributed, or cloud-based tools for automating algorithm selection and hyper-parameter optimization.
- Neural Network Automation Frameworks: Auto-MEKAGGP applies grammar-based genetic programming to automate multi-label classification, while Auto-Net uses SMAC and PyTorch for neural architecture search.
- Centralized Frameworks: Auto-Sklearn combines meta-learning, SMAC-based Bayesian optimization, and ensembles, while TPOT explores feature-engineering and learning pipelines through genetic programming.
- Centralized Frameworks: ML-Plan constructs full pipelines from Weka and Scikit-Learn algorithms using a hierarchical task-network search over a tree-structured space.
- Centralized Frameworks: SmartML uses dataset meta-features and nearest-neighbor similarity search against a knowledge base populated by previous runs to guide algorithm selection.
- Distributed Frameworks: Distributed frameworks address centralized CASH infeasibility by using multiple-node clusters, exemplified by MLbase’s Spark-based search-space pruning.
- Neural Network Automation Frameworks: Auto-Keras and Auto-Net target neural architectures competitive with human-designed architectures, but reported results so far are not significant.
6. OTHER AUTOMATION ASPECTS IN MODEL BUILDING LIFE CYCLE
Automation beyond modeling addresses pre-modeling and post-modeling tasks across data understanding, validation, preparation, experimentation, packaging, and deployment, while retaining human intervention for some decisions.
- Current AutoML tools reduce effort in modeling, but pre-modeling and post-modeling still require human intervention and interpretability for correct decisions.
- Data Understanding: Data understanding formulates alerts and identifies actions, but humans remain essential to confirm the actions taken.
- Data Understanding: Sanity checking identifies unsuitable features, such as those with 70% missing values, before model training.
- Data Understanding: Feature-based analysis evaluates metrics and statistics over data subsets defined by feature conditions to identify slices affecting model quality and behavior.
- Data Validation: Data validation separates preparation from training and includes feature addition, cleaning, and automatic correction of invalid data.
- Data Validation: Data X-Ray diagnoses systematic data errors at scale using MapReduce and Bayesian analysis, while MacroBase detects unusual stream behavior and reaches up to 2 million events per second per query on one core.
- Data Preparation: Data preparation is time-consuming because it combines many preprocessing algorithms with feature synthesis and complementary-data discovery.
- Data Preparation: Feature Tools automates relational feature engineering through deep feature synthesis using entity, forward, and backward relationships between tables.
7. OPEN CHALLENGES AND FUTURE DIRECTIONS
The survey identifies open challenges spanning data preparation, validation, optimization, deployment, scalability, composability, usability, and continuous delivery. These challenges reflect limitations in current AutoML systems and motivate more integrated, adaptive, and scalable approaches.
- Scalability: Centralized CASH frameworks tied to single-node libraries do not scale to large data volumes, while distributed automated solutions remain limited.The survey identifies a need for methods that automatically build and tune models over massive datasets.
- Optimization Techniques: No single hyper-parameter optimization method consistently outperforms alternatives across datasets, search spaces, and evaluation metrics.The survey calls for systems that can select or tune optimization strategies rather than relying on a fixed method.
- Composability: Composable pipelines combining Weka and Scikit-Learn did not significantly outperform Auto-Weka and Auto-Sklearn, especially with big datasets and small time budgets.The survey attributes the challenge to enlarged search spaces, communication overhead, and the need for cost-aware search.
- User friendliness: Current AutoML tools often require sophisticated technical skills, limiting usability and wider acceptance among domain experts and lay users.Interactive, lightweight web interfaces are suggested as one possible response.
- Continuous delivery pipeline: Continuous delivery for machine learning has received limited attention, leaving production deployments without proper unit and integration testing.The paper notes that data scientists often push models directly into production.
- Data Validation: Current AutoML frameworks remain limited in automatically correcting diverse data-quality problems, while data repairing is NP-hard.Most solutions detect problems and notify users rather than performing broad automatic correction.
- Data Preparation: Feature extraction remains a time-consuming pipeline stage, and automated feature transformation, encoder selection, and feature-selection guidance require further research.The survey notes limited automation across domain transformations, encoder architectures, and comparative feature-importance methods.
- Model Deployment and Life Cycle: Integrating separate tools across the machine learning lifecycle remains difficult, including automatic workflow selection and coordination of data understanding, validation, preparation, and deployment.Existing tools ease parts of production automation, but broader whole-pipeline integration is still needed.
8. CONCLUSION
The survey reviews state-of-the-art AutoML frameworks and identifies research directions needed to advance the field. It aims to support researchers and practitioners in understanding these challenges and advancing AutoML.
- 8. CONCLUSION: The survey comprehensively covers AutoML frameworks, research efforts, and open challenges across the machine learning production process.It is presented as a resource for researchers and practitioners seeking insight into future directions.