Source-linked AI summary
FLAML: A Fast and Lightweight AutoML Library
Chi Wang, Qingyun Wu, Markus Weimer, Erkang Zhu
TL;DR
AutoML must select learners and hyperparameters for ad-hoc datasets while using limited computational resources. FLAML jointly models factors affecting trial cost and error, then applies adaptive search strategies. It outperforms leading AutoML systems on benchmark tasks under equal or smaller budgets, while leaving meta-learning and ensemble integration for future work.
Problem
Existing AutoML systems can require long runtimes or large resources, motivating low-cost selection of learners and hyperparameters for ad-hoc datasets.
Method
FLAML jointly considers learner, hyperparameter, sample-size, and resampling choices, using adaptive search strategies that exploit their effects on trial cost and error.
Results
FLAML outperforms top open-source AutoML libraries and a commercial service on a majority of benchmark tasks under equal or smaller time budgets.
Takeaways & Limitations
FLAML provides a lightweight approach for robust AutoML on ad-hoc datasets without relying on expensive preparation such as meta-learning.
Takeaways & Limitations
FLAML does not use meta-learning or ensembles at first order, leaving their integration with cost optimization as future work.
Abstract
from arXiv · showhide
We study the problem of using low computational cost to automate the choices of learners and hyperparameters for an ad-hoc training dataset and error metric, by conducting trials of different configurations on the given training data. We investigate the joint impact of multiple factors on both trial cost and model error, and propose several design guidelines. Following them, we build a fast and lightweight library FLAML which optimizes for low computational resource in finding accurate models. FLAML integrates several simple but effective search strategies into an adaptive system. It significantly outperforms top-ranked AutoML libraries on a large open source AutoML benchmark under equal, or sometimes orders of magnitude smaller budget constraints.
1 INTRODUCTION
FLAML targets AutoML in low-resource settings by jointly considering trial cost and model error, then uses adaptive search strategies to find accurate models efficiently. Evaluations report strong performance against existing AutoML systems under equal or smaller budgets.
- Motivation: Low-cost AutoML is needed for small teams and applications where only limited computational resources can be allocated to model selection.The paper motivates this setting with frequent model selection on changing datasets and examples such as database systems.
- Problem: Existing AutoML systems can require substantial time or resources, underperforming a tuned random forest baseline on 36-51% of benchmark tasks within one CPU hour.The underperformance ratio is higher when the budget is smaller.
- Problem: Trial cost and error jointly depend on learner, hyperparameters, training-data size, and resampling strategy, creating a cost-error tradeoff that prior systems do not handle holistically.Optimizing only error can trigger expensive trials, while optimizing only cost can produce cheap but erroneous trials.
- Approach: FLAML adaptively chooses learners, hyperparameters, sample sizes, and resampling strategies while exploiting their compound effects on trial cost and error.Its search generally moves from cheap, inaccurate trials toward expensive, accurate trials, with little overhead beyond trial cost.
- Evaluation: FLAML outperforms three open-source AutoML libraries and a commercial service on a majority of benchmark tasks under equal or smaller time budgets.The evaluation spans open-source AutoML and regression benchmarks with budgets from one minute to one hour.
2 RELATED WORK
Prior AutoML systems commonly search large spaces using Bayesian optimization, genetic programming, randomized grid search, or meta-learning, often with ensembles. FLAML instead emphasizes efficient, robust single-learner search without first-order reliance on meta-learning or ensembles.
- Open-source AutoML: Auto-sklearn uses Bayesian optimization and meta-learning, TPOT uses genetic programming, and H2O AutoML uses randomized grid search with manually ordered learners.These libraries also use model ensembles to improve accuracy.
- Commercial AutoML: Commercial AutoML platforms provide end-to-end services that consume uncleaned raw data and produce trained models and predictions.The passage lists platforms including SageMaker, DataRobot, Google Cloud AutoML Tables, AzureML, and others.
- Common approaches: Large-search-space AutoML depends strongly on trial order, with meta-learning often proposed to improve ordering using prior datasets and experiments.Meta-learning assumes that prior learner and hyperparameter performance indicates future performance on new tasks.
- Common approaches: Ensembles can boost accuracy but increase inference latency.This trade-off is identified as a common characteristic of ensemble-based AutoML systems.
- FLAML’s positioning: FLAML avoids first-order reliance on meta-learning and ensembles to support immediate customization, plug-in use on new scenarios, and simpler deployment.The paper also cites model complexity, debuggability, and explainability as reasons to prefer single learners.
- Scope: Neural architecture search targets neural networks and commonly addresses unstructured data, whereas FLAML addresses a different search space and application setting.The paper suggests its cost-minimization principles may still be applicable to NAS.
3 API, FORMULATION AND ANALYSIS
This section defines FLAML’s API and formulation, then analyzes how learner, hyperparameter, sample-size, and resampling choices jointly affect error and cost. It derives search properties and organizes FLAML into adaptive components that repeatedly select configurations within a budget.
- 3 API, FORMULATION AND ANALYSIS: FLAML provides a scikit-learn-style Python API with configurable learners, metrics, time budgets, and estimator lists.Users can add customized learners and metrics through the API.
- 3 API, FORMULATION AND ANALYSIS: The fit() method automatically produces an accurate model for an ad-hoc featurized dataset under a given error metric.FLAML does not innovate on featurization, but can support feature preprocessors.
- 3.1 Formulation: A learning configuration χ = (l, h, s, r) combines a learner, its hyperparameters, training sample size, and resampling strategy.A trial returns validation error and a model, while its cost is mainly the CPU time for training and testing.
- 3.2 Analysis: The unobservable test error is approximated by validation error, while hyperparameter domains depend on the selected learner.The analysis identifies several non-blackbox relations among the search variables that existing systems rarely leverage.
- 3.2 Analysis: Learner complexity, regularization, and sample size jointly influence error, while trial cost is approximately proportional to sample size and cost-related hyperparameters.Small samples suit low-complexity configurations; large samples are needed to compare high-complexity configurations.
- 3.2 Analysis: Sample size and resampling jointly affect validation quality: cross-validation reduces variance on small samples, whereas holdout is cheaper and close to test error on large samples.The analysis assumes the gap between test and validation error decreases with sample size and is smaller for cross-validation than holdout.
4 FLAML
FLAML organizes AutoML as an adaptive, low-cost search over learners, hyperparameters, sample sizes, and resampling strategies. Its ECI-based prioritization and complementary search components target both efficient anytime performance and strong final accuracy.
- Design Overview: FLAML separates the ML layer of candidate learners from an AutoML layer containing proposers and a controller.The controller selects configurations, runs trials, and observes validation error and cost until the budget is exhausted.
- Design Overview: FLAML jointly searches learner, hyperparameters, sample size, and resampling strategy while coupling hyperparameters with sample size and ordering learner selection before them.This organization reflects the compound effects of these variables on trial cost and error.
- Search Strategy: ECI estimates the cost for a learner to find an improvement or improve its current configuration with a larger sample size.ECI combines learner-specific error gaps, observed improvement efficiency, and estimated trial costs.
- Search Strategy: FLAML selects cross-validation below specified dataset and computational thresholds and otherwise uses holdout resampling.The default cross-validation setting is 5-fold.
- Search Strategy: Learners are sampled with probability proportional to 1/ECI, prioritizing choices expected to improve error cheaply while preserving every learner’s chance of being searched.ECI updates dynamically as observations accumulate, producing self-correcting prioritization when estimates are too high or too low.
- Search Strategy: Random ECI sampling and random restarts help escape local optima, while the proposer shifts from cheap early configurations toward higher complexity and larger samples when needed.The AutoML-layer overhead is negligible relative to the trial cost, with per-iteration operations linear in hyperparameter dimensionality.
5 EXPERIMENTS
The experiments evaluate FLAML against trial-based AutoML libraries across 53 datasets and budgets from one minute to one hour. FLAML generally outperforms competitors at equal budgets and remains competitive with smaller budgets, while ablations and selectivity estimation assess its search strategy and application performance.
- 5.1 Comparative Study: The evaluation covers 53 datasets—39 classification and 14 regression—and compares FLAML with four AutoML libraries and HpBandSter.Experiments use one CPU core and budgets ranging from one minute to one hour.
- 5.1 Comparative Study: FLAML outperforms top open-source and commercial AutoML systems on a majority of tasks given equal or smaller budgets.The comparison uses the open-source AutoML and regression benchmarks with time budgets from one minute to one hour.
- 5.1 Comparative Study: 62%-83% of datasets match or improve on competitors when FLAML uses 1m versus their 10m, and 72%-89% when using 10m versus their 1h.These comparisons measure FLAML’s performance under smaller time budgets than the baselines.
- 5.2 Ablation Study: Removing learner prioritization, small-data trials, or adaptive resampling degrades search performance across classification and regression datasets.The gap from roundrobin grows before convergence, while FLAML’s early advantage over fulldata narrows as its sample size increases.
- 5.3 Application to Selectivity Estimation: FLAML outperforms other AutoML libraries and the manual configuration on selectivity estimation with a one-CPU-minute budget.On 10D-Forest, FLAML is the only AutoML solution that outperforms Manual.
6 FUTURE WORK
Future work focuses on incorporating prior experience without sacrificing robustness on ad-hoc datasets and on understanding cost-error tradeoffs when ensembles are used.
- 6 FUTURE WORK: FLAML does not use meta learning to optimize each task instance from previous experience.The authors identify integrating meta learning while preserving robustness on ad-hoc datasets as future work.
- 6 FUTURE WORK: The authors also propose studying the cost-error tradeoff introduced by model ensembles.Ensembles are identified as a separate direction for future investigation.
APPENDIX
The appendix describes FLAML’s low-overhead, resource-aware search design and extensions, and reports that short-budget FLAML can match or outperform several baselines given much longer budgets.
- Search-cost estimation: FLAML assigns untried learners estimated costs from the fastest observed learner and predefined offline-calibrated multiples.The estimates adapt to input-dependent trial costs while requiring only calibration of each learner’s fastest configuration.
- Resource-aware execution: FLAML runs one configuration at a time, giving the learner all available cores and RAM while minimizing latency between iterations.The design keeps computation beyond trial cost negligible and starts with inexpensive models so proposers receive feedback early.
- Parallel search: When extra cores are available, FLAML can extend this design with multiple learner search threads selected and updated using ECI.Resources are released after iterations, allowing learner selection to reflect updated cost estimates.
- Optional extensions: FLAML omits stacked ensembling by default to avoid storage, ensemble-building, and model-retraining overhead, while allowing users to enable it when those costs are acceptable.The option requires retaining cross-validation predictions and spending additional computation time.
- Benchmark comparison: In one minute, FLAML is better than or equal to auto-sklearn, H2O AutoML, and TPOT at one hour on more than half of the tasks.The comparison uses a 0.1% tolerance ratio to treat marginal scaled-score differences as close enough.