PaperlayerView exact revision on arXiv

Source-linked AI summary

Practical and sample efficient zero-shot HPO

Fela Winkelmolen, Nikita Ivkin, H. Furkan Bozkurt, Zohar Karnin

arXiv:2007.13382v1stat.MLcs.LG

TL;DR

Zero-shot HPO seeks a small, complementary set of hyperparameter configurations, but existing approaches require costly evaluations across datasets and configurations. The paper proposes adaptive surrogate-based and multi-fidelity methods for selecting that set. It reports that both methods have a superior offline-budget versus overall-accuracy tradeoff compared with existing baselines.

  • Problem

    Existing zero-shot HPO methods require evaluations across many datasets and configurations, making offline training costly as hyperparameter spaces change and budgets constrain dataset or configuration coverage.

    Source: §1 · pp. 1–2

  • Method

    The paper proposes OBO, which adaptively selects dataset–configuration evaluations using a surrogate, and a multi-fidelity subset-selection method for settings where surrogate work is impractical.

    Source: §1 · p. 2

  • Results

    The two proposed methods have an offline training budget versus overall accuracy tradeoff superior to existing baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The study's precomputed tables and zero-shot configurations are presented as resources for HPO research and as a way to improve performance versus runtime for the algorithms studied.

    Source: §1 · p. 2

    The authors note that meta-data collection, useful feature choice, and relevant distance metrics remain unclear for meta-data-based task similarity approaches.

    Source: §2 · p. 3

Abstract

from arXiv · show

Zero-shot hyperparameter optimization (HPO) is a simple yet effective use of transfer learning for constructing a small list of hyperparameter (HP) configurations that complement each other. That is to say, for any given dataset, at least one of them is expected to perform well. Current techniques for obtaining this list are computationally expensive as they rely on running training jobs on a diverse collection of datasets and a large collection of randomly drawn HPs. This cost is especially problematic in environments where the space of HPs is regularly changing due to new algorithm versions, or changing architectures of deep networks. We provide an overview of available approaches and introduce two novel techniques to handle the problem. The first is based on a surrogate model and adaptively chooses pairs of dataset, configuration to query. The second, for settings where finding, tuning and testing a surrogate model is problematic, is a multi-fidelity technique combining HyperBand with submodular optimization. We benchmark our methods experimentally on five tasks (XGBoost, LightGBM, CatBoost, MLP and AutoML) and show significant improvement in accuracy compared to standard zero-shot HPO with the same training budget. In addition to contributing new algorithms, we provide an extensive study of the zero-shot HPO technique resulting in (1) default hyper-parameters for popular algorithms that would benefit the community using them, (2) massive lookup tables to further the research of hyper-parameter tuning.

1 Introduction

Zero-shot HPO addresses costly offline configuration selection by using information across datasets and can make subsequent search embarrassingly parallel. The introduction presents adaptive surrogate-based and multi-fidelity methods, plus a normalized score and extensive precomputed tables.

1 Introduction

  • Zero-shot HPO uses a set of configurations learned offline from multiple datasets, making subsequent search embarrassingly parallel.This also addresses limitations of low evaluation budgets and rich search spaces for learning a surrogate from scratch.Source: §1 · p. 2
  • Existing zero-shot methods evaluate N configurations on D datasets, requiring ND training jobs.The authors identify repeated training costs as especially problematic when algorithm versions or deep-network architectures change.Source: §1 · p. 2
  • Offline Bayesian Optimization adaptively selects dataset–configuration pairs while learning a surrogate to find a set of hyperparameter configurations.OBO applies Bayesian optimization to the set-selection objective rather than to choosing one configuration.Source: §1 · p. 2
  • Both proposed methods have an offline-budget versus overall-accuracy tradeoff that is superior to existing baselines.The introduction reports this comparison for OBO and the multi-fidelity approach.Source: §1 · p. 2
  • The second approach uses a multi-fidelity algorithm when surrogate selection, tuning, or testing is impractical or a surrogate performs poorly in a rich hyperparameter space.It is presented as a budget-efficient solution for offline subset selection.Source: §1 · p. 2
  • The study precomputed tables for five machine-learning problems using over six million training jobs, and reports that its zero-shot configurations can improve performance versus runtime.The authors also describe the tables as a resource intended to make HPO research more approachable.Source: §1 · p. 2
  • The paper proposes Relative Error Difference (RED) as a normalized score that has the properties the authors seek for aggregating performance across tasks.The authors report that reviewed alternatives do not have all those properties.Source: §1 · p. 2

2 Related work

Related work uses task similarity, offline configuration evaluations, and transfer between related tasks to inform hyperparameter optimization. The authors also describe applications beyond fixed configuration portfolios and note unresolved choices in meta-data-based similarity.

2 Related work

  • Meta-data-based task similarity methods leave unclear how to collect meta-data, which features to use, and how to choose a relevant distance metric.The authors describe these as unresolved practical questions for that approach.Source: §2 · p. 3
  • Wistuba et al. (2015a) optimize meta-loss using differentiable surrogates and a discrete relaxation, while Wistuba et al. (2015b) use greedy selection after evaluating a full grid on all datasets.The first work uses the resulting configurations to initialize Bayesian optimization.Source: §2 · p. 3
  • Pfisterer et al. (2018) select multiple default configurations greedily from random candidates using surrogate evaluations, but provide no details about the surrogate type or its savings.Their approach combines a discrete random candidate set with iterative selection.Source: §2 · p. 3
  • Related-task transfer has also been applied to pipeline selection, SAT solver configuration, dataset-dependent hyperparameters, and search-space pruning.Other work models related-task evaluations directly in a surrogate, an area the authors describe as still challenging and under active research.Source: §2 · p. 3

3 Zero-shot HPO

The Naive baseline samples configurations, evaluates each on every dataset, and greedily selects a set minimizing aggregate meta-loss. Although optimal subset selection is NP-hard, the paper uses supermodularity to give a greedy approximation guarantee.

3 Zero-shot HPO

  • Naive greedily adds configurations to the selected set to minimize the zero-shot meta-loss over the dataset collection.The objective uses the best loss achieved by the selected configurations on each dataset.Source: §3 · pp. 3–4
  • The Naive baseline samples N configurations and runs ND training jobs, with N set by the budget as floor(B/D).It then uses the resulting validation losses to select K configurations jointly.Source: §3 · p. 3
  • Optimal selection of a size-K subset is NP-hard, while the meta-loss is monotone decreasing and supermodular.The paper states that the loss reduction from adding a configuration is at least as large for a smaller selected set as for a larger one.Source: §3 · p. 4
  • Greedy selection obtains a (1 −e−1)-approximation for the stated objective, and the paper extends the guarantee to approximate local improvements.The extension gives a (1 −e−1+ϵ)-approximation for the stated monotone increasing submodular maximization problem.Source: §3 · p. 4

4 Offline Bayesian optimization (OBO)

Offline Bayesian Optimization uses a probabilistic surrogate to choose which configuration and dataset evaluations to make while optimizing the set of zero-shot configurations. Its acquisition strategy accounts for the cost of completing configuration evaluations, and the authors select a global random-forest surrogate.

4 Offline Bayesian optimization (OBO)

  • At each step, OBO uses a probabilistic surrogate to choose a configuration and location in the selected set, then chooses a dataset to evaluate.The method uses expected improvement to prioritize candidate configuration-location choices.Source: §4 · p. 4
  • OBO estimates a candidate configuration's performance across all datasets and selects it by expected improvement per training job.The acquisition accounts for the evaluations still required to complete that configuration across datasets.Source: §4 · p. 4
  • The authors use random forests because they seek a fast-learning surrogate for rich spaces with categorical and conditional features.They report that one global model with dataset ID as an input outperformed variants that included dataset-specific models.Source: §4 · p. 5

5 Multi-fidelity (MF) zero-shot HPO

The multi-fidelity method combines HyperBand-style resource allocation with sequential selection of a set of configurations. It adapts evaluation across locations while reusing information and balancing resources.

5 Multi-fidelity (MF) zero-shot HPO

  • For K = 1, querying a configuration on a randomly selected dataset gives an unbiased estimate of mean loss, framing the task as best-arm identification under a fixed query budget.The number of configurations can be chosen by trading breadth of exploration against evaluation depth, as in HyperBand.Source: §5 · p. 5
  • For K > 1, greedy location-by-location selection seeks marginal improvements and gives a (1 + e^-1+ϵ) approximation to the optimal size-K subset.Each location is selected based on its improvement over the losses achieved by earlier locations.Source: §5 · p. 5
  • For K = 1, the problem fits HyperBand's framework, so standard HyperBand or an extension using a surrogate such as BOHB can be used.The paper extends this setup to K > 1 through sequential greedy selection.Source: §5 · p. 5
  • To reduce comparison noise, configurations evaluated with equal-sized dataset subsets use the same datasets, drawn from a shared random order.A query at resource r computes loss on the first r datasets in that order.Source: §5 · p. 5
  • Because losses and variances shrink at later locations, those locations can receive fewer resources; exploring locations simultaneously avoids fixed-stage allocations and accommodates reused or leftover resources.The method designs its HyperBand instances to reuse information from earlier locations.Source: §5 · pp. 5–6
  • Asynchronous HyperBand promotion first selects a location according to predetermined resource allocation, then promotes the unpromoted candidate with the lowest loss after appending it to the preceding zero-shot choices.The location selection and candidate promotion are modified to support concurrent exploration of locations.Source: §5 · p. 6

6 Relative error difference (RED)

Relative error difference (RED) normalizes errors against a dataset-specific reference so losses can be meaningfully aggregated across datasets.

6 Relative error difference (RED)

  • Zero-shot HPO can use any loss and aggregate loss, but losses should be meaningfully comparable across datasets before aggregation.The experiments use average RED.Source: §6 · p. 6
  • RED normalizes an unnormalized error metric against a reference loss computed for each dataset, treating a 0/0 ratio as 0.A reduction from 40% to 30% error is equivalent under RED to a reduction from 4% to 3%.Source: §6 · p. 6
  • The authors describe RED as providing robust aggregation compared with other normalization schemes.The comparison is discussed further in Appendix A.Source: §6 · p. 7

7 Experimental setup

The experiments compare zero-shot HPO methods across four supervised-learning algorithms and a zero-shot AutoML search space, using precomputed configuration-performance tables and a leave-one-dataset-out evaluation.

7 Experimental setup

  • Table 1 reports the sizes of the precomputed tables for the experimental settings.The study covers XGBoost, CatBoost, LightGBM, MLP, and ZAML, a zero-shot AutoML search space.Source: Table 1 · p. 7, §7 · p. 7
  • The tables cover public classification datasets with at least 3,000 rows and store validation and test miss-classification rates, training time, and the hyperparameters used.The study applies minimal preprocessing and splits datasets into 50% training, 25% validation, and 25% test sets.Source: §7 · p. 7
  • Evaluation uses average RED and leave-one-dataset-out transfer, reporting the test metric of the configuration with the best validation error at each evaluation budget.Each dataset is evaluated using all other datasets as source tasks; the reference metric averages test results for the ten configurations with lowest validation error.Source: §7 · p. 7
  • The four compared methods—Naive, Surrogate, OBO, and MF—each use a budget of 3,000 training jobs.The Surrogate baseline uses one default-parameter random forest per dataset; OBO uses a 20-tree random forest.Source: §7 · pp. 7–8
  • Preliminary surrogate-model comparisons used only the XGBoost and MLP tables; other settings used only the final algorithm versions, reducing the stated risk of overfitting to the experimental settings.Source: §7 · p. 8

8 Results

The proposed algorithms outperform their respective baselines in the reported experiments. The authors also report a substantial budget gap versus random search and a narrower OBO–MF difference on ZAML.

8 Results

  • OBO consistently outperforms Surrogate, and MF proves superior to Naive in the reported comparisons.Table 2 reports OBO versus Surrogate average RED (± standard error), lower is better; Table 3 reports MF versus Naive on the same basis.Source: §8 · p. 8, Table 2 · p. 8, Table 3 · p. 8
  • Random search requires a 10- to 50-times higher budget to match zero-shot HPO performance.These comparisons are provided in Appendix G.Source: §8 · p. 8
  • OBO outperforms MF in most settings, but the gap disappears for ZAML; the authors conjecture that surrogate benefits diminish in more complex search spaces that are harder to model.Source: §8 · p. 8
  • The authors expect the published configurations and tables to help practitioners improve performance-versus-runtime tradeoffs for the studied algorithms.They also present the tables as resources for further HPO research.Source: §8 · p. 8

Broader Impact

The paper frames zero-shot HPO as a way to reduce offline optimization costs and make precomputed configurations and results more accessible to practitioners and researchers.

Broader Impact

  • The proposed algorithms target the offline cost of computing zero-shot HPO configurations.The paper also provides precomputed configurations for popular public algorithms and training-result tables for benchmarking.Source: Broader Impact · p. 9
  • Precomputed configurations are intended to reduce the difficulty and cost of practitioners performing HPO.Source: Broader Impact · p. 9
  • The training-result tables can support research benchmarking without requiring prohibitive computational budgets.Source: Broader Impact · p. 9

A Comparison of RED to other metrics

The section compares normalization metrics for aggregating performance across datasets. It argues that RED satisfies both desired properties: robustness to rescaling and to datasets where configurations perform similarly.

A Comparison of RED to other metrics

  • All normalization schemes except no normalization are invariant under linear rescaling, while no normalization can let metrics with the largest range dominate an aggregate.Source: A Comparison of RED to other metrics · p. 11
  • Only RED is stated to fulfill both desired properties for aggregating metrics across datasets.The comparison considers rescaling robustness and robustness to simple datasets; Table 4 is identified as a metric normalization comparison.Source: A Comparison of RED to other metrics · p. 11, Table 4 · p. 11
  • For datasets where configurations perform similarly, normalization schemes other than RED and no normalization magnify the differences.The section notes that relative differences can matter more than absolute error-rate changes.Source: A Comparison of RED to other metrics · p. 11
  • The authors use average aggregation to approximate expected performance on new datasets, while noting that median or P90 may suit distribution shifts or other objectives.Their preliminary experiments point toward average as the superior choice in their settings.Source: A Comparison of RED to other metrics · p. 11

B Proofs

The appendix gives proofs for the greedy method and details the anytime HyperBand variant, search spaces, released configurations, and evaluation datasets. It also describes the datasets and generated files used to report results.

C Details of anytime HyperBand

  • Anytime HyperBand treats the number of datasets as its resource and promotes configurations through exponentially increasing rungs, while sometimes launching new candidates at higher rungs.The new starts help address regions that appear poor when evaluated with few resources; the rung resource levels increase by powers of η up to D.Source: C Details of anytime HyperBand · p. 13

D HP search space

  • The ZAML pipeline detects numeric, categorical, and text columns, preprocesses them, then trains with XGBoost, LightGBM, or MLP.Text uses fixed TF-IDF; numeric and categorical preprocessing choices are also described.Source: D HP search space · p. 14

E Zero-shot HP configurations

  • The appendix reports five zero-shot configurations for each setting, obtained by applying the Naive algorithm to the full training-results table.The configurations are presented for users to apply and reproduce the paper's results.Source: E Zero-shot HP configurations · p. 14

F Datasets and generated tables.

  • The configurations were evaluated on public datasets from Kaggle, OpenML, UCI, and the AutoML Challenge, with 50/25/25 train-validation-test splits.The appendix also describes generated validation and test error matrices, configuration mappings, and dataset mappings.Source: F Datasets and generated tables · pp. 14–15

G Additional results

Additional results compare zero-shot configurations with baselines, specify how RED is computed for the plots, and examine aggregation strategy and the number of configurations considered.

G Additional results

  • The additional pairwise tables report average RED and standard error for OBO versus MF, Surrogate versus Naive, MF versus Surrogate, and OBO versus Naive; lower RED is better.The supplied passages identify the comparisons and metric but do not provide table values.Source: G Additional results · p. 15, Table 12 · p. 20, Table 13 · p. 20
  • For the additional plots, each dataset’s RED is relative to the average test miss-classification rate of the ten best generated configurations, selected by validation miss-classification rate.The aggregate is leave-one-dataset-out, and shaded regions show one standard error in each direction, corresponding to 68% confidence intervals of the mean.Source: G Additional results · p. 15
  • Average-based configurations match—and usually outperform—P90-based configurations even when performance is assessed by P90 across datasets.Figure 3 compares average and P90 aggregation; the reported result suggests P90 aggregation is unlikely to provide a benefit.Source: G Additional results · p. 15, Figure 3 · p. 23
  • Considering more random configurations produces better zero-shot configurations, supporting more sample-efficient zero-shot HPO.The ablation compares the full generated set with 1/4 and 1/16 of the data; the full set generally performs better.Source: G Additional results · p. 15, Figure 4 · p. 23
  • Figures 1 and 2 compare OBO and multi-fidelity configurations with naive baselines and random search; the multi-fidelity figure also includes a budget-matched subsampled baseline.The captions note that the full-data naive baselines use orders of magnitude more training jobs, while random search requires a much larger online budget to find competitive configurations.Source: Figure 1 · p. 21, Figure 2 · p. 22
Loading 2007.13382v1…