PaperlayerView exact revision on arXiv

Source-linked AI summary

A System for Massively Parallel Hyperparameter Tuning

Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Moritz Hardt, Benjamin Recht, Ameet Talwalkar

arXiv:1810.05934v5cs.LGstat.ML

TL;DR

Large hyperparameter spaces, long training times, and distributed production workloads create a need for scalable tuning methods. The paper introduces ASHA, an asynchronous early-stopping algorithm, and integrates it with production systems optimizations. ASHA outperforms state-of-the-art alternatives and scales linearly with workers, while addressing limitations of synchronous parallelization.

  • Problem

    Large, interacting hyperparameter spaces and long model-training times make production-quality, massively parallel tuning necessary in distributed computing settings.

  • Method

    The paper introduces ASHA, a simple asynchronous hyperparameter optimization method using aggressive early stopping, and develops systems optimizations for production deployment.

  • Results

    ASHA outperforms Fabolas, PBT, BOHB, and Vizier and scales linearly with the number of workers in distributed experiments.

  • Takeaways & Limitations

    ASHA provides a theoretically principled approach to simple and robust massively parallel hyperparameter optimization with production-oriented support.

  • Takeaways & Limitations

    Synchronous SHA is sensitive to stragglers and dropped jobs because each rung must complete before the next begins.

Abstract

from arXiv · show

Modern learning models are characterized by large hyperparameter spaces and long training times. These properties, coupled with the rise of parallel computing and the growing demand to productionize machine learning workloads, motivate the need to develop mature hyperparameter optimization functionality in distributed computing settings. We address this challenge by first introducing a simple and robust hyperparameter optimization algorithm called ASHA, which exploits parallelism and aggressive early-stopping to tackle large-scale hyperparameter optimization problems. Our extensive empirical results show that ASHA outperforms existing state-of-the-art hyperparameter optimization methods; scales linearly with the number of workers in distributed settings; and is suitable for massive parallelism, as demonstrated on a task with 500 workers. We then describe several design decisions we encountered, along with our associated solutions, when integrating ASHA in Determined AI's end-to-end production-quality machine learning system that offers hyperparameter tuning as a service.

1 INTRODUCTION

Modern ML models require tuning across large, interacting hyperparameter spaces, while long training times and increasing parallelism make sequential search impractical. The paper introduces ASHA and integrates it into a production-quality distributed tuning system.

  • Modern models may require evaluating thousands of hyperparameter settings because their high-dimensional parameters interact in unknown ways.
  • Training models can take days or weeks, and each candidate hyperparameter configuration requires a new model training run.
  • Distributed computation addresses the impracticality of sequential evaluation as hyperparameter spaces and per-model training times increase.
  • The paper targets production-quality hyperparameter tuning in distributed computing, with massive parallelism as a cornerstone design criterion.
  • ASHA is a simple, practical method for massive parallelism that uses aggressive early stopping and is designed for large-scale searches.
  • ASHA outperforms mature methods including Fabolas, PBT, BOHB, and Vizier, while scaling linearly with worker count in parallel experiments.
  • The production integration addresses usability, autoscaling, and efficient ML-job scheduling through systems design decisions and optimizations.

2 RELATED WORK

Related work includes adaptive configuration selection, adaptive early evaluation, hybrid methods, and established parallel tuning systems. This paper instead focuses specifically on ASHA and production optimizations for massively parallel tuning.

  • Sequential Methods: Hyperparameter optimization methods either select promising configurations adaptively or early-stop poor configurations while allocating resources to promising ones.
  • Sequential Methods: SHA and Hyperband use adaptive configuration evaluation and have achieved state-of-the-art performance on several empirical tasks.
  • Hybrid Methods: Hybrid approaches combine adaptive configuration selection with evaluation, including Fabolas, Bayesian learning-curve methods, and BOHB.
  • Parallel Methods: PBT exploits partial training to improve a model population but lacks theoretical guarantees and is primarily designed for neural networks.
  • Parallel Methods: Vizier supports multiple optimization and early-stopping methods, but its strategies offer approximately 3× speedup versus order-of-magnitude speedups observed for SHA.
  • Hyperparameter Optimization Systems: Existing distributed systems provide generic algorithm support, whereas this work specializes in ASHA and introduces optimizations for its production performance, usability, and robustness.
  • Hyperparameter Optimization Systems: The paper’s resource-management approach uses adaptive scheduling specifically for ASHA, contrasting with generic shared-compute management and user-specified autoscaling.

3 ASHA ALGORITHM

ASHA adapts Successive Halving for massively parallel hyperparameter tuning by replacing synchronized promotions with asynchronous scheduling and aggressive early stopping. This design reduces straggler bottlenecks, supports rapid returns in large-scale settings, and extends naturally across finite- and infinite-horizon resource regimes.

  • Successive Halving (SHA): SHA repeatedly evaluates configurations, retains the top 1/η, increases each survivor’s resource by η, and continues until reaching maximum resource R.Resources may represent SGD iterations, training examples, or random features.
  • Successive Halving (SHA): Higher early-stopping rate s increases the starting budget r0 by η per increment, while each rung’s total budget is divided among fewer configurations at higher rungs.Different s values define SHA brackets with different initial resource allocations.
  • Parallelizing SHA: Synchronous SHA waits for every configuration in a rung, making it sensitive to stragglers and dropped jobs while keeping promotion estimates independent across brackets.These properties hamper straightforward parallelization in the large-scale regime.
  • Asynchronous SHA (ASHA): ASHA promotes configurations whenever possible instead of waiting for rung completion, and adds new configurations to the base rung when no promotion is available.This asynchronous policy increases the supply of configurations that can later reach upper rungs.
  • Asynchronous SHA (ASHA): 13/9×time(R) is sufficient for ASHA to return a fully trained configuration in the nine-machine example, compared with 3×time(R) for SHA’s toy bracket.With iterative training, checkpointing and resumption can reduce ASHA’s return time to time(R).
  • Algorithm Discussion: ASHA trades a small number of incorrect early promotions for removal of the synchronous promotion bottleneck, with the expected mispromoted fraction vanishing as evaluated configurations increase.The later-rung dependence is expected to be weak even though promoted configurations are no longer independently sampled.
  • Algorithm Discussion: ASHA can generalize from finite to infinite horizons by continuing promotions beyond R, whereas SHA requires rerunning larger-budget brackets; ASHA also grows brackets incrementally and uses intermediate losses to reduce latency.SHA returns outputs only after bracket completion, while ASHA can update the current best configuration from intermediate results.

4 EMPIRICAL EVALUATION

The evaluation compares ASHA and related hyperparameter optimization methods across sequential, distributed, neural architecture search, and massively parallel settings. ASHA generally matches or outperforms alternatives, including faster discovery of good configurations and strong performance with 500 workers.

  • Sequential Experiments: Hyperband tends to outperform Fabolas across four benchmarks and exhibits lower variance across experimental trials.
  • Limited-Scale Distributed Experiments: ASHA evaluated over 1000 configurations in just over 40 minutes with 25 workers and found benchmark-1 error below 0.21 in approximately single-model training time.ASHA required nearly 400 minutes to reach the same target sequentially, but achieved only a 10× speedup because the task was relatively simple.
  • Limited-Scale Distributed Experiments: ASHA finds a good configuration 1.5× as fast as synchronous SHA and BOHB on benchmark 1, while BOHB finds a slightly better final configuration.On benchmark 2, ASHA performs significantly better than synchronous SHA and BOHB when training-time variance exacerbates synchronous straggler sensitivity.
  • Tuning Neural Network Architectures: ASHA finds CNN and RNN architectures meeting target thresholds nearly twice and nearly three times as fast as SHA and BOHB, respectively.Final CNN test error averages 3.24% for ASHA versus 3.42% for SHA and 3.36% for BOHB; RNN validation perplexity is 63.5 versus 64.3 and 64.2.

5 PRODUCTIONIZING ASHA

The production system operationalizes ASHA through usability defaults, autoscaling, fair-share scheduling, and resumable state. These choices balance training latency, configuration throughput, cluster utilization, and recovery needs.

  • System design considerations: ASHA productionization addresses usability, autoscaling, job scheduling, and reproducibility as core system considerations.The implementation discussion focuses on streamlining the interface, balancing latency and throughput, scheduling jobs efficiently, and preserving reproducibility.
  • Selecting ASHA default settings: ASHA defaults promote the top 1/4 of configurations at each rung and allow up to five rungs with minimum resource R/256.The default elimination rate is η = 4, and the most aggressive bracket begins at r = R/256.
  • Selecting ASHA default settings: The standard configuration runs brackets s = 0, 1, 2 to increase robustness while favoring faster early stopping.The two least aggressive brackets are excluded, while more conservative or aggressive bracket sets remain available.
  • Stopping criteria: Using n as the stopping criterion matches random and grid search interfaces and avoids requiring prior knowledge for compute-time or performance thresholds.The production system explicitly stops after the requested number of configurations.
  • Automatic scaling of parallel training: Paleo models diminishing parallel-training returns and supports limiting GPUs per configuration to a target efficiency.For Inception, requiring at least 75% efficiency limits each configuration to at most 16 GPUs.
  • Resource scheduling: A centralized fair-share scheduler allocates resources across jobs while determining each job’s appropriate parallelism.For ASHA, the scheduler derives resource requirements from algorithm inputs and parallel-scaling profiles.
  • Algorithm level resource allocation: ASHA’s adaptive maximum parallelism is κ|S|, improving cluster utilization over static allocation without adversely impacting performance.The task stack grows with bottom-rung configurations and gains promotable configurations as workers finish tasks.
  • Reproducibility and recovery: Reproducible checkpoints and complete bracket states allow jobs to resume after crashes and support evaluating additional configurations.Resuming a bracket can improve selection estimates because wider rungs better estimate the top 1/η configurations.

6 CONCLUSION

The paper presents ASHA as a simple, robust, theoretically principled method for massively parallel tuning and supplements it with production-system solutions. Empirically, ASHA outperforms several state-of-the-art methods across benchmark tasks.

  • Conclusion: ASHA provides simple and robust massively parallel hyperparameter optimization within a production-quality tuning system.The method is presented as theoretically principled and designed for integration into production systems.
  • Conclusion: ASHA outperforms Fabolas, PBT, BOHB, and Vizier across a suite of hyperparameter tuning benchmarks.The conclusion reports this as the paper’s empirical result across state-of-the-art comparison methods.

A APPENDIX

The appendix provides supplementary comparisons and implementation details for robustness experiments, sequential-method evaluation, and the main empirical results.

  • Appendix: Supplementary material evaluates stragglers and dropped jobs, compares synchronous SHA with ASHA, reports the Fabolas comparison, and adds details for Section 4 results.These materials extend the paper’s robustness, sequential-comparison, and empirical analyses.

A.1 Comparison of Synchronous SHA and ASHA

The appendix compares synchronous SHA and ASHA under simulated stragglers and dropped jobs using 25 workers. ASHA completes more configurations when training-time variability is high.

  • Experimental setup: The simulations use η = 4, r = 1, R = 256, and n = 256 for synchronous SHA and ASHA, with ASHA using s = 0.Both methods are evaluated under matched settings, while ASHA uses the maximum early-stopping rate.
  • Simulation model: Stragglers are simulated by multiplying expected runtime by 1 + |z|, while dropped jobs occur independently with probability p at each time unit.The runtime perturbation uses normally distributed z, and drop simulations apply throughout each job’s runtime.
  • Results: With 25 workers, ASHA trains many more configurations to completion than synchronous SHA when training-time standard deviation is high.Figure 7 reports averages over 25 simulations for each combination of variability and drop probability.

A.2 Comparison with Fabolas in Sequential Setting

The experiments compare Hyperband with Fabolas across SVM and CNN tuning tasks, using resource definitions suited to each task. Hyperband (by rung) is competitive with or outperforms Fabolas, while aggressive early stopping drives much of Hyperband’s progress.

  • Experimental Tasks: Hyperband was compared with Fabolas on SVM, CIFAR-10 CNN, and SVHN CNN tuning tasks.The SVM resource was training datapoints; the CNN resource was training iterations.
  • Experimental Tasks: Fabolas is not directly applicable to the CNN tasks because it was designed for datapoints as the resource.The authors nevertheless treat it as a reasonable competitor because freeze-thaw Bayesian optimization performs poorly on deep learning tasks.
  • Results: Hyperband (by rung) outperforms Fabolas by leveraging intermediate validation losses rather than waiting for complete SHA brackets.The bracket-based variant records the incumbent only after each bracket completes.
  • Results: Hyperband (by rung) is competitive with Fabolas and often finds a better configuration with less variance.This accounting scheme records incumbents after each SHA rung, using intermediate validation losses.
  • Results: Most Hyperband progress comes from bracket 0, which uses the most aggressive early-stopping rate.Hyperband traverses brackets in decreasing early-stopping rate order.

A.3 Experiments in Section 4.1 and Section 4.2

These experiments define evaluation and implementation settings for comparing hyperparameter optimization methods, including SHA, BOHB, Hyperband, ASHA, and PBT. They also document PBT constraints needed for fair architecture-tuning comparisons.

  • Evaluation Setup: The experiments use standard CIFAR-10 splits, validation error for algorithm decisions, test error for reporting, and AWS g2.2xlarge instances.Configurations are evaluated on the validation set, while final performance is reported on the test set.
  • Algorithm Settings: SHA and BOHB use n = 256, η = 4, s = 0, and r = R/256 with R = 30000 SGD iterations.Hyperband runs five SHA brackets, while ASHA and asynchronous Hyperband use the same settings as their synchronous counterparts.
  • PBT Implementation: PBT uses truncation selection, replacing the bottom 20% with uniformly sampled configurations from the top 20%.Inherited weights and hyperparameters are copied before exploration perturbs or resamples them.
  • PBT Limitations: PBT cannot directly handle architecture-changing hyperparameters because perturbed architectures invalidate inherited weights.The adaptation fixes architecture-affecting hyperparameters during exploration.
  • PBT Limitations: Configurations are restricted to within 2000 iterations of one another to avoid bias against slower-training configurations.Without this restriction, comparisons would favor configurations trained for more iterations.

A.4 Experimental Setup for the Small CNN Architecture Tuning Task

The small CNN architecture-tuning benchmark varies network architecture and evaluates models on SVHN using established train, validation, and test splits. Its search space is also reused for the CIFAR-10 architecture task.

  • Search Space: The benchmark tunes a multilayer CNN whose search space includes the number of layers and number of filters.The number of layers counts convolutional layers before two fully connected layers; the final CNN layer has twice the specified filter count.
  • Datasets: The same small architecture search space is used for SVHN and CIFAR-10 experiments.The SVHN task uses the dataset splits described by Sermanet et al. (2012).
  • Datasets: SVHN evaluation follows train, validation, and test splits created according to Sermanet et al. (2012).

A.5 Experimental Setup for Neural Architecture Search Benchmarks

The neural architecture search experiments sample CNN and RNN cell architectures from established search spaces and extend the evaluation to a near-state-of-the-art LSTM language-modeling space. ASHA is compared with PBT using large-scale GPU resources.

  • NAS Benchmarks: The NAS benchmarks use the CNN and RNN cell search spaces from Liu et al. (2019).Architectures are sampled randomly and trained with the evaluation hyperparameters used by Liu et al. (2019).
  • LSTM Benchmark: The LSTM search space is built around Merity et al. (2018)’s near-state-of-the-art DropConnect architecture.ASHA and PBT are each run with 16 GPUs on one AWS p2.16xlarge instance.
  • Algorithm Settings: The LSTM benchmark uses η = 4, r = 1 epoch, R = 256 epochs, and s = 0 for ASHA, SHA, and BOHB.PBT uses a population size of 20 and explores or exploits every 8 epochs.
  • Results: ASHA initially trails PBT but soon catches up and finds a better final configuration.At the end, the ASHA and PBT min/max ranges do not overlap.
  • Results: ASHA’s best configuration reaches validation and test perplexities of 58.7 and 56.3 after fine-tuning.Before fine-tuning, the corresponding perplexities are 60.2 and 58.1.
Loading 1810.05934v5…