PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning

Baohe Zhang, Raghu Rajan, Luis Pineda, Nathan Lambert, André Biedenkapp, Kurtland Chua, Frank Hutter, Roberto Calandra

arXiv:2102.13651v1cs.LGcs.AIcs.NEcs.SYeess.SY

TL;DR

MBRL algorithms’ many interacting components and hyperparameters make configuration expertise-intensive, and the paper applies automatic HPO to this problem. Its experiments report that HPO outperforms manually tuned settings and that dynamic tuning can improve performance over static tuning, while configuration transferability remains a consideration.

  • Problem

    MBRL’s complex modeling and planning components create many tunable choices, making configuration expertise-intensive; the paper identifies HPO as not yet evaluated in this setting.

    Source: §1 · pp. 1–2

  • Method

    The paper applies automatic HPO to MBRL, including dynamic tuning methods that change hyperparameters during training.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    HPO significantly outperforms manually tuned MBRL settings, and dynamic hyperparameters further improve performance compared with static settings.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Experiments support that automatic HPO can help researchers find settings for faster learning and better final rewards across different tasks.

    Source: §6 · p. 9

    Hyperparameter settings may not transfer across environments, and dynamic configurations may be less transferable across runs than static ones.

    Source: §4.1 · p. 4

Abstract

from arXiv · show

Model-based Reinforcement Learning (MBRL) is a promising framework for learning control in a data-efficient manner. MBRL algorithms can be fairly complex due to the separate dynamics modeling and the subsequent planning algorithm, and as a result, they often possess tens of hyperparameters and architectural choices. For this reason, MBRL typically requires significant human expertise before it can be applied to new problems and domains. To alleviate this problem, we propose to use automatic hyperparameter optimization (HPO). We demonstrate that this problem can be tackled effectively with automated HPO, which we demonstrate to yield significantly improved performance compared to human experts. In addition, we show that tuning of several MBRL hyperparameters dynamically, i.e. during the training itself, further improves the performance compared to using static hyperparameters which are kept fixed for the whole training. Finally, our experiments provide valuable insights into the effects of several hyperparameters, such as plan horizon or learning rate and their influence on the stability of training and resulting rewards.

1 Introduction

MBRL’s multiple interacting components and design choices make configuration expertise-intensive, while the paper investigates automatic and dynamic HPO as a way to reduce that burden and improve performance. It reports that HPO can outperform experts, dynamic tuning can improve on static tuning, and optimal settings vary across scenarios.

1 Introduction

  • MBRL algorithms combine dynamics modeling, trajectory prediction and propagation, and nested policy or action-sequence optimization, each with tunable design parameters.These choices can significantly affect performance and require substantial human expertise to configure for new tasks.Source: §1 · p. 1
  • MBRL’s low-data regime and many interacting hyperparameters challenge HPO, while potentially changing requirements during learning challenge stationary tuning.The paper proposes dynamic HPO to address these challenges and reduce the need for human expertise.Source: §1 · pp. 1–2
  • Automatic HPO can outperform human experts in tuning MBRL and achieve new state-of-the-art performance.The introduction also reports an experiment in which an agent was so successful that it broke the simulator.Source: §1 · p. 2
  • Dynamic HPO can further improve performance compared with static HPO.Source: §1 · p. 2
  • The best hyperparameter configurations differ across MBRL scenarios, and hyperparameter choices can have a dramatic effect.Source: §1 · p. 2

2 Related Work

Prior HPO work includes static search methods, multi-fidelity approaches, and dynamic optimization, especially for model-free RL. The paper identifies MBRL as an unevaluated setting for HPO in the cited work.

HPO

  • Bayesian optimization uses prior evaluations to guide sequential hyperparameter selection, but full evaluations can make it costly.Multi-fidelity HPO uses cheaper evaluations to estimate performance at higher fidelity.Source: §2 · p. 2
  • Hyperband and BOHB use multi-fidelity evaluation to make hyperparameter search more efficient, while the methods discussed so far search static settings.Successive Halving retains better-performing configurations as fidelity increases; Hyperband runs it with different minimum budgets, and BOHB adds a surrogate model to guide search.Source: §2 · p. 2

HPO for model-free RL

  • Population based training dynamically changes configurations during a run, whereas prior model-free RL HPO studies also include static methods such as BOHB.PBT was used to optimize hyperparameters dynamically and achieved better performance than random search on A3C-like methods across tasks.Source: §2 · p. 2

HPO for model-based RL

  • Wang et al. (2019) benchmarked MBRL agents using static grid searches but reported optimized-agent performance without quantifying the grid searches’ impact.Source: §2 · p. 3

3 Background

The background introduces PBT as a dynamic HPO method and describes the model-based RL loop: learn dynamics, plan with the learned model, execute, and repeat.

3 Background

  • PBT is introduced as an evolutionary approach that optimizes hyperparameters during population members’ training.Source: §3 · p. 3, §3.1 · p. 3

3.1 Population Based Training

  • PBT ranks population members periodically, replaces weaker members with stronger ones and perturbs copied hyperparameters to explore nearby configurations.The passage notes uncertainty about whether training data are also copied during replacement, an issue examined in the experiments.Source: §3.1 · p. 3

3.2 Model-based Reinforcement Learning

  • MBRL learns a dynamics model from data, optimizes a policy using that model, executes the policy for limited steps, and repeats the optimization and execution across trials.Execution is limited to reduce the effect of compounding modeling errors.Source: §3.2 · p. 3

4 HPO for MBRL

The section frames HPO choices around transferability, evaluation stochasticity, and computational cost, then specifies the methods considered. Dynamic configurations can be tailored to a particular environment and run, making transferability a key consideration.

4.1 Comphrehensive Considerations

  • Hyperparameter settings may not transfer across environments because MBRL dynamics models are environment-specific and can require different settings.The choice is between configurations intended to transfer across related tasks and configurations optimized for a particular task.Source: §4.1 · p. 4
  • Dynamic configurations make more parameter choices than static ones and may be less transferable across runs, increasing the difficulty of tuning them by hand.Source: §4.1 · p. 4

4.2 Design Choices

  • The study compares Hyperband, PBT, and random search, and evaluates PBT variants that copy training data or backtrack model parameters and hyperparameters.PBT with backtracking is intended to reduce typical performance drops and explore from previously good configurations.Source: §4.2 · p. 4
  • Using model loss as the HPO objective may be problematic because its correlation with final returns can be weak; return evaluations that are too short or long also pose trade-offs.Short evaluations may favor volatile settings, whereas long evaluations may slow identification of well-performing configurations.Source: §4.2 · p. 4
  • Because traditional HPO can add substantial computational overhead to expensive MBRL problems, the section considers multi-fidelity methods and PBT for efficient parallelization.Source: §4.2 · p. 4

5 Experimental Evaluation

Across PETS experiments, the authors compare automatic and human-tuned hyperparameters, static and dynamic tuning, and the effects of hyperparameter choices. Automatic tuning improves on human-tuned baselines, while configuration performance varies by training duration, environment, and hyperparameter group.

5 Experimental Evaluation

  • The experiments compare automatic HPO with human-expert settings and assess dynamic tuning against static tuning in PETS.The setup uses four MuJoCo tasks and Daisy, with 80 trials for Pusher and 300 for the other environments.Source: §5 · p. 4, §5.1 · p. 5

5.2 Importance of HPO

  • HPO-tuned agents improved final returns over human-tuned baselines by 1.5–2× on Hopper and 10× on HalfCheetah.The gains were less significant in some simpler environments, but automatic tuning always outperformed manual tuning and saved significant overhead.Source: Figure 1 · p. 5, §5.2 · p. 6

5.3 Need for Dynamic Tuning

  • Hyperband configuration rankings had very small or negative correlations across training durations, suggesting different settings can work best at different fidelities.Configurations found at the lowest fidelity were less correlated with those at the highest than with those at the middle fidelity in nearly all environments; Hopper model-training parameters were an exception.Source: §5.3 · p. 6, Figure 2 · p. 6

5.4 Breaking the Simulation

  • Some tuned HalfCheetah agents learned a rolling gait, cartwheeling to extreme speeds and rewards that could cause simulator overflow.The reported overflow was a MuJoCo simulation bug.Source: §5.4 · p. 6

5.5 Effects of Hyperparameters

  • The more impactful hyperparameter group differed by environment: CEM settings mattered more on Hopper, while model-training settings mattered more on HalfCheetah.Model-training configurations varied widely in score, and static search often produced agents with near-zero reward; CEM tuning had fewer low-reward runs, consistent with inaccurate models being more damaging than weak planners in these experiments.Source: §5.5 · p. 7

5.6 Trends of Hyperparameters

  • On Hopper, dynamic tuning outperformed static tuning at comparable budgets, and plan horizon increased later across methods.Both PBT variants learned decreasing learning-rate schedules; on HalfCheetah, optimization methods did not always converge to the same CEM schedules, and PBT reduced CEM population size.Source: §5.6 · pp. 7–8

5.7 Copying History across PBT Members

  • Without copying history during PBT exploitation, HalfCheetah performance dropped 25% for CEM Optimizer hyperparameters and 75% for Model Training hyperparameters.The authors report that history affects model training and implicitly influences planning.Source: Figure 5 · p. 9, §5.7 · p. 9
  • For PETS, the authors recommend dynamic tuning when final performance is the priority, but static settings when transferability across runs is required.They also recommend copying history during PBT exploitation in MBRL.Source: §5.7 · p. 9

6 Conclusion

The experiments show that HPO can improve PETS performance over manual settings, with dynamic configurations outperforming static ones. The authors encourage broader use of automatic and dynamic tuning and identify further search directions.

6 Conclusion

  • HPO significantly outperforms manually tuned settings, and dynamic hyperparameters clearly outperform static configurations.The paper evaluates these design choices on PETS and reports better learning speed and final rewards across different tasks.Source: §6 · p. 9
  • The authors encourage automatic configuration, especially dynamic tuning, to reduce reliance on human expertise and account for RL non-stationarity.Source: §6 · p. 9
  • Future research directions include jointly and dynamically searching hyperparameters with neural architectures and using HPO to search alternative surrogate losses.Source: §6 · p. 9

A.1 Environment Overview

The environment overview summarizes task dimensions, horizons, and trial counts, and provides screenshots of the test environments.

A.1 Environment Overview

  • Table 1 summarizes each environment’s observation and action dimensions, task horizon, and number of trials.Source: Table 1 · p. 12, §A.1 · p. 12
  • Figure 6 provides screenshots of the environments used in the experiments.Source: Figure 6 · p. 12, §A.1 · p. 12
  • Hyperband samples configurations, evaluates them at successive budgets, and retains top performers as budgets increase.Source: §A.1 · p. 12

A.2 Implementation Details of Search Strategies

The appendix specifies how random search, Hyperband, PBT, and PBT-BT allocate evaluations and update configurations.

A.2 Implementation Details of Search Strategies

  • Random search evaluates 40 independent configurations per experiment at full budget, using 3,200 trials for Pusher/Reacher and 12,000 for the other environments.Source: §A.2 · p. 12
  • Hyperband uses random sampling and Successive Halving to reject weak configurations, with budgets chosen to approximately equalize stage costs.The authors use 15 iterations and η = 3; total trials are 3,130 for Pusher/Reacher and 11,982 for the other environments.Source: §A.2 · pp. 12–13
  • PBT replaces the bottom 20% of runs with top-20% runs, then perturbs hyperparameters for 75% of replacements and resamples the rest.Exploitation and exploration occur every 4 trials for Pusher/Reacher and every 5 trials in the other environments.Source: §A.2 · p. 13
  • PBT-BT maintains elites across timesteps and replaces weak current members with sampled elites every 30 PBT timesteps.It also permits exploitation across different timesteps and modifies replacement hyperparameters to avoid duplicates.Source: §A.2 · p. 13

A.3 Search Space

The search-space setup uses a common space across environments for fair comparison, with default values based on Chua et al. (2018).

A.3 Search Space

A.4 Joint Optimization of All Hyperparameters

Jointly tuning model-training and CEM-optimizer hyperparameters doubled the search budget relative to tuning one set at a time. PBT performed best on Hopper, while the static methods outperformed it on HalfCheetah.

A.4 Joint Optimization of All Hyperparameters

  • Joint optimization used 80 configurations for PBT and random search and 30 Hyperband iterations, twice the cost of optimizing one hyperparameter set.The separate searches used 40 configurations or 15 Hyperband iterations.Source: §A.4 · p. 14
  • The authors suggest PBT may struggle in higher-dimensional searches without sufficient compute or a good heuristic.They attribute this possibility to PBT’s evolutionary-search basis and report that its HalfCheetah result lagged the static methods.Source: §A.4 · p. 14
  • PBT performed best on Hopper but worse than both static tuning methods on HalfCheetah under joint optimization.The joint search doubled computational cost; the trends for learning rate, plan horizon, and model weight decay resembled those found when tuning the sets separately.Source: Figure 7 · p. 14, §A.4 · p. 14

A.5 Additional Experimental Results on Other Environments

Across the additional environments, dynamic tuning helped Daisy converge faster, while its benefit was limited on simpler Pusher and Reacher tasks. The HalfCheetah analysis also shows that model-fitting loss can diverge from planning performance, especially across training and unseen data.

A.5 Additional Experimental Results on Other Environments

  • PBT-BT was dropped to reduce compute after it showed similar behavior in most cases and could get stuck in local optima.The authors also describe Pusher as relatively simple, with only a small PBT improvement over random search and Hyperband during the search.Source: §A.5 · p. 14
  • On Daisy, PBT’s model-training schedule converged faster and achieved better rewards than the other HPO methods, while its final-return advantage was less pronounced than on Hopper and HalfCheetah.PBT decreased plan horizon and increased weight decay over time; training epochs also declined, which the authors say may reflect efforts to combat overfitting.Source: Figure 8 · p. 15, §A.5 · p. 15, Figure 9 · p. 17
  • On Pusher, PBT’s dynamic schedules performed better early in training, possibly with a low plan horizon and larger learning rate; all three methods showed similar hyperparameter trends.The three methods also shared an increasing plan-horizon trend.Source: §A.5 · pp. 14–15, Figure 9 · p. 17
  • On Reacher, the three search strategies produced similar results and hyperparameter trends, with no improvement from dynamic tuning.The authors characterize Reacher as lower-dimensional and easier to solve than Pusher in their setting.Source: §A.5 · p. 15, Figure 9 · p. 17

A.6 Objective Mismatch

  • On HalfCheetah, low plan horizons performed well in the first 10 trials but did not learn the other gaits associated with higher returns.The authors trained 40 models with different horizons; Figure 10 plots the average performance and horizon of the top and bottom five members.Source: §A.6 · p. 15, Figure 10 · p. 18
  • On-policy, bottom members had lower NLL despite poorer behavior; off-policy, well-performing models had lower NLL than poorly performing ones, consistent with overfitting to training data.The unseen test data came from other policies, split into three return-based subsets of 300 trajectories each.Source: §A.6 · p. 16, Figure 11 · p. 18, Figure 12 · p. 19

Related papers

Loading 2102.13651v1…