Source-linked AI summary
Evolving Rewards to Automate Reinforcement Learning
Aleksandra Faust, Anthony Francis, Dar Mehta
TL;DR
Continuous-control objectives can be poor direct RL rewards, while conventional reward shaping requires tedious hand-tuning. AutoRL evolves parameterized rewards by training a population of agents and selecting rewards against task objectives. Across four MuJoCo tasks and two RL algorithms, it improves on baselines, with stronger gains on complex tasks.
Problem
Direct task metrics can be problematic RL rewards, while shaping rewards for tractable learning and desirable behavior requires repeated manual tuning.
Source: §1 · p. 1Method
Results
Across four MuJoCo tasks and two RL algorithms, AutoRL outperforms hand-tuned baselines and often hyperparameter-tuned baselines, with stronger results on complex tasks.
Source: §4.1 · pp. 6–7Takeaways & Limitations
Simpler task objectives can produce comparable or superior policies while reducing reward-engineering effort, and reward tuning may be more beneficial under limited training budgets.
Source: §4.1 · p. 6AutoRL requires O(nmc) processors and runs ng nmc times longer than vanilla RL.
Source: §3 · p. 4
Abstract
from arXiv · showhide
Many continuous control tasks have easily formulated objectives, yet using them directly as a reward in reinforcement learning (RL) leads to suboptimal policies. Therefore, many classical control tasks guide RL training using complex rewards, which require tedious hand-tuning. We automate the reward search with AutoRL, an evolutionary layer over standard RL that treats reward tuning as hyperparameter optimization and trains a population of RL agents to find a reward that maximizes the task objective. AutoRL, evaluated on four Mujoco continuous control tasks over two RL algorithms, shows improvements over baselines, with the the biggest uplift for more complex tasks. The video can be found at: \url{https://youtu.be/svdaOFfQyC8}.
1. Introduction
Direct task metrics can be difficult RL rewards, while hand-crafted reward shaping requires repeated tuning. This paper asks whether evolutionary reward search can automate that process across tasks, algorithms, and objectives.
Introduction
- Directly rewarding task metrics can require long exploration and may produce undesirable behaviors, motivating iterative reward shaping.Practitioners add progress feedback or adjust reward-term weights, then retrain and inspect the outcome.Source: §1 · p. 1
- Earlier AutoRL work addressed navigation with DDPG, leaving its usefulness for other algorithms, tasks, and objectives open.That work first optimized reward parameters for a sparse goal objective, then searched for a network architecture.Source: §1 · p. 2
- AutoRL evolves reward parameters to improve task-objective performance across four continuous-control benchmarks and two RL algorithms.The evaluation also compares metric-based task objectives with standard returns and reports improvements over hand-tuned and, on complex tasks, hyperparameter-tuned baselines.Source: §1 · p. 2
- 489% gain over hyperparameter tuning is reported for SAC on Humanoid with a single-task objective.The introduction also reports that simpler single-task objectives produce results comparable to carefully hand-tuned standard returns.Source: §1 · p. 2
2. Related Work
Prior work applies automation to model search and reward design, but tuning RL itself has been limited. AutoRL uses large-scale optimization for reward and network tuning, focusing here on metric-based objectives.
AutoML and RL
- RL tuning has been limited in AutoML; one evolutionary approach mutated actor-network weights [Khadka and Tumer, 2018].AutoML searches neural-network architectures using RL or evolutionary algorithms [Zoph and Le, 2017; Zoph et al., 2018a; Cai et al., 2018; Zoph et al., 2018b; Real et al., 2017, 2018b; Liu et al., 2017], with an evolutionary edge reported [Real et al., 2018a].Source: §2 · p. 2
Reward design
- Reward-design alternatives include curriculum learning [Florensa et al., 2017; Ivanovic et al., 2018; Gur et al., 2019], bootstrapping [Silver et al., 2018], and inverse reinforcement learning.AutoRL keeps task difficulty constant, trains from scratch, and needs no demonstrations; it could also be combined with curriculum learning and bootstrapping.Source: §2 · p. 2
- AutoRL applies large-scale hyperparameter-optimization tools to reward and network tuning, whereas this paper focuses on metric-based objectives.Earlier AutoRL work used sparse objectives such as reaching a goal or achieving waypoints, which can be easy to achieve in simple settings.Source: §2 · p. 2
3. AutoRL
AutoRL searches parameterized proxy rewards by evaluating the resulting policies against a trajectory-quality objective. It trains a population of agents and uses accumulated reward-search experience to select further trials.
Problem formulation
- The trajectory metric G(T) evaluates policy quality, while the immediate reward R supplies feedback during learning.The objective metric orders policies by human-interpretable trajectory quality; RL agents learn best from dense feedback, so G helps select R.Source: §3 · p. 3
- AutoRL seeks a policy maximizing task fitness J across agents trained with parameterized proxy rewards R(θ).Proxy rewards may add learning-guiding features or reweight the task objective to better match the value-function gradient.Source: §3 · p. 3
Algorithm
- AutoRL trains parallel agents with different reward parameters and estimates each resulting policy’s fitness using Monte Carlo rollouts.The first parameterizations are selected randomly, and the learned policy optimizes cumulative return for its fixed reward parameterization.Source: §3 · p. 3
- AutoRL records reward-parameter fitness and policy-evaluation results, then uses Gaussian Process Bandits to choose subsequent trials.It updates the current best policy as evaluation results arrive and stops after the maximum number of trials.Source: §3 · p. 4
- AutoRL requires O(nmc) processors and runs ng nmc times longer than vanilla RL.The number of concurrent trials must remain below the population size to leave completed experience available for selecting parameters.Source: §3 · p. 4
4. Results
AutoRL outperforms hyperparameter-tuned baselines on task objectives and standard returns across the reported SAC tasks and several PPO tasks. Task-objective policies are often comparable to standard-reward policies, while reward tuning also yields stronger trial performance, especially on complex tasks.
4. Results
- 489% improvement over HP tuning on Humanoid; task-objective AutoRL also beats HP on all SAC tasks and PPO Walker and Humanoid.AutoRL uses non-tuned learning hyperparameters.Source: §4 · p. 5
- AutoRL beats HP tuning on all SAC tasks and on PPO Walker and Humanoid when evaluated on standard returns.Both methods also beat the hand-tuned baseline.Source: §4 · p. 5
- 13 out of 16 standard-return conditions favor standard-reward optimization, while task-objective policies are comparable on other tasks and travel farthest on Humanoid.Figure 2 cross-evaluates standard returns and task objectives for SAC and PPO; Standup and Ant are scaled.Source: §4 · p. 5, Figure 2 · p. 6
- On Humanoid, the standard-reward policy loops through jumping and falling, while the height-reached policy rises stably to just above kneeling.Non-scalarized multi-objective optimization is left for future work.Source: §4 · pp. 5–6
- Reward tuning produces more high-performing trials and higher peak performance than hyperparameter tuning on both SAC and PPO.With limited computational training budgets, reward tuning explores better policies and is more likely to produce good ones.Source: §4 · p. 6
4.1 Conclusion
- The benchmark conclusions hold more strongly for complex environments such as Humanoid, where AutoRL offers less hand engineering and better results.The evaluation covers four MuJoCo tasks and two RL algorithms.Source: §4 · p. 5, §4.1 · pp. 6–7
Appendix A. AutoRL Parameter Settings
The appendix identifies the environments and tuned reward parameters and reports implementation details for SAC and PPO.
Appendix A. AutoRL Parameter Settings
- The SAC and PPO implementations use TF Agents; PPO training steps count gradient updates, with 25 training repetitions every environment-specific N episodes.Approximate environment steps are 19.8 million for Ant and Walker, and 158.4 million for Standup and Humanoid.Source: Appendix A · p. 10
- Table 1 is the environments table for the continuous-control benchmarks used in the study.The environments are Ant, Walker, Humanoid Standup, and Humanoid, ordered by increasing complexity.Source: Table 1 · p. 10
- Table 2 is the table of tuned reward parameters.The standard rewards contain predefined components, and AutoRL changes their parameterization.Source: Table 2 · p. 10
Appendix B. True Objective Selection
For Humanoid Standup, sparse success objectives miss valuable intermediate learning behaviors because few agents reach the target height.
Appendix B. True Objective Selection
- A sparse Standup objective sets success at approximately 1.4 m, a height few agents achieve.Intermediate performance plateaus occur below that target.Source: Appendix B · p. 10
- Agents plateau at 0.4 m, 0.6 m, and 0.75 m, where they may have learned behaviors such as sitting up or rising to one knee.Source: Appendix B · p. 10
- The sparse objective treats those intermediate behaviors as failures despite their value during learning.Source: Appendix B · p. 10
B.1 Single-Task Objectives
Metric-based objectives provide a dense signal for optimizing single-task goals, while standard returns combine multiple objectives through hand-engineered reward weights. AutoRL can optimize parameterized proxy rewards against either objective type.
B.1 Single-Task Objectives
- Figure 3 tracks training performance for approximately 100 agents and the final-value distribution across 1000 studies on Humanoid Standup.The figure concerns optimization over a dense true objective.Source: Figure 3 · p. 11
- Metric-based objectives use distance traveled for Ant, Walker, and Humanoid, and height achieved for Humanoid Standup, providing a continuous signal that reaches the true objective in the limit.They order policies by the task metric without requiring component weights.Source: §B.1 · p. 11
B.2 Multi-Objective Standard Returns
- Standard returns combine multiple objectives through hand-engineered reward weights, which may not align with the true value function under environment dynamics.The section proposes optimizing a reparameterization of the standard reward against the standard return.Source: §B.2 · p. 11
B.3 Single-Task Objective Evaluation
On single-task objectives, AutoRL generally outperforms hand-tuning and hyperparameter tuning, with SAC generally outperforming PPO. The section also reports the resulting parameterized reward weights for each algorithm.
B.3 Single-Task Objective Evaluation
- SAC generally outperforms PPO, and AutoRL outperforms hand-tuning and hyperparameter tuning on task objectives.For Humanoid Standup, no policy fully stood; hyperparameter tuning and AutoRL with SAC reached a consistent crouch.Source: §B.3 · p. 11
- For Humanoid Standup, neither method fully stood, but hyperparameter tuning and AutoRL with SAC rose to a consistent crouch.The passage also reports that AutoRL with SAC had the fastest travel.Source: §B.3 · p. 11
- Tables 4 and 5 present AutoRL parameterized reward weights for SAC and PPO, respectively.Source: Table 4 · p. 12, Table 5 · p. 12
B.4 Multi-Objective Standard Return Evaluation
On standard returns, AutoRL is consistently superior with SAC and superior for Walker and Humanoid with PPO. In Humanoid Standup, only AutoRL with SAC fully stood, though it entered a falling-and-standing loop.
B.4 Multi-Objective Standard Return Evaluation
- AutoRL was consistently superior for SAC and superior for Walker and Humanoid in PPO when evaluated on standard returns.The evaluation collected returns over the complete standard rewards.Source: §B.4 · p. 12
- For Humanoid Standup, AutoRL with SAC was the only policy to fully stand, but it entered a falling-and-standing loop.Hyperparameter tuning rose to a consistent crouch.Source: §B.4 · p. 12
- Videos showed SAC generally outperforming PPO and AutoRL outperforming hand-tuning and hyperparameter tuning on standard returns.AutoRL with SAC also had the fastest travel.Source: §B.4 · p. 12
B.5 Cross-Evaluation of Single-Task Objectives and Standard Returns
Cross-evaluation compares policies trained on task objectives with standard returns, and vice versa. The objectives showed similar performance overall, with task-objective optimization especially strong for Humanoid travel and standard-reward optimization scoring highest on standard returns in most conditions.
B.5 Cross-Evaluation of Single-Task Objectives and Standard Returns
- Standard rewards use predefined components, whereas task-objective optimization evaluates a different objective, normally the achievement objective.The cross-evaluation results compare performance across these distinct evaluation targets.Source: §B.5 · p. 12
- Task objectives and standard returns showed similar performance in cross-evaluation, suggesting task objectives can obtain good policies with reduced reward-engineering effort.Policies optimized for task objectives were evaluated on standard returns, and policies optimized on standard returns were evaluated on task objectives.Source: §B.5 · p. 12
- On Humanoid, task-objective agents traveled farthest with both PPO and SAC, while task-objective and standard-reward optimization were comparable on the other tasks.AutoRL optimized over standard rewards produced the highest scores when evaluated on standard rewards in most conditions.Source: §B.5 · p. 12