Source-linked AI summary
Meta-Gradient Reinforcement Learning
Zhongwen Xu, Hado van Hasselt, David Silver
TL;DR
Reinforcement learning relies on sampled, bootstrapped returns whose design strongly influences algorithm behavior and performance. The paper introduces an online gradient-based meta-learning algorithm that adapts return functions while interacting with the environment, achieving higher performance on 57 Atari 2600 games over 200 million frames.
Problem
Reinforcement learning lacks a teacher or oracle for the true value function, so algorithms rely on returns whose discounting and bootstrapping choices are crucial to performance.
Method
A gradient-based meta-learning algorithm adapts return-function meta-parameters online while the agent interacts and learns from a single environment.
Results
The meta-gradient agent achieved higher performance than previously observed on 57 Atari 2600 games, including median scores of 292.9% versus Rainbow's 153% on human starts and 287.6% versus 223% on no-ops.
Takeaways & Limitations
The approach can automatically adapt return components such as discounting and bootstrapping, with potential applicability to broader learning updates and changing environments.
Takeaways & Limitations
Because the return function changes during training, value functions and policies can become inaccurate when they remain specialized to older returns.
Abstract
from arXiv · showhide
The goal of reinforcement learning algorithms is to estimate and/or optimise the value function. However, unlike supervised learning, no teacher or oracle is available to provide the true value function. Instead, the majority of reinforcement learning algorithms estimate and/or optimise a proxy for the value function. This proxy is typically based on a sampled and bootstrapped approximation to the true value function, known as a return. The particular choice of return is one of the chief components determining the nature of the algorithm: the rate at which future rewards are discounted; when and how values should be bootstrapped; or even the nature of the rewards themselves. It is well-known that these decisions are crucial to the overall success of RL algorithms. We discuss a gradient-based meta-learning algorithm that is able to adapt the nature of the return, online, whilst interacting and learning from the environment. When applied to 57 games on the Atari 2600 environment over 200 million frames, our algorithm achieved a new state-of-the-art performance.
1 Meta-Gradient Reinforcement Learning Algorithms
Meta-gradient reinforcement learning treats the return as a differentiable function whose meta-parameters can adapt online, using cross-validation to improve prediction or control updates. The approach instantiates this idea with TD(λ) and A2C, while conditioning value and policy networks on changing meta-parameters to address non-stationarity.
- Meta-gradient procedure: Online cross-validation applies updates to one experience sample, evaluates the resulting parameters on subsequent experience, and uses the meta-gradient to adjust η.The update gradient tracks how η affected θ′, while the later sample supplies the meta-objective.
- Return parameterization: The return gη(τt) is parameterized by meta-parameters η, including discount γ and bootstrapping λ, which can be differentiated through the learning update.This reframes normally fixed return-design choices as quantities that can be learned online.
- Meta-gradient prediction: The TD(λ) prediction instantiation adjusts η toward better predictive accuracy using a mean-squared-error meta-objective on a second trajectory.The λ-return is treated as constant with respect to θ in the semi-gradient update, while the update remains differentiable with respect to η.
- Meta-gradient control: The A2C control instantiation differentiates an actor-critic update whose objective combines policy control, value prediction, and entropy regularization, then optimizes a policy-gradient meta-objective.The meta-objective evaluates the updated policy on held-out experience using returns computed under fixed meta-parameters η′.
- Conditioned value and policy functions: Because adapting η makes the return non-stationary, conditioning vθ and πθ on an embedding of η lets the agent represent policies and values appropriate to different returns.This reduces the need for the approximators to catch up after meta-parameters shift, though it makes approximation harder.
2 Illustrative Examples
The illustrative MRPs show that meta-gradients can adapt γ and λ to state-dependent signal, noise, and observability patterns. The learned returns alternate between high and low parameter values across states.
- γ adaptation: In the 10-step MRP, low γ is beneficial in noisy states, while high γ preserves information in signal states.Signal states provide +0.1 rewards; noise states produce rewards sampled from N(0, 1).
- λ adaptation: In the 9-step MRP, low λ suits well-known bottleneck states, whereas high λ avoids bootstrapping in noisy, partially observed fan-out states.Fan-out and fan-in transitions form zero-mean reward pairs, while bottleneck states are visited frequently and have more accurate value functions.
- Learned adaptation: The meta-gradient algorithm adapted both λ and γ so returns alternated between high and low values in odd and even states.Figure 1 reports this behavior for meta-gradient prediction with TD(λ).
3 Deep Reinforcement Learning Experiments
Experiments applied meta-gradient learning to IMPALA on 57 Atari games under two evaluation protocols and compared variants adapting γ, λ, or both. Meta-gradient agents improved over IMPALA and exceeded Rainbow on the reported protocols, while conditioning value and policy functions on η was important.
- Experiment setup: The evaluation covered 57 Atari games using human-starts and no-ops-starts protocols, reporting median human-normalised scores.Human starts sample initial states from human play; no-ops starts use random no-op actions and are also used during training.
- Experiment setup: The comparison included baseline IMPALA and variants meta-learning λ, γ, or both γ and λ, under γ = 0.99 and γ = 0.995.Table 1 also matched the cross-validated discount factor γ′ to the comparison discount factor.
- Experiment results: 30%–80% median-performance gains over IMPALA occurred across both start conditions and both tested discount factors.The result is reported relative to the IMPALA baseline algorithm.
- Experiment results: 183% was achieved without conditioning value and policy functions on η, versus 211.9% for tuned IMPALA and 267.9% for the full meta-gradient algorithm.The comparison used a cross-validation discount factor of γ′ = 0.995.
- Experiment results: With 200M frames, meta-gradient achieved 292.9% on human starts and 287.6% on no-ops, compared with Rainbow’s 153% and 223%.The authors note architectural differences, including the deeper neural network used in their work.
4 Related Work
Related work spans meta-learning algorithms, online cross-validation, automated λ selection, and learning return meta-parameters. The paper distinguishes its approach from concurrent work by conditioning value and policy functions and using separate samples for cross-validation.
- Meta-learning: Early meta-learning evolved learning procedures, used recurrent neural networks as meta-learners, or learned recurrent optimisers and meta-policies.MAML instead learns an initialization that adapts quickly to other tasks.
- Return meta-learning: Concurrent work learned an auxiliary reward meta-parameter but neither conditioned value and policy functions nor separated update and cross-validation samples.The paper identifies sample reuse as potentially problematic in stochastic domains because update noise can become correlated.
- Hyper-parameter adaptation: Prior methods adapted learning rates through gradient-based, Bayesian-optimisation, or evolution-based hyper-parameter tuning.Sutton’s online cross-validation work was limited to adapting learning rates for linear updates.
- Bootstrapping adaptation: Earlier λ research analysed bias, variance, and MSE, derived error-bound schedules, or used Bayesian model averaging and maximum-likelihood estimation.These studies focused specifically on automatically adapting the bootstrapping parameter λ.
5 Conclusion
The paper presents online learning of return-function meta-parameters and demonstrates improved performance on 57 Atari 2600 games. The proposed framework extends beyond γ and λ to other return components and learning updates.
- Conclusion: The algorithm learns return-function meta-parameters online while interacting with a single environment.The authors report that it adapts the return and produces better performance.
- Conclusion: Adjusting meta-parameters in a state-of-the-art deep learning algorithm produced much higher performance than previously observed on 57 Atari 2600 games.The evaluation used games from the Arcade Learning Environment.
- Conclusion: The method can target discounting, bootstrapping, other return components, and more generally the learning update itself.The authors frame these choices as meta-parameters that agents may tune automatically and change over time.
A Detailed Hyper-Parameters used in the Atari Experiments
The Atari experiments used fixed hyper-parameters across all games, with settings detailed in Table 2 and selected through a search on six games.
- The meta-gradient hyper-parameters were obtained by searching six games: Beamrider, Breakout, Pong, Q*bert, Seaquest, and Space Invaders.
- All hyper-parameters were fixed across all Atari games.
- Table 2 details the important hyper-parameters used in the Atari experiments.
B Implementation Details
The implementation computes online meta-gradients through differentiable reinforcement-learning updates and includes on-policy correction for stale IMPALA data.
- The return and temporal-difference quantities are rewritten to support the meta-gradient update computation.
- The return used with IMPALA can be an on-policy v-trace return that corrects slightly stale data toward the current policy.
- First-order optimizer updates are linear and differentiable, allowing automatic differentiation to compute the meta-gradient.
B.3 Data Efficiency
The method reuses experience for both agent training and meta-learning, while its IMPALA comparison reports a substantial but bounded running-speed overhead.
- B.3 Data Efficiency: Training and meta-learning can reuse the same experiences by swapping the roles of τ and τ′ for updating and validation.
- B.4 Running Speed: 13k environment steps/second was achieved by the method versus 20k for the IMPALA baseline on the same hardware and software.
- B.4 Running Speed: The meta-gradient updates introduced 35% additional compute overhead, while 200 million frames per game took about 4 hours.
- B.3 Data Efficiency: IMPALA experts trained a separate agent on each environment rather than using a multi-task setting, and population-based training was not used.
C Results of Grid Search of Discount Factor γ on Atari Experiments
The grid search examined four discount factors within IMPALA, and the results indicate that the discount factor has a major effect on Atari performance.
- γ was grid-searched at 0.99, 0.995, 0.998, and 0.999 within the IMPALA framework.
- The search selected a γ′ for the meta-objective to provide a useful proxy for the true return.
- The discount factor γ has a huge impact on agent performance.
- Table 3 compares IMPALA performance across discount factors using human-normalised scores.
D Additional Experiment Results
The experiments compare meta-gradient adaptation of the discount factor alone or together with bootstrapping across individual Atari 2600 games against IMPALA baselines.
- Meta-gradient adaptation of γ is evaluated against IMPALA across individual Atari 2600 games.
- Figure 2 compares γ adaptation with IMPALA using γ = 0.99.
- Relative gain is computed against max(human, baseline) − random, with improvements above 200% capped for visualization.
- Adapting both γ and λ is evaluated against an IMPALA baseline using γ = 0.995.
D.2 Training Curves
Training curves show score comparisons against baselines alongside the online changes in the meta-parameters for representative γ-only and γ, λ experiments.
- The γ-only experiment with γ′ = 0.99 compares training scores against baseline and tracks γ for each game.
- The γ, λ experiment compares training scores against baseline and tracks both γ and λ for each game.