PaperlayerView exact revision on arXiv

Source-linked AI summary

Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluation

Yunhao Tang, Tadashi Kozuno, Mark Rowland, Rémi Munos, Michal Valko

arXiv:2106.13125v2cs.LGstat.ML

TL;DR

Gradient-based meta-RL needs Hessian estimates, but policy-dependent value-function distributions make unbiased estimation challenging. This paper unifies higher-order derivative estimation through off-policy evaluation and derives new estimates. It studies the bias–variance properties of the resulting estimates, including TayPO-based estimates, theoretically and empirically.

  • Problem

    Estimating value-function Hessians in meta-RL is challenging because the distributions defining value functions depend on policy parameters, complicating bias characterization.

    Source: §1 · p. 2

  • Method

    The paper frames higher-order value-function derivative estimation as differentiating off-policy evaluation estimates, unifying prior methods and deriving Taylor-expansion-based estimates.

    Source: §1 · p. 2, §2.2 · p. 3

  • Results

    The framework interprets prior methods as special cases and the paper studies TayPO-based estimates theoretically and empirically.

    Source: §6 · p. 10

  • Takeaways & Limitations

    Future advances in off-policy evaluation could be imported into meta-gradient estimates, and the framework motivates applying these estimates in broader meta-RL settings.

    Source: §6 · p. 10

    The convergence impact of Hessian-estimate and correlated-sample bias remains theoretically unclear and is left for future work.

    Source: §2.2 · p. 3, §4.2 · p. 7

Abstract

from arXiv · show

Model-agnostic meta-reinforcement learning requires estimating the Hessian matrix of value functions. This is challenging from an implementation perspective, as repeatedly differentiating policy gradient estimates may lead to biased Hessian estimates. In this work, we provide a unifying framework for estimating higher-order derivatives of value functions, based on off-policy evaluation. Our framework interprets a number of prior approaches as special cases and elucidates the bias and variance trade-off of Hessian estimates. This framework also opens the door to a new family of estimates, which can be easily implemented with auto-differentiation libraries, and lead to performance gains in practice.

1 Introduction

Meta-RL aims to help agents transfer prior experience to new tasks, but gradient-based adaptation requires estimating value-function Hessians whose policy-dependent distributions complicate unbiased estimation. The paper unifies prior approaches through off-policy evaluation and uses the framework to explain bias–variance trade-offs and derive new estimates.

1 Introduction

  • State-of-the-art RL can require orders of magnitude more samples than humans, motivating methods that help agents use prior knowledge on new tasks.Meta-RL formalizes this transfer by training across environments so agents can learn faster on previously unseen environments.Source: §1 · p. 1
  • Gradient-based meta-RL differentiates through policy-gradient updates, reducing the problem to estimating Hessians of value functions.The policy-dependent distributions defining RL value functions mean straightforward differentiation can yield Hessian estimates with improperly characterized bias.Source: §1 · p. 2
  • Prior Hessian-estimation methods include unbiased estimates with potential variance reduction and biased, lower-variance estimates, but their relationships were unclear.The methods were derived through seemingly different approaches, leaving open whether other alternatives existed.Source: §1 · p. 2
  • The off-policy-evaluation framework interprets prior methods as special cases, clarifies estimate bias–variance trade-offs, and yields new estimates based on Taylor expansions.Table 1 presents prior higher-order derivative methods as instances of differentiating off-policy evaluation estimates.Source: §1 · p. 2, Table 1 · p. 2
  • The framework and new method can be implemented in auto-differentiation libraries, and experiments validate important claims about the framework.The paper presents these implementation and experimental goals in its introduction.Source: §1 · p. 2

2 Background

The background defines task-based RL and gradient-based meta-RL, then explains why estimating Hessians is central to practical meta-gradient computation. With finite samples, multiplying Hessian and policy-gradient estimates can be biased even when each estimate is unbiased.

2 Background

  • The section introduces notation and background on RL and meta-RL.Source: §2 · p. 2

2.1 Task-based reinforcement learning

  • Each episode resamples a task, and rewards depend on the state, action, and task; the policy's value is expected discounted reward from the starting state.The setup assumes deterministic rewards, smoothly parameterized policies, and finite-horizon termination.Source: §2.1 · pp. 2–3

2.2 Gradient-based meta-reinforcement learning

  • MAML optimizes expected value after a task-specific parameter update, aiming for fast adaptation to new environments.For a policy-gradient update, differentiating the update introduces the value-function Hessian with respect to policy parameters.Source: §2.2 · p. 3
  • Practical meta-gradients multiply a Hessian estimate by a policy-gradient estimate, making accurate Hessian estimation a central challenge.Even unbiased component estimates can produce a biased product when their finite-sample estimates are correlated; the paper leaves fuller treatment of this bias to future work.Source: §2.2 · p. 3

3 Deriving Hessian estimates with off-policy evaluation

Off-policy value estimates retain dependence on policy parameters, enabling higher-order derivatives to be estimated by differentiating those estimates. The framework covers unbiased estimates and Taylor-based estimates that trade bias against variance.

3.1 Off-policy evaluation: maintaining higher-order dependencies on parameters

  • Off-policy evaluation uses behavior-policy data to estimate target-policy values across a range of policies, retaining their dependence on policy parameters.The paper assumes target and behavior policies assign positive probability to every state-action pair.Source: §3.1 · p. 4

3.2 Warming up: deriving unbiased estimates with variance reduction

  • Under differentiability, boundedness, and neighborhood conditions, differentiating an unbiased off-policy value estimate gives unbiased estimates of the value function’s derivatives.The result applies to the derivative order of interest, including the Hessian.Source: §3.2 · p. 4
  • The doubly robust (DR) estimate is unbiased for all πθ, subsumes step-wise importance sampling when Q(x, a) ≡ 0, and can reduce variance with a properly chosen critic.Source: §3.2 · p. 4

3.3 Trading-off bias and variance with Taylor expansions

  • TayPO-K uses at most K importance ratios; increasing K lowers the bias bound but generally raises variance, while on-policy it preserves derivatives through order K.Its sample-based value estimate has a residual bias that decays exponentially as target and behavior policies become closer.Source: §3.3 · pp. 5–6
  • LVC [24] is equivalent to TayPO-K with K = 1, preserving first-order dependence but biasing the Hessian approximation even on-policy.Source: §3.3 · p. 6

4 From Hessian estimates to meta-gradient estimates

The paper describes how to implement off-policy Hessian estimates with automatic differentiation and how to use them in meta-gradients. It also notes that the convergence effects of meta-gradient bias remain theoretically unclear.

4.1 Auto-differentiating off-policy evaluation estimates for Hessian estimates

  • Automatic differentiation through an off-policy value-estimation function maintains its higher-order dependence on policy parameters, allowing the Hessian estimate to be computed by differentiating the output.The doubly-robust estimator’s parameter dependence is built through its recursive computation.Source: §4.1 · p. 7

4.2 Practical implementations of meta-gradient estimates

  • Hessian-vector products let the meta-gradient be computed by differentiating a scalar objective instead of explicitly forming a D^2-sized Hessian.Source: §4.2 · p. 7
  • The convergence effect of bias in meta-gradient estimates remains theoretically unclear; the paper studies bias and variance empirically and leaves further theory for future work.Prior work links lower meta-gradient variance with faster convergence [49] [50].Source: §4.2 · p. 7

5 Experiments

The experiments examine estimator bias–variance trade-offs and test the second-order estimate in high-dimensional meta-RL. Locomotion gains are marginal overall, while the 2-D control experiment reports more stable training with PROMP-TayPO-2.

5 Experiments

  • The experiments use a tabular random-MDP study to assess bias–variance trade-offs and high-dimensional meta-RL studies to evaluate practical performance gains, focusing on estimates with K ≤ 2.The tabular setting permits exact higher-order derivatives; the high-dimensional studies include continuous 2-D control and simulated locomotion.Source: §5 · p. 7

5.2.1 Continuous control in 2D environments

  • PROMP-TayPO-2 achieves more stable 2-D control training curves than the baselines, while FMAML and PROMP reach slightly lower performance.MAML and EMAML learn quickly initially but ultimately become unstable; TRPO-DiCE generally underperforms, potentially due to bias.Source: Figure 1 · p. 8, §5.2.1 · p. 9

5.1 Investigating the bias and variance trade-off of different estimates

  • As off-policyness increases, most estimates lose accuracy, but the second-order estimate degrades more slowly and dominates across the tested values.The truncated DR estimate is heavily biased and performs poorly unless πθ ≈ µ.Source: §5.1 · p. 8, Figure 1 · p. 8
  • At ε = 0, the second-order and DR Hessian estimates are unbiased and more accurate than the first-order estimate, while step-wise IS remains suboptimal for gradients.The on-policy Hessian result is reported as validating the theoretical analysis.Source: §5.1 · p. 8
  • With ε = 0.5, the first-order estimate leads at small sample sizes, but the second-order estimate surpasses it as sample size increases.The authors attribute the crossover to the first-order estimate’s higher bias; Figure 1(c) reports Hessian accuracy as a function of N.Source: §5.1 · p. 8, Figure 1 · p. 8

5.2 High-dimensional meta-RL problems

  • The high-dimensional comparison uses PROMP-TayPO-2, which applies the second-order estimate to PROMP, alongside TRPO-DiCE, TRPO-MAML, TRPO-FMAML, and TRPO-EMAML baselines.All use plain stochastic gradient ascent for inner-loop updates; their inner-loop loss implementations produce different Hessian estimates.Source: §5.2 · pp. 8–9

5.2.2 Large scale locomotion experiments

  • PROMP-TayPO-2 slightly outperforms PROMP in some locomotion tasks, with slightly faster learning and sometimes higher final performance, but overall empirical gains are marginal.Under the default setup, the authors suggest variance may be a major factor and the first-order estimate is near optimal.Source: Figure 2 · p. 9, §5.2.2 · p. 9

6 Conclusion

The paper unifies higher-order value-function derivative estimates through off-policy evaluation, providing a way to reason about their bias and variance. It also explains why unbiased estimates of an adaptation update do not generally yield unbiased meta-gradients.

6 Conclusion

  • The framework derives prior meta-gradient estimation methods through differentiation of off-policy evaluation estimates and yields TayPO-based estimates.The authors state that advances in off-policy evaluation could be imported into meta-gradient estimates.Source: §6 · pp. 9–10

A Bias with practical estimations of meta-gradients

  • An unbiased estimate of the adapted parameter does not generally give an unbiased meta-gradient because the policy gradient evaluated at that parameter is highly nonlinear.A direct plug-in estimate f(bθ′) can therefore be biased even when bθ′ is an unbiased estimate of θ′.Source: A Bias with practical estimations of… · p. 14

B Proof on the bias of TMAML

The TMAML proof shows that its baseline can reduce Hessian-estimate variance at the cost of bias. The accompanying discussion describes conditions on off-policyness and estimator parameterization used to support the framework’s assumptions.

B Proof on the bias of TMAML

  • TMAML may reduce variance in DiCE [21] Hessian estimates by adding baselines, but at the cost of bias.The proof notes that a term arising from the summation starting at s = t rather than s = t + 1 generally does not evaluate to zero.Source: B Proof on the bias of TMAML · p. 14

C Further discussions on the assumptions

  • The framework assumes bounded importance ratios, a common condition that trust-region optimization can naturally produce.The discussion gives ρθ_t < R for a finite constant R as the bounded-ratio condition.Source: C Further discussions on the assumptions · p. 15
  • The required smoothness of πθ ↦ bVπθ holds for TayPO-K because it is polynomial in πθ, and for finite-horizon DR estimators because they are polynomials of order at most H.The assumptions depend on the functional form of the estimates and the two-stage parameterization through πθ.Source: C Further discussions on the assumptions · p. 15

D Proof of results in the main paper

The proofs establish conditions for unbiased higher-order derivative estimates and give recursive forms for differentiating doubly robust estimates. They also bound TayPO-K derivative bias and characterize the on-policy case.

D Proof of results in the main paper

  • Under the stated assumptions, derivatives of the estimator are unbiased estimates of the corresponding value-function derivatives.The proof uses conditions that permit exchanging derivatives and expectations.Source: D Proof of results in the main paper · p. 15
  • Differentiating the doubly robust estimate recursively yields gradient and Hessian estimates, including a recursive form for the DR Hessian.The derivation differentiates the DR identity as a function of the policy parameters.Source: D Proof of results in the main paper · p. 16
  • The upper bound on TayPO-K derivative bias decreases as K increases, and on-policy TayPO-K preserves derivatives through order K.The on-policy claim applies when μ = πθ.Source: D Proof of results in the main paper · p. 17

E Further details on sampled-based TayPO-K estimates

The appendix describes sampled TayPO-K estimates and a subsampling alternative, then turns to high-level implementation details for Hessian estimates and meta-gradients.

E Further details on sampled-based TayPO-K estimates

  • A naive Monte Carlo TayPO-K estimate has O(T^K) terms; for K ≤ 2, exact computation may be tractable, and Algorithm 4 has O(T^2) complexity.The passage assumes a finite trajectory length T or the effective horizon Tγ = 1/(1 − γ).Source: E Further details on sampled-based… · p. 17
  • Subsampling random times along a trajectory reduces computation to linear time, at the cost of potentially larger variance.The described procedure first generates a full trajectory under μ.Source: E Further details on sampled-based… · p. 17

F High-level code for implementations of Hessian estimates and meta-gradients

F.1 Implementing Hessian estimates with off-policy evaluation subroutines

The section describes how off-policy evaluation estimates can be differentiated and used in meta-RL, and reports how derivative-estimate accuracy varies with sample size in tabular MDPs.

F.1 Implementing Hessian estimates with off-policy evaluation subroutines

  • Established trajectory-based off-policy evaluation routines can be differentiated to estimate Hessians, with parameter dependencies retained in the auto-differentiation implementation.The pseudocode is presented as a high-level JAX implementation.Source: §F.1 · p. 18, Figure 3 · p. 18

F.2 Examples of off-policy evaluation estimates

  • The examples include DR, step-wise importance sampling as a special case of DR, TayPO-1, TayPO-2, and truncated DR.Truncated DR replaces importance ratios with capped ratios to control variance, but introduces bias unless samples are near on-policy.Source: §F.2 · pp. 18–19

F.3 Implementing meta-RL estimates

  • A scalar inner-loop objective updates the parameters, while auto-differentiating the resulting outer objective produces meta-gradient estimates.The inner objective can be replaced by an off-policy evaluation estimate.Source: §F.3 · p. 19

G.1 Tabular MDP

  • The tabular experiments estimate gradients and Hessians from N trajectories of length 20, then compare them with ground truth using a scale-insensitive accuracy measure.The MDPs have 10 states, 5 actions, and discount factor γ = 0.8.Source: §G.1 · p. 20
  • As sample size grows, the second-order estimate quickly surpasses the first-order and DR estimates, with a better bias-variance trade-off especially in the medium-data regime.At small sample sizes, it slightly underperforms the first-order estimate; DR generally outperforms step-wise IS through variance reduction.Source: §G.1 · p. 20

G.2 Large scale experiments

The supplied passages describe a sample-size comparison, implementation choices, and a 2-D control sensitivity study; they do not provide a full large-scale benchmark comparison.

G.2 Large scale experiments

  • In the adopted code base, the learning rate was changed from the suggested α = 10^-3 to α = 10^-4 for more stable learning.The code base was released by [24].Source: §G.2 · p. 20
  • The second-order estimate achieves a better bias-variance trade-off overall as sample size varies in the gradient-estimation plots.Figure 4 reports accuracy between estimates and ground truth.Source: Figure 4 · p. 21
  • PROMP-TayPO-2 replaces PROMP’s first-order inner-loop loss with the second-order estimate to alleviate bias introduced by the first-order term.PROMP uses PPO as its outer-loop optimizer.Source: §G.2 · p. 21
  • The large-scale experiments use a convex mixture of first- and second-order estimates, with β = 0.3 reported as the best-performing setting.Source: §G.2 · p. 21
  • Figure 5 examines sensitivity to second-order coefficients and off-policyness in 2-D control; the second-order estimate is generally more robust across 10-run averages.Source: Figure 5 · p. 22

G.3 Ablation study

On 2-D control environments, performance improved as β increased toward the full second-order estimate, while second-order estimates were more robust to increasing off-policyness.

G.3 Ablation study

  • As β rises from 0 to 1, performance on the 2-D control environment improves stably.β = 0 gives the first-order estimate; β = 1 recovers the full second-order estimate.Source: §G.3 · p. 22
  • Performance appears to plateau near β ≈ 0.9, potentially marking the best bias–variance trade-off between the two extremes.Source: §G.3 · p. 22
  • As ε increases, both estimate types degrade, but second-order estimates perform more robustly.The comparison is on 2-D control environments in high-dimensional setups.Source: §G.3 · p. 22
Loading 2106.13125v2…