Source-linked AI summary
Stochastic Variance Reduction for Policy Gradient Estimation
Tianbing Xu, Qiang Liu, Jian Peng
TL;DR
Model-free policy gradients can be sample-inefficient because on-policy simulation produces high-variance gradient estimates. The paper incorporates SVRG into a trust-region policy-optimization framework and reports faster, stronger performance than TRPO on several Mujoco tasks.
Problem
On-policy policy gradients often have high-variance estimates and require expensive repeated simulation, limiting sample efficiency.
Source: §I · p. 1Method
SVRPO applies SVRG with a control variate inside a trust-region Newton-CG framework, using outer-loop anchor-policy rollouts and inner-loop variance-reduced optimization.
Source: A. Stochastic Variance Reduced Gradient · p. 2, B. Stochastic Variance Reduced Policy… · p. 3, §VI · p. 5Results
SVRPO significantly improves sample efficiency and achieves better performance than TRPO on many Mujoco tasks, with similar returns on the remaining tasks.
Source: A. Comparison with TRPO in Mujoco Tasks · p. 3Takeaways & Limitations
Variance reduction is reported as the major factor behind the method’s stronger results, while Fisher curvature information further accelerates convergence.
Source: B. Understanding SVRPO · p. 5
Abstract
from arXiv · showhide
Recent advances in policy gradient methods and deep learning have demonstrated their applicability for complex reinforcement learning problems. However, the variance of the performance gradient estimates obtained from the simulation is often excessive, leading to poor sample efficiency. In this paper, we apply the stochastic variance reduced gradient descent (SVRG) to model-free policy gradient to significantly improve the sample-efficiency. The SVRG estimation is incorporated into a trust-region Newton conjugate gradient framework for the policy optimization. On several Mujoco tasks, our method achieves significantly better performance compared to the state-of-the-art model-free policy gradient methods in robotic continuous control such as trust region policy optimization (TRPO)
I. INTRODUCTION
Policy-gradient methods suit large or continuous action spaces but rely on on-policy samples and often suffer high-variance estimates, making model-free sampling inefficient. The paper proposes improving TRPO with SVRG to address this sample-efficiency problem.
- Policy-gradient methods directly optimize expected return and work efficiently for large or continuous action spaces.Source: §I · p. 1
- On-policy sampling and high gradient-estimation variance make policy-gradient methods less sample efficient than off-policy methods that reuse historical samples.This matters because model-free reinforcement learning requires repeatedly running a simulator to collect data.Source: §I · p. 1
- TRPO provides robust policy-gradient optimization through trust-region techniques across challenging tasks.Source: §I · p. 1
- The paper applies SVRG to TRPO updates and reports improved efficiency and performance on Mujoco robotic-control tasks.Source: §I · p. 1
B. Policy Gradient Methods
Policy-gradient methods optimize a parameterized policy using gradients of expected return estimated from rollouts. Baselines reduce estimation variance without introducing bias, but the estimates remain simulation-based and noisy.
- Policy-gradient methods optimize a parameterized policy with respect to expected return through gradient descent.Source: B. Policy Gradient Methods · p. 2
- Because Q^π is difficult to compute, practitioners estimate the policy gradient from M rollout trajectories sampled from the current policy π_w.Source: B. Policy Gradient Methods · p. 2
- Subtracting a state-dependent baseline from future returns reduces gradient-estimation variance without introducing estimation bias.REINFORCE uses a constant baseline, whereas A2C uses an estimated value function V^π.Source: B. Policy Gradient Methods · p. 2
C. Trust Region Policy Optimization
TRPO replaces unconstrained parameter-space gradient steps with a surrogate-objective optimization constrained by KL divergence between consecutive policies. Fisher curvature and line search support stable, approximately solved updates.
- TRPO optimizes a local surrogate objective under an expected KL-divergence constraint rather than using the Euclidean parameter metric.The surrogate matches the exact objective at w = w_l and approximates it near w_l.Source: C. Trust Region Policy Optimization · p. 2
- The KL bound δ limits changes between consecutive policies, preserving locality of the approximation and update stability.As δ approaches zero, TRPO reduces to Natural Policy Gradient.Source: C. Trust Region Policy Optimization · p. 2
- TRPO uses the Fisher information matrix as curvature information and approximately solves the constrained problem with a natural-gradient step.Source: C. Trust Region Policy Optimization · p. 2
- A line search tunes the step size η so the expected KL-divergence constraint is satisfied.Source: C. Trust Region Policy Optimization · p. 2
III. TRUST REGION STOCHASTIC VARIANCE REDUCTION POLICY OPTIMIZATION
The method estimates a TRPO surrogate from rollouts and uses density-ratio-weighted advantages to compute its gradient. These estimates provide the inputs for the paper’s trust-region variance-reduction policy optimization.
- Each TRPO iteration samples transitions under the anchor policy π_w_l and approximates the surrogate objective with those samples.Source: §III · p. 2
- The surrogate-gradient contribution of sample (s_t, a_t, r_t) is ρ_t∇_w log π_w(a_t|s_t) Â^π_w_l(s_t,a_t).Here ρ_t is the density ratio between π_w and π_w_l, while  estimates the advantage.Source: §III · p. 2
A. Stochastic Variance Reduced Gradient
SVRG reduces stochastic policy-gradient variance by anchoring mini-batch difference estimates to a nearby parameter with a precomputed full gradient. This preserves unbiasedness while enabling more accurate gradients with small batches.
- Small mini-batches improve computation efficiency but increase variance, whereas SVRG provides more accurate gradient estimates with small batches.Source: A. Stochastic Variance Reduced Gradient · p. 2
- SVRG maintains a nearby snapshot parameter with a precomputed accurate gradient and estimates the current gradient using mini-batch differences.The difference can be small when the current and snapshot parameters are close, reducing estimator variance.Source: A. Stochastic Variance Reduced Gradient · pp. 2–3
- SVRG provides an unbiased exact-gradient estimate whose variance is small near the snapshot, unlike the potentially much larger variance of naive stochastic gradients.Source: A. Stochastic Variance Reduced Gradient · p. 3
B. Stochastic Variance Reduced Policy Optimization
SVRPO applies SVRG to TRPO’s surrogate objective within a double-loop trust-region optimization procedure. It uses backtracking line search, subsampled Fisher information, and conjugate gradients for practical updates.
- SVRPO optimizes TRPO’s surrogate objective with SVRG around the anchor policy, using outer sampling epochs and inner variance-reduced updates.The KL constraint is handled through step-size selection during the inner optimization.Source: B. Stochastic Variance Reduced Policy… · p. 3
- Backtracking line search repeatedly halves the step size until expected utility improves, while bounding the initial step by the KL trust-region distance.Source: B. Stochastic Variance Reduced Policy… · p. 3
- A subsampled trust-region Newton conjugate-gradient procedure uses an empirical Fisher covariance and conjugate gradients to compute the inverse-curvature–gradient product efficiently.Source: B. Stochastic Variance Reduced Policy… · p. 3
IV. EXPERIMENTAL RESULTS
Across six MuJoCo control tasks, SVRPO generally reaches comparable performance with fewer samples and often achieves higher final returns than TRPO. The experiments use matched settings and attribute additional exploration to controlled mini-batch and Fisher subsampling noise.
b) Performance and Sample Efficiency
- SVRPO achieves higher final returns than TRPO on Swimmer, Hopper, Walker, and Ant, while obtaining similar or slightly better returns on Humanoid and similar final returns on Half-Cheetah.Source: A. Comparison with TRPO in Mujoco Tasks · p. 3
a) Experimental Settings
- Experiments use matched common hyperparameters and Gaussian MLP policies in OpenAI rllab, with larger three-layer networks for Ant and Humanoid and two-layer networks for other tasks.Source: A. Comparison with TRPO in Mujoco Tasks · p. 3
b) Performance and Sample Efficiency
- SVRPO needs no more than 40% of TRPO’s samples for similar returns on Swimmer and Walker, while reaching about 4000 versus TRPO’s about 2900 on Walker.Source: A. Comparison with TRPO in Mujoco Tasks · p. 3, Figure 1 · p. 4
- For Hopper, fewer than 200 SVRPO iterations match TRPO’s result after 500 iterations, and SVRPO reaches about 3000 versus TRPO’s about 2200 on Ant after 1000 iterations.Source: A. Comparison with TRPO in Mujoco Tasks · p. 3
c) Implicit Exploration with parameter noise
- Subsampled Fisher estimates and mini-batch variance reduction introduce controlled noise that supports smooth exploration around promising parameter regions.Source: A. Comparison with TRPO in Mujoco Tasks · pp. 3, 5
B. Understanding SVRPO
SVRPO’s gains come from combining variance-reduced policy gradients with Fisher-information curvature, improving returns and sample efficiency across Mujoco tasks.
B. Understanding SVRPO
- Variance-reduced gradients are the major contributor to SVRPO’s better results than TRPO, whereas replacing SVRG with SGD produces less reliable policy improvement.The SGD variant uses policy gradients with line search and can become stuck in poor local policies because of high-variance estimates.Source: B. Understanding SVRPO · p. 5
- Fisher-information curvature accelerates convergence, while removing it leaves Swimmer and Walker stuck in poor policies requiring many more samples.The comparison replaces the Fisher matrix with the identity matrix in the update.Source: B. Understanding SVRPO · p. 5
- SVRPO combines variance reduction with the Fisher information matrix to achieve better returns with improved sample efficiency on Mujoco tasks.The combined design is presented as the source of improvements in tasks including Swimmer and Walker.Source: B. Understanding SVRPO · p. 5
V. RELATED WORK
Related work frames policy search as optimization over parameterized policies, with Fisher information and stochastic variance reduction serving as tools for faster convergence.
V. RELATED WORK
- Policy search optimizes parameterized policies for applications including robotic learning, continuous-action control, and high-dimensional state spaces.The literature spans locomotion, manipulation, safe navigation, and model-based policy search.Source: §V · p. 5
- REINFORCE uses stochastic gradient descent, while Natural Gradient and TRPO use Fisher information to accelerate convergence and SVRG uses control variates for variance reduction.These optimization methods target nonlinear policies in continuous, high-dimensional parameter spaces.Source: §V · p. 5
VI. CONCLUSION AND DISCUSSION
The paper develops SVRPO to address high-variance policy-gradient estimation, combining control-variate variance reduction, trust-region Newton-CG optimization, and mini-batch updates for Mujoco control.
VI. CONCLUSION AND DISCUSSION
- SVRPO combines variance-reduced policy gradients with a control variate in a trust-region Newton-CG framework for model-free robotic control.The method also uses stochastic mini-batch noise for exploration and multiple updates to reuse trajectory information.Source: §VI · p. 5
- SVRPO achieves better performance and sample efficiency than TRPO on Mujoco tasks including Swimmer, Walker, Hopper, and Ant.The supplied conclusion identifies these tasks as examples of the reported improvement.Source: §VI · p. 5
- Figures 2 and 3 examine SVRG’s performance advantage and Fisher information’s convergence acceleration for SVRPO on Swimmer and Walker.Figure 2 compares SVRG and SGD within SVRPO against TRPO, while Figure 3 focuses on Fisher-information effects.Source: Figure 2 · p. 6, Figure 3 · p. 6
- Sub-sampled Fisher-information estimation trades computation for accuracy, supporting scalability to reinforcement-learning and robotics problems with long horizons and many samples.The authors present this trade-off as especially useful for large-scale problems.Source: §VI · p. 6