Source-linked AI summary
Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, Honglak Lee
TL;DR
Model-free reinforcement learning can require very large datasets, while inaccurate dynamics models can degrade model-based learning in complex environments. STEVE combines model-free and model-based learning by uncertainty-aware interpolation across rollout horizons. It reduces sample complexity by an order of magnitude on several challenging tasks, while remaining limited to deterministic policies and environments and omitting estimated bias terms from its weighting objective.
Problem
Model-free reinforcement learning requires tens to hundreds of millions of samples, while imperfect dynamics models can degrade learning in complex environments.
Method
STEVE uses ensembles to estimate uncertainty and dynamically reweights candidate targets from model rollouts of different horizon lengths.
Results
STEVE reduces sample complexity by an order of magnitude on several challenging tasks and matches or outperforms MVE-DDPG and DDPG on every task.
Takeaways & Limitations
STEVE’s uncertainty-aware target reweighting supports effective model use without degrading performance in the complex environments evaluated.
Takeaways & Limitations
STEVE operates only with deterministic policies and environments, and its weighting objective ignores bias terms because reliable bias estimators were unavailable.
Abstract
from arXiv · showhide
Integrating model-free and model-based approaches in reinforcement learning has the potential to achieve the high performance of model-free algorithms with low sample complexity. However, this is difficult because an imperfect dynamics model can degrade the performance of the learning algorithm, and in sufficiently complex environments, the dynamics model will almost always be imperfect. As a result, a key challenge is to combine model-based approaches with model-free learning in such a way that errors in the model do not degrade performance. We propose stochastic ensemble value expansion (STEVE), a novel model-based technique that addresses this issue. By dynamically interpolating between model rollouts of various horizon lengths for each individual example, STEVE ensures that the model is only utilized when doing so does not introduce significant errors. Our approach outperforms model-free baselines on challenging continuous control benchmarks with an order-of-magnitude increase in sample efficiency, and in contrast to previous model-based approaches, performance does not degrade in complex environments.
1 Introduction
Deep model-free reinforcement learning can require tens to hundreds of millions of samples, while model-based methods improve efficiency but may fail when learned dynamics are inaccurate. STEVE addresses this by using ensemble uncertainty to adaptively interpolate rollout horizons, limiting harmful model use.
- Motivation: Tens to hundreds of millions of samples can make deep model-free reinforcement learning infeasible for practical applications such as robotics.Data collection is especially expensive in real-world control problems.
- Motivation: Model-based reinforcement learning improves sample efficiency through imagined rollouts, improved TD targets, or policy inputs, but its successes are concentrated in environments with simple dynamics.PILCO is cited as evidence that orders-of-magnitude sample reductions are possible.
- Problem: In noisy, complex environments, dynamics-model mistakes can cause the wrong policy to be learned and hinder performance.Prior work addresses this by mixing real and imagined data or using ensembles of environment models.
- Evidence: On a toy value-estimation task, noisy-model STEVE converges correctly with a 2× speedup over TD learning, whereas MVE diverges from model errors.With a perfect model, both MVE and STEVE solve the task with 5× fewer samples than TD learning.
- Approach: STEVE extends MVE by interpolating across rollout horizons and favoring estimates with lower uncertainty.It replaces the model and Q-function with ensembles and estimates uncertainty from ensemble variance.
2 Background
The paper frames reinforcement learning as estimating discounted returns and Q-values from replay-buffer transitions, then describes model-based value expansion as replacing standard TD targets with model-rollout targets. MVE can reduce target error under ideal assumptions, but its horizon is sensitive to model and Q-function errors, motivating adaptive horizon selection.
- 2 Background: Reinforcement learning seeks a policy maximizing the expected discounted sum of rewards along sampled trajectories.The setup uses deterministic policy actions and Markovian environment transitions in the described formulation.
- 2.1 Value Estimation with TD-learning: Qπ(s, a) satisfies a recursion relating immediate reward to the discounted value of the successor state, with termination stopping further value propagation.The successor is defined by the environment dynamics, and d(s′) indicates whether it is terminal.
- 2.1 Value Estimation with TD-learning: The Q-function is approximated with a deep neural network trained by minimizing mean squared error between Q estimates and TD targets from replay-buffer transitions.Target computation uses an older parameter copy, θ−.
- 2.1 Value Estimation with TD-learning: In continuous action spaces, a parameterized policy network approximates Q-function maximization, and the paper uses DDPG as its base TD-learning algorithm.The policy minimizes the negative Q-value.
- 2.2 Model-Based Value Expansion (MVE): MVE replaces the standard TD target with a target obtained by rolling out a learned transition, termination, and reward model for H steps.When H = 0, MVE reduces to TD-learning.
- 2.2 Model-Based Value Expansion (MVE): Under a perfect model and similarly biased Q-values, an MVE rollout of horizon H decreases target error by a factor of γ2H.In practice, model errors and nonuniform Q-function bias require tuning H.
- 2.2 Model-Based Value Expansion (MVE): MVE’s effectiveness is limited by task-specific rollout-horizon tuning: H = 10 was used for HalfCheetah-v1 but reduced to H = 3 for Walker2d-v1.The paper motivates dynamically adjusting the horizon to balance model error and Q-function error.
3 Stochastic Ensemble Value Expansion
STEVE forms a single value target by interpolating candidate targets from model rollouts of different horizons. It uses ensembles to estimate uncertainty and inverse-variance weighting to favor more reliable estimates while accounting for model and Q-function errors.
- Candidate targets: STEVE produces H +1 candidate targets from one H-step rollout by considering every rollout horizon from 0 through H.Standard TD learning uses the zero-horizon target, whereas MVE uses the H-step target.
- Target interpolation: The method interpolates all candidate targets rather than using a fixed horizon, balancing errors from the learned Q-function against errors from longer model rollouts.The weights are selected to minimize mean-squared error relative to the true Q-value under the paper's approximation.
- Uncertainty estimation: STEVE estimates uncertainty by maintaining independently trained ensembles for the Q-function, reward function, and dynamics model.Each parameterization is initialized independently and trained on different minibatch data subsets.
- Uncertainty estimation: For each rollout length, model, reward-function, and Q-function combinations generate M · N · L candidate values whose empirical means and variances determine the combined target.The procedure evaluates every ensemble combination at each partial rollout length before aggregating the results.
- Target construction: Inverse-variance weighting forms T STEVE_H, which replaces the ordinary TD target when learning the value function.The paper derives the weighting by estimating variance terms from the ensemble while ignoring unreliable bias estimates.
- Ensemble choice: Although alternative Bayesian or dropout-based model families could provide uncertainty estimates, the paper focuses on ensembles because those alternatives degraded base-model accuracy and ensembles are easier to train.The uncertainty-estimation technique itself applies to any sampleable family of models.
4 Experiments
Experiments across continuous-control tasks show that STEVE improves sample efficiency and performance, especially in complex environments, while remaining competitive in wall-clock time despite additional computation.
- Comparison of Performance: STEVE showed significant improvements in performance and sample efficiency across a wide range of continuous-control environments.The experiments compared STEVE with model-free and model-based baselines using learning curves.
- Comparison of Performance: Within 5M frames, STEVE was the only algorithm to show significant learning on Humanoid-v1, RoboschoolHumanoid-v1, RoboschoolHumanoidFlagrun-v1, and BipedalWalkerHardcore-v2.
- Ablation Study: Ablations found that only STEVE converged faster and higher than the baseline, supporting the role of uncertainty-aware target reweighting.The compared variants included ensemble MVE, uniform-weighted Mean-MVE, TD(λ) variants, and COV-STEVE.
- Ablation Study: Increasing rollout horizon increased STEVE’s sample efficiency even though the Humanoid-v1 dynamics model had high error.
- Wall-Clock Comparison: STEVE-DDPG learned more quickly than DDPG and MVE-DDPG on Humanoid-v1 and achieved higher reward than both baselines there.On RoboschoolHumanoidFlagrun-v1, STEVE-DDPG performed comparably to DDPG.
5 Discussion
STEVE’s gains are attributed to dynamic reweighting across rollout horizons, which adapts model usage to the relative biases of the learned model and Q-function.
- Discussion: STEVE matched or outperformed MVE-DDPG and DDPG on every task and remained competitive with model-free algorithms in wall-clock time.Its sample-efficiency gains offset the additional computation per epoch.
- Discussion: STEVE’s model usage stayed near 50% in most environments but changed markedly in Humanoid-v1 and Swimmer-v1.The authors associate these exceptions with environment complexity and the speed at which an accurate model can be learned.
6 Related Work
Related approaches combine model-free learning with imagined data, uncertainty estimates, or multi-step targets, but STEVE adapts target weighting to individual rollouts. Its main distinction is combining model-free and model-based estimates while accounting for uncertainty and model bias.
- Target weighting: STEVE generalizes TD(λ) by weighting multiple targets according to each rollout's characteristics rather than using fixed weights.TD(λ) applies exponentially decaying weights, whereas STEVE adapts them across examples and during training.
- Off-policy correction: STEVE differs from Retrace(λ) by learning a predictive environment model for on-policy rollouts instead of reweighting off-policy data.Both methods use more complex target weighting than standard TD(λ), but Retrace(λ) is an off-policy correction method.
- Scope: SVG methods provide a related hybrid framework, but STEVE currently operates only with deterministic policies and environments.Extending STEVE to stochastic continuous-control settings is identified as future work.
- Model ensembles: Unlike ME-TRPO's purely model-based policy-gradient approach, STEVE interpolates between model-free and model-based estimates using an ensemble.ME-TRPO uses an ensemble to avoid overfitting to any one model, while STEVE uses model uncertainty to select among estimates.
- Uncertainty adaptation: STEVE adapts to uncertainty in individual estimates, whereas prior real-and-imagined-data methods adjust only the overall ratio and omit model-prediction uncertainty.This comparison concerns Kalweit and Boedecker's ensemble-based adjustment of real versus imagined data.
- Uncertainty-aware models: Uncertainty-aware model-based methods include Bayesian neural-network and dropout approaches, while I2A has been evaluated only on discrete domains.These works motivate uncertainty modeling, but differ from STEVE in policy setting, model use, or evaluation domain.
7 Conclusion
The paper presents STEVE as an uncertainty-aware method for merging model-free and model-based reinforcement learning. It reports better performance than model-free approaches with an order-of-magnitude reduction in sample complexity on several challenging tasks.
- Conclusion: STEVE outperforms model-free approaches while reducing sample complexity by an order of magnitude on several challenging tasks.The conclusion characterizes STEVE as an uncertainty-aware approach for merging model-free and model-based reinforcement learning.
- Future directions: Future work includes more complex world models and alternative techniques for calculating uncertainty and estimating bias.The authors also report evidence that STEVE may help reduce overestimation bias.
A Toy Problem: A Tabular FSM with Model Noise
The toy experiment studies tabular TD learning in a deterministic finite-state chain while introducing controlled noise into the learned dynamics model. It compares value estimation under oracle and noisy model conditions using STEVE and MVE ensembles.
- Environment: The toy environment has states s0 through s100, one action, deterministic forward transitions, and a terminal reward of +100 after otherwise −1 rewards.There is only one possible deterministic policy because every state offers the same single action.
- Value estimation: The experiment estimates state values with tabular TD learning, initializing nonterminal values randomly from 0 to 99 and fixing the terminal value at 0.Updates use the standard undiscounted one-step TD rule on randomly sampled nonterminal transitions.
- Model conditions: The noisy model transitions to a random state 10% of the time instead of correctly advancing one step, while the oracle model matches the true environment.Other noise mechanisms reportedly produced qualitatively similar results.
- Evaluation: Figure 1 evaluates prediction quality using mean squared error between estimated and true state values.The passage specifies the plotted metric but does not provide numerical outcomes.
- Implementation: Both STEVE and MVE use ensembles of size 8 for the dynamics model and Q-function, with MVE targets averaged across ensembled rollouts and predictions.This establishes the shared ensemble configuration for the comparison.
B The TD-k Trick
The TD-k trick trains the Q-function on intermediate rollout states to address the mismatch between replay-buffer data and states generated by on-policy model rollouts. It is compatible with STEVE in principle, but the combined method was not rigorously studied and initial experiments were not promising.
- TD-k trick: The TD-k trick trains the Q-function using every intermediate state of a model rollout.This expands training beyond only the sampled states associated with the rollout endpoint.
- Motivation: Replay-buffer states may poorly overlap with on-policy rollout states, causing high Q-function error on model-generated states that can grow with rollout length.The trick is motivated by reducing this distribution-mismatch problem.
- Relation to STEVE: TD-k is orthogonal to STEVE because STEVE tends to ignore estimates from states with poorly learned Q-values.Better Q-values could still provide more information for computing STEVE's target.
- Scope: STEVE+TD-k was beyond the work's scope, received no rigorous treatment, and had unpromising initial experimental results.The authors leave deeper exploration of the connection to future work.
C Implementation Details
The implementation uses feedforward neural networks, replay-buffer training, multiple environment agents, and fixed-size ensembles with specified update schedules and checkpointing.
- Model architectures and optimization: Policy, reward, and termination networks used four 128-unit layers, while the transition model used eight 512-unit layers; all models used ReLU activations.Parameters were optimized with Adam at a learning rate of 3e-4, and environments reset after 1000 timesteps.
- Policy training: Policies trained from replay buffers of size 1e6 with minibatches of 512, beginning with 1e5 random-interaction frames and then using four policy updates per environment frame.Policy checkpoints were saved every 500 updates and frozen for use as θ−.
- Data collection: Eight training agents collected frames using mostly greedy actions, with probability ϵ = 0.05 of sampling from a normal distribution around the policy’s pre-tanh logit; two greedy agents handled evaluation.
- Dynamics-model training: Dynamics models used replay buffers of size 1e6, minibatches of 1024, 1e5 random-interaction frames, 1e5 pre-training updates, and four model updates per subsequent environment frame.Dynamics-model checkpoints were saved every 500 updates.
- Ensemble configuration: All ensembles contained four independently minibatch-trained members, with M, N, and L each set to 4 in every experiment.Ensemble members shared the same replay buffer while receiving independently sampled minibatches.