Source-linked AI summary
Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning
Noah Y. Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, Martin Riedmiller
TL;DR
Batch RL must learn from fixed, potentially conflicting data even though standard off-policy methods can fail in continuous control. The paper regularizes policy iteration toward a learned behavior prior, using an advantage-weighted model to favor supported actions that work for the current task. The approach improves stability and performance across continuous-control, simulated-robot, and real-robot settings, including tasks without direct trajectories.
Problem
Batch RL learns from fixed data, but naïve off-policy methods can fail in continuous control and conflicting behavior data can be misleading.
Method
The method learns a behavior prior and regularizes policy improvement toward it, with an advantage-weighted model favoring observed actions that are successful for the current task.
Results
The approach improves over competitive baselines across standard continuous-control benchmarks and multi-task learning with simulated and real-world robots.
Takeaways & Limitations
The learned prior supports stable offline learning from conflicting and sub-optimal data, including some robot-manipulation tasks without any trajectory for the task itself.
Takeaways & Limitations
The fixed dataset is collected beforehand by arbitrary behavior policies, which may generate trajectories unrelated to the target task.
Abstract
from arXiv · showhide
Off-policy reinforcement learning algorithms promise to be applicable in settings where only a fixed data-set (batch) of environment interactions is available and no new experience can be acquired. This property makes these algorithms appealing for real world problems such as robot control. In practice, however, standard off-policy algorithms fail in the batch setting for continuous control. In this paper, we propose a simple solution to this problem. It admits the use of data generated by arbitrary behavior policies and uses a learned prior -- the advantage-weighted behavior model (ABM) -- to bias the RL policy towards actions that have previously been executed and are likely to be successful on the new task. Our method can be seen as an extension of recent work on batch-RL that enables stable learning from conflicting data-sources. We find improvements on competitive baselines in a variety of RL tasks -- including standard continuous control benchmarks and multi-task learning for simulated and real-world robots.
1 INTRODUCTION
Batch RL learns from fixed data without new environment interaction, which is valuable when real-world data collection is costly. Standard off-policy methods can fail in continuous control, especially with conflicting multi-policy data; the paper proposes an advantage-weighted prior to keep learning near relevant, successful behavior.
- Motivation: Batch RL learns a policy from a fixed dataset without collecting new environment interactions.This setting is attractive for robotics and other domains where experimentation, hardware, and supervision are costly.
- Problem: Naïvely applying off-policy algorithms to continuous-control batch RL can perform dramatically worse than the behavior policy that generated the data.The central challenge is exploiting recorded information without inferring unsupported conclusions.
- Problem: Conflicting trajectories from policies solving different tasks or task variants further exacerbate batch-learning failures.Consistent but irrelevant behavioral patterns can mislead algorithms more than unpredictable random behavior.
- Contribution: The proposed solution learns an advantage-weighted behavior prior and constrains policy improvement toward relevant, previously successful actions.This is intended to keep the learned policy close to supported data while filtering behavior that is poor for the current task.
- Contribution: The method targets stable learning from conflicting data sources and is evaluated across continuous-control and simulated and real-robot tasks.The introduction reports improvements over competitive baselines across these settings.
2 BACKGROUND AND NOTATION
The paper formulates continuous-control RL as policy optimization in an MDP and extends the setup to multiple reward-conditioned tasks. In batch RL, the fixed dataset may come from arbitrary behavior policies, including policies unrelated to the target task.
- RL formulation: The environment is modeled as an MDP with continuous states, actions, transition dynamics, and a reward function.The objective is to find a policy maximizing cumulative discounted return.
- RL formulation: The state-action value function measures expected discounted return after taking an action and then following a policy.The paper relates this value function to the RL objective and optimal policy.
- Multi-task setting: For multi-task experiments, the policy and Q-function are conditioned on a task index, with an objective summing returns across tasks.Each task has its own reward function.
- Batch RL assumptions: Batch RL uses trajectory snippets from a fixed dataset collected beforehand by arbitrary behavior policies.The behavior policies may target the task of interest or generate trajectories unrelated to it.
3 A LEARNED PRIOR FOR OFFLINE OFF-POLICY RL FROM IMPERFECT DATA
The method stabilizes offline off-policy RL by constraining learned policies to actions supported by the batch, using a learned behavior prior in policy improvement. An advantage-weighted prior further emphasizes data that is better for the current task.
- Prior-constrained policy improvement: The learned policy is restricted toward the batch’s empirical state-conditional action support to avoid unsupported actions and overoptimistic Q-values.Policy iteration constrains improvement using a learned prior representing the batch distribution.
- Policy evaluation: Policy evaluation minimizes squared temporal-difference error using rewards that may be computed post-hoc rather than optimized by the behavior policies.The value estimate is updated from fixed trajectory data.
- Policy evaluation: The value estimate uses sampled actions from the current policy to approximate the expectation in V.The experiments use M = 20 samples, and the authors report stable learning when the policy is appropriately regularized.
- Prior-constrained policy improvement: Policy improvement solves a constrained optimization problem that keeps the new policy close to a separately learned prior.The prior is used instead of simply constraining the policy to remain close to its previous version.
- Prior learning: A raw behavior prior models observed actions, but it can prevent improvement over imperfect or unsuccessful behavior.This limitation motivates the advantage-weighted behavior model for diverse, multi-task datasets.
- Advantage-weighted behavior model: The advantage-weighted prior weights observed behavior snippets by their off-policy advantage over the current policy, emphasizing supported actions that are good for the current task.With the unit-step weighting choice, trajectories are progressively filtered toward snippets that do not underperform the current policy.
- Optimization variants: The prior and policy can be optimized with EM-style or stochastic value-gradient procedures, while a good enough prior permits skipping policy improvement when ϵ = 0.The simplified case learns the advantage-weighted behavior model and its Q-function.
4 RELATED WORK
Prior batch-RL work attributes continuous-control failures to inaccurate Q-values and out-of-distribution actions. This paper instead uses policy iteration with an adaptive, advantage-weighted behavior prior to remain near relevant data while handling conflicting behaviors.
- Batch-RL failure modes: Continuous-control off-policy algorithms commonly fail in fixed-batch settings because Q-values are inaccurate for state-action pairs absent from the data.Greedy exploitation can propagate these errors through Bellman backups and produce inappropriate actions.
- Existing approaches: Prior methods mitigate batch-RL failures by correcting optimistic Q-values and restricting policies toward actions likely to occur in the dataset.Examples include replacing maximization with generative action models or constraining policy actions near learned data distributions.
- Proposed approach: This work replaces Q-learning maximization with policy-iteration expectations and constrains improvement toward an adaptive prior that can focus on task-relevant behavior.The authors report that this combination avoids requiring special backup treatment when the policy is appropriately constrained and sufficiently sampled.
- Conflicting data: Conflicting or consistent-but-irrelevant behavioral patterns can mislead batch-RL methods more than unpredictable random behavior.The paper motivates adapting the prior to the target task rather than matching the entire empirical behavior distribution.
5 EXPERIMENTS
Experiments evaluate the method on continuous-control benchmarks, multi-task block manipulation, newly specified tasks, and a real Sawyer arm using fixed logged data. Behavioral priors, especially ABM, improve stability and performance, while real-robot offline learning reproduces tasks in under 12 hours.
- Experimental settings: The evaluation covers DeepMind control-suite benchmarks, seven simulated Sawyer manipulation tasks, newly specified block-placement tasks, and the corresponding real-robot setting.All compared algorithms use the same networks and implementation framework, differing in update rules.
- Control-suite experiments: In control-suite offline learning, BM+MPO performs as well as or better than complex baselines, and ABM+SVG provides further improvement in some domains.Standard MPO and SVG can learn some tasks with enough data but remain unstable.
- Simulated robot experiments: Behavioral-modelling priors improve learning across the seven simulated tasks, while ABM achieves high performance where the broader BM prior is hindered by conflicting trajectories.The RL policy using ABM still outperforms the prior itself.
- New offline tasks: ABM+MPO successfully learns bring-to-center and bring-to-corner tasks from data collected only for seven initial stacking tasks.The result requires trajectory snippets leading to successful execution of the new tasks to be present in the dataset.
- Real-robot experiments: The real Sawyer experiment re-learns all seven tasks from logged data in less than 12 hours, compared with over 200 hours for the original training run.The offline policy achieves similar performance on all tasks according to distributions of returns over 50 test episodes.
6 CONCLUSION
The approach uses an advantage-weighted behavior model as a prior for off-policy policy learning from logged experience. It is reported as robust to substantial sub-optimal data and effective across continuous-control and robot-manipulation settings.
- The approach learns an advantage-weighted behavior model that represents the data distribution and regularizes the reinforcement-learning policy toward it.This prior is intended to keep policy updates aligned with actions supported by logged experience.
- The method is designed for stable learning from logged experience with off-policy reinforcement-learning algorithms.
- The approach is reported to remain robust when the dataset contains large amounts of sub-optimal data.
- The method compares favorably with strong baselines on standard continuous-control benchmarks.
- Robot-manipulation experiments demonstrate that some tasks can be learned without observing a single trajectory for those tasks.
B.1 MPO
The MPO policy-improvement step first forms a non-parametric target policy by reweighting the prior with estimated action values. It then estimates the normalization and optimizes the parametric policy under trust-region constraints.
- MPO constructs the target policy by multiplying the prior action distribution by exponentiated estimated action values, scaled by a temperature and normalization term.The prior is πprior, action values are ˆQπi, η controls the temperature, and Z normalizes the distribution.
- The normalization constant is approximated from M actions sampled from the prior for each state.The implementation estimates Z using sampled prior actions rather than evaluating the integral directly.
- The temperature η depends on the desired KL constraint and is optimized by gradient steps using Adam.Optimization starts at η = 3 and projects η to at least 0.001 after each step.
- The parametric policy is fitted by minimizing KL(ˆπ∥πθ), equivalently maximizing the weighted log likelihood of sampled actions.The implementation uses M = 20 action samples, which may provide a crude representation of the behavior model at a state.
- A Lagrangian relaxation enforces a trust-region constraint on the policy update, with θ and the multiplier optimized by alternating Adam steps.The stated constraint is KL(πθi∥πθ) < ϵtrust, and α is projected to remain positive.
C.1 HYPERPARAMETERS AND NETWORK ARCHITECTURE
The experiments use shared network architectures and tuned algorithm-specific hyperparameters for the proposed method and the BCQ and BEAR baselines. BEAR uses a KL constraint to improve comparability.
- Hyperparameters for MPO, SVG, and BCQ single-task experiments are reported separately, with modified parameters used for multitask experiments.
- BCQ and BEAR were re-implemented in the same framework as the proposed algorithm using the same network architecture.Algorithm-specific hyperparameters were tuned by coarse grid search on control-suite tasks while following the original papers’ recommended ranges.
- BCQ uses perturbative actions with range [0.25, 0.25] and a VAE latent dimensionality of 64.
- BEAR uses a KL constraint instead of maximum mean discrepancy, with the same Lagrangian-multiplier optimization used for the proposed method.The authors report no instability issues from using the KL constraint.
C.3 DETAILS ON THE ROBOT EXPERIMENT SETUP
The robot experiments use a Sawyer arm with camera-tracked cubes and velocity control, evaluating seven staged manipulation tasks. MPO and SVG configurations use distinct hyperparameter tables, while multitask networks branch by task.
- MPO hyperparameters include 20 sampled actions per state, ϵ = 0.1, discount factor γ = 0.99, and Adam learning rate 2 × 10^-4.
- The robot tasks are used as an evaluation test-bed, and the authors make no claim to have contributed them specifically for this paper.
- The robot setup uses a Sawyer arm, Robotiq gripper, basket-mounted cameras, and three cubes whose proportions and basket sizes match simulation and reality.
- Robot control commands end-effector Cartesian velocities and gripper velocity, with execution at 20 Hz and episode termination above 20 N wrist force.
- Seven tasks are used, with the first six serving as auxiliary exploration tasks for the final Stack and Leave task.The task set includes reaching, grasping, lifting, placing, stacking, and stacking followed by moving the arm away.
- The multitask architecture branches into separate task heads selected by a one-hot task vector, with transitions duplicated in replay for independent task sampling.
D ADDITIONAL EXPERIMENTAL RESULTS
Additional experiments show that ABM improves performance over standard behavior-model priors, especially when conflicting data makes difficult tasks harder. Expanded plots also show ABM producing less constrained RL policies.
- Control suite: The learned advantage-weighted behavior model is far superior to the standard behavior-model prior on DeepMind control-suite tasks.The expanded plots show ABM in the left column and the resulting RL policy in the right column.
- Control suite: ABM produces less constrained RL policies than the standard behavior-model prior.
- Robot simulation: On simulated robot stacking, the simple behavioral model performs well on prerequisite tasks such as reaching and lifting.The task set contains seven intentions, with earlier tasks needed for most later tasks.
- Robot simulation: Conflicting data prevents the simple behavioral model from achieving high reward on more difficult stacking tasks, although it significantly improves the regularized policy.
D.3 PERFORMANCE TABLES FOR CONTROL SUITE AND ROBOT SIMULATION
Performance tables compare methods on control-suite and simulated-robotics tasks, while accompanying figures examine priors, policies, and a value-function-based simplification. ABM improves over BM particularly on difficult tasks, and RL policy improvement adds further gains there.
- Performance tables: ABM provides a performance boost over BM alone, particularly on difficult tasks such as block stacking and quadruped.Final performance is averaged over the final 10% of episodes.
- Performance tables: The RL policy further improves performance on difficult control-suite and simulated-robotics tasks.
- Simplified algorithm: The simplified ABM (ϵ = 0, |τ| = 2) option roughly recovers ABM+MPO performance but incurs some loss on the most complicated domains.Short trajectory snippets are vital in this setting so the prior can select the best action in each state.
- Expanded plots: Figure 7 compares the learned prior in the left column with the RL policy in the right column for each control-suite task.
- Expanded plots: Figure 8 compares prior execution on the left with the algorithm learned using that prior on the right across seven simulated block-stacking intentions.
- Value-function simplification: Learning the value function from the ABM prior removes its dependence on the RL policy and achieves similar, sometimes slightly better, performance on harder tasks.MPO policy improvement can still be performed using this value function while regularizing to the new prior.