Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control

Kendall Lowrey; Aravind Rajeswaran; Sham Kakade; Emanuel Todorov; Igor Mordatch

Source-linked AI summary

Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control

Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, Igor Mordatch

arXiv:1811.01848v3Nov 5, 2018cs.LGcs.AIcs.ROstat.ML

TL;DR

Agents with limited resources must continually act and learn in complex worlds where exact optimal behavior is infeasible. POLO combines local trajectory optimization, global value-function approximation, and uncertainty-aware exploration, enabling control of complex humanoid and dexterous-hand agents while assuming an accurate internal dynamics model.

Problem
Agents must continually act and learn in complex, diverse worlds despite limited memory and computation and incomplete coverage of possible situations.
Method
POLO combines local trajectory optimization, global value-function approximation, and uncertainty- and reward-aware exploration for continual control and learning.
Results
POLO enabled control of complex agents including a 3D humanoid and a five-fingered hand, while empirical evaluation examined coordinated exploration, reduced MPC horizons, and faster, more stable value-function learning.
Takeaways & Limitations
Trajectory optimization and uncertainty-aware exploration can be combined with value-function learning to support complex continual-control tasks.
Takeaways & Limitations
The work assumes access to an accurate internal dynamics model and leaves improving that model through real-world interaction as future work.

Abstract

from arXiv · show

We propose a plan online and learn offline (POLO) framework for the setting where an agent, with an internal model, needs to continually act and learn in the world. Our work builds on the synergistic relationship between local model-based control, global value function learning, and exploration. We study how local trajectory optimization can cope with approximation errors in the value function, and can stabilize and accelerate value function learning. Conversely, we also study how approximate value functions can help reduce the planning horizon and allow for better policies beyond local solutions. Finally, we also demonstrate how trajectory optimization can be used to perform temporally coordinated exploration in conjunction with estimating uncertainty in value function approximation. This exploration is critical for fast and stable learning of the value function. Combining these components enable solutions to complex simulated control tasks, like humanoid locomotion and dexterous in-hand manipulation, in the equivalent of a few minutes of experience in the real world.

1 Introduction

POLO addresses continual acting and learning under limited resources by coupling local trajectory optimization, global value-function learning, and exploration. The framework uses this coupling to support directed exploration and learning across diverse tasks despite an accurate nominal dynamics model assumption.

1 Introduction: An agent with limited memory and computation must act and learn in complex worlds with diverse tasks, even when it has only a nominal dynamics model.The state space and task set may be too large for an agent that has experienced all situations and can act optimally everywhere.
1 Introduction: Trajectory optimization can provide a vehicle for exploration and learning when more powerful planning procedures are computationally infeasible or insufficient.The setting assumes the agent has dynamics knowledge and computational resources for local search, while recognizing that complex environments may limit optimizer effectiveness.
1 Introduction: POLO couples local trajectory optimization, global value-function learning, and exploration for continual acting and learning.The framework is motivated by agents becoming moderately competent on infrequent tasks and more skillful through repeated experience.
1 Introduction: POLO uses trajectory optimization to execute temporally coordinated action sequences based on hypotheses about potential reward regions.This directed exploration contrasts with ϵ−greedy and Boltzmann strategies that explore one timestep at a time.
1 Introduction: The framework studies continual control with an accurate internal nominal dynamics model, linking local trajectory optimization for fast improvement with learning-based generalization.The paper identifies improving an approximate internal model over time from real-world interaction as a future direction.

2 The POLO framework

POLO combines local trajectory optimization, global value-function learning, and uncertainty-aware exploration for continual acting and learning with an internal dynamics model. The framework uses these components to reduce approximation-error sensitivity, accelerate value learning, and coordinate exploration.

POLO combines local trajectory optimization, global value function approximation, and uncertainty- and reward-aware exploration.The framework is designed for agents that must act and learn continually in complex worlds with limited computational resources.
Value Function Approximation: Greedy actions from an approximate value function can suffer performance degradation that scales with the effective problem horizon, especially when γ ≈1.The induced greedy policy can exploit approximation errors, and the bound can be tight for some MDPs and approximations.
Trajectory Optimization and MPC: MPC with a terminal approximate value function is less susceptible to approximation errors than greedy action selection.For H > 1, planning over multiple steps reduces the sensitivity described for one-step greedy control.
Accelerating Value Learning: The H-step Bellman operator contracts errors by γ^H, allowing global information to propagate for H steps and potentially accelerating value-function convergence.POLO realizes this propagation through trajectory optimization, while its overall procedure uses trajectory-optimization-generated targets for value fitting.
Planning to Explore: MPC enables temporally coordinated exploration by optimizing trajectories toward hypothesized rewarding regions rather than selecting exploratory actions independently at each timestep.POLO estimates uncertainty by independently training K value-function approximators and uses their posterior samples for action selection.
Final Algorithm: The final procedure constructs value targets by solving an N-step trajectory-optimization problem, which the paper reports accelerates convergence and stabilizes learning.The targets are computed for each state in the buffer and value network, then used to fit the value approximation.

3 Empirical Results and Discussion

The empirical results evaluate POLO’s coordinated exploration, horizon reduction, and value-learning benefits across point-mass, humanoid, and in-hand manipulation tasks. POLO achieves temporally coordinated exploration, outperforms pure MPC at matched horizons, and accelerates and stabilizes value-function learning.

POLO’s evaluation asks whether uncertainty-aware trajectory optimization coordinates exploration, reduces MPC’s planning horizon, and accelerates stable value-function learning.
3.1 Trajectory optimization for exploration: POLO achieves more region coverage than random or greedy exploration in a reward-free 2D navigation task by using uncertainty with 32-step MPC.The value function assigns low values to visited regions and preserves high values for unexplored regions.
3.2 Value function approximation for trajectory optimization: POLO uniformly dominates pure MPC at matched planning horizons on humanoid getup and in-hand manipulation tasks.The experiments use 12000 and 2500 environment timesteps, respectively.
3.2 Value function approximation for trajectory optimization: A learned POLO value function enables MPC to increase humanoid height without per-time-step rewards, indicating task-relevant value estimates.
3.3 Trajectory optimization for value function learning: Increasing the N-step trajectory-optimization horizon lets the agent learn the value function with fewer world interactions.The reported difference of 0.2 reward relative to MPC corresponds to approximately 50% performance improvement.
3.3 Trajectory optimization for value function learning: Longer planning horizons mitigate degradation when a value function trained on a nominal humanoid model is used with models having larger heads.The result suggests a possible role for MPC in transfer between tasks or robot platforms.

4 Related Work

The related work situates POLO among planning, exploration, model-free reinforcement learning, and learned-model approaches. It distinguishes POLO’s temporally intentional MPC-based exploration and accurate internal-model setting from these alternatives.

Planning and learning: Prior planning methods combine search with approximate value functions or use trajectory libraries, but do not explicitly cover POLO’s full continual learning setting.
Planning and exploration: Per-time-step exploration methods can become intractable for sparse or delayed rewards in high-dimensional settings and may produce inefficient back-and-forth wandering.
Planning and exploration: Parameter-space exploration generates correlated behavior but treats exploration as deviation from an objective rather than an intentional act.
Model-free RL: POLO reports 96 seconds of experience and 24 CPU core hours for getting-up tasks, compared with approximately 5 days of experience and 128 CPU core hours reported for model-free methods.
Model-free RL: For similar in-hand reorientation tasks, POLO required 1 CPU hour versus 3 years of simulated experience and 500 CPU hours reported for policy-gradient methods.
Learned internal models: Other model-based approaches learn internal dynamics models for variance reduction or short-horizon action selection, whereas POLO uses an internal model with MPC and value learning.

5 Conclusions and Future Work

The paper concludes that POLO combines trajectory optimization, value-function learning, and uncertainty-aware exploration to control complex simulated agents. Its principal scope boundary is the assumption of an accurate internal dynamics model.

POLO combines trajectory optimization and value-function learning with uncertainty-aware exploration to control 3D humanoid and five-fingered hand agents.
The experiments assume access to an accurate internal dynamics model.Future work is to study internal-model approximation errors and improve the model using real-world interaction data.

A Appendix: Experimental Details, Humanoid

The humanoid experiments use a torque-actuated MuJoCo model with specified observations, MPPI settings, and an ensemble value-function learner. Training and evaluation use fixed timestep, horizon, and interaction-budget details.

The humanoid model has 27 degrees of freedom, uses direct torque actuation, and advances with a 0.008-second timestep.Actuation inputs are limited to ±1.0 while the original gear ratios remain unchanged.
The value function uses easily observed robotics quantities, including directions, heights, root velocities, and touch sensors.
The humanoid value-function configuration uses γ=0.99, a planning horizon of 64, 120 MPPI rollouts, noise σ=0.2, and temperature 1.25.
POLO approximates humanoid values with an ensemble of 6 two-layer neural networks using tanh nonlinearities.Training applies 64 ADAM gradient steps to minibatches of size 32 every 16 experienced timesteps.
Reset experiments use 20 episodes of 600 timesteps, totaling 12000 agent timesteps or 96 seconds.No-reset experiments use the same total timestep budget and share an actuator-force control cost.

A.1 Humanoid Getup

The humanoid getup task requires the agent to stand from a supine position and reach a root height of 1.1 meters. Its non-sparse reward makes learning difficult by favoring delayed standing over immediate sitting.

The agent starts supine and must bring its root height to 1.1 meters.
In the non-sparse case, the agent must forgo immediate sitting reward to discover the delayed standing reward.
The sparse reward assigns 1.0 when Rootz exceeds 1.1 and otherwise uses a height-dependent reward.

A.2 Humanoid Walk

The humanoid walking task rewards maintaining a target height or walking at 1.0 meters per second along a straight path. A target speed is used to encourage stable gaits rather than maximum speed.

The walking reward penalizes deviation from a target height of 1.1 meters or from a target speed of 1.0 meters/second and the world’s x-axis.
The target speed is chosen instead of maximum-speed reward to encourage stable walking gaits.

A.3 Humanoid Box

The humanoid box task requires pushing a 0.9 meter cube to a specified point. Very low friction makes most pushes send the box out of reach, so POLO learns to limit the initial push.

The task places a 0.9 meter cube in front of the humanoid, which must push it to a specific point.
Because friction is very low and most pushes send the box out of reach, POLO learns to limit the initial push.
The reward includes terms for humanoid height, box distance from the humanoid, and box distance from the target under different conditions.
The box starts 1.5 meters in front of the humanoid and must be navigated to a position 2.5 meters in front.

B Appendix: Experimental Details, Hand Manipulation

The hand-manipulation setup uses a position-controlled Adroit hand to reorient a free dice object toward randomized desired configurations. Experiments use MPPI with an ensemble value function and periodic updates.

Hand Manipulation: A position-controlled five-fingered hand reorients a free dice object whose desired configuration changes every 75 timesteps.
Hand Manipulation: The reward penalizes Cartesian position error and quaternion orientation difference between the dice and its goal.
Experimental Details: The setup uses 80 MPPI trajectories, an ensemble of 6 networks, and value updates every 25 interaction steps.
Experimental Details: Each value-function update uses 16 gradient steps with batch size 16, after coarse hyperparameter search.

C Lemma 2

Lemma 2 characterizes when trajectory optimization with an approximate terminal value function can support near-optimal control. Its bound requires closed-loop optimization in general, while deterministic dynamics remove that requirement, and MPC can optimize the approximate value over H steps.

The bound compares trajectories generated by the approximate-value MPC policy and the optimal policy over a finite horizon H.The comparison starts from a state s and uses trajectories of length H.
For the bound to hold generally, actions must be optimized in closed loop over feedback policies rather than open-loop actions.This requirement is stated as necessary for equation (13) and the overall bound.
Under deterministic dynamics, equation (13) holds without the closed-loop requirement, so trajectory optimization methods such as iLQG and MPPI can be used.For finite horizon H, optimal open-loop trajectories and local feedback policies have the same performance in this setting.
With stochastic dynamics and additive Gaussian noise, local dynamic programming methods such as iLQG and DDP efficiently optimize trajectories while providing local feedback policies.
Because MPC is globally non-parametric, it can simultaneously optimize the approximate value function over H steps, unlike some parametric policy classes.This property supports the paper's use of MPC in deriving the bound.