Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning

Anusha Nagabandi; Gregory Kahn; Ronald S. Fearing; Sergey Levine

Source-linked AI summary

Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning

Anusha Nagabandi, Gregory Kahn, Ronald S. Fearing, Sergey Levine

arXiv:1708.02596v2Aug 8, 2017cs.LGcs.AIcs.RO

TL;DR

The paper targets the high sample complexity of model-free reinforcement learning and the difficulty of using expressive neural networks in sample-efficient model-based learning. It combines neural network dynamics with MPC and uses the resulting controller to initialize model-free policies. On MuJoCo locomotion tasks, the hybrid approach achieves 3−5× sample-efficiency gains, while the model-based approach learns useful gaits from random-action data.

Problem
Model-free reinforcement learning often requires millions of samples, while sample-efficient model-based methods have been difficult to extend to expressive deep neural network dynamics models.
Method
The paper combines medium-sized neural network dynamics models with MPC and uses model-based controller rollouts to behaviorally initialize a model-free policy.
Results
3−5× sample-efficiency gains were achieved on swimmer, cheetah, hopper, and ant MuJoCo benchmarks compared with pure model-free learning.
Takeaways & Limitations
The model-based method quickly extracts realistic gaits and can support different trajectory-following tasks, while model-based initialization improves model-free learning on benchmark locomotion tasks.
Takeaways & Limitations
The model-based approach cannot always reach extremely high rewards on its own, whereas model-free methods can become experts with more data.

Abstract

from arXiv · show

Model-free deep reinforcement learning algorithms have been shown to be capable of learning a wide range of robotic skills, but typically require a very large number of samples to achieve good performance. Model-based algorithms, in principle, can provide for much more efficient learning, but have proven difficult to extend to expressive, high-capacity models such as deep neural networks. In this work, we demonstrate that medium-sized neural network models can in fact be combined with model predictive control (MPC) to achieve excellent sample complexity in a model-based reinforcement learning algorithm, producing stable and plausible gaits to accomplish various complex locomotion tasks. We also propose using deep neural network dynamics models to initialize a model-free learner, in order to combine the sample efficiency of model-based approaches with the high task-specific performance of model-free methods. We empirically demonstrate on MuJoCo locomotion tasks that our pure model-based approach trained on just random action data can follow arbitrary trajectories with excellent sample efficiency, and that our hybrid algorithm can accelerate model-free learning on high-speed benchmark tasks, achieving sample efficiency gains of 3-5x on swimmer, cheetah, hopper, and ant agents. Videos can be found at https://sites.google.com/view/mbmf

I. INTRODUCTION

The paper addresses the high sample complexity of model-free reinforcement learning and the difficulty of applying expressive neural dynamics models in model-based learning. It combines neural network dynamics, MPC, and model-based initialization of model-free policies for complex locomotion.

Model-free reinforcement learning can learn complex skills but often requires millions of samples, while efficient model-based methods have traditionally relied on simpler function approximators.
Multi-layer neural network dynamics models achieve excellent sample complexity when combined with design decisions such as data aggregation.
The model-based controller uses MPC with random-sampling shooting to acquire locomotion gaits for swimmer, half-cheetah, hopper, and ant systems from entirely random-action data.
Less than four hours of random-action data was needed for each system, and the learned models supported trajectory following through sparse user-defined waypoints.The figure describes 7e5 time steps for the quadrupedal waypoint-following task.
The hybrid method uses model-based rollouts to initialize a model-free learner, addressing the model-based method’s lower asymptotic performance caused by model bias.
3−5× sample-efficiency gains were achieved on swimmer, cheetah, hopper, and ant benchmarks compared with pure model-free learning.
The work contributes neural-network model-based learning for contact-rich locomotion, empirical evaluation of dynamics-model design choices, and model-based initialization of model-free learning.

III. PRELIMINARIES

The preliminaries define reinforcement learning through discounted future rewards and introduce learned dynamics-based action selection. MPC repeatedly replans with the learned model, while the method predicts state changes rather than directly predicting the next state.

Reinforcement learning selects actions to maximize discounted future rewards under an unknown transition function, assuming the reward function is available here for planning.
A learned dynamics function ˆfθ takes the current state and action and estimates the next state at time t+∆t for model-based action selection.
MPC solves the finite-horizon action optimization at every time step, executes only the first action, and replans using updated state information.
The algorithm organizes its learned dynamics function, training procedure, policy extraction, and subsequent reinforcement-learning improvement as separate stages.
The neural dynamics model predicts the state change over ∆t and forms the next-state estimate as ˆs_t+1 = s_t + ˆfθ(s_t,a_t).Increasing ∆t provides more information per data point but also increases discretization and dynamics complexity.

B. Training the Learned Dynamics Function

The dynamics model is trained from random-action trajectories transformed into normalized state-difference examples. Training minimizes prediction error, while multi-step validation evaluates long-horizon open-loop predictions without being used for training.

Training data comes from trajectories generated by sampling initial configurations and executing random actions, despite differing from trajectories later produced for reward-based planning.
Trajectory slices use (s_t,a_t) as inputs and s_t+1−s_t as labels, with normalization and Gaussian noise added before storing the dataset D.
The dynamics model is trained with stochastic gradient descent by minimizing prediction error, while mean squared error is also computed on held-out validation trajectories.
H-step validation propagates the learned model forward using a fixed action sequence and compares predicted states with the corresponding ground-truth states.
The multi-step validation error is used to analyze experimental results but is not used during training.

C. Model-Based Control

The method uses a learned dynamics model within finite-horizon MPC to select actions by evaluating sampled action sequences. It repeatedly executes only the first selected action and replans from the updated state.

MPC optimizes finite-horizon action sequences using the learned dynamics model to predict future states and maximize cumulative reward.Exact optimization is difficult for nonlinear dynamics and rewards, so the method uses approximate control.
Random-sampling shooting generates K candidate action sequences, predicts their state sequences, and selects the sequence with the highest expected cumulative reward.
The algorithm begins with random trajectories, trains the dynamics model, and iteratively adds controller-generated data while retraining the model.
At each timestep, the controller executes the first action from the selected sequence, then recalculates the optimal sequence using the updated state.
For higher-dimensional action spaces and longer horizons, random-sampling MPC may be insufficient, motivating investigation of alternative methods.
Changing the reward function allows the trained model-controller combination to accomplish varied runtime goals without live task-specific retraining.

D. Improving Model-Based Control with Reinforcement

On-policy data aggregation improves the learned model by alternating controller rollouts with retraining on aggregated data. The resulting model-based controller can also provide demonstrations for initializing a model-free learner.

On-policy data aggregation mitigates mismatch between the training data distribution and the model-based controller’s state-action distribution.
The algorithm combines initial random data with separately stored MPC-generated data, retraining the dynamics model on both datasets.Model weights are warm-started from the previous iteration during retraining.
The model-based reinforcement learning algorithm learns complex gaits using very small numbers of samples compared with purely model-free learners.
The model-based controller supplies expert trajectories that are collected into D* and used to train a neural-network policy through behavioral cloning.
DAGGER alternates policy training, on-policy rollouts, expert action queries, and policy retraining to address the data distribution problem.

B. Model-Free Reinforcement Learning

The model-free learner is initialized with a policy trained to imitate the model-based controller, then fine-tuned with TRPO. Experiments evaluate this approach across MuJoCo locomotion agents and model-design choices.

TRPO uses the imitation policy as its initial policy instead of a randomly initialized policy.The choice avoids requiring a critic or value function for initialization.
The evaluated MuJoCo agents are swimmer, hopper, half-cheetah, and ant, with state and action dimensions varying across systems.
The study evaluates model-design decisions on swimmer and half-cheetah locomotion tasks, using the best outcome from each evaluation thereafter.
20 training epochs caused swimmer to reach only half of the performance achieved in the other training-step experiments.
Mostly aggregated on-policy rollouts significantly improved performance when training the learned dynamics function.
Too short a controller horizon harmed performance, while half-cheetah performed worse with longer horizons.

B. Trajectory Following with the Model-Based Controller

The model-based controller uses learned dynamics with MPC to follow varied trajectories and produce locomotion gaits from limited data. Its short-horizon control can succeed despite open-loop prediction errors, while hybrid model-based/model-free learning improves sample efficiency on benchmark tasks.

Model prediction and control: Open-loop multi-step predictions eventually diverged on some half-cheetah state elements, but short-horizon MPC still controlled the agent successfully.The controller uses the model locally rather than relying on long open-loop predictions.
Trajectory-following capability: The learned dynamics model supported trajectory-following tasks for swimmer, ant, and half-cheetah agents, including curved paths and U-turns.The model was trained once per agent using only random initial trajectories, then reused at run time for different tasks.
Reward design: The trajectory-following reward penalized perpendicular distance from the desired path and encouraged forward movement along it, without specifying limb motions.The model-based algorithm therefore had to discover a suitable gait for the desired center-of-mass trajectory.
Benchmark comparison: The controller was evaluated against TRPO on standard MuJoCo locomotion tasks requiring swimmer, half-cheetah, hopper, and ant agents to learn fast forward gaits.The benchmark reward functions primarily incentivized speed, creating challenges for the short-horizon controller.
Benchmark comparison: 20× fewer data points produced a stable swimmer gait than model-free TRPO, while hybrid Mb-Mf learning enabled TRPO to reach maximum performance 3× faster.Across the benchmark agents, the combined approach achieved 3−5× sample-efficiency gains over pure model-free methods.
Hybrid fine-tuning: The hybrid approach fine-tuned quickly learned model-based gaits with model-free learning, addressing the model-based method’s insufficient final rewards.For hopper, the pure model-based method learned double or triple hops quickly but plateaued because forward velocity alone did not keep it upright over longer periods.

VII. DISCUSSION

The method learns neural-network dynamics models efficiently for complex locomotion, then uses them for trajectory following or model-free fine-tuning. Its model-based controller quickly produces competent gaits, while hybrid fine-tuning improves task rewards and sample efficiency.

Model-based learning: The algorithm learns neural-network dynamics models for complex simulated locomotion using relatively few samples.The authors report performance exceeding the complexity demonstrated in prior neural-network model-based methods.
Model-based learning: Design choices for training dynamics models were evaluated experimentally, and the learned model quickly produced an effective gait.The resulting gait could support different trajectory-following tasks at run-time.
Model-based learning: The model-based approach can extract complex, realistic gaits quickly, but it does not always reach extremely high rewards on its own.The paper contrasts rapid competence from model-based learning with the higher eventual expertise of model-free methods.
Practical implications: Using 14% of the data required by model-free learning for ant walking, the model-based approach enabled travel in any direction and along arbitrary desired trajectories.The model can be trained once and applied to multiple tasks, which the authors identify as relevant to real-robot applications.
Future work: Further work should integrate model-based learning more tightly with model-free learners to seek additional sample-efficiency gains.The proposed approach is described as simple and effective, but tighter integration remains an open direction.
Future work: Real-world deployment must address communication delays, computational limits, and the feasibility of the paper’s random-sampling MPC controller.The authors suggest optimization-based real-time controllers as future work, especially for high-dimensional action spaces.

APPENDIX

The appendix specifies the data collection, state initialization, neural-network architecture, optimization settings, and data-aggregation parameters used in the experiments.

Data collection: Initial datasets were collected from rollouts using random actions sampled approximately uniformly from [−1, 1].Starting-state noise was added to promote exploration of the state space.
Data collection: Swimmer trajectory-following used additional heading noise sampled uniformly from (−π, π) to support arbitrary future trajectories.This was the exception to the standard starting-state noise used for the agents.
Dynamics model: The dynamics network used two hidden layers of dimension 500 with ReLU activations.Inputs and outputs were standardized before training, using Adam with learning rate 0.001 and batch size 512.
Experimental parameters: The appendix records whether data aggregation was performed and notes that some rollout lengths varied with agent termination conditions.The listed parameter tables mark unavailable aggregation or rollout values as N/A where applicable.

B. Experimental Details for Hybrid Mb-Mf approach

For the hybrid Mb-Mf approach, MPC rollouts are converted into an imitation policy that initializes TRPO, while trajectory-following rewards evaluate predicted action sequences against desired paths.

Moving forward: MPC rollouts for forward motion were collected with small Gaussian action noise, then used to train a Gaussian policy for TRPO initialization.Noise with distribution N(0, 0.005) promoted exploration while preserving useful behavior.
Moving forward: The imitation policy used tanh nonlinearities and two hidden layers of dimension 64, with Adam optimization at learning rate 0.0001.Its standard-deviation parameter was also important for subsequent TRPO performance.
Moving forward: TRPO used batch size 50000, base eps 1e−5, discount factor 0.995, and step size 0.5 for all agents.These were the reported common parameters across the benchmark agents.
Trajectory following: Trajectory-following reward evaluation assigns value to an action sequence using sparse desired center-of-mass positions.The desired trajectory is represented through waypoints converted into line segments.
Trajectory following: The trajectory reward penalizes perpendicular distance from the desired path while encouraging forward progress along it.The reward is computed over predicted states generated by the learned dynamics model for each candidate action sequence.
Trajectory following: The trajectory-following reward is an example that can be extended with penalties for behaviors such as jumping too high or falling.The paper notes that standard MuJoCo rewards likewise include tuned components such as terminal conditions.

2) Moving Forward:

The appendix states that moving-forward experiments use the standard MuJoCo reward functions for individual time steps and combine them into sequence rewards.

Moving Forward: Moving-forward experiments use the standard MuJoCo reward functions r_t(s_t, a_t).The resulting sequence reward R corresponds to the rewards associated with a given action sequence.