PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Difference Models: Model-Free Deep RL for Model-Based Control

Vitchyr Pong, Shixiang Gu, Murtaza Dalal, Sergey Levine

arXiv:1802.09081v2cs.LG

TL;DR

Model-free RL is often sample-inefficient, whereas model-based RL can suffer asymptotic bias from imperfect dynamics models. The paper introduces temporal difference models, goal-conditioned value functions trained with model-free learning and used for model-based control. Across continuous-control tasks, TDMs combine model-based-like sample complexity with final performance comparable to or better than model-free methods.

  • Problem

    Model-free RL uses experience inefficiently, while model-based RL can lose final performance because learned dynamics are imperfect.

  • Method

    TDMs are variable-horizon, goal-conditioned value functions trained with off-policy model-free learning and used as implicit dynamics models for MPC-like control.

  • Results

    Across continuous-control tasks, TDMs achieve sample complexity comparable to model-based methods while attaining asymptotic performance comparable to or better than model-free methods.

  • Takeaways & Limitations

    TDMs provide a model-free-trained route to efficient model-based control while reducing the effects of model bias.

  • Takeaways & Limitations

    The long-horizon formulation does not optimize intermediate rewards and is therefore limited to terminal-reward problems.

Abstract

from arXiv · show

Model-free reinforcement learning (RL) is a powerful, general tool for learning complex behaviors. However, its sample efficiency is often impractically large for solving challenging real-world problems, even with off-policy algorithms such as Q-learning. A limiting factor in classic model-free RL is that the learning signal consists only of scalar rewards, ignoring much of the rich information contained in state transition tuples. Model-based RL uses this information, by training a predictive model, but often does not achieve the same asymptotic performance as model-free RL due to model bias. We introduce temporal difference models (TDMs), a family of goal-conditioned value functions that can be trained with model-free learning and used for model-based control. TDMs combine the benefits of model-free and model-based RL: they leverage the rich information in state transitions to learn very efficiently, while still attaining asymptotic performance that exceeds that of direct model-based RL methods. Our experimental results show that, on a range of continuous control tasks, TDMs provide a substantial improvement in efficiency compared to state-of-the-art model-based and model-free methods.

1 INTRODUCTION

Deep model-free RL can learn complex behaviors but often requires too much real-world experience, while model-based RL learns efficiently yet suffers asymptotic model bias. TDMs are introduced to bridge these trade-offs through goal-conditioned value functions usable for model-based control.

  • Deep RL often requires extensive experience, limiting applications where data must be collected directly on physical systems.
  • Model-based RL gains supervision from state transitions and learns efficiently, but imperfect dynamics models can produce highly suboptimal final policies.
  • The paper asks whether model-based learning efficiency can be combined with model-free asymptotic performance.
  • TDMs use variable-horizon goal-conditioned value functions that interpolate between direct models and more model-free formulations.
  • TDMs are trained efficiently and used with MPC-like control, improving sample complexity over fully model-free methods while exceeding purely model-based methods in final performance.

2 PRELIMINARIES

The paper reviews Q-functions, off-policy Q-learning, model-based dynamics prediction, and implicit dynamics constraints as foundations for connecting model-free learning with planning.

  • RL formalizes decision making using states, actions, transition dynamics, initial-state distributions, and reward functions.
  • Q-functions: Q-learning estimates optimal action values from transition tuples collected by arbitrary exploration policies through off-policy optimization.
  • Model-based RL and optimal control: Model-based RL learns a next-state predictor and can use it for planning, including receding-horizon model-predictive control.
  • Model-based RL and optimal control: Implicit dynamics encode transitions as constraints requiring C(s_i, a_i, s_i+1) = 0 exactly when s_i+1 = f(s_i, a_i).
  • Goal-conditioned value functions: Goal-conditioned value functions condition Q-values on goal vectors, enabling value prediction for multiple task descriptions.

3 TEMPORAL DIFFERENCE MODEL LEARNING

TDM learning connects goal-conditioned value functions to implicit dynamics models by using distance-to-goal rewards and an explicit planning horizon. Longer horizons support progressively more model-free, coarser-resolution planning, with a terminal-reward scope limitation.

  • 3 TEMPORAL DIFFERENCE MODEL LEARNING: TDMs connect model-based MPC optimization to goal-conditioned value functions and extend that connection toward model-free learning.
  • 3.1 FROM GOAL-CONDITIONED VALUE FUNCTIONS TO MODELS: With state goals and distance-based rewards, the τ = 0 value function directly identifies whether the next state reaches the goal.
  • 3.1 FROM GOAL-CONDITIONED VALUE FUNCTIONS TO MODELS: The resulting goal-conditioned value function can serve as an implicit model for MPC, although the basic connection lacks long-horizon capability.
  • 3.2 LONG-HORIZON LEARNING WITH TEMPORAL DIFFERENCE MODELS: TDMs replace discounted reward aggregation with a horizon input τ, using distance-to-goal reward at τ = 0 and decrementing τ on earlier steps.
  • 3.2 LONG-HORIZON LEARNING WITH TEMPORAL DIFFERENCE MODELS: As TDMs become effective over longer horizons, planning can optimize every Kth state and action, eventually using a single effective time step.
  • 3.2 LONG-HORIZON LEARNING WITH TEMPORAL DIFFERENCE MODELS: The long-horizon formulation does not optimize intermediate rewards, limiting it to terminal-reward problems despite allowing arbitrary terminal-state rewards.

4 TRAINING AND USING TEMPORAL DIFFERENCE MODELS

TDMs use off-policy Q-learning with relabeled goals and horizons, vector-valued distance prediction, and state prediction architectures to support efficient training and MPC-style policy extraction.

  • 4 TRAINING AND USING TEMPORAL DIFFERENCE MODELS: TDMs can be trained with standard off-policy Q-learning while relabeling goals and horizons for each transition.
  • 4 TRAINING AND USING TEMPORAL DIFFERENCE MODELS: Relabeling enables simultaneous learning of short- and long-horizon behaviors for arbitrary goals from the same behavioral-policy transitions.
  • 4 TRAINING AND USING TEMPORAL DIFFERENCE MODELS: Vector-valued rewards train per-dimension distance predictions, increasing supervision when the distance factors additively across state dimensions.
  • 4 TRAINING AND USING TEMPORAL DIFFERENCE MODELS: When task rewards depend on selected state features, TDMs can predict only those dimensions to simplify the prediction problem.
  • 4.2 POLICY EXTRACTION WITH TDMS: Defining Q(s, a, s_g, τ) = −∥f(s, a, s_g, τ) − s_g∥ trains f to predict the state reached after τ steps and supports explicit MPC.
  • 4 TRAINING AND USING TEMPORAL DIFFERENCE MODELS: The algorithm alternates noisy MPC-based exploration, replay-buffer storage, relabeling, and repeated Q-function updates.

5 RELATED WORK

The paper situates TDMs among approaches combining model-based and model-free reinforcement learning, while distinguishing its goal-conditioned-value-function formulation from prior methods.

  • Prior hybrid methods simulate experience, improve model-free gradients, or correct model errors, whereas TDMs propose an equivalence through goal-conditioned value functions.
  • Earlier theory established model extraction from tabular value functions and convergence equivalence for linear approximators, but the paper distinguishes its contribution from these results.
  • TDMs predict future states under a policy pursuing a goal, unlike multi-step models that require optimization over action sequences.
  • Compared with HER, TDMs use multiple temporal scales and explicitly connect goal-conditioned value functions with model-based and model-free learning.
  • Other methods enhance model-free supervision through multi-step reward predictions or auxiliary pixel-control objectives, but these differ substantially from TDMs.

6 EXPERIMENTS

Experiments compare TDMs with model-free, model-based, and goal-conditioned baselines across simulated and real-world continuous-control tasks, while testing key design choices.

  • 6.1 TDMS VS MODEL-FREE, MODE-BASED, AND DIRECT GOAL-CONDITIONED RL: TDMs learn as quickly as or faster than model-based methods and achieve final policies at least as good as model-free methods across tasks.
  • 6.1 TDMS VS MODEL-FREE, MODE-BASED, AND DIRECT GOAL-CONDITIONED RL: TDMs require fewer samples than model-free baselines on ant tasks and drastically fewer samples on the other evaluated tasks.
  • 6.1 TDMS VS MODEL-FREE, MODE-BASED, AND DIRECT GOAL-CONDITIONED RL: Model-based learning is faster than model-free baselines, but its final performance is worse on harder cheetah and ant tasks because of model bias.
  • 6.1 TDMS VS MODEL-FREE, MODE-BASED, AND DIRECT GOAL-CONDITIONED RL: The experiments cover reaching, puck pushing, target-velocity cheetah locomotion, ant target control, and real-world Sawyer reaching.
  • 6.1 TDMS VS MODEL-FREE, MODE-BASED, AND DIRECT GOAL-CONDITIONED RL: HER does not improve over DDPG on tasks DDPG already solves, but significantly outperforms DDPG-Sparse on sparse tasks.
  • 6.1 TDMS VS MODEL-FREE, MODE-BASED, AND DIRECT GOAL-CONDITIONED RL: On the 7-DoF Sawyer arm, TDMs outperform DDPG while using fewer samples, demonstrating performance on a real-world analogue of reaching.
  • 6.2 ABLATION STUDIES: The vectorized reward formulation learns substantially faster than the scalar variant, while horizon choice trades off learning behavior between model-based and model-free approaches.
  • 6.2 ABLATION STUDIES: Performance degrades when τmax is too low, whereas learning slows when τmax is too high.

7 CONCLUSION

The conclusion presents TDMs as a connection between model-based and model-free RL that combines efficient training with favorable asymptotic performance. It also identifies extensions to planning, representations, and stochastic settings as future directions.

  • TDMs function both as goal-conditioned value functions and implicit dynamics models, enabling efficient off-policy training while reducing model-bias effects.
  • TDMs achieve asymptotic performance comparable to model-free algorithms with sample complexity comparable to purely model-based methods.
  • Future work includes integrating TDMs with constrained model-predictive control or trajectory-optimization methods.
  • Applying TDMs to image-based tasks may require effective metric embeddings because simple distance metrics may not work for complex state representations.
  • The presentation assumes deterministic environments, although the paper states that stochastic extensions could predict expected distances to goals.

A EXPERIMENT DETAILS

The experiment-details section introduces the experimental setups and reports an update-count study for TDMs on the ant target-position task.

  • Figure 4 varies the number of updates per step I for TDMs on the ant target-position task.
  • The Figure 4 experiment uses a maximum distance of 5 instead of 6, so its numbers should be lower than those reported in the paper.

A.1 GOAL STATE AND τ SAMPLING STRATEGY

Training efficiency depends on how goal states and time horizons are sampled for each transition tuple. The appendix lists several goal-sampling strategies.

  • Goal State and τ Sampling Strategy: Q-learning remains valid for any goal state sg and horizon τ assigned to a transition tuple.The sampling procedure can nevertheless affect learning efficiency.
  • Goal State and τ Sampling Strategy: Candidate goal-sampling strategies include future states on the same trajectory, replay-buffer states, and goals sampled from a uniform range.Future-state sampling selects sg = st+k for a random k > 0.

A.2 MODEL-FREE SETUP

The model-free setup uses DDPG as the base algorithm for learning TDMs, together with large replay buffers and soft target networks. TDMs and baselines use different Polyak coefficients.

  • Model-Free Setup: DDPG is used as the base off-policy model-free algorithm for learning TDMs Q(s, a, g, sτ).The setup uses experience replay with 1 million transitions.
  • Model-Free Setup: Soft target networks use Polyak averaging coefficients of 0.999 for DDPG and TDM, and 0.95 for HER and DDPG-Sparse.HER and DDPG-Sparse also use a penalty on tanh pre-activation.
  • Model-Free Setup: The TDM reward uses the L1 distance −∥st+1 − sg∥1.This choice ensures consistency between scalar and vectorized TDMs.

A.3 MODEL-BASED SETUP

The model-based baseline learns a neural dynamics model from normalized replay data and uses model predictive control to select actions. MPC evaluates random action sequences through the learned model.

  • Model-Based Setup: The dynamics model uses ReLU activations, two hidden layers of size 300, and no output activation.It predicts state differences rather than full states and minimizes mean squared prediction error.
  • Model-Based Setup: States, actions, and state differences are normalized before model input or loss computation.The model is trained from replay-buffer minibatches of size 128.
  • Model-Based Setup: MPC simulates 512 random action sequences of length 15 and executes the first action from the highest-reward sequence.The sequences are simulated through the learned dynamics model.

A.4 TUNED HYPERPARAMETERS

The appendix describes TDM parameterization, policy extraction, task-specific goal choices, and benchmark settings. TDMs use a distance-based value representation, while control optimization exploits task goals and state structure.

  • TDM Network Architecture and Vector-Based Supervision: TDMs parameterize Q(s, a, sg, τ) as −∥f(s, a, sg, τ) − sg∥1, making outputs non-positive and encouraging prediction of the state reached after τ steps.The vector f has the same dimension as the goal.
  • TDM Network Architecture and Vector-Based Supervision: Vectorized supervision trains each component fj against the corresponding goal-coordinate distance, with a∗ chosen by maximizing Q at the next state.The supervision is applied separately for each state dimension j.
  • Task and Reward Descriptions: Reaching and pushing use task-specific policy extraction, while benchmark horizons are 50 for pusher and ant, 100 for reaching, and 99 for half-cheetah.The benchmarks use MuJoCo physics and OpenAI Gym environments.
  • Task and Reward Descriptions: For the 7-DoF reacher, only the first 14 terminal state dimensions are optimized, while the target XYZ position is fixed.The method samples 10,000 candidate joint-angle and joint-velocity vectors and selects the best value.
  • Task and Reward Descriptions: Pusher goal states first target the puck's current location and then switch to the target location after the hand reaches the puck.The hand goal is set to the puck location, and no other state dimensions are optimized.
  • Task and Reward Descriptions: Half-cheetah sets the goal velocity to the desired velocity v∗, sampled uniformly from [−6, 6], making optimal control trivial.The resulting finite-horizon TDM policy tends to jump at the last time step.
  • Task and Reward Descriptions: The ant task lowers the gear ratio to 30 for all joints to reduce frequent flipping early in training, and Sawyer adds measured torques to its state.Ant rewards use the L1 norm between actual and desired torso position and velocity, while Sawyer uses L1 distance to the desired XYZ position.
Loading 1802.09081v2…