When to use parametric models in reinforcement learning?

Hado van Hasselt; Matteo Hessel; John Aslanides

Source-linked AI summary

When to use parametric models in reinforcement learning?

Hado van Hasselt, Matteo Hessel, John Aslanides

arXiv:1906.05243v1Jun 12, 2019cs.LGcs.AIstat.ML

TL;DR

The paper asks when parametric models help reinforcement learning relative to experience replay. It compares their shared planning capabilities and differing failure modes, then evaluates replay and model-based strategies in grid worlds and Atari 2600. The replay-based agent achieved state-of-the-art data efficiency and outperformed prior model-based results in the reported Atari comparison.

Problem
The paper asks when learning and using a parametric model improves learning efficiency compared with using actual observed data through replay.
Method
The paper compares replay and parametric-model planning, including forward behavioural planning, backward credit assignment, and Atari evaluations.
Results
The replay-based agent attained state-of-the-art data efficiency on Atari 2600 and bested prior model-based results.
Takeaways & Limitations
Replay can be competitive with or better than parametric-model approaches when models generate fictional transitions from observed states for otherwise model-free updates.
Takeaways & Limitations
The discussed equivalences are limited to linear prediction and do not extend straightforwardly to nonlinear functions or control.

Abstract

from arXiv · show

We examine the question of when and how parametric models are most useful in reinforcement learning. In particular, we look at commonalities and differences between parametric models and experience replay. Replay-based learning algorithms share important traits with model-based approaches, including the ability to plan: to use more computation without additional data to improve predictions and behaviour. We discuss when to expect benefits from either approach, and interpret prior work in this context. We hypothesise that, under suitable conditions, replay-based algorithms should be competitive to or better than model-based algorithms if the model is used only to generate fictional transitions from observed states for an update rule that is otherwise model-free. We validated this hypothesis on Atari 2600 video games. The replay-based algorithm attained state-of-the-art data efficiency, improving over prior results with parametric models.

1. Model-based reinforcement learning

The paper defines a generic framework spanning model-based and replay-based reinforcement learning. Planning uses extra computation without new data, while learning updates depend on newly observed experience.

The framework separates model-free algorithms from model-based algorithms that learn an environment model and use it to plan.The distinction is acknowledged as not fully unambiguous.
Planning improves predictions or behaviour through additional computation without consuming additional data.Learning instead refers to updates that depend on newly observed experience.
Models map a state and action to a reward and next state, and may be perfect, learned, stochastic, or expectation-based.The paper notes that iterating expectation models can be unhelpful under stochastic dynamics when expected states lack useful semantics.
Experience replay stores observed transitions for later updates, can resemble an accurate deterministic model on observed states, but cannot query arbitrary unseen states.Replay is therefore less flexible than a parametric model for generating transitions outside the stored data.
A generic algorithm alternates environment interaction, model and agent updates, and P planning steps using model-generated transitions.With P = 0 it is model-free; if only model-generated updates affect the agent, it is purely model-based.
Dyna combines learning and planning, while Dyna-Q results when both real and modelled transitions update action values with Q-learning.DQN can also fit the framework when replay buffers are treated as limited non-parametric models.

2. Model properties

Parametric models and replay both support planning, but differ in flexibility, computation, and exposure to model error. The paper argues that models are especially useful for forward behavioural planning and potentially backward credit assignment, while experiments compare these strategies with replay.

2. Model properties: Replay-based Q-learning and Dyna-Q scale similarly as planning updates per real environment step increase in a grid-world experiment.The comparison uses total steps to complete 25 episodes against updates per environment step, with both axes logarithmic.
2. Model properties: Replay and parametric models can be equivalent when a perfect model generates transitions only from previously observed states.Replay remains limited to states present in memory, whereas models can query arbitrary states.
2.2. Equivalences: An imperfect model generating fictional transitions from observed states should not improve learning over replay when all else is equal.This motivates comparing replay directly with model-based updates under the same state sampling distribution.
2.2. Equivalences: These equivalences are limited to linear prediction and do not extend straightforwardly to nonlinear functions or control.The limitation leaves open when parametric models should replace replay.
2.3. When do parametric models help learning?: Parametric models can plan from unseen or counterfactual states to select future behaviour, which standard replay cannot readily replicate.This forward behavioural planning is used in model-predictive control and need not treat imagined transitions as real experience.
2.3. When do parametric models help learning?: More forward planning improved behaviour in a deterministic four-room grid world by using planned action values from a learned model and Q-function.The method interprets deeper search as a multi-step greedy policy for choosing actions.
2.3. When do parametric models help learning?: Backward planning was faster than forward planning in deterministic and stochastic four-room environments, while later deterministic performance was slightly below replay.The stochastic variant used a 20% probability of transitioning to a random adjacent cell.

3. A failure to learn

The paper analyzes instability when model-generated transitions are combined with replay-based state sampling, showing that model error can cause divergence even when replay alone is stable. It discusses mitigations but notes that no definitive general solution exists.

3. A failure to learn: Divergence is possible rather than inevitable, and the paper next compares replay with a model-based agent that was stable enough to achieve strong Atari sample efficiency.This distinction motivates evaluating theoretical failure modes alongside empirical performance.
3. A failure to learn: The deadly triad can destabilize learning when function approximation, bootstrapping, and off-policy learning are combined.The issue arises when the state-sampling distribution does not match the policy-conditioned transition dynamics.
3. A failure to learn: Replay of uniformly sampled transitions from complete episodes is stable under the stated temporal-difference setup.The replay buffer defines an empirical model whose transition and state distributions correspond to the same empirical policy.
3. A failure to learn: Prioritized replay can reduce replay stability, while proposed model-based mitigations remain unsatisfactory or limited in scope.The paper notes that iterated learned models may become increasingly unrealistic and that existing stable off-policy methods are often specialized.
3. A failure to learn: A learned model can diverge when it generates transitions from states sampled according to a replay distribution.Model error can make the transition dynamics and sampling distribution inconsistent, producing a spectral radius above one and divergent parameters.
3. A failure to learn: The failure occurs because model-generated transitions may lead to uncommon or impossible states under the sampling distribution.Those states influence value estimates through function generalisation even when they are not sampled directly for updates.

4. Model-based algorithms at scale

The paper compares SimPLe’s parametric-model approach with Rainbow DQN’s experience replay under a similarly data-limited Atari setting. A lightly modified Rainbow matched SimPLe earlier and surpassed it substantially with fewer model-generated samples.

4. Model-based algorithms at scale: SimPLe demonstrated data-efficient Atari learning by updating its policy only with transitions sampled from a learned parametric model.It performed relatively well after 102,400 interactions within each game.
4. Model-based algorithms at scale: Rainbow DQN combines DQN with several enhancements and learns action values from mini-batches sampled from experience replay.Its components include double Q-learning, dueling networks, prioritized replay, noisy networks, and distributional reinforcement learning.
Model and replay comparison: The original Rainbow configuration targeted 50M interactions and sampled 32 transitions every 4 real interactions.In the paper’s notation, this corresponds to M = 4, P = 32, and K = 12.5M.
Model and replay comparison: The data-efficient Rainbow configuration used K = 100,000, M = 1, and P = 32, replaying 3.2 million samples versus SimPLe’s 15.2 million model samples.Replay sampling was also faster computationally than generating transitions with a learned model.
4.2. Empirical results: The experiment measured median human-normalised episode returns across 26 Atari games as a function of environment interactions.The comparison includes SimPLe as a dashed horizontal reference, with error bars computed over five independent seeds.
4.2. Empirical results: Rainbow’s large-data hyperparameters were poorly suited to extreme data efficiency, whereas the slightly tweaked data-efficient version performed better.The authors increased multi-step returns from 3 to 20 and reduced the replay warm-up from 20,000 to 1,600 steps.
4.2. Empirical results: 70,000 interactions were sufficient for the tuned Rainbow to match SimPLe, while performance was roughly 25% higher by 100,000 interactions.The Rainbow agent was superior to SimPLe in 17 of 26 games.

5. Conclusions

The paper argues that replay can outperform parametric models when models are used in replay-like updates, while parametric models may be more useful for backward credit assignment or forward behavioural planning. Its conclusions are bounded by the paper’s focus on common parametric-model and replay methods.

5. Conclusions: Replay-based learning is hypothesised to be preferable when parametric models generate transitions from sampled observed states for replay-like updates.The paper attributes this to potential instability from model errors under that state-sampling distribution.
5. Conclusions: At-scale Atari 2600 experiments produced state-of-the-art data efficiency for the replay-based agent, surpassing prior model-based results.The conclusion specifically compares the replay-based agent with Kaiser et al. (2019).
5. Conclusions: Parametric models may be most useful for backward planning for credit assignment or forward planning for behaviour.Forward planning for credit assignment was less effective, whereas forward planning for behaviour was considered less harmful because plans were not treated as real experience.
5. Conclusions: The paper does not cover all possible ways to plan with learned models, including abstract models and alternative model uses.These topics are explicitly identified as outside the paper’s scope.
5. Conclusions: The discussion focuses mainly on parametric models and replay, leaving non-parametric models as a potential alternative and future direction.Such models could interpolate and generalise to unseen states, but high-dimensional distance metrics raise practical questions.

A. Divergence example

The divergence example shows that instability depends on a mismatch between model dynamics and the state-sampling distribution, not necessarily on model inaccuracy. Divergence can therefore occur even with true dynamics.

A. Divergence example: In the two-state Markov reward process, divergence occurs when state 1 is sampled sufficiently more often than the model transitions into state 1.The example uses two states, zero rewards, discount γ = 0.99, and a single feature x(s) = s.
A. Divergence example: Oversampling state 2 is less harmful for the example’s specific function approximation.This asymmetry is tied to the particular function-approximation choice rather than stated as a general property.
A. Divergence example: Model-based updates can diverge because the model dynamics and state-sampling distribution mismatch, even when the model represents true dynamics.For true dynamics with P(St+1 = 1|St) = 0.5, divergence likelihood can be examined as a function of samples used to estimate d.

B. Experiment details: Scalability of planning

The scalability experiments compare forward Dyna and replay-based Q-learning agents using neural-network function approximation and learned transition, reward, and termination models.

B. Experiment details: Scalability of planning: The maze exposes a deterministic 5 × 5 local view, with walls encoded as 1s and free cells as 0s.The agent chooses among up, down, left, and right; actions into walls have no effect.
B. Experiment details: Scalability of planning: Both agents approximate Q-values with a multilayer perceptron containing two fully connected hidden layers of size 20 and ReLU activations.The final Q-value layer has four linear outputs, one per action.
B. Experiment details: Scalability of planning: The forward Dyna agent uses separate neural networks to model state transitions, rewards, and terminations.Their output layers have 25, 1, and 1 outputs, respectively.

C. Experiment details: Benefits of Planning

The benefits-of-planning experiments use a deterministic, fully observed four-rooms environment with randomized starts, a fixed goal, and tabular state representations.

C. Experiment details: Benefits of Planning: Each episode randomizes the agent’s starting position while keeping the goal position fixed.The environment’s dynamics are deterministic and fully observed.
C. Experiment details: Benefits of Planning: The agent has four cardinal movement actions plus a no-op action.The experiments use a tabular state-index representation.
C. Experiment details: Benefits of Planning: Both experiments learn an exact Bayesian tabular model.

D. Additional results on Atari

The Atari ablations examine how multi-step target length, update frequency, and replay sampling affect data-efficient Rainbow DQN performance.

Bootstrapping length: N = 5 produced much worse performance than N = 10 or N = 20, although it remained comparable to SimPLe.N denotes the multi-step bootstrapping length, with update frequency fixed at M = 1.
Bootstrapping length: N = 10 and N = 20 both performed well, with no statistically significant difference between them.The comparison used Welch’s test across 5 replicas for each hyper-parameter evaluation, at significance level 0.1.
Update frequency: Updating every step performed much better than updating every 4 steps when the bootstrapping length was fixed at N = 20.The performance gap from changing update frequency was larger than the gap from varying N.

E. Table of results

Table 1 reports end-of-training mean episode returns for Rainbow and SimPLe across 26 Atari games, alongside human and random reference scores.

Game-by-game results: Table 1 compares mean episode returns for Human, Random, SimPLe, and Rainbow on each of 26 Atari games.The higher score between SimPLe and Rainbow is marked in bold for each game.
Evaluation protocol: Rainbow results were measured at the end of training and averaged across 5 seeds, while SimPLe results came from Kaiser et al. (2019).Human and Random scores serve as reference scores for normalizing the learning curves.

F. Atari hyper-parameters

The section documents the hyper-parameters of canonical Rainbow DQN and its data-efficient variant, while noting that several shared parameters were not exhaustively tuned.

Parameter settings: Table 2 lists the hyper-parameters used by canonical Rainbow DQN and the data-efficient Rainbow variant.The table is provided for completeness and reproducibility.
Tuning scope: Several parameters shared by both variants may substantially affect data efficiency, but the authors did not fine-tune them exhaustively.The stated goal was a clean comparison rather than maximal performance.
Implementation notes: The target-network update period is defined by the number of updates rather than frames, making this update more frequent in frame terms for the data-efficient variant.The accompanying implementation note contrasts the canonical and data-efficient schedules.