PaperlayerView exact revision on arXiv

Source-linked AI summary

DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction

Aviral Kumar, Abhishek Gupta, Sergey Levine

arXiv:2003.07305v1cs.LGstat.ML

TL;DR

Deep RL can be unstable, and bootstrapped value-learning methods may not receive corrective feedback from the experience their policies collect. DisCor estimates target-value accuracy to reweight replay transitions, and the paper reports improvements across varied RL settings, including multi-task learning and noisy rewards.

  • Problem

    Bootstrapped Q-learning methods may not correct value-function errors from their collected experience, while interactions between policy-induced data distributions and learned errors can contribute to instability and suboptimal convergence.

    Source: §1 · p. 1

  • Method

    DisCor estimates target-value error and reweights replay-buffer transitions using a distribution intended to provide corrective feedback.

    Source: §1 · p. 2, §8 · p. 13

  • Results

    DisCor improves performance across a range of RL problems, with a final success rate 50% higher than SAC on MT10 and several robotic manipulation tasks.

    Source: §1 · p. 2, §8 · p. 13

  • Takeaways & Limitations

    The distribution correction can be combined with a variety of approximate dynamic programming algorithms and improves results in challenging settings including multi-task learning and Atari games.

    Source: §8 · p. 13

    Because DisCor only reweights the replay buffer, it is limited to transitions collected by the behavior policy.

    Source: §8 · p. 13

Abstract

from arXiv · show

Deep reinforcement learning can learn effective policies for a wide range of tasks, but is notoriously difficult to use due to instability and sensitivity to hyperparameters. The reasons for this remain unclear. When using standard supervised methods (e.g., for bandits), on-policy data collection provides "hard negatives" that correct the model in precisely those states and actions that the policy is likely to visit. We call this phenomenon "corrective feedback." We show that bootstrapping-based Q-learning algorithms do not necessarily benefit from this corrective feedback, and training on the experience collected by the algorithm is not sufficient to correct errors in the Q-function. In fact, Q-learning and related methods can exhibit pathological interactions between the distribution of experience collected by the agent and the policy induced by training on that experience, leading to potential instability, sub-optimal convergence, and poor results when learning from noisy, sparse or delayed rewards. We demonstrate the existence of this problem, both theoretically and empirically. We then show that a specific correction to the data distribution can mitigate this issue. Based on these observations, we propose a new algorithm, DisCor, which computes an approximation to this optimal distribution and uses it to re-weight the transitions used for training, resulting in substantial improvements in a range of challenging RL settings, such as multi-task learning and learning from noisy reward signals. Blog post presenting a summary of this work is available at: https://bair.berkeley.edu/blog/2020/03/16/discor/.

1. Introduction

The paper attributes deep RL instability to interactions between policy-induced data distributions and value-function errors. It argues that bootstrapped methods may lack corrective feedback, and proposes DisCor to reweight replay data toward it.

1. Introduction

  • Deep RL methods can be sensitive to hyperparameters and converge inconsistently or unstably; the paper hypothesizes that policy-induced data distributions interact pathologically with learned value-function errors.Errors from training on one distribution can worsen the distribution induced by the next policy.Source: §1 · p. 1
  • Prior work had not extensively studied how data distributions induced by the latest value function relate to errors in future value functions trained on that data.The paper places its analysis within prior theoretical work on approximate dynamic programming methods.Source: §1 · p. 1
  • Bootstrapped Q-learning and actor-critic methods may lack corrective feedback because visited states can have targets corrupted by upstream errors in less-visited states.In supervised learning, policy-selected hard negatives can instead supply transitions that help correct model errors.Source: §1 · p. 1
  • DisCor reweights replay-buffer transitions using an estimated target-value accuracy distribution to mitigate error accumulation.The approach is presented as usable with most modern approximate dynamic programming algorithms, including DQN and SAC.Source: §1 · p. 2
  • 50% higher final success rate than SAC on MT10 and several robotic manipulation tasks is reported for DisCor.Experiments also cover continuous-control tasks and discrete-action Atari games.Source: §1 · p. 2

2. Preliminaries

The preliminaries describe value-based approximate dynamic programming as repeated Bellman projection under a sampling distribution. In common methods, that distribution comes from the current policy or a replay buffer, and its choice can affect stability and convergence.

2. Preliminaries

  • An MDP is defined by state and action spaces, transition dynamics, rewards, and a discount factor; dπ(s) denotes a policy's infinite-horizon discounted marginal state distribution.The corresponding state-action marginal is dπ(s, a) = dπ(s)π(a|s).Source: §2 · p. 2
  • Approximate dynamic programming methods such as Q-learning and actor-critic seek optimal state and state-action value functions by iterating the Bellman optimality operator.Function approximation projects the operator's values onto a Q-function family under a sampling distribution.Source: §2 · p. 2
  • Fitted Q-iteration alternates fitting the action-value function to current data by minimizing Bellman error with collecting data using a policy derived from that value function.The paper uses this setup to simplify its analysis.Source: §2 · p. 2
  • Common sampling distributions are the current policy's state-action marginal or a replay-buffer mixture of past policies, and these choices can lead to suboptimal convergence.The paper identifies the sampling distribution as important to stability and efficiency, even with online data collection.Source: §2 · p. 2
  • The analysis uses tabular MDPs with all transitions available and reweights them to simulate data distributions, removing sampling error while measuring error against Q∗.A two-hidden-layer feed-forward network represents the Q-function.Source: §2 · p. 2

3. Corrective Feedback in Q-Learning

Corrective feedback occurs when policy-driven data collection exposes the model’s errors, but bootstrapped ADP can fail to correct errors in its target values. On-policy and replay-buffer distributions can therefore produce slow or unstable learning, even when all transitions are available.

3. Corrective Feedback in Q-Learning

  • In bootstrapped ADP, erroneous values at backup states can yield incorrect targets that repeated observations of the current transition do not correct.Because data distributions are coupled to learned value functions, the algorithm may fail to revisit the states responsible for target errors.Source: §3 · p. 3
  • ADP visitation can correlate positively with increasing value error for prolonged periods, unlike supervised bandit learning, whose correlation is negative until convergence and then fluctuates around zero.The bandit comparison reaches convergence around iteration 25.Source: §3 · p. 3, Figure 1 · p. 3

3.1.1 Didactic Example

  • With on-policy Q-learning on a tree MDP, infrequently visited leaves are slow to correct, so their erroneous values become targets for higher-level states.Updating leaves first and then moving upward produces fewer incorrect backups and lower errors.Source: §3.1.1 · p. 4, Figure 3 · p. 5

3.1 A Didactic Example and Theoretical Analysis of an Absence of Corrective Feedback

  • With function approximation, states suffering from erroneous targets may be visited more often while the states causing those errors do not become more frequent.This distribution mismatch can prevent corrective feedback despite sufficient convergence guarantees in tabular settings.Source: §3.1.1 · pp. 4–5

3.1.2 Theoretical Analysis

  • On-policy and replay-buffer Q-learning can require exponentially many Bellman backups to converge, even when the function class can represent Q* and all transitions are available.The analysis contrasts this with the proposed method’s polynomial iteration requirement in that scenario.Source: §3.1.2 · p. 5

3.2 Consequences of Absent Corrective Feedback

  • On-policy sampling can converge to a suboptimal Q-function, while replay-buffer Q-learning can be unstable and learning can worsen with sparse or noisy rewards.These failures are reported even without sampling error; replay-buffer degradation can occur when the latest policy is near-optimal.Source: §3.2 · pp. 5–6

4. Optimal Distributions for Optimizing Corrective Feedback

The paper derives an optimal training distribution intended to minimize value error and then approximates it with quantities available during learning. Practical weights reweight replay-buffer transitions, with a conservative bound that may down-weight some low-error transitions.

4. Optimal Distributions for Optimizing Corrective Feedback

  • The optimal distribution p_k is derived by minimizing value error at each iteration, and assigns greater probability to high-Bellman-error pairs when the resulting Q-value is close to Q*.The derivation uses a soft-min upper bound on the objective and constraints that keep p_k a valid distribution.Source: §4 · pp. 6–7

4.1 Tractable Approximation to Optimal Distribution

  • Because Q_k and Q* are unavailable when p_k is chosen, Δ_k serves as a tractable upper-bound proxy for |Q_k − Q*|, while prior-iteration Bellman errors bound the other unknown term.The stated bound holds after a sufficiently large iteration k_0; c_1 and c_2 can be computed over replay-buffer transitions.Source: §4.1 · p. 7
  • Importance weights reweight replay-buffer samples toward p_k because directly sampling from p_k through online interaction is challenging.To reduce high-variance weights, the method instead projects p_k toward the buffer distribution under a KL-divergence objective.Source: §4.1 · p. 8

4.2 Putting it All Together

  • The practical weights use a worst-case lower bound: they may down-weight extra transitions with low error accumulation, but never up-weight transitions with high error.The authors describe this as a conservative distribution and leave less conservative weight expressions for future work.Source: §4.2 · p. 8

5. Distribution Correction (DisCor) Algorithm

DisCor uses estimated target-value errors to weight Bellman backups, focusing learning on transitions with more accurate supervision. Its implementation adds an error model and a weighted Q-function backup to standard ADP methods.

5. Distribution Correction (DisCor) Algorithm

  • DisCor weights Bellman backups to down-weight transitions whose bootstrapped target estimates have high error, focusing learning on samples with more accurate supervision.The paper links this focus to preventing error accumulation and providing corrective feedback.Source: §5 · p. 8

Algorithm 1 DisCor (Distribution Correction)

  • The algorithm collects transitions into a replay buffer, evaluates Q and Δ on sampled transitions, computes w_k, and minimizes weighted Bellman error.The pseudocode describes this sequence alongside the standard Q-learning updates.Source: Algorithm 1 DisCor (Distribution… · p. 9
  • DisCor adds a parametric error model Δ_φ and uses weights w_k from Equation 8 in a weighted Q-function backup.The algorithm trains Δ_φ using the recursion in Equation 4 and is implemented on top of standard DQN/SAC pseudocode.Source: Algorithm 1 DisCor (Distribution… · p. 9

6. Related Work

This section distinguishes DisCor’s focus on data-distribution interactions from prior analyses of convergence and function approximation. It also contrasts the setting with offline sampling and related prioritization methods.

Related work

  • Unlike work centered on Bellman-backup convergence or time-varying distributions without function approximation, this paper studies time-varying data distributions with function approximation.The distribution depends on prior and latest Q-functions in the authors’ setting.Source: §6 · p. 9
  • The paper treats interaction between data distribution and value-function fitting error as distinct from deep-network non-convergence and generalization issues.Its focus is the relationship between the training distribution and fitting error.Source: §6 · p. 9
  • Where prior offline work addresses sampling schemes for convergence, this work examines sampling-distribution effects during online interaction; unlike PER, it weights transitions by estimated error to the optimal value function.The paper says prioritizing high Bellman error may fail in many cases and seeks corrective feedback through reweighting.Source: §6 · p. 9

7. Experimental Evaluation of DisCor

The evaluation tests corrective feedback and comparisons with prior methods across tabular MDPs, continuous control, multi-task learning, and Atari. DisCor generally improves performance or learning speed in the reported challenging settings.

7. Experimental Evaluation of DisCor

  • The evaluation asks whether DisCor provides corrective feedback, compares it with prior reweighting methods, tests challenging settings, and examines the effects of its approximations.Experiments include tabular MDPs, six robotic manipulation tasks, MT10, and three Atari games.Source: §7 · p. 10

7.1 Analysis of DisCor on Tabular Environments

  • On tabular MDPs, DisCor consistently provides corrective feedback and outperforms on-policy and replay-buffer sampling; its approximation improves performance over uniform weighting and PER-like prioritization.The oracle version performs somewhat better, while high-Bellman-error prioritization struggles on sparse-reward domains.Source: §7.1 · p. 10

7.2 Continuous Control Experiments

  • On six Meta-World manipulation tasks, DisCor often achieves better success rates and learns good performance earlier than unweighted SAC and PER.PER performs poorly in these experiments; the authors note that prioritizing high-Bellman-error states may increase error accumulation when target values are incorrect.Source: Figure 7 · p. 11, §7.2 · pp. 11–12

7.3 Multi-Task Reinforcement Learning

  • On MT10, DisCor achieves 50% higher success rates than unweighted SAC and makes progress on 7/10 tasks versus 3/10 for SAC.On MT50, DisCor continues learning while unweighted SAC tends to saturate at a suboptimal success rate for about 4M environment steps.Source: Figure 8 · p. 12, §7.3 · p. 12

7.4 Arcade Learning Environment

  • On three Atari games, DisCor usually improves learning speed and performance over unweighted DQN.The evaluation uses Pong, Breakout, and Asterix with sticky actions.Source: §7.4 · p. 12, Figure 9 · p. 13

8. Discussion, Future Work and Open Problems

DisCor applies distribution correction by reweighting replay-buffer transitions using estimated target-value accuracy, and improves performance across varied RL problems. Its scope is limited to transitions collected by the behavior policy; changing exploration is proposed as future work.

8. Discussion, Future Work and Open Problems

  • DisCor reweights replay-buffer transitions according to estimated target-value accuracy and improves performance across robotic manipulation, multi-task RL, and Atari.The method can be combined with a variety of ADP algorithms.Source: §8 · p. 13
  • Because it only reweights collected transitions, DisCor is limited by the transitions the behavior policy actually collects.The authors propose directly modifying exploration as a direction for changing which transitions are collected.Source: §8 · p. 13
  • The authors identify better characterization of learning dynamics and corrective feedback as a direction for developing more stable algorithms.They also propose studying exploration-policy changes to produce larger sample-efficiency and asymptotic-performance gains.Source: §8 · p. 13

Appendix A. Detailed Proof of Theorem 4.1 (Section 4)

The derivation turns the objective for choosing a corrective-feedback distribution into a tractable optimization, then uses implicit gradients and KKT conditions to obtain the distribution. Its resulting probabilities favor high Bellman error when the updated value is close to optimal.

Appendix A. Detailed Proof of Theorem 4.1 (Section 4)

  • The derivation upper-bounds the objective with a Fenchel–Young decomposition and uses a soft-min choice that yields an entropy-based objective.It further upper-bounds state-action marginal entropy by uniform-distribution entropy to keep the marginal entropy from falling drastically.Source: Appendix A · p. 17
  • The constrained optimization makes p_k a valid distribution, and the derivation uses the implicit function theorem to differentiate Q_k with respect to p_k.The theorem’s IFT gradient is examined with a non-parametric Q-function representation to avoid model-specific Jacobian or inverse-Hessian calculations.Source: Appendix A · pp. 17–18
  • The IFT gradient links Bellman residuals and sampling density: residuals should be higher where p_k is lower.The passage states this relationship for nonzero residuals under the full-coverage regime.Source: Appendix A · p. 18
  • Setting the Lagrangian gradient to zero and applying KKT conditions yields p_k that favors high Bellman error when the post-update Q-value is close to Q*.The derivation’s summary lists the objective transformation, IFT gradient, and zero-gradient condition as its key steps.Source: Appendix A · p. 18

Appendix B. Proofs for Tractable Approximations in Section 4.1

The appendix supports DisCor’s tractable approximations with bounds on its error estimator and a surrogate replay reweighting objective. It also provides proofs and assumptions for the paper’s ADP analysis, including on-policy suboptimal convergence.

Appendix B. Proofs for Tractable Approximations in Section 4.1

  • The replay-buffer surrogate q_k trades closeness to the target distribution p_k against closeness to replay distribution µ, avoiding direct importance weights that can have high variance and require unknown densities.The temperature parameter trades off bias and variance.Source: Appendix B · p. 19

Proof

  • The proof explains that the effect of initialization shrinks at rate γ and gives k0 ≈59 for γ = 0.95 and k0 ≈460 for γ = 0.99.For improving policies, the cited limiting bound involving policy divergence tends to zero.Source: Proof · pp. 20–21

Appendix B. Proofs for Tractable Approximations in Section 4.1

  • The recursive estimator formed from Bellman residuals upper-bounds value error pointwise, and DisCor’s randomly initialized estimator satisfies the bound after sufficiently many iterations.The proof summary states that ∆k is an upper bound on |Qk − Q*| and that initial error matters only infinitesimally after a large enough k.Source: Appendix B · p. 20, Proof · p. 21

Appendix C. Missing Proofs From Section 3

  • The missing-proof appendix lays out a generic ADP procedure and assumes a linear Q-function representation for its analysis.It states that the representation is not limiting for problems with sufficiently large state and action spaces.Source: Appendix C · pp. 21–22

C.1 Suboptimal Convergence of On-policy Q-learning

  • The on-policy convergence subsection establishes that projected Bellman optimality under a Boltzmann policy can have one or more fixed points.The passage presents this as justification for empirical observations of suboptimal convergence with on-policy distributions.Source: §C.1 · p. 22

C.2 Proof of Theorem 3.1

The proof constructs a sparse-reward tree MDP where on-policy and replay-buffer distributions can require exponentially many Bellman iterations, despite a near-exact Q-function representation. In the same family, DisCor’s error-guided distribution permits polynomially many iterations.

C.2 Proof of Theorem 3.1

  • Corollary C.1.1 gives features Φ ∈ R^(2H×O(H²/ε²)) satisfying Assumption C.1 and ||I₂H−ΦΦᵀ||∞≤ϵ; for shared weights, Φ′ has the same stated dimensions.Assumption C.1 bounds the Q-function approximation error by δ≥0. Φ′ pads Φ with zeros by level, and one weight vector concatenates the earlier level-specific weights.Source: §C.2 · pp. 22–23

Proof

  • The construction is a deterministic binary tree with H levels, two actions per state, and a single rewarding state-action pair at the leaves; all transitions are available for backups.This sparse-reward setting isolates iteration complexity from exploration.Source: Proof · p. 23
  • On the tree MDP family, on-policy and replay-buffer sampling can require exponentially many exact Bellman iterations, while DisCor requires only poly(H) iterations for ε-accurate convergence.The DisCor result assumes linear function approximation and features satisfying Assumption C.1.Source: §C.2 · p. 23, Proof · pp. 25–26
  • DisCor’s proof progresses from leaves toward the root, updating a level at a time while keeping already-updated levels within ε-allowable error.The argument relies on accurate target values for the updates and Q* being expressible to ε-accuracy in the chosen function class.Source: Proof · pp. 25–26

Appendix D. Extended Related Work

  • The related-work discussion distinguishes this online error-accumulation problem from offline RL, which learns from fixed experience and constrains the policy near the behavior policy.It also distinguishes the issue from replay-buffer size and generalization effects.Source: Appendix D · p. 26

E.1 DisCor in Practice

  • The practical DisCor implementation maintains error networks alongside Q-networks and uses estimated errors to weight Q-function Bellman-error training.The implementation also updates target networks and, when explicitly modeled, the policy.Source: §E.1 · p. 27, Algorithm 3 DisCor: Deep RL Version · p. 27

E.2 Experimental Hyperparameter Choices

DisCor’s experimental setup uses a moving-average temperature for transition weights and an error network with one extra hidden layer. The appendix then introduces additional experiment results.

E.2 Experimental Hyperparameter Choices

  • DisCor uses a moving average of the temperature for computing transition weights, avoiding the need to tune temperature values; all experiments initialize τ0 to 10.0.The temperature is the method’s main introduced hyperparameter.Source: §E.2 · p. 28
  • The error network has one more hidden layer than its corresponding Q-network, and its learning rate matches the Q-function’s rate.The stated rates are 3e−4 for SAC and 0.0025 for DQN.Source: §E.2 · p. 28

Appendix F. Additional Experiments

  • The appendix presents additional experimental results that were not included in Section 7.Experiments generally used four random seeds, except the method’s Atari DQN runs, which used three per game.Source: Appendix F · p. 28, §E.2 · p. 28

F.1 Tabular Environment Analysis

Across the additional experiments, DisCor performs well on difficult MetaWorld tasks and noisy-reward benchmarks, while standard Gym benchmarks show roughly comparable methods. Multi-task results show continued learning or higher returns, and DisCor also outperforms AFM and DQN with three-step returns in the tested comparisons.

F.1 Tabular Environment Analysis

  • The tabular analysis evaluates five gridworld variants spanning observation and reward types, plus CliffWalk, Sparsegraph, and MountainCar.It compares exact-transition weighting with conventional online sampled data collection.Source: §F.1 · p. 28

F.2 MetaWorld Tasks

  • DisCor outperforms SAC and PER on all six tested MetaWorld tasks, achieving nearly 50% more return than SAC.The reported metric is evaluation average return, rather than success rate.Source: §F.2 · p. 29, Figure 16 · p. 31

F.3 OpenAI Gym Benchmarks

  • On standard Gym benchmarks, DisCor, SAC, and PER are roughly comparable; with stochastic reward noise, DisCor outperforms both baselines in asymptotic performance and sample efficiency.The noisy-reward evaluation reports deterministic ground-truth reward.Source: §F.3 · pp. 30–31, Figure 18 · p. 32

F.4 MT10 Multi-Task Experiments

  • On MT10, DisCor’s returns are around 30% higher than SAC’s, and it achieves non-trivial returns on 7/10 tasks after 1000k steps, versus 3/10 for SAC.The task-count comparison is reported in Figure 19.Source: Figure 19 · p. 32, §F.4 · p. 32

F.5 MT50 Multi-Task Experiments

  • On MT50, DisCor continues learning during a period when SAC plateaus between 4M and 8M training steps.The benchmark requires a single policy to solve 50 tasks together.Source: §F.5 · p. 32, Figure 20 · p. 33

F.6 Comparison with AFM

  • DisCor outperforms AFM on all three tested benchmarks, both with regular Gym rewards and with stochastic reward noise.Figure 21 presents regular benchmarks in its top row and noisy-reward benchmarks in its bottom row.Source: §F.6 · p. 33, Figure 21 · p. 34

F.7 DQN with multi-step returns

  • DisCor outperforms DQN with three-step returns on all three evaluated games; adding three-step returns to DisCor also outperforms applying them to DQN.The comparison uses average return with sticky actions.Source: §F.7 · p. 33, Table 1 · p. 33

F.8 Code for the Method

  • The appendix includes a simplified code example for training DisCor based on the official SAC implementation.The code is shown in Figure 22.Source: §F.8 · p. 34, Figure 22 · p. 35

Related papers

Loading 2003.07305v1…