PaperlayerView exact revision on arXiv

Source-linked AI summary

BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning

Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, Keith Ross

arXiv:1910.12179v4cs.LGcs.AIstat.ML

TL;DR

Batch reinforcement learning aims to learn high-performing policies from existing data without further environment interaction, while conventional Q-function methods can perform poorly or diverge in this setting. BAIL estimates values from batch data and uses selected actions to train a policy through imitation learning. On MuJoCo training batches, it wins most batches, outperforms comparison methods, and runs faster than Q-learning algorithms.

  • Problem

    Batch reinforcement learning seeks high-performing policies from a given dataset without further environment interaction, while conventional Q-function methods can perform poorly or diverge because of extrapolation error.

    Source: §1 · p. 1

  • Method

    BAIL estimates a value function from batch data, selects high-return actions near the data's upper envelope, and trains a policy with imitation learning.

    Source: §4 · p. 3, §4.1 · p. 3, §4.2 · p. 5

  • Results

    BAIL wins 20 of 22 training batches, with performance 42% higher than BCQ and 101% higher than BC, and is computationally much faster than BCQ and BEAR.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The results show that state-of-the-art performance is possible with a simple, computationally fast imitation-learning algorithm, and the upper-envelope idea may find applications in other machine-learning domains.

    Source: §6 · p. 9

    The formulation assumes deterministic environment dynamics and rewards.

    Source: §3 · p. 3

Abstract

from arXiv · show

There has recently been a surge in research in batch Deep Reinforcement Learning (DRL), which aims for learning a high-performing policy from a given dataset without additional interactions with the environment. We propose a new algorithm, Best-Action Imitation Learning (BAIL), which strives for both simplicity and performance. BAIL learns a V function, uses the V function to select actions it believes to be high-performing, and then uses those actions to train a policy network using imitation learning. For the MuJoCo benchmark, we provide a comprehensive experimental study of BAIL, comparing its performance to four other batch Q-learning and imitation-learning schemes for a large variety of batch datasets. Our experiments show that BAIL's performance is much higher than the other schemes, and is also computationally much faster than the batch Q-learning schemes.

1 Introduction

Batch reinforcement learning seeks high-performing policies from existing data without further environment interaction, but conventional Q-function methods can perform poorly or diverge from extrapolation error. BAIL combines value estimation with imitation learning and is reported to outperform comparison methods across MuJoCo batches while running faster than Q-learning baselines.

1 Introduction

  • Batch reinforcement learning learns from a given dataset without further environment interaction and can reuse collected data to improve a policy or support growing-batch learning.The paper also notes its potential relevance when partially trained policies cannot safely collect online data.Source: §1 · p. 1
  • Conventional Q-function methods such as DDPG can learn poorly or diverge in batch reinforcement learning because of extrapolation error.The introduction distinguishes Q-function methods, including BCQ and BEAR, from imitation-learning methods such as MARWIL and AWR.Source: §1 · p. 1
  • BAIL estimates a V function, selects dataset state-action pairs whose Monte Carlo returns are close to the upper envelope, then trains a policy with vanilla imitation learning.Its value estimates are updated only with batch data.Source: §1 · pp. 1–2
  • Across 22 non-expert training batches, BAIL wins 20 and achieves overall performance 42% or more higher than the other algorithms; it is 30–50 times faster than Q-learning algorithms.Source: §1 · p. 2
  • The paper contributes BAIL and the upper-envelope concept, plus experiments comparing five batch methods over diverse datasets.It reports providing open-source code for reproducibility and plans to make datasets available for benchmarking.Source: §1 · p. 2

2 Related work

Prior batch reinforcement learning work includes Q-function and imitation-learning approaches, alongside broader studies of batch DRL. This section contrasts their design choices and describes related benchmarks and complementary methods.

2 Related work

  • Batch RL has a long history in tabular and function-approximation settings [26] and remains an active research area [14] [29] [4] [12] [13].Source: §2 · p. 2
  • In Q-function-based batch methods, extrapolation error and overestimation in Q updates contribute to the difficulty of training deep networks effectively [8] [6] [30].The broader difficulty has also been discussed in recent work [18].Source: §2 · p. 2
  • BCQ [8] and BEAR [16] constrain actions considered during Q-function maximization, while adding multiple components to address batch-learning challenges.BCQ uses a state-dependent VAE, a perturbation model, and modified clipped Double Q-learning; BEAR uses MMD, kernel selection, a tanh-Gaussian model, and a test policy distinct from its learned actor.Source: §2 · p. 2
  • MARWIL and AWR [23] are imitation-learning approaches that weight training using advantage estimates, whereas BAIL selects actions using an upper envelope.Self-Imitation Learning uses data with cumulative discounted returns above current value estimates for exploration [22], and the idea has also been explored in multi-agent settings.Source: §2 · p. 2
  • REM [1] uses random convex combinations of Q-network heads and can outperform data-collecting policies on Atari; the paper describes REM and BAIL as orthogonal.Discrete-action batch algorithms have been benchmarked on Atari [7], while MuJoCo has been a major continuous-action benchmark and other environments have been proposed [5].Source: §2 · p. 3

3 Batch Deep Reinforcement Learning

The paper formulates batch deep reinforcement learning using an MDP and assumes deterministic dynamics and rewards. Its batch-learning objective is to learn a high-performing policy from a fixed dataset without further environment interaction.

3 Batch Deep Reinforcement Learning

  • The environment is modeled as an MDP with state and action spaces, dynamics, rewards, an initial-state distribution, and a discount factor.The functions g(s, a) and r(s, a) specify the next state and reward.Source: §3 · p. 3
  • The paper assumes deterministic environments, where the next state and reward are functions of the current state and action, as in the simulated MuJoCo locomotion tasks.The functions are not assumed to be known.Source: §3 · p. 3
  • Batch RL uses a dataset of transitions to learn a high-performing policy without further interaction, and the data may come from episodic training or a fixed deterministic execution policy.The algorithm, policy, and random seeds that generated the batch are not assumed to be known.Source: §3 · p. 3

4 Best-Action Imitation Learning (BAIL)

BAIL seeks high-performing actions from batch data by estimating an upper envelope of returns, selecting actions near it, and training a policy with imitation learning. For MuJoCo continuing tasks, it augments finite-episode returns; the heuristic has similar performance to oracle returns in the evaluated batches.

4.1 Upper envelope of the data

  • BAIL fits a regularized upper envelope above episodic Monte Carlo returns, balancing closeness to the data against regularization.As λ increases, the envelope approaches a constant through the highest-return data point; the penalty method uses K = 1000 in the tested environments.Source: §4.1 · pp. 3–4

4.2 Selecting the best actions

  • BAIL-ratio selects batch state-action pairs whose returns exceed a fraction of the envelope, then trains a policy by imitation learning.The selection fraction is 25%; BAIL-difference performs similarly, with BAIL-ratio sometimes a little better.Source: §4.2 · pp. 4–5

4.3 Augmented and oracle returns for the MuJoCo benchmark

  • For MuJoCo’s continuing tasks, BAIL augments finite-episode returns to approximate discounted infinite-horizon returns near episode ends.The heuristic extends the return using a state near the episode’s last state; oracle returns use episodes extended to 2,000 time steps.Source: §4.3 · p. 5
  • The return-augmentation heuristic has similar performance to oracle returns in the evaluated Hopper-v2 batches.It estimates continuation beyond the artificial episode boundary using rewards from the same policy.Source: §4.3 · pp. 5–6

5 Experimental results

Across diverse MuJoCo datasets, BAIL outperformed the comparison methods on training batches and required substantially less computation than BCQ and BEAR. Execution batches were a different case: behavior cloning performed strongly on these fixed-policy datasets.

5 Experimental results

  • The experiments compared five algorithms across 62 diverse batches and five MuJoCo environments, using available author code and recommended hyperparameters where possible.Datasets followed procedures from the BCQ and BEAR papers; batches generated with different random seeds could yield drastically different results, so each scenario used two seeds.Source: §5 · p. 6, §5.1 · p. 6

5.1.1 Training batches

  • Training batches comprised 22 non-expert datasets generated using DDPG or SAC, including datasets from early, suboptimal stages of policy training.They included DDPG batches at noise levels σ = 0.5 and σ = 0.1, SAC batches, and two random seeds per scenario.Source: §5.1.1 · pp. 6–7

5.1.2 Execution batches

  • Execution batches comprised 40 datasets generated from fixed SAC policies, with mediocre or near-optimal training, exploration noise or noise-free generation, and two seeds per scenario.Source: §5.1.2 · p. 7

5.2 Performance comparison

  • BAIL won 20 of 22 training batches, averaging 42% better performance than BCQ and 101% better than BC, with about half BCQ’s normalized standard deviation across batches.BCQ won 8 of 22 batches; the performance ratios average BAIL-to-baseline ratios across the 22 batches.Source: §5.2 · p. 8
  • BEAR occasionally performed very poorly under one shared set of recommended hyperparameters, while MARWIL performed similarly to BC on MuJoCo.The authors note that the BEAR paper used different hyperparameters for different environments.Source: §5.2 · p. 8
  • On execution batches, BC was the overall winner and most robust, while BAIL and BCQ had similar overall performance and BAIL had slightly stronger average performance than BCQ.The authors attribute BC’s strong showing to execution data generated by a single fixed policy, which is easy for BC to learn.Source: §5.2 · p. 8
  • Naively selecting the highest-return data points or constructing the value function by regression performed substantially worse than BAIL’s upper-envelope approach.The authors also suggest that identifying and imitating high-performing actions avoids the need to tune policy constraints, which can be too loose or too tight.Source: §5.2 · p. 8

5.3 Comparison of run times for Batch DRL algorithms

  • BAIL took about 1–2 hours per training seed, roughly 35 times faster than BCQ and 50 times faster than BEAR on a CPU node.The corresponding times were 36–72 hours for BCQ and 60–100 hours for BEAR.Source: §5.3 · p. 8

6 Conclusion

The conclusion highlights BAIL’s training-batch performance and computational speed, while distinguishing execution batches where BC already performs well. It also identifies possible directions for extending the approach.

6 Conclusion

  • BAIL won 20 of 22 training batches, improved performance by 42% over BCQ and 101% over BC, and was much faster than BCQ and BEAR.On execution batches, BC did well with little room for improvement, though BAIL and BCQ were almost as good and sometimes beat BC by a small amount.Source: §6 · p. 9
  • The authors describe the upper envelope of data as a potentially novel concept that may find applications in other machine-learning domains.Source: §6 · p. 9
  • Proposed future directions include combining batch methods with exploration for online algorithms and developing methods more robust across batches and hyperparameters.The authors suggest such robustness might reduce computation time and be safer for real-world deployment.Source: §6 · p. 9

Broader Impact

The broader impact passage describes batch reinforcement learning as a way to reuse collected data without further environment interaction, with potential applications to growing-batch learning and safety-critical settings.

Broader Impact

  • Batch RL reuses data collected by a policy to possibly improve that policy without further environment interaction.Source: Broader Impact · p. 10
  • In a growing-batch algorithm, a batch method seeks an exploitation policy from replay-buffer data, then exploration adds fresh data before the process repeats.Source: Broader Impact · p. 10
  • Batch RL may be necessary in safety-critical systems where partially trained policies cannot be deployed online to collect data.The passage also notes potential mechanisms for training robots and self-driving vehicles on complex tasks.Source: Broader Impact · p. 10

A Proofs of Theorems

The proof passages address convergence properties of the regularized upper-envelope parameters and invoke a standard convergence result for penalty functions.

A.1 Proof of Theorem 4.1

  • The proof argues by contradiction to show that the norm of the upper-envelope weight vector tends to zero as λ grows.It bounds the objective using a feasible constant function and derives a contradiction if the weight norm stays bounded away from zero.Source: §A.1 · p. 13
  • For λ = 0, the proof invokes finite-sample neural-network expressivity results for the finite set of inputs, citing [36].Source: §A.1 · p. 13

A.2 Proof of Theorem 4.2

  • The proof of Theorem 4.2 identifies the loss as a penalty-function form and applies the standard convergence theorem for penalty functions.Source: §A.2 · p. 14

B.1 Pseudo-Code and Early Stopping Scheme for Upper Envelope Training

BAIL’s procedure trains an upper envelope, selects data above a threshold, and updates a policy from selected samples, with validation-based early stopping. The experiments use shared hyperparameters across environments and batches.

B.1 Pseudo-Code and Early Stopping Scheme for Upper Envelope Training

  • Training restores the best-validation parameters after four consecutive epochs with worse validation loss.A second parameter set records the lowest validation loss seen so far.Source: §B.1 · p. 15
  • Initialize parameters and batch data, compute returns, train the upper envelope, select samples above a threshold, and update the policy from selected mini-batches.The selection threshold is set so that p% of batch data are selected.Source: §B.1 · p. 15

B.2 Hyper-parameters of BAIL

  • BAIL and Progressive BAIL share hyperparameters except for the data-selection percentage; Table 2 lists BAIL’s hyperparameters.Source: §B.2 · p. 16, Table 2 · p. 16

C Experimental Details

  • The comparison covers five MuJoCo environments and four baselines, including Humanoid, described as the most challenging MuJoCo environment.The passage notes Humanoid is not attempted in most other batch-DRL papers.Source: C Experimental Details · p. 16

C.1 Hyper-parameter consistency

  • Each algorithm uses the same hyperparameters across environments and batches, avoiding separate environment-specific searches for the comparison.The passage characterizes this as consistent with common online-RL practice and the BCQ paper.Source: §C.1 · p. 16

C.2 Reproduction of the Baseline Algorithms

The experiments reproduce baseline implementations and evaluate policies consistently, while testing BAIL’s return-augmentation heuristic against oracle returns. On Hopper-v2, the heuristic typically achieves oracle-level performance.

C.2 Reproduction of the Baseline Algorithms

  • BCQ’s final-buffer results are consistent with the BCQ paper, and the reproduced BEAR curve for the authors’ mediocre Hopper-v2 dataset matches Kumar et al. [16].Source: §C.2 · p. 17, Figure 3 · p. 17
  • BEAR can perform much worse than reported in [16] [35] under the single recommended hyperparameter setting used across environments; MARWIL was replicated from described details with shared network architecture.Source: §C.2 · p. 17

C.3 Common Hyper-parameters across all batch RL algorithms

  • All algorithms use 400 × 300 policy networks with ReLU units and a policy-network learning rate of 1 · 10−3.Source: §C.3 · p. 17

C.4 Evaluation methodology employed for all batch RL algorithms

  • Policies are evaluated with ten test episodes at the same frequency, and MuJoCo-error terminations receive zero returns, accounting for some zero-valued BEAR results.The authors suggest repeated errors may relate to using one hyperparameter setting across environments.Source: §C.4 · p. 17

D.1 Augmented return versus oracle performance

  • On Hopper-v2 across seven diverse batches, return augmentation typically achieves oracle-level performance.Oracle episodes run to 2,000 steps, providing at least 1,000 actual rewards per return and estimates described as essentially exact due to discounting.Source: Figure 4 · p. 18, §D.1 · p. 18

D.2 Ablation study for data selection

The upper envelope is reported as critical to BAIL’s data selection, both against raw-return ranking and standard regression. On execution datasets, BAIL and BCQ trade off average performance and wins.

D.2 Ablation study for data selection

  • The learning curves show that BAIL’s upper envelope is critical compared with selecting the same percentage of state-action pairs with the highest returns.Source: Figure 5 · p. 19, §D.2 · p. 19

D.3 Ablation study using standard regression instead of an upper envelope

  • The learning curves show that BAIL’s upper envelope is critical compared with using standard regression instead.Source: Figure 6 · p. 20, §D.3 · p. 20

E Performance for execution batches

F.1 DDPG training batches

The appendix presents learning-curve figures for DDPG and SAC batches, SAC execution batches, and Humanoid, plus upper-envelope visualizations and computing infrastructure details.

F.1 DDPG training batches

  • Figures 7 and 8 report batch-DRL performance on DDPG training batches at σ = 0.5 and σ = 0.1, respectively.All policies train for 100 epochs except BAIL, which trains its upper envelope for 50 epochs and policy for 50 epochs.Source: Figure 7 · p. 22, Figure 8 · p. 22

F.2 SAC training batches

  • Figure 9 reports batch-DRL performance on SAC training batches.Policies train for 100 epochs except BAIL, which trains its upper envelope for 50 epochs and policy for 50 epochs.Source: Figure 9 · p. 23

F.3 SAC mediocre execution batches

  • Figures 10 and 11 report performance on SAC mediocre execution batches with σ = 0 and σ = σ(s), respectively.Policies train for 100 epochs except BAIL, which trains its upper envelope for 50 epochs and policy for 50 epochs.Source: Figure 10 · p. 24, Figure 11 · p. 24

F.4 SAC optimal execution batches

  • Figures 12 and 13 report performance on SAC optimal execution batches with σ = 0 and σ = σ(s), respectively.Policies train for 100 epochs except BAIL, which trains its upper envelope for 50 epochs and policy for 50 epochs.Source: Figure 12 · p. 25, Figure 13 · p. 25

F.5 Learning curves for Humanoid

  • Figure 14 presents learning curves for batch-DRL algorithms on Humanoid-v2 using SAC batches.Source: Figure 14 · p. 26

G Visualization of the Upper Envelopes

  • Figure 15 orders states by increasing upper-envelope value; the curves rise monotonically and lie above most, but not all, data points.Each curve is trained with one million returns, represented by orange dots.Source: Figure 15 · p. 27

H Computing Infrastructure

Related papers

Loading 1910.12179v4…