PaperlayerView exact revision on arXiv

Source-linked AI summary

RLIF: Interactive Imitation Learning as Reinforcement Learning

Jianlan Luo, Perry Dong, Yuexiang Zhai, Yi Ma, Sergey Levine

arXiv:2311.12996v2cs.AIcs.RO

TL;DR

Interactive imitation learning is convenient for control but typically relies on near-optimal expert interventions, while standard RL requires task rewards. RLIF learns from intervention feedback instead, and the authors report that it outperforms DAgger variants, especially with suboptimal experts.

  • Problem

    Interactive imitation learning addresses behavioral cloning’s distribution shift but generally relies on near-optimal interventions, while RL typically uses specified reward functions.

    Source: §1 · p. 1

  • Method

    RLIF trains an off-policy reinforcement learning policy on interactive intervention data, using the intervention signal itself to derive rewards.

    Source: §1 · p. 2

  • Results

    RLIF is on average 2-3x better than best-performing DAgger variants, with the difference more pronounced as expert suboptimality increases.

    Source: §1 · p. 2

  • Takeaways & Limitations

    RLIF can learn from suboptimal interventions without ground-truth rewards, and the paper reports that RL can recover the optimal policy with additional samples where imitation learning performs poorly due to suboptimality.

    Source: §1 · p. 2

    RLIF requires an RL method able to train on data collected by both the policy and the intervening expert, combining off-policy and on-policy data.

    Source: §7 · p. 9

Abstract

from arXiv · show

Although reinforcement learning methods offer a powerful framework for automatic skill acquisition, for practical learning-based control problems in domains such as robotics, imitation learning often provides a more convenient and accessible alternative. In particular, an interactive imitation learning method such as DAgger, which queries a near-optimal expert to intervene online to collect correction data for addressing the distributional shift challenges that afflict na\"ive behavioral cloning, can enjoy good performance both in theory and practice without requiring manually specified reward functions and other components of full reinforcement learning methods. In this paper, we explore how off-policy reinforcement learning can enable improved performance under assumptions that are similar but potentially even more practical than those of interactive imitation learning. Our proposed method uses reinforcement learning with user intervention signals themselves as rewards. This relaxes the assumption that intervening experts in interactive imitation learning should be near-optimal and enables the algorithm to learn behaviors that improve over the potential suboptimal human expert. We also provide a unified framework to analyze our RL method and DAgger; for which we present the asymptotic analysis of the suboptimal gap for both methods as well as the non-asymptotic sample complexity bound of our method. We then evaluate our method on challenging high-dimensional continuous control simulation benchmarks as well as real-world robotic vision-based manipulation tasks. The results show that it strongly outperforms DAgger-like approaches across the different tasks, especially when the intervening experts are suboptimal. Code and videos can be found on the project website: https://rlif-page.github.io

1 INTRODUCTION

The paper proposes RLIF, which learns from intervention feedback without ground-truth rewards or optimal expert corrections. It reports stronger performance than DAgger variants, especially with suboptimal experts.

1 INTRODUCTION

  • Interactive imitation learning addresses behavioral cloning’s distributional shift by querying experts online, but relies on near-optimal interventions and cannot improve over the expert.The paper frames combining RL’s reward-maximizing behavior with interactive imitation’s accessible assumptions as its research question.Source: §1 · pp. 1–2
  • The intervention decision itself can provide a reward signal, allowing RL to learn without assuming that the human’s correction is optimal.The paper motivates this with the intuition that detecting a mistake can be easier than optimally correcting it.Source: §1 · p. 2
  • RLIF applies reinforcement learning to DAgger-style intervention data, assigning negative reward to actions that lead to intervention.The policy learns to avoid interventions rather than reproduce potentially suboptimal corrections.Source: §1 · p. 2
  • RLIF is on average 2-3x better than best-performing DAgger variants and scales to a real-world robotic task involving human feedback.The performance difference is more pronounced as expert suboptimality increases.Source: §1 · p. 2

2 RELATED WORK

The paper distinguishes RLIF from interactive imitation and prior RL-assisted imitation by deriving reward from interactive intervention choices rather than requiring optimal actions or task rewards.

Interactive imitation learning

  • Interactive imitation methods address distribution shift with online human interventions but generally assume those interventions are near-optimal; RLIF relaxes that assumption using intervention-derived rewards.The related-work discussion cites interactive imitation methods including Hoque et al. (2022), Kelly et al. (2018), Hoque et al. (2021), Menda et al. (2018), and Ross et al. (2011).Source: §2 · p. 3

Imitation learning with reinforcement learning

3 PRELIMINARIES AND PROBLEM SETUP

The section establishes the MDP and imitation-learning setup, then frames RLIF around the assumption that intervention timing conveys information about policy suboptimality.

Behavioral cloning and interactive imitation learning

  • Naïve behavioral cloning can accumulate quadratic regret as errors shift the states visited, while interactive imitation reduces error to linear in the horizon by collecting online expert labels.DAgger-style methods collect training data on states visited by the learned policy; intervention variants let an expert take over and override its action.Source: §3 · p. 3
  • When expert actions are suboptimal, DAgger-like methods’ regret gap expands; applying RL may address this and potentially outperform the expert.The section describes this as the motivation for applying RL in the interactive-imitation setting.Source: §3 · p. 4

Reinforcement learning

  • The setup uses an infinite-horizon MDP with state and action spaces, transitions, rewards, a discount factor, and policy value functions.The objective is to learn a policy maximizing expected cumulative reward from the initial-state distribution.Source: §3 · p. 4

Problem setup

  • RLIF targets learning without a supplied reward function or optimal intervention actions, assuming instead that experts are more likely to intervene when the learned policy takes a bad action.The intervention choice is intended to indicate that preceding behavior deviated from optimal behavior.Source: §3 · p. 4

4 INTERACTIVE IMITATION LEARNING AS REINFORCEMENT LEARNING

RLIF replaces supervised learning with reinforcement learning on aggregated intervention data, using intervention timing to derive rewards and an off-policy algorithm to train on collected data.

4 INTERACTIVE IMITATION LEARNING AS REINFORCEMENT LEARNING

  • RLIF labels the transition before an intervention with -1 reward and other transitions with 0, then trains an RL policy on the aggregated data.All transitions are retained, and the expert may take over for several steps after intervening.Source: §4 · p. 4
  • Off-policy RL uses the reward labels and collected data to avoid situations that trigger intervention, without requiring task rewards or optimal expert actions.The method still requires intervention timing to correlate with policy suboptimality.Source: §4 · p. 5
  • The practical implementation uses RLPD, an off-policy actor-critic suited to combining near-expert offline data with online experience; RLIF itself does not modify the RL algorithm.The paper describes RLIF as a meta-algorithm that changes the dataset on which an RL algorithm operates.Source: §4 · p. 5

5 EXPERIMENTS

Experiments compare RLIF with DAgger-style methods under different expert quality and intervention strategies, then evaluate human interventions on real-world robot tasks.

5.1 INTERVENTION STRATEGIES

  • Simulated interventions include uniform-random choices and value-based choices triggered probabilistically by expert–agent action gaps under a reference value function.A smaller δ makes the simulated expert more willing to intervene on slightly suboptimal robot actions.Source: §5.1 · pp. 5–6

5.2 PERFORMANCE ON CONTINUOUS CONTROL BENCHMARK TASKS

  • Evaluation spans Walker2d, Pen, and Hopper benchmarks alongside real-robot peg insertion and cloth unfolding tasks.The benchmarks involve high-dimensional dynamics and underactuation; the robot tasks add image inputs, contact dynamics, and real-world sensor noise.Source: Figure 4 · p. 6, §5.2 · p. 6
  • Up to 5x: RLIF consistently outperforms DAgger and HG-DAgger across tasks, with a larger gap for more suboptimal experts.With value-based interventions, RLIF also raises a 15% expert to a score over 100.0 on the 2D-walker task.Source: Table 1 · p. 7, §5.2 · pp. 7–8
  • RLIF performance depends on the reference value function: as the reference policy deteriorates, its value estimates become less calibrated and policy performance worsens.Source: §5.2 · p. 8

5.3 REAL-WORLD VISION-BASED ROBOTIC MANIPULATION TASK

  • 100% success on peg insertion within six interaction rounds and 95% on cloth unfolding within seven rounds were achieved on the real robot.The peg task used image observations and a 1.5 mm tolerance.Source: §5.3 · p. 8

6 THEORETICAL ANALYSIS

The analysis compares RLIF and DAgger using intervention-derived rewards when the true task reward is unavailable. Under its assumptions, it bounds their suboptimality and provides a non-asymptotic sample-complexity result for RLIF.

6 THEORETICAL ANALYSIS

  • The theoretical analysis is organized around intervention-reward assumptions and bounds on the suboptimality gap for RLIF and DAgger.It compares each learned policy’s value with the optimal policy’s value.Source: §6 · p. 8, §6.1 · p. 8

6.1 THEORETICAL SETTINGS AND ASSUMPTIONS

  • The analysis uses intervention feedback instead of the true task reward, assigning reward 1 to non-intervened steps and assuming the expert’s policy is not intervened.The expert may use a reference policy to determine intervention; δ is described as the expert’s confidence level.Source: §6.1 · pp. 8–9

6.2 MAIN RESULT

  • Under Assumption 6.1, RLIF’s suboptimality gap is no larger than DAgger’s; the RLIF bound assumes Vπref is known and includes δϵ/(1−γ).The paper also gives a DAgger bound for the case where Vπref is unknown.Source: §6.2 · p. 9
  • The paper also provides a non-asymptotic sample-complexity analysis for learning a policy that maximizes intervention reward.The stated result is in Appendix C.Source: §6.2 · p. 9

7 DISCUSSION AND LIMITATIONS

The discussion presents RLIF as learning from intervention signals without requiring an optimal expert or ground-truth reward, while noting practical constraints around training data and online deployment.

7 DISCUSSION AND LIMITATIONS

  • RLIF derives reward from the expert’s decision to intervene, rather than requiring ground-truth task rewards or strongly relying on an optimal expert.The authors contrast this with conventional reinforcement learning and imitation learning, respectively.Source: §7 · p. 9
  • Training requires an RL method that can use data from both the policy and intervening expert, combining off-policy and on-policy data.The authors report using the off-the-shelf offline RL algorithm RLPD without modification.Source: §7 · p. 9
  • Interactive deployment partly forfeits imitation learning’s avoidance of online deployment, and safety challenges from online exploration remain future work.The authors note expert oversight may still be safer in settings such as autonomous driving.Source: §7 · p. 9

A.1 GRID WORLD NAVIGATION AS A DIAGNOSTIC TASK

The appendix describes a grid-world diagnostic, robot and simulation setups, intervention strategies, training plots, an ablation, baseline comparisons, and experiment parameters.

A.1 GRID WORLD NAVIGATION AS A DIAGNOSTIC TASK

  • In the 6×6 grid world, RLIF’s learned policy matches the optimal start-to-goal path using intervention feedback, although its value function differs from the true value function.Value iteration is used, and the task reward is not observed by RLIF.Source: §A.1 · p. 13, Figure 7 · p. 13

A.2 ROBOT EXPERIMENT DETAILS

  • The robot experiments use a 7-DoF Franka arm for peg insertion and cloth unfolding, with visual feedback and operator takeover counted as negative reward on the preceding step.Success rates are based on 20 trials.Source: §A.2 · p. 13

A.3 EXPERIMENT SETUP DETAILS

  • Simulation experiments use initialized offline datasets and pretrained experts; Adroit performance is measured by success rate, while Walker2d and Hopper use normalized returns.The initial datasets are subsets of D4RL datasets, and the same expert intervenes across strategies at each task and level.Source: §A.3 · pp. 13–14, §A.2 · p. 13, Table 2 · p. 14

A.4 INTERVENTION STRATEGIES

  • Random interventions sample uniformly at 30%, 50%, and 85% rates (intervened steps/total steps), with (I,k) windows (U(t+1,t+10), U(1,5)), (U(t+1,t+5), U(3,7)), and (U(t+1,t+2), U(12,16)), respectively; value-based intervention is likely when expert values exceed policy values by a threshold.Source: §A.4 · p. 14

A.5 EXPERIMENT PLOTS

  • The appendix presents representative RLIF and HG-DAgger training plots across expert levels and intervention strategies.Figure 9 is titled as training plots for RLIF, DAgger, and HG-DAgger.Source: §A.5 · p. 14, Figure 9 · p. 16

A.6 ABLATION EXPERIMENT RESULTS

  • The walker2d ablation varies the reference Q function and expert level.The table caption defines a 10% Qπref as one trained with a 10% expert’s offline dataset.Source: Table 3 · p. 16

A.7 ADDITIONAL BASELINES

  • RLIF consistently performs better than HG-DAgger and DAgger across individual expert levels and averaged over expert levels on the continuous-control tasks.The comparison is reported in Table 4.Source: Table 4 · p. 16

A.8 EXPERIMENT HYPERPARAMETERS

  • The appendix reports training parameters, including 100 rounds with five trajectories per round, and lists simulation and robot-task parameter tables.Tables 5 and 6 cover simulation tasks and Franka insertion, respectively.Source: §A.8 · p. 17, Table 5 · p. 17, Table 6 · p. 17

B.1 SUBOPTIMALITY GAP OF RLIF

The appendix bounds RLIF’s suboptimality under Assumption 6.1, compares the DAgger bound, and gives a bandit construction for a tight gap.

B.1 SUBOPTIMALITY GAP OF RLIF

  • Under Assumption 6.1, RLIF’s suboptimality bound includes an additive δϵ/(1−γ) term when V^πref is known.The theorem concerns a policy optimal for the RLIF reward.Source: §B.1 · p. 18

B.2 SUBOPTIMALITY GAP OF DAGGER (ROSS ET AL., 2011)

  • When V^πref is unknown, the DAgger gap is bounded by V^π⋆(µ)−V^πexp(µ) plus δϵ/(1−γ).The bound therefore includes the expert’s suboptimality relative to the optimal policy.Source: §B.2 · p. 19

B.3 TIGHT SUBOPTIMALITY GAP

  • Example B.3 presents a lower-bound analysis of RLIF using a one-state, two-action bandit with rewards 1 for a1 and 0 for a2; the optimal policy is [1, 0]^T.Source: §B.3 · p. 20

C NON-ASYMPTOTIC ANALYSIS

The non-asymptotic analysis bounds RLIF’s sample complexity using offline-RL tools and intervention-data coverage. It states that the total complexity does not exceed that of solving the original RL problem with true rewards.

C NON-ASYMPTOTIC ANALYSIS

  • The analysis uses LCB-VI to handle offline data and gradually mixed expert-policy data, and notes that RLIF can use other RL subroutines.The authors identify online and hybrid settings as possible alternatives to LCB-VI.Source: C NON-ASYMPTOTIC ANALYSIS · p. 21
  • The analysis bounds the concentrability coefficient for an intervention-data distribution formed by mixing reference-policy data with a full-support distribution.The coefficient is important to the final non-asymptotic bound.Source: C NON-ASYMPTOTIC ANALYSIS · p. 21
  • The total sample complexity for learning a policy that maximizes V^π does not exceed the complexity of solving the original RL problem with true rewards.Corollary C.3 states that, under Assumption 6.1, an algorithm returns an ϵ-optimal policy.Source: C NON-ASYMPTOTIC ANALYSIS · pp. 21–22

D SUPPORTING THEORETICAL RESULTS

This section supplies supporting results from concentration inequalities and offline reinforcement learning convergence analysis.

D SUPPORTING THEORETICAL RESULTS

  • The supporting results include a Hoeffding bound for independent bounded variables and an offline-RL convergence result under stated conditions.The cited offline-RL result concerns learning with a concentrability coefficient.Source: D SUPPORTING THEORETICAL RESULTS · p. 22

E PROPERTIES OF Πopt

The section lists metric properties for δ on Πopt and states a monotonicity property as δ varies.

E PROPERTIES OF Πopt

  • The section states that a quantity is monotonic decreasing in δ, though the supplied passage does not identify that quantity.Source: E PROPERTIES OF Πopt · pp. 22–23
  • The stated metric properties include zero distance from a policy to itself and positive distance between distinct policies in Πoptδ.Source: E PROPERTIES OF Πopt · p. 23
Loading 2311.12996v2…