PaperlayerView exact revision on arXiv

Source-linked AI summary

IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data

Ajay Mandlekar, Fabio Ramos, Byron Boots, Silvio Savarese, Li Fei-Fei, Animesh Garg, Dieter Fox

arXiv:1911.05321v2cs.ROcs.AIcs.LG

TL;DR

Learning from large demonstration datasets is challenging when trajectories are diverse and suboptimal. IRIS combines goal-conditioned short-sequence imitation with high-level goal selection, and its evaluations report better performance than baselines across manipulation tasks. The paper concludes that performant policies can be recovered from large offline manipulation datasets, subject to similar training and testing task-instance distributions.

  • Problem

    Large-scale demonstrations can be diverse and suboptimal, whereas conventional imitation learning assumes near-optimal, unimodal data and can deteriorate when those assumptions do not hold.

    Source: §I · p. 1

  • Method

    IRIS pairs a low-level goal-conditioned controller that imitates short demonstration sequences with a high-level mechanism that proposes and selects goals.

    Source: §I · p. 1, A. Overview · p. 3

  • Results

    Across Lift and Cans, IRIS outperforms baselines, which attain 18% on Lift and fail to solve Cans.

    Source: §VII · p. 6

  • Takeaways & Limitations

    IRIS recovers performant policies from large manipulation datasets by combining goal-conditioned imitation with high-level goal selection.

    Source: §VIII · p. 6

    Training and testing task-instance distributions must be similar; generalizing across can locations requires sufficient variety in training initial locations.

    Source: §VIII · p. 6

Abstract

from arXiv · show

Learning from offline task demonstrations is a problem of great interest in robotics. For simple short-horizon manipulation tasks with modest variation in task instances, offline learning from a small set of demonstrations can produce controllers that successfully solve the task. However, leveraging a fixed batch of data can be problematic for larger datasets and longer-horizon tasks with greater variations. The data can exhibit substantial diversity and consist of suboptimal solution approaches. In this paper, we propose Implicit Reinforcement without Interaction at Scale (IRIS), a novel framework for learning from large-scale demonstration datasets. IRIS factorizes the control problem into a goal-conditioned low-level controller that imitates short demonstration sequences and a high-level goal selection mechanism that sets goals for the low-level and selectively combines parts of suboptimal solutions leading to more successful task completions. We evaluate IRIS across three datasets, including the RoboTurk Cans dataset collected by humans via crowdsourcing, and show that performant policies can be learned from purely offline learning. Additional results at https://sites.google.com/stanford.edu/iris/ .

I. INTRODUCTION

The introduction asks whether policies can be learned from large demonstration datasets without system interaction, despite variation in demonstration quality and solution approaches. IRIS addresses this with selective imitation of local sequences and is reported to outperform imitation-learning and batch-RL baselines.

I. INTRODUCTION

  • Reinforcement learning for manipulation faces costly exploration in large state and action spaces and requires carefully shaped, task-specific rewards.Imitation learning can use demonstrations in place of a reward function and lessen the burden of exploration.Source: §I · p. 1
  • Imitation learning has mainly used small datasets collected by one decision maker, despite the availability of mechanisms for collecting thousands of demonstrations in days.The paper identifies large-scale, diverse supervision as a setting for policy learning.Source: §I · p. 1
  • The paper asks whether robust, performant policies can be learned purely from externally provided demonstrations, such as varied human approaches to pick-and-place.The motivating example uses demonstrations of placing a can on a shelf from several initial poses.Source: §I · p. 1
  • Large-scale demonstrations can vary in both quality and solution approach, while conventional imitation learning assumes near-optimal, unimodal data and can deteriorate on lower-quality or multimodal demonstrations.Different approaches to grasping a can can be valid but vary in efficiency.Source: §I · p. 1
  • IRIS learns offline from diverse, suboptimal demonstrations by selectively imitating local sequences, and significantly outperforms imitation-learning and batch-RL baselines.The evaluation spans a pedagogical dataset, a highly suboptimal dataset, and a crowdsourced dataset, using sparse task-completion rewards at demonstration ends.Source: §I · p. 1

II. RELATED WORK

Related work covers imitation and reinforcement-learning approaches, while the preliminaries formalize offline learning from goal-reaching trajectories that may be suboptimal and multimodal.

II. RELATED WORK

  • Prior imitation methods include offline behavioral cloning and online inverse reinforcement learning; offline methods can suffer from covariate shift, while online methods require interaction.Methods using off-policy deep RL with demonstrations still require interaction, and batch RL had not addressed robot manipulation with diverse demonstrations in the cited work.Source: §II · p. 2
  • IRIS differs from HIRO by targeting fixed offline data and training its low-level policy with supervised imitation rather than an off-policy RL update.The related work notes that off-policy RL updates can be unsuitable for offline learning.Source: §II · p. 2
  • RoboTurk uses crowdsourced human supervision to collect successful demonstrations in simulation, which IRIS uses without collecting additional experience.The cited self-supervised collection approaches gathered extensive robot interaction, although most interactions were unsuccessful.Source: §II · p. 2

III. PRELIMINARIES

  • The task formulation uses a sparse reward of one upon entering a goal state, so maximizing expected return corresponds to solving tasks quickly and consistently.Each initial state represents a task instance, and goal states are absorbing.Source: §III · p. 2
  • The offline dataset contains goal-reaching trajectories from multiple policies, with no additional MDP experience available to the learning method.A trajectory is goal-reaching when its final state is a goal state.Source: §III · pp. 2–3
  • Demonstrations may be slower than necessary and may solve tasks through multiple modes, so the learned policy should not imitate every recorded action indiscriminately.The examples include different ways to grasp a can, including knocking it down before picking it up.Source: §III · p. 3

A. Overview

IRIS divides control into a high-level mechanism that sets goals and a low-level controller that attempts to reach each goal over a fixed interval. The high level proposes and ranks goals, enabling selective imitation of local dataset sequences.

A. Overview

  • The high level selects a goal that remains fixed for T timesteps, then the low-level controller acts toward it in closed loop before control returns to the high level.This cycle repeats.Source: A. Overview · p. 3
  • A conditional VAE models future observations from the current state and proposes goals, while a value function selects the most promising proposal.The low-level recurrent network outputs actions from the current observation and selected goal.Source: A. Overview · p. 3
  • The components together support selective imitation of local sequences from the dataset.The training loop is given in Algorithm 1.Source: A. Overview · p. 3

B. Low-Level Goal-Conditioned Imitation Controller

The low-level goal-conditioned RNN learns to reproduce short action sequences toward their final observations. At the high level, a conditional generative model proposes future-state goals and a value function ranks them.

B. Low-Level Goal-Conditioned Imitation Controller

  • The low-level goal-conditioned RNN is trained on length-T trajectory segments, treating each segment’s final observation as the goal and imitating its demonstrated actions.This yields unimodal imitation over short sequences for reaching different goals.Source: B. Low-Level Goal-Conditioned… · p. 3

C. High-Level Goal Selection Mechanism

  • The high-level mechanism uses a conditional VAE to propose goals and a value function to model their expected returns and select among them.The conditional VAE is trained on current and future observations sampled from dataset trajectories.Source: C. High-Level Goal Selection Mechanism · p. 3
  • The goal VAE learns to reconstruct a future observation from the current observation and a latent code, with a KL regularizer matching the latent distribution to a standard normal prior.At test time, sampling from that prior lets the decoder generate conditional goal proposals.Source: C. High-Level Goal Selection Mechanism · p. 4
  • The value network uses a BCQ-style update with a state-conditioned action VAE and a temporal-difference loss for Q.The target considers candidate actions proposed by the action VAE and maximizes Q over those candidates.Source: C. High-Level Goal Selection Mechanism · p. 4

V. IRIS: CHALLENGES OF PURELY OFFLINE DATA

IRIS separates short, goal-conditioned action sequences from the modeling of diverse solution approaches. Its high-level goal selection evaluates candidate goals to account for suboptimal data, while training remains constrained to the offline data distribution.

Learning from diverse solution approaches

  • The goal-conditioned controller imitates short, fine-grained action sequences, while the generative goal model proposes reachable future observations to represent diverse solution approaches.This separates unimodal sequence reproduction from modeling trajectory diversity, enabling selective imitation.Source: §V · p. 4

Learning from suboptimal data

  • Because the low-level controller acts for few timesteps, IRIS relies on high-level value learning to select goals with the highest expected task returns.The method argues that potential improvements in reaching a nearby goal within a small number of steps are negligible.Source: §V · p. 4

Learning from off-policy datasets

  • IRIS constrains learning to the training-data distribution: the controller imitates its sequences, the goal model proposes its observations, and the Q-network is queried on in-distribution state-action pairs.The Q-network constraint is intended to mitigate extrapolation error.Source: §V · p. 4

A. Tasks and Datasets

The experiments use a pedagogical navigation task and two robot manipulation datasets, including crowdsourced can demonstrations with diverse, suboptimal strategies. The reported comparisons cover model performance and dataset size, with table metrics evaluated across randomized task instances and seeds.

Performance comparisons

  • Figure 3 compares IRIS and baselines on Lift and Cans and compares IRIS across dataset sizes; Table I reports success rate, successful-rollout length, and discounted return.Table evaluations use 100 randomized task instances and three random seeds; most models decrease or maintain successful-rollout lengths versus the original trajectories.Source: Figure 3 · p. 5, Table I · p. 5

Graph Reach

  • The Graph Reach dataset contains 250 varied paths, and selective imitation of path segments can recover the straight-line route from start to goal.The paths are generated on a 5x5 grid; demonstrations average 3844 timesteps.Source: A. Tasks and Datasets · pp. 4–5

Robosuite Lift

  • The Lift dataset contains 137 demonstrations from one human, whose intentionally inefficient movements create low-value state-action pairs for algorithms to avoid.The demonstrations average 622 timesteps.Source: A. Tasks and Datasets · p. 5

RoboTurk Can Pick and Place

  • The Can Pick and Place dataset uses the fastest 225 of over 1100 crowdsourced trajectories, which vary in grasping strategy; its image variant uses front-view camera observations.The demonstrations average 589 timesteps.Source: A. Tasks and Datasets · p. 5

B. Experiment Details

The experiment compares IRIS with behavioral-cloning and batch-reinforcement-learning baselines, as well as two IRIS variants. All training is offline, with no additional samples collected.

Baselines

  • The comparison includes Behavioral Cloning, which regresses over dataset state-action pairs, and its recurrent variant, BC-RNN.Source: B. Experiment Details · p. 5
  • IRIS is also compared with BCQ, a batch reinforcement-learning baseline for continuous control.Source: B. Experiment Details · p. 5

IRIS variants

  • Two IRIS variants test removing the high-level Q-function or replacing the Goal VAE with a deterministic goal-prediction network; all training is offline.No algorithm is allowed to collect additional samples.Source: B. Experiment Details · p. 5

VII. RESULTS

IRIS variants outperform baselines on the challenging Lift and Cans tasks, while the results examine demonstration reuse, dataset size, and image observations.

VII. RESULTS

  • On Graph Reach, all models solve consistently while IRIS and BCQ variants solve faster; Fig. 4 shows IRIS reusing demonstrated modes as BCQ extrapolates.BC converges to a dataset mode that is slow to reach the goal; IRIS also varies across rollouts.Source: §VII · pp. 5–6, Figure 4 · p. 6
  • 70-80% on Lift and 20-30% on Cans are IRIS success rates, versus baseline 18% on Lift and no Cans solutions.The authors attribute the gap to goal-directed imitation, which the baselines lack.Source: §VII · p. 6
  • IRIS, no Q achieves 10% higher success rate than IRIS, no Goal VAE on Cans; the VAE variant replaces deterministic goal prediction.The authors hypothesize that value selection can avoid goals where the demonstrator moves away from the cube or drifts side to side on Lift.Source: §VII · p. 6
  • The two-level decomposition reproduces demonstrated behaviors over time while leaving the high-level component flexibility to choose which behaviors to reproduce.Value selection significantly improves Lift performance but does not improve Cans performance.Source: §VII · p. 6
  • The medium-sized Lift dataset matches full-data asymptotic performance, whereas medium-sized Cans data restricts performance, indicating greater data benefits with more task-instance variation.The small datasets, containing the best 10% of trajectories, lead to poor performance.Source: §VII · p. 6
  • On RoboTurk Cans images, all IRIS variants outperform their low-dimensional counterparts, with the best reaching 42.7% success using a 32-dimensional landmark representation.The representation uses 16 landmark locations extracted from 128 by 128 RGB images.Source: §VII · p. 6

VIII. CONCLUSION

IRIS selectively imitates local sequences from diverse, suboptimal demonstrations and recovers performant policies from large manipulation datasets. The authors identify similarity between training and testing task-instance distributions as a limitation.

VIII. CONCLUSION

  • IRIS learns offline from diverse, suboptimal demonstrations by selectively imitating local sequences.The framework decomposes policy learning into goal-conditioned imitation and high-level goal selection.Source: §VIII · p. 6
  • IRIS significantly outperforms other baselines on large manipulation datasets through its decomposition into goal-conditioned imitation and high-level goal selection.Source: §VIII · p. 6
  • Expected test-time generalization is bounded by task-instance coverage in training, such as the range of initial can locations represented.The authors identify domain adaptation for novel test-time scenarios as future work.Source: §VIII · p. 6
Loading 1911.05321v2…