PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation

Suraj Nair, Eric Mitchell, Kevin Chen, Brian Ichter, Silvio Savarese, Chelsea Finn

arXiv:2109.01115v2cs.ROcs.AIcs.LG

TL;DR

The paper addresses how to ground natural-language task instructions for vision-based manipulation using scalable offline data. It introduces LOReL, which learns language-conditioned rewards from crowd-sourced annotations on sub-optimal robot trajectories and uses them for control. In simulation it outperforms language-conditioned imitation and goal-image comparisons, while also completing language-specified skills on a real robot.

  • Problem

    Grounding natural-language instructions in robot observations requires diverse language-annotated interaction data, while goal images are effortful and can over- or under-specify tasks.

  • Method

    LOReL combines sub-optimal offline robot datasets with crowd-sourced labels to train a classifier that predicts whether a state transition completes a language instruction, producing a reward for offline RL.

  • Results

    25% more effective than language-conditioned imitation techniques and ∼30% more effective than goal-image comparisons in simulated manipulation tasks; on a real robot, it achieved 66% average success across five language-conditioned skills.

  • Takeaways & Limitations

    LOReL supports language-conditioned visuomotor manipulation from sub-optimal offline data and generalizes from scripted instructions to unseen natural language zero-shot.

  • Takeaways & Limitations

    LOReL currently captures state-change tasks but not path-dependent tasks, and the paper focuses on short-horizon skills rather than composing long-horizon tasks.

Abstract

from arXiv · show

We study the problem of learning a range of vision-based manipulation tasks from a large offline dataset of robot interaction. In order to accomplish this, humans need easy and effective ways of specifying tasks to the robot. Goal images are one popular form of task specification, as they are already grounded in the robot's observation space. However, goal images also have a number of drawbacks: they are inconvenient for humans to provide, they can over-specify the desired behavior leading to a sparse reward signal, or under-specify task information in the case of non-goal reaching tasks. Natural language provides a convenient and flexible alternative for task specification, but comes with the challenge of grounding language in the robot's observation space. To scalably learn this grounding we propose to leverage offline robot datasets (including highly sub-optimal, autonomously collected data) with crowd-sourced natural language labels. With this data, we learn a simple classifier which predicts if a change in state completes a language instruction. This provides a language-conditioned reward function that can then be used for offline multi-task RL. In our experiments, we find that on language-conditioned manipulation tasks our approach outperforms both goal-image specifications and language conditioned imitation techniques by more than 25%, and is able to perform visuomotor tasks from natural language, such as "open the right drawer" and "move the stapler", on a Franka Emika Panda robot.

1 Introduction

The paper targets language-conditioned visuomotor manipulation from offline robot data, addressing limitations of goal-state specifications and the difficulty of grounding language in observations. LOReL uses crowd-sourced labels and sub-optimal offline data to learn language-conditioned rewards for control.

  • Motivation: Goal-state specifications require human effort and can over- or under-specify manipulation tasks.A single goal image may encode irrelevant object and robot positions or represent only one instance of success.
  • Results: LOReL also performs language-conditioned visuomotor tasks on a real Franka Emika Panda robot.The introduction cites tasks such as opening the right drawer and moving the stapler.
  • Motivation: Grounding natural-language instructions in high-dimensional robot observations requires diverse interaction data paired with language annotations.Human teleoperated data is costly and time consuming to collect at scale.
  • Approach: LOReL combines autonomously collected offline data with post-hoc crowd-sourced labels and learns a classifier that predicts whether a state transition completes an instruction.The classifier provides a language-conditioned reward for offline reinforcement learning without assuming optimal actions.
  • Results: 25% more effective than language-conditioned imitation techniques and ∼30% more effective than goal-image comparisons in simulated object manipulation tasks.The comparison used data collected by a random policy; pretrained language models also supported zero-shot generalization to unseen instructions.

2 Related Work

Prior work grounds language for instruction following, rewards, offline imitation, and goal-conditioned control, but this paper focuses on image-based robotic manipulation without requiring predefined motion primitives or near-optimal data.

  • Language grounding: Instruction-following research spans formal or natural language agents, often studied in simulated games, navigation, or object interaction.This work instead studies vision-based robotic manipulation control.
  • Language grounding: Earlier approaches mapped semantic language parses to predefined motion primitives or actions, whereas this work learns control directly from images and language.The proposed setting does not assume predefined motion primitives.
  • Language-conditioned rewards: Language-conditioned reward methods have commonly learned rewards online from demonstrations or successful task examples and then optimized policies with online reinforcement learning.Those approaches may use the agent’s own online experience to train the reward.
  • Offline learning: Prior offline language-conditioned robot agents use demonstrations or human play data and generally treat the data as near optimal for behavior cloning.LOReL differs by learning from sub-optimal offline data.
  • Goal-conditioned control: Goal-conditioned learning is another common vision-based robotics approach, with methods ranging from model-free goal relabeling to model-based planning.Task specifications in this literature include task IDs, demonstrations, and rewards.

3 Preliminaries

The paper formulates language-conditioned control as an MDP problem with an unobserved task reward inferred from language-labeled offline trajectories. Its reward model captures state-change tasks but not path-dependent tasks.

  • Problem formulation: The agent operates over K tasks in an MDP with RGB-image states, robot actions, stochastic dynamics, task rewards, and a finite horizon.The task reward is binary for completing a task from an initial state.
  • Problem formulation: Multiple language instructions may describe one task, while one instruction may describe multiple task instances.For example, different phrases can describe picking up the blue marker, and “pick up the marker” can refer to different markers.
  • Data assumptions: The offline dataset contains trajectories paired with language instructions that describe at least one completed task, although those tasks need not be among the target tasks.The data may therefore include unrelated behaviors such as doing nothing.
  • Reward and policy: The learned reward model conditions on an instruction, an initial state, and a current state to infer task completion, and the policy maximizes expected cumulative reward.The policy is language conditioned and produces robot actions.
  • Scope: The formulation captures tasks reflected in state changes but excludes path-dependent tasks such as closing a drawer slowly.The limitation follows from rewarding state transitions rather than full action trajectories.

4 Language-conditioned Offline Reward Learning (LOReL)

LOReL learns a language-conditioned reward by classifying whether transitions between initial and final visual states satisfy an instruction, then uses that reward with visual model-predictive control to select actions. Training uses positive transitions, mismatched-instruction negatives, reversed-state negatives, augmentation, and a pretrained language encoder.

  • Learning the Reward Function: LOReL learns a language-conditioned reward classifier from initial/final states, annotations, and potentially sub-optimal offline trajectories.The method does not assume that actions in the dataset are optimal; it uses annotated state changes to identify task completion.
  • Learning the Reward Function: Positive examples pair an instruction with transitions whose early initial state and late final state satisfy that command.The positive-selection scheme also labels nearby initial and final states as positives using a noisy temporal window.
  • Learning the Reward Function: Negative examples use transitions labeled with a different instruction and reversed initial/final states, with the latter encouraging temporal-progress sensitivity.Balanced batches contain both negative types; reversed-state negatives discourage rewards that focus only on objects.
  • Learning the Reward Function: Visual augmentation and language-embedding noise are used to reduce classifier over-fitting and sparse or incorrect reward signals.Augmentations include affine transformations, color jitter, and uniform noise in the language embedding space.
  • Learning the Reward Function: A fixed pretrained distilBERT encoder represents commands before classification, supporting generalization to unseen natural-language instructions.The encoder maps commands to fixed-length vectors in R768.
  • Learning Language Conditioned Policies with Visual Model Predictive Control: Visual model-predictive control predicts future states for sampled action sequences, ranks them with LOReL, and iteratively executes the best sequence.The learned visual dynamics model supplies predictions, while the cross-entropy method optimizes action sequences to maximize language-conditioned reward.

5 Experiments

Experiments evaluate LOReL against prior methods in simulation, test generalization to unseen language, and deploy it on a real robot. LOReL outperforms comparison methods in simulated tasks, maintains performance under rephrasing, and completes language-conditioned skills on a Franka Panda.

  • Simulated Domain: Simulation uses 50,000 randomly collected episodes in Meta-World, procedurally annotated into 2,311 unique instructions across six target tasks.The domain contains a Sawyer robot, drawer, faucet, and two mugs.
  • 5.1 Does LOReL enable effective language-conditioned behavior compared to prior work?: Over 25%, LOReL outperforms LCBC, LCRL, and LPIPS/Pixel goal-image specifications on six simulated language-conditioned tasks.Success rates and standard errors are computed over three seeds of 100 trials; pixel-cost goal images perform comparably to a random policy, while LPIPS remains about 30% worse than LOReL.
  • 5.2 Can LOReL generalize zero-shot to unseen natural language commands?: Pretrained language representations preserve generalization under rephrasing: changing a verb or noun causes an average 5% success-rate drop, while changing both or using human language causes at most a 10% drop.Without the pretrained language model, unseen-instruction performance drops by up to 23% rather than 10%.
  • 5.3 Can LOReL be used to learn language-conditioned visuomotor skills on a real robot?: The real-robot study uses a Franka Emika Panda over an IKEA desk with a 3,000-episode, 150,000-frame offline dataset and 6,000 crowd-sourced annotations.The annotations contain 1,699 unique instructions, and the dataset comes from a sub-optimal online-RL replay buffer.
  • 5.3 Can LOReL be used to learn language-conditioned visuomotor skills on a real robot?: 66% average success enables LOReL to complete five language-conditioned skills on the real robot.Removing reversed-state negatives reduces performance by 30%, indicating their importance for capturing temporal progress and avoiding object-based overfitting.
  • 5.3 Can LOReL be used to learn language-conditioned visuomotor skills on a real robot?: LOReL succeeds 7/10 times on a longer drawer-opening command and 5/10 times on a more complex stapler-moving command.These tests replace shorter commands with more descriptive vocabulary and longer instruction phrasing.

6 Limitations and Future Work

The paper reports that LOReL handles visuomotor tasks and unseen language but remains limited to short-horizon tasks represented by changes in state. Future work targets path-dependent behavior, long-horizon composition, and unified language and goal-image specification.

  • Limitations: LOReL cannot currently capture path-dependent tasks whose success is not represented by a state change, such as moving in a circle slowly.The authors propose training on full video clips as one possible direction.
  • Future Work: Composing short-horizon language skills into long-horizon tasks remains unresolved and requires longer-horizon data, stronger planners, and more capable visual dynamics models.The limitation concerns extending the demonstrated short-horizon language-conditioned skills to real-world task sequences.
  • Future Work: Goal images remain self-supervised and can be effective in some settings, so the paper identifies unifying language and goal-image specification as future work.This frames language as an alternative rather than a complete replacement for goal images.

A Method Implementation Details

LOReL learns language-conditioned rewards from state changes and uses visual model-predictive control to select actions. The implementation combines image and language encoders, with separate imitation, Q-learning, and goal-image baselines.

  • LOReL reward learning: The reward classifier predicts whether a state change completes a language instruction from the initial state, current state, and instruction.
  • LOReL reward learning: LOReL concatenates image and sentence embeddings, then maps them through fully connected layers to a sigmoid reward prediction.The image encoder produces an embedding of size L, while the sentence encoder produces a 768-dimensional vector.
  • Training details: Training uses different positive-state sampling ranges in simulation and on the real robot, with α = 0 and α = 0.25 respectively.Simulation uses episode initial and final states; real-robot positives use states from the first and last quarters of episodes.
  • Training details: Visual augmentation applies color jitter and affine transformations, while instruction embeddings receive uniform noise to reduce classifier overfitting.
  • Action selection: Model-predictive control samples action sequences, predicts their outcomes with learned dynamics, ranks them by reward, and refits the sampling distribution iteratively.Simulation uses 20-action sequences, 200 samples, the top 10%, and three refinement iterations; the real robot uses five-action sequences and 48 samples per iteration.
  • Baselines: The imitation baseline minimizes action mean-squared error, while the language-conditioned Q-learning baseline predicts discounted returns and selects high-value sampled actions.
  • Baselines: Pixel and LPIPS baselines use the same planner and visual dynamics model as LOReL but optimize similarity to a goal image.Pixel uses negative ℓ2 pixel distance; LPIPS uses LPIPS similarity.

B Environment/Data Details

Experiments use simulated and real-robot environments with offline interaction data, including random-policy and autonomously trained RL-agent data. Tasks are specified by language or goal images and evaluated through repeated randomized trials.

  • Simulation environment: The simulated environment uses a Sawyer arm interacting with a drawer, faucet, and two mugs from 64 × 64 RGB observations.Episodes last 20 timesteps and use delta end-effector control.
  • Simulation data: The simulation dataset contains 50,000 randomly collected episodes, totaling 1M frames, with randomized resets and procedurally recorded object movements.
  • Robot environment: The real environment uses a Franka Emika Panda over an Ikea desk with four camera views and delta end-effector control.Each camera returns a 64 × 64 × 3 RGB image; grasping is not used in evaluation tasks.
  • Robot data: The robot dataset contains 3,000 autonomously trained RL episodes with meaningful, near-random, and highly sub-optimal behavior.
  • Robot data: Crowdsourcing produced 6,000 annotations, and filtering reduced the training set to 1,600 episodes.Each episode received two annotations; episodes judged uninformative were removed.
  • Evaluation: Simulation evaluates six language-conditioned manipulation tasks, including drawer, faucet, and mug movements, with corresponding goal-image comparisons.
  • Evaluation: Each simulated method uses three seeds and 100 randomized trials per task to compute success rates.
  • Evaluation: A second simulation experiment rephrases instructions and samples among human-provided unseen language variants.

C.4 Robot Tasks

The real-robot evaluation covers drawer opening, stapler movement, marker reaching, and cabinet reaching. Success is defined by task-specific spatial or contact criteria under randomized initial conditions.

  • Robot tasks: The robot evaluates five tasks: opening either drawer, moving a stapler, reaching markers in the left drawer, and reaching the cabinet.
  • Success criteria: Drawer success requires opening the drawer at least 1 inch, while stapler success requires any translation on the table.
  • Success criteria: Marker-reaching success requires the gripper tip to come within 1 inch of any marker in the left drawer.
  • Success criteria: Cabinet-reaching success is defined as making contact with the cabinet.
  • Initialization: Initial robot positions are task-specific, with drawers slightly open and markers randomized within the left drawer.Each episode randomizes the initial position by up to 5 cm from its main initialization.
  • Annotation: Each episode is annotated from video by two crowdworkers, yielding 6,000 annotations across 3,000 episodes.

C.5 Experiment 3 Details

The third real-robot experiment compares LOReL with an ablation that omits flipped initial/final negatives, using repeated trials and more sophisticated instructions for selected tasks.

  • Experiment protocol: The real-robot experiment runs 10 trials per task for LOReL and for an ablation trained without flipped initial/final negatives.
  • Experiment protocol: Each real-robot episode lasts 30 timesteps, with the agent planning five actions at a time.
  • Instruction variation: Tasks (1) and (3) additionally receive 10 trials with more sophisticated instructions.
  • Instruction variation: Figure 9 lists the 25 most frequent instructions and their occurrence counts in the filtered robot dataset.

D.1 Additional Ablations

Additional ablations show that negative-example design, filtering “do nothing” episodes, and noisy positive labels materially affect reward learning and real-robot performance.

  • Negative examples: 66% to 36%: removing flipped negatives sharply reduced average real-robot success, while simulation performance changed from 56% to 55%.Flipped negatives help prevent overfitting to scene objects and capture temporal progress; their impact is larger with 1.6K real-robot episodes than with 50K simulation episodes.
  • Negative examples: 56% to 27%: removing randomly chosen negatives substantially reduced average simulation success.These negatives teach the agent how different language instructions differ.
  • Real-robot ablations: Table 3 identifies filtering “do nothing” episodes and training with noisy positives as important real-robot ablations.The caption summarizes both design choices as important for real-robot performance.
  • Data filtering: Filtering “do nothing” episodes can make reward training considerably easier, especially when such episodes dominate batches and negative examples.The real-robot dataset contains nearly half “do nothing” data, whereas simulation contains approximately 20%.
  • Positive examples: Generating more positive examples through noisy labeling is important for real-robot performance with only 1.6K filtered episodes.The training-curve ablation also reports that removing noisy positives makes fitting easier but can worsen the reward used for planning.

D.2 LOReL Training Curves

LOReL training curves compare the full method with ablations in simulation and on real-robot data, while additional results examine unseen-command performance and ranked trajectories.

  • LOReL training curves: Figure 12 compares LOReL training and test accuracy curves with simulation and real-robot ablations.Simulation ablations remove randomly chosen or flipped negatives; real-robot ablations remove filtering, flipped negatives, or noisy positives.
  • LOReL training curves: Removing random negatives makes fitting easier but produces a worse reward and worse planning performance in simulation.For planning, the last checkpoint before test accuracy begins decreasing is used for each ablation.
  • LOReL training curves: Without filtering, real-robot data are harder to fit; without noisy positives, fitting is easier but the planning reward can worsen.These observations come from the real-robot curves in Figure 12.
  • Unseen instruction generalization: Figure 13 reports per-task success rates for original commands and unseen variations.The variations include unseen verbs, nouns, verb-noun combinations, and human-generated natural-language commands.
  • Trajectory ranking: Figures 14–18 show qualitative examples of trajectories ranked under language instructions on the real robot.The instructions cover opening drawers, moving the stapler, and reaching the marker or cabinet.
Loading 2109.01115v2…