PaperlayerView exact revision on arXiv

Source-linked AI summary

Yell At Your Robot: Improving On-the-Fly from Language Corrections

Lucy Xiaoyang Shi, Zheyuan Hu, Tony Z. Zhao, Archit Sharma, Karl Pertsch, Jianlan Luo, Sergey Levine, Chelsea Finn

arXiv:2403.12910v1cs.ROcs.AIcs.LG

TL;DR

Long-horizon dexterous tasks are difficult to supervise robustly because demonstrations are costly and indirect knowledge is not grounded in robot-specific behaviors. YAY Robot uses occasional verbal corrections to control low-level skills immediately and fine-tune the high-level policy over time, improving performance on three real-robot manipulation tasks. The approach achieves reported gains from 15% to 50% during on-the-fly correction and from 15% to 45% after high-level-policy training.

  • Problem

    Robust high-level supervision is difficult because complete long-horizon demonstrations are costly, while external language or video knowledge is indirectly grounded in robot behaviors.

  • Method

    YAY Robot temporarily applies human language corrections through a language-conditioned low-level policy and fine-tunes the high-level policy on recorded interventions.

  • Results

    15% to 50% real-time improvement and 15% to 45% improvement after high-level training were reported across three bi-manual, multi-stage manipulation tasks.

  • Takeaways & Limitations

    Verbal corrections provide supervision for both immediate adaptation and iterative improvement of long-horizon robotic behavior without additional teleoperation.

  • Takeaways & Limitations

    The approach critically relies on a performant low-level policy that can react successfully to many distinct language commands.

Abstract

from arXiv · show

Hierarchical policies that combine language and low-level control have been shown to perform impressively long-horizon robotic tasks, by leveraging either zero-shot high-level planners like pretrained language and vision-language models (LLMs/VLMs) or models trained on annotated robotic demonstrations. However, for complex and dexterous skills, attaining high success rates on long-horizon tasks still represents a major challenge -- the longer the task is, the more likely it is that some stage will fail. Can humans help the robot to continuously improve its long-horizon task performance through intuitive and natural feedback? In this paper, we make the following observation: high-level policies that index into sufficiently rich and expressive low-level language-conditioned skills can be readily supervised with human feedback in the form of language corrections. We show that even fine-grained corrections, such as small movements ("move a bit to the left"), can be effectively incorporated into high-level policies, and that such corrections can be readily obtained from humans observing the robot and making occasional suggestions. This framework enables robots not only to rapidly adapt to real-time language feedback, but also incorporate this feedback into an iterative training scheme that improves the high-level policy's ability to correct errors in both low-level execution and high-level decision-making purely from verbal feedback. Our evaluation on real hardware shows that this leads to significant performance improvement in long-horizon, dexterous manipulation tasks without the need for any additional teleoperation. Videos and code are available at https://yay-robot.github.io/.

I. INTRODUCTION

YAY Robot addresses the difficulty of training robust high-level policies for long-horizon manipulation by using natural-language corrections during deployment and subsequent policy fine-tuning. The system combines language-conditioned low-level skills with a high-level instruction policy, enabling immediate adaptation and iterative improvement from occasional human feedback.

  • Motivation: Long-horizon manipulation requires sequencing multiple primitives, motivating hierarchical policies that command low-level behaviors through language.Packing items requires repeated grasping, maneuvering, and insertion steps.
  • Motivation: Scalable high-level supervision remains difficult because complete demonstrations are costly, while knowledge from LLMs, VLMs, language-only data, and human videos is indirectly grounded.These sources may leave policies brittle or insufficiently aware of robot-specific skill effectiveness.
  • Core idea: YAY Robot treats human language corrections as high-level feedback analogous to DAgger, allowing the policy to address errors from both low-level execution and high-level decisions.The approach supports robust recovery from failures during long-horizon tasks.
  • Core idea: YAY Robot integrates real-time adaptation with continuous improvement by temporarily applying user commands and later fine-tuning the high-level policy on those interventions.The system is designed to improve through user interaction rather than relying on one-time correction or post-hoc relabeling.
  • Policy design: The language-conditioned low-level policy supports free-form commands ranging from fine-grained movements to high-level preferences within one neural network.Examples include moving a sponge slightly lower or changing gripper orientation.
  • Results: 15% to 50% real-time improvement and 15% to 45% improvement after high-level training were reported across three bi-manual, multi-stage manipulation tasks.The tasks were bag packing, trail-mix preparation, and cleaning gummies from a plate.
  • Positioning: Unlike shared-autonomy systems requiring perpetual corrections, YAY Robot is intended to operate autonomously while accepting flexible language interventions for dexterous manipulation.The paper contrasts its flexible input with systems limited to fixed spatial movements.

III. YELL AT YOUR ROBOT

YAY Robot formulates partially observed robot manipulation around hierarchical language-conditioned policies trained from demonstrations and corrective interaction data. The low-level policy remains fixed during post-training while human corrections fine-tune the high-level policy.

  • Preliminaries: The manipulation problem is formulated as a partially observed Markov Decision Process, with the robot receiving observations rather than true states.The MDP is represented as M = (S, A, P, p0).
  • Preliminaries: The base dataset contains observation, action, and language-instruction sequences for training both high-level and low-level policies.Observations include RGB images and robotic proprioceptive information.
  • Preliminaries: The dataset is assumed to cover the visuomotor skills needed for tasks together with mistakes and recovery behaviors.This coverage supports learning both task execution and correction-relevant behavior.
  • Iterative improvement: During post-training, the correction dataset records observations and user language but omits actions because the low-level policy is frozen.Only the high-level policy parameters are updated.
  • Low-level policy: An expanded language-conditioned skill set gives the high-level policy flexibility to orchestrate recovery behaviors and lets the low-level policy execute adjustments.The design goes beyond the minimal skills required for nominal task completion.
  • Low-level policy: The low-level policy maps the current observation and language instruction to an action and is trained with a standard behavior-cloning objective.The loss compares predicted continuous actions with ground-truth actions using an l1 or l2 loss.

C. High-Level Policy for Autonomous Instruction Generation

YAY Robot uses a history-conditioned high-level policy to generate language instructions for a low-level controller, while deployment-time user commands can override those instructions and supply training data for later fine-tuning.

  • C. High-Level Policy for Autonomous Instruction Generation: A hierarchical policy reuses primitive skills by generating language instructions that guide the low-level policy.The high-level policy is trained to compose behaviors for longer tasks.
  • C. High-Level Policy for Autonomous Instruction Generation: The architecture processes RGB images and joint positions, with a Vision Transformer predicting language embeddings and ACT generating target motor actions.The resulting pipeline translates commands such as “Pick up the bag” into joint movements.
  • C. High-Level Policy for Autonomous Instruction Generation: The high-level policy conditions instruction prediction on a brief history of observations because one observation can support multiple reasonable commands.Examples include “tilt the scoop down” and “go a bit higher.”
  • C. High-Level Policy for Autonomous Instruction Generation: πH(l_t|o_{t−k:t}) predicts a language embedding from temporal observation context and is trained end-to-end with cross-entropy loss and cosine-similarity logits.The temporal context is the observation sequence available up to time t.
  • D. Policy Integration and Adaptation to Human Feedback: During deployment, a human can issue a corrective language command to address erroneous behavior or express a preference.The intervention directly enters the language-conditioned control pipeline.
  • D. Policy Integration and Adaptation to Human Feedback: The deployment policy selects the low-level action from the user command when an intervention occurs and otherwise uses the high-level policy’s instruction.This conditional routing temporarily overrides autonomous instruction generation.
  • D. Policy Integration and Adaptation to Human Feedback: Each intervention is stored as (o_{t−k:t}, l_user) in D_corr and later used to fine-tune πH.The high-level instruction is the maximum-probability output l_H = arg max πH(· | o_{t−k:t}).

E. Continuous Improvement from Human Feedback

YAY Robot continuously improves its high-level policy by incorporating human corrective feedback into iterative fine-tuning alongside the base dataset.

  • Human language feedback is incorporated into the high-level policy in a process conceptually similar to Human-Gated DAgger.
  • The policy is fine-tuned on the base dataset together with the correction dataset to expose it to mistakes requiring corrective instructions.
  • After each user-interaction and feedback-collection iteration, the policy is fine-tuned to reflect the new data.
  • π(n)H and π(n+1)H denote the high-level policy before and after the n-th fine-tuning iteration.
  • D(n)corr is the corrective-feedback dataset obtained at iteration n, and the process is intended to improve autonomous handling without frequent interventions.

IV. PRACTICAL INSTANTIATION AND IMPLEMENTATION

The implementation streamlines language-annotated data collection and separates instructions from corrections while targeting correction skills relevant to robot failures.

  • Traditional post-hoc annotation is laborious because long-horizon tasks contain numerous skill segments requiring timestamps.
  • Live narration records intended skills before teleoperation, then synchronizes Whisper transcriptions with robot trajectories.
  • The dataset includes successful executions and errors followed by recoveries, but training on all data could teach the model to make mistakes intentionally.
  • Foot pedals distinguish instruction segments from correction segments and enable filtering of data preceding corrections.
  • Policy rollouts identify robot-relevant skills for correction collection because operator and robot mistakes may differ.

B. Low-Level Policy

The system uses language-conditioned low-level control and captures temporal visual context to support real-time human intervention and autonomous instruction generation.

  • The low-level policy uses ACT with visual and language inputs for precise, complex robotic actions.
  • EfficientNet b3 encodes camera RGB images, while FiLM layers fuse visual and DistilBERT language features.
  • The high-level policy uses a frozen CLIP-initialized ViT followed by Transformer and MLP layers to generate language commands.
  • Historical observations use up to four images spaced one second apart to provide broader temporal context.
  • Training compares output language embeddings with dataset commands using cosine-similarity logits and cross-entropy loss, with targets offset toward upcoming instructions.
  • Users can interrupt the robot verbally and provide corrections, which are recorded with observations for later high-level-policy fine-tuning.

V. EXPERIMENTS

Experiments evaluate YAY Robot on real-robot, long-horizon bimanual manipulation tasks using autonomous and interactive comparisons, with task and sub-task success measurements.

  • Robot Setup and Tasks: The experiments use ALOHA, a low-cost open-source bimanual setup with a 14-dimensional action space and four RGB cameras.
  • Quantitative Evaluations: Language corrections produce a 20% success-rate improvement over the base policy across three long-horizon bimanual manipulation tasks.
  • Robot Setup and Tasks: Three tasks emphasize precision, coordination, and contact-rich manipulation involving deformable and transparent objects.
  • Methods and Evaluation Protocol: The interactive evaluation lets a human override high-level commands with natural-language instructions to test on-the-fly behavior modulation.
  • Methods and Evaluation Protocol: ACT Flat-BC provides a flat-policy comparison and is trained on the same data as the other baselines.
  • Iterative Improvement: Success rates improve significantly with each iteration of verbal feedback collection and fine-tuning, approaching oracle performance at each task stage.
  • Methods and Evaluation Protocol: Evaluations use 20 trials per task and measure both overall and sub-task success rates, with no physical interventions in the interactive condition.

C. Key Results

Language corrections improve task performance both during execution and after fine-tuning, while hierarchical and language-based high-level policies outperform key alternatives on long-horizon manipulation tasks.

  • Language Corrections: 25%-50% in Bag Packing, 30%-45% in Trail Mix Preparation, and 15%-25% in Plate Cleaning are the stage-wise improvements from real-time language corrections.The high-level policy rarely issued corrections before fine-tuning despite being trained on correction skills.
  • Iterative Fine-Tuning: 20%-45% in Bag Packing, 15%-20% in Trail Mix Preparation, and 15%-25% in Plate Cleaning are the stage-wise gains after fine-tuning on language corrections.The fine-tuned policy autonomously generates corrective instructions such as widening the gripper or targeting specific plate regions.
  • Policy Structure: Hierarchical policies generally achieve higher success rates than Flat-BC, with larger differences in later stages of each task.The later-stage gap indicates better handling of compounding errors.
  • High-Level Ablations: Replacing the learned high-level policy with a scripted sequence decreases task performance by up to 30%, because the scripted policy cannot react to mistakes or unforeseen scenarios.The comparison uses a predefined sequence despite selecting the most optimal skill sequence observed in rollouts.
  • High-Level Ablations: GPT-4V frequently makes spatial and object-state errors, while replacing language embeddings with one-hot encodings produces inferior performance and occasional unreasonable outputs.The language ablation involves a dataset containing 1,200 unique strings for the Bag task.
  • Data Quality: Training on larger mixed-quality datasets can deteriorate performance relative to smaller higher-quality datasets, with less stable or slightly higher training loss.The robot can also exhibit suboptimal behavior when observations appear in-distribution.

VI. CONCLUSION AND LIMITATIONS

The framework enables robots to respond to and improve from verbal corrections, achieving higher success than the base policy on three long-horizon bimanual tasks. Its effectiveness depends critically on a performant low-level policy that can respond to many distinct language commands.

  • Conclusion: Across three long-horizon bimanual manipulation tasks, verbal corrections alone produce a 20% higher success rate than the base policy.The evaluated tasks are bag packing, trail-mix preparation, and plate cleaning.
  • Limitations: The approach critically relies on a performant low-level policy that can successfully react to many distinct language commands.The authors state that further improvements must come from improving low-level policy performance and flexibility.

APPENDIX

The appendix describes the teleoperated base data, smaller language-only post-training data, task success criteria, command representations, and command-frequency analysis for bag packing.

  • Dataset Collection: The base dataset consists of human teleoperated trajectories annotated with language commands recorded through a USB microphone and foot paddle.The demonstrations use leader arms to control puppet arms in a bimanual setup.
  • Dataset Collection: The post-training dataset is collected from language-only human interventions after 2-3 iterations and contains 4%-11% as many skill segments as the base dataset.The post-training dataset is significantly smaller than the base dataset.
  • Data Visualization: Correction skills occur less frequently than task-oriented skills in bag-packing data but are more diverse, including wiggling, rotating, and shaking commands.The visualization covers the 200 most frequent commands.
  • Task Evaluation: Evaluation uses 20 trials per method across all three tasks, with task-specific and sub-task success criteria.Plate-cleaning success measures how many of 12 gummies are removed.
  • Task Evaluation: Bag-packing success requires sequentially grasping and inserting one, two, or three objects while keeping previously inserted objects inside the bag.The three-item criterion also requires releasing the bag without dropping any object.
  • Task Evaluation: Trail-mix success requires scooping and pouring one, two, or three ingredients, with the two-ingredient criterion allowing spills below one-quarter of the scoop.The three-ingredient criterion additionally requires releasing the bag.
  • Command Representation: The one-hot ablation indexes 1054 unique bag-packing commands, predicts a 1054-dimensional logit vector, and selects the highest-probability index at inference.Training uses cross-entropy loss.

3) GPT-4V:

GPT-4V is prompted with task, camera, and instruction information, but its initially reasonable predictions often give way to incorrect spatial and object-state reasoning across camera configurations.

  • Observed Behavior: GPT-4V makes reasonable predictions at the beginning but usually develops incorrect spatial reasoning after 2-3 predictions.It may infer that a sponge has been picked up when it remains on the table.
  • Camera Ablations: Using only a front-camera view does not resolve GPT-4V’s mistakes, including asking to lower a gripper already touching the table.The model changes its prediction after textual reminders but can still produce an impossible instruction.
  • Camera Ablations: More convenient camera angles likewise fail to correct errors in spatial relationships, such as treating an object outside the bag as already inside.The authors therefore evaluate GPT-4V with the learned policy’s multi-camera setup.
  • Prompting: GPT-4V receives task information, camera setup, and 50 useful instructions and corrections for the multi-camera bagging task.The setup uses top, front, left-wrist, and right-wrist cameras at regular intervals.
Loading 2403.12910v1…