Yell At Your Robot: Improving On-the-Fly from Language Corrections

Motivation

• Long-horizon manipulation requires sequencing multiple primitives, motivating hierarchical policies that command low-level behaviors through language.Packing items requires repeated grasping, maneuvering, and insertion steps.Source: §I · p. 1
• Scalable high-level supervision remains difficult because complete demonstrations are costly, while knowledge from LLMs, VLMs, language-only data, and human videos is indirectly grounded.These sources may leave policies brittle or insufficiently aware of robot-specific skill effectiveness.Source: §I · pp. 1–2

Core idea

• YAY Robot treats human language corrections as high-level feedback analogous to DAgger, allowing the policy to address errors from both low-level execution and high-level decisions.The approach supports robust recovery from failures during long-horizon tasks.Source: §I · p. 2
• YAY Robot integrates real-time adaptation with continuous improvement by temporarily applying user commands and later fine-tuning the high-level policy on those interventions.The system is designed to improve through user interaction rather than relying on one-time correction or post-hoc relabeling.Source: §I · p. 2

Policy design

• The language-conditioned low-level policy supports free-form commands ranging from fine-grained movements to high-level preferences within one neural network.Examples include moving a sponge slightly lower or changing gripper orientation.Source: §I · p. 2

Results

• 15% to 50% real-time improvement and 15% to 45% improvement after high-level training were reported across three bi-manual, multi-stage manipulation tasks.The tasks were bag packing, trail-mix preparation, and cleaning gummies from a plate.Source: §I · p. 2

Positioning

• Unlike shared-autonomy systems requiring perpetual corrections, YAY Robot is intended to operate autonomously while accepting flexible language interventions for dexterous manipulation.The paper contrasts its flexible input with systems limited to fixed spatial movements.Source: §II · p. 3

Preliminaries

• The manipulation problem is formulated as a partially observed Markov Decision Process, with the robot receiving observations rather than true states.The MDP is represented as M = (S, A, P, p0).Source: A. Preliminaries · p. 3
• The base dataset contains observation, action, and language-instruction sequences for training both high-level and low-level policies.Observations include RGB images and robotic proprioceptive information.Source: A. Preliminaries · p. 3
• The dataset is assumed to cover the visuomotor skills needed for tasks together with mistakes and recovery behaviors.This coverage supports learning both task execution and correction-relevant behavior.Source: A. Preliminaries · p. 3

Iterative improvement

• During post-training, the correction dataset records observations and user language but omits actions because the low-level policy is frozen.Only the high-level policy parameters are updated.Source: A. Preliminaries · p. 3

Low-level policy

• An expanded language-conditioned skill set gives the high-level policy flexibility to orchestrate recovery behaviors and lets the low-level policy execute adjustments.The design goes beyond the minimal skills required for nominal task completion.Source: B. Low-Level Language-Conditioned… · p. 3
• The low-level policy maps the current observation and language instruction to an action and is trained with a standard behavior-cloning objective.The loss compares predicted continuous actions with ground-truth actions using an l1 or l2 loss.Source: B. Low-Level Language-Conditioned… · p. 3

C. High-Level Policy for Autonomous Instruction Generation

• A hierarchical policy reuses primitive skills by generating language instructions that guide the low-level policy.The high-level policy is trained to compose behaviors for longer tasks.Source: C. High-Level Policy for Autonomous… · p. 3
• The architecture processes RGB images and joint positions, with a Vision Transformer predicting language embeddings and ACT generating target motor actions.The resulting pipeline translates commands such as “Pick up the bag” into joint movements.Source: Figure 4 · p. 4
• The high-level policy conditions instruction prediction on a brief history of observations because one observation can support multiple reasonable commands.Examples include “tilt the scoop down” and “go a bit higher.”Source: C. High-Level Policy for Autonomous… · p. 4
• πH(l_t|o_{t−k:t}) predicts a language embedding from temporal observation context and is trained end-to-end with cross-entropy loss and cosine-similarity logits.The temporal context is the observation sequence available up to time t.Source: C. High-Level Policy for Autonomous… · p. 4

D. Policy Integration and Adaptation to Human Feedback

• During deployment, a human can issue a corrective language command to address erroneous behavior or express a preference.The intervention directly enters the language-conditioned control pipeline.Source: D. Policy Integration and Adaptation… · p. 4
• The deployment policy selects the low-level action from the user command when an intervention occurs and otherwise uses the high-level policy’s instruction.This conditional routing temporarily overrides autonomous instruction generation.Source: D. Policy Integration and Adaptation… · p. 4
• Each intervention is stored as (o_{t−k:t}, l_user) in D_corr and later used to fine-tune πH.The high-level instruction is the maximum-probability output l_H = arg max πH(· | o_{t−k:t}).Source: D. Policy Integration and Adaptation… · p. 4

• Human language feedback is incorporated into the high-level policy in a process conceptually similar to Human-Gated DAgger.Source: E. Continuous Improvement from Human… · p. 4
• The policy is fine-tuned on the base dataset together with the correction dataset to expose it to mistakes requiring corrective instructions.Source: 1) Finetuning High-Level Policy · p. 4
• After each user-interaction and feedback-collection iteration, the policy is fine-tuned to reflect the new data.Source: 2) Iterative Improvement · p. 4
• π(n)H and π(n+1)H denote the high-level policy before and after the n-th fine-tuning iteration.Source: 2) Iterative Improvement · p. 4
• D(n)corr is the corrective-feedback dataset obtained at iteration n, and the process is intended to improve autonomous handling without frequent interventions.Source: 2) Iterative Improvement · p. 4

• Traditional post-hoc annotation is laborious because long-horizon tasks contain numerous skill segments requiring timestamps.Source: 1) Language Annotation · p. 5
• Live narration records intended skills before teleoperation, then synchronizes Whisper transcriptions with robot trajectories.Source: 1) Language Annotation · p. 5
• The dataset includes successful executions and errors followed by recoveries, but training on all data could teach the model to make mistakes intentionally.Source: 2) Filtering Mistakes · p. 5
• Foot pedals distinguish instruction segments from correction segments and enable filtering of data preceding corrections.Source: 2) Filtering Mistakes · p. 5
• Policy rollouts identify robot-relevant skills for correction collection because operator and robot mistakes may differ.Source: 3) Collection of Low-Level Correction… · p. 5

• The low-level policy uses ACT with visual and language inputs for precise, complex robotic actions.Source: B. Low-Level Policy · p. 5
• EfficientNet b3 encodes camera RGB images, while FiLM layers fuse visual and DistilBERT language features.Source: B. Low-Level Policy · p. 5
• The high-level policy uses a frozen CLIP-initialized ViT followed by Transformer and MLP layers to generate language commands.Source: C. High-Level Policy · p. 5
• Historical observations use up to four images spaced one second apart to provide broader temporal context.Source: C. High-Level Policy · p. 5
• Training compares output language embeddings with dataset commands using cosine-similarity logits and cross-entropy loss, with targets offset toward upcoming instructions.Source: C. High-Level Policy · p. 5
• Users can interrupt the robot verbally and provide corrections, which are recorded with observations for later high-level-policy fine-tuning.Source: D. Post-Training · p. 5

Robot Setup and Tasks

• The experiments use ALOHA, a low-cost open-source bimanual setup with a 14-dimensional action space and four RGB cameras.Source: A. Robot Setup and Tasks · pp. 6–7

Quantitative Evaluations

• Language corrections produce a 20% success-rate improvement over the base policy across three long-horizon bimanual manipulation tasks.Source: Figure 6 · p. 7

Robot Setup and Tasks

• Three tasks emphasize precision, coordination, and contact-rich manipulation involving deformable and transparent objects.Source: A. Robot Setup and Tasks · p. 7

Methods and Evaluation Protocol

• The interactive evaluation lets a human override high-level commands with natural-language instructions to test on-the-fly behavior modulation.Source: B. Methods and Evaluation Protocol · p. 7
• ACT Flat-BC provides a flat-policy comparison and is trained on the same data as the other baselines.Source: B. Methods and Evaluation Protocol · pp. 7–8

Iterative Improvement

• Success rates improve significantly with each iteration of verbal feedback collection and fine-tuning, approaching oracle performance at each task stage.Source: Figure 7 · p. 8

Methods and Evaluation Protocol

• Evaluations use 20 trials per task and measure both overall and sub-task success rates, with no physical interventions in the interactive condition.Source: B. Methods and Evaluation Protocol · p. 8

Language Corrections

• 25%-50% in Bag Packing, 30%-45% in Trail Mix Preparation, and 15%-25% in Plate Cleaning are the stage-wise improvements from real-time language corrections.The high-level policy rarely issued corrections before fine-tuning despite being trained on correction skills.Source: C. Key Results · p. 8

Iterative Fine-Tuning

• 20%-45% in Bag Packing, 15%-20% in Trail Mix Preparation, and 15%-25% in Plate Cleaning are the stage-wise gains after fine-tuning on language corrections.The fine-tuned policy autonomously generates corrective instructions such as widening the gripper or targeting specific plate regions.Source: C. Key Results · p. 8

Policy Structure

• Hierarchical policies generally achieve higher success rates than Flat-BC, with larger differences in later stages of each task.The later-stage gap indicates better handling of compounding errors.Source: C. Key Results · p. 8, Table I · p. 8

High-Level Ablations

• Replacing the learned high-level policy with a scripted sequence decreases task performance by up to 30%, because the scripted policy cannot react to mistakes or unforeseen scenarios.The comparison uses a predefined sequence despite selecting the most optimal skill sequence observed in rollouts.Source: Figure 8 · p. 8, C. Key Results · p. 9
• GPT-4V frequently makes spatial and object-state errors, while replacing language embeddings with one-hot encodings produces inferior performance and occasional unreasonable outputs.The language ablation involves a dataset containing 1,200 unique strings for the Bag task.Source: Figure 8 · p. 8, C. Key Results · p. 9

Data Quality

• Training on larger mixed-quality datasets can deteriorate performance relative to smaller higher-quality datasets, with less stable or slightly higher training loss.The robot can also exhibit suboptimal behavior when observations appear in-distribution.Source: C. Key Results · p. 9, Table II · p. 9

Conclusion

• Across three long-horizon bimanual manipulation tasks, verbal corrections alone produce a 20% higher success rate than the base policy.The evaluated tasks are bag packing, trail-mix preparation, and plate cleaning.Source: §VI · p. 10

Limitations

• The approach critically relies on a performant low-level policy that can successfully react to many distinct language commands.The authors state that further improvements must come from improving low-level policy performance and flexibility.Source: §VI · p. 10

Dataset Collection

• The base dataset consists of human teleoperated trajectories annotated with language commands recorded through a USB microphone and foot paddle.The demonstrations use leader arms to control puppet arms in a bimanual setup.Source: 1) Base Dataset · p. 14
• The post-training dataset is collected from language-only human interventions after 2-3 iterations and contains 4%-11% as many skill segments as the base dataset.The post-training dataset is significantly smaller than the base dataset.Source: 2) Post-Training Dataset · p. 14, Table IV · p. 14

Data Visualization

• Correction skills occur less frequently than task-oriented skills in bag-packing data but are more diverse, including wiggling, rotating, and shaking commands.The visualization covers the 200 most frequent commands.Source: 3) Data Visualization · p. 14, Figure 11 · p. 15

Task Evaluation

• Evaluation uses 20 trials per method across all three tasks, with task-specific and sub-task success criteria.Plate-cleaning success measures how many of 12 gummies are removed.Source: 1) Task Success Criterion · pp. 14–15
• Bag-packing success requires sequentially grasping and inserting one, two, or three objects while keeping previously inserted objects inside the bag.The three-item criterion also requires releasing the bag without dropping any object.Source: 1) Task Success Criterion · p. 15
• Trail-mix success requires scooping and pouring one, two, or three ingredients, with the two-ingredient criterion allowing spills below one-quarter of the scoop.The three-ingredient criterion additionally requires releasing the bag.Source: 1) Task Success Criterion · p. 15

Command Representation

• The one-hot ablation indexes 1054 unique bag-packing commands, predicts a 1054-dimensional logit vector, and selects the highest-probability index at inference.Training uses cross-entropy loss.Source: 3) GPT-4V · p. 15

Observed Behavior

• GPT-4V makes reasonable predictions at the beginning but usually develops incorrect spatial reasoning after 2-3 predictions.It may infer that a sponge has been picked up when it remains on the table.Source: 3) GPT-4V · p. 15, Figure 13 · p. 18, Figure 14 · p. 18

Camera Ablations

• Using only a front-camera view does not resolve GPT-4V’s mistakes, including asking to lower a gripper already touching the table.The model changes its prediction after textual reminders but can still produce an impossible instruction.Source: 3) GPT-4V · p. 16, Figure 15 · p. 19
• More convenient camera angles likewise fail to correct errors in spatial relationships, such as treating an object outside the bag as already inside.The authors therefore evaluate GPT-4V with the learned policy’s multi-camera setup.Source: 3) GPT-4V · p. 16, Figure 16 · p. 16

Prompting

• GPT-4V receives task information, camera setup, and 50 useful instructions and corrections for the multi-camera bagging task.The setup uses top, front, left-wrist, and right-wrist cameras at regular intervals.Source: Figure 12 · p. 17