Source-linked AI summary
"No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomy
Yuchen Cui, Siddharth Karamcheti, Raj Palleti, Nidhya Shivakumar, Percy Liang, Dorsa Sadigh
TL;DR
Existing language-following systems lack online adaptivity and often require large amounts of language-aligned demonstration data or restrictive assumptions. LILAC maps language into human-usable control spaces that online corrections refine within shared autonomy. In a user study, it achieved higher task success than two learned baselines and users preferred it on several measures.
Problem
Existing systems lack online language adaptation and either require large amounts of language-aligned demonstrations or make restrictive assumptions about the environment and perception.
Source: §1 · p. 2Method
Results
6 of 7 survey metrics significantly favored LILAC (p<0.05); it had the highest success rate across subtasks and significantly outperformed LILA and imitation learning on grasping, transfer, and full task completion (p<0.05).
Source: §6 · p. 7Takeaways & Limitations
The authors present LILAC as a potential answer to the need for language-driven interaction systems with adaptivity and sample efficiency.
Source: §4.2 · p. 4, §7 · p. 8Corrections are interpreted independently, so the current framework cannot correctly interpret anaphora or implicit coreference such as “the other way” or “undo that.”
Source: §7 · p. 8
Abstract
from arXiv · showhide
Systems for language-guided human-robot interaction must satisfy two key desiderata for broad adoption: adaptivity and learning efficiency. Unfortunately, existing instruction-following agents cannot adapt, lacking the ability to incorporate online natural language supervision, and even if they could, require hundreds of demonstrations to learn even simple policies. In this work, we address these problems by presenting Language-Informed Latent Actions with Corrections (LILAC), a framework for incorporating and adapting to natural language corrections - "to the right," or "no, towards the book" - online, during execution. We explore rich manipulation domains within a shared autonomy paradigm. Instead of discrete turn-taking between a human and robot, LILAC splits agency between the human and robot: language is an input to a learned model that produces a meaningful, low-dimensional control space that the human can use to guide the robot. Each real-time correction refines the human's control space, enabling precise, extended behaviors - with the added benefit of requiring only a handful of demonstrations to learn. We evaluate our approach via a user study where users work with a Franka Emika Panda manipulator to complete complex manipulation tasks. Compared to existing learned baselines covering both open-loop instruction following and single-turn shared autonomy, we show that our corrections-aware approach obtains higher task completion rates, and is subjectively preferred by users because of its reliability, precision, and ease of use.
1 INTRODUCTION
Prior instruction-following approaches are sample-inefficient and lack online adaptivity; LILAC lets users refine language-conditioned control spaces during execution and outperforms learned baselines in a user study.
1 INTRODUCTION
- Existing instruction-following systems either need large language-aligned demonstration sets or assume known dynamics, perfect object localization, and affordance prediction.These assumptions can be used to connect task and motion planners to robot actions.Source: §1 · p. 2
- Long-horizon manipulation involves precise states, and errors such as missed grasps or misalignment can be predictable and recoverable with a suitable adaptation mechanism.The motivating example requires aligning a book with a bookshelf with less than a few millimeters of clearance.Source: §1 · p. 2
- Prior work on dialogue, post-hoc corrections, and implicit feedback does not provide real-time natural-language corrections, motivating an online adaptive and sample-efficient approach.The cited strands include dialogue [47] [48], post-hoc corrections [7] [8] [12] [43], and implicit feedback [22].Source: §1 · p. 2
- LILAC lets users issue real-time language corrections that update the control space and reflect their intent during execution.The framework is built within shared autonomy, with each utterance shaping the control afforded to the user.Source: §1 · p. 2
- LILAC learns complex manipulation tasks from 10–20 demonstrations, compared with thousands to tens of thousands for fully autonomous imitation or reinforcement learning approaches.The authors attribute these gains to splitting agency so the human and robot share the burden of reasoning over actions.Source: §1 · p. 2
- LILAC achieved higher task success than language-informed latent actions and fully autonomous language-conditioned imitation learning, and users found it more reliable, precise, and easy to use.The evaluation was a within-subjects study with 12 users on a Franka Emika Panda arm.Source: §1 · p. 2
2 MOTIVATING EXAMPLE
The motivating example contrasts a static learned joystick mapping with LILAC’s ability to refine the control space mid-task when accumulated errors prevent precise insertion.
2 MOTIVATING EXAMPLE
- Learned latent actions map joystick axes to end-effector controls so each axis represents semantically meaningful movement through task space.The mapping is learned from a small number of task-specific demonstrations as an alternative to independently controlling groups of end-effector axes.Source: §2 · p. 3
- LILA’s single static mapping can compound navigation errors until the end-effector is misaligned with the shelf and clean insertion is impossible.The user can diagnose the problem during execution and provide a correction.Source: §2 · p. 3
- A “tilt down a little bit” correction switches the user into a new control space where left on the joystick precisely adjusts orientation, allowing task completion.This correction addresses end-effector alignment during the book-insertion task.Source: §2 · p. 3
3 RELATED WORK
Related work addresses language-conditioned policies, language corrections, and other forms of feedback, but differs from LILAC in when corrections are used, required assumptions, and data demands.
3 RELATED WORK
- Prior language-correction methods use hand-designed grounding and known dynamics, post-hoc cost optimization [43], or post-hoc waypoint edits with massive paired data [7] [8].LILAC instead operates online in real time without massive data, prior environment dynamics, or full state knowledge.Source: §3 · p. 3
- Language-conditioned imitation learning is expressive but can require hundreds or thousands of examples for simple tasks, while LILA [23] uses shared autonomy with orders of magnitude fewer demonstrations.LILA is the starting point for LILAC.Source: §3 · p. 3
- Other corrective-feedback research includes an online meta-algorithm that excludes language [42], while language corrections may transfer across tasks because phrases like “to the left” are often state-independent.The broader feedback modalities include physical corrections, teleoperation, critiques, and trajectory preferences.Source: §3 · p. 3
4 LILAC: FRAMING CORRECTIONS
LILAC extends language-informed latent actions with online natural-language corrections in shared autonomy. It maps low-dimensional user controls to robot actions using state and language, with gating to handle differing state-dependence across utterances.
4 LILAC: FRAMING CORRECTIONS
- LILAC extends LILA with online directional and referential language corrections that update the user’s control space during execution.The framework is designed to generalize across correction types and states.Source: §4 · p. 3
4.1 Problem Statement
- The policy maps robot and environment state, low-dimensional user controls, an initial instruction, and a correction stack to high-dimensional robot actions.Users can add corrections, remove the latest correction, or provide a control input at each timestep.Source: §4.1 · p. 4
4.2 Modeling: Inference & Learning
- The model fuses state and language representations to generate orthonormal action bases, which map low-dimensional user inputs to robot actions.Training uses a state-and-language conditional autoencoder to induce latent control manifolds from demonstrated actions.Source: §4.2 · pp. 4–5
4.3 Gating Instructions vs. Corrections
- A language-dependent gate controls state information: referent-free corrections can be state-independent, while referential instructions require environmental grounding.GPT-3 predicts the gate value; the authors motivate this design by noting that it can reduce the need for repeated examples of the same correction across different states.Source: §4.3 · p. 5
4.4 Reproducibility
- GPT-3 labels training utterances for context-dependence before joint model training; the authors also release the pipeline and report laptop-CPU training.The reported training setup uses 50 epochs and a batch size of 512.Source: §4.4 · p. 5
5 USER STUDY PRELIMINARIES
The study compares LILAC with imitation learning and LILA across complex tabletop manipulation tasks. It measures task progress and user preferences in a within-subjects study with 12 participants.
Tasks
- The five tasks ranged from throwing away paper and transferring a pen to opening a drawer, inserting a book, and watering a plant.The drawer task required fine-grained end-effector orientation control; the book had only a few millimeters of insertion clearance.Source: §5 · p. 6
Measures
- Task progress was measured at four stages: reaching the object, grasping it, transferring it to the target, and completing the full task.These subtasks provide a denser measure of progress than full-task success alone.Source: §5 · p. 6
Data collection
- The authors collected 50 full-task demonstrations and, for LILAC, correction demonstrations with directional and referential language in under two hours of robot interaction.The correction demonstrations sampled intermediate states during replay of the full-task demonstrations.Source: §5 · p. 6
Study design
- Twelve participants used all three methods on three of five tasks each, with method order shuffled and two trials per task.The study used a Franka Emika Panda robot; participants completed a survey after each strategy.Source: §5 · pp. 5, 7
Hypotheses
- The study tested whether LILAC improves subtask and full-task success and is preferred over both baselines on usability measures.Usability was surveyed on a seven-point Likert scale.Source: §5 · p. 7
Baselines
- LILAC and both baselines were trained on the same data; LILA omits the GPT-3 gating component, while imitation learning predicts actions for open-loop control.The imitation policy uses a one-second history window.Source: §5 · pp. 6–7
6 USER STUDY RESULTS
LILAC achieved the highest success rates across subtasks and was significantly better than both baselines on grasping, transfer, and full-task completion. Users also preferred it on most survey measures, while trajectory examples show how online corrections help recover from precision failures.
Study results
- LILAC had the highest success rate across all subtasks and significantly outperformed imitation learning and LILA on grasping, transfer, and full-task completion (p < 0.05).Users significantly preferred LILAC on six of seven survey metrics, including ease of use, intuitiveness, and willingness to use again (p < 0.05).Source: §6 · p. 7
Trajectory visualizations
- For open-drawer and water-plant, LILAC recovered from precision failures after users issued corrections, completing the grasp and pouring motions.Imitation learning failed early, while LILA reached objects but could not precisely aim and grasp them.Source: §6 · pp. 7–8
- On open-drawer, LILAC trajectories stayed closer to the training state distribution than LILA trajectories.The comparison shows end-effector position across users; orientation is omitted.Source: Figure 7 · p. 8, §6 · p. 8
7 DISCUSSION
The discussion identifies limits in how LILAC interprets and uses corrections, while concluding that users preferred it and it outperformed both learned baselines in the user study.
Limitations
- Because each correction is interpreted independently, the current framework cannot correctly handle anaphora or implicit coreference such as “undo that.”The authors identify context-sensitive corrections as future work.Source: §7 · p. 8
- Some participants overused corrections, leaving the high-level control space and completing much of the task through one-axis-at-a-time end-effector control.The authors say more natural and intuitive high-level control spaces will be crucial for scaling to more complex, temporally extended tasks.Source: §7 · p. 8
- Low-level corrections alone may be insufficient for tasks requiring simultaneous control of 3+ degrees of freedom or for long tasks where sequential corrections make slow progress.Making the underlying control spaces more natural and intuitive is identified as important for scaling.Source: §7 · p. 8
- “Rotate” and “tilt” can be ambiguous because users may mean their own reference frame or the robot’s.The authors identify this as another limitation of the current approach.Source: §7 · p. 8
Conclusion
- Users preferred LILAC, which was also more objectively performant than language-conditioned imitation learning and language-informed shared autonomy.The conclusion presents reliability, precision, and ease of use as the qualities users valued.Source: §7 · p. 8