PaperlayerView exact revision on arXiv

Source-linked AI summary

LILA: Language-Informed Latent Actions

Siddharth Karamcheti, Megha Srivastava, Percy Liang, Dorsa Sadigh

arXiv:2111.03205v1cs.ROcs.AIcs.CLcs.HCcs.LG

TL;DR

Shared-autonomy latent actions can fail to disambiguate behaviors that overlap in state, while language-conditioned robotics often leaves execution to an autonomous robot policy. LILA uses language to shape a low-dimensional controller; user studies report improved performance and sample efficiency against the stated baselines, along with qualitative user preference.

  • Problem

    State-conditioned latent actions can fail to distinguish different intended behaviors in overlapping states, while full-autonomy language approaches have reported concerns including sample efficiency and robustness.

    Source: §1 · p. 1

  • Method

    LILA trains language- and state-conditioned latent-action models and lets users combine natural-language instructions with low-dimensional control.

    Source: §1 · p. 2

  • Results

    LILA was more reliable, performant, and sample efficient than imitation learning and end-effector control baselines, and users qualitatively preferred it; it trained on half as many task demonstrations as imitation learning.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    In the studied assistive teleoperation setting, language-informed control spaces supported user agency while providing intuitive low-dimensional control.

    Source: §6 · p. 8

    LILA uses language to disambiguate tasks but has no mechanism for generalizing to completely unseen tasks or language specifications.

    Source: §6 · p. 8

Abstract

from arXiv · show

We introduce Language-Informed Latent Actions (LILA), a framework for learning natural language interfaces in the context of human-robot collaboration. LILA falls under the shared autonomy paradigm: in addition to providing discrete language inputs, humans are given a low-dimensional controller $-$ e.g., a 2 degree-of-freedom (DoF) joystick that can move left/right and up/down $-$ for operating the robot. LILA learns to use language to modulate this controller, providing users with a language-informed control space: given an instruction like "place the cereal bowl on the tray," LILA may learn a 2-DoF space where one dimension controls the distance from the robot's end-effector to the bowl, and the other dimension controls the robot's end-effector pose relative to the grasp point on the bowl. We evaluate LILA with real-world user studies, where users can provide a language instruction while operating a 7-DoF Franka Emika Panda Arm to complete a series of complex manipulation tasks. We show that LILA models are not only more sample efficient and performant than imitation learning and end-effector control baselines, but that they are also qualitatively preferred by users.

1 Introduction

LILA adds language to shared-autonomy latent actions so a low-dimensional controller can reflect the user’s objective. The paper reports that users preferred LILA and that it outperformed the specified autonomy and end-effector-control baselines.

1 Introduction

  • State-conditioned latent-action controllers can fail when different tasks overlap in state space because state alone does not disambiguate the intended behavior.For example, “grab the milk” and “shift the milk to the side” can call for different behavior in overlapping states.Source: §1 · p. 1
  • LILA uses language and state to condition low-dimensional control spaces, allowing utterances to shape joystick dimensions around task-relevant behavior.The cereal-bowl example links joystick dimensions to distance from the bowl and end-effector angle relative to it.Source: §1 · p. 2
  • LILA models were more reliable, performant, and sample efficient than imitation learning and end-effector control baselines, and users qualitatively preferred them.The study evaluated 10 users completing diverse manipulation tasks.Source: §1 · p. 2

2 Related Work

Prior work uses shared autonomy and learned latent actions to make high-dimensional robot control more manageable, while language-robotics approaches commonly place execution under full robot autonomy. LILA uses language to disambiguate the user-controlled latent action space.

Shared Autonomy & Assistive Teleoperation

  • Learned latent actions provide low-dimensional, semantically meaningful controls for high-DoF manipulators, but struggle to distinguish behaviors with overlapping states.Direct end-effector control can require frequent mode switching, while latent action spaces can support diverse behaviors from a low-DoF interface.Source: §2 · p. 3

Language-Informed Robotics

  • Language-robotics methods map instructions to planning primitives, imitation policies, rewards, or adaptive behavior, but typically leave execution to a robot policy.The cited approaches are characterized as full autonomy after the user provides an instruction.Source: §2 · p. 3
  • Full-autonomy learning can require hundreds to thousands of demonstrations or millions of reinforcement-learning episodes, while precise recovery remains difficult.LILA instead places behaviors that are harder for robots but easier for people onto the user within shared autonomy.Source: §2 · p. 3

3 Formalizing Language for Assistive Teleoperation

The formalization treats assistive teleoperation as a language-augmented shared-autonomy task in which users provide low-dimensional control. Language conditions the decoder so it can distinguish objectives that state-conditioned latent actions alone may not recover.

3 Formalizing Language for Assistive Teleoperation

  • The formulation defines the task as a fully observable, language-augmented MDP, with the user’s utterance implicitly specifying a reward over states and actions.The tuple is (S, U, A, T, R, γ), and the human and robot collaboratively maximize the reward.Source: §3 · pp. 3–4
  • A latent-action decoder maps the user’s low-dimensional input and current state to a high-dimensional robot action, but state alone may not distinguish intended behaviors.Different actions such as picking up, shifting, or pouring a jug can be appropriate when the end-effector is near it.Source: §3 · p. 4
  • LILA conditions the decoder on state, utterance, and user input so language can help disambiguate the resulting control space.The decoder is expressed as Dec(s, u, z) mapping to a high-DoF action.Source: §3 · p. 4

4 Language-Informed Latent Actions (LILA)

LILA learns a language-conditional autoencoder that reconstructs demonstrated actions from state, utterance, and latent action. Its language integration uses pretrained embeddings and FiLM, while inference retrieval maps user utterances to training exemplars.

4 Language-Informed Latent Actions (LILA)

  • Demonstration trajectories are split into (utterance, state, action) triples to train a conditional autoencoder that reconstructs the demonstrated action.The encoder maps the triple to a latent action, and the decoder predicts the original action using state, utterance, and latent action.Source: §4 · p. 4

4.1 Integrating Language within the Latent Actions Architecture

  • The decoder concatenates state and latent action, processes them through ReLU layers, then fuses language into an intermediate representation using FiLM.A distilled RoBERTa-Base model produces mean-pooled utterance embeddings that set FiLM’s affine parameters.Source: §4.1 · p. 5
  • At inference, each user utterance is embedded and matched to its nearest training exemplar, which replaces the new wording as LILA’s input.This retrieval approach is intended to reduce mismatch between training and test language inputs.Source: §4.1 · p. 5
  • Because novel or unachievable utterances map to language seen during training, this retrieval approach limits LILA’s ability to perform novel tasks.Users can stop execution if the robot does not align with their objective.Source: §4.1 · p. 5

5 User Study

The user study compares LILA with end-effector control and imitation learning on real-world manipulation tasks. LILA significantly outperforms both across subtasks, provides smoother control, and receives stronger user ratings, although full-task completion remains difficult.

Environment

  • The study evaluates five manipulation tasks with a 7-DoF Franka Emika Panda Arm and scores partial success across reaching, grasping, transport, and task completion.The tasks include placing objects and pouring marbles between cups or bowls.Source: §5 · pp. 5–6

Demonstration Collection

  • LILA and imitation learning use language-demonstration training data; imitation learning was increased to 30 demonstrations per task after failing at rudimentary reaching with 15.Crowdworkers supplied task instructions from demonstration videos, and the 15 least noisy annotators' utterances formed the ground-truth language data.Source: §5 · p. 6

Participants & Procedure

  • In the within-subjects study, each of 10 students tried two randomly chosen tasks with all three methods, while researchers recorded success, time, and controller logs and collected surveys.For language-based methods, a proctor entered each participant's utterance for nearest-neighbor retrieval from the training utterances.Source: §5 · pp. 6–7

Quantitative Results

  • LILA significantly outperforms both methods across subtasks and provides smoother 2-DoF input and 6-DoF end-effector control, while full-task completion remains a fine-control challenge.The results also identify imitation learning's poor performance as likely related to sample inefficiency in the low-data regime.Source: Figure 3 · p. 6, §5 · p. 7

Qualitative Results

  • Users rate LILA above imitation learning and end-effector control across most qualitative metrics, including helpfulness and willingness to use the method again.The survey used a 7-point Likert scale.Source: §5 · p. 7, Figure 4 · p. 7
  • For three of five tasks, example trajectories show LILA smoothly approaching targets, end-effector users diverging from intended paths, and imitation learning often stopping shortly after reaching objects.Across all tasks, LILA produces smoother end-effector trajectories than end-effector control.Source: §5 · p. 7, Figure 5 · p. 8

6 Discussion

The discussion presents LILA as a language-informed latent-action framework for assistive teleoperation, reporting efficiency and performance advantages alongside user preference. It also identifies limits in task and language generalization and a desire for language-based corrections during execution.

Summary

  • Compared with imitation learning, LILA trains on half the task demonstrations while achieving significantly higher success rates; compared with end-effector control, it yields high success and intuitive low-dimensional spaces.The authors report that users prefer LILA to alternative methods across the board.Source: §6 · p. 8

Limitations and Future Work

  • LILA currently uses language to disambiguate tasks but does not generalize to completely unseen tasks or language specifications.The authors point to dynamically adapting to new utterances and tasks as future work.Source: §6 · p. 8
  • Users wanted to modulate behavior with language during execution, including correcting direction or specifying how to grasp an object.The authors frame this as a desire for multi-resolution language control.Source: §6 · p. 8

Shared Autonomy and LILA

  • Users retain responsibility for robot motion in shared autonomy and can stop, reset, issue a new instruction, or switch to a more complex control mode when behavior is unsafe or misaligned.The discussion describes the language-informed latent control spaces as interpretable and quickly graspable by humans.Source: §6 · p. 8

A COVID-19 Impact Statement

COVID-19 restrictions constrained reliable lab access and the ability to run a larger, more diverse user study. The authors nevertheless considered the 10-student results compelling and hoped to conduct more studies when restrictions eased.

COVID-19 Impact

  • COVID-19 safety procedures and university access restrictions limited reliable lab access and recruitment for a larger user study.Most participants had pre-existing Engineering Building access; the restrictions limited a study with more than 10 participants from a more diverse population.Source: A COVID-19 Impact Statement · p. 13
  • The authors believed the 10 students' statistics and significant results remained compelling, while expressing interest in additional studies after local restrictions relaxed.The participants were five women and five men, with a reported age range of 23.2 ± 1.87.Source: A COVID-19 Impact Statement · p. 13

B Related Work Discussion

The discussion contrasts LILA with language-to-planning and end-to-end learning approaches, then positions it between them. It also identifies language grounding and other modalities as potential routes to broader generalization.

B Related Work Discussion

  • The paper says its main-body discussion of language-informed robotics is cursory and omits much of the existing work and its data, implementation, and feature-engineering considerations.Source: B Related Work Discussion · p. 13
  • Logical-form approaches can map language with tens of examples, but some rely on planners and full world dynamics, while policy learning can require hundreds to thousands of demonstrations.The navigation policy example used DAgger for 25 iterations and an order of magnitude more data than initially provided.Source: B Related Work Discussion · pp. 13–14
  • End-to-end approaches include simulated reinforcement learning and real-world imitation learning, with the latter still requiring thousands of demonstrations despite data-augmentation and synthetic-data techniques.Source: B Related Work Discussion · p. 14
  • LILA aims to retain sample efficiency without hand-engineered features or strong known-dynamics assumptions, while its experiments mostly use language for disambiguation.The authors describe it as a first step toward richer language and harder generalization.Source: B Related Work Discussion · p. 14
  • Language-and-gesture grounding and multimodal cues are proposed as possible routes to scaling LILA to multiple objects, referring expressions, compositional instructions, and new contexts.The cited examples include learning object localizations from language and gestures and using gaze, gesture, or intonation.Source: B Related Work Discussion · p. 14

C Model Architectures & Training

LILA inserts language into latent-action networks through FiLM and trains with consistency and noise augmentations. The section also contrasts the LILA and imitation-learning architectures and reports qualitative imitation-learning behavior on LILA-style demonstrations.

C Model Architectures & Training

  • FiLM uses language embeddings to modulate intermediate encoder and decoder representations through a learned affine transformation, without increasing the core network’s intrinsic parameter count.The modulation module is a separate two-layer GELU MLP; language concatenation experiments did not yield decreasing training loss.Source: C Model Architectures & Training · p. 15
  • Imitation learning uses a single state-conditioned MLP with one FiLM module, whereas LILA uses an encoder-decoder structure with separate FiLM modules.The imitation-learning network is matched to the combined LILA encoder and decoder in parameter count and number of layers.Source: C Model Architectures & Training · p. 15
  • A sliding-window procedure trains the decoder to predict the same actions from nearby states given the same latent action and language.This is intended to enforce consistency of behavior across nearby states.Source: C Model Architectures & Training · p. 15
  • Noise augmentation perturbs each initial state and recomputes its corresponding action from the next state; the reported noise standard deviation is 0.01.The authors describe it as a simulated version of the DART paradigm.Source: C Model Architectures & Training · p. 15
  • In two training-instruction examples, imitation learning trained on LILA-style sweeping demonstrations follows the ideal task trajectory less closely than the pure-data model, and neither completes the task.Source: Figure 7 · p. 16
  • The authors report tripling noise augmentation for imitation learning to obtain even slightly meaningful behavior.Source: C Model Architectures & Training · p. 16

D Demonstration Collection

The paper uses smooth, forward-only demonstrations for imitation learning and sweeping, reversible demonstrations for latent-action models. It addresses whether this difference gives LILA more training data than its baseline.

D Demonstration Collection

  • Imitation-learning demonstrations are smooth, forward-only task executions, while latent-action demonstrations add back-and-forth sweeping motions intended to support reversible control.For a fruit-basket task, the sweeping version includes retreating and reaching again before grasping, and lowering the object before carrying it to the basket.Source: D Demonstration Collection · p. 16
  • The authors identify a fairness concern: sweeping demonstrations may provide LILA with more state-action pairs per demonstration than imitation learning.Source: D Demonstration Collection · p. 16
  • After collecting 30 demonstrations per task for imitation learning versus 15 for LILA, imitation learning saw 40% more state-action pairs, even before accounting for its tripled data augmentation.Source: D Demonstration Collection · p. 17

E Additional Experiments: Imitation Learning Baseline

In equal-footing tests on three tasks, LILA succeeds with 10 demonstrations while imitation learning fails; more demonstrations and augmentation improve imitation learning but do not solve the tasks. The section also examines real-robot timing as a source of cascading imitation-learning errors.

E Additional Experiments: Imitation Learning Baseline

  • With 10 demonstrations per task on three tasks, LILA fully succeeds while imitation learning completely fails, though imitation learning begins to show semantically meaningful but unsuccessful movements.The tasks are Pick Cereal, Pick Fruit Basket, and Pour Cup.Source: E Additional Experiments: Imitation… · pp. 17–18
  • Three-times noise augmentation slightly improves imitation learning over the reference, while five-times augmentation adds no further improvement.Source: E Additional Experiments: Imitation… · p. 18
  • Imitation learning improves from 10 to 30 demonstrations per task but still cannot solve the tasks; at 20 demonstrations it grasps the cup but cannot finish.The authors estimate that 50–100 demonstrations per task might be needed for imitation learning to solve them, nearly two orders of magnitude more than LILA needed.Source: E Additional Experiments: Imitation… · p. 18
  • For the cross-disambiguation task, a standard latent-action model with a 1-DoF controller achieved 0% task success without an additional conditioning input such as language.The task requires at least a 2-DoF controller to disambiguate the four directions.Source: Figure 10 · p. 19
  • On the real robot, inconsistent state-action sampling can make the policy execute a bad action, with timing slips cascading to arbitrarily bad final states.The fixed-interval assumption used in the simplified trajectory analysis may not hold on the 7-DoF Franka Emika Panda; the authors note that the nearest-neighbor argument is a simplification.Source: E Additional Experiments: Imitation… · pp. 18–19, Figure 9 · p. 18
  • Imitation learning on LILA’s sweeping demonstrations performed worse than on pure demonstrations, supporting the authors’ claim that the LILA comparison is fair.They suggest that opposing actions in the same state confuse imitation learning.Source: E Additional Experiments: Imitation… · p. 19

F Additional Experiments: No-Language Ablation

The no-language ablations show that task disambiguation depends on language or additional controller dimensions in the tested cross setting. The crowdsourced language annotations were filtered before training, with 15 utterances per task used as the training data.

F Additional Experiments: No-Language Ablation

  • Without language, the latent-action model made no progress in the expert trials and produced random behavior when control approached an object.The authors attribute this to the model's inability to identify which task to perform.Source: F Additional Experiments: No-Language… · p. 20
  • 0% task success rate was achieved by the 1-DoF without-language controller, while both the 1-DoF language controller and 2-DoF without-language controller enabled task disambiguation.The comparison used 100 demonstrations across four tasks for the cross-disambiguation task.Source: F Additional Experiments: No-Language… · p. 20
  • The authors argue no-language baselines are uninformative for the multi-task user study because controller DoF limits task completion, making above 0% completion impossible.Source: F Additional Experiments: No-Language… · p. 20

G Crowdsourcing & User Study

  • Crowdworkers supplied short utterances for seven videos and were not told the possible tasks or object names.They were paid $1.20 for their responses.Source: G Crowdsourcing & User Study · p. 20
  • The annotation procedure filtered spam, noise, and extremely out-of-domain utterances before training, using utterances from 15 crowdworkers for each of five tasks.Examples of filtered responses included both overly detailed directions and poorly formed text.Source: G Crowdsourcing & User Study · p. 21
  • The training data comprised 15 utterances for each of the five tasks.The repository also included the utterances filtered out by the procedure.Source: G Crowdsourcing & User Study · p. 21
Loading 2111.03205v1…