PaperlayerView exact revision on arXiv

Source-linked AI summary

RT-H: Action Hierarchies Using Language

Suneel Belkhale, Tianli Ding, Ted Xiao, Pierre Sermanet, Quon Vuong, Jonathan Tompson, Yevgen Chebotar, Debidatta Dwibedi, Dorsa Sadigh

arXiv:2403.01823v2cs.ROcs.AI

TL;DR

RT-H addresses the difficulty of sharing data across semantically diverse tasks by inserting fine-grained language motions between high-level tasks and low-level robot actions. It outperforms RT-2 across diverse tasks, language-motion corrections reach near-perfect success rates, and language-motion interventions outperform teleoperation-based methods, while absolute success rates still leave room for improvement.

  • Problem

    As tasks become more semantically diverse, high-level task language shares less structure between tasks, making task-to-action learning require more demonstration data.

  • Method

    RT-H predicts fine-grained language motions from high-level tasks, then conditions action prediction on the inferred motion, with both stages using visual input and task language.

  • Results

    RT-H outperforms RT-2 by 15% across a wide range of tasks, while language-motion interventions outperform IWR by 50% and improve generalization to scene and object variations.

  • Takeaways & Limitations

    Language motions provide a flexible intermediate interface for correcting robot behavior and learning from human interventions across diverse multi-task datasets.

  • Takeaways & Limitations

    Absolute success rates still leave room for improvement, even after training on language-motion corrections.

Abstract

from arXiv · show

Language provides a way to break down complex concepts into digestible pieces. Recent works in robot imitation learning use language-conditioned policies that predict actions given visual observations and the high-level task specified in language. These methods leverage the structure of natural language to share data between semantically similar tasks (e.g., "pick coke can" and "pick an apple") in multi-task datasets. However, as tasks become more semantically diverse (e.g., "pick coke can" and "pour cup"), sharing data between tasks becomes harder, so learning to map high-level tasks to actions requires much more demonstration data. To bridge tasks and actions, our insight is to teach the robot the language of actions, describing low-level motions with more fine-grained phrases like "move arm forward". Predicting these language motions as an intermediate step between tasks and actions forces the policy to learn the shared structure of low-level motions across seemingly disparate tasks. Furthermore, a policy that is conditioned on language motions can easily be corrected during execution through human-specified language motions. This enables a new paradigm for flexible policies that can learn from human intervention in language. Our method RT-H builds an action hierarchy using language motions: it first learns to predict language motions, and conditioned on this and the high-level task, it predicts actions, using visual context at all stages. We show that RT-H leverages this language-action hierarchy to learn policies that are more robust and flexible by effectively tapping into multi-task datasets. We show that these policies not only allow for responding to language interventions, but can also learn from such interventions and outperform methods that learn from teleoperated interventions. Our website and videos are found at https://rt-hierarchy.github.io.

I. INTRODUCTION

RT-H addresses the difficulty of sharing data across semantically diverse tasks by inserting fine-grained language motions between high-level tasks and low-level actions. It uses this hierarchy to improve multi-task learning, support language-based corrections, and learn from those corrections.

  • Motivation: High-level language shares structure between similar tasks, but this sharing becomes harder as task descriptions become more diverse.Examples include “pick coke can” versus “pick an apple” and “pick coke can” versus “pour a cup”.
  • Action hierarchy: Language motions such as “move arm forward” provide an intermediate representation between high-level tasks and low-level actions.For example, “pick coke can” can be decomposed into moving the arm forward, grasping the can, and moving the arm up.
  • Method: RT-H predicts a language motion from visual observations and the task, then predicts the action using the observation, task, and inferred motion.The model uses a single VLM based on RT-2 and extracts more than 2500 language motions automatically from robot proprioception.
  • Results: RT-H outperforms RT-2 by 15% across a wide range of tasks and reports near-perfect success after language-motion correction.The paper also reports a 50% improvement over IWR when fine-tuning with language-motion interventions and better generalization to scene and object variations.
  • Correction: Language motions enable contextual human corrections during execution and allow the model to be trained on those corrections.Users can intervene on the language motion, while the action query converts the correction into a contextual action.

A. Formalizing Action Hierarchies

The action hierarchy formalizes an intermediate natural-language action between task descriptions and low-level robot actions. RT-H learns both hierarchy levels with one VLM and obtains language motions automatically, while the abstraction level trades prediction difficulty against action guidance.

  • A. Formalizing Action Hierarchies: An action hierarchy contains an intermediate natural-language action z that is more fine-grained than the task and more coarse-grained than the low-level action.The formalization represents each demonstration as observations, intermediate actions, and low-level actions.
  • A. Formalizing Action Hierarchies: The high-level policy πh maps observations and task descriptions to intermediate actions, while πl maps observations, tasks, and intermediate actions to low-level actions.The composed policy is π(a, z|o, g) = πh(z|o, g)πl(a|o, g, z).
  • A. Formalizing Action Hierarchies: RT-H models intermediate actions with language motions such as “move arm forward” and “rotate arm right”, and the hierarchy can extend to multiple levels.The text orders multiple intermediate representations by their degree of fine-grainedness.
  • B. RT-H: Model and Training Details: RT-H learns both hierarchy policies with a single VLM co-trained on internet-scale data and based on the PaLI-X 55B architecture used by RT-2.Images are encoded into tokens, and actions are represented through discretized action dimensions and tokenized action outputs.
  • B. RT-H: Model and Training Details: The model replaces RT-2’s action query with language-motion and action queries sampled at equal rates while freezing the ViT encoder during co-training.Using one model lets both queries benefit from the PaLI-X training mixture.
  • C. Extracting Language Motions: The automated extraction procedure combines motion components, producing over 2500 language motions from a simple set of known motions.A trajectory that moves the arm forward while closing the gripper becomes “move arm forward and close gripper.”
  • C. Extracting Language Motions: The extraction procedure is fixed across experiments and datasets, making its design a one-time developer cost without manual annotation.The language motions are derived directly from actions and therefore provide predictive information for the action query.
  • C. Extracting Language Motions: Finer-grained language motions provide more action guidance but are harder to predict, creating a trade-off in choosing the abstraction level.The paper presents its chosen motion vocabulary as a balance between language-motion accuracy, action-query accuracy, and labeling cost.

IV. RT-H: INFERENCE & CORRECTION

RT-H performs sequential language-motion and action inference, supports direct language corrections, and is evaluated against flat, clustered, one-hot, and intervention-based alternatives on eight challenging tasks. Its hierarchy improves multi-task performance and supports contextual correction and generalization evaluations.

  • Inference: At test time, RT-H infers a language motion before computing the action, doubling sequential query time and creating lag for large models.The paper discusses asynchronous and other inference modes to address this challenge.
  • Correction: Users can directly enter a language-motion correction by keyboard or microphone, which the action query converts into a scene- and task-contextual action.The current predicted motion can also be displayed to help users choose corrections.
  • Experimental questions: The evaluation asks whether language hierarchies improve performance, provide contextual motions, outperform teleoperated corrections, and improve robustness to out-of-distribution settings.These questions are labeled Q1 through Q4 in the experimental design.
  • Dataset: The Diverse+Kitchen dataset contains 100K demonstrations, combining Kitchen’s 70K demonstrations with Diverse’s 30K demonstrations across more than 24 semantic task categories.Evaluation uses eight representative hard tasks requiring complex motion sequences and high precision.
  • Results: RT-H outperforms RT-2 by 15% on average and achieves higher performance on 6/8 Diverse+Kitchen tasks.Replacing language motions with one-hot labels significantly reduces performance, while K-means clusters cause a minor drop.
  • Results: Training on language-motion corrections improves policy performance, and RT-H generalizes better than RT-2 to scene and object variations.The experiments separately examine correction learning and robustness to novel settings.

A. RT-H on Diverse Multi-Task Datasets

RT-H improves action prediction and on-robot performance over flat RT-2 on diverse multi-task datasets, while remaining robust to querying variants.

  • On-Robot Performance: RT-H surpasses RT-2 by 15% on average across eight Diverse+Kitchen tasks without additional human annotation.RT-H makes more progress toward success on 7/8 tasks, while RT-2 has nonzero performance on 4/8 and RT-H on 6/8.
  • Ablations: RT-H-Joint performs comparably to RT-H, indicating robustness to the exact language-motion querying mechanism.
  • Offline Performance: RT-H has roughly 20% lower action-prediction MSE than RT-2, while RT-H-Joint is 5–10% lower than RT-2.
  • Offline Performance: RT-H and RT-H-Joint achieve lower end-to-end validation MSE than RT-2 on both Kitchen and Diverse+Kitchen datasets.Table I compares best-checkpoint action-prediction MSE across models and datasets.

B. Contextual & Flexible Language Motions

RT-H’s language motions adapt their execution to scene and task context while allowing the robot to follow diverse, including out-of-distribution, motion instructions.

  • Contextuality: The same language motion produces task- and scene-dependent variations in speed, secondary axes, rotation, and gripper position.Examples include “move arm forward” adding rotation or object-directed motion, and “rotate arm right” changing vertical behavior by task.
  • Contextuality: RT-H preserves a high-level motion instruction while adjusting actions to the relevant object and task context.
  • Flexibility: RT-H correctly executes two different language motions for the same napkin-pulling task.The demonstrated corrections are “right and down” and “up and backward.”
  • Flexibility: RT-H often follows completely out-of-distribution language motions, including general motions not seen for the task during training.For example, it correctly responds to moving the arm away from the jar during “close pistachio jar.”
  • Implications: This flexibility supports language-motion corrections when tasks or scenes are out of distribution and require novel motion sequences.

C. Training on Online Corrections

Training on language-motion corrections substantially improves RT-H and outperforms teleoperation-based correction methods, though hierarchy and correction data introduce distinct failure modes.

  • Correction Results: RT-2-IWR’s average success rate degrades from 25% to 13% after teleoperation corrections.The authors attribute this likely to limited data and action distributions that differ from the base policy’s training data.
  • Correction Results: RT-H-Intervene substantially outperforms RT-2-IWR despite using the same amount of correction data.It improves by 60–70% on the harder precise tasks of opening and closing the pistachio jar.
  • Correction Results: RT-H-InterveneAction improves over RT-H by 9% on average, but action fine-tuning can cause policy degeneration.Model-generated actions included in interventions may be suboptimal and bias the learned action distribution.
  • Correction Results: Language-motion corrections raise RT-H’s average success rate from 40% to 63% with 30 correction episodes per task.
  • Failure Modes: Systemic language-motion prediction errors can produce oscillatory or incorrect behavior, although the authors report these issues are debuggable through relabeling or intervention.
  • Failure Modes: Correction collection is limited by the underlying action predictor, especially as tasks diverge from the training distribution.Incorrect responses can make language correction slower than teleoperation for the operator.

D. Generalization

RT-H is more robust than RT-2 to new scenes and novel objects, and qualitative results show that a few language corrections can support unseen tasks.

  • New Scenes: RT-H and RT-H-Joint are more robust than competing models to new buildings, lighting, backgrounds, and floors.The largest performance deltas occur on “place upright” and “open / close drawer.”
  • New Objects: RT-H achieves 65% success on “pick” and “move” with objects unseen during training, compared with 55% for RT-2.The evaluation covers 50 scenarios involving objects such as pears, coconut water, and Oreos.
  • New Objects: RT-H progresses farther through task stages than RT-2 on average under novel-object evaluation.
  • Evaluation Setup: The novel-object evaluation compares RT-2 and RT-H trained on Kitchen data across the “pick” and “move” tasks.
  • New Tasks: For unseen tasks, qualitative demonstrations show RT-H succeeding after a few well-timed language-motion corrections.The demonstrations use motions shown across varied contexts.

VI. CONCLUSION

RT-H uses language motions as an intermediate layer between high-level tasks and low-level actions, improving multi-task learning, correction, and robustness to scene and object variations. The paper reports stronger performance than RT-2 and identifies remaining limitations in success rates and hierarchy design.

  • Conclusion: RT-H maps high-level tasks to language motions before predicting low-level actions, with both stages conditioned on visual input and the task.This hierarchy is intended to share task structure across disparate tasks and support language-motion corrections.
  • Conclusion: RT-H supports language-motion corrections that alter behavior within the context of the scene and task, and these corrections outperform teleoperation-based corrections.The conclusion presents correction as a distinct use of the intermediate language layer.
  • Conclusion: The authors report that RT-H outperforms RT-2 and action-hierarchy ablations on diverse multi-task data and is more robust to scene and object variations.These findings are presented as evidence for the promise of language-based action hierarchies.
  • Conclusion: RT-H and its action-hierarchy variant outperform RT-2 on the same tasks in a new building with novel backgrounds, lighting, and flooring.RT-2 particularly struggles with placing upright and opening or closing drawers in these new scenes.
  • Limitations & Future Work: Absolute success rates still leave room for improvement, even after training on corrections.The authors propose scaling both offline datasets and the correction pipeline as future work.
  • Limitations & Future Work: Future work must determine the best abstraction level for intermediate layers and may extend the single intermediate layer to multiple steps of action reasoning.The proposed hierarchy could include task-level reasoning before language motions and actions.

APPENDIX

The appendix describes RT-H’s transformer implementation, query structure, training setup, and action-hierarchy ablations. These components distinguish language-motion representations from clustered or one-hot alternatives.

  • Implementation: RT-H and RT-2 use a Pali-X 55B multimodal encoder-decoder transformer with images encoded by a fixed 22B ViT during robot-data co-training.The encoded images and prompt pass through the encoder, while each query’s output is autoregressively decoded.
  • Method Implementations: RT-H first predicts a language motion from the task, then uses the predicted motion, task, and observation to predict a tokenized action.The two queries separately specify skill prediction and action prediction.
  • Method Implementations: RT-H co-trains language-motion and action queries alongside the original pre-training mixture after pretraining on the same large vision-language dataset as RT-2.The implementation uses a learning rate of 1e-3, batch size 1024, warmup, and square-root-normalized decay.
  • Ablations: RT-H-Cluster replaces language motions with 256 K-means action-cluster centers converted to integer tokens.This ablation tests language motions against embeddings tuned to dataset-specific actions.

B. Corrections

RT-H shifts human correction from low-level actions to language motions, allowing typed or spoken corrections to directly condition action prediction. The correction-training recipe separately allocates data across pretraining, demonstrations, and corrections.

  • Corrections: Human corrections can be typed or spoken as language motions and passed directly into the action query instead of the inferred motion.This shifts the correction burden up one level in the action hierarchy, from actions to language motions.
  • Corrections: 50% of training samples use pre-training queries, while demonstration language-motion and action queries each use 23%.Correction-data language-motion queries account for the remaining 4%.
  • Corrections: The correction dataset is upweighted by 50:1 because the demonstration-to-correction dataset size ratio is roughly 300:1.This weighting compensates for the much smaller correction dataset.
  • Corrections: RT-2-IWR uses the same recipe for teleoperated corrections, replacing language-motion training queries with action queries.This creates the comparison between language-motion and action-level correction training.

C. Datasets

The study uses Kitchen and Diverse+Kitchen datasets spanning common household-object tasks and many varied instructions. Diverse+Kitchen extends Kitchen with 24 semantic tasks in 30K additional demonstrations.

  • Datasets: Kitchen contains 6 semantic evaluation tasks in 70K demonstrations across common object categories and 542 unique instructions.The categories include cans, bottles, and fruits.
  • Datasets: Kitchen evaluation tasks include knocking objects over, drawer placement, moving one object near another, picking objects, opening or closing drawers, and placing objects upright.These tasks define the semantic task families used in evaluation.
  • Datasets: Diverse+Kitchen adds 24 semantic tasks in 30K demonstrations and contains 165 unique instructions.It includes all demonstrations from Kitchen.
  • Datasets: The Kitchen instructions cover actions such as retrieving napkins, placing bowls or cups upright, opening jars, handling basket tools, dispensing cereal, and pouring snacks.The listed instructions illustrate the dataset’s varied object-manipulation behaviors.
  • Datasets: Additional instructions include placing jar lids, putting tools inside baskets, positioning bowls under or away from dispensers, placing food in bowls, and squeezing honey.These examples broaden the task set beyond simple pick-and-place actions.
  • Datasets: Other listed behaviors include swiveling a cereal dispenser, picking up tongs, placing tongs in a basket, scooping snacks, moving objects near objects, and squeezing honey into a bowl.The dataset contains substantial variation in manipulation goals and object relations.
  • Datasets: The dataset has large data imbalances between tasks, with additional demonstrations for knock-over, move, pick, and place tasks forming a small fraction of the overall data.The evaluation tasks are bolded in the dataset description.

D. Detailed Results

RT-H improves staged performance across diverse tasks, generalizes better to novel objects, and produces context-sensitive, multimodal language motions, although some individual manipulation stages remain difficult.

  • Interventions: Many tasks require correction at only one or two stages, so RT-H-Intervene can improve performance with few language corrections and little new data.The reported pattern links localized stage failures with efficient intervention-based improvement.
  • Generalization: RT-H outperforms RT-2 in both final success and every individual stage for pick and move tasks involving novel objects.The evaluation measures staged cumulative success rates for generalization to novel objects.
  • Contextuality: The contextuality analysis groups actions by language motion and examines means and standard deviations across arm, rotation, and gripper dimensions.The dominant action dimension has the largest mean and variance for each basic language motion.
  • Multimodality: RT-H predicts multiple contextually valid language motions, ranging from subtle directional variants to distinct behaviors such as moving left or closing the gripper.Beam-search examples show that different predicted motions can each be reasonable for the same scene and task.

E. Frequently Asked Questions

The FAQ explains RT-H’s conceptual advantage, distinguishes its hierarchy from long-horizon planning, and identifies contextuality, labeling, and tool-use expansion as important boundaries. It also reports robust intervention integration and similar performance across asynchronous and synchronous inference.

  • Conceptual Motivation: RT-H structures high-dimensional action prediction into two simpler stages, using language motions as a bottleneck that can support transfer across tasks and datasets.The proposed mechanism is predicting motions from tasks and then actions from the inferred motions, rather than mapping tasks directly to every action dimension.
  • Comparison with Planning: Unlike SayCan, which decomposes long-horizon instructions into medium-horizon tasks, RT-H learns mappings from medium-horizon tasks to short-horizon motions.RT-H’s hierarchy operates one level below the task decomposition used by LLM planning frameworks.
  • Contextuality: Predefined language-motion primitives are impractical for precise manipulation because a single motion requires substantial contextual variation across action dimensions.The paper therefore argues that VLMs are better suited to representing these contextual motions.
  • Automated Labeling: Current large VLMs could not produce reasonable language motions zero-shot in the authors’ exploration, likely because they lack strong spatial priors for robot behaviors.The automated labeling procedure is presented as a scalable way to teach spatial concepts such as left and forward.
  • Future Scope: Extending language motions to object-referential tool-use phrases remains challenging because obtaining them without expensive human annotation is unresolved.Suggested examples include “rotate the screw” and “grasp the pot handle.”
  • Intervention Training: Adding intervention data did not produce observed model collapse: offline metrics on different data splits remained stable after co-training.The authors attribute this capacity to the large parameter counts of RT-H and related VLMs.
  • Inference: RT-H’s asynchronous inference has lower offline MSE than RT-H-Joint and RT-2, while asynchronous and synchronous variants show fairly minor performance differences.Synchronous inference substantially increases inference time, making online evaluation difficult.
Loading 2403.01823v2…