PaperlayerView exact revision on arXiv

Source-linked AI summary

Action-Free Reasoning for Policy Generalization

Jaden Clark, Suvir Mirchandani, Dorsa Sadigh, Suneel Belkhale

arXiv:2502.03729v2cs.ROcs.AI

TL;DR

Robot policies trained by imitation learning struggle to generalize, while human videos lack the action labels imitation learning typically uses. RAD trains on robot data with reasoning and action labels and action-free human videos with reasoning labels. It reports better performance on tasks seen only in action-free data and on tasks unseen in both sources.

  • Problem

    Robot imitation policies face a generalization challenge, and human videos lack action labels while grounded action representations can rely on assumptions about embodiment mapping.

    Source: §I · pp. 1–2

  • Method

    RAD co-trains a policy on robot demonstrations with reasoning and action labels and action-free human videos with reasoning labels only.

    Source: §I · p. 2

  • Results

    20% better performance is reported on tasks seen only in action-free data, and RAD outperforms baselines by 15% on tasks unseen in both robot and human data.

    Source: §I · p. 2

  • Takeaways & Limitations

    RAD enables transfer across the embodiment gap and supports learning tasks represented only in human videos or absent from both training sources.

    Source: §V · p. 7

    The collected human-hand demonstrations limit expressed hand motions to Cartesian axes, constraining the motion types represented in this study.

    Source: §V · p. 7

Abstract

from arXiv · show

End-to-end imitation learning offers a promising approach for training robot policies. However, generalizing to new settings remains a significant challenge. Although large-scale robot demonstration datasets have shown potential for inducing generalization, they are resource-intensive to scale. In contrast, human video data is abundant and diverse, presenting an attractive alternative. Yet, these human-video datasets lack action labels, complicating their use in imitation learning. Existing methods attempt to extract grounded action representations (e.g., hand poses), but resulting policies struggle to bridge the embodiment gap between human and robot actions. We propose an alternative approach: leveraging language-based reasoning from human videos-essential for guiding robot actions-to train generalizable robot policies. Building on recent advances in reasoning-based policy architectures, we introduce Reasoning through Action-free Data (RAD). RAD learns from both robot demonstration data (with reasoning and action labels) and action-free human video data (with only reasoning labels). The robot data teaches the model to map reasoning to low-level actions, while the action-free data enhances reasoning capabilities. Additionally, we will release a new dataset of 3,377 human-hand demonstrations with reasoning annotations compatible with the Bridge V2 benchmark and aimed at facilitating future research on reasoning-driven robot learning. Our experiments show that RAD enables effective transfer across the embodiment gap, allowing robots to perform tasks seen only in action-free data. Furthermore, scaling up action-free reasoning data significantly improves policy performance and generalization to novel tasks. These results highlight the promise of reasoning-driven learning from action-free datasets for advancing generalizable robot control. Project page: https://rad-generalization.github.io

I. INTRODUCTION

Robot imitation policies need to generalize beyond their training settings, but scaling robot demonstrations may be impractical and human videos lack action labels. RAD uses language reasoning from both sources, with action-free human-video reasoning supporting policies that transfer across embodiments and generalize to unseen tasks.

I. INTRODUCTION

  • Robot demonstrations may be impractical to collect at the scale needed for comprehensive generalization, while human videos are abundant but lack action labels.Extracting hand poses or affordances as action-like labels can require assumptions about scenes and mapping human actions to robot actions.Source: §I · pp. 1–2
  • RAD trains on robot demonstrations with reasoning and action labels alongside action-free human videos labeled only with reasoning.Robot data teaches the model to map reasoning to low-level actions, while human-video data augments its reasoning knowledge.Source: §I · p. 2
  • 20% better performance on tasks seen only in action-free data and 15% better performance on tasks unseen in both robot and human data are reported for RAD.The comparisons are respectively against models not fine-tuned with RAD and baselines.Source: §I · p. 2

II. RELATED WORK

  • RAD treats language reasoning as an action representation, rather than relying only on language as a goal or instruction representation.Prior work has used language to condition policies or decompose instructions, and fine-grained language to guide policies [30] [13] [5] [6] [16] [11] [28] [2] [40].Source: §II · p. 2
  • Prior human-video approaches use visual pretraining, reward learning, video priors, or alignment, while some intermediate-representation methods assume human-hand behavior matches robot actions.The paper positions detailed reasoning steps about human-video demonstrations as an alternative to generating intermediate representations for action prediction [19] [36] [14] [8] [18] [27] [1] [35] [26] [29] [37] [12] [23] [21] [4].Source: §II · p. 2

III. REASONING THROUGH ACTION-FREE DATA

RAD annotates action-free data with language reasoning, then trains a reasoning-based policy using both robot demonstrations and action-free samples. Its objective combines action-and-reasoning supervision from robot data with reasoning-only supervision from action-free data, which can provide partial reasoning chains of varying quality.

III. REASONING THROUGH ACTION-FREE DATA

  • RAD has two steps: annotate action-free data with language reasoning, then train with robot data containing actions and reasoning and action-free data containing reasoning only.This is the method's overview of the pipeline.Source: §III · p. 3

A. Problem: Learning Reasoning in Action-free Data

  • Multi-task imitation learning seeks to learn expert actions conditioned on observations and task specifications.The reasoning-based objective models the joint distribution of actions and intermediate language reasoning given observations and task specifications.Source: A. Problem: Learning Reasoning in… · p. 3
  • Action-free samples provide additional supervision for the joint action-reasoning distribution using observations, goals, and one or more reasoning steps, without action labels.The number of labeled reasoning steps can vary across samples.Source: A. Problem: Learning Reasoning in… · p. 3
  • Variable-length reasoning labels let the framework use samples with different confidence levels and embodiment gaps, improving reasoning steps independently from action prediction.Samples may have only higher-level reasoning labels or also higher-quality lower-level reasoning labels.Source: A. Problem: Learning Reasoning in… · p. 3
  • The training objective combines robot-data reasoning-and-action likelihood with an auxiliary reasoning likelihood for action-free data.A sample's available reasoning steps are modeled from preceding steps, the observation, and the task.Source: A. Problem: Learning Reasoning in… · p. 3

B. Reasoning Steps in RAD

RAD organizes reasoning as a progression from task-level plans toward spatially grounded information and low-level robot actions. The progression includes language motions, gripper position, and object locations as intermediate steps.

B. Reasoning Steps in RAD

  • GripperPosition predicts the end-effector's pixel position, while VisibleObjects predicts scene-object bounding box coordinates.Source: B. Reasoning Steps in RAD · p. 3
  • The reasoning sequence moves from TaskPlan through SubtaskReasoning and Subtask to MoveReasoning and MovePrimitive, then to GripperPosition and VisibleObjects before the Action output.TaskPlan outlines subtasks; MovePrimitive is a language prediction, GripperPosition predicts end-effector pixel position, VisibleObjects predicts object bounding boxes, and Action predicts an end-effector position delta.Source: B. Reasoning Steps in RAD · p. 3
  • The stages trace increasing physical and spatial groundedness, from high-level task and subtask reasoning through language motions to spatial information and low-level action.RAD instantiates reasoning steps from prior work [40].Source: B. Reasoning Steps in RAD · p. 3

C. Labeling Reasoning in Action-free Data

RAD labels action-free human videos by generating scene, object, and movement information, then synthesizing language reasoning. Human hand tracking supplies movement primitives where robot actions are unavailable, though rotational primitives are less reliably detected than gripper and positional movements.

C. Labeling Reasoning in Action-free Data

  • The labeling pipeline generates scene descriptors and object boxes for human and robot data, then uses Gemini to synthesize natural-language reasoning from these and movement primitives.Robot data can also contribute action labels; the resulting reasoning is tokenized for co-finetuning on mixed human and robot data.Source: Figure 3 · p. 4
  • For robot demonstrations, the ECoT-style procedure extracts grounded tags and movement primitives from detections and actions, then uses Gemini to label earlier reasoning steps [40].The paper describes this automated procedure as the basis for its human-video labeling modifications.Source: C. Labeling Reasoning in Action-free… · p. 4
  • For human videos, HaMeR hand-pose tracking supplies movement primitives by labeling the dominant axis of hand-pose change, but rotational primitives are less reliable than gripper and positional ones [22].Human-video action labels are unavailable, so movement primitives and gripper positions cannot be extracted from explicit actions.Source: C. Labeling Reasoning in Action-free… · p. 4

D. Training on Partial Reasoning Chains

RAD reuses a reasoning-and-action prediction architecture for robot demonstrations, while action-free data contributes reasoning labels only. The supplied passages also distinguish RAD from a hand-location-only comparison.

D. Training on Partial Reasoning Chains

  • Robot demonstrations train the model to predict language reasoning followed by action tokens, while action-free data supplies reasoning-only labels.The approach uses the ECoT and OpenVLA architecture, a pretrained 7B-parameter vision-language transformer.Source: D. Training on Partial Reasoning Chains · pp. 4–5
  • ECoT-GT is finetuned on the same human video data as RAD but uses human hand locations rather than full reasoning data.Fig. 4 compares this baseline with RAD and RAD-A where human-video data was trained on without new robot data.Source: Figure 4 · p. 5
  • The action-free-data setup retains reasoning supervision while omitting robot action labels.In RAD, action-free data labels contain only reasoning.Source: D. Training on Partial Reasoning Chains · p. 5

IV. EXPERIMENTS

The experiments test whether RAD supports human-to-robot transfer, generalization beyond both human and robot training data, and transfer from out-of-domain human videos. Evaluation spans three generalization axes and compares RAD with reasoning and hand-tracking baselines.

IV. EXPERIMENTS

  • The experiments ask whether RAD learns human-video-only tasks, generalizes beyond both training datasets, and transfers from out-of-domain human videos.These are the paper's Q1, Q2, and Q3 evaluation questions.Source: §IV · p. 5

A. Evaluating Generalization

  • Evaluations use a 6-DoF WidowX arm, with Toy Sink tests and additional plain-tabletop, cluttered-desk, home, and kitchen environments.The Toy Sink is used for Sections IV-B and IV-C; additional environments support testing on unstructured settings in IV-D.Source: A. Evaluating Generalization · p. 5
  • Generalization is assessed compositionally, on unseen objects, and in new scenes with novel backgrounds or distractors.Compositional tests interpolate among seen objects, tasks, and scenes; new-object and new-scene tests extrapolate beyond training data.Source: A. Evaluating Generalization · p. 5
  • The comparisons include Bridge V2-trained ECoT, hand-tracking ECoT-GT, full-reasoning RAD, and axis-specific RAD-A.ECoT-GT generates only the GripperPosition portion of the reasoning chain; RAD uses the full chain from human videos.Source: A. Evaluating Generalization · p. 5

B. Can RAD enable transfer from human-to-robot embodiments?

RAD transfers reasoning learned from human videos to robot tasks, including tasks unseen in both human and robot data. Across compositional, new-object, and new-scene generalization, it improves on ECoT; axis-specific comparisons also show transfer without new robot demonstrations.

C. Can RAD train more generalizable policies?

B. Can RAD enable transfer from human-to-robot embodiments?

  • On human-video-only tasks, RAD-A exceeds zero-shot ECoT and ECoT-GT across generalization areas despite no new robot demonstrations.For compositional tasks, RAD-A outperforms ECoT by 23% and ECoT-GT by 20%; for new scenes, full RAD outperforms them by 27% and 30%.Source: B. Can RAD enable transfer from… · p. 6

C. Can RAD train more generalizable policies?

  • For compositionally novel tasks, RAD outperforms ECoT by 5%, with better reasoning reported on multi-step tasks.The section evaluates 10 tasks absent from both human and robot data across all three generalization axes.Source: C. Can RAD train more generalizable… · p. 6
  • For unseen objects, RAD achieves 30% higher success than ECoT, and for novel scenes it reaches an 18% higher success rate.The novel-scene comparison includes large distractors; the authors report that RAD's reasoning informed downstream action prediction.Source: C. Can RAD train more generalizable… · p. 6

D. Can RAD leverage data from new environments?

RAD is evaluated with human videos collected outside the Bridge V2 environment and with added in-domain or out-of-domain demonstrations. Both environment transfer and additional data mixtures improve reported performance over their respective comparisons.

D. Can RAD leverage data from new environments?

  • Human videos from a new tabletop environment yield models that outperform ECoT by 16% and ECoT-GT by 13% when evaluated in the original Bridge Toy Sink environment.The authors also report improved reasoning about grasp points, such as where to pick up a controller.Source: D. Can RAD leverage data from new… · p. 7
  • The data-scaling comparison adds 100 in-distribution or 250 out-of-domain demonstrations for the tape-pickup task to the original RAD mix.That original mix already contained 40 demonstrations of the task.Source: D. Can RAD leverage data from new… · p. 7
  • Adding in-distribution demonstrations improves success by 30%, while adding out-of-domain demonstrations improves success by 25% over the original RAD model.The in-distribution data comes from the Toy Sink setup; out-of-domain videos come from real-world kitchen and office environments.Source: D. Can RAD leverage data from new… · p. 7
  • RAD models better reasoned about bringing the gripper to tape height, while ECoT models often moved too low and knocked the tape over.The tape was unusually tall relative to objects in Bridge V2.Source: D. Can RAD leverage data from new… · p. 7

V. DISCUSSION

RAD uses reasoning labeled on both robot and human videos to train generalist robot policies. The authors report transfer across the embodiment gap, generalization to tasks absent from both data sources, and positive responses to out-of-domain data.

V. DISCUSSION

  • RAD predicts reasoning that can be labeled on both robot and human video data, providing a way to train generalist robot policies from human videos.Source: §V · p. 7
  • RAD enables vision-language-action models to learn tasks represented only in human video and generalize to tasks absent from both robot and human data.The authors also report that RAD responds positively to out-of-domain environments, enabling learning of tasks from environments separate from the target domain.Source: §V · p. 7
  • The study focuses on pick-and-place tasks with rigid objects, while more fine-grained and dexterous manipulation remains future work.The collected human videos also restrict hand motions to Cartesian axes; more accurate hand-pose estimation may partially mitigate this limitation.Source: §V · p. 7

APPENDIX

The appendix describes the human-video collection and reasoning-generation pipeline, demonstration mixtures, and model training setup. It also outlines the evaluation-results material presented in the appendix.

APPENDIX

  • The appendix organizes its material into dataset and reasoning generation, model and training details, and examples of results with reported success rates.Source: APPENDIX · p. 11

A. Dataset Details

  • Human videos were recorded in the Bridge V2 Toy Sink setup from a main-view camera and a second camera behind the WidowX gripper for hand tracking.HaMeR tracking was used to derive movement primitives and gripper locations for reasoning generation, following the [40] pipeline.Source: A. Dataset Details · p. 11, Figure 6 · p. 11
  • RAD-A used 392 compositional-generalization, 304 new-object, and 280 new-scene demonstrations; RAD and ECoT-GT used those plus 640 additional demonstrations, totaling 1,616.Source: A. Dataset Details · p. 11
  • Table I tasks used 40 demonstrations each, Table II used 100 additional Toy Sink demonstrations for in-distribution evaluation, and OOD data comprised 50 demonstrations from five scenes.Source: A. Dataset Details · p. 11

B. Training Details

  • RAD uses the Prismatic VLM architecture with a LLaMA 2 7B language backbone, and models are fine-tuned to convergence with a 2e-4 learning rate and LoRA batch size 2.Training used two to eight L40s or A40 GPUs; ECoT-GT omits the stop-token loss and changes the query prompt to ask for hand location.Source: B. Training Details · p. 11

C. Results

This appendix section presents examples of human-video tasks and the environments and task types used in evaluations. It also specifies trial procedures and partial-credit rules for selected tasks.

C. Results

  • Each task was evaluated 10 times, with objects randomly placed in different spots across the trials.Source: C. Results · p. 11
  • Pick-and-place trials received 0.5 partial credit for a successful pickup followed by incorrect placement; controller pickup also received partial credit if the object slipped after lifting.Source: C. Results · p. 11
  • Fig. 7 shows examples of the human-video tasks collected.Source: Figure 7 · p. 12
  • Fig. 8 shows demonstrations collected outside Bridge V2 to assess RAD's response to data from different scene types.Source: Figure 8 · p. 12
  • Fig. 9 depicts real-world environment data used to train RAD for Section IV-D.Source: Figure 9 · p. 12
  • Fig. 10 gives example tasks for compositional novelty, new objects, and new scenes.Source: Figure 10 · p. 13

Related papers

Loading 2502.03729v2…