PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Embodiment Causal Transfer from Human Videos to Robot Manipulation

Xingyu Zhang, Yingchao Li, Peizheng Guo, Jie Hu, Changwen Zheng, Wenwen Qiang

arXiv:2610.21798v1cs.ROcs.AI

TL;DR

Robot manipulation needs substantial interaction data, but diverse real-robot experience is costly to collect and human motions cannot directly supervise robot actions across embodiments. CBST transfers human-video representations through robot-grounded task variables and contextual causal-beam selection to condition a world-action model. The paper reports task-success improvements over corresponding backbone baselines on robot manipulation benchmarks.

  • Problem

    Real-robot interaction data are costly to scale, while human videos are abundant but their actions cannot be directly reused as robot supervision across embodiment differences.

    Source: §1 · p. 1

  • Method

    CBST grounds human-video representations in robot-supervised task variables and uses contextual causal-beam constraints to select information for conditioning a frozen world-action model.

    Source: §1 · p. 2

  • Results

    CBST improves task success over corresponding backbone baselines on robot manipulation benchmarks.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Within a predefined manipulation model, causal-beam selection transfers contextual task dependencies without requiring identical human and robot motion realizations.

    Source: §3.3 · p. 4, §6 · p. 9

    The shared task-level schema is a hypothesis space, not a guarantee that a beam valid in one embodiment remains valid in another; selection must be evaluated for the instantiated mechanisms and context.

    Source: §3.3 · p. 4

Abstract

from arXiv · show

Scaling robot manipulation models requires large amounts of interaction data, yet collecting diverse real-robot experience remains costly. Human manipulation videos, in contrast, provide abundant and diverse task experience that could potentially benefit robot learning. However, differences in morphology, action spaces, and control mechanisms make human motions unsuitable as direct supervision for robot actions, raising a scientific question: what knowledge in human videos should be transferred across embodiments to robot manipulation? To address this question, we analyze subtask transfer through the lens of causal beams, focusing on which contextual conditions and interaction dependencies should be retained to support local task effects. Based on this perspective, we propose Causal-Beam-Guided Subtask Transfer (CBST). CBST learns manipulation-related representations from human videos and grounds them in task-level semantic variables using robot demonstrations. Within a predefined manipulation model, causal-beam constraints guide contextual information selection by requiring retained assignments to sustain local mechanism outputs while preserving nontrivial responsiveness to retained inputs. The selected semantic information conditions a world-action model, allowing the robot to generate actions from its own observations without requiring human-robot motion correspondence. Experiments on robot manipulation benchmarks show improvements in task success over the corresponding world-action model baselines.

1 Introduction

Human videos offer diverse task experience, but embodiment differences prevent direct reuse of human actions as robot supervision. The paper asks which task information can transfer while avoiding both endpoint-only summaries and embodiment-specific trajectories.

1 Introduction

  • Human manipulation videos are abundant and diverse, while collecting broad real-robot interaction data is costly and human actions cannot directly supervise robot actions.Source: §1 · p. 1
  • Subtask boundaries alone do not determine which local conditions and interaction relations to retain: endpoints can omit them, while full trajectories preserve embodiment-specific motion.Source: §1 · pp. 1–2
  • Causal beams select context-dependent inputs whose actual assignments sustain a mechanism’s output while its projected mechanism remains responsive to retained inputs.Source: §1 · p. 2
  • CBST grounds human-video representations in robot-supervised task variables, then selects contextual causal dependencies to condition a robot world-action model.Robot actions are generated from the robot’s own state and action space.Source: §1 · p. 2

2 Related Works

Related work spans motion-oriented human-to-robot transfer, higher-level task abstractions, and predictive world models. CBST is situated among approaches that transfer beyond direct motion correspondence.

Human-to-robot transfer

Temporal abstraction

  • Temporal-abstraction methods organize behavior into extended policies, intermediate subgoals, or latent plans for hierarchical planning.Examples include the options framework, Video Language Planning [Du et al., 2023], and hierarchical latent world models [Zhang et al., 2026].Source: §2 · p. 3

World models and World-Action Models

  • World models learn predictive dynamics for planning, while newer World-Action Models jointly represent future observations and robot actions.Examples include PlaNet [Hafner et al., 2019], Cosmos Policy [Kim et al., 2026], and LingBot-VA [Li et al., 2026].Source: §2 · p. 3

3 Problem Definition and Analysis

The section formulates robot control as a world-action model problem and analyzes cross-embodiment transfer using contextual causal beams over task-level manipulation variables. The shared model is a hypothesis space, not an assertion that mechanisms or valid beams are identical across embodiments.

3.1 Problem Formulation

  • A world-action model conditions on robot interaction history and task instruction to jointly generate actions and predict future observations in closed-loop control.The history contains prior observations and actions; executing an action chunk updates it with newly acquired observations.Source: §3.1 · p. 3

3.2 Causal Beams

  • Causal-beam selection partitions each mechanism’s parents into retained and remaining inputs, requiring retained assignments to sustain the actual output and the projection to remain responsive.The local responsiveness condition does not establish that every retained parent is individually necessary or has an intervention effect on the downstream outcome.Source: §3.2 · pp. 3–4

3.3 Causal Analysis of Cross-Embodiment Transfer

  • The candidate manipulation SCM shares task-level variable roles and possible dependencies across embodiments, but mechanisms may differ and not every candidate dependency is active in each context.The schema represents task-relevant variables and dependencies rather than a complete physical model.Source: §3.3 · p. 4
  • The local task variables distinguish starting conditions X, manipulation attempt I, resulting object-state change D, and realized effect Y.Separating an attempt from the object change distinguishes attempted manipulation from what is actually realized.Source: §3.3 · p. 4
  • In the bowl example, valid causal projections depend on execution context: matching the observed endpoint alone does not ensure that the dependencies producing it are preserved.The example motivates evaluating contextual beam conditions under robot-side reference mechanisms rather than directly reusing a human-side beam.Source: §3.3 · p. 5
  • The causal subtask representation records the realized local effect, retained parents and their actual values, and retained parent–child relations.Source: §3.3 · p. 5

4 Method

CBST estimates task-level semantics from robot observations, selects context-specific dependencies under causal-beam constraints, and uses the resulting subtask information to condition a frozen WAM. Training combines human-video action classification, robot-demonstration semantic supervision, and mechanism-consistency and structural-selection objectives.

4.1 Overall Architecture

  • CBST maps robot observations to semantic estimates, selects contextual dependencies under causal-beam constraints, and conditions WAM action generation with the resulting subtask information.The WAM combines this condition with robot observations to generate action chunks and future-observation predictions.Source: §4 · p. 6, §4.1 · p. 6
  • The selector chooses compatible parent subsets whose assignments satisfy sustenance and responsiveness conditions under fixed reference mechanisms.If selection is infeasible, the selected fields are omitted while task and state conditioning remain; the WAM and text encoder stay frozen.Source: §4.1 · p. 6, §4.2 · p. 7

4.2 Learning Objectives and Training

  • Human-video action-category classification initializes the visual-temporal encoder, and robot demonstrations supervise semantic predictions and mechanism consistency.The mechanism-consistency loss encourages predicted parent variables to reproduce annotated outputs under frozen reference mechanisms.Source: §4.2 · p. 6
  • Structural supervision trains the selector to assign probability to acceptable parent subsets, while a separate regularizer favors compact selections.The beam loss permits multiple acceptable subsets; compactness is a preference, not an additional beam-validity condition.Source: §4.2 · pp. 6–7

5 Experiments

CBST is evaluated with frozen WAM backbones across simulation benchmarks, with additional real-world and pretraining evaluations. Results report task success improvements in the supplied simulation comparisons, while the pretraining study finds higher SSv2 accuracy with ordered than shuffled clips.

5.1 Experimental Setup

  • Evaluation uses LingBot-VA on RoboTwin 2.0 and LIBERO, and Predict2.5 on LIBERO and RoboCasa, reporting task success averaged across evaluated tasks.Comparisons include the underlying WAMs and representative published policies.Source: §5.1 · p. 7

5.2 Results on Simulation Benchmarks

  • 94.00% and 92.54% success on RoboTwin 2.0 Easy and Hard; LIBERO-Long success is 98.60% with LingBot-VA and 97.33% with Predict2.5.On RoboCasa, Predict2.5 + CBST reaches 72.44% with 50 demonstrations per task, compared with 71.1% for the backbone.Source: §5.2 · p. 7, Table 3 · p. 8

5.3 Ablation Studies

  • The RoboTwin Easy ablations vary human-video initialization and temporal order, conditioning information, and mechanism-consistency supervision while holding robot adaptation data and evaluation settings fixed.Conditioning variants share the same trained predictor.Source: §5.3 · p. 8, Table 4 · p. 8

5.4 Real-World Experiments

  • The real-world evaluation compares Predict2.5 with and without CBST on ten AgileX object-placement tasks, including cross-workspace handovers, using 20 trials per task and full-task success after release.The supplied passage describes the setup but does not give the full-task success results.Source: §5.4 · p. 8, Figure 5 · p. 8

5.5 Human-Video Pretraining

  • 25.88% accuracy with ordered SSv2 pretraining versus 22.15% with frame-shuffled clips after three epochs.The comparison holds architecture, labels, sampled frame content, and optimization budget fixed; only temporal order differs.Source: §5.5 · p. 8, Figure 6 · p. 9
  • On RoboTwin Easy, ordered pretraining yields 94.00% success, versus 92.94% for shuffled pretraining and 92.80% without human-video pretraining.Robot-specific prediction heads and structural selection are learned during adaptation.Source: §5.5 · pp. 8–9

6 Conclusion

CBST uses causal-beam constraints to select contextual information for human-to-robot transfer. It combines human-video representations with robot-grounded task semantics and improves task success over corresponding backbone baselines.

6 Conclusion

  • CBST uses causal-beam constraints to guide contextual information selection for human-to-robot transfer.The selected information is supplied to a frozen world-action model for closed-loop control.Source: §6 · p. 9
  • Human-video pretraining initializes manipulation-related representations, which robot demonstrations ground in task-level semantic variables.Source: §6 · p. 9
  • Experiments on robot manipulation benchmarks show task-success improvements over the corresponding backbone baselines.Source: §6 · p. 9

AI Usage Statement

The appendices define the causal-beam construction and illustrate its context dependence, then document the datasets, comparisons, and implementation used for evaluation. The paper also states that generative AI was used only for grammar and language polishing.

AI Usage Statement

  • The authors state that generative AI tools were used for grammar and language polishing, not research ideation, methodology, analysis, or substantive writing.Source: AI Usage Statement · p. 13

A.1 Formal Details of Causal Beam Construction

  • The causal-beam construction partitions mechanism parents into retained and remaining sets, requiring retained values to sustain the actual output and the projection to remain responsive to retained inputs.Selections across mechanisms must also satisfy compatibility conditions.Source: §A.1 · p. 13

A.2 An Illustrative Construction of a Causal Beam

  • Two contexts of the same binary SCM yield different valid projections, illustrating that retained parents depend on context rather than general irrelevance.In the all-ones context, both parents are retained for each mechanism; in the second context, smaller retained sets suffice.Source: §A.2 · pp. 13–14

B.1.1 Something-Something V2

  • SSv2 action classification pretrains the visual-temporal representation, which initializes the robot planner; human labels are not robot action labels or direct supervision for causal parent sets.SSv2 has 174 action classes and 220,847 released videos.Source: §B.1.1 · pp. 14–15

B.1 Datasets and Benchmark Composition

  • Evaluation spans 50 bimanual RoboTwin tasks, four ten-task LIBERO suites, and a 24-task RoboCasa protocol, with the benchmark-specific settings described separately.RoboTwin Easy and Hard are reported separately; RoboCasa excludes kitchen navigation.Source: §B.1.2 · p. 15, §B.1.3 · p. 16, §B.1.4 · p. 17

B.2 Baseline Details

  • Comparisons include the two CBST WAM backbones and external methods, while reported results preserve their source, model version, demonstration budget, evaluation setting, and uncertainty conventions.The RoboTwin, LIBERO, and RoboCasa comparison sources are specified separately.Source: §B.2 · pp. 17–18

B.3 Implementation Details

  • The planner combines shared-view ResNet18 features, proprioception, and task features through a GRU, then adapts robot-specific heads while keeping the WAM frozen.Human-pretrained components initialize the planner; robot demonstrations provide semantic and structural supervision.Source: §B.3 · pp. 18–19

B.4 Evaluation Metrics and Protocols

The evaluation reports task success rates and aggregates them across tasks under benchmark-specific protocols. The appendices distinguish episode budgets, execution settings, and sources of randomness.

B.4 Evaluation Metrics and Protocols

  • Task success uses official episode indicators, and the aggregate score averages task success rates, equaling pooled success when each task has the same episode count.Source: §B.4 · p. 19
  • RoboTwin Easy and Hard are aggregated separately, LIBERO-Long averages ten tasks, and RoboCasa averages 24 tasks.Published baselines retain their sources’ aggregation and uncertainty conventions.Source: §B.4 · p. 19
  • The protocol records action-chunk length, actions executed per request, episode limits, and training and evaluation randomness separately.Error-bar meanings are specified in the corresponding table or figure caption.Source: §B.4 · p. 20, Table 15 · p. 20

C Additional Simulation Results

The appendices expand benchmark comparisons and ablations while retaining source-specific reporting conventions. They also document qualitative executions and define the controls used to analyze pretraining, objectives, and conditioning.

C Additional Simulation Results

C.1 RoboTwin 2.0

  • Table 16 groups RoboTwin Easy and Hard success rates by task horizon and retains source-reported aggregates rather than recomputing them from rounded task entries.Source: §C.1 · p. 20

C.2 LIBERO

  • Table 17 combines four-suite LIBERO comparisons and preserves source-dependent baseline differences alongside CBST results for both WAM backbones.CBST’s Average is the arithmetic mean of its four suite success rates.Source: §C.2 · p. 20, Table 17 · p. 21

C.3 RoboCasa

  • Table 18 reports the 24-task RoboCasa comparison with demonstration budgets retained because methods use different amounts of data.The CBST entry counts overlapping demonstrations only once.Source: §C.3 · p. 20, Table 18 · p. 22

C.4 Additional Qualitative Examples

  • Figure 7 presents three additional recorded executions as temporally ordered frames, with task names and recording sources given in the captions.The examples include two RoboTwin tasks and one LIBERO task.Source: §C.4 · p. 21, Figure 7 · p. 22

D Additional Ablation Studies

  • The additional ablation analysis separates changes to pretraining, robot-adaptation objectives, and inference-time conditioning.This distinction indicates which comparisons need independently trained planners and which use a shared checkpoint.Source: D Additional Ablation Studies · p. 21

D.1 Variant Definitions

  • The variants remove human-video pretraining, shuffle human-video frame order, omit selected dependencies, provide the full candidate structure or random subsets, or remove mechanism-consistency loss.State-only conditioning omits dependency and interval-interaction fields while keeping state predictions unchanged.Source: §D.1 · pp. 22–23

D.2 Results across Backbones and Benchmarks

  • Table 20 extends the principal ablations across WAM backbones and benchmarks, with RoboTwin Easy and Hard separate and published backbone scores treated as references, not paired ablation measurements.Source: §D.2 · p. 23

D.3 Temporal Architecture

These appendices specify temporal-module comparisons, training diagnostics, sensitivity configurations, computational-cost measurements, and the AgileX platform and task formulation. They describe evaluation scope and measurement choices but provide no numeric comparison outcomes in the supplied passages.

D.3 Temporal Architecture

  • The temporal-module comparison covers GRU, Transformer, TCN, and MLP under controlled inputs and supervision, with planner parameter counts and update latency reported on LIBERO-Long.The comparison retains the same visual encoder, observation samples, robot supervision, and WAM interface; parameter counts are reported rather than assumed equal.Source: §D.3 · pp. 23–24, Table 21 · p. 24

D.4 Training and Validation Curves

  • Training curves separate human-pretraining losses and accuracy from robot-adaptation objectives and heldout diagnostics, while preserving logged training boundaries and the stated model-selection rule.Human-pretraining plots compare correctly ordered and shuffled clips; adaptation plots report semantic-state, interval-variable, mechanism-consistency, and structural-selection objectives.Source: §D.4 · p. 24

D.5 Sensitivity to Observation History

  • Observation-history sensitivity varies frame-sampling density and lookback duration independently, retaining the current observation and keeping each setting consistent across adaptation and evaluation.Other planner and WAM settings remain unchanged.Source: §D.5 · p. 24

D.6 Sensitivity to Structural Compactness

  • Structural-compactness sensitivity varies λ_size, including a zero setting that removes explicit compactness preference, and reports control success, retained-parent fraction, and WAM token count separately.Token counts use each backbone’s native tokenizer because equal parent counts need not produce equal token counts.Source: §D.6 · p. 24, Table 22 · p. 25

D.7 Computational Cost

  • Computational-cost measurements isolate planner and conditioning components from WAM generation and measure end-to-end request latency directly under fixed input and hardware settings.Planner time includes visual-temporal encoding, semantic prediction, and parent-set selection; simulator stepping and video recording are excluded.Source: §D.7 · p. 25

E.1 Platform and Task Formulation

  • The real-world platform uses two AgileX manipulators over a shared tabletop, with tasks defined by object and target and coordination through overlapping reach when one arm cannot reach both.No particular handover trajectory is prescribed.Source: §E.1 · p. 25

E.2 Task Definitions

The task inventory describes placement outcomes, while the detailed real-world results are organized by task and method. The supplied passages do not provide task names or result values.

E.2 Task Definitions

  • The task inventory covers ten task entries whose names specify placement outcomes rather than a required handover.Target locations are fixed within a task configuration; changes across configurations are documented separately from initial-pose variation.Source: §E.2 · p. 25
  • Task names encode placement outcomes, not whether or how the arms must hand over an object.The target is fixed within each task configuration.Source: §E.2 · p. 25
  • Detailed real-world results report 20 trials per method for each task and express gains in percentage points.The results are presented in Table 24; its values are not included in the supplied passage.Source: Table 24 · p. 26

E.3 Baseline and Evaluation Protocol

The evaluation compares a Predict2.5 Cosmos Policy baseline with its CBST-augmented counterpart under a controlled protocol and scores completed placements. Qualitative videos illustrate selected executions but do not establish task-suite success rates or execution frequency.

E.3 Baseline and Evaluation Protocol

  • The controlled comparison adds CBST contextual structural conditioning while holding the task-adapted checkpoint, observations, control settings, and evaluation conditions fixed.The published ALOHA reference uses Predict2, and official results are not substituted for evaluation on the AgileX tasks.Source: §E.3 · p. 26
  • Full-task success requires stable final placement after release; grasping or transfer without completing placement does not count.Handover is neither an additional task objective nor independently rewarded.Source: §E.3 · p. 26
  • With 20 trials per task, individual task rates change in 5-percentage-point increments and the ten-task average in 0.5-point increments.Improvements are reported in percentage points; a three-task visualization mean is labeled as a subset mean, not the ten-task result.Source: §E.3 · p. 26

E.4 Qualitative Executions

  • The qualitative examples show plate transport involving grasp transfer and remote-control placement by one arm, but do not measure how often those strategies occur across tasks.Figure 8 presents additional AgileX executions in temporal order, with timestamps and event annotations.Source: §E.4 · p. 26, Figure 8 · p. 27
  • Selected successful videos are qualitative examples, not estimates of task success rates, control frequency, or execution latency.The timestamps identify frames rather than control frequency or latency.Source: §E.4 · pp. 26–27

Related papers

Loading 2610.21798v1…