PaperlayerView exact revision on arXiv

Source-linked AI summary

EXE: Egocentric Cross-Embodiment Pretraining via Factorized Latent Actions for VLA Models

Qingyuan Cai, Bo Zhang, Yongzhen Huang, Jianyu Chen

arXiv:2610.33685v1cs.ROcs.CV

TL;DR

Robot-data scarcity and embodiment-specific kinematics make action-level use of egocentric human video difficult. EXE factorizes latent actions into shared and private components and reports consistent gains across simulated and real-world robot tasks.

  • Problem

    Human and robot actions share manipulation primitives but use distinct kinematics, making it difficult to extract action-level knowledge from human video in a form that transfers to robot control.

    Source: §I · p. 1

  • Method

    EXE factorizes actions into shared positional and embodiment-specific kinematic subspaces, aligning shared codes across domains with spatial and language-guided semantic signals.

    Source: §I · p. 1, CONCLUSION · p. 7

  • Results

    EXE improves success over no-pretraining baselines across RoboTwin 2.0, RoboCasa GR1, and real-world tasks; on RoboCasa GR1, it reaches 56.6% versus EXEbase’s 52.0% and StarVLA-GR00T’s 47.8%.

    Source: §I · p. 2, 2) Results on RoboCasa GR1 · p. 6

  • Takeaways & Limitations

    The shared action representation transfers manipulation knowledge from egocentric human video to robots with grippers or dexterous hands.

    Source: CONCLUSION · p. 7

Abstract

from arXiv · show

Vision-Language-Action (VLA) models are bottlenecked by the scarcity of robot data. Egocentric human manipulation videos offer an abundant, diverse, and semantically-rich alternative that inherently captures manipulation intent. However, the embodiment gap makes it difficult to extract action-relevant representations from these videos. From the perspective of action execution, human hands and robots share analogous manipulation primitives---reaching, grasping, placing---yet exhibit distinct kinematics tied to their embodiments. This might motivate our key approach: egocentric video pretraining can yield shared action representations capturing transferable manipulation skills, while human-specific motion patterns should be disentangled to avoid interfering with robot policy learning. To this end, we propose EXE, an Egocentric Cross(X)-Embodiment pretraining framework that learns a factorized latent action representation from egocentric human videos, separating shared manipulation priors from embodiment-specific signals. By leveraging the shared action space, the robot policy acquires generalizable manipulation semantics from diverse human demonstrations without being biased by human-specific kinematics. This shared knowledge transfers across embodiments with different kinematic structures, enabling the same pretraining to benefit both parallel grippers and dexterous hands. EXE yields consistent gains over baseline without pretraining on RoboTwin 2.0, RoboCasa GR1, and real-world manipulation tasks.

I. INTRODUCTION

EXE addresses the challenge of extracting transferable action knowledge from egocentric human video by separating shared manipulation semantics from embodiment-specific motion. Its factorized pretraining improves success rates across gripper, dexterous-hand, and real-world tasks.

I. INTRODUCTION

  • Robot-data scarcity limits VLA generalization, while prior video-pretraining approaches do not transfer the action information embedded in human hand motions.The open question is how to convert egocentric video action knowledge into representations a VLA can absorb and translate into robot actions.Source: §I · p. 1
  • Human and robot actions share manipulation primitives but differ in kinematics, making full-action sharing or visual-transition latents vulnerable to embodiment geometry or visual noise.The proposed division shares wrist or end-effector position while keeping finger and gripper geometry private.Source: §I · p. 1
  • Language-guided relational distillation complements contrastive spatial alignment by bringing semantically equivalent actions together even when their trajectories differ.The language signal supplies pairwise semantic similarity as a soft target.Source: §I · pp. 1–2
  • EXE’s feature-group-routed tokenizer quantizes positional actions with a jointly trained cross-domain codebook and embodiment-specific kinematics with per-domain codebooks.This separates transferable task semantics from domain-specific geometry.Source: §I · p. 2
  • EXE trains a factorized tokenizer, pretrains VLMs on human video and robot data using shared action tokens, then fine-tunes on downstream robot tasks.Ablations report independent contributions from feature-group factorization and language-guided alignment.Source: §I · p. 2
  • EXE consistently improves success rates over baselines without pretraining on RoboTwin 2.0, RoboCasa GR1, and real-world tasks.The reported benefits span gripper and dexterous-hand embodiments.Source: §I · p. 2

II. RELATED WORK

Related work draws on egocentric video and several approaches for converting human motion into robot-executable signals. EXE factorizes actions and transfers shared representations through a staged VLM-pretraining and robot-fine-tuning pipeline.

II. RELATED WORK

  • Egocentric video offers diverse first-person hand–object interactions, while prior VLA approaches convert human motion through pose retargeting, shared representations, reconstruction, or latent-action vocabularies.Explicit approaches require accurate pose estimation and hand-designed embodiment mappings.Source: §II · p. 2
  • EXE addresses limitations of full-action sharing and visual entanglement by aligning a cross-embodiment positional subspace while keeping embodiment-specific kinematics separate.The approach avoids pose retargeting and paired demonstrations.Source: §II · p. 2

III. METHOD

  • The method trains a factorized tokenizer, pretrains a VLM to predict shared action codes from human video and robot demonstrations, then fine-tunes a lightweight action expert on robot data.Stage 3 uses continuous action regression and conditions on Stage 2 representations through Bridge Features.Source: §III · pp. 2–3
  • The tokenizer’s shared group uses one cross-domain codebook, while its specific group uses per-domain codebooks.The figure labels the corresponding shared and specific encoders and decoders.Source: Figure 1 · p. 3

A. Stage1: Training Factorized Action Tokenizer

Stage 1 factorizes actions into a shared positional subspace and embodiment-specific kinematics, then trains separate codebooks with cross-domain alignment so semantically equivalent motions can share codes.

A. Stage1: Training Factorized Action Tokenizer

  • The tokenizer routes transferable motion features to a shared codebook and embodiment-dependent features to per-domain codebooks, keeping kinematic dimensions out of the shared representation.The shared group captures wrist or end-effector position; the specific group retains rotation and finger-joint or gripper features.Source: A. Stage1: Training Factorized Action… · p. 3
  • The action dimensions are partitioned by semantic role into a shareable positional group and a specific group containing the remaining embodiment-dependent dimensions.The groups are extracted from each action chunk and passed through separate encoder–quantizer–decoder pipelines.Source: A. Stage1: Training Factorized Action… · p. 3
  • Human and robot action signals are expressed in the egocentric camera frame, and the tokenizer produces one discrete token per timestep in each chunk.Human inputs are MANO hand poses; robot inputs are end-effector poses, including position, rotation, and gripper state.Source: A. Stage1: Training Factorized Action… · p. 3
  • Two complementary cross-domain losses align the shared codebook: InfoNCE groups human and robot latents assigned the same code, while language-guided distillation aligns semantic similarities despite differing trajectories.The text similarity matrix is used only as a distributional shaping signal for the shared latent space and is discarded after tokenizer training.Source: A. Stage1: Training Factorized Action… · pp. 3–4
  • The jointly trained tokenizer combines per-domain reconstruction with cross-domain alignment, then freezes its shared and domain-specific codebooks for later stages.Its objective also includes masked-latent consistency on the specific latent and a soft-entropy diversity regularizer for codebook utilization.Source: A. Stage1: Training Factorized Action… · p. 4

B. Stage2: Cross-Embodiment VLM Pretraining

Stage 2 transfers Stage 1 action knowledge into the VLM by training it to predict discrete action codes from image and language context. Human examples supervise shared codes, while robot examples supervise shared and robot-specific codes.

B. Stage2: Cross-Embodiment VLM Pretraining

  • The VLM learns to map an image and language instruction to the discrete action-code indices produced by the frozen Stage 1 tokenizer.The VLM is Florence-2-large, with its vocabulary extended for shared-code entries and action placeholders appended to the prompt.Source: B. Stage2: Cross-Embodiment VLM… · p. 4
  • Human-data steps train the VLM to predict shared codes, while robot-data steps train it to predict both shared and robot-specific codes.Training interleaves egocentric videos paired with hand-pose actions and robot manipulation demonstrations at a configurable ratio.Source: B. Stage2: Cross-Embodiment VLM… · p. 4
  • The action expert is excluded from Stage 2, which focuses exclusively on shaping the VLM’s internal representations.Source: B. Stage2: Cross-Embodiment VLM… · p. 4

C. Stage3: VLA Fine-tuning

Stage 3 converts the pretrained VLM into a robot-action policy by conditioning an action transformer on VLM features and fine-tuning it with robot data. At deployment, the model outputs continuous actions without the tokenizer or language model.

C. Stage3: VLA Fine-tuning

  • The action transformer decodes continuous robot actions from noisy action tokens, conditioned on VLM hidden states and Bridge Features from action-placeholder positions.Training uses continuous action regression; the discrete tokenizer and latent-action code supervision are removed.Source: C. Stage3: VLA Fine-tuning · p. 4
  • Stage 3 loads the Stage 2 VLM checkpoint, attaches an action transformer, and fine-tunes the components end to end on robot-specific actions.The VLM supplies context while the action expert learns to decode its representations into continuous robot actions.Source: C. Stage3: VLA Fine-tuning · p. 4
  • At inference, the VLM conditions action generation from an image and instruction, while the action transformer decodes continuous actions from sampled noise.The tokenizer, codebook, and language model are absent at deployment; discrete action tokens are a training-time mechanism for Stages 1–2.Source: C. Stage3: VLA Fine-tuning · p. 5

A. Implementation Details

Implementation uses a three-stage training schedule, with tokenizer training, shared-action-token VLM pretraining, and robot-data post-training. The stages use a common batch size, learning rate, and hardware setup.

A. Implementation Details

  • Stage 1 trains the factorized tokenizer for 50k steps with shared and per-domain codebooks of size 64 and quantization dimension 32.Source: A. Implementation Details · p. 5
  • Stage 2 trains the Florence-2-large backbone for 30k iterations to predict shared action tokens from visual and language inputs, excluding large-scale cross-embodiment robot data.Stages 2 and 3 use the X-VLA framework.Source: A. Implementation Details · p. 5
  • Stage 3 randomly initializes the action expert and post-trains it on robot data for 60k iterations, inheriting Stage 2 representations.All stages use batch size 32, learning rate 6e−4, 20k warmup steps, behavior cloning with ℓ1 loss, and eight NVIDIA A800 GPUs.Source: A. Implementation Details · p. 5

B. Benchmark and Metrics

The evaluation spans simulated gripper and dexterous-hand manipulation, real-world bimanual manipulation, and analysis of the tokenizer’s shared and embodiment-specific latent spaces. The tokenizer analysis examines cross-domain alignment, domain separation, and codebook usage.

B. Benchmark and Metrics

  • RoboTwin 2.0 evaluates 50 bimanual tabletop tasks under easy and randomized hard splits, while RoboCasa GR1 covers 24 dexterous-hand household tasks with articulated objects and multi-stage goals.RoboTwin reports average success across both splits; each task is tested for 50 episodes per split.Source: B. Benchmark and Metrics · p. 5
  • Real-world evaluation on an AgileX Piper bimanual platform tests rubbish sweeping, using 2,450 teleoperated real-world demonstrations and rolling out the policy for 10 physical-platform deployment trials.The Stage 3 action expert is trained for 30k steps with a batch size of 32, warm-started from the Stage 2 VLM checkpoint; success rate is reported.Source: B. Benchmark and Metrics · p. 5
  • EXEbase uses the same X-VLA framework and Florence-2-large backbone but omits Stages 1–2 pretraining, so its comparison with EXE isolates those stages’ benefit.Source: B. Benchmark and Metrics · p. 5

C. Tokenizer Analysis

  • Shared-code tokens from human and robot actions overlap in compact clusters, while independently trained specific-group codebooks form distinct domain clusters.The visualization uses jointly embedded shared-code features; the specific group captures domain-private features such as rotation, gripper, and hand joints.Source: C. Tokenizer Analysis · p. 5
  • A shared code activated in both domains corresponds to similar top-to-bottom placing trajectories, linking shared-token clusters to a common spatial intent.The trajectory projections compare a robot chunk and a human chunk quantized to the same shared code.Source: C. Tokenizer Analysis · p. 5, Figure 3 · p. 6
  • Both codebooks activate nearly all codes, and nearly all co-activated shared codes are used by both human and robot domains.Table I defines CO as codes co-activated in both domains and distinguishes human, RoboTwin 2.0, and RoboCasa GR1 domains.Source: C. Tokenizer Analysis · p. 5, Table I · p. 5

D. Main Results

On RoboTwin 2.0, EXE with EgoLive pretraining improves over EXEbase and outperforms the listed VLA baselines. The results support gains on bimanual tabletop manipulation from egocentric human-video pretraining.

1) Results on RoboTwin 2.0:

2) Results on RoboCasa GR1:

EXE improves success on RoboCasa GR1 and on two real-world evaluation groups on the AgileX Piper platform. The real-world comparisons include both familiar objects and novel objects used as distractors.

3) Results in the Real World:

  • 30% success for EXE versus 10% for EXEbase when unseen objects were placed as distractors.The authors characterize this as stronger scene-level generalization under visual distractions from unseen objects.Source: 3) Results in the Real World · p. 6

2) Results on RoboCasa GR1:

3) Results in the Real World:

  • 80% success for EXE versus 60% for EXEbase on in-distribution real-world objects and positions.The gain was most pronounced on visually colorful objects; the Stage 3 expert was fine-tuned on 2,450 teleoperated demonstrations. Figure 4 shows the AgileX Piper bimanual platform.Source: 3) Results in the Real World · p. 6, Figure 4 · p. 7

E. Ablation Study

The ablation study reports that the full EXE pipeline reaches 76.2% on RoboTwin, with alignment losses, the Bridge Feature, co-training data balance, and human-data source all examined. The conclusion frames EXE as transferring manipulation semantics through shared positional and embodiment-specific action subspaces.

E. Ablation Study

  • 73.8% with shared-code InfoNCE and 74.1% after adding language-guided relational distillation in Stage 1 tokenizer training.The language embedding guides semantically equivalent actions to nearby codes even when their spatial trajectories differ.Source: E. Ablation Study · p. 7
  • A balanced 1:1 human-to-robot co-training ratio achieves 76.2%, exceeding the 74.3% at 2:1 and 75.5% at 3:1 ratios.The authors suggest that overweighting robot data biases the backbone toward the robot domain and erodes the transferable shared representation.Source: E. Ablation Study · p. 7
  • 76.2% for full EXE versus the 73.2% EXEbase result on RoboTwin; removing any component degrades performance.The Bridge Feature adds shared action-token features from Stage 2 co-training to Stage 3 fine-tuning.Source: Table V · p. 7, E. Ablation Study · p. 7
  • VITRA-1M, EgoDex, and EgoLive each improve over EXEbase on RoboTwin 2.0; EgoLive has the best average success rate at 76.2%.The authors suggest EgoLive’s naturalistic hand-manipulation scenarios provide spatial-intent coverage aligned with the benchmark.Source: Table VI · p. 7, E. Ablation Study · p. 7

CONCLUSION

  • EXE combines contrastive learning and language-guided relational distillation to align actions sharing spatial similarity and task semantics.Its factorized representation separates shared positional information from embodiment-specific kinematics, supporting transfer to grippers and dexterous hands.Source: CONCLUSION · p. 7
Loading 2610.33685v1…