PaperlayerView exact revision on arXiv

Source-linked AI summary

One Policy, Many Embodiments: Unified Camera-Centric Action Geometry Pre-training for Heterogeneous Embodied Manipulation

Xiaomi Embodied Intelligence Team, University of Macau, :, Shaoqing Xu, Fang Li, Guozhi Zhan, Zhixiang Duan, Yuhan Wang, Yuechen Luo, Shengyin Jiang, Hanbing Li, Zhiying Du, Longlong Wang, Longmei Jiang, Weixiang Liang, Ying Gong, Yong Pan, Ziping Zhao, Zhiyuan Chen, Yangwei You, Kun Ma, Qinyuan Liu, Hangjun Ye, Zhi-xin Yang

arXiv:2608.26058v1cs.RO

TL;DR

Embodied datasets vary in morphology, camera geometry, and action spaces, so a single low-level command parameterization cannot serve as their shared target. UCAG-P shares camera-observable manipulation geometry across embodiments and supports unified manipulation and transfer across varied evaluations.

  • Problem

    A single low-level command parameterization cannot serve as the shared prediction target across datasets with differing embodiments and action spaces.

    Source: §1 · p. 3, §4.1 · p. 8

  • Method

    UCAG-P uses camera-observable anchor motion as a shared policy target and translates geometric predictions into executable commands for each target embodiment.

    Source: §1 · p. 4, §4.1 · p. 9, §4.2 · p. 9

  • Results

    Across single-arm, dual-arm, humanoid, out-of-distribution, and real-world evaluations, UCAG-P supports unified manipulation, cross-embodiment and human-to-robot transfer, and data-efficient adaptation.

    Source: §6 · p. 14

  • Takeaways & Limitations

    The authors conclude camera-centric action geometry provides a practical interface for scaling reusable robot policies across heterogeneous embodiments and demonstration sources.

    Source: §6 · p. 14

    Cross-embodiment transfer remains challenging, as camera-centric motion alone does not eliminate morphology and kinematic mismatch.

    Source: §6 · p. 14

Abstract

from arXiv · show

Scaling generalist vision-language-action (VLA) policies is severely bottlenecked by the inherent heterogeneity of embodied data, which spans diverse robot morphologies, camera configurations, and low-level action spaces. Existing paradigms typically address this mismatch through explicit action retargeting, human-to-robot video synthesis, or dataset-specific adaptation branches, fundamentally hindering the joint learning of a unified policy. We introduce UCAG-P, a camera-centric unified action formulation that structurally aligns heterogeneous embodied datasets into a shared geometric action space. Rather than treating robot-specific commands as the shared policy target, UCAG-P represents manipulation through camera-observable anchor motion in image and camera-frame coordinates, treating robot arms, humanoids, and human hands as different embodiments of a common action schema. A geometry-conditioned action translator combines predicted motion with target-embodiment kinematics to produce executable controls. The resulting decoupled architecture allows a shared VLA policy to learn transferable manipulation geometry while retaining embodiment-specific controllability. UCAG-P is trained on 4.03K hours of robot and simulation data and 2.34K hours of human demonstrations. A single checkpoint reaches 98.3% on LIBERO, 88.7% and 89.2% on RoboTwin Easy and Hard, 82.0% zero-shot on LIBERO-Plus, and 62.0% on RoboCasa GR-1, without benchmark-specific fine-tuning.

1 Introduction · 1 Introduction

UCAG-P aligns heterogeneous robot and human demonstrations through camera-observable manipulation geometry, then translates shared motion predictions into executable controls for each target embodiment. This decoupled formulation enables joint pre-training without requiring human-to-robot synthesis or explicit retargeting.

1 Introduction

  • Embodied datasets vary in morphology, camera geometry, proprioception, control frequency, and action space, so the same behavior can appear as end-effector deltas, joint commands, gripper states, or hand motion.The introduction identifies this heterogeneity, alongside data scale, as a bottleneck for generalist robot learning.Source: §1 · p. 3
  • Prior approaches separate embodiment-specific policies, retain robot-specific targets, or require conversion pipelines; camera-frame end-effector actions still fail to directly capture human grasp behavior.Figure 2 contrasts these paradigms with UCAG-P’s shared anchor motion before embodiment-specific translation.Source: §1 · p. 3, Figure 2 · p. 3
  • Shared actions represent 3D camera-frame trajectories of the wrist or end effector and grasp center, which a geometry-conditioned translator maps into embodiment-specific executable controls.The translator uses camera-to-base transforms, Jacobians, and robot state.Source: Figure 3 · p. 4, §1 · p. 4
  • UCAG-P jointly pre-trains across robot trajectories, simulation data, and human-hand demonstrations using a shared camera-centric action formulation.The introduction reports training on 11 datasets, more than 6,300 hours of demonstrations, and nine embodiments.Source: §1 · p. 4
  • Human-hand motion directly supervises policy learning as a distinct embodiment, without explicit human-to-robot video synthesis or action retargeting.This lets human demonstrations join the shared action space while preserving target-platform controllability.Source: §1 · p. 4
  • UCAG-P is validated across simulated single-arm, bimanual, humanoid, out-of-distribution, cross-embodiment, and real-robot settings without dataset-specific fine-tuning.The overview also names LIBERO, LIBERO-Plus, RoboTwin, and RoboCasa GR-1 among the evaluation benchmarks.Source: §1 · p. 4, Figure 3 · p. 4

2 Related Work

Related work addresses heterogeneous embodied data through generalist VLA policies, human-demonstration learning, and varied action representations, but often retains embodiment-specific interfaces. UCAG-P instead shares camera-centric manipulation geometry and translates predictions into embodiment-specific executable commands.

Generalist VLA and Cross-Embodiment Robot Learning

  • Prior cross-embodiment VLA methods address heterogeneous data with conditioning, normalization, specialized heads, latent alignment, or canonical templates, but often retain robot-specific action interfaces [4] [19] [20] [48] [58] [59] [65].UCAG-P follows the generalist VLA direction while learning from heterogeneous robot and human demonstrations before translating predictions into executable commands.Source: §2 · p. 5

Human Demonstrations for Robot Learning

  • Human demonstrations support robot learning through visual pre-training, imitation, retargeting, reconstruction, robot-video generation, or pseudo-actions, while UCAG-P trains converted camera-centric supervision jointly with robot trajectories [1] [3] [9] [21] [35] [37] [45] [50] [68].This shared action space lets unpaired human and robot data supervise transferable manipulation geometry; both converted human actions and robot actions contribute to translation when available.Source: §2 · p. 5

Action Representations for Manipulation

  • Existing manipulation policies use tokens, action chunks, trajectories, deltas, waypoints, keypoints, and voxel actions, but UCAG-P separates shared camera-centric anchor motion from target-robot execution [4] [11] [13] [42] [51] [69] [72].The shared prediction is translated into the target robot’s native command space, preserving executable control across heterogeneous data.Source: §2 · p. 5

3 Pre-training Data

UCAG-P pre-training combines heterogeneous robot, simulation, and human data in a shared camera-centric geometric representation while preserving source-specific supervision. The section describes the corpus, its curation into a common masked training interface, and the complementary data roles across three training stages.

3 Pre-training Data

  • 1,020,672 episodes spanning 6,373.586 hours across eleven subsets form the embodied corpus, supplemented by a separate vision-language mixture.The embodied sources are real-robot demonstrations, simulated robot trajectories, and egocentric human-hand videos.Source: §3 · p. 5

3.1 Embodied Corpus Composition

  • Robot and simulation data account for 63.29% of the corpus, while 2,339.727 hours of human demonstrations account for 36.71%.Human videos lack robot commands; wrist and grasp-center motion is converted into camera-centric pseudo-actions that share the geometric action space with robot trajectories.Source: §3.1 · pp. 6–7

3.2 Data Curation and Preprocessing

  • A four-stage curation pipeline indexes domains, standardizes visual and language inputs, derives geometric and control targets, and builds fixed-horizon samples with validity masks.Domain identities and valid-dimension indicators are retained so unavailable labels and padded dimensions do not become supervision.Source: §3.2 · p. 7
  • RGB inputs are standardized to 224 × 224 by default, and trajectories are windowed at horizon H = 30 by default with domain-specific stride and temporal-validity masks.Different camera counts are packed into fixed view slots with missing views zero-filled and disabled; short terminal segments are padded and excluded from loss.Source: §3.2 · p. 7
  • Robot anchors p0 and p1 are recovered in camera coordinates from available pose, gripper, calibration, and kinematic signals, while human wrist and fingertip anchors leave executable-command targets invalid.Single-arm and bimanual samples share a layout, with inactive arms and unavailable coordinates masked.Source: §3.2 · p. 7

3.3 Supervision and Training Roles

  • Vision-language and human data establish shared visual-geometric prediction in Stage 1, simulation and robot data additionally supervise action translation in Stage 2, and Stage 3 jointly optimizes motion and robot control.Vision-language samples supply instruction, spatial and affordance grounding, trajectory understanding, and text/JSON coordinates; missing labels are masked, and their scale is not measured in hours.Source: §3.3 · p. 8

4 Method

UCAG-P replaces incompatible native action targets with shared camera-observable geometric motion, then translates that motion into embodiment-specific controls. Its architecture, staged training, and masked objectives support learning from heterogeneous annotations while preserving executable robot control.

4.1 Problem Formulation

  • UCAG-P separates shared prediction of camera-observable geometric action chunks from geometry-conditioned translation into each target embodiment’s native executable controls.This avoids requiring all datasets to share one low-level command parameterization, despite annotations ranging from end-effector and joint commands to humanoid and human-hand motion.Source: §4.1 · pp. 8–9

4.2 Camera-Centric Action Space

  • The shared action space tracks a semantic wrist/end-effector and gripper/grasp-center anchor pair in camera-frame coordinates across robot arms, humanoids, and human demonstrations.For human hands, the grasp center is the midpoint of the thumb and index fingertips; unavailable metric-depth or calibration components are masked rather than treated as valid targets.Source: §4.2 · p. 9

4.3 UCAG-P Architecture

  • A Qwen3-VL-4B-Instruct [2] backbone and shared motion head feed a geometry-conditioned translator that uses camera-to-base transforms, local Jacobians, and embodiment tokens to emit executable commands.The translator outputs a sparse command layout, passing only slots active for the target embodiment to its downstream controller.Source: §4.3 · pp. 10–11

4.4 Three-Stage Training

  • Training proceeds from camera-centric motion specialization, to translation trained on ground-truth trajectories, then joint robot-human optimization using policy-predicted trajectories.The second stage isolates geometry-to-command learning from motion-prediction errors; the joint stage excludes auxiliary general VLM data and uses pooled VLM context for translation.Source: §4.4 · p. 11

4.5 Training objectives and Inference

  • Masked L1 objectives supervise available geometric and executable-command targets, and inference predicts camera-centric motion before translating it into active target-embodiment command slots.Only active command dimensions contribute to translator loss, and the downstream controller executes only the target embodiment’s active slots.Source: §4.5 · p. 11

5 Experiments

Experiments test whether a single UCAG-P checkpoint can work across heterogeneous embodiments and control interfaces, withstand distribution shift, and transfer to new embodiments and physical robots. The evaluation spans simulation and real-world manipulation without benchmark-specific fine-tuning.

5.1 Experimental Setup

  • A single final checkpoint is evaluated across simulation benchmarks without benchmark-specific fine-tuning, using the Qwen3-VL-4B-Instruct backbone [2].The checkpoint follows the three-stage training procedure; the pre-training corpus combines real-robot demonstrations, simulated trajectories, egocentric human-hand videos, and vision-language supervision.Source: §5.1 · p. 12

5.2 Experimental Results

  • 98.3% on LIBERO, 88.66% on RoboTwin Easy, 89.20% on RoboTwin Hard, and 62.0% on RoboCasa GR-1 show unified cross-embodiment performance.The same checkpoint is used across the reported benchmarks; UCAG-P exceeds Qwen-VLA across evaluated datasets, while benchmark specialists achieve slightly higher peak metrics. On RoboCasa GR-1, UCAG-P is below ZR-0 (69.3%) and JoyAI-RA (63.2%), but above earlier baselines included in the table [22, 70].Source: Table 3 · p. 12, §5.2 · p. 12
  • 82.0% is UCAG-P’s zero-shot total score on LIBERO-Plus across seven perturbation categories.Robot-state, lighting, and background perturbations are particularly robust, while camera, sensor-noise, and object-layout shifts remain more challenging; no LIBERO post-training is used.Source: Table 4 · p. 13, §5.2 · p. 13
  • 60%, 90%, and 75% success on bread grasping, drawer opening, and bowl stacking compare with π0.5’s 20%, 85%, and 65%, respectively.With the same demonstration budget, observation setup, and controller, UCAG-P initialization improves real-robot adaptation; direct ALOHA-to-ARX replacement yields 35.0% zero-shot success, below source-embodiment performance.Source: Figure 7 · p. 13, §5.2 · p. 13
  • Figure 8 qualitatively shows task-relevant geometry maintained across real-world and simulation executions despite changes in appearance, morphology, and control interface.Bimanual and humanoid examples illustrate the geometry-conditioned translator converting shared motion predictions into coordinated embodiment-specific commands.Source: Figure 8 · p. 14, §5.2 · p. 14

6 Conclusion and Limitations … A.4 Question-Answer Training Examples

UCAG-P aligns heterogeneous embodied data through camera-observable anchor motion, then translates shared geometric predictions into embodiment-specific commands. The supplementary material details its limitations, implementation, sparse control interfaces, and training examples.

6 Conclusion and Limitations

  • UCAG-P decouples transferable task geometry from robot-specific control by predicting camera-observable anchor motion and translating it into embodiment-specific commands.The same formulation supports unified manipulation and transfer across embodiments and demonstration sources, including human-to-robot transfer and data-efficient adaptation.Source: §6 · p. 14
  • UCAG-P depends on reliable geometry estimation, and controlled, small-scale real-world tests leave broader robustness across robots, cameras, objects, and long-horizon tasks unestablished.Calibration, depth, kinematics, or hand-keypoint errors can propagate; non-zero ALOHA-to-ARX transfer still falls short of the source embodiment.Source: §6 · p. 14

Appendix

  • The appendix covers architecture, evaluation, qualitative analysis, transfer results, pooling ablations, benchmark breakdowns, human-to-robot visualizations, and failure cases.Its training and evaluation material includes the schedule, head architectures, sparse command layout, protocols, and question–answer examples.Source: Appendix · p. 21, A Training & Evaluation Details · p. 21

A Training & Evaluation Details

  • The training and evaluation details omitted from the main paper are organized around optimization, control layout, and evaluation protocols.These supplementary details include the training schedule, head architectures, sparse command layout, and question–answer examples.Source: A Training & Evaluation Details · p. 21, Appendix · p. 21

A.1 Training Schedule and Optimization

  • Training uses 224 × 224 images and 30-step action chunks in Stages 1 and 3, while Stage 2 trains the translator from trajectory, state, and calibration inputs without images.The schedule uses AdamW, cosine decay, 1,000 warmup steps, and learning rates of 1 × 10^-5 for VLM components and 1 × 10^-4 for the action model.Source: §A.1 · p. 21, Table A.1 · p. 21

A.2 Motion and Action Head Architectures

  • The motion head predicts camera-centric motion, and a geometry-conditioned action head combines that prediction with embodiment geometry to generate executable commands.The motion head maps 2,560-dimensional token features through two residual MLP blocks to 30-dimensional vectors; the action head uses pooled VLM context, camera pose, Jacobian, and motion to produce a 30-step, 80-dimensional qpos chunk.Source: §A.2 · pp. 21–22, Figure A.1 · p. 21, Figure A.2 · p. 22

A.3 Sparse Command Layout

  • Fixed-width motion and action tensors let embodiments share batches without conflating physical semantics, while validity masks exclude unavailable dimensions from loss and control.Inactive blocks and padded dimensions are masked, so only commands supported by the target embodiment are dispatched at inference.Source: §A.3 · pp. 22–23

A.4 Question-Answer Training Examples

  • Three-stage question–answer training covers affordance, anchor, and trajectory prediction; motion-to-action translation; and joint motion and command prediction with source-specific masks.Semantic answers use concise language, while geometric and control targets use structured numerical outputs, including p0 and p1 coordinates.Source: §A.4 · p. 24, Figure A.3 · p. 24

A.5 Evaluation Protocols · B More Experimental Results · B.1 Transfer to a New Embodiment

Evaluation protocols control checkpoint selection, task definitions, and execution conditions to isolate embodiment and distribution shifts. In the ARX transfer setting, UCAG-P achieves 35.0% success under a controlled replacement of the source ALOHA embodiment.

A.5 Evaluation Protocols

  • Evaluation protocols isolate embodiment and data-distribution changes while controlling checkpoint selection, task definitions, and execution conditions.For ARX transfer, initial end-effector poses are aligned by inverse kinematics.Source: §A.5 · p. 24
  • Simulation benchmarks use the same final checkpoint without benchmark-specific fine-tuning across LIBERO, LIBERO-Plus, RoboTwin 2.0, and RoboCasa GR-1.RoboTwin reports Easy and Hard results on ALOHA; ARX transfer fixes scenes, layouts, perturbation seeds, and camera extrinsics, while RoboCasa GR-1 tests executable humanoid control.Source: §A.5 · p. 24
  • Real-world Piper evaluations use 100 demonstrations per task and 20 closed-loop trials, with budgets, optimization, observations, controller, and success criteria matched across comparisons.Bread pickup tests human-to-robot transfer using MediaPipe hand keypoints converted to camera-centric anchors; drawer opening and bowl stacking test SFT adaptation from robot demonstrations.Source: §A.5 · p. 24
  • Closed-loop task success is primary; LIBERO, RoboTwin, and RoboCasa GR-1 report average success, while LIBERO-Plus reports the mean across seven perturbation categories.Real-world comparisons evaluate π0 [4] and π0.5 [43] on the same Piper platform under matched deployment conditions; open-loop prediction errors are supporting diagnostics only.Source: §A.5 · p. 24

B More Experimental Results

B.1 Transfer to a New Embodiment

  • 35.0% success on ARX under UCAG-P’s controlled ALOHA-to-ARX replacement, with scenes, layouts, seeds, and camera extrinsics fixed and initial poses inverse-kinematics-aligned.The result is reported in Table B.1.Source: §B.1 · p. 25, Table B.1 · p. 25

B.2 Ablation on Pooled VLM Features · B.3 Full LIBERO Suite Results

UCAG-P averages 98.25% across the four LIBERO suites, with consistent suite scores and a remaining gap in Long-Horizon tasks. Adding pooled VLM features to the action head improves RoboCasa GR-1 success from 58.3% to 62.0%.

B.2 Ablation on Pooled VLM Features

  • The full action head uses pooled VLM hidden states in addition to motion and geometric inputs; the baseline excludes VLM features.Source: §B.2 · p. 25
  • Adding pooled VLM features raises RoboCasa GR-1 success from 58.3% to 62.0%, a gain of 3.7 percentage points.The passage attributes the improvement to complementary visual and language information for downstream action prediction.Source: §B.2 · p. 25, Table B.2 · p. 25

B.3 Full LIBERO Suite Results

  • UCAG-P’s average across the four LIBERO suites is 98.25%, only 0.35 percentage points below the highest reported average.The average is the arithmetic mean of the Spatial, Object, Goal, and Long-Horizon scores.Source: §B.3 · p. 25
  • UCAG-P achieves the best Goal score on the standard LIBERO suites at 99.2%.Source: §B.3 · p. 25
  • UCAG-P’s scores across the four LIBERO suites remain within a 2.8-point range.The passage describes this as consistent performance across spatial reasoning, object interaction, goal following, and long-horizon execution.Source: §B.3 · p. 25
  • UCAG-P reaches 96.4% on Long-Horizon tasks, compared with the best result of 97.6%.Source: §B.3 · p. 25

B.4 Full RoboCasa GR-1 Task-Level Results

UCAG-P averages 62.0% success across 24 RoboCasa GR-1 tasks, ranking second behind JoyAI-RA; it leads on six individual tasks, while task variation leaves room to improve execution.

B.4 Full RoboCasa GR-1 Task-Level Results

  • 62.0% average success across 24 RoboCasa GR-1 tasks ranks UCAG-P second, 1.2 points behind JoyAI-RA and ahead of ABot-M0, VP-VLA, and GR00T-N1.6 by 3.7, 8.2, and 14.4 points.The task-level results are reported in Table B.4.Source: §B.4 · p. 25, Table B.4 · p. 26
  • UCAG-P achieves the highest task-level success on six tasks, including cup-to-drawer, cuttingboard-to-cardboardbox, and several plate-placement tasks.Source: §B.4 · p. 25
  • The results indicate transfer of the shared representation to humanoid control, while task-to-task variation leaves room for stronger contact-rich and articulated-object execution.Source: §B.4 · p. 25

B.5 Full RoboTwin Task-Level Results … C.3 Qualitative Manipulation Examples

UCAG-P achieves strong RoboTwin performance across Clean and Randomized settings, while OpenMicrowave remains a task-specific weakness. The accompanying visualizations show how human demonstrations provide camera-centric motion supervision and how the representation is used across simulated and real-world manipulation examples.

B.5 Full RoboTwin Task-Level Results

  • 11%/13% on OpenMicrowave identifies articulated opening as a task-specific limitation despite UCAG-P’s strong aggregate RoboTwin performance.The values correspond to Clean/Randomized settings.Source: §B.5 · p. 27
  • 88.66% Clean and 89.20% Randomized; the latter is the best randomized average, 1.22 points above ZR-0 and 0.54 above UCAG-P Clean.Each task is evaluated over 100 episodes using success rate (SR, %); the small cross-setting variation indicates robustness to randomized scenes.Source: Table B.5 · p. 27, §B.5 · p. 27

C More Visualizations

  • Section C provides qualitative manipulation examples and visualizes converting human-hand observations into the shared camera-centric anchor representation.Source: C More Visualizations · p. 28

C.1 Human-to-Robot Conversion Details

  • For bread pickup, MediaPipe maps the wrist to p0 and thumb–index midpoint to p1; camera-frame anchors supervise shared motion while unavailable channels and executable robot-action slots are masked from action loss.This conversion preserves task-relevant approach and grasp geometry without requiring robot-specific kinematics.Source: §C.1 · p. 28, Figure C.1 · p. 28

C.2 Human Demonstration Example

  • The bread-pickup example shows video collection, hand-keypoint detection, and p0/p1 estimation provide geometric robot-learning supervision without expressing human actions in robot joint coordinates.The illustrated conversion uses MediaPipe hand keypoints [29].Source: §C.2 · p. 29, Figure C.2 · p. 29

C.3 Qualitative Manipulation Examples

  • Paired examples span simulated benchmarks and real-world Piper tasks, showing reuse of the camera-centric representation where sensing and contact add variation.The simulated examples include LIBERO, RoboTwin, and RoboCasa GR-1.Source: §C.3 · p. 29, Figure C.3 · p. 29

D Failure cases and Limitation Analysis

UCAG-P’s limitations stem from errors in estimated geometry, remaining cross-embodiment mismatch, and a controlled, limited-scale real-world evaluation. Figure D.1 illustrates failures in anchor detection, trajectory prediction, and action translation.

Failure cases and limitations

  • Errors in camera calibration, depth, embodiment kinematics, or MediaPipe hand-keypoint localization can propagate through the camera-centric target and translator.Figure D.1 shows representative failures involving anchor detection, trajectory prediction, and action translation.Source: D Failure cases and Limitation Analysis · p. 30, Figure D.1 · p. 30
  • Although UCAG-P achieves non-zero ALOHA-to-ARX transfer, the gap from the source embodiment shows that camera-centric motion does not eliminate morphology and kinematic mismatch.Source: D Failure cases and Limitation Analysis · p. 30
  • Real-world evaluation is controlled and limited in scale, supporting human-to-robot transfer and data-efficient SFT adaptation but leaving broader robustness untested.The paper identifies evaluation across more robots, camera setups, object categories, and long-horizon tasks as needed.Source: D Failure cases and Limitation Analysis · p. 30

Related papers

Loading 2608.26058v1…