Source-linked AI summary
One Policy, Many Embodiments: Unified Camera-Centric Action Geometry Pre-training for Heterogeneous Embodied Manipulation
Xiaomi Embodied Intelligence Team, University of Macau, :, Shaoqing Xu, Fang Li, Guozhi Zhan, Zhixiang Duan, Yuhan Wang, Yuechen Luo, Shengyin Jiang, Hanbing Li, Zhiying Du, Longlong Wang, Longmei Jiang, Weixiang Liang, Ying Gong, Yong Pan, Ziping Zhao, Zhiyuan Chen, Yangwei You, Kun Ma, Qinyuan Liu, Hangjun Ye, Zhi-xin Yang
TL;DR
Generalist VLA learning is hindered by heterogeneous embodiments and incompatible action spaces. UCAG-P shares camera-centric anchor geometry across robot and human demonstrations, then translates predictions into embodiment-specific controls. A single checkpoint performs across multiple benchmarks without benchmark-specific fine-tuning, while remaining sensitive to calibration, estimation, and morphology mismatches.
Problem
Heterogeneous embodied datasets differ in morphology, camera geometry, proprioception, control frequency, and low-level action spaces, limiting shared action supervision.
Method
UCAG-P represents manipulation with camera-observable wrist, end-effector, and grasp-center motion, then translates shared geometric predictions into embodiment-specific executable commands.
Results
UCAG-P validates one formulation across single-arm, dual-arm, humanoid, out-of-distribution, and real-world evaluations without dataset-specific fine-tuning.
Takeaways & Limitations
Camera-centric action geometry provides a practical interface for scaling reusable robot policies across heterogeneous embodiments and demonstration sources.
Takeaways & Limitations
UCAG-P depends on reliable camera calibration, depth, embodiment kinematics, and hand-keypoint localization, while morphology and kinematic mismatch still challenge transfer.
Abstract
from arXiv · showhide
Scaling generalist vision-language-action (VLA) policies is severely bottlenecked by the inherent heterogeneity of embodied data, which spans diverse robot morphologies, camera configurations, and low-level action spaces. Existing paradigms typically address this mismatch through explicit action retargeting, human-to-robot video synthesis, or dataset-specific adaptation branches, fundamentally hindering the joint learning of a unified policy. We introduce UCAG-P, a camera-centric unified action formulation that structurally aligns heterogeneous embodied datasets into a shared geometric action space. Rather than treating robot-specific commands as the shared policy target, UCAG-P represents manipulation through camera-observable anchor motion in image and camera-frame coordinates, treating robot arms, humanoids, and human hands as different embodiments of a common action schema. A geometry-conditioned action translator combines predicted motion with target-embodiment kinematics to produce executable controls. The resulting decoupled architecture allows a shared VLA policy to learn transferable manipulation geometry while retaining embodiment-specific controllability. UCAG-P is trained on 4.03K hours of robot and simulation data and 2.34K hours of human demonstrations. A single checkpoint reaches 98.3% on LIBERO, 88.7% and 89.2% on RoboTwin Easy and Hard, 82.0% zero-shot on LIBERO-Plus, and 62.0% on RoboCasa GR-1, without benchmark-specific fine-tuning.
1 Introduction
UCAG-P addresses heterogeneous manipulation data by sharing camera-observable anchor motion rather than embodiment-specific controls. Its decoupled translator converts this shared geometry into executable commands while enabling joint robot–human training and benchmark-general evaluation.
- Heterogeneous datasets differ in morphology, camera geometry, proprioception, control frequency, and low-level action space, making shared action supervision difficult.
- Existing approaches isolate embodiments, retain robot-specific targets, or require retargeting and conversion assumptions for human demonstrations.
- UCAG-P shares camera-observable wrist, end-effector, and grasp-center motion, treating robot arms, humanoids, and human hands as embodiments of a common action schema.
- A decoupled embodiment-specific translator maps shared geometric actions into executable robot controls while preserving target-platform controllability.
- UCAG-P is validated across simulated, cross-embodiment, out-of-distribution, and real-robot settings without dataset-specific fine-tuning.
2 Related Work
Related work uses generalist VLA scaling, human demonstrations, and diverse manipulation action representations, but these approaches commonly remain tied to embodiment-specific execution interfaces.
- VLA policies connect visual observations and language instructions to action prediction, with progress driven by larger datasets, stronger multimodal backbones, and scalable action generation.
- Human videos provide scalable object-centric manipulation observations but usually lack robot-native executable labels.
- Prior methods make human behavior usable through visual pre-training, imitation learning, retargeting, trajectory reconstruction, video generation, or pseudo-action extraction.
- Manipulation policies use discrete tokens, action chunks, flow or diffusion trajectories, end-effector deltas, waypoints, keypoints, and voxel actions.
- These action representations are generally effective within intended interfaces but remain tied to executable control spaces.
3 Pre-training Data
UCAG-P combines real-robot, simulated, and egocentric human data into a common multimodal training interface while preserving source-specific valid supervision. Its staged pipeline standardizes observations, derives camera-centric geometry, and supports both shared prediction and executable-control learning.
- 1,020,672 episodes and 6,373.586 hours form an embodied corpus spanning eleven dataset subsets and single-arm, dual-arm, humanoid, and human-hand embodiments.
- 3,767.511 hours of simulated trajectories and 266.348 hours of real-robot demonstrations provide robot observations and executable commands.
- 2,339.727 hours of egocentric human-hand video contain no robot commands and contribute camera-centric pseudo-actions from wrist and grasp-center anchors.
- The preprocessing pipeline indexes episodes, standardizes visual and language observations, derives geometric and executable-control targets, and creates fixed-horizon samples with validity masks.
- Vision-language and human data supervise shared visual-geometric prediction, while simulation and real-robot data additionally supervise action translation and executable control.
4 Method
UCAG-P separates shared camera-centric geometric prediction from embodiment-specific execution, enabling heterogeneous robot and human demonstrations to use a common action schema. It represents manipulation with semantic anchor motion and translates predictions into target-specific commands using geometry and kinematics.
- Problem formulation: UCAG-P separates shared geometric prediction from embodiment-specific execution, avoiding a single low-level command parameterization across incompatible action spaces.The base policy predicts a shared camera-centric action chunk, while a translator produces commands in the target embodiment’s native control space.
- Camera-centric action space: The shared action space represents manipulation with a semantic anchor pair: p0 is the wrist or end-effector, and p1 is the gripper or grasp center.Robot anchors come from simulator state or forward kinematics and gripper geometry; human anchors use wrist and hand keypoints.
- Camera-centric action space: Each action step describes anchor motion in 3D camera-frame coordinates over horizon H, optionally including camera motion and gripper or hand state.Displacements are measured relative to the first frame, while camera-motion terms are zero for fixed-camera datasets; unavailable quantities are masked.
- Training framework: UCAG-P uses a shared motion head and geometry-conditioned translator within a unified VLA policy trained across single-arm, bimanual, humanoid, and human-hand data.Its three training stages specialize camera-centric prediction, train translation separately, and then optimize both components jointly.
- UCAG-P architecture: The translator combines predicted camera-centric motion with target state, camera-to-base geometry, local Jacobians, and embodiment tokens to generate executable commands.The camera-to-base transform aligns camera-frame motion with the robot body frame, while the Jacobian relates joint changes to end-effector motion.
5 Experiments
UCAG-P is evaluated as a single unified checkpoint across heterogeneous embodiments, distribution shifts, simulation benchmarks, and real-world manipulation tasks. It remains competitive across settings and supports transfer without benchmark-specific fine-tuning.
- Evaluation scope: A single unified checkpoint evaluates heterogeneous simulation datasets and real-world manipulation tasks without benchmark-specific fine-tuning.The evaluation covers single-arm, bimanual, humanoid, out-of-distribution, cross-embodiment, and real-world settings.
- Unified manipulation: 98.3% on LIBERO and 88.66% Easy and 89.20% Hard on RoboTwin demonstrate competitive unified manipulation across single-arm, bimanual, and humanoid settings.On RoboCasa GR-1, UCAG-P reaches 62.0%, below stronger specialist or generalist methods but above earlier baselines.
- Distribution shift: 82.0% zero-shot on LIBERO-Plus shows robustness across seven perturbation categories, with robot-state, lighting, and background shifts stronger than camera, sensor-noise, and object-layout shifts.The latter perturbations remain more challenging.
- Real-world transfer: 60%, 90%, and 75% success on bread grasping, drawer opening, and bowl stacking compares with 20%, 85%, and 65% for π0.5.The controlled Piper comparisons use matched demonstration budgets, observation setups, and controllers.
- Qualitative analysis: Successful real-world and simulation executions preserve task-relevant object–effector relationships despite changes in scene appearance, robot morphology, and control interface.The qualitative examples include bimanual and humanoid tasks.
6 Conclusion and Limitations
UCAG-P introduces a unified camera-centric formulation that separates transferable manipulation geometry from embodiment-specific control. Across diverse evaluations, the same formulation supports unified manipulation and transfer, while reliable geometric estimation and morphology differences remain limitations.
- Conclusion: UCAG-P represents manipulation through camera-observable anchor motion and translates shared geometric predictions into embodiment-specific commands.This decouples transferable task geometry from robot-specific control interfaces.
- Conclusion: The same formulation supports unified manipulation, cross-embodiment transfer, human-to-robot transfer, and data-efficient adaptation across simulation and real-world evaluations.The reported evaluation spans single-arm, dual-arm, humanoid, out-of-distribution, and real-world settings.
- Limitations: Reliable camera calibration, depth estimation, embodiment kinematics, and hand-keypoint localization are required because their errors can propagate into targets and downstream translation.These dependencies constrain the reliability of the geometric interface.
- Limitations: Non-zero ALOHA-to-ARX transfer still leaves a gap from the source embodiment, indicating that camera-centric motion alone does not eliminate morphology-related challenges.Cross-embodiment transfer therefore remains challenging.
7 Contributions and Acknowledgments
The supplementary material documents UCAG-P’s training schedule, prediction heads, canonical motion and action layouts, masking scheme, and question–answer supervision interface. These components organize heterogeneous samples into fixed-width representations while preserving embodiment-specific semantics.
- Supplementary scope: The supplementary material covers training schedules, head architectures, sparse command layouts, evaluation protocols, transfer results, ablations, and qualitative visualizations.It records optimization, control-layout, and evaluation details omitted from the main paper.
- Prediction heads: The motion head maps VLM action-token features to a 30-step camera-centric motion prediction, while the action head combines that motion with embodiment geometry to produce executable commands.The motion head uses two residual MLP blocks; the action head incorporates camera pose, Jacobian, motion, and VLM-conditioned features.
- Masking and layouts: Fixed-width tensors let samples from different embodiments share batches, while dimension-level validity masks exclude unsupported or unavailable channels from loss and control.Padding values are structural placeholders rather than supervised zeros.
- Motion layout: The canonical motion representation uses 10-dimensional blocks for manipulator and camera information, including anchor displacements, gripper state, rotation, and padding.Single-arm, fixed-camera, and partially annotated samples mask the corresponding unused channels.
- Action layout: The canonical 80-dimensional action layout reserves eight ordered 10-dimensional blocks whose active subsets encode arms, end effectors, hands, waist, and mobile-base commands.Embodiment-specific masks select supported blocks and exclude excess degrees of freedom from training and control.
- Training interface: Three question–answer training stages cover affordance and anchor prediction, action translation from camera-centric motion and robot context, and joint supervision of motion and executable actions.Available loss masks accommodate heterogeneous supervision.
A.5 Evaluation Protocols
The evaluation protocols control checkpoint selection, task definitions, and execution conditions while isolating embodiment and distribution changes. They combine standardized simulation benchmarks with matched real-world Piper comparisons and closed-loop success metrics.
- Protocol design: Checkpoint selection, task definitions, and execution conditions remain controlled while the protocols isolate embodiment and data-distribution changes.This establishes a common basis for the reported comparisons.
- Simulation: Simulation uses one final checkpoint without benchmark-specific fine-tuning across LIBERO, LIBERO-Plus, RoboTwin, and RoboCasa GR-1.The ARX transfer setting fixes scenes, layouts, perturbation seeds, and camera extrinsics, with initial poses aligned by inverse kinematics.
- Real-world evaluation: Real-world Piper experiments use 100 demonstrations per task and 20 closed-loop trials with matched budgets, schedules, observations, controllers, and success criteria.Bread pickup evaluates human-to-robot transfer, while drawer opening and bowl stacking evaluate robot-demonstration adaptation.
- Metrics: Closed-loop task success is the primary metric, with average success for LIBERO, RoboTwin, and RoboCasa GR-1 and seven-category means for LIBERO-Plus.Open-loop prediction errors serve only as supporting diagnostics.
- Cross-embodiment protocol: 35.0% success on ARX measures cross-embodiment transfer under a controlled replacement of ALOHA while scenes, layouts, seeds, and camera extrinsics remain fixed.Initial end-effector poses are aligned by inverse kinematics.
B.2 Ablation on Pooled VLM Features
Adding pooled VLM hidden states to the action head improves RoboCasa GR-1 performance over a baseline without VLM features. The result supports complementary visual-language context for downstream action prediction.
- The ablation compares an action head without VLM features against a full model using pooled VLM hidden states.
- 3.7 percentage points: pooled VLM features increase RoboCasa GR-1 success from 58.3% to 62.0%.The full model conditions the action head on pooled VLM hidden states alongside motion and geometric inputs.
- Pooled VLM context provides complementary visual and language information for downstream action prediction.
B.4 Full RoboCasa GR-1 Task-Level Results
UCAG-P achieves strong task-level performance on RoboCasa GR-1, ranking second across 24 tasks while exceeding three comparison methods. The task variation nevertheless leaves room for improved contact-rich and articulated-object execution.
- 62.0% average success across 24 RoboCasa GR-1 tasks ranks UCAG-P second, 1.2 points behind JoyAI-RA.
- UCAG-P exceeds ABot-M0, VP-VLA, and GR00T-N1.6 by 3.7, 8.2, and 14.4 points, respectively.
- UCAG-P attains the highest task-level success on six tasks, including cup-to-drawer and several plate-placement tasks.
- Variation across tasks leaves room for stronger contact-rich and articulated-object execution.
B.5 Full RoboTwin Task-Level Results
UCAG-P performs strongly across Clean and Randomized RoboTwin settings, with only small aggregate variation between them. OpenMicrowave remains a clear task-specific weakness despite the overall performance.
- Full RoboTwin Task-Level Results: 89.20%: UCAG-P achieves the best Randomized RoboTwin average, exceeding its 88.66% Clean score by 0.54 points.
- Full RoboTwin Task-Level Results: 88.66% on Clean is within 0.04 points of the strongest baseline.
- Full RoboTwin Task-Level Results: The Randomized result exceeds ZR-0 by 1.22 points, indicating strong robustness to randomized scene conditions.
- Full RoboTwin Task-Level Results: OpenMicrowave scores of 11% on Clean and 13% on Randomized identify articulated opening as a task-specific limitation.
- Qualitative and Human-Demonstration Examples: Human-hand observations are converted into camera-centric anchor trajectories, enabling geometric motion supervision without robot joint-coordinate labels.For bread pickup, MediaPipe keypoints map the wrist to p0 and the thumb-index midpoint to p1.
- Qualitative and Human-Demonstration Examples: Qualitative examples span simulated LIBERO, RoboTwin, and RoboCasa GR-1 tasks alongside real-world Piper robot tasks.
D Failure cases and Limitation Analysis
UCAG-P failures arise from errors in anchor detection, trajectory prediction, and action translation. Its limitations include dependence on reliable geometric estimates, incomplete cross-embodiment transfer, and controlled real-world evaluation.
- Failure Cases: Errors in anchor detection, trajectory prediction, and action translation produce representative UCAG-P failure cases.
- Limitation Analysis: Reliable camera calibration, depth estimation, embodiment kinematics, and hand-keypoint localization are required because their errors can propagate downstream.
- Limitation Analysis: Camera-centric motion does not eliminate morphology and kinematic mismatch, despite non-zero ALOHA-to-ARX transfer.
- Limitation Analysis: Real-world evaluation is controlled and limited in scale, requiring broader testing across robots, camera setups, object categories, and long-horizon tasks.