Source-linked AI summary
ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining
Hao Li, Ganlong Zhao, Yufei Liu, Haotian Hou, Guoquan Ye, Tongyan Fang, Chunxiao Liu, Siyuan Huang, Jianbo Liu, Xiaogang Wang, Hongsheng Li
TL;DR
Robot data are costly to collect, while human and robot demonstrations differ in action spaces, embodiments, timing, and supervision quality. ACE-EGO-0 unifies these sources with pseudo-action trajectories, shared action representations, and reliability-aware training, achieving strong simulation and real-world results.
Problem
Robot demonstration collection is costly, while human and robot data differ in action spaces, embodiments, temporal dynamics, and supervision quality.
Method
ACE-EGO-0 converts egocentric videos into robot-format pseudo-actions and unifies heterogeneous data through camera-space actions, morphology conditioning, time alignment, and reliability-aware training.
Results
72.8% average success on RoboCasa, 91.12%/90.62% on RoboTwin Easy/Hard, and 78.3% across six real bimanual tasks were achieved.
Takeaways & Limitations
Adding human videos produced further gains beyond robot data, while human supervision improved unified pretraining, fine-tuning, and transfer to bimanual manipulation.
Takeaways & Limitations
Evaluation focuses on tabletop manipulation, and the pretraining pool lacks dexterous-hand data and force/torque sensing.
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) models benefit from large-scale and diverse embodied data, yet scaling robot trajectory collection is costly and labor-intensive. Recent advances show that large-scale egocentric human videos provide complementary real-world supervision in pretraining. However, joint training on human and robot data remains challenging due to divergences in action spaces, embodiment structures, temporal dynamics, and supervision quality. We introduce ACE-EGO-0, a unified VLA pretraining framework jointly leveraging heterogeneous data sources. To extract large-scale pretraining supervision from egocentric human videos, we build a scalable egocentric video-to-action pipeline that converts raw human videos into robot-format pseudo-action trajectories. To make these labels comparable with robot demonstrations, ACE-EGO-0 uses a unified action representation based on camera-space actions, morphology conditioning, and time-aligned action chunking. To robustly leverage noisy pseudo-action supervision from egocentric human videos, we formulate a reliability-aware training objective with a human auxiliary loss that concentrates supervision on reliable signals. We instantiate ACE-EGO-0 on 4.53K hours of robot and simulation data, together with 1.48K hours of pseudo-action-labeled egocentric human data. Experiments show that incorporating large-scale human supervision under reliability-aware weighting consistently improves both unified joint pretraining and supervised fine-tuning. ACE-EGO-0 achieves state-of-the-art performance on RoboCasa GR1 TableTop and RoboTwin 2.0, while demonstrating strong transfer to real-world bimanual manipulation.
1 Introduction
ACE-EGO-0 is a unified VLA pretraining framework that addresses heterogeneous embodied data through shared action representations and reliability-aware supervision. It combines scalable pseudo-action-labeled egocentric human videos with robot and simulation data, achieving strong benchmark and real-world bimanual performance.
- Introduction: ACE-EGO-0 bridges spatial, structural, and temporal discrepancies across human and robot data with a unified action representation.It constructs a canonical action space using an observation-centric coordinate frame for robot end-effector and reconstructed human hand pseudo-action trajectories.
- Introduction: ACE-EGO-0 addresses supervision-quality mismatch with a reliability-aware training objective for noisy human pseudo-action supervision.The framework is designed to concentrate human supervision on reliable signals while jointly pretraining on heterogeneous embodied data.
- Introduction: 1.48K hours of pseudo-action-labeled human video are combined with 4.53K+ hours of robot demonstrations and simulation rollouts into a 6.0K+ hour pretraining dataset.The human data comes from a scalable five-stage processing pipeline applied to six diverse egocentric video datasets.
- Introduction: 72.8% average success is achieved on RoboCasa GR1 TableTop, while RoboTwin 2.0 reaches 91.12% and 90.62% average success on Easy and Hard splits.The reported evaluations cover RoboCasa and RoboTwin 2.0 benchmarks.
- Introduction: ACE-EGO-0 demonstrates strong real-world bimanual performance on long-horizon, contact-rich tasks on an ARX platform.Ablations confirm the contributions of morphology conditioning, time-aligned action chunking, and reliability-aware human supervision.
2 Related Work
Prior VLA research has progressed from task-specific imitation toward generalist models trained on diverse robot data, while addressing mismatches across datasets and embodiments. Egocentric human videos offer scalable manipulation supervision, but extracted pseudo-actions introduce distinct quality challenges because of tracking and estimation noise.
- Robot-data scaling: Generalist VLA models increasingly train on large, diverse robot datasets, connecting real-robot demonstrations with web-scale vision-language pretraining.RT-1 demonstrated large-scale real-robot training for language-conditioned manipulation, while RT-2 linked web-scale vision-language pretraining to robot action prediction.
- Representation alignment: Prior methods address cross-dataset representation mismatch through shared action formats, embodiment-aware modules, universal or latent action spaces, and spatial grounding.These approaches target differences in action representation and robot kinematics before or within a shared policy backbone.
- Human-video supervision: Egocentric human videos provide scalable, cost-effective manipulation experience with diverse interactions, environments, and long-tail behaviors beyond robot teleoperation.Methods derive supervision through latent or inverse-dynamics actions, trajectory reconstruction, retargeting, inverse kinematics, visual translation, or morphology-agnostic formulations.
- Supervision quality: Human-video action labels are noisy pseudo-actions affected by tracking jitter, occlusions, and estimation bias, creating a supervision-quality mismatch distinct from representation heterogeneity.Existing frameworks commonly bypass direct action-level training or naively use these noisy pseudo-actions.
3 Method
ACE-EGO-0 addresses heterogeneous embodied pretraining through unified action representation and reliability-aware supervision. It aligns spatial, structural, and temporal action properties while weighting noisy human pseudo-actions separately from high-fidelity robot supervision.
- Unified Action Representation: ACE-EGO-0 unifies heterogeneous trajectories through camera-space coordinates, morphology conditioning, and time-aligned action chunking.These mechanisms align spatial, structural, and temporal dimensions across robot and human data.
- Spatial Alignment: Camera-space actions place robot and human trajectories in a shared head-camera frame, reducing platform-specific world-to-camera transformations.Robot end-effector poses and human hand-centric proxies are represented through a shared robot-compatible action interface.
- Structural Alignment: Morphology tokens encode robot URDF graphs or learned human surrogate embeddings, conditioning the action expert on embodiment-specific structure.This shared morphology space addresses differences in kinematic chains, joint limits, physical dimensions, and human visual or dataset priors.
- Temporal Alignment: Action chunks use a target physical duration rather than a fixed step count, while task-and-horizon bucketing reduces padding and stabilizes updates.The strategy groups semantically related samples and samples with similar chunk lengths across datasets with different control frequencies.
- Reliability-Aware Training: Reliability-aware training uses a robot primary flow-matching loss and a reliability-weighted human auxiliary Huber loss to discount noisy pseudo-actions.Static channel priors and dynamic smoothness factors concentrate human supervision on reliable position channels while reducing the influence of noisy rotations and gripper states.
4 Heterogeneous Pretraining Data
ACE-EGO-0 combines over 6.0K hours of heterogeneous robot, simulation, and pseudo-action-labeled egocentric human data. A five-stage video-to-action pipeline addresses structural and behavioral discrepancies to produce robot-compatible camera-space supervision with quality control.
- Data sources: More than 6.0K hours combine sensor-logged robot demonstrations, simulation rollouts, and pseudo-action-labeled egocentric human videos.The sources differ in coordinate frames, kinematic structures, control frequencies, and action-label quality.
- Data sources: Robot and simulation data span humanoid, single-arm wheeled, and mobile bimanual embodiments, with control frequencies ranging from 10 to 30 Hz.The robot portion includes AgiBot, Galaxea, DigitalWorld, RoboCasa, and more than 1,800 hours of self-collected Galbot demonstrations.
- Data sources: Six egocentric video sources capture diverse kitchens, homes, workshops, and long-tail manipulation behaviors difficult to cover through robot teleoperation alone.The sources are Ego4D, EgoExo4D, EPIC-KITCHENS-100, HOI4D, EgoDex, and Xperience-10M; their labels are inferred from vision rather than physical sensors.
- Video-to-action pipeline: The five-stage pipeline bridges missing metric 3D hand trajectories and behaviorally unsuitable clips through curation, filtering, reconstruction, action formatting, and fidelity-based quality control.Figure 4 specifies dataset curation, ego-interaction and captioning-based video selection, 3D hand reconstruction, robot-convention action parameterization, and multiple quality filters.
- Video-to-action pipeline: Human actions are stored as 16-dimensional bimanual vectors comprising per-hand position, XYZ Euler orientation, gripper, and activity values.Each hand contributes 8 dimensions, yielding 8D × 2 hands = 16D total.
5 Experiments
ACE-EGO-0 is evaluated on RoboCasa GR1 TableTop, RoboTwin 2.0, and a real ARX bimanual platform, achieving strong simulation and real-world manipulation performance. Ablations show gains from morphology conditioning, embodied pretraining, and human-video supervision.
- RoboCasa GR1 TableTop: 72.8% average success rate on RoboCasa GR1 TableTop surpasses DIAL (70.2%), JoyAI-RA (63.2%), ABot-M0 (58.3%), and FLARE (55.0%).The benchmark covers 24 humanoid tabletop tasks, including pick-and-place rearrangement and articulated-object interaction.
- RoboTwin 2.0: 91.12% average success rate on RoboTwin 2.0 Easy/Clean and 90.62% on Hard/Randomized surpass JoyAI-RA by 0.64% and 1.34%, respectively.The benchmark covers 50 bimanual tasks using clean and randomized demonstrations.
- Real-robot evaluation: 78.3% average success rate across six ARX tasks outperforms π0.5 (71.7%) by 6.6% and GR00T-N1.7 (35.6%), with ACE-EGO-0 leading on five tasks.The tasks span single-arm, long-horizon, contact-rich bimanual, and language-grounded manipulation.
- Component ablation: Removing morphology tokens reduces RoboCasa success from 72.8% to 70.9% (−1.9%), showing that morphology conditioning contributes despite a shared camera-space action format.The ablation removes one component at a time from the full model and evaluates checkpoints trained for 190K steps.
- Data source ablation: 72.8% success follows adding human videos to robot pretraining, after Qwen initialization reaches 65.4% and robot data raises it to 68.3% (+2.9%); human videos add +4.5%.Human-video trajectories cover 0.296 m2 versus 0.062 m2 for robot demonstrations, a 4.8× broader action-space area.
6 Conclusion
ACE-EGO-0 unifies human and multi-embodiment robot data through a shared action representation and reliability-aware supervision for noisy pseudo-action labels. On a 6.0K+ hour data pool, it achieves strong benchmark results and improves data-scarce fine-tuning.
- Core framework: ACE-EGO-0 aligns human and robot data across spatial, structural, and temporal dimensions using camera-space actions and reliability-aware training.The framework jointly addresses representational heterogeneity and supervision-quality mismatch.
- Data and evaluation: 6.0K+ hours of data, including 1.48K hours of egocentric human video, support ACE-EGO-0 across multiple robot platforms and simulation environments.The pool spans heterogeneous robot, simulation, and human-video sources.
- Benchmark results: 72.8% on RoboCasa GR1 TableTop and 91.12%/90.62% on RoboTwin 2.0 Easy/Hard splits outperform all compared methods.These results are reported for the unified framework trained on the combined data pool.
- Benchmark results: 4× improvement in data-scarce regimes results from human-augmented fine-tuning.The conclusion specifically attributes this gain to adding human data during fine-tuning.
- Real-world transfer: 78.3 on a real bimanual ARX platform demonstrates ACE-EGO-0’s transfer to real-world bimanual manipulation.The supplied passage reports the value but truncates the associated metric label.
7 Limitations
ACE-EGO-0’s evaluation is currently centered on tabletop manipulation, leaving its generality under broader embodiments, task types, and sensing modalities open. The pretraining pool also lacks dexterous hand data and force/torque sensing.
- Evaluation scope: ACE-EGO-0 is evaluated primarily on tabletop manipulation, so its generality for mobile manipulation, whole-body humanoid control, and deformable-object tasks remains untested.These settings would introduce more diverse spatial conventions and longer task horizons.
- Data coverage: The current pretraining pool does not include dexterous hand data, limiting coverage of fine-grained hand control.
- Data coverage: The current pretraining pool lacks force/torque sensing, leaving that sensing modality unrepresented.
A Additional Method Details … B Training Details
The additional methods standardize robot and human actions in a shared camera-space representation, condition policies on robot or human-source morphology, and weight human supervision by reliability. The supplied passages also describe graph construction, morphology encoding, validation, and auxiliary-loss implementation details.
- A.1 Camera-Space Action Standardization and Layout: Robot and human trajectories are expressed in head-camera coordinates and mapped to a unified 22-dimensional bimanual action vector.Each vector concatenates symmetric 11-dimensional left- and right-arm blocks.
- A.1 Camera-Space Action Standardization and Layout: Robot poses undergo calibrated camera-frame transformation, while human hand trajectories use a wrist-origin hand-centric frame and projection checks mask invalid tracking.Frames with non-positive depth or out-of-bounds projections are removed using an action validity mask.
- A.2 Robot Kinematic Graph Construction: Robot morphology is represented by a cached joint-centric kinematic graph whose 29-dimensional node features encode local, actuation, topological, and chain relationships.The graph is constructed once per canonical robot name and supplies morphology conditioning rather than entering the shared vision-language trunk.
- A.3 Morphology Encoder: The morphology encoder applies residual message passing over the kinematic tree and pools global-body and end-effector-chain summaries into a shared morphology token.The body summary captures global embodiment, while the chain summary emphasizes manipulation-relevant paths.
- A.4 Human Surrogate Morphology Embeddings: Human-video sources use learned surrogate morphology embeddings that capture stable differences in capture conditions, visual domains, annotation quality, and action statistics.The embeddings are projected into the same morphology-token space used by URDF-conditioned robots.
- A.5 Reliability-Aware Human Auxiliary Loss Details: Human auxiliary supervision decomposes reliability into channel, dataset, and local step factors, then normalizes the loss by effective supervision weight.This makes the loss scale-invariant to valid-entry counts and concentrates training on reliable channels.
- A.5 Reliability-Aware Human Auxiliary Loss Details: Step-level reliability softly attenuates unusually large motion jumps or jerks, while per-source, per-hand priors reflect trajectory survival and normalized jerk quality.Temporal smoothing is applied to auxiliary velocity targets after reliability weights are computed.
B.1 Architecture, Training, and Evaluation Protocol … C.2 Real-Robot Success Criteria
ACE-EGO-0 combines a Qwen3-VL-4B-Instruct backbone with a ∼600M-parameter flow-matching action expert, trained through large-scale pretraining and task-specific fine-tuning. Evaluation spans simulated benchmarks and real-robot trials using strict full-sequence completion criteria, with dataset groups sampled separately by source type.
- B.1 Architecture, Training, and Evaluation Protocol: ACE-EGO-0 uses Qwen3-VL-4B-Instruct for vision-language processing and a ∼600M-parameter flow-matching Diffusion Transformer for action prediction.Head and wrist images are processed at 256×256 resolution, and actions are decoded in 4 flow-matching steps at inference.
- B.2 Hyperparameters: Pretraining uses 128×A800 (80GB) GPUs, while task-specific fine-tuning uses 16×A800 GPUs, with AdamW and cosine scheduling for both.Table 8 contains the full optimizer hyperparameters, learning rates, and schedule.
- B.1 Architecture, Training, and Evaluation Protocol: RoboCasa evaluates 50 rollouts per task across 24 tasks, RoboTwin 2.0 evaluates 100 trials per task across 50 tasks, and real-robot experiments use 30 trials per task.RoboTwin 2.0 evaluations cover both Easy and Hard settings.
- B.3 Dataset Mixtures and Sampling: The dataset pool comprises named dataset groups whose sampling weights and preprocessing paths can be controlled independently.Ego4D combines cooking and non-cooking splits, with hours computed as frames/(fps×3600) from LeRobot metadata.
- B.3 Dataset Mixtures and Sampling: Human-video and robot sources are sampled at the dataset-group level and routed to distinct supervision objectives.Human-video groups use camera-space pseudo-actions and the reliability-aware human loss, whereas robot groups use the main robot action objective.
- C.1 Real-Robot Task Descriptions: The real-robot evaluation suite contains six tasks ordered by increasing complexity, including Pick Tea, Scoop Coffee, Category Sorting, Sweep Cubes, and Stack Bowls.The supplied task description lists these task names and their associated manipulation settings.
- C.2 Real-Robot Success Criteria: A real-robot trial succeeds only when the robot completes the full task sequence without human intervention, as judged by a human evaluator.Per-task success definitions are provided for the real-robot tasks.
- C.2 Real-Robot Success Criteria: Real-robot success requires task-specific completion outcomes, such as placing the tea box inside the basket, depositing coffee into the designated cup, and sorting all objects into correct bins.Other criteria include sweeping all cubes into a dustpan and vertically stacking all three bowls.
C.3 Qualitative Results
Qualitative rollouts on the real ARX bimanual platform show ACE-EGO-0 executing successful long-horizon, multi-step manipulation with bimanual coordination and contact-rich tool use.
- C.3 Qualitative Results: Figure 8 shows successful real-ARX rollouts demonstrating long-horizon multi-step manipulation, bimanual coordination, and contact-rich tool use.Each row visualizes key frames from a successful episode in a real-world setting.
C.4 Full RoboCasa GR1 TableTop Results
Table 9 reports per-task success rates across all 24 RoboCasa GR1 TableTop tasks.
- Table 9 reports per-task success rates for all 24 RoboCasa GR1 TableTop tasks.
C.5 Full RoboTwin 2.0 Results
This section reports ACE-EGO-0’s full RoboTwin 2.0 evaluation across all 50 tasks. Results are presented as per-task success rates under Easy/Clean and Hard/Randomized conditions, with 100 trials per task.
- Evaluation scope: Table 10 reports per-task success rates for all 50 RoboTwin 2.0 tasks.The benchmark evaluates the full task suite rather than a subset.
- Evaluation settings: Success rates are reported in percentage for Easy/Clean and Hard/Randomized settings.Easy denotes the clean setting, while Hard denotes the randomized setting.
- Evaluation protocol: 100 trials per task are used for the RoboTwin 2.0 evaluation.This trial count applies to the reported full benchmark results.