PaperlayerView exact revision on arXiv

Source-linked AI summary

Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videos

Qixiu Li, Yu Deng, Yaobo Liang, Lin Luo, Lei Zhou, Chengtang Yao, Lingqi Zeng, Zhiyuan Feng, Huizhi Liang, Sicheng Xu, Yizhong Zhang, Xi Chen, Hao Chen, Lily Sun, Dong Chen, Jiaolong Yang, Baining Guo

arXiv:2510.21571v1cs.ROcs.AIcs.CVcs.LG

TL;DR

Dexterous-hand VLA pretraining is constrained by scarce, costly robot data, motivating the conversion of unstructured human videos into aligned training episodes. The paper builds a fully automatic pipeline and a 1M-episode dataset, obtaining strong zero-shot behavior and improved robot performance after limited fine-tuning, with performance scaling as pretraining data grows.

  • Problem

    Dexterous-hand VLA pretraining lacks large, diverse datasets because robotic manipulation data are costly to collect and especially scarce for dexterous hands.

  • Method

    The paper treats human hands as robot end-effectors and automatically converts unstructured human videos into atomic VLA episodes with 3D motion and language labels.

  • Results

    The resulting dataset contains 1M episodes and 26M frames; pretrained models show strong zero-shot performance, improve after limited robot-data fine-tuning, and scale with pretraining data.

  • Takeaways & Limitations

    Real-life human activity videos provide a scalable source for VLA pretraining and can support generalization to unseen scenes, objects, and environments within the evaluated scope.

  • Takeaways & Limitations

    Current pretraining data contain inaccuracies from limitations in 3D reconstruction and VLM capabilities, and experiments primarily focus on single-handed manipulation.

Abstract

from arXiv · show

This paper presents a novel approach for pretraining robotic manipulation Vision-Language-Action (VLA) models using a large corpus of unscripted real-life video recordings of human hand activities. Treating human hand as dexterous robot end-effector, we show that "in-the-wild" egocentric human videos without any annotations can be transformed into data formats fully aligned with existing robotic V-L-A training data in terms of task granularity and labels. This is achieved by the development of a fully-automated holistic human activity analysis approach for arbitrary human hand videos. This approach can generate atomic-level hand activity segments and their language descriptions, each accompanied with framewise 3D hand motion and camera motion. We process a large volume of egocentric videos and create a hand-VLA training dataset containing 1M episodes and 26M frames. This training data covers a wide range of objects and concepts, dexterous manipulation tasks, and environment variations in real life, vastly exceeding the coverage of existing robot data. We design a dexterous hand VLA model architecture and pretrain the model on this dataset. The model exhibits strong zero-shot capabilities on completely unseen real-world observations. Additionally, fine-tuning it on a small amount of real robot action data significantly improves task success rates and generalization to novel objects in real robotic experiments. We also demonstrate the appealing scaling behavior of the model's task performance with respect to pretraining data scale. We believe this work lays a solid foundation for scalable VLA pretraining, advancing robots toward truly generalizable embodied intelligence.

1 Introduction

Dexterous-hand VLA pretraining lacks large, diverse robot datasets, while unstructured human videos offer abundant activity data but require alignment into robotic formats. The paper introduces an automated pipeline that performs this alignment and supports scalable pretraining with strong zero-shot and fine-tuned performance.

  • Motivation: Dexterous-hand VLA pretraining remains largely underexplored because existing robotic data are costly, limited in scale and diversity, and especially scarce for dexterous hands.Large generic pretraining supports commonsense knowledge and domain generalization, but comparable resources for dexterous manipulation are lacking.
  • Motivation: Unstructured real-life human videos contain diverse activities and environments but lack reliable segmentation, language instructions, and 3D action labels.The paper frames transforming these videos into robotic V-L-A formats as its central question.
  • Approach: The fully automatic framework treats the human hand as a robot end-effector and generates atomic action segments, metric-scale 3D hand and camera motion, and language instructions.It addresses both task alignment and label alignment for existing robotic V-L-A data.
  • Results: The resulting dataset contains about 1 million episodes and 26 million frames spanning diverse objects, skills, concepts, and environmental variations.The model uses a dexterous hand VLA architecture with a Causal Action Transformer for pretraining.
  • Results: The pretrained model shows strong zero-shot performance in unseen scenes, while limited real-robot fine-tuning improves task success and generalization to novel objects and backgrounds.Experiments also show task performance scales with the amount of pretraining data.
  • Implications: The approach is presented as a tractable path toward scaling low-cost human-video pretraining, complemented by modest real-robot data or reinforcement learning.The authors state that the pipeline has no technical barriers preventing further scaling and plan to open-source the dataset and models.

2 Related Works

Prior robotic VLA systems rely heavily on costly robot demonstrations and large robot datasets, while human-video methods commonly learn representations, affordances, or latent actions rather than explicit 3D action labels for dexterous VLA pretraining.

  • Robotic VLA Pretraining: Most robotic VLA models requiring diverse language-instructed tasks pretrain on large datasets such as Open X-Embodiment, which contains over 1M trajectories from more than twenty robots.The review focuses on action pretraining or action proxies rather than general vision-language pretraining.
  • Dexterous Hand Manipulation: Concurrent methods using 3D hand labels generally rely on controlled, scripted videos and privileged capture devices such as RGBD sensors or VR/AR headsets.The paper instead targets unstructured, in-the-wild human videos.
  • Robot Learning from Human Videos: Human-video approaches have learned vision-language representations, affordances, point trajectories, masks, or explicit actions for robot policy training.These methods span representation learning, imitation learning, and other forms of action guidance.
  • Temporal Action Segmentation: Temporal action segmentation methods detect and classify action windows, but existing approaches still struggle with localization accuracy and did not meet the paper’s requirements.Earlier methods typically assume predefined action classes.

3 Transforming Human Hand Video to VLA Data

The paper converts unscripted human videos into robotic-style VLA episodes through 3D motion recovery, atomic action segmentation, and instruction labeling. The resulting pipeline is fully automatic and produces large-scale data without using source-dataset action annotations.

  • Data Format: Existing robot episodes pair language instructions and video frames with frame-aligned 3D end-effector action chunks; this framework applies the same format to human hands.The two human hands are treated as the robot end-effector.
  • 3D Motion Labeling: 3D motion labeling reconstructs framewise hand poses and camera trajectories from monocular videos, including wrist 6D pose and full joint angles.The pipeline estimates camera properties and tracks hand and camera motion, including for moving cameras.
  • 3D Motion Labeling: World-space hand sequences can be transformed into any frame’s camera space, supporting static-camera-style representations and later segmentation and instruction labeling.Long videos are processed as overlapping 20-second clips and then recomposed.
  • Atomic Action Segmentation: Atomic action segmentation detects minima in smoothed 3D wrist speed and uses them as cutting points for short-horizon hand-action clips.The method requires no additional model inference or preannotated text labels, though repetitive actions may be over-segmented.
  • Instruction Labeling: Instruction labeling samples eight frames per segment, overlays projected hand trajectories, and prompts GPT to describe meaningful actions in imperative language.The model also filters clips that do not contain meaningful manipulation.
  • Instruction Labeling: Atomic-level clips and trajectory overlays improve GPT captioning accuracy compared with fixed-length segments because they reduce multiple-action ambiguity and clarify the relevant hand motion.This finding supports using motion-based segmentation before language annotation.
  • Dataset Construction: The constructed dataset contains 1M episodes and 26M frames from Ego4D, Epic-Kitchen, EgoExo4D, and SSV2, covering activities such as cooking, cleaning, construction, repairing, crafting, and painting.The source datasets’ human action annotations are not used because they may have unsuitable granularity or imprecise temporal boundaries.

4 Dexterous Hand VLA Model

The dexterous hand VLA combines a vision-language backbone with a diffusion action expert to predict future hand-action chunks from visual, linguistic, camera, and end-effector information. It uses camera-coordinate hand actions, unified dual-hand instructions, causal denoising, and trajectory-aware augmentation to support variable-length manipulation episodes and robot fine-tuning.

  • The model predicts future end-effector action sequences from visual observations, proprioceptive state, and language instructions.
  • 4.1.1 Model Architecture: The architecture couples a VLM backbone with a diffusion action expert that iteratively denoises action chunks using cognition features and causal attention.The VLM receives visual and linguistic instructions plus camera FoV; the action expert also receives end-effector state and valid action masks.
  • 4.1.2 Hand Action Space: Hand actions are represented in the current camera frame using relative wrist translation, rotation, and MANO joint angles for both hands.The action representation includes 3D relative wrist translation, 3D relative rotation, and Euler angles for 15 hand joints.
  • 4.1.2 Hand Action Space: The VLM uses a unified left-hand/right-hand instruction format, while the action expert receives noisy actions for both hands.Each hand’s language field is either None or the instruction for its corresponding atomic action chunk.
  • 4.1.1 Model Architecture: Causal attention prevents zero-padded positions beyond short episode ends from influencing earlier action predictions.This addresses clips as short as approximately 1 second when prediction chunks extend beyond the episode boundary.
  • Trajectory-aware image and action augmentation varies camera field of view, aspect ratio, crop center, and perspective while transforming actions consistently.The augmentation is applied during pretraining to enhance generalization.
  • 4.3 Fine-tuning for Robotic Dexterous Manipulation: Fine-tuning maps robot end-effector poses and joints into the human-hand action space, while acknowledging that alternative mappings remain open.The method does not perform direct pose transfer and uses fine-tuning to mitigate action-space differences.

5 Experiments

Experiments analyze the dataset’s visual and language diversity, evaluate hand-action prediction and construction choices, and compare downstream robotic performance across pretraining methods and scales. The results show that diverse, aligned human-video pretraining improves generalization, with performance increasing as pretraining data scale grows.

  • 5.1 Pretraining Data Analysis: The dataset’s visual observations provide broad real-world scene coverage, with similarity to OpenImages increasing rapidly as episode count grows.The steeper similarity slope indicates more uniform coverage than the fragmented distribution observed in OXE.
  • 5.1 Pretraining Data Analysis: Language-instruction analysis compares distributions of nouns, verbs, and adjectives across VLA datasets to assess task-related diversity.Curves closer to the upper-right indicate more distinct words occurring with sufficient frequency.
  • 5.2 Performance Analysis: Our method consistently outperforms baselines on the constructed hand-action benchmark and generalizes more strongly than models trained on EgoDex data.The authors attribute weaker instruction following in original human annotations to temporal or granularity misalignment between text and actions.
  • 5.2 Performance Analysis: Removing trajectory-aware augmentation or replacing causal attention with bidirectional attention substantially reduces performance.The authors connect augmentation to visual diversity and causal attention to the characteristics of the pretraining data.
  • 5.2 Performance Analysis: Fixed-interval episode segmentation and removing hand-trajectory overlays during captioning degrade grasping performance.Fixed intervals can combine multiple actions in one clip, while trajectories guide GPT in aligning captions with actions.
  • 5.3 Results and Comparisons: Human-video pretraining achieves stronger few-shot and unseen-task performance than alternative methods, while increasing pretraining scale improves both seen- and unseen-task success rates.EgoDex performs worse than the model trained on only 10% of this dataset despite containing more episodes and frames, and latent-action pretraining fails in unseen environments.
  • 5.3 Results and Comparisons: Higher hand-action prediction performance positively correlates with downstream robot task success after fine-tuning.This supports using the hand-action benchmark as a proxy for rapidly prototyping pretrained VLA models.

6 Discussion and Future Work

The framework is presented as an initial exploration whose pipeline can scale to more diverse video sources, while current data and experiments remain bounded by reconstruction quality, short-horizon skills, and mostly single-handed tasks.

  • Future Work: The automatic pipeline is readily extensible to more diverse video sources, including Howto100M, for larger and more comprehensive VLA pretraining.The authors plan to improve data quality with advanced reconstruction and additional filtering mechanisms.
  • Limitations: Current pretraining data contains inaccuracies from limitations of 3D reconstruction algorithms and inherent VLM capabilities.The authors identify reconstruction improvements and noisy-sample filtering as future directions.
  • Limitations: Current data construction and model training primarily target short-horizon, atomic manipulation skills rather than higher-level task structures for long-horizon planning and reasoning.Extending the framework to organize higher-level task structures is identified as an important future direction.
  • Limitations: Current robotic experiments primarily focus on single-handed manipulation, although the framework naturally supports bimanual operations.A simple hand-over experiment demonstrates feasibility on two-handed tasks, while broader bimanual scenarios remain future work.

7 Conclusion

The paper presents a scalable approach for pretraining robotic manipulation VLA models from unstructured real-life human activity videos. Its pipeline, model, and experiments support zero-shot transfer, improved fine-tuned performance, and favorable data-scaling behavior.

  • Conclusion: The paper converts unstructured real-life human videos into atomic-level VLA data aligned with existing robotic demonstrations through a fully automatic pipeline.It treats the human hand as a dexterous robot end-effector and designs a hand VLA model with tailored training strategies.
  • Conclusion: The pretrained model exhibits strong zero-shot performance in unseen real-world environments and high task success after fine-tuning on limited robot data.The conclusion also reports improved generalization and favorable scaling behavior.
  • Conclusion: The results demonstrate a promising and scalable approach toward learning more generalizable embodied robots.This conclusion is stated within the scope of the reported pretraining and robotic experiments.

A More Implementation Details

The implementation estimates camera and hand geometry, segments videos into atomic actions, generates diverse captions, and adapts visual and action processing for robust hand-VLA training.

  • Camera Processing: Camera intrinsics are estimated with DroidCalib for moving cameras and DeepCalib for static cameras under assumptions about the principal point and focal lengths.DroidCalib uses a unified camera model with an additional distortion parameter for ultra-wide-angle and fisheye cameras.
  • Camera Processing: HaWoR reconstructs camera-space 3D hands using estimated focal information, while its motion-infilling module is discarded because interpolation is less reliable on incomplete reconstructions.The reconstruction operates jointly within each video chunk.
  • Camera Processing: MegaSAM estimates metric-scale camera poses, with MoGe-2 replacing its original depth modules for more accurate, stable, and efficient inference.The modification addresses limited camera baselines and complex scene dynamics.
  • Action Segmentation: Long videos are segmented at local minima of smoothed world-space 3D wrist speed, using a 0.5-seconds window to suppress noise-induced extrema.Gaussian smoothing is applied before detecting minima.
  • Language Annotation: GPT-4.1 captions atomic video clips from sampled frames and produces five meaning-preserving paraphrases to increase language-description diversity.The captions are generated after atomic-level segmentation.
  • Model Input: The VLM resizes images to 2242 without center cropping and incorporates aspect-ratio and camera-intrinsics cues because the SigLIP encoder uses a fixed 1:1 ratio.FoV cues are also relevant under trajectory-aware augmentation.
  • Action Expert: The action expert is a roughly 136M-parameter Diffusion Transformer using causal self-attention for action tokens, QKNorm, and RMSNorm to improve training stability.These design choices are applied within the transformer blocks.
  • Normalization: State and action inputs are standardized dimension-wise to zero mean and unit variance using fixed unified normalization parameters weighted by dataset frame-sampling probabilities.Dataset-specific statistics are combined before pretraining and evaluation.

A.3 Training Details

Training uses diffusion-based action prediction with augmentation, input dropout, coordinate alignment, and hand-pose retargeting to connect human pretraining data with real-robot control.

  • Training Setup: Training uses 16-step action chunks, 100 diffusion noise steps, eight noisy samples per VLM pass, AdamW, weight decay 1e-1, and gradient clipping 1.0.These optimizer settings are applied consistently during pretraining and fine-tuning.
  • Pretraining: Trajectory-aware augmentation and 0.1-probability dropout of state and cognition inputs encourage reliance on vision-language information and reduce state overfitting.A single GPT-generated instruction is randomly selected when trajectories have multiple annotations.
  • Fine-tuning: Real-robot and human coordinate systems are aligned during fine-tuning, while pretrained and non-pretrained models use 20K and 60K steps respectively because the latter converge more slowly.The non-pretrained model produces highly jittery actions within the shorter schedule.
  • Inference: Inference uses DDIM with 10 sampling steps and CFG scale 5.0, executes 6 of 16 predicted actions at a time, and converts poses to joint angles through inverse kinematics.The resulting joint angles are mapped to the dexterous robot hand.
  • Teleoperation: A leader-follower arm system sends measured leader-arm joint angles directly to follower arms for precise end-effector 6D-pose control during teleoperation.The leader arms share the Realman robot arms’ joint topology.
  • Hand Pose Retargeting: MANUS glove measurements are retargeted to robot-hand joint angles by optimizing the mapping at each timestep.The pipeline implements two optimizers for this retargeting process.
  • Hand Pose Retargeting: The DexPilot optimizer matches five wrist-to-fingertip and ten interfinger glove vectors to corresponding robot vectors obtained through forward kinematics.Its objective is constrained by robot joint limits and includes scaling, temporal smoothness, and fingertip-contact weighting.
  • Hand Pose Retargeting: The retargeting objective enforces robot joint limits, scales for different hand sizes, smooths motion temporally, and increases fingertip-contact weight as fingertip-wrist distance decreases.These terms jointly shape the robot-hand motion toward the glove measurements and contact configurations.

B.1 Hand Action Prediction Benchmark

The benchmark reconstructs target-object geometry and evaluates predicted hand actions using annotated RGB-D scenes and human judgments of general-action plausibility.

  • RGB-D images are undistorted before object positions and captions are manually annotated.The annotated positions prompt SAM-2 to obtain object masks.
  • Object masks, depth images, and camera intrinsics reconstruct each target object as a camera-coordinate 3D point cloud.
  • Synthetic SMPL-X hands with attached arms are rendered approximately 20 cm from target objects in natural resting poses.The hand is placed closer to the camera than the object to prevent incorrect occlusion relationships.
  • General hand-action predictions are evaluated through a randomized user study covering 117 scenes and 30 trials per participant.Method identities and presentation order were anonymized and randomized to reduce bias.

C.1 Human Hand V-L-A Data

The constructed hand V-L-A data spans varied environments and hand actions, with moving cameras adding noticeable frame-to-frame observation diversity.

  • The dataset covers a wide range of environments and hand actions.
  • Moving cameras create noticeable variation across frames, further increasing the diversity of the pretraining data.

C.2 Hand Action Prediction Results

The pretrained VLA model generalizes to unseen real-life environments and predicts diverse human hand motions.

  • The pretrained VLA model demonstrates strong generalization across unseen real-life environments.
  • The model predicts diverse human hand motions in those unseen scenes.

C.3 Real-Robot Execution Results

The experiments include visualized robot executions for general, functional, and sequential manipulation tasks, including a bimanual handover.

  • Figures VII–IX present additional visual results of real-robot task executions.
  • The visualized tasks include general pick-and-place, functional grasping, sweeping, pouring, and bimanual handover.
  • Figure VIII includes executions with unseen backgrounds and objects, including objects from previously unseen categories.The figure caption distinguishes new objects within seen categories from objects belonging to unseen categories.
Loading 2510.21571v1…