Spatial-Aware VLA Pretraining through Visual-Physical Alignment from Human Videos

1 Introduction

• Existing VLA models rely on 2D visual inputs for action in 3D environments, while largely overlooking spatial understanding and consequently showing poor spatial grounding and limited generalization.The introduction frames the missing correspondence between perception and embodied action as a substantial gap.Source: §1 · pp. 1–2
• The proposed Spatial-Aware VLA Pretraining uses large-scale human videos to teach models 3D spatial awareness before robot policy learning.It extracts 3D cues, including hand-object relationships and motion trajectories, to align 2D vision with 3D spatial understanding.Source: §1 · p. 2, Figure 1 · p. 2
• Hand3D comprises human manipulation videos annotated with 3D visual and action information for visual–physical alignment supervision.The annotations provide coarse spatial grounding and fine-grained motion supervision through 3D trajectories.Source: §1 · p. 2
• VIPA-VLA is a dual-encoder architecture that augments semantic visual representations with explicit 3D spatial features and improves spatial grounding and generalization in downstream robot tasks.The pretrained model is adapted to robotic tasks via post-training.Source: §1 · p. 2

2.1 VLA models

• VLA research includes robot-action fine-tuning, action experts, video-based representation learning, and large-scale multimodal pretraining, while this work focuses on 3D physical-space understanding from human videos.The cited approaches include GR00T-N1[7] and π0.5[32], which combine internet and robot data for pretraining.Source: §2.1 · p. 3

2.2 3D Aware Models

• Existing 3D vision-language models improve spatial perception but primarily address static observations, without explicitly connecting 3D perception to the physical action space needed for robot policy learning.The paper distinguishes its pretraining approach, which uses human-video annotations and a 3D encoder to align 2D observations with physical space before policy learning.Source: §2.2 · p. 3

2.3 Learning from human videos

• Human-video methods learn implicit activity features, align human and robot actions, or extract interaction cues, but the paper argues these approaches do not explicitly bridge perception and action in 3D physical space.Direct human-to-robot action alignment is described as subject to embodiment mismatch; this work instead uses human videos to supervise grounded correspondences before downstream policy learning.Source: §2.3 · p. 3

3 Method

• The section first provides preliminaries on vision-language models and vision-language-action models.These preliminaries are in Section 3.1.Source: §3 · p. 3
• The Visual Physical Alignment discussion covers the Hand3D spatial-aware dataset and VIPA-VLA model architecture.The section identifies these as the dataset and architecture components of Section 3.2.Source: §3 · p. 3
• The section also presents the Spatial-Aware VLA pretraining strategy as part of its Visual Physical Alignment method.The provided section overview names this strategy alongside the dataset and architecture.Source: §3 · p. 3

3.2.1 Hand3D

• Hand3D derives 3D visual annotations from point clouds, object proposals, and hand poses, then calibrates point-cloud scale against absolute hand-joint positions.Applying the scale factor yields hands and objects in a consistent 3D physical coordinate system.Source: §3.2.1 · p. 5
• Hand3D-visual converts dense geometry into VQA-style labels covering spatial relations, task completion, hand movement, and camera movement.The labels represent spatial offsets by direction and Euclidean distance, and camera motion by rotation and translation.Source: §3.2.1 · pp. 5–6
• From nine heterogeneous human-video sources, the pipeline sampled approximately 4K clips for Hand3D-visual and produced around 300K instruction–answer pairs.Hand3D-action includes 4M video–instruction–motion pairs before filtering and a resulting 1M curated dataset.Source: §3.2.1 · pp. 6–7

3.2.2 VIPA-VLA

• VIPA-VLA combines semantic visual embeddings with Cut3R spatial embeddings through cross-attention, while discretized wrist coordinates become motion tokens.The motion-token scheme uses 1024 bins per coordinate axis over ranges of [-0.5, 0.5] for x and y and [0, 1] for z.Source: §3.2.2 · p. 7

3.2.3 Spatial-Aware VLA Pretraining

• Spatial-Aware VLA Pretraining first trains the fusion layer on 3D visual VQA data, then trains the LLM to predict motion tokens from fused visual and text inputs.Pretrained parameters are frozen in Stage 1; in Stage 2, the semantic and spatial encoders are frozen.Source: §3.2.3 · p. 8

3.3 Post-Training

• For downstream tasks, VIPA-VLA conditions a diffusion-transformer action head on visual-language context and robot state to predict executable action chunks.During training, only the LLM backbone and action head are updated.Source: §3.3 · pp. 8–9

4.1 Implementation Details

• Training has three stages: Hand3D-visual pretraining, Hand3D-action pretraining, then post-training with the visual and 3D encoders frozen.The first two stages optimize the fusion layer; the second also optimizes the LLM backbone.Source: §4.1 · pp. 9–10

4.2 Simulation Robot Tasks

• On LIBERO, VIPA-VLA performs strongly across all task suites and surpasses several prior methods, with results comparable to π-series and GR00T.The comparison includes methods that explicitly model spatial reasoning, such as SpatialVLA, TraceVLA, and MolmoAct.Source: §4.2 · p. 10
• On RoboCasa, VIPA-VLA achieves the best overall performance using 50 human demonstrations per task, including a +9.9% improvement in Doors/Drawers.The benchmark covers 24 tasks across Pick & Place, Doors / Drawers, and Others, with 50 trials per task.Source: §4.2 · p. 10, Table 4 · p. 10

4.3 Real Robot Tasks

• Real-robot evaluation uses three manipulation tasks, 50 teleoperated post-training trajectories per task, and 10 randomized evaluation trials per task.The tasks test multi-object placement, wiping irregular targets, and fine-grained 3D motion; evaluation also reports averaged sub-task success.Source: §4.3 · p. 11
• VIPA-VLA achieves the best overall performance across the real-world manipulation tasks, with higher sub-task success but lower complete-task success than InternVL3.5 on Put-Three-Obj.The authors attribute this difference to baseline models failing early, while VIPA-VLA progresses more consistently across stages.Source: §4.3 · p. 11, Table 5 · p. 11
• VIPA-VLA significantly outperforms all baselines in unseen visual environments.Evaluation reports sub-task and whole-task success rates.Source: §4.3 · p. 11, Table 6 · p. 12

4.4 Ablation Studies

• Ablations on LIBERO find that removing either human-video Spatial-Aware Pretraining or the dual-encoder architecture consistently lowers performance across task suites.The authors associate pretraining with 3D motion and perception priors, and the dual encoder with more effective fusion of 3D visual representations.Source: §4.4 · pp. 12–13, Table 7 · p. 13
• On Hand3D-test, Hand3D pretraining improves spatial grounding over InternVL3.5, and VIPA-VLA gains further from its 3D visual encoder and fusion mechanism.The evaluation uses unseen-video VQA pairs and measures distance and direction accuracy; after spatial-aware pretraining, predictions have fewer severe errors and are highly accurate in about 30% of cases.Source: §4.4 · pp. 13–14, Table 8 · p. 13

• The paper addresses the 2D–3D gap in VLA models through Spatial-Aware VLA Pretraining from large-scale human videos.Source: §5 · p. 14
• Hand3D provides 3D visual and action annotations from human manipulation, while VIPA-VLA uses a dual-encoder architecture for visual-physical alignment.Source: §5 · p. 14
• The authors report improvements in spatial reasoning and downstream robotic manipulation, and propose combining the paradigm with robot-data pretraining in future work.Source: §5 · p. 14