Source-linked AI summary
PointVLA: Injecting the 3D World into Vision-Language-Action Models
Chengmeng Li, Junjie Wen, Yan Peng, Yaxin Peng, Feifei Feng, Yichen Zhu
TL;DR
2D-pretrained VLAs have limited 3D spatial reasoning, while retraining with scarce 3D data is costly and discarding 2D data is impractical. PointVLA adds point-cloud conditioning through modular injection into selected blocks while preserving pretrained representations, and experiments demonstrate few-shot, safety-related, adaptive, and long-horizon capabilities.
Problem
Existing VLAs rely on 2D inputs that limit 3D spatial reasoning, while retraining with 3D data is computationally prohibitive and discarding 2D datasets is impractical.
Method
PointVLA preserves pretrained 2D representations by treating point clouds as complementary conditioning and injecting 3D features into selected action-expert blocks.
Results
PointVLA demonstrates few-shot multi-task learning with four tasks and 20 demonstrations each, along with strong performance in dynamic long-horizon packing and real-world bimanual experiments.
Takeaways & Limitations
The experiments support augmenting pre-trained robot models with new modalities without costly retraining while retaining practical and safety-related capabilities.
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) models excel at robotic tasks by leveraging large-scale 2D vision-language pretraining, but their reliance on RGB images limits spatial reasoning critical for real-world interaction. Retraining these models with 3D data is computationally prohibitive, while discarding existing 2D datasets wastes valuable resources. To bridge this gap, we propose PointVLA, a framework that enhances pre-trained VLAs with point cloud inputs without requiring retraining. Our method freezes the vanilla action expert and injects 3D features via a lightweight modular block. To identify the most effective way of integrating point cloud representations, we conduct a skip-block analysis to pinpoint less useful blocks in the vanilla action expert, ensuring that 3D features are injected only into these blocks--minimizing disruption to pre-trained representations. Extensive experiments demonstrate that PointVLA outperforms state-of-the-art 2D imitation learning methods, such as OpenVLA, Diffusion Policy and DexVLA, across both simulated and real-world robotic tasks. Specifically, we highlight several key advantages of PointVLA enabled by point cloud integration: (1) Few-shot multi-tasking, where PointVLA successfully performs four different tasks using only 20 demonstrations each; (2) Real-vs-photo discrimination, where PointVLA distinguishes real objects from their images, leveraging 3D world knowledge to improve safety and reliability; (3) Height adaptability, Unlike conventional 2D imitation learning methods, PointVLA enables robots to adapt to objects at varying table height that unseen in train data. Furthermore, PointVLA achieves strong performance in long-horizon tasks, such as picking and packing objects from a moving conveyor belt, showcasing its ability to generalize across complex, dynamic environments.
1. Introduction
PointVLA addresses the spatial limitations of 2D-pretrained VLAs by adding point-cloud information while preserving existing representations. Experiments show advantages in few-shot multitasking, real-versus-photo discrimination, height adaptability, and long-horizon manipulation.
- Existing robot foundation models rely mainly on 2D visual inputs, limiting spatial reasoning for precise perception, depth awareness, and object manipulation.
- PointVLA integrates point clouds into pre-trained VLAs through a 3D modular block while preserving the vision-language backbone and minimizing disruption to action-expert features.
- Few-shot multi-tasking: PointVLA performs four instructed tasks using only 20 demonstrations per task and significantly outperforms the baseline in this few-shot multi-task setting.
- Real-vs-photo discrimination and height adaptability: PointVLA distinguishes real objects from photographic representations and adapts robot actions to unseen table heights, addressing safety and geometric variation.
- Long-horizon tasks: PointVLA handles long-horizon manipulation involving multiple objects picked from a moving conveyor belt and packed into a box.
- The experiments support PointVLA’s performance and generalizability across diverse scenarios, while motivating additional modalities for pre-trained VLA models.
2. Related Works
Related work develops VLAs from vision-language backbones and explores 3D robot learning, but existing approaches can depend on simulation or costly retraining. PointVLA targets integration of 3D inputs without discarding established 2D data.
- Vision-Language-Action models: VLAs extend vision-language models trained on internet-scale image-text data to robotic control and generalist robot policies.
- Robot learning with 3D modalities: Existing 3D robot-learning methods address scene understanding, generalization, and control, but some rely on simulation, creating a sim-to-real gap.
- Robot learning with 3D modalities: Discarding existing 2D robot data or fully retraining foundation models with 3D inputs is computationally expensive and resource-intensive.
3. Methodology
PointVLA treats point clouds as complementary conditioning for a frozen VLA, injecting 3D features into selected action-expert blocks. Skip-block analysis identifies blocks that can accept this information with limited disruption.
- The VLA combines a vision-language model that processes images and instructions with an action expert that translates state information into robot actions.
- Motivation: Because 3D datasets are much smaller than 2D pretraining corpora, the method aims to preserve rich 2D representations while integrating sparse 3D data.
- PointVLA framework: The framework keeps the vanilla action expert frozen and integrates point-cloud representations through a modular network.
- Motivation: Point clouds are treated as complementary conditioning rather than a primary modality, decoupling 3D processing from the core 2D encoder.
- Point cloud injector: The injector aligns point-cloud embeddings with the action expert and uses a bottleneck to compress action information before selected-block adaptation.
- Point cloud injection: 3D features are injected only into less critical action-expert blocks to reduce computational cost and interference with pretrained 2D representations.
- Point cloud encoder: The point-cloud encoder uses hierarchical convolutions and max pooling to progress from low-level features toward high-level scene representations.
- Skip block analysis: The skip-block analysis evaluates single-block skips and multiple consecutive skips beginning at the 11th block.
4. Experiment
PointVLA is evaluated across real-world bimanual setups, few-shot multi-tasking, long-horizon packing, real-vs-photo discrimination, height adaptability, and RoboTwin simulation. Across these settings, it achieves strong performance while preserving the ability to learn from the 2D-pretrained VLA.
- Experimental scope: The experiments cover real-robot and simulation benchmarks, including few-shot multi-tasking, long-horizon packing, two 3D-enabled generalization tests, and RoboTwin.Real-world studies use bimanual UR5e and AgileX platforms, while RoboTwin provides a mobile bimanual simulation benchmark.
- Few-shot multi-tasking: PointVLA outperforms all baselines in the few-shot multi-tasking scenario, with DexVLA remaining on par with or inferior to PointVLA.The few-shot evaluation uses four tasks and 20 demonstrations per task.
- Long-horizon packing: 0.64 higher Avg. Len. than DexVLA is achieved by PointVLA on the long-horizon bimanual UR5e packing task.The task sequentially picks and places two detergent bags from a moving conveyor before sealing the box.
- Real-vs-photo discrimination: PointVLA recognizes that a photographed detergent is not a real object, whereas OpenVLA and DexVLA attempt to grasp the nonexistent object.The point-cloud input lets the model identify the apparently occupied location as empty in 3D space.
- Simulation benchmark: PointVLA achieves the highest average success rate across RoboTwin tasks with either 20 or 50 demonstrations.The comparison includes Diffusion Policy and 3D Diffusion Policy, with RGB added to DP3 for fairness.
- Input integration: PointVLA conditionally integrates point clouds, unlike the direct RGB addition to DP3 that can negatively impact purely 3D-model performance.The experiments report that conditional 3D integration significantly enhances performance compared with relying solely on 2D visual input.
5. Conclusion
PointVLA augments pre-trained VLAs with 3D point-cloud inputs while preserving 2D representations, avoiding full retraining. Simulated and real-world experiments show few-shot multi-task capability and strong performance on dynamic long-horizon packing.
- PointVLA integrates a modular 3D feature injector into pre-trained VLAs while preserving their 2D representations.The method uses skip block analysis to select injection locations without full retraining.
- 4 tasks with only 20 demonstrations each demonstrate PointVLA’s few-shot multi-task learning capability.
- PointVLA performs strongly in long-horizon tasks such as dynamic item packing across simulated and real-world environments.Real-world evaluations include UR5e and AgileX bimanual robots.