PaperlayerView exact revision on arXiv

Source-linked AI summary

Any3D-VLA: Enhancing VLA Robustness via Diverse Point Clouds

Xianzhe Fan, Shengliang Deng, Xiaoyang Wu, Yuxiang Lu, Zhuoling Li, Mi Yan, Yujia Zhang, Zhizheng Zhang, He Wang, Hengshuang Zhao

arXiv:2602.00807v2cs.CVcs.RO

TL;DR

VLAs largely rely on 2D visual backbones, limiting spatial understanding in challenging scenes. ANY3D-VLA trains on diverse point-cloud sources and fuses 3D features with corresponding 2D features. The authors report improved performance across simulation and real-world evaluations, including robustness when depth quality and scale vary.

  • Problem

    VLA spatial understanding is largely inherited from 2D visual backbones, and the paper identifies scarce 3D data and cross-environment depth differences as challenges to incorporating 3D information.

    Source: §1 · p. 2

  • Method

    ANY3D-VLA trains with diverse point-cloud sources and compresses, aligns, and fuses point-cloud features with corresponding 2D patch representations.

    Source: §1 · p. 2, §5.2 · pp. 4–5

  • Results

    Across simulation and real-world evaluations, ANY3D-VLA reports superior performance and robustness to noisy or scale-biased depth; real-world zero-shot maximum overall accuracy is 62.5%, outperforming the best baseline by 29.2%.

    Source: §1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    The authors conclude that native sparse 3D point representations fused with 2D features are a promising direction for VLA manipulation.

    Source: §8 · p. 9

    Real-world experiments cover only one robotic arm and a limited set of objects.

    Source: §7 · p. 8

Abstract

from arXiv · show

Existing Vision-Language-Action (VLA) models typically take 2D images as visual input, which limits their spatial understanding in complex scenes. How can we incorporate 3D information to enhance VLA capabilities? We conduct a pilot study across different observation spaces and visual representations. The results show that explicitly lifting visual input into point clouds yields representations that better complement their corresponding 2D representations. To address the challenges of (1) scarce 3D data and (2) the domain gap induced by cross-environment differences and depth-scale biases, we propose Any3D-VLA. It unifies the simulator, sensor, and model-estimated point clouds within a training pipeline, constructs diverse inputs, and learns domain-agnostic 3D representations that are fused with the corresponding 2D representations. Simulation and real-world experiments demonstrate Any3D-VLA's advantages in improving performance and mitigating the domain gap. Our project homepage is available at https://xianzhefan.github.io/Any3D-VLA.github.io.

1. Introduction

The paper argues that point clouds complement 2D representations for VLA spatial understanding and proposes ANY3D-VLA to train and fuse them across diverse depth sources. Simulation and real-world evaluations report improved performance and robustness to noisy or scale-biased depth.

  • VLA spatial understanding remains largely inherited from 2D backbones, leaving models brittle around small objects, viewpoint changes, and occlusions.Source: §1 · p. 2
  • The pilot study finds that explicitly projecting visual input into point clouds complements 2D representations more effectively than implicit or reconstruction-based spatial priors for fine-grained spatial relationships.Source: §1 · p. 2
  • Prior point-cloud VLA approaches face limits including scarce 3D data, cross-environment differences, depth-scale biases, and dependence on pretrained encoders or non-parametric tokenizers.Source: §1 · p. 2
  • ANY3D-VLA lifts and compresses visual inputs into point clouds, then aligns and fuses their embeddings with corresponding 2D patch features for existing VLA backbones.The paper also constructs a large-scale RGBD pre-training dataset with diverse depth sources.Source: §1 · p. 2
  • In real-world zero-shot evaluation, ANY3D-VLA reaches a maximum overall accuracy of 62.5%, outperforming the best baseline by 29.2%.The paper also reports a maximum overall accuracy of 93.3% in post-training.Source: §1 · p. 2

2. Related Work

Prior work injects 3D information into VLAs through learned spatial priors, depth inputs, or point clouds. Depth sources differ in quality and scale, while point-cloud approaches vary in how they encode and combine 3D with 2D inputs.

  • Some VLAs use spatial priors from pretrained encoders while retaining RGB-only inputs at inference, whereas others directly incorporate depth maps.Examples include Spatial Forcing and DepthVLA for learned priors, and 3D-CAVLA for depth-based scene awareness.Source: §2 · pp. 2–3
  • Point-cloud VLA methods include feature injection and 3D spatial constraints, but many rely on non-pretrained encoders or non-parametric tokenizers, use only point clouds, or keep them relatively independent from 2D inputs.Source: §2 · p. 3
  • Simulator depth can be high-quality metric depth, while sensor depth varies in noise and quality and model-estimated depth may suffer scale drift.These differences can introduce a sim-to-real gap, although depth estimation from RGB can support larger-scale data acquisition.Source: §2 · p. 3

3. Dataset and Benchmark

The authors build a simulator-based RGBD dataset and a held-out-scene evaluation benchmark for VLA tasks.

  • The simulator pre-training dataset draws on 290 categories and 10,680 instances from the Objaverse LVIS subset, with randomized object layouts on a 40cm × 50cm tabletop.Source: §3 · p. 3
  • RGB and aligned depth are rendered from a single viewpoint with randomized lighting, materials, backgrounds, and camera extrinsics; depth comes either from simulation rendering or RGB-based metric-depth estimation.The camera intrinsics match a RealSense D435, and depth images are stored at 256 × 256 × 1.Source: §3 · p. 3
  • The evaluation benchmark contains 95 randomly generated scenes across 15 pre-training object categories, with layouts and backgrounds unseen during pre-training.Source: §3 · p. 3

4. Pilot Study: Different Observation Spaces and Visual Representations

The pilot study compares five observation-space designs under controlled simulation conditions. Point cloud–2D patch fusion performs best, supporting the use of explicit 3D structure alongside 2D features.

4. Pilot Study: Different Observation Spaces and Visual Representations

  • The comparison uses five perception pipelines that construct observations from the same world state, differing in how explicitly geometry is provided and represented.Source: §4 · p. 3, Table 1 · p. 4
  • The controlled study uses simulator ground-truth metric depth, freezes the image encoder, and holds the fusion layer, remaining architecture, and training strategy constant.Source: §4 · p. 3
  • Point cloud–2D patch fusion achieves the best simulator performance, improving Single-Trial SR by 4.3% over the second-best result of 56.8%.The paper argues that compressed point clouds fused with corresponding 2D patches provide more direct and stable spatial constraints and interaction geometry.Source: §4 · p. 4, Table 2 · p. 4
  • The study finds that adding depth as a 2D channel or injecting implicit and reconstruction-based priors yields relatively limited gains, even with perfect simulator depth.The authors attribute this to weak metric alignment and spatial hallucinations in implicit methods, and loss of 3D topology when depth is treated as a 2D channel.Source: §4 · p. 4

5. ANY3D-VLA

ANY3D-VLA combines point-cloud–2D patch fusion with hybrid point-cloud training. Its visual module compresses and encodes point clouds, aligns 3D features to image patches, and adds them to 2D features through gated residual fusion.

5. ANY3D-VLA

  • ANY3D-VLA combines point cloud–2D patch fusion with hybrid point-cloud training.Source: §5 · p. 4

5.1. Overall Architecture

  • An InternLM2 1.8B VLM and a conditional flow-matching action expert are connected by Progressive Action Generation to generate end-effector actions.Source: §5.1 · p. 4

5.2. Visual Observation Module

  • 3D compression bins workspace points into fixed-resolution cells, retaining one representative per cell before the compact cloud is encoded into point-wise features.The point-cloud encoder takes point coordinates plus color and normal attributes; an image encoder independently maps RGB images to patch tokens.Source: §5.2 · pp. 4–5
  • 3D point features are assigned to image patches and scatter-mean aggregated; a gated residual fusion adds aligned 3D tokens to corresponding 2D tokens.The design treats 3D representations as corrections to, rather than replacements for, 2D representations.Source: §5.2 · p. 5

5.3. Training Strategy

  • Training uses GRIT for object bounding-box tokens and synthetic RGBD data for grasp-pose and action supervision, while hybrid point-cloud inputs span simulator, sensor, and RGB-estimated sources.The VLM predicts intermediate discrete tokens before the action expert generates continuous action chunks; exposure to varied point-cloud sources is intended to encourage depth-source-invariant geometry.Source: §5.3 · p. 5

6. Experiments Centered on ANY3D-VLA

Experiments assess ANY3D-VLA in real-world and simulation settings, across point-cloud sources, and in few-shot adaptation with limited real-world data.

6.1.2. ZERO-SHOT COMPARISONS IN THE REAL WORLD

  • 62.5% overall average success rate for (Setting 2, DA3), a 29.2% improvement over SpatialVLA’s 33.3%, leads the zero-shot real-world comparison.Figure 2 compares simulator-only versus hybrid training and RealSense versus DA3 inference.Source: Figure 2 · p. 6, §6.1.2 · p. 7

6.1.1. REAL-WORLD SETUP

  • Against π0.5, GraspVLA, and SpatialVLA, real-world zero-shot tests cover 47 objects and 120 trials across four challenge sets.Depth Anything 3 was selected for real-world experiments by jointly considering prediction accuracy and inference latency.Source: §6.1.1 · p. 6, §6.1.2 · p. 6

6.1.3. REAL-WORLD POST-TRAINING

  • Hybrid point-cloud training with DA3 inference achieves 93.3% on Task 1 and 86.7% on Task 2, outperforming baselines on both post-training tasks.At the same inference source, hybrid training consistently beats RealSense-only training.Source: Table 3 · p. 7, §6.1.3 · p. 7

6.2. Inference Efficiency and Latency

  • Despite point-cloud estimation overhead, action chunking amortizes perception latency; 1.7–2.0 FPS remains feasible for tabletop manipulation.The policy uses chunks of four actions and executes roughly 2–3× longer motions per step than higher-frequency policies.Source: §6.2 · pp. 7–8, Table 4 · p. 7

6.3. Diverse Point-Cloud Inputs as Data Augmentation

  • Across simulation and real-world comparisons, hybrid point-cloud training consistently matches or outperforms single-source training under comparable conditions.The authors describe diverse point-cloud inputs as data augmentation that helps mitigate the sim-to-real gap.Source: Table 5 · p. 8, §6.3 · p. 8

6.4. Ablation Study

  • 61.1% Single-Trial SR with full fusion, versus 45.3% (2D-only) and 44.2% (3D-only), supports combining 2D semantics and 3D geometry.The 3D-only model showed weaker language-target localization, while 2D-only was less stable with small objects or heavy occlusion.Source: Table 6 · p. 8, §6.4 · p. 8

6.5. LIBERO and CALVIN Benchmarks

  • ANY3D-VLA improves over GraspVLA by 13.9% on LIBERO and increases CALVIN average length by 0.71, while remaining slightly better than SpatialVLA overall.Source: §6.5 · p. 8

7. Limitations and Future Work

The paper identifies limits in its real-world evaluation and in its chosen 3D representation strategy. It presents the implementation as one promising option, not the only viable approach.

7. Limitations and Future Work

  • Real-world experiments cover only one robotic arm and a limited set of objects.The authors propose extending evaluation to more robot platforms and environments, including complex, long-horizon tasks.Source: §7 · p. 8
  • The method uses relatively simple 3D grid compression and representation extraction, and stronger 3D backbones may yield additional gains.The authors chose not to use more complex architectures to preserve interpretable comparisons of observation spaces and representation design.Source: §7 · p. 9
  • The authors do not exhaust strategies for using depth and point clouds; compressed native sparse 3D representations are presented as promising, not exclusive.The paper frames its specific implementation as one viable choice among many.Source: §7 · p. 9

8. Conclusion

The paper studies observation spaces and visual representations for VLA manipulation. It concludes that combining compressed point clouds with 2D features is an effective path toward stronger spatial perception and manipulation.

8. Conclusion

  • ANY3D-VLA outperforms existing baselines in complex scenarios across simulation and real-world experiments.The paper concludes that integrating mature point-cloud representations with 2D representations is an effective path to stronger spatial perception and manipulation.Source: §8 · p. 9
  • Fusing compressed point clouds with corresponding 2D patches provides direct, stable spatial constraints and interaction geometry for manipulation.The authors contrast this with implicit and reconstruction-based priors, which improve visual representations to some extent but have clear limitations in their comparative experiments.Source: §8 · p. 9
  • A hybrid point-cloud training strategy addresses the data-scaling bottleneck, and the model remains stable when inference uses noisy point clouds.The approach does not require expensive depth hardware at deployment or stringent data collection, according to the authors.Source: §8 · p. 9

Impact Statement

These sections describe Any3D-VLA’s point-cloud and 2D-feature pipeline, its training and evaluation settings, and the controlled scope of its real-world deployment. The paper also recommends safety monitoring and human oversight for deployment beyond its controlled experiments.

Impact Statement

  • The work targets manipulation robustness under noisy depth and viewpoint variation, while warning that deployment may introduce safety risks and affect some manual-labor jobs.Experiments are controlled; the authors recommend safety monitoring, human oversight, and risk assessment for real-world deployment.Source: Impact Statement · p. 9

A. Extended Related Work

  • The paper does not claim the first conceptual use of 2D–3D fusion; it focuses on adapting fusion to VLA manipulation and training with heterogeneous point-cloud sources.It also uses off-the-shelf depth estimators without instruction conditioning, leaving conditioned depth estimation as future work.Source: A. Extended Related Work · p. 13

B. Dataset and Benchmark Details

C. Pilot Study Details

  • The point-cloud–2D setting lifts RGBD observations into camera-coordinate point sets, compresses them in 3D, then aligns and fuses their features with 2D patches.Construction synchronizes geometry and color cropping, back-projects valid pixels using camera intrinsics, and voxelizes points at 1 cm.Source: C. Pilot Study Details · p. 14, D. Point-Cloud Construction and 3D… · p. 16

E. Vision Encoder Details

  • The visual encoders combine DINOv2 and SigLIP2 image features with Concerto point-cloud features, including a learnable empty token for patches without corresponding points.Most Concerto parameters are frozen, with its last four sparse-convolution layers unfrozen for partial fine-tuning.Source: E. Vision Encoder Details · p. 16

F.1. Joint Training of VLM and Action Expert

  • Joint training mixes GRIT and synthetic VLA samples: GRIT supervises bounding boxes, while synthetic samples supervise boxes, grasp poses, and action flow matching.Depth maps and point clouds receive no explicit supervision; they affect the model through fused features.Source: §F.1 · pp. 16–17

F.2. Data Augmentation and Pose Perturbations

  • Training uses image-padding and pose perturbations, mixes point-cloud sources across pre- and post-training, and reports depth-source noise and scale variation.The reported optimization setup uses AdamW with a 1.6 × 10−4 learning rate and 16 H800 GPUs for three days.Source: §F.2 · p. 17, §F.3 · p. 17, Table 7 · p. 17

G. Evaluation of ANY3D-VLA in the Simulator with Point Clouds from Different Sources

  • In simulation, ANY3D-VLA remains competitive with imperfect estimated point clouds; real-world deployment uses a Franka Panda arm and a front-view RealSense camera.The evaluation uses the same RGB input across point-cloud sources, and the deployment setup places test objects in a 40cm × 50cm × 20cm workspace.Source: G. Evaluation of ANY3D-VLA in the… · p. 17, Table 8 · p. 18, §I · p. 19

J. Qualitative Analysis and Limitations

The qualitative analysis contrasts baseline localization failures with issues shared across models. Point-cloud representations mitigate several baseline problems, but grasping and collision failures remain.

J. Qualitative Analysis and Limitations

  • Baselines show grasp-position drift, worse offsets under occlusion, and localization failures when targets resemble the background; point-cloud representations substantially mitigate these issues.The authors suggest these baseline errors indicate reliance on non-robust spatial shortcut cues.Source: J. Qualitative Analysis and Limitations · p. 19
  • All models can struggle with smooth-surface slips, precise grasping under occlusion, and occasional gripper collisions that leave the robot stuck.The authors suggest tactile feedback or finer grip-force control, active perception or more cameras, and reinforcement learning or collision-constrained planning, respectively.Source: J. Qualitative Analysis and Limitations · p. 19

K. Comparisons on LIBERO and CALVIN Benchmarks

Across LIBERO and CALVIN, ANY3D-VLA improves on GraspVLA, while adding its 3D branch also improves π0.5 on both benchmarks.

K. Comparisons on LIBERO and CALVIN Benchmarks

  • The simulator evaluations use metric depth; π0.5 and SpatialVLA fine-tune public pretrained weights, while GraspVLA and ANY3D-VLA first pretrain on synthetic RGBD data.GraspVLA's released pretrained weights were tailored to a dual-view setup and grasping tasks.Source: K. Comparisons on LIBERO and CALVIN… · p. 19
  • ANY3D-VLA's best average success rate is 13.9% above GraspVLA's and 0.3% above SpatialVLA's on LIBERO.The reported measure is the average across tasks and initial states.Source: K. Comparisons on LIBERO and CALVIN… · p. 19, Table 10 · p. 20
  • On CALVIN's zero-shot evaluation in environment D, ANY3D-VLA improves average length by 0.71 over GraspVLA.The test comprises 1,000 instruction chains of five sequential tasks.Source: K. Comparisons on LIBERO and CALVIN… · p. 20, Table 11 · p. 20
  • CALVIN trains on environments A–C and zero-shot evaluates in D, while LIBERO jointly fine-tunes across its four suites.LIBERO evaluates each task from 50 random initial states; CALVIN uses 1,000 five-task instruction chains.Source: K. Comparisons on LIBERO and CALVIN… · pp. 19–20
  • The authors attribute ANY3D-VLA's remaining gap to π0.5 to differences in training data and benchmark environment configurations.The stated differences include action space, gripper parameters, and camera field of view.Source: K. Comparisons on LIBERO and CALVIN… · p. 20
  • LIBERO average success rate rises from 82.9% to 87.1% with DA3 depth and 87.3% with Simulator depth; CALVIN average completion length rises from 3.22 to 3.41 and 3.43, respectively.These are results from introducing a 3D branch into π0.5.Source: K. Comparisons on LIBERO and CALVIN… · p. 20

L. Additional Discussion: Limitations and Future Work

The discussion identifies unresolved representation and training choices, and suggests extending the approach beyond VLA policies.

L. Additional Discussion: Limitations and Future Work

  • The authors identify disentangling localization from spatial understanding as a future direction for manipulation-oriented representations.They note that spatial foundation models can provide image-plane correspondences and localization cues without necessarily producing manipulation-optimal representations.Source: L. Additional Discussion: Limitations… · p. 20
  • The demonstrated simple mixture of point-cloud sources is beneficial, but source scheduling and weighting for robustness remain open questions.The authors suggest exploring more refined strategies during pre-training and post-training.Source: L. Additional Discussion: Limitations… · p. 21
  • The authors propose exploring the method on non-VLA robot policies that use depth or point clouds for geometric reasoning.The cited policy examples include [Ke et al., 2025; Ze et al., 2024; Qian et al., 2025; Gervet et al., 2023; Murali et al., 2025; Fang et al., 2020].Source: L. Additional Discussion: Limitations… · p. 21
Loading 2602.00807v2…