GaussVLA: Geometry-Aware Spatial Reasoning for Vision-Language-Action Model

Md Selim Sarowar; Md Tanvir Islam; Sungho Kim; Sangtae Ahn

Source-linked AI summary

GaussVLA: Geometry-Aware Spatial Reasoning for Vision-Language-Action Model

Md Selim Sarowar, Md Tanvir Islam, Sungho Kim, Sangtae Ahn

arXiv:2608.24959v1Aug 25, 2026cs.ROcs.CV

TL;DR

Conventional VLA representations lack explicit geometric structure, limiting reliability on tasks requiring precise spatial reasoning. GaussVLA addresses this with 3D Gaussian visual tokens and structured reasoning-aware action decoding, improving spatially demanding manipulation while remaining lightweight.

Problem
Conventional VLA policies encode RGB observations as flat 2D tokens without explicit metric 3D geometry, making precise geometric reasoning and robustness less reliable.
Method
GaussVLA combines Gaussian Spatial Tokenization, which converts depth and visual features into compact 3D Gaussian primitives, with Depth-Aware Chain-of-Thought reasoning integrated into action generation.
Results
GaussVLA improves LIBERO average success by 8.7% relative to π0 and by 13.5% relative to CoT-VLA while using only 200M parameters.
Takeaways & Limitations
GaussVLA provides structured spatial cues and intermediate geometric supervision for spatially demanding manipulation while remaining lightweight.
Takeaways & Limitations
The study focuses on language-conditioned robot manipulation from demonstrations, with expert action trajectories defined over a fixed action horizon and dimension.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models encode visual observations as flat 2D patch tokens that carry no intrinsic geometric structure, and augmenting them with dense monocular depth injects per-pixel scalar values that encode neither surface orientation nor geometric confidence. This leaves the policy with limited structured spatial reasoning for action prediction. We propose GaussVLA, a Mamba-based VLA that incorporates two custom modules: Gaussian Spatial Tokenizer (GST) to lift frozen semantic and depth features into compact 3D Gaussian tokens, pools geometrically salient regions with learned queries, and \emph{Depth-Aware Chain-of-Thought (DA-CoT)} that performs structured, non-autoregressive geometric reasoning under language and flow-time conditioning. Across both simulation and real-world evaluations, GaussVLA demonstrates strong spatial-manipulation performance while remaining parameter-efficient. On LIBERO, it achieves 93.5% average success and 100.0% success on the Spatial suite with only 200M parameters, improving over SpatialVLA by 19.7% relative average success while remaining significantly more parameter-efficient.

1 Introduction

GaussVLA addresses the limited geometric structure and implicit spatial reasoning of conventional VLA policies. It combines 3D Gaussian spatial tokens with non-autoregressive, action-conditioned geometric reasoning in an efficient backbone.

Conventional VLA visual tokens lack explicit metric 3D geometry and treat patches equally despite differences in depth, surface orientation, and geometric reliability.
Per-pixel monocular depth primarily encodes distance, leaving surface orientation and geometric reliability to be inferred implicitly from potentially ambiguous depth patterns.
Current action pipelines encode spatial relationships implicitly, while autoregressive 2D reasoning adds sequential latency and remains decoupled from action decoding.
GST converts visual and depth features into compact 3D Gaussian primitives whose means encode position, covariances encode local geometry, and opacity represents depth-aware confidence.
DA-CoT uses learnable queries to cross-attend to GST tokens under language and flow-time conditioning, anchoring non-autoregressive reasoning to action prediction.
GaussVLA integrates GST and DA-CoT in an efficient sequence-modeling backbone and reports comparable state-of-the-art performance with greater computational efficiency.

2 Related Works

Related work explores discrete, diffusion, geometric, and reasoning-based VLA designs, but each retains efficiency, representation, or spatial-reasoning limitations. GaussVLA combines structured geometry with efficient sequence modeling to address these trade-offs.

Autoregressive action-token methods preserve language grounding but accumulate prediction errors over long sequences and become computationally costly as chunk length increases.
Diffusion-based policies capture multimodal, smooth trajectories but require multiple denoising steps, slowing execution.
Decoupled diffusion or flow-matching action heads improve continuous control, while adding computational overhead through extra heads or repeated denoising.
RGB-D and point-cloud approaches improve spatial awareness but can lose fine-grained morphology and remain sensitive to depth-estimation errors and sensor noise.
Textual 2D Chain-of-Thought reasoning is interpretable but incurs autoregressive latency, whereas recurrent latent reasoning sacrifices auditability and can degrade under excessive unrolling.
Mamba-style selective state-space models provide linear-time sequence processing as an efficient alternative to transformers' quadratic attention and growing KV cache.
GaussVLA addresses the representation and reasoning trade-off by supervising short structured CoT in metric 3D coordinates and coupling its Mamba core to a Gaussian field.

3 Proposed Method: GaussVLA

GaussVLA builds a geometry-aware manipulation policy by converting dual-stream semantic and depth features into compact 3D Gaussian tokens, then applying task- and flow-conditioned non-autoregressive spatial reasoning before action decoding.

3 Proposed Method: GaussVLA: GaussVLA processes RGB observations and language instructions through dual-stream visual encoding, structured tokenization, reasoning, and flow-matching action prediction.Semantic and depth features are transformed into Gaussian spatial tokens, refined with reasoning tokens, and decoded by a Mamba-based policy.
3.3 Gaussian Spatial Tokenization (GST): GST converts dense 2D patch features and predicted depth into compact 3D Gaussian tokens encoding semantic content, spatial position, and confidence-aware local geometry.Patch-center coordinates and corrected depth support back-projection, while Gaussian parameters represent position, geometric structure, and reliability.
3.3 Gaussian Spatial Tokenization (GST): Learned spatial queries aggregate geometry-aware patch tokens into a compact representation passed to DA-CoT and the action decoder.The pooling module uses learnable queries and spatial attention to produce the final GST features.
3.4 Depth-Aware Chain-of-Thought (DA-CoT): DA-CoT uses learnable reasoning queries to attend over GST tokens under joint language and flow-time conditioning, producing structured non-autoregressive spatial summaries.Each reasoning token summarizes a distinct GST subset conditioned on the task and flow time.
3.5 GaussVLA: GaussVLA aligns geometry-aware observation tokens, DA-CoT reasoning tokens, and flow-state action tokens within a single conditional policy backbone.The integrated design refines action states using contextual states, DA-CoT tokens, and time embeddings before predicting conditional flow velocity.

4 Experiments

GaussVLA is evaluated across simulation, robustness, long-horizon control, and real-world manipulation, with strong performance and modest efficiency costs. Ablations show GST provides the dominant gain, while DA-CoT adds complementary improvements.

LIBERO: 93.5% average success rate is achieved on LIBERO, including 100.0% on Spatial, 95.8% on Object, and 95.3% on Goal.This is the best overall average and exceeds QueST by 4.9 points.
Meta-World and CALVIN: GaussVLA remains competitive across Meta-World difficulty splits and ranks second behind GR-1 on CALVIN for one-to-four consecutive tasks.Its average CALVIN sequence length is 1.474, while very long-horizon recovery remains challenging.
Robustness: 0.33 average SR is obtained on LIBERO-PRO despite larger shifts in appearance, position, semantics, instructions, and environments.The benchmark remains highly challenging for large pretrained VLM-based VLAs as well.
Real-world evaluation: 58.8% average success rate is achieved on real-world SO-101 multi-task evaluation, compared with 46.3% for SpatialVLA and 35.7% for ACT.The largest cited category gain is on Sorting, where GaussVLA exceeds SpatialVLA by 15.7 points.
Ablations: GST raises LIBERO performance from 78.1% to 90.5% and LIBERO-PRO performance from 11.2% to 29.0% when added alone.DA-CoT alone improves the vanilla model to 82.1% on LIBERO and 16.7% on LIBERO-PRO; combining both reaches 93.5% and 33.3%.
Efficiency and sensitivity: 12.97 ms latency, 179M trainable parameters, and 4.83 GFLOPs are reported for full GaussVLA versus 10.85 ms, 158M, and 3.50 GFLOPs for vanilla.Increasing GST queries improves performance up to Ng = 128, while DA-CoT benefits saturate at Nr = 4.

5 Conclusion

GaussVLA addresses flat 2D VLA representations with structured 3D Gaussian tokens and depth-aware reasoning. Experiments report improved spatial manipulation, generalization, and physical execution with a lightweight model.

5 Conclusion: GaussVLA lifts frozen semantic and depth features into compact anisotropic 3D Gaussian tokens that provide structured spatial cues and task-relevant geometry.DA-CoT supplies intermediate geometric supervision before action prediction.
5 Conclusion: 8.7% relative improvement over π0 and 13.5% over CoT-VLA are reported on LIBERO with only 200M parameters.This corresponds to about 97% fewer parameters than the cited 7B-scale VLA baselines.
5 Conclusion: GST provides the dominant performance gain, while DA-CoT contributes complementary improvements in reasoning and robustness.Results on Meta-World, CALVIN, and real-world SO-101 further support simulation generalization and physical robot execution.

A.1 Architecture Hyperparameter

The reported GaussVLA runs use a fixed architecture and training configuration listed in Table 6.

A.2.1 Architecture Hyperparameter: Table 6 lists the architecture and training hyperparameters used for all reported GaussVLA runs.The defaults apply unless otherwise stated.

A.2.1 Hardware

Real-world evaluation uses a single SO-101 arm, an RGB-D camera, and consumer-grade workstation hardware for deployment.

A.2.1 Hardware: A single SO-101 6-DOF arm with a parallel-jaw gripper is evaluated using a Realsense RGB-D camera at 640×480 and 60 Hz.Control runs at 15 Hz with action-chunk replanning every H=10 steps.
A.2.1 Hardware: Inference runs on one GeForce 1080 Ti external workstation, demonstrating deployment viability on consumer-grade hardware.This differs from the RTX Pro 6000 Blackwell used for main-paper latency benchmarking.

A.2.2 Tasks

The evaluation includes pick-and-place, stacking, sorting, and camera-shifted in-distribution and out-of-distribution pick-and-place tasks.

Pick-and-Place requires lifting a 3-cm cube from a 30×30 cm workspace and placing it in a box.
Stacking requires ordering cubes and a small wheel according to language-specified object instructions.
Sorting separates a cube and small wheel into left and right bins.
Pick-Place ID uses the training camera pose, whereas OOD shifts the camera by ±5 cm and ±10° rotation.

A.2.3 Training

Training uses 50 teleoperated demonstrations per task while keeping the SigLIP and Depth-Anything-V2 encoders fully frozen.

50 demonstrations per task are collected via teleoperation.
SigLIP and Depth-Anything-V2 visual and depth encoders remain fully frozen during training.

A.3 Per-Component Latency Breakdown

The reported analyses characterize inference latency, geometric representation quality, backbone and decoder effects, GST and DA-CoT ablations, and action-horizon trade-offs. Together, they show that structured geometry-aware processing improves performance, while latency can be amortized across action chunks.

Per-Component Latency Breakdown: 12.97 ms is the total end-to-end inference latency per step under batch size 1 and BF16 precision on an NVIDIA RTX Pro 6000 Blackwell.
Per-Component Latency Breakdown: 8.50 ms comes from frozen encoders, while trainable modules add 4.47 ms; with H = 10, amortization yields approximately 5.32 ms per control step, or about 188 Hz.
Geometric Representation Probes: DA-CoT tokens encode richer 3D geometric structure than flat 2D patches or pooled GST representations, according to LIBERO linear-probe regression R2.
Backbone and Action Decoder Ablation: 73.36 LIBERO average with Mamba improves over 67.49 while reducing computation from 9.82 to 3.28 GFLOPs; the action-token query decoder reaches 78.1 at 3.50 GFLOPs.
GST Ablation: 90.5 LIBERO average and 100.0 spatial performance with full GST exceed scalar-depth concatenation, which reduces the average from 78.1 to 73.3; LIBERO-PRO reaches 29.0.
Confidence-Aware Pooling: Removing confidence bias lowers LIBERO average from 93.5 to 91.6 and LIBERO-PRO average from 33.3 to 29.7, isolating a measurable robustness contribution from opacity-weighted pooling.
Meta-World Successful Frames: A Very Hard Meta-World example demonstrates object localization, grasping, and precise placement during a spatially demanding manipulation task.