DVGT-2: Vision-Geometry-Action Model for Autonomous Driving at Scale

Sicheng Zuo; Zixun Xie; Wenzhao Zheng; Shaoqing Xu; Fang Li; Hanbing Li; Long Chen; Zhi-Xin Yang; Jiwen Lu

Source-linked AI summary

DVGT-2: Vision-Geometry-Action Model for Autonomous Driving at Scale

Sicheng Zuo, Zixun Xie, Wenzhao Zheng, Shaoqing Xu, Fang Li, Hanbing Li, Long Chen, Zhi-Xin Yang, Jiwen Lu

arXiv:2604.00813v3Apr 1, 2026cs.CVcs.AIcs.RO

TL;DR

Autonomous-driving geometry models face a gap between dense 3D understanding and efficient online planning because batch reconstruction redundantly processes historical frames. The paper introduces streaming DVGT-2 within a VGA paradigm, jointly reconstructing geometry and planning while reusing cached temporal features. DVGT-2 delivers strong geometry and planning performance with reduced latency, while global ego-pose estimation remains less competitive and bounded by accumulated drift and limited temporal context.

Problem
Existing geometry-reconstruction methods use computationally expensive batch processing and redundant historical-frame computation, making them unsuitable for online real-time driving.
Method
DVGT-2 adopts the VGA paradigm, jointly predicting dense 3D pointmaps, ego-poses, and future trajectories with temporal causal attention, cached features, and sliding-window streaming.
Results
DVGT-2 achieves strong geometry reconstruction and planning performance across diverse datasets, including direct application to closed-loop NAVSIM and open-loop nuScenes without fine-tuning.
Takeaways & Limitations
Dense geometry can serve as a foundation for end-to-end driving while streaming inference reduces latency and supports efficient online trajectory planning.
Takeaways & Limitations
Global ego-pose estimation is less competitive because lightweight pose reasoning, accumulated relative-pose drift, and a fixed historical window limit accuracy and global context.

Abstract

from arXiv · show

End-to-end autonomous driving has evolved from the conventional paradigm based on sparse perception into vision-language-action (VLA) models, which focus on learning language descriptions as an auxiliary task to facilitate planning. In this paper, we propose an alternative Vision-Geometry-Action (VGA) paradigm that advocates dense 3D geometry as the critical cue for autonomous driving. As vehicles operate in a 3D world, we think dense 3D geometry provides the most comprehensive information for decision-making. However, most existing geometry reconstruction methods (e.g., DVGT) rely on computationally expensive batch processing of multi-frame inputs and cannot be applied to online planning. To address this, we introduce a streaming Driving Visual Geometry Transformer (DVGT-2), which processes inputs in an online manner and jointly outputs dense geometry and trajectory planning for the current frame. We employ temporal causal attention and cache historical features to support on-the-fly inference. To further enhance efficiency, we propose a sliding-window streaming strategy and use historical caches within a certain interval to avoid repetitive computations. Despite the faster speed, DVGT-2 achieves superior geometry reconstruction performance on various datasets. The same trained DVGT-2 can be directly applied to planning across diverse camera configurations without fine-tuning, including closed-loop NAVSIM and open-loop nuScenes benchmarks.

1 Introduction

The paper proposes Vision-Geometry-Action (VGA), which uses dense 3D geometry as a precise foundation for autonomous-driving decisions. DVGT-2 makes this paradigm practical for online planning through streaming inference and historical-feature reuse.

Motivation: VGA advocates dense, pixel-aligned 3D geometry as the foundational representation for precise spatial control and safe driving decisions.The paper contrasts this with sparse perception representations and language descriptions, which may not capture complete geometric details.
Problem: Existing geometry-reconstruction methods redundantly recompute overlapping historical frames during online processing, causing unacceptable latency.These methods typically process entire multi-frame sequences in computationally expensive batches.
Method: DVGT-2 uses relative temporal positional encoding and jointly predicts local-frame dense points and relative ego-pose to aggregate historical geometric cues without repetitive computation.The design supports continuous streaming reconstruction and planning.
Training: DVGT-2 is trained on nuScenes, OpenScene, Waymo, KITTI, and DDAD to provide a foundation for the VGA paradigm.The training mixture spans diverse driving datasets.
Results: Despite faster inference, DVGT-2 achieves superior geometry reconstruction across datasets and transfers directly to planning across camera configurations without fine-tuning.It performs strongly on closed-loop NAVSIM and open-loop nuScenes benchmarks.

2 Related Work

Prior end-to-end driving systems connect perception to planning through sparse representations, trajectory modeling, or language-based reasoning. The paper positions VGA as an alternative that explicitly models dense geometry to connect visual inputs with driving actions.

End-to-End Autonomous Driving: End-to-end autonomous driving maps raw sensor inputs directly to planning trajectories or control signals while jointly optimizing the system.Early systems integrated perception, prediction, and planning into a single framework.
VLA for Autonomous Driving: VLA approaches apply vision-language models to scene understanding, reasoning, and high-level driving decisions that guide downstream planners or end-to-end models.Their uses include predicting meta-actions or providing intermediate supervision.
Paradigm Comparison: The figure contrasts conventional sparse perception, VLA language descriptions, and VGA dense 3D geometry as representations for end-to-end driving.VGA uses dense geometry to facilitate safe planning.
Vision-Geometry-Action: VGA explicitly models fine-grained dense geometry to connect visual inputs to driving actions through structural and dynamic scene understanding.This framework is presented as an alternative to sparse representations and language-based approaches.
Visual Geometry Models: Visual geometry models have progressed from depth estimation toward direct dense pointmap regression from unposed images and multi-view inputs.The related work cites DUSt3R, VGGT, and π3 as examples of this progression.

3 Proposed Approach

The proposed approach combines dense geometry reconstruction with trajectory planning in a streaming Vision-Geometry-Action framework. DVGT-2 uses cached historical features and a fixed sliding window to support efficient online inference while jointly predicting geometry and actions.

Vision-Geometry-Action Model: Sparse representations such as bounding boxes and map elements discard environmental context, while voxelized occupancy introduces quantization errors.These limitations restrict scene modeling for planning.
Vision-Geometry-Action Model: VLA models provide scene descriptions alongside trajectories, but natural language remains too ambiguous and coarse-grained to capture precise geometric details.This limits high-fidelity scene understanding and robust trajectory planning.
Vision-Geometry-Action Model: VGA jointly reconstructs dense 3D pointmaps and ego-poses while predicting the future ego-trajectory from multi-frame images.The framework treats dense geometry as the bridge between visual inputs and driving actions.
Streaming Geometry Reconstruction: Batch reconstruction computes pairwise interactions across frames with O(T^2) complexity and redundantly reprocesses overlapping history during online inference.This creates prohibitive latency for real-time autonomous driving.
Streaming Geometry Reconstruction: Full-history streaming reduces computation to O(T) but retains the entire sequence, causing memory and computation to grow linearly with sequence length.Its global first-frame coordinate system makes it unsuitable for continuous infinite-horizon driving.
Streaming Driving Visual Geometry Transformer: DVGT-2 combines an image encoder, temporal-causal geometry transformer, prediction heads, and a fixed-size cache to jointly produce pointmaps, ego-poses, and trajectories online.The sliding-window strategy maintains constant O(W) per-frame complexity and avoids recomputing historical frames.

4 Experiments

DVGT-2 is evaluated for geometry reconstruction, trajectory planning, and online inference efficiency across diverse driving datasets and benchmarks. It combines strong reconstruction and planning performance with substantially reduced latency and memory use, while global pose estimation remains less competitive.

Experimental Setup: DVGT-2 is trained and evaluated on mixed data from nuScenes, OpenScene, Waymo, KITTI, and DDAD using online multi-view sequences.Evaluation includes geometry, ego-pose, open-loop planning, and closed-loop NAVSIM benchmarks.
Geometry Reconstruction: DVGT-2 achieves state-of-the-art ray depth performance across multiple datasets, outperforming general vision-geometry and driving-specific models.Ray depth measures distance from a 3D point to the current ego center and indicates local geometric accuracy.
Geometry Reconstruction: Despite cumulative errors from local-to-global aggregation, DVGT-2 delivers global point reconstruction performance comparable to existing state-of-the-art methods.Global pointmaps are constructed by iteratively aggregating local predictions using predicted relative ego-poses.
Limitations: Global ego-pose estimation is less competitive because lightweight pose reasoning, accumulated relative-pose drift, and fixed-window context limit long-term consistency.These choices trade global pose accuracy and context for more efficient per-frame inference.
Inference Efficiency: With a fixed window size of 4, DVGT-2 processes 16-frame sequences at 0.27s per frame and maintains O(1) memory for infinite-length online inference.VGGT and DVGT encounter OOM after about 10 frames, while StreamVGGT reaches OOM around 30 frames; DVGT-2 maintains stable latency around 260ms per frame.
Planning: DVGT-2 achieves 88.6 PDMS on NAVSIM v1 and 88.9 EPDMS on NAVSIM v2, while its fine-tuned variant establishes new state-of-the-art results on both benchmarks.The foundation model supports closed-loop planning after training on a large mixture of driving datasets.
Planning: DVGT-2 achieves nuScenes L2 errors comparable to state-of-the-art models and a significantly lower collision rate than models trained with high-level semantic labels.The reported result supports planning without relying on semantic-label supervision.

5 Conclusion

The paper introduces DVGT-2 as a streaming model for the Vision-Geometry-Action paradigm, using dense 3D geometry to support trajectory planning. Its streaming design reduces computational bottlenecks while retaining strong geometry reconstruction and planning performance across diverse benchmarks.

Conclusion: DVGT-2 pioneers the Vision-Geometry-Action paradigm by jointly predicting dense 3D geometry and trajectories for end-to-end autonomous driving.The model uses temporal causal attention, feature caching, and a sliding-window streaming strategy for on-the-fly inference.
Conclusion: DVGT-2 achieves strong geometry reconstruction with significantly reduced latency and strong planning performance across open-loop and closed-loop benchmarks.The conclusion frames dense geometry as the foundation representation for spatial and temporal cues in planning.

A.1 Ablation on Window size

Window size affects global geometry modeling but has little effect on local ray-depth accuracy. Increasing the window helps global point accuracy up to a point, after which accumulated pose error outweighs broader temporal context.

Window Size: Increasing the window size from 2 to 6 improves global point accuracy by enlarging the temporal receptive field.The larger window helps model inter-frame relations and global structures.
Window Size: A window size of 8 slightly degrades global point accuracy because accumulated relative-pose error outweighs the benefit of broader temporal context.The degradation arises during local-to-global point transformation.
Window Size: Ray depth Abs Rel remains constant across window sizes, indicating that window size affects global and inter-frame geometry rather than local geometry accuracy.The ablation separates temporal-context effects from local ray-depth prediction.

A.2 Planning on All Datasets

DVGT-2 provides strong trajectory planning on OpenScene while remaining competitive on NuScenes and Waymo, but performance drops on KITTI and DDAD because of domain mismatch in driving trajectories.

0.20m L2 error is the lowest reported value on OpenScene, which contributes over 75% of the training data.The model maintains competitive performance on NuScenes and Waymo.
KITTI and DDAD have notably higher trajectory errors exceeding 2.0m.The passage attributes this to higher driving speeds and a domain gap in ego-vehicle trajectory distributions.
Anchor clustering and trajectory-distribution modeling are biased toward OpenScene, producing sub-optimal KITTI and DDAD planning.The same bias also affects ego-pose prediction on those datasets.

A.3 Geometry Reconstruction on KITTI

On KITTI, DVGT-2 achieves state-of-the-art ray-depth reconstruction, while trajectory-domain mismatch harms ego-pose and global point accuracy.

DVGT-2 achieves state-of-the-art KITTI ray-depth estimation on both Abs Rel and δ<1.25 metrics.The passage attributes this superiority to local pointmap prediction, which models local geometry effectively.
Domain gaps in ego-vehicle trajectories and KITTI’s small training share lead to sub-optimal ego-pose prediction.These factors particularly affect performance relative to the model’s dominant training distribution.
Predicted ego-pose errors degrade global point reconstruction because local points require ego poses for transformation into global coordinates.Accumulated pose errors therefore affect global point accuracy even when local geometry is modeled effectively.

B Additional Implementation Details

The implementation uses a large geometry-transformer architecture with intermediate-token dense prediction, clustered anchors, and staged optimization on mixed driving data.

The model has approximately 1.8 billion parameters and uses QKNorm and LayerScale in each geometry-transformer attention layer.LayerScale is initialized at 0.01 for training stability.
Intermediate tokens from blocks 4, 11, 17, and 23 feed a DPT head for dense prediction.The architecture separately augments visual tokens for ego-pose prediction and trajectory planning.
Ego-pose and trajectory heads use anchors pre-computed by clustering the training data.The anchors support both prediction tasks.
Training proceeds in two mixed-dataset stages for 160K and 80K iterations, followed by 40K NAVSIM finetuning for DVGT-2-NAVSIM.Optimization uses AdamW with cosine scheduling, a 1e-4 peak learning rate, 8K-iteration warmup, and gradient clipping at 1.

C Dataset Details

DVGT-2 is trained on five driving datasets with varied camera and sequence configurations, while NAVSIM finetuning uses a fixed eight-view, four-frame setup.

The general model mixes nuScenes, OpenScene, Waymo, KITTI, and DDAD, with videos sampled at 2Hz.The training batch ratio is 6:77:6:5:6 across those datasets.
Training randomizes image aspect ratio, camera-view count, sequence length, and batch size within hardware limits.The number of camera views ranges from 2 to 8, with a maximum of 48 images per GPU.
NAVSIM finetuning fixes OpenScene inputs to 8 views, 4 frames, aspect ratio 1.6, and batch size 1.
A validation video demonstrates online geometry reconstruction and trajectory planning from multi-view image sequences.The demo is described as showing high fidelity and consistency.