Geometry-Grounded Unified 3D Perception for Autonomous Driving

• A unified representation is motivated by the shared need for semantic discrimination, metric geometry, multi-camera consistency, and temporal correspondence across depth, detection, and occupancy tasks.These tasks read out surface, instance, and voxel-wise scene states, respectively.Source: §1 · p. 2
• General-purpose recognition backbones do not explicitly encode metric 3D structure, so geometry is commonly introduced through task-specific downstream modules.Geometry-oriented pretraining can improve geometric awareness but does not yield a multi-view, temporally consistent scene representation.Source: §1 · p. 2
• Vanilla VGGT lacks explicit calibration priors for absolute metric scale and uses costly global cross-image attention that does not separate temporal from cross-view interactions.These gaps motivate a driving-oriented adaptation with calibration-aware encoding and structured spatiotemporal attention.Source: §1 · pp. 2–3
• GeoUP adapts VGGT with self-, temporal-, and view-attention plus raymap encodings, supporting metric depth, 3D detection, and semantic occupancy from a shared latent.The paper also introduces joint multi-task and multi-dataset training across benchmarks.Source: §1 · p. 3
• GeoUP achieves state-of-the-art performance across detection, occupancy, and depth estimation on nuScenes, Argoverse 2, Waymo, KITTI, and DDAD.The paper also reports improved EPDMS for DriveSuprim using GeoUP as its visual backbone on NAVSIMv2.Source: §1 · p. 3

2.1 Camera-based Backbone Pretraining

• Recognition-oriented backbones provide semantic features but emphasize 2D appearance rather than metric 3D structure; geometry-aware approaches often remain single-frame or task-specific.The paper positions its approach as making geometry part of the shared representation for calibrated streaming driving scenes.Source: §2.1 · pp. 3–4

2.2 Task-specific Geometry Modeling

• Depth, detection, and occupancy methods commonly model geometry within separate task-specific pipelines, whereas this work treats them as readouts of one geometry-grounded latent.The readouts cover surface-, instance-, and volume-level perception.Source: §2.2 · p. 4

2.3 Visual Geometry Models

• Visual geometry foundation models learn transferable priors from reconstruction and correspondence objectives, but are not directly tailored to calibrated, temporally continuous driving scenes.GeoUP uses the visual geometry latent as a shared backbone for depth estimation, 3D detection, and semantic occupancy.Source: §2.3 · p. 4

3 Method

• GeoUP adapts VGGT to calibrated streaming multi-camera driving scenes, combining geometry-aware tokens with a spatiotemporal backbone for depth, detection, and occupancy.Source: §3 · p. 4, Figure 2 · p. 5

3.1 Geometry-Grounded Backbone

• Patch-center Plücker rays derived from camera intrinsics and poses are projected into token embeddings, while camera tokens are added to each frame-view pair.Source: §3.1 · p. 5
• The backbone factorizes cross-image interaction into self-attention within images, temporal aggregation across frames from the same camera, and view exchange across cameras at each timestamp.Source: §3.1 · p. 6, Figure 3 · p. 6

3.2 Multi-task Perception Heads

• Task-specific heads decode metric depth, 3D boxes, and semantic occupancy; the camera branch predicts camera-to-reference poses under auxiliary supervision.Source: §3.2 · pp. 6–7

3.3 Training Strategy

• Training first finetunes VGGT with depth and camera branches, then adds detection and occupancy using annotation-availability masks and sets depth and camera loss weights to 0.1.Source: §3.3 · pp. 7–8

4.1 Datasets and Metrics

• Detection spans nuScenes, Argoverse 2, and Waymo; occupancy is evaluated on Occ3D-nuScenes, and depth on KITTI and DDAD using task-specific metrics.The benchmarks cover differing camera configurations, perception ranges, and object categories.Source: §4.1 · p. 8

4.2 Implementation Details

• GeoUP uses compact VGGT-12, selected from every other pretrained layer; unless specified, temporal windows are 4 frames, and joint training samples nuScenes, Argoverse 2, Waymo, DDAD, and KITTI at an 8:8:9:3:1 ratio.The multi-dataset model is trained for twice the training epochs of the single-dataset setting.Source: §4.2 · pp. 8–9

4.3 Main Results

• Joint training improves detection on nuScenes, Argoverse 2, and Waymo; GeoUP† also achieves the best occupancy results and strongest DDAD depth results.On Occ3D-nuScenes, GeoUP† reaches 42.3% mIoU and 47.0% RayIoU; on DDAD, it reaches 0.123 Abs Rel and 87.6% δ < 1.25.Source: Table 4 · p. 9, §4.3 · pp. 9–10, Table 5 · p. 10
• GeoUP improves DriveSuprim EPDMS over the Depth Anything backbone by 0.8 points with the original evaluator and 0.9 points with the corrected evaluator.The planner decoder and training configuration remain unchanged; only the visual backbone is replaced.Source: Table 6 · p. 10, §4.3 · p. 11

4.4 Ablation Study

• Four input frames capture most gains across detection, occupancy, and depth; temporal task heads outperform current-only heads, and performance largely saturates beyond four frames.Pretrained VGGT initialization improves over random initialization, while GeoUP adaptations outperform VGGT-24; joint detection-occupancy training improves both tasks.Source: Table 8 · p. 11, §4.4 · pp. 12–13, Table 9 · p. 12
• Temporal attention outperforms global attention, view attention helps when combined with temporal attention, and Plücker ray embeddings yield the best detection and occupancy metrics.The ablations attribute weaker standalone view attention to limited overlap across cameras.Source: Table 7 · p. 11, §4.4 · p. 12

4.5 Qualitative Analysis

• Across frames, GeoUP maintains steadier static-vehicle boxes and detects a moving vehicle in all four frames, while RayDN misses it in two intermediate frames.Qualitative comparisons also show more complete road layout and surrounding structures in occupancy, plus reconstructed depth point maps retaining 3D structure.Source: §4.5 · pp. 13–14, Figure 4 · p. 13, Figure 5 · p. 13

5 Conclusion and limitations

• GeoUP’s shared 3D scene latent supports metric depth estimation, 3D object detection, and semantic occupancy prediction.The framework combines temporal-view attention factorization, calibration-aware raymap injection, and joint multi-task and multi-dataset training.Source: §5 · p. 14
• Extensive experiments demonstrate state-of-the-art performance across perception tasks.The conclusion presents these results as validating geometry-grounded representation learning for unified autonomous driving perception.Source: §5 · p. 14
• GeoUP’s visual geometry backbone is computationally heavy, limiting inference speed, and its perception heads remain task-specific.The paper identifies lightweight models or compression and a unified decoder as potential directions for addressing these limitations.Source: §5 · p. 14

A Additional Dataset and Implementation Details

• Evaluation uses nuScenes for detection and occupancy, Waymo and Argoverse 2 for detection, and KITTI and DDAD for depth evaluation.KITTI evaluates monocular depth transfer to forward-view scenes; DDAD evaluates dense metric depth and cross-dataset geometric generalization.Source: A Additional Dataset and… · p. 23
• Multi-dataset training uses all five datasets for depth and camera-pose supervision, three for detection, and nuScenes alone for occupancy.Detection uses dataset-specific point-cloud ranges and category heads, while the detection box-regression branch is shared.Source: A Additional Dataset and… · p. 23
• The depth head combines metric-depth regression with gradient regularization, while the detection head uses Hungarian matching, focal classification, box regression, and denoising.The detection head follows RayDN-style objectives and DETR-style assignment and denoising training.Source: A Additional Dataset and… · p. 24
• An auxiliary 2D head provides image-space localization supervision using classification, centerness, box-regression, overlap, and projected-center losses.The stated losses include Quality Focal Loss, Gaussian focal loss, L1 losses, and GIoU loss.Source: A Additional Dataset and… · p. 24
• The occupancy head predicts sparse occupied points and semantic labels, while the camera head uses an L1 camera loss.Occupancy uses focal classification and Smooth-L1 point regression with β = 0.2.Source: A Additional Dataset and… · p. 24
• The unified objective masks losses according to annotation availability and down-weights depth and camera objectives with λdep = λcam = 0.1.The stated weighting applies in the unified training stage.Source: A Additional Dataset and… · p. 24

B Efficiency Analysis

• 2.18 to 0.81 FPS as the temporal window grows from 1 frame to 4 frames.The runtime comparison includes the shared geometry backbone and all task readout heads at 672 × 224 resolution.Source: Table 11 · p. 25, B Efficiency Analysis · p. 25
• 292.7 ms (66.5%) in the 1-frame setting and 1062.1 ms (87.8%) in the 4-frame setting are attributed to the geometry backbone.Detection, depth, and occupancy heads remain around 28.6 ms, 32.5 ms, and 81.6 ms, respectively, across the two settings.Source: Table 12 · p. 25, B Efficiency Analysis · p. 25
• The paper identifies backbone optimization, a lighter base model, and reduced multi-frame computation as direct efficiency directions.It also proposes a more unified decoder that shares computation across task readouts and processes heads in parallel.Source: B Efficiency Analysis · p. 25

C Additional Quantitative Results

• Joint multi-dataset training improves detection and occupancy across benchmarks compared with single-dataset training.Training uses nuScenes, Argoverse 2, Waymo, DDAD, and KITTI, with each dataset supervising heads for which it has annotations.Source: Table 13 · p. 26, C Additional Quantitative Results · p. 26
• Removing camera-pose supervision consistently degrades detection and occupancy performance on nuScenes.The camera-pose branch provides auxiliary supervision during training and is not directly used by downstream perception heads during inference.Source: Table 14 · p. 26, C Additional Quantitative Results · p. 26

• GeoUP's qualitative outputs are reported as reliable across diverse sensor configurations, driving scenes, and task settings.Source: D More Qualitative Results · p. 26
• On Occ3D-nuScenes, GeoUP produces more complete occupancy predictions than OPUS-V2.Source: Figure 7 · p. 27
• Figure 8 shows point maps generated from predicted depth on KITTI and DDAD.Source: Figure 8 · p. 27
• Figure 9 shows 3D detection on nuScenes, Argoverse 2, and Waymo, with ground truth in blue and predictions in red.Source: Figure 9 · p. 28