PaperlayerView exact revision on arXiv

Source-linked AI summary

Drive3R: Block-wise Streaming Geometric Foundation Models for Autonomous Driving

Tianyi Yang, Haojun Yang, Jipeng Xiao, Xuesong Chen, Shaoshuai Shi, Li Jiang, Zhipeng Zhang

arXiv:2610.15112v1cs.CV

TL;DR

Long driving logs make global-attention geometric reconstruction computationally intractable, while existing streaming approaches face drift or continuity problems. Drive3R uses block-wise streaming and confidence-aware re-anchoring, and reports state-of-the-art results across nuScenes, KITTI, and Waymo.

  • Problem

    Global-attention feed-forward models cannot feasibly process continuous driving logs of thousands of frames under GPU-memory limits and quadratic attention complexity.

    Source: §1 · pp. 1–2

  • Method

    Drive3R combines bidirectional attention within streaming blocks with cross-block feature reuse and confidence-triggered cache resets stitched by SE(3) alignment.

    Source: §1 · p. 2, §3.4 · p. 6

  • Results

    Drive3R establishes state of the art across nuScenes, KITTI, and Waymo; on nuScenes, it improves DVGT-2 AbsRel from 0.1806 to 0.1081.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results demonstrate the value of adapting feed-forward geometric foundation models to continuous driving logs for dense geometry and ego-motion estimation.

    Source: §4.1 · p. 8, §5 · p. 10

    The formulation assumes continuous driving logs, metric-scale geometry, and available calibrated camera intrinsics and extrinsics.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

High-fidelity closed-loop simulation for autonomous driving increasingly relies on dense multi-view 3D reconstruction. While recent feed-forward geometric foundation models have revolutionized per-scene optimization, applying them to continuous, long-horizon driving sequences is computationally intractable due to GPU memory limits and the quadratic complexity of global attention. Existing scaling strategies are fundamentally limited, as causal streaming suffers from compounding scale drift and trajectory deviation, while rigid "chunk-then-align" heuristics fragment global context and produce trajectory discontinuities. To address these limitations, we introduce Drive3R, a block-wise streaming geometric foundation model. Its block-wise causal attention resolves the scalability bottleneck of fully bidirectional attention while avoiding the loss of local future context in purely causal streaming. Its confidence-aware re-anchoring further mitigates long-range drift by detecting unreliable streaming states, re-initializing the KV cache, and stitching refreshed segments through SE(3) alignment. Extensive experiments demonstrate that Drive3R establishes a new state-of-the-art on nuScenes, KITTI, and Waymo. Specifically, on nuScenes, it improves over the recent DVGT-2 baseline from 0.1806 to 0.1081 in AbsRel, from 0.6750 to 0.5344 in point-cloud Accuracy, and from 1.5812 to 0.9092 in ATE.

1 Introduction

Long driving logs make global-attention reconstruction intractable, while causal streaming and chunk-then-align have drift or continuity drawbacks. Drive3R combines block-wise streaming with confidence-aware re-anchoring and reports state-of-the-art results across nuScenes, KITTI, and Waymo.

1 Introduction

  • Dense, accurate 3D geometry from multi-view images is a critical prerequisite for next-generation driving neural simulators.The motivation contrasts dense image-based geometry with the sparsity of LiDAR initialization.Source: §1 · p. 1
  • Global bidirectional attention makes single-pass inference over driving logs of thousands of frames computationally intractable.The stated bottlenecks are GPU memory limits and quadratic attention complexity.Source: §1 · pp. 1–2
  • Causal streaming accumulates errors into scale drift and trajectory deviation, while rigid chunk-then-align can sever motion and create boundary discontinuities.Chunk alignment also lacks coherent feature propagation between chunks.Source: §1 · p. 2
  • Drive3R uses bidirectional attention within blocks and causal attention with cross-block KV-cache reuse to preserve local context and propagate historical geometry.This design is presented as a scalable alternative to fully bidirectional inference and isolated chunk processing.Source: §1 · pp. 2–3, Figure 2 · p. 2
  • Confidence-aware re-anchoring detects potentially degraded streaming states, resets the KV cache, and stitches refreshed segments through SE(3) alignment on overlapping frames.The mechanism is intended to mitigate long-range drift while maintaining continuous geometry recovery.Source: §1 · p. 3
  • Drive3R sets state of the art on nuScenes, KITTI, and Waymo, improving nuScenes AbsRel from 0.1806 (DVGT-2) to 0.1081.It also surpasses the cited baselines on nuScenes point-cloud Accuracy and ATE.Source: §1 · p. 3

2 Related Work

Related work moves from feed-forward multi-view reconstruction to long-sequence strategies, but global attention, causal streaming, and fixed chunk alignment each have limitations. Drive3R combines local bidirectional reasoning with streaming and dynamic re-anchoring.

2 Related Work

  • Feed-forward reconstruction Transformers regress pointmaps, depths, and camera poses in one pass, extending from image pairs to joint multi-view inference.VGGT and π3 scaled the approach to scenes with dozens of frames using global bidirectional attention.Source: §2 · p. 3
  • Global bidirectional attention has O(N2) complexity, making these models incompatible with continuous long-sequence inference.DVGT adapts the paradigm to surround-view driving inputs and outputs metric-scale geometry and ego-vehicle poses.Source: §2 · p. 3
  • Per-frame causal methods avoid repeated full-sequence reconstruction but sacrifice bidirectional context, allowing errors to accumulate into scale drift and trajectory deviation.Examples named include CUT3R, StreamVGGT, and DVGT-2.Source: §2 · p. 3
  • Chunk-then-align methods preserve local bidirectional attention but can sever dynamic motion and produce trajectory discontinuities at chunk boundaries.Drive3R instead combines scalable streaming with local bidirectional reasoning and dynamic re-anchoring to correct accumulated drift online.Source: §2 · p. 4

3 Method

Drive3R formulates long-horizon driving geometry as online ego-pose and dense point-cloud prediction, then combines intrinsic-aware multi-view decoding with block-wise streaming and confidence-triggered re-anchoring. Its training supervises geometric outputs and derives confidence targets from trajectory drift.

3.1 Problem Formulation and Overview

  • The task is online prediction of ego-pose trajectories and dense 3D point clouds from calibrated surround-view streams, with long-horizon, metric-scale consistency as driving-specific constraints.Camera intrinsics and camera-to-ego extrinsics are provided as geometric priors.Source: §3.1 · p. 4

3.2 Overall Architecture

  • The architecture adds camera-ray priors to visual tokens, decodes them spatio-temporally, then branches to dense geometry, metric scale, ego pose, and streaming confidence.Its decoder combines spatial attention within camera views with spatio-temporal attention across views and temporally available blocks.Source: §3.2 · pp. 4–5, Figure 3 · p. 4

3.3 Block-wise Streaming Transformer

  • Blocks group K consecutive frames and all camera views; attention is bidirectional within a block but causal across blocks, while KV caching reuses history for incremental inference.This preserves local multi-frame and multi-camera context while avoiding recomputation of the entire history for each new block.Source: §3.3 · p. 5

3.4 Confidence-Aware Re-Anchoring

  • A block confidence score controls cache reuse or low-confidence reset from overlapping frames; the refreshed segment is SE(3)-aligned and transformed into the existing global frame.The confidence branch estimates whether the current streaming state remains reliable; alignment uses predicted poses on overlapping frames.Source: §3.4 · p. 6

3.5 Training Losses

  • Training combines dense geometry, metric-scale, camera-pose, and ego-pose supervision with a confidence target derived from block-averaged trajectory error and binary cross-entropy.The trajectory discrepancy combines translation and rotation errors measured relative to the segment's first frame.Source: §3.5 · pp. 6–7

4 Experiments

The experiments evaluate dense geometry and ego-motion on Waymo and nuScenes, long-sequence ego-motion on KITTI, and ablations of re-anchoring and block length.

4.1 Dense Geometry and Ego-Motion Estimation

  • Waymo achieves the best depth and point-cloud reconstruction (AbsRel 0.1947, δ < 1.25 0.8429, Accuracy 0.5941); nuScenes leads on five of six metrics.The nuScenes gains span depth, point-cloud reconstruction, and ATE.Source: Table 1 · p. 7, Table 2 · p. 7, §4.1 · pp. 7–8
  • Red-box regions show cleaner road surfaces, sharper vehicle shapes, and more consistent foreground structures than per-frame streaming baselines.StreamVGGT produces noisy, incomplete geometry, while DVGT-2 blurs object boundaries and fine structures.Source: Figure 4 · p. 8, §4.1 · p. 8

4.2 Long-sequence Ego-Motion Estimation

  • ATE averages 41.7 on KITTI, 44.8% lower than DVGT-2 [34]; after unreliable states trigger cache reinitialization, refreshed segments are stitched by overlap alignment.Figure 5 shows estimated trajectories more consistent with ground truth on sequences 09 and 10.Source: Table 3 · p. 8, §4.2 · pp. 8–9, Figure 5 · p. 9

4.3 Experimental Analysis

  • Without confidence-aware re-anchoring, ATE rises from 4.1422 to 18.1769, demonstrating the value of dynamic anchor selection.The ablation is on 36-frame, 6-view nuScenes sequences.Source: Table 4 · p. 9, §4.3 · p. 9
  • Increasing block size from 1 to 3 improves Accuracy, Completeness, and CD; larger blocks improve Accuracy but hurt Completeness, motivating the choice of size 3.CD is the sum of Accuracy and Completeness errors; the offline setting runs out of memory.Source: §4.3 · p. 9, Figure 6 · p. 9

5 Conclusion

Drive3R combines block-wise local reasoning, KV-cache reuse, and confidence-aware re-anchoring for long-horizon autonomous driving reconstruction. Experiments report strong performance across dense depth, point-cloud reconstruction, and ego-motion estimation, with online Gaussian scene representations and closed-loop simulation identified as future applications.

5 Conclusion

  • Drive3R preserves local multi-frame and multi-camera reasoning within blocks, reuses historical features through a KV cache, and uses confidence-aware re-anchoring to correct residual long-range drift.The conclusion presents this combination as bridging offline bidirectional reconstruction and causal streaming.Source: §5 · p. 10
  • Experiments show strong performance on dense depth, point-cloud reconstruction, and ego-motion estimation.Source: §5 · p. 10
  • Drive3R may serve as a foundation for online Gaussian scene representations and closed-loop autonomous driving simulation.Source: §5 · p. 10

Related papers

Loading 2610.15112v1…