PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Consistent Video Geometry Estimation

Zhu Yu, Jingnan Gao, Runmin Zhang, Lingteng Qiu, Zhengyi Zhao, Rui Peng, Yichao Yan, Kejie Qiu, Siyu Zhu, Zilong Dong, Si-Yuan Cao, Hui-Liang Shen

arXiv:2605.30060v3cs.CV

TL;DR

Video geometry estimation still struggles to combine spatial accuracy, long-term temporal consistency, and adaptable inference across video contexts. ViGeo addresses this with dynamic chunking attention and completion-based supervision, achieving state-of-the-art performance across video geometry estimation benchmarks.

  • Problem

    Existing video geometry models use fixed temporal access patterns, while sparse, noisy, or scale-ambiguous supervision limits dense and temporally consistent estimation.

  • Method

    ViGeo combines dynamic chunking attention with completion-based refinement to estimate depth, surface normals, and point maps across streaming, full-sequence, and long-video settings.

  • Results

    ViGeo achieves state-of-the-art performance across multiple video geometry benchmarks while maintaining strong spatial sharpness and temporal consistency.

  • Takeaways & Limitations

    ViGeo provides a practical foundation for scalable video geometry estimation within a unified model.

  • Takeaways & Limitations

    High-resolution and 4D video geometry estimation remain challenging, with high-resolution inputs adding computational cost for long sequences.

Abstract

from arXiv · show

This work presents ViGeo, a feed-forward foundation model for recovering spatially dense and temporally consistent geometry from video sequences. Built upon a plain transformer architecture without task-specific architectural modifications, ViGeo supports streaming, full-sequence, and long-video inference within a unified model. The key design is dynamic chunking attention, which exposes the model to both bidirectional and causal temporal contexts during training and allows it to adapt its attention pattern at test time without retraining. To improve supervision quality, we further introduce a completion-based data refinement framework. This framework trains a video depth completion teacher that conditions on sparse and noisy annotations and exploits video/multi-view context to produce dense, temporally coherent, and geometrically reliable training targets. Beyond depth and point maps, ViGeo also predicts surface normals within the same framework. Trained solely on public datasets, ViGeo achieves state-of-the-art performance across online, offline, and long-video depth estimation, surface normal estimation, and video point map estimation.

1 Introduction

ViGeo is a unified feed-forward foundation model for dense, spatially accurate, and temporally consistent video geometry estimation. Its dynamic attention and completion-based refinement address adaptable inference and unreliable supervision across depth, surface normal, and point map prediction.

  • Motivation: Video geometry estimation requires spatial accuracy and temporal consistency over long sequences, yet unified high-fidelity reconstruction and scalable inference remain challenging.These requirements support applications including robotics, augmented reality, autonomous navigation, and video editing.
  • Motivation: Existing models use fixed temporal access patterns, with offline methods relying on future frames and online methods restricted to causal context.This prevents models from adapting attention behavior to the available video context at inference time.
  • ViGeo: ViGeo uses a plain transformer with dynamic chunking attention to train on bidirectional and causal contexts and adapt attention at inference without retraining.The model supports full-sequence, streaming, and long-video inference within a unified framework.
  • Data refinement: A completion-based data refinement framework trains a video depth completion teacher to convert sparse and noisy LiDAR/SfM annotations into dense, temporally coherent, geometrically reliable targets.The framework treats raw annotations as imperfect geometric observations requiring completion and rectification.
  • Evaluation: ViGeo jointly predicts depth, surface normals, and point maps, and achieves state-of-the-art results on most metrics while remaining competitive on the rest using only public datasets.Evaluation covers streaming, offline, and long-video depth estimation, plus surface normal and point map estimation.

2 Related Work

Prior work progresses from monocular geometry estimators toward temporally coherent video geometry and large-scale training, while data engines address limited high-quality 3D supervision. ViGeo builds on this landscape with multi-view depth completion for dense, accurate, temporally consistent annotations.

  • Dense monocular geometry estimation: Early monocular geometry methods are largely limited to in-domain datasets, restricting generalization to unseen environments.
  • Dense monocular geometry estimation: MiDaS introduces an affine-invariant objective that unifies diverse data sources for joint training and improves zero-shot relative depth estimation.
  • Dense video geometry estimation: Dense video geometry estimation targets temporally coherent and spatially accurate geometry, using multi-image depth optimization, global alignment, or DUSt3R-based approaches.
  • Large-scale data training: Large-scale training with advanced backbones has become a powerful paradigm for 3D geometry estimation, but high-quality labeled 3D data remain scarce.
  • Large-scale data training: ViGeo’s data engine uses multi-view depth completion to combine images and sparse measurements into dense, accurate, temporally consistent depth annotations for large-scale training.

3 Method

ViGeo is a unified feed-forward framework for dense video geometry estimation, combining a plain Transformer with dynamic chunking attention for flexible temporal inference. Its completion-based refinement pipeline generates coherent pseudo-labels from noisy depth data, while multi-task losses supervise depth, point maps, and surface normals.

  • 3.1 Architecture: ViGeo uses a plain ViT-style Transformer whose early layers extract frame-level visual tokens and later layers combine intra-frame and dynamic chunking attention.The decoder predicts point maps, depth maps, and surface normal maps for every video frame.
  • 3.2 Dynamic Chunking Attention: Dynamic chunking attention enables bidirectional attention within chunks and causal attention across chunks, allowing one model to support offline, streaming, and chunk-based inference.Full-sequence attention results when L = 1, while single-frame chunks produce strictly causal streaming attention.
  • 3.2 Dynamic Chunking Attention: The same chunking formulation supports scalable long-video processing through KV caching, which reuses past states across chunks to help control memory growth.It also accommodates streaming inputs that arrive in short multi-frame packets.
  • 3.3 Completion-Based Data Refinement: The completion-based refinement pipeline filters unreliable depth measurements, constructs coarse dense priors, and uses a video depth completion teacher to produce dense, temporally coherent pseudo-labels.The pseudo-labels replace raw measurements as supervision for training ViGeo on real-captured data.
  • 3.3 Completion-Based Data Refinement: The refinement stages improve geometric supervision: Poisson reconstruction densifies sparse depth, while the video completion teacher removes artifacts and produces denser, more coherent point clouds aligned with image structures.Raw measurements may contain missing regions and outliers, whereas Poisson reconstruction can introduce flying points and geometric artifacts.
  • 3.4 Training Objective: ViGeo is trained with a multi-task geometry loss that includes a geometry-derived normal loss computed from predicted point maps to encourage locally coherent surface geometry.The teacher uses the same loss formulation as LDCM.

4 Experiments · 4.1 Evaluation Protocol

The evaluation protocol spans multiple datasets and inference settings for depth, point maps, surface normals, camera pose, and 3D reconstruction. Metrics are computed over valid pixels or aligned geometric predictions using task-specific error and accuracy measures.

  • 4.1.1 Evaluation Datasets: Inputs are resized to meet model resolution requirements and predictions are resized back before metric computation, unless otherwise specified.The protocol evaluates multiple benchmark datasets and inference settings, with pixel-wise metrics computed over valid frame-pixel pairs.
  • 4.1.1 Evaluation Datasets: Sintel evaluates monocular and video depth, video point maps, surface normals, and camera pose across full training-split sequences.Sequences contain 21–50 frames at 1024 × 436; depth is capped at 70 meters, and pose uses the clean pass with provided trajectories.
  • 4.1.1 Evaluation Datasets: Bonn and KITTI support monocular, video, long-video depth, and video point map evaluation with fixed standard and long-video sampling protocols.Bonn uses five sequences with 110 standard or 400 consecutive long-video frames; KITTI uses 13 sequences with 110 standard or 300 consecutive long-video frames.
  • 4.1.1 Evaluation Datasets: HAMMER evaluates long-video depth and surface normals, while NYUv2 evaluates monocular surface normals on its official test split.HAMMER uses 11 sequences, sampling 300 consecutive frames for long-video depth and 110 frames with stride 2 for normals; NYUv2 evaluates 654 images cropped to 565 × 427.
  • 4.1.2 Evaluation Metrics: Depth and video point map evaluation reports absolute relative error Rel, threshold accuracy δ1, and point-wise relative error Relp 0.25.For scale-ambiguous dense predictions, predictions are first aligned using a scalar scale factor s.
  • 4.1.2 Evaluation Metrics: Surface normal evaluation reports mean and median angular errors, along with δ11.25◦ for pixels whose angular error is below 11.25◦.Angular error is computed between predicted and ground-truth surface normals.
  • 4.1.2 Evaluation Metrics: Camera pose uses similarity-aligned trajectories and reports absolute trajectory error plus translational and rotational relative pose error at frame interval ∆= 1.ATE is defined as root mean squared translation error, while rotation errors are measured in degrees.
  • 4.1.2 Evaluation Metrics: 3D reconstruction aligns predicted point clouds to ground truth with a similarity transform and point-to-point ICP, then reports accuracy, completeness, and normal consistency.Accuracy and completeness use nearest-neighbor distances in both directions, with mean and median statistics; normal consistency uses bidirectional normal agreement.

4.2 Main Results

ViGeo is evaluated for video depth estimation across Sintel, Bonn, and KITTI against diverse offline and online baselines. The results include scale-invariant depth and long-video depth evaluation using absolute relative error and δ1 accuracy.

  • Video depth estimation: ViGeo is evaluated for video depth estimation on Sintel, Bonn, and KITTI.The study reports quantitative results for these three datasets.
  • Video depth estimation: The comparison covers offline and online baselines, including video depth estimators and 3D reconstruction or depth foundation models.Named baselines include VideoDepthAnything, DepthCrafter, FlashDepth, VGGT, and Pi3.
  • Video depth estimation: Scale-invariant video depth is evaluated with absolute relative error (Rel) and threshold accuracy (δ1).Table 3 reports these metrics on Sintel, Bonn, and KITTI.
  • Long-video depth estimation: Scale-invariant long-video depth is evaluated on Bonn, KITTI, and HAMMER using absolute relative error (Rel) and threshold accuracy (δ1).Table 4 reports these metrics for long-video depth estimation.

Method

ViGeo is evaluated across long-video depth, dense point-map, surface-normal, 3D reconstruction, and camera-pose estimation. Across these settings, it uses dynamic chunking or chunk-based inference to maintain geometry quality under online and long-sequence constraints.

  • Long-video depth estimation: ViGeo processes 300–400-frame Bonn, KITTI, and HAMMER sequences, while VGGT and streaming variants fail with out-of-memory errors.Dynamic chunking attention adapts temporal access at inference time for long-video depth estimation.
  • Surface normal estimation: ViGeo achieves the best mean and median surface-normal results on Sintel, HAMMER, and NYUv2, compared with image- and video-based estimators.Evaluation reports Mean, Med, and δ11.25° under standard protocols.
  • 3D reconstruction: In online 3D reconstruction, ViGeo improves accuracy and completeness over streaming baselines on 7-Scenes and NRGBD while remaining competitive offline.The model recovers spatially coherent geometry under causal inference.
  • Camera pose estimation: Under online camera-pose estimation, ViGeo outperforms streaming baselines across all pose metrics and achieves competitive offline trajectory accuracy.The evaluation uses ATE and relative pose errors on Sintel.

4.3 Analysis and Ablation

The ablations show that refined supervision improves video geometry learning, while dynamic chunking attention enables long-video processing and supports stronger inference capacity than VGGT at comparable speed.

  • Dynamic Chunking Attention: Dynamic chunking attention is evaluated against full-sequence and causal attention under both offline full-sequence and online streaming inference.The ablation compares variants trained with different temporal attention schemes across both inference modes.
  • Completion-Based Data Refinement: Refined supervision improves most metrics under the same architecture and training protocol by converting sparse, noisy observations into dense, temporally coherent targets.The targets are produced by a video depth completion teacher.
  • Completion-Based Data Refinement: The full refinement pipeline produces cleaner, more complete point clouds with better structural coherence than raw measurements or Poisson reconstruction.Raw measurements contain missing regions and outliers, while Poisson reconstruction may introduce flying points and geometric distortions.
  • Long-Video Inference: Standard full-sequence inference scales quadratically in memory and causes OOM errors on long sequences of approximately 400 frames, whereas KV-cache chunking supports arbitrarily long videos.The inference ablation varies chunk length C ∈{16, 32, 48, 64}.
  • Efficiency Comparison: 700–800 images is ViGeo’s full-attention input capacity versus 200–250 images for VGGT, while running speed remains comparable at 10.00 FPS versus 10.96 FPS.Both models have the same parameter count; speed is measured on 16 images at 518 × 518 resolution using a 96 GB NVIDIA H20 GPU.

4.4 Qualitative Results

ViGeo qualitatively recovers cleaner, denser, and more temporally consistent geometry across point-cloud, video-depth, monocular-depth, and data-refinement settings. Compared with existing methods and raw measurements, it better preserves structures, boundaries, layouts, and coherent geometry.

  • Point cloud reconstruction: Across outdoor, indoor, and object-centric scenes, ViGeo recovers cleaner and more complete 3D structures than existing feed-forward approaches.Pi3 often introduces structural noise and checkerboard artifacts, whereas VGGT tends to produce incomplete or fragmented geometry.
  • Point cloud reconstruction: ViGeo produces accurate, realistic point clouds with coherent global geometry, fine local details, preserved object boundaries, and preserved scene layouts.These results demonstrate dense and spatially consistent geometry recovery from videos.
  • Video depth estimation: Across consecutive frames, ViGeo maintains stable depth structures for moving objects and camera-induced scene changes, unlike the noisy and fluctuating outputs of VGGT and Pi3.DepthCrafter is smoother but over-smooths large background regions.
  • Completion-based data refinement: Completion-based refinement converts raw depth maps with large missing regions into dense, geometrically consistent labels and recovers complete surfaces with sharp structures.The refined point clouds restore structures fragmented or absent in raw sensor measurements.

5 Discussion

ViGeo unifies streaming, full-sequence, and long-video inference for dense, temporally consistent video geometry, while its completion-based data refinement framework offers a reusable data engine. Remaining challenges include computational cost, richer 4D representations, privacy and misuse risks, dataset bias, and the need for deployment validation and safeguards.

  • Limitations: High-resolution and 4D video geometry estimation remain challenging, with high-resolution inputs adding computational cost particularly for long sequences.More explicit 4D representations could further improve temporal consistency in dynamic scenes.
  • Contributions: ViGeo supports streaming, full-sequence, and long-video inference within a single feed-forward model for dense and temporally consistent geometry.The paradigm may benefit robotic perception, autonomous navigation, AR/VR, video editing, and 3D scene understanding.
  • Contributions: Its completion-based data refinement framework can serve as a reusable data engine for converting sparse and noisy real-world annotations.
  • Risks: Reconstructing real-world scenes from videos raises privacy concerns and may enable unauthorized mapping or surveillance.
  • Risks: Training on public datasets may transmit dataset biases and reduce reliability in underrepresented environments.Safety-critical deployments require careful validation under target conditions and appropriate safeguards.

6 Conclusion

ViGeo is a feed-forward geometry foundation model that uses dynamic chunking attention and refined supervision to estimate temporally consistent video depth and surface normals. Across multiple benchmarks, it achieves state-of-the-art performance with strong spatial sharpness and temporal consistency.

  • 6 Conclusion: ViGeo estimates temporally consistent depth and surface normals as a feed-forward geometry foundation model.The model unifies these video geometry tasks within a single framework.
  • 6 Conclusion: Dynamic chunking attention unifies streaming and full-sequence inference within one transformer model.This design supports both inference modes without separate task-specific architectures.
  • 6 Conclusion: The data refinement framework converts sparse and noisy annotations into dense, coherent, and geometrically reliable targets.The refined supervision addresses missing regions and reconstruction artifacts in the raw measurements.
  • 6 Conclusion: Across multiple benchmarks, ViGeo achieves state-of-the-art performance while maintaining strong spatial sharpness and temporal consistency.Qualitative results show sharper, more accurate, and temporally stable depth compared with existing methods.
  • 6 Conclusion: The refined pseudo-labels produce spatially dense and geometrically consistent labels that recover solid surfaces and sharp structural details.These details are fragmented or absent in raw sensor outputs.
Loading 2605.30060v3…