PaperlayerView exact revision on arXiv

Source-linked AI summary

PURE: Refining Feed-forward 3D Reconstruction with Unlabeled Post-training

Yaolong Zhu, Peng Li, Jiahao Lu, Chengfeng Zhao, Mengfei Li, Chao Wang, Wenyue Chen, Zekai Gu, Chengwei Ren, Ronggang Wang, Yuan Liu

arXiv:2610.14698v1cs.CV

TL;DR

Feed-forward 3D reconstruction models rely on costly supervision and remain fragile on dynamic in-the-wild videos. PURE post-trains them on unlabeled video by distilling structural priors and refining pose, and experiments report consistent improvements across architectures. The authors conclude that this offers a practical, scalable path for real-world reconstruction, though gains remain bounded by the base architecture.

  • Problem

    Feed-forward models rely on costly depth, camera-pose, or synthetic-rendering supervision, while adapting them to in-the-wild videos without annotations remains challenging.

    Source: §1 · p. 2

  • Method

    PURE post-trains pre-trained feed-forward 3D models on unconstrained unlabeled videos, distilling structural priors from 2D monocular models and refining poses through correspondences.

    Source: §1 · p. 2

  • Results

    PURE consistently improves performance and robustness across state-of-the-art architectures on synthetic and real-world dynamic datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that annotation-free post-training offers a practical, scalable pathway for advancing feed-forward 3D reconstruction in real-world settings.

    Source: §1 · p. 2

    Because PURE does not increase model capacity, gains are ultimately bounded by the underlying model architecture.

    Source: §4.5 · p. 8

Abstract

from arXiv · show

Feed-forward 3D reconstruction models have achieved remarkable progress through large-scale training and strong learned priors. However, they remain fragile when applied to dynamic in-the-wild videos, where non-rigid motion, occlusions, and illumination changes are prevalent. A key limitation lies in their reliance on expensive depth and camera pose annotations, which hinders scaling to diverse real-world dynamic data and restricts generalization. To address this issue, we propose \textbf{PURE}, an unlabeled post-training refinement framework that improves depth and camera estimation without requiring ground-truth supervision. Instead of relying on annotations, PURE elegantly distills robust structural knowledge from state-of-the-art 2D priors into 4D feed-forward architectures. PURE leverages relative depth constraints, affine-invariant geometry distillation, and pose refinement based on feature correspondences to stabilize geometric predictions under challenging dynamic conditions. Extensive experiments on both synthetic and real-world dynamic benchmarks demonstrate that PURE delivers consistent improvements across multiple state-of-the-art architectures. Furthermore, our data-efficient paradigm effectively mitigates domain shifts, successfully unlocking the architectural limits of these base models in challenging, in-the-wild dynamic scenes.

1 Introduction

Feed-forward 3D models are fast and scalable but remain dependent on costly supervision and struggle to generalize to dynamic in-the-wild videos. PURE refines them using unlabeled videos and 2D geometric priors, with reported improvements across architectures and a practical, scalable path for real-world reconstruction.

1 Introduction

  • Classical SfM and visual SLAM use iterative optimization over sparse correspondences and bundle adjustment, but their rigid-scene, parallax, and pipeline requirements constrain robustness and scalability in dynamic videos.The paper frames accurate video geometry and camera motion as a fundamental problem with applications including robotics, autonomous driving, augmented reality, and digital content creation.Source: §1 · p. 2
  • Feed-forward models regress depth and camera parameters efficiently, but costly depth, pose, or synthetic-rendering supervision limits their generalization to real-world dynamic scenes.Existing unsupervised photometric approaches degrade under illumination changes, non-Lambertian surfaces, and highly dynamic objects; per-scene optimization is computationally expensive.Source: §1 · p. 2
  • PURE refines pre-trained feed-forward models on unconstrained videos without ground-truth supervision or slow test-time optimization, distilling structural knowledge from 2D monocular priors.Its relative-depth constraints preserve ordinal depth, local affine-invariant distillation maintains surface structure, and correspondence-based pose refinement enforces multi-view consistency.Source: §1 · p. 2
  • Across synthetic and real-world dynamic datasets, PURE consistently improves performance and robustness across state-of-the-art architectures, supporting a practical and scalable pathway for real-world reconstruction.Evaluations include video depth estimation, monocular depth prediction, and camera pose evaluation.Source: §1 · p. 2

2 Related Work

Optimization-based reconstruction can achieve accuracy but depends on iterative refinement and conditions that limit scalability in dynamic, unconstrained videos. Feed-forward models scale better, yet their dynamic-scene brittleness motivates annotation-free refinement; the figures contrast improved geometry and the transfer of monocular cues while preserving temporal consistency.

2.1 Optimization-based 3D Reconstruction

  • Optimization-based systems extend reconstruction to dense geometry and dynamic content, but their computational cost and careful system-design needs limit scalability to diverse in-the-wild videos.Classical SfM and visual SLAM also rely on rigid scenes, sufficient parallax, and reliable feature matches.Source: §2.1 · p. 2

2.2 Feed-forward 3D Reconstruction

  • Feed-forward reconstruction spans pair-wise and video-based models, but pair-wise processing scales poorly to long sequences and video models remain brittle on dynamic in-the-wild videos without annotations.Figure 1 reports more accurate and temporally consistent geometry after annotation-free post-training; Figure 2 contrasts MoGe's frame-wise relative geometry with feed-forward models' temporal scale consistency.Source: §2.2 · pp. 2–3, Figure 1 · p. 3, Figure 2 · p. 3

3 Methodology

PURE post-trains feed-forward reconstruction models on unlabeled video, refining local geometry and camera poses through complementary constraints. Its methodology combines relative depth and geometry distillation, correspondence-based pose estimation, and training on real and synthetic dynamic scenes.

3 Methodology

  • PURE refines pretrained models’ predicted depth, point maps, and camera poses through local geometry optimization and dense correspondence constraints.The framework’s geometry branch uses ordinal depth, affine-invariant constraints, and intrinsic distillation; its pose branch uses confidence-filtered GIM matches.Source: §3 · pp. 3–4, Figure 3 · p. 5

3.1 Relative Local Geometry Optimization

  • Relative depth ordering and local affine-invariant constraints transfer monocular teachers’ structural cues while avoiding absolute-scale and shift errors that can destabilize geometry.Pseudo 3D points combine the pretrained model’s stable ray directions with monocular depth, also regularizing intrinsics without explicit labels.Source: §3.1 · p. 4

3.2 Pose Estimation via Correspondences

  • GIM correspondences support pose refinement, with dynamic or low-confidence matches removed, cycle consistency filtering ambiguous matches, and inverse-density sampling promoting spatially uniform constraints.The filtering pipeline aims to improve the reliability and global robustness of correspondence constraints.Source: §3.2 · p. 4

3.3 Model Training

  • VGGT, Pi3, and DepthAnything3 are fine-tuned; Pi3’s point head is replaced with a jointly trained multi-level convolutional head to address instability and grid-like artifacts.Only a subset of intermediate layers is updated for VGGT and DepthAnything3 to balance adaptation with training stability.Source: §3.3 · pp. 4–5
  • Training uses unlabeled SpatialVID-HQ and Sekai-Real-Walking-HQ sequences, with MoGe depth, Grounded SAM 2 segmentation, and GIM Roma correspondences providing pseudo-supervision.Synthetic datasets and Waymo driving videos are also included to help prevent catastrophic forgetting and support training across diverse dynamic environments.Source: §3.3 · p. 5

4 Experiments

Experiments evaluate video-depth and camera-pose estimation on dynamic benchmarks, then examine test-set post-training, component contributions, and data scaling. The results show broad improvements, with gains from more data eventually limited by the base model's capacity.

4.1 Implementation Details

  • PURE makes negligible architectural changes, except for a minor Pi3 head replacement, and preserves the base models' inference computational cost.VGGT and DA3 fine-tune intermediate global attention layers; Pi3 also trains a replaced point prediction head.Source: §4.1 · p. 6

4.2 Video Depth Estimation

  • Post-training improves video-depth performance across nearly all datasets and backbones, including Pi3 despite its replaced, smaller retrained point head.The evaluation covers Sintel, Bonn, KITTI, and DyCheck.Source: §4.2 · p. 6, Table 2 · p. 7

4.3 Camera Pose Estimation

  • Camera-pose accuracy improves across all evaluated datasets after annotation-free post-training.Evaluation uses ATE, translational RPE, and rotational RPE after Sim(3) trajectory alignment.Source: §4.3 · p. 6, Table 3 · p. 7

4.4 Post-Training with Test-Set Data

  • Post-training on test-set data improves every evaluated benchmark, but less than post-training on large-scale unlabeled videos.The comparison includes depth and camera-pose performance on three benchmarks.Source: §4.4 · p. 6, Table 4 · p. 7

4.5 Ablation Study

  • Removing unlabeled-video data degrades depth and pose; removing ordinal or local-geometry losses also worsens results, while the full model is best across metrics.The ordinal loss supports relative depth ordering, while local geometry distillation enforces cross-view geometric consistency.Source: §4.5 · p. 7
  • Normalized average Abs Rel is 0.8452 for Small, 0.7461 for Medium, and 0.7229 for Large, with gains saturating as data grows.The average is across four datasets and is normalized to the base model, set to 1.0; saturation is attributed to unchanged model capacity.Source: §4.5 · pp. 7–8

5 Conclusion

PURE adapts feed-forward 3D reconstruction models to unconstrained dynamic videos by distilling relative geometric priors and enforcing correspondence-based pose consistency. The conclusion presents unlabeled post-training as a scalable pathway to more robust 3D reconstruction in the wild.

  • PURE is an annotation-free post-training framework for adapting feed-forward 3D reconstruction models to unconstrained dynamic videos.Source: §5 · p. 8
  • PURE distills relative geometric priors and enforces correspondence-based pose consistency to enhance depth and camera estimation across state-of-the-art architectures.Source: §5 · p. 8
  • The work presents unsupervised post-training as a highly effective, scalable pathway for robust 3D reconstruction in the wild.Source: §5 · p. 8

Method Depth Camera Pose

The supplied passages report scaling behavior with increasing post-training data and list depth and camera trajectory metrics.

  • The listed metrics are Abs Rel, δ<1.25, ATE, Trans, and Rot.Source: Method Depth Camera Pose · p. 9
  • The metric list includes both depth measures, Abs Rel and δ<1.25, and trajectory measures, ATE, Trans, and Rot.Source: Method Depth Camera Pose · p. 9
  • Performance improves significantly with more unlabeled post-training data, then gradually saturates at larger scales due to fixed model capacity.Source: Figure 5 · p. 9

Related papers

Loading 2610.14698v1…