PaperlayerView exact revision on arXiv

Source-linked AI summary

Pow3R: Empowering Unconstrained 3D Reconstruction with Camera and Scene Priors

Wonbong Jang, Philippe Weinzaepfel, Vincent Leroy, Lourdes Agapito, Jerome Revaud

arXiv:2503.17316v1cs.CV

TL;DR

Existing feed-forward 3D models generally lack a unified way to exploit camera and scene priors available at test time. Pow3R extends DUSt3R with lightweight conditioning for arbitrary subsets of intrinsics, pose, and depth, trained with random modality subsets. Across reconstruction, completion, depth, stereo, and pose tasks, the model consistently benefits from auxiliary information and achieves state-of-the-art results in reported evaluations.

  • Problem

    Existing feed-forward 3D models generally lack a unified mechanism to use arbitrary camera and scene priors at test time.

  • Method

    Pow3R extends DUSt3R with lightweight modules for arbitrary subsets of camera intrinsics, pose, and sparse or dense depth, trained with random modality subsets.

  • Results

    Pow3R consistently improves when auxiliary camera or scene priors are provided and achieves state-of-the-art results across reported depth, reconstruction, and completion evaluations.

  • Takeaways & Limitations

    A single model can exploit varying amounts of available camera and scene information while also supporting high-resolution processing and sparse-to-dense depth completion.

Abstract

from arXiv · show

We present Pow3r, a novel large 3D vision regression model that is highly versatile in the input modalities it accepts. Unlike previous feed-forward models that lack any mechanism to exploit known camera or scene priors at test time, Pow3r incorporates any combination of auxiliary information such as intrinsics, relative pose, dense or sparse depth, alongside input images, within a single network. Building upon the recent DUSt3R paradigm, a transformer-based architecture that leverages powerful pre-training, our lightweight and versatile conditioning acts as additional guidance for the network to predict more accurate estimates when auxiliary information is available. During training we feed the model with random subsets of modalities at each iteration, which enables the model to operate under different levels of known priors at test time. This in turn opens up new capabilities, such as performing inference in native image resolution, or point-cloud completion. Our experiments on 3D reconstruction, depth completion, multi-view depth prediction, multi-view stereo, and multi-view pose estimation tasks yield state-of-the-art results and confirm the effectiveness of Pow3r at exploiting all available information. The project webpage is https://europe.naverlabs.com/pow3r.

1. Introduction

Pow3R extends DUSt3R into a unified 3D vision model that can use arbitrary camera and scene priors at test time. Random modality subsets during training support flexible inference, while auxiliary information improves performance and enables capabilities such as high-resolution processing.

  • DUSt3R provides the foundation through pointmaps and a transformer architecture that supports camera, pose, and depth estimation from uncalibrated images.Its pointmap representation and pretrained transformer are identified as central to its versatility.
  • Pow3R addresses the lack of a unified 3D model by accepting any subset of camera intrinsics, sparse or dense depth, and relative poses.It is designed to support multiple 3D perception tasks within one feed-forward model.
  • Random subsets of modalities are supplied during training so one Pow3R model can operate under different prior-information conditions at test time.
  • With no priors, Pow3R performs on par with DUSt3R; when priors are available, it outperforms DUSt3R and supports extreme cropping through intrinsic-aware inference.

2. Related work

Prior 3D methods either optimize scenes independently or target particular tasks, while Pow3R combines camera and depth guidance within a learning-based reconstruction model. Its architecture places different priors into the encoder or decoder alongside image features.

  • Structure-from-Motion and MVS: Structure-from-Motion optimizes camera parameters and sparse 3D maps scene by scene, while multi-view stereo reconstructs dense surfaces through multi-view triangulation.Learning-based components have been added to these pipelines, but challenging conditions and outliers remain concerns.
  • Guiding 3D: Guidance denotes optionally supplying 3D priors, typically camera parameters or sparse depth, to assist a prediction task.Existing examples include camera-guided depth estimation and sparse-depth completion, usually with task-specific architectures.
  • Guiding 3D: Pow3R is presented as a learning-based model that can integrate any subset of sparse depth, camera intrinsics, and camera extrinsics.
  • Most learning-based 3D methods specialize in tasks such as depth estimation, depth completion, pose estimation, or novel view synthesis.
  • Pow3R architecture: In Pow3R, intrinsics and depth enter the encoder, whereas pose enters the decoder to guide pointmap regression.

3. Methodology

Pow3R extends DUSt3R with a single feed-forward architecture that accepts arbitrary subsets of camera and depth priors. It injects these modalities into encoder and decoder transformers, predicts multiple pointmaps, and supports native-resolution processing and direct relative-pose recovery.

  • Overall architecture: Pow3R takes two images plus any subset of intrinsics, relative pose, and sparse or dense depth to regress a 3D reconstruction.The auxiliary inputs include K1, K2, P1,2, D1, and D2 with associated validity masks for depth.
  • Overall architecture: Pow3R follows DUSt3R but predicts an additional pointmap for the second image in its own coordinate system.The three pointmaps enable recovery of information about both cameras in one forward pass.
  • Overall architecture: A shared Siamese ViT encoder processes each image with optional intrinsics and depth, while paired decoders optionally receive relative pose.The decoders communicate through cross-attention and predict X1,1, X2,1, and X2,2 with confidence maps.
  • Training objective: The model uses linear prediction heads and a scale-invariant, confidence-aware pointmap regression loss.The confidence weighting reduces penalties on harder areas and encourages extrapolation.
  • Adding Versatile Conditioning: Auxiliary modalities are embedded with dedicated MLPs and injected tokenwise into transformer blocks, with relative pose added to decoder CLS tokens.Intrinsics become ray patches, sparse depth is patchified, and experiments find first-block injection sufficient.
  • Downstream capabilities: Camera intrinsics allow Pow3R to process image crops natively, avoiding DUSt3R’s fixed-resolution downscaling, while the second pointmap enables single-pass relative-pose estimation.Relative pose is recovered by Procrustes alignment between the two pointmaps of the second image; this is reported as similar to RANSAC with PnP and an order of magnitude faster.

4. Experimental Evaluation

Pow3R is evaluated across multiple 3D vision tasks and auxiliary-input settings, showing that camera and scene priors improve predictions, enable high-resolution processing, and support controllable inference.

  • Pow3R is evaluated across focal, depth, rotation, multi-view depth, reconstruction, pose, and ablation experiments, mostly without retraining.
  • Guiding the output prediction: Pow3R’s performance consistently improves when additional camera or scene priors are provided, including synergistic effects among modalities.Depth completion on NYUv2 achieves state-of-the-art results across varying sparsity despite the dataset being absent from training; KITTI mean absolute error is below 30cm.
  • High-Resolution Processing: High-resolution pointmap regression uses a sliding-window approach with crop intrinsics, while naive high-resolution input to a low-resolution network degrades performance.The evaluation compares sliding-window processing against naive high-resolution input without downsampling.
  • Controllability: When supplied focal length or relative pose becomes too inaccurate, Pow3R refuses the guidance and produces low confidence instead.The model follows guidance near the ground-truth value but not when it differs substantially.
  • Multi-View Depth Estimation: Pow3R with both pose and intrinsics achieves state-of-the-art multi-view depth performance on ETH3D, DTU, Tanks and Temples, and their average.Intrinsics improve performance more than ground-truth pose alone, whose impact is described as negligible.
  • Multi-View Stereo: 19% relative improvement results from adding ground-truth intrinsics on DTU, while combining intrinsics and pose yields a 36% relative improvement.Adding pose alone slightly degrades results by 4%, whereas jointly providing both modalities demonstrates a synergetic effect.

5. Conclusion

Pow3R guides large 3D vision models with camera and scene priors, yielding consistent gains when auxiliary information is available and enabling high-resolution processing and sparse-to-dense completion.

  • Pow3R guides large 3D vision models such as DUSt3R with camera and scene priors in an implicit manner.
  • Auxiliary information produces consistent performance gains and unlocks high-resolution processing and sparse-to-dense depth completion.A qualitative Megadepth result shows Pow3R reconstructing a statue correctly where DUSt3R fails to capture it accurately.

A. Further Experiments on the Impact of Guiding

Additional experiments on Infinigen reproduce the study of auxiliary-modality effects while avoiding Habitat artifacts and training-set contamination concerns.

  • The Infinigen experiment repeats the auxiliary-modality study used on Habitat.Infinigen is described as artifact-free relative to Habitat and absent from the training set, reducing concerns about contamination or overfitting.

B. Multi-View Depth Estimation results

Pow3R performs multi-view depth estimation competitively, with camera pose and intrinsics providing the strongest reported configuration. Its results surpass DUSt3R and most competing classical and learning-based methods when both priors are available.

  • State-of-the-art comparison: Pow3R with both camera pose and intrinsics outperforms DUSt3R and most classical and learning-based approaches on multi-view depth evaluation.The comparison includes methods using poses, depth ranges, or ground-truth intrinsics.
  • Baselines: The reproduced DUSt3R results are slightly worse than the original report, while Pow3R with RGB only matches the original results.The supplementary text distinguishes reported, reproduced, and Pow3R results.

C. High-resolution processing with Pow3R

Pow3R supports high-resolution and native-resolution processing through crop-based strategies that use camera intrinsics or coarse depth as guidance. These schemes avoid training a network directly at prohibitively expensive high resolutions.

  • Sliding-window processing: Pow3R uses camera intrinsics to process crops in a sliding-window scheme and upsample pointmaps to native image resolution.Intrinsics are densified as rays and injected into the encoder, supporting arbitrary aspect ratios and high-resolution images.
  • Computational trade-off: Pow3R’s crop-based multi-stage schemes process full-resolution images without training directly on high-resolution images.Direct high-resolution processing can degrade performance, while high-resolution training may be computationally prohibitive.
  • Asymmetric sliding: The asymmetric sliding strategy combines a downsampled coarse image with the same image represented as high-resolution crops.This strategy is described for monocular pointmap upsampling.
  • Coarse-to-fine strategy: The coarse-to-fine strategy conditions two high-resolution crops on a coarse depthmap estimated during an initial downscaled pass.Crop pointmaps are scale-invariant, so their scale is recovered using a median scale factor.
  • KITTI case study: The coarse-to-fine approach handles KITTI’s 370 × 1226 resolution and atypical aspect ratio in a zero-shot setting.The supplied passage states that it produces detailed and accurate outputs, but the sentence is truncated.

D. Controllability

Pow3R follows supplied camera guidance when it is close to the ground truth but rejects substantially incorrect guidance. Extreme deviations produce broken pointmaps with very low confidence.

  • Guidance adherence: Pow3R follows incorrect focal or pose guidance near the ground-truth value but stops following it when the deviation becomes too large.Controllability is evaluated by comparing predicted focal ratios and pose geodesic angles against the supplied ground truth.
  • Failure boundary: At the breaking point, the model outputs broken pointmaps with very low associated confidence maps.This behavior is reported for extreme auxiliary-information errors.
  • Practical impact: Sparse depth produces the largest observed improvement in global 3D reconstruction error, especially for image pairs with large depths of field.Intrinsics and pose also provide noticeable improvements.

E. Noises in the ground-truth depth annotation: NYUd - Section 4.1 of the main paper

Pow3R is evaluated on depth data containing inaccurate NYUd ground-truth annotations, especially around edges and fine structures. The paper reports that it can inpaint sparse depth consistently despite these annotation issues.

  • Annotation noise: NYUd ground-truth depth errors often occur at edges and fine-structural details that are difficult to annotate accurately.Figure E marks regions whose ground-truth and predicted depth discrepancy exceeds a defined threshold.
  • Zero-shot evaluation: Pow3R achieves state-of-the-art sparse depth completion on NYUv2 despite not being trained on that dataset.The reported comparison spans varying input sparsity depth ratios.
  • Depth completion: Sparse depth excludes inaccurate annotations, while Pow3R produces high-quality depthmaps when inpainting the remaining sparse measurements.The visualization reports errors in log scale and uses a zero-shot setting.
  • Guided reconstruction: Pow3R extends DUSt3R by incorporating sparse depth, camera intrinsics, and camera pose as auxiliary inputs alongside RGB images.The paper connects this guidance mechanism with improved prediction quality and full-resolution processing.

G. More Qualitative Results

Pow3R produces more detailed qualitative depthmaps than DUSt3R when given camera and sparse-depth priors, improving reconstruction of challenging scene elements.

  • Pow3R consistently produces better depthmaps with fewer failure cases than DUSt3R when auxiliary priors are provided.The comparisons use camera information and 2048 sparse depth values, except where the evaluation specifies otherwise.
  • Pow3R reconstructs a gate more smoothly and accurately than DUSt3R on an indoor Megadepth scene.DUSt3R struggles to generate the gate properly, while Pow3R uses camera intrinsics, pose, and 2048 sparse point clouds.
  • Pow3R more precisely reconstructs a horse’s head than DUSt3R on a BlendedMVS scene.DUSt3R captures the overall scene but fails on this specific object when Pow3R receives camera and sparse-point priors.
  • Pow3R preserves fine details such as a glass door window and small refrigerator objects better than DUSt3R on an ARKit scene.DUSt3R generally produces a good depthmap, but misses these localized structures.
  • Pow3R captures car and tree details better than DUSt3R on a Waymo outdoor scene using camera priors and 2048 sparse LiDAR points.DUSt3R already generates a good RGB-only depthmap, while Pow3R improves detail capture with auxiliary inputs.

DUSt3R

With camera and sparse-depth priors, Pow3R generally improves 3D reconstruction and camera localization over DUSt3R, especially in scenes with difficult viewpoints or depth ambiguity.

  • Pow3R achieves better reconstruction and more accurate camera registration than DUSt3R on an outdoor Megadepth scene.DUSt3R struggles with scale ambiguity and improper registration from two extreme viewpoints.
  • Pow3R generates the 3D scene and cameras accurately on a BlendedMVS indoor scene where DUSt3R mispredicts depth and struggles with a statue.The comparison provides camera intrinsics, extrinsics, and 2048 sparse depth values to Pow3R.
  • Pow3R excels at reconstructing depth of field and camera locations on an outdoor Megadepth scene using camera intrinsics, pose, and 2048 sparse depths.DUSt3R performs worse on both aspects in the qualitative comparison.
  • Both Pow3R and DUSt3R produce faithful reconstructions from two diverging RealEstate10K viewpoints, while Pow3R predicts camera locations better.This evaluation supplies camera intrinsics and extrinsics without depth because RealEstate10K lacks point clouds and depthmaps.
  • Pow3R predicts camera locations more accurately than DUSt3R on an ARKit scene despite both producing reasonable reconstructions.The viewpoints are almost non-overlapping, and both methods generate a reasonable 3D scene.
Loading 2503.17316v1…