Source-linked AI summary
Cost Volume Pyramid Based Depth Inference for Multi-View Stereo
Jiayu Yang, Wei Mao, Jose M. Alvarez, Miaomiao Liu
TL;DR
Multi-view stereo needs accurate high-resolution depth inference without the memory burden of fixed-resolution cost volumes. CVP-MVSNet builds a coarse-to-fine cost-volume pyramid, beginning with global coarse sampling and refining pixelwise depth residuals. On benchmark datasets, it is reported as 6x faster while achieving better performance than state-of-the-art methods.
Problem
Fixed-resolution cost volumes require memory cubic to image resolution, limiting high-resolution multi-view stereo depth inference.
Method
CVP-MVSNet constructs a coarse cost volume over the full depth range, then iteratively builds partial cost volumes around current estimates to refine depth residuals.
Results
6x faster than Point-MVSNet with six times less memory and similar accuracy for the same depth-map output size, while also producing high-resolution depth maps with better accuracy.
Takeaways & Limitations
The cost-volume pyramid provides a compact, lightweight framework that handles high-resolution images for high-quality 3D reconstruction.
Takeaways & Limitations
The method assumes calibrated reference and neighboring source images with camera intrinsics and extrinsics.
Abstract
from arXiv · showhide
We propose a cost volume-based neural network for depth inference from multi-view images. We demonstrate that building a cost volume pyramid in a coarse-to-fine manner instead of constructing a cost volume at a fixed resolution leads to a compact, lightweight network and allows us inferring high resolution depth maps to achieve better reconstruction results. To this end, we first build a cost volume based on uniform sampling of fronto-parallel planes across the entire depth range at the coarsest resolution of an image. Then, given current depth estimate, we construct new cost volumes iteratively on the pixelwise depth residual to perform depth map refinement. While sharing similar insight with Point-MVSNet as predicting and refining depth iteratively, we show that working on cost volume pyramid can lead to a more compact, yet efficient network structure compared with the Point-MVSNet on 3D points. We further provide detailed analyses of the relation between (residual) depth sampling and image resolution, which serves as a principle for building compact cost volume pyramid. Experimental results on benchmark datasets show that our model can perform 6x faster and has similar performance as state-of-the-art methods. Code is available at https://github.com/JiayuYANG/CVP-MVSNet
1. Introduction
Learning-based MVS improves feature matching but fixed-resolution cost volumes impose severe memory costs, limiting high-resolution depth inference. CVP-MVSNet addresses this with a compact coarse-to-fine cost-volume pyramid and reports faster, accurate reconstruction.
- Fixed-resolution cost volumes require memory cubic in image resolution, limiting their use for high-resolution multi-view stereo.
- Point-MVSNet predicts depth residuals iteratively on 3D points, but runtime increases almost linearly with the number of iteration levels.
- CVP-MVSNet builds a compact cost volume at the coarsest image resolution, then constructs partial cost volumes iteratively around current depth estimates.
- The method builds its cost-volume pyramid coarse-to-fine using an analysis of depth residual search range and image resolution.
- 6x faster than Point-MVSNet, CVP-MVSNet handles high-resolution images with less memory and better accuracy on benchmark datasets.
2. Related Work
Related MVS methods use volumetric, depth-map, or cost-volume representations, each presenting trade-offs in memory, flexibility, or resolution. CVP-MVSNet retains iterative refinement while moving computation onto regular image-grid cost volumes and partial searches.
- Traditional Multi-View Stereo: Volumetric MVS represents scenes with voxels and uses photometric consistency, but space discretization is memory intensive.
- Deep learning-based MVS: Deep learning-based volumetric systems provide promising results but cannot handle large-scale scenes because volumetric representations are memory expensive.
- Deep learning-based MVS: MVSNet-style fixed-resolution cost volumes have memory requirements cubic to image resolution, while recurrent regularization addresses high-resolution constraints sequentially.
- Related cost-volume methods: Point-MVSNet predicts depth coarse-to-fine on point clouds by aggregating information from k nearest neighbors in 3D space.
- Related cost-volume methods: CVP-MVSNet instead builds cost volumes on a regular image grid and uses partial cost volumes to predict depth residuals, producing compact and faster models.
3. Method
CVP-MVSNet infers a reference-view depth map from multi-view images using a feed-forward network over feature and cost-volume pyramids. It estimates depth coarsely from the full range, then refines it across resolutions with residual cost volumes.
- Design Rationale: The feature-extraction and cost-volume design reduces memory requirements while supporting high-resolution depth inference.The paper reports significant memory reduction from its feature pipeline and describes regular-grid cost volumes as compact and efficient.
- Feature Pyramid: The system builds an image pyramid for every input view and extracts shared multi-scale CNN features at each level.The feature maps have spatial dimensions H/2^l × W/2^l and F = 16 channels in the experiments.
- Cost Volume Pyramid: At the coarsest resolution, uniform fronto-parallel depth hypotheses span the scene’s entire depth range to form an initial cost volume.Features are warped between views using differentiable homographies and bilinear interpolation; matching cost is the variance across the reference and source-view features.
- Cost Volume Pyramid: At finer levels, the method upsamples the current estimate and constructs partial cost volumes by searching pixelwise depth residuals around it.The residual interval is Δd_p = s_p/M, where s_p is the pixel’s local search range and M is the number of residual samples.
- Depth Map Inference: Multi-scale 3D convolutions aggregate each cost volume into probability volumes, followed by soft-argmax for coarse depth and iterative residual refinement.The coarse level uses absolute depth, while finer levels use residual depth hypotheses.
- Cost Volume Pyramid: The local residual search range is determined by projecting points two pixels along the epipolar line in both directions and intersecting their back-projected rays with the reference visual ray.This ties depth sampling to image resolution and supplies the search range used for refinement.
4. Experiments
Experiments on DTU and Tanks and Temples show that CVP-MVSNet achieves strong reconstruction quality while reducing runtime and memory, including for high-resolution depth inference. Ablations identify suitable pyramid-level and depth-refinement interval settings.
- DTU dataset: CVP-MVSNet outperforms learning-based methods on DTU in Mean Completeness and Overall reconstruction quality, while achieving second-best Mean Accuracy.Only Galliani et al. provides slightly better Mean Accuracy among the compared geometric-based and learning-based methods.
- Qualitative results: Qualitative DTU results show better completeness than Point-MVSNet, smoother surfaces, and more high-frequency details in edgy areas.The comparisons use point clouds and normal maps, including scan 9 and additional DTU results.
- Efficiency: 6 times faster and 6 times smaller GPU memory usage than Point-MVSNet are reported for similar depth-map output performance.For the same input-image size, CVP-MVSNet achieves the best reconstruction with the shortest time and reasonable GPU memory usage.
- Tanks and Temples: 5% higher mean f-score than Point-MVSNet and only 1% lower than P-MVSNet are reported on Tanks and Temples without fine-tuning.The comparison uses a model trained on DTU and a simple fusion process following MVSNet.
- Ablation study: The 2-level pyramid performs best, while more than 2 levels make the coarsest resolution too small for a good initial depth map.The ablation varies the number of pyramid levels on downsampled images.
- Ablation study: Performance drops when the evaluation depth-refinement interval is too small, with tested intervals corresponding to 0.25 to 2 pixels.The interval is linked to depth sampling determined by corresponding pixel offsets in source views.
5. Conclusion
The paper concludes that CVP-MVSNet is a compact, lightweight framework for high-resolution multi-view stereo depth inference. It reports better benchmark performance while identifying further memory reduction in learning-based structure-from-motion as future work.
- CVP-MVSNet is proposed as a cost volume pyramid-based framework for depth inference in multi-view stereo.
- The model is compact, lightweight, fast, handles high-resolution images, and produces high-quality depth maps for 3D reconstruction.
- The model achieves better performance than state-of-the-art methods through extensive evaluation on benchmark datasets.
- Future work will integrate the approach into learning-based structure-from-motion to further reduce memory requirements.