Source-linked AI summary
Planar Prior Assisted PatchMatch Multi-View Stereo
Qingshan Xu, Wenbing Tao
TL;DR
Dense MVS reconstruction remains difficult in low-textured areas because photometric consistency is unreliable. The paper embeds triangulated planar priors into PatchMatch MVS using a probabilistic graphical model and a multi-view aggregated cost combining photometric and planar evidence. Experiments on ETH3D report state-of-the-art dense 3D reconstructions and strong performance in low-textured regions.
Problem
MVS depth estimation fails in low-textured areas because photometric consistency is unreliable, limiting complete dense 3D reconstruction.
Method
The method embeds triangulated planar models into PatchMatch MVS through a probabilistic graphical model and a cost combining photometric consistency with planar compatibility.
Results
The method yields state-of-the-art dense 3D reconstructions on the ETH3D benchmark and estimates depth in both planar and non-planar regions.
Takeaways & Limitations
Combining planar priors with PatchMatch preserves PatchMatch efficiency while helping recover depth in low-textured areas.
Takeaways & Limitations
The method is comparable to ACMM, and combining it with multi-scale geometric consistency is identified as future work.
Abstract
from arXiv · showhide
The completeness of 3D models is still a challenging problem in multi-view stereo (MVS) due to the unreliable photometric consistency in low-textured areas. Since low-textured areas usually exhibit strong planarity, planar models are advantageous to the depth estimation of low-textured areas. On the other hand, PatchMatch multi-view stereo is very efficient for its sampling and propagation scheme. By taking advantage of planar models and PatchMatch multi-view stereo, we propose a planar prior assisted PatchMatch multi-view stereo framework in this paper. In detail, we utilize a probabilistic graphical model to embed planar models into PatchMatch multi-view stereo and contribute a novel multi-view aggregated matching cost. This novel cost takes both photometric consistency and planar compatibility into consideration, making it suited for the depth estimation of both non-planar and planar regions. Experimental results demonstrate that our method can efficiently recover the depth information of extremely low-textured areas, thus obtaining high complete 3D models and achieving state-of-the-art performance.
Planar Prior Assisted PatchMatch Multi-View Stereo
The paper lists Qingshan Xu and Wenbing Tao as its authors.
- Qingshan Xu is listed as an author.
- Wenbing Tao is listed as an author.
- The listed contact address uses the hust.edu.cn domain.
Introduction
The paper targets failures of dense 3D reconstruction in low-textured areas, where photometric consistency is unreliable. It combines planar priors with PatchMatch multi-view stereo through a probabilistic graphical model and a matching cost that considers both photometric and planar evidence.
- MVS seeks dense 3D scene models from calibrated images, but depth estimation often fails in low-textured areas.
- Unreliable photometric consistency motivates piecewise-planar assumptions for low-textured surfaces.
- PatchMatch methods are efficient but still fail in low-textured areas because they do not explicitly model planar priors.
- The method triangulates sparse credible correspondences into planar models whose primitive sizes can adapt across planar and non-planar regions.
- A probabilistic graphical model induces a matching cost combining photometric consistency with planar compatibility.
Related Work
Prior PatchMatch multi-view stereo work uses sampling and propagation for efficient depth estimation but struggles where photometric consistency provides weak discrimination. Later methods use multi-scale schemes and geometric consistency to improve discrimination in low-textured areas.
- PatchMatch multi-view stereo exploits sampling and propagation to estimate depth efficiently.
- Photometric consistency remains unreliable in low-textured areas, causing depth-estimation failures.
- Multi-scale methods propagate discrimination from low-resolution images, while ACMM adds view selection and multi-scale geometric consistency.
Planar Prior Assisted PatchMatch MVS
The framework augments efficient PatchMatch MVS with triangulated planar priors and a probabilistic matching model, combining photometric consistency with planar compatibility. It targets depth recovery in low-textured areas while retaining photometric influence where texture is reliable.
- Framework: The method estimates each reference-image depth map using PatchMatch MVS with sparse credible correspondences as additional input.The pipeline uses a checkerboard propagation pattern and first obtains correspondences through conventional PatchMatch MVS and thresholding.
- Planar prior construction: Sparse correspondences are triangulated into triangular primitives whose vertices define plane parameters for depth and surface normals.Primitive sizes adapt to scene structure: smaller triangles preserve non-planar regions, while larger planar regions receive shared plane parameters.
- Planar prior assistance: A probabilistic graphical model combines photometric consistency, planar compatibility, source-image visibility, and neighboring-pixel information to derive the matching cost.The model uses source-image observations and visibility variables to infer hypotheses and view selection information.
- Photometric modeling: The likelihood models photometric consistency through source-image matching costs and view-selection weights, but that evidence remains unreliable in low-textured areas.The matching cost aggregates source-image patch comparisons weighted by view selection.
- Planar prior assistance: Photometric consistency dominates hypothesis changes in well-textured areas, while the planar prior becomes more influential when photometric evidence is unreliable in low-textured areas.This design also allows photometric consistency to correct planar-model errors in non-planar regions.
- Geometric refinement: Geometric consistency is added after plane-aware depth estimation to address errors from unreliable planar priors caused by erroneous sparse correspondences.The overall pipeline generates initial maps, constructs planar models, applies the planar-assisted cost, and then further optimizes the maps geometrically.
1: for each image do
For each image, the method initializes and iteratively updates depth hypotheses through photometric and geometric PatchMatch stages, then fuses the estimates into a final point cloud.
- Each image is set as the reference image in turn, with source images used for depth estimation.
- The method randomly initializes a hypothesis map and iteratively performs hypothesis sampling, propagation, and updates.The photometric stage updates hypotheses via Equation (1), followed by refinement via Equation (12).
- A subsequent geometric stage repeats hypothesis sampling, propagation, updating, and refinement using geometric iterations.The geometric stage updates the hypothesis map via Equation (13) and refines it via Equation (12).
- Fusion unprojects each image’s depth estimates into world coordinates and combines consistent 3D points into the final point cloud.Consistency uses projected depths, normals, and image coordinates across neighboring images.
Experiments
Experiments on ETH3D evaluate depth maps, reconstructed point clouds, qualitative quality, and runtime. ACMP improves low-textured depth estimation, balances point-cloud accuracy and completeness, and adds little computational cost.
- Evaluation setup: The evaluation uses ETH3D high-resolution imagery, with depth metrics on training datasets and point-cloud metrics on benchmark test datasets.Depth accuracy is measured by percentages of pixels below 2cm and 10cm error; point clouds are evaluated by accuracy, completeness, and F1 score.
- Depth estimation: ACMP\G estimates low-textured depth well while also handling non-planar regions through photometric consistency in its planar-prior matching cost.The comparison covers 13 high-resolution ETH3D training datasets.
- Depth estimation: ACMP combines ACMP\G with geometric consistency to reduce noise caused by inappropriate planar models.Without geometric consistency, planar priors can introduce noise into estimated depth maps.
- Depth estimation: ACMP performs much better than ACMM on the offi. dataset because it adaptively captures discrimination at different sizes using triangular primitives.The comparison attributes ACMM’s weaker result to limited scales and lost image information at its coarsest scale.
- Point-cloud evaluation: At the 2cm threshold, ACMP achieves the best F1 score among the compared methods, while openMVS achieves the best completeness with lower accuracy.OpenMVS’s relaxed consistent-view requirement produces noisy point clouds, whereas ACMP provides a better accuracy–completeness tradeoff.
- Runtime analysis: Planar-model construction occupies very little runtime, and the other major stages have similar costs because they use the same PatchMatch pipeline.The method without geometric consistency achieves better reconstruction with very little additional computational cost than other original-resolution PatchMatch methods.
Conclusion
The proposed framework combines planar priors with PatchMatch multi-view stereo to improve depth estimation in low-textured areas. Its performance is comparable to ACMM, while outperforming ACMH as a separate module.
- The framework assists depth estimation in low-textured areas by combining planar models with PatchMatch multi-view stereo.It embeds planar models through a probabilistic graphical model and uses a matching cost that combines photometric consistency with planar priors.
- The method achieves performance comparable to ACMM on the reported evaluation.
- As a separate module, the method performs much better than ACMH in the authors’ experiments.The authors suggest combining it with multi-scale geometric consistency in future work to improve reconstruction performance.