Source-linked AI summary
Planar Prior Assisted PatchMatch Multi-View Stereo
Qingshan Xu, Wenbing Tao
TL;DR
Depth estimation in low-textured regions remains a challenge for dense multi-view stereo reconstruction. This paper embeds planar priors in PatchMatch MVS using a probabilistic graphical model and a matching cost that combines photometric consistency with planar compatibility; the authors report state-of-the-art dense reconstructions on ETH3D.
Problem
Depth estimation in low-textured regions remains challenging for dense MVS, and PatchMatch methods without explicit planar priors still encounter failures there.
Source: Introduction · p. 1Method
The framework embeds planar models in PatchMatch MVS through a probabilistic graphical model and a multi-view matching cost combining photometric consistency with planar compatibility.
Source: Introduction · p. 2Results
The authors report state-of-the-art dense 3D reconstructions on ETH3D and show that their method estimates depth in low-textured areas well.
Source: Introduction · p. 2, Experiments · p. 6Takeaways & Limitations
The method achieves a better accuracy-completeness tradeoff, which the authors say makes its reconstructed 3D models applicable in actual environments.
Source: Experiments · p. 7The authors report performance comparable to ACMM and identify combining their method with multi-scale geometric consistency as future work.
Source: Conclusion · p. 8
Abstract
from arXiv · showhide
The completeness of 3D models is still a challenging problem in multi-view stereo (MVS) due to the unreliable photometric consistency in low-textured areas. Since low-textured areas usually exhibit strong planarity, planar models are advantageous to the depth estimation of low-textured areas. On the other hand, PatchMatch multi-view stereo is very efficient for its sampling and propagation scheme. By taking advantage of planar models and PatchMatch multi-view stereo, we propose a planar prior assisted PatchMatch multi-view stereo framework in this paper. In detail, we utilize a probabilistic graphical model to embed planar models into PatchMatch multi-view stereo and contribute a novel multi-view aggregated matching cost. This novel cost takes both photometric consistency and planar compatibility into consideration, making it suited for the depth estimation of both non-planar and planar regions. Experimental results demonstrate that our method can efficiently recover the depth information of extremely low-textured areas, thus obtaining high complete 3D models and achieving state-of-the-art performance.
Introduction
The paper proposes planar-prior-assisted PatchMatch MVS to improve depth estimation in low-textured areas while retaining efficient depth-map estimation. Its matching cost combines photometric consistency and planar compatibility, and the authors report state-of-the-art dense reconstructions on ETH3D.
Introduction
- Low-textured areas challenge MVS because photometric consistency is unreliable, while planar-regularized global methods are inefficient and can become trapped in local optima.PatchMatch MVS is efficient but does not explicitly model planar priors, so it can also fail in low-textured areas.Source: Introduction · p. 1
- Sparse credible correspondences are triangulated into planar models, which can also adapt to non-planar regions through differently sized triangular primitives.A probabilistic graphical model combines planar compatibility, which constrains predicted depths to an appropriate range, with photometric consistency that reflects depth changes in well-textured areas; multi-view geometric consistency rectifies errors from unreliable planar priors.Source: Introduction · p. 2
- The proposed framework combines PatchMatch efficiency with planar priors to improve depth estimation in low-textured areas.The authors report state-of-the-art dense 3D reconstructions on the ETH3D benchmark (Sch¨ops et al. 2017).Source: Introduction · p. 2
Related Work
Prior work uses PatchMatch MVS for efficient depth estimation and planar models to support reconstruction, but photometric consistency is unreliable in low-textured areas. This work combines triangulation-based planar priors with PatchMatch MVS while accounting for cross-view visibility and geometry constraints.
PatchMatch Multi-View Stereo
- PatchMatch MVS methods use sampling and propagation for depth estimation, but their reliance on photometric consistency often fails in low-textured areas.Prior methods also jointly estimate depth and view selection, use diffusion-like propagation, or adopt adaptive checkerboard sampling and pixelwise view selection.Source: Related Work · p. 2
- Multi-scale and superpixel-plane methods seek reliable discrimination in low-textured regions, while this work uses triangulation-based piecewise planar priors to adapt to different areas.Multi-scale approaches can be limited by predefined scales, particularly for large low-textured regions.Source: Related Work · p. 2
Planar Priors
- Earlier planar-prior methods use global smoothness energies, planar classification, or stereo triangulation, whereas embedding triangulated priors in MVS must account for visibility and geometry across views.Second-order smoothness formulations lead to triple-clique representations that make optimization difficult; stereo triangulation methods can reduce disparity search space and recover low-textured surfaces.Source: Related Work · p. 2
Planar Prior Assisted PatchMatch MVS
The method estimates each reference image’s depth by combining sparse-correspondence-derived planar models with photometric consistency in a PatchMatch MVS pipeline. It uses geometric consistency to further optimize depth maps and follows a conventional pipeline to fuse them.
Planar Prior Assisted PatchMatch MVS
- The method estimates each reference image’s depth using sparse correspondences and builds on checkerboard propagation, which Xu and Tao (2019) report as more efficient and effective than sequential propagation.It first generates and thresholds correspondences, triangulates them into planar models, then combines those priors with photometric consistency in PatchMatch MVS.Source: Planar Prior Assisted PatchMatch MVS · pp. 2–3
Planar Model Construction
- Sparse credible correspondences are triangulated into adaptive triangular primitives whose planes preserve local structure and provide priors for depth estimation.Primitives are relatively small in well-textured regions and large in low-textured regions; pixels within a primitive share its plane parameters.Source: Planar Model Construction · p. 3
Planar Prior Assistance
- A probabilistic graphical model combines planar priors and photometric consistency in a novel multi-view matching cost, letting each guide hypothesis updates where the other is unreliable.Photometric consistency has greater influence in well-textured areas, while the planar prior becomes more influential in low-textured areas.Source: Planar Prior Assistance · p. 4
Geometric Consistency
- Multi-view geometric consistency addresses errors from unreliable planar priors by regularizing reprojection error with an occlusion-robust truncation threshold.The regularizer uses the reprojection error between the reference and source images induced by a plane hypothesis.Source: Geometric Consistency · p. 5
The Algorithm
- The pipeline generates initial depth maps, selects credible correspondences, triangulates planar models, estimates plane-aware maps, and then optimizes them with geometric consistency.The reported iteration counts are 3 for photometric, 3 for planar-prior photometric, and 2 for geometric-consistency processes.Source: The Algorithm · p. 5
Fusion
- Because individual depth maps contain noise and outliers, the method follows a conventional PatchMatch pipeline for fusion.Source: Fusion · p. 5
1: for each image do
The method builds planar priors through photometric and planar-prior PatchMatch stages, then uses their hypothesis maps to initialize geometric refinement. It fuses the resulting depth estimates across neighboring images to produce the final point cloud.
1: for each image do
- Photometric PatchMatch estimates hypotheses, sparse credible correspondences generate planar models, and planar-prior PatchMatch maps initialize a final geometric refinement stage.Each stage performs hypothesis sampling and propagation, updates the map with its stage-specific cost, and refines the hypotheses.Source: §1 · p. 5
- Fusion retains estimates with relative depth difference below 0.01, normal difference below 10°, and reprojection error below 2 pixels, then averages 3D points with two consistent neighbors.Each image is used as a reference in turn; depth estimates are unprojected, projected into neighboring images, and checked for consistency.Source: §1 · p. 5
Experiments
On ETH3D, ACMP combines a planar-prior-assisted matching cost with geometric consistency to improve depth estimation and produce point clouds with a strong accuracy–completeness tradeoff. The experiments also report its runtime and evaluate depth maps and reconstructed point clouds.
Datasets and Evaluation Metrics
- The evaluation uses ETH3D high-resolution multi-view training and test datasets, measuring depth-map errors below 2 cm and 10 cm and point-cloud accuracy, completeness, and F1.Training depth maps are compared with ground truth; reconstructed test point clouds are submitted to the benchmark server.Source: Experiments · pp. 5–6
Depth Map Evaluation
- On ETH3D training datasets, ACMP\G outperforms COLMAP, openMVS, and ACMH, estimating low-texture depths while handling non-planar regions.Its aggregated matching cost considers both photometric consistency and planar compatibility.Source: Table 1 · p. 6, Figure 4 · p. 6, Experiments · p. 6
- ACMP’s geometric consistency addresses noise from inappropriate planar models, while adaptive triangular primitives help it outperform ACMM on the offi. dataset.ACMM’s limited scales and loss of image information at its coarsest scale can impair low-texture depth estimates.Source: Experiments · pp. 6–7, Figure 4 · p. 6
Point Cloud Evaluation
- At the 2 cm threshold, ACMP achieves the best F1 score among the compared methods and balances point-cloud accuracy and completeness better than openMVS.openMVS has the best completeness, but its relaxed consistent-view requirement produces noisy point clouds with low accuracy.Source: Table 2 · p. 7, Experiments · p. 7
Runtime Analysis
- Planar-model construction takes little runtime, and ACMP\G achieves better reconstruction than original-resolution PatchMatch methods with little computational cost.The planar-prior assistance and geometric-consistency stages have running times close to sparse-correspondence generation because they use the same PatchMatch pipeline.Source: Table 3 · p. 7, Experiments · p. 7
Conclusion
The conclusion presents a planar-prior-assisted PatchMatch MVS framework that integrates planar models through a probabilistic graphical model and a matching cost combining photometric consistency with planar priors. It reports state-of-the-art ETH3D performance, results comparable to ACMM, and better performance than ACMH.
Conclusion
- On ETH3D, the method achieves state-of-the-art performance, is comparable to ACMM, and is much better than ACMH.Source: Conclusion · pp. 7–8
- Sparse credible correspondences form planar models that capture low-textured depth ranges but remain biased, especially in non-planar regions.Source: Conclusion · p. 7
- A probabilistic graphical model embeds planar priors into PatchMatch MVS, yielding a multi-view cost that combines photometric consistency and planar compatibility for planar and non-planar regions.Source: Conclusion · p. 7
- The authors attribute ACMM's performance gain mainly to multi-scale geometric consistency and propose combining that framework with their method in future work.Source: Conclusion · p. 8