Source-linked AI summary
Multi-Scale Geometric Consistency Guided Multi-View Stereo
Qingshan Xu, Wenbing Tao
TL;DR
Multiview stereo remains difficult in low-textured and other challenging regions because visual similarity and propagation can be unreliable. The paper proposes ACMH and ACMM, combining structured-region sampling, joint view selection, multiscale geometric consistency, and detail restoration; experiments report state-of-the-art performance while recovering low-textured regions and details.
Problem
Multiview stereo must estimate accurate and complete depth maps despite low texture, occlusions, repetitive patterns, and reflective surfaces.
Method
The method combines ACMH’s structured-region hypothesis sampling and joint view selection with ACMM’s multiscale geometric consistency guidance and detail restorer.
Results
Experiments on extensive datasets report state-of-the-art performance, recovering depth in low-textured areas while preserving details.
Takeaways & Limitations
The approach obtains smooth and consistent depth maps while retaining fine details across challenging multiview stereo settings.
Abstract
from arXiv · showhide
In this paper, we propose an efficient multi-scale geometric consistency guided multi-view stereo method for accurate and complete depth map estimation. We first present our basic multi-view stereo method with Adaptive Checkerboard sampling and Multi-Hypothesis joint view selection (ACMH). It leverages structured region information to sample better candidate hypotheses for propagation and infer the aggregation view subset at each pixel. For the depth estimation of low-textured areas, we further propose to combine ACMH with multi-scale geometric consistency guidance (ACMM) to obtain the reliable depth estimates for low-textured areas at coarser scales and guarantee that they can be propagated to finer scales. To correct the erroneous estimates propagated from the coarser scales, we present a novel detail restorer. Experiments on extensive datasets show our method achieves state-of-the-art performance, recovering the depth estimation not only in low-textured areas but also in details.
1. Introduction
The paper targets efficient, accurate multiview stereo under challenging conditions such as low texture, occlusions, repetitive patterns, and reflective surfaces. It introduces ACMH and ACMM to improve propagation, view selection, low-textured estimation, and detail preservation.
- Multi-view stereo remains challenging with large-scale data, low texture, occlusions, repetitive patterns, and reflective surfaces.
- Diffusion-like checkerboard propagation improves parallelism but can produce less competitive reconstruction results than sequential propagation.
- ACMH adaptively samples candidate hypotheses from structured regions and jointly infers credible aggregation views for each pixel.
- Low-textured patches become more discriminative at coarser image scales, motivating multiscale depth estimation as guidance for finer-scale matching.
- ACMM combines ACMH with multiscale geometric consistency and a detail restorer to estimate low-textured depths while preserving fine details.
- Extensive evaluation reports state-of-the-art performance on the Strecha dataset and ETH3D benchmark.
2. Related Work
Prior multiview stereo approaches use voxel, surface-evolution, patch-based, or depth-map representations, each with limitations. PatchMatch methods improve search efficiency, while this paper addresses propagation, view selection, and multiscale limitations.
- Voxel-based methods are constrained by predefined grid resolution, surface-evolution methods depend on good initialization, and patch-based methods can lack model completeness.
- Sequential propagation offers limited parallelism, whereas checkerboard diffusion-like propagation updates half the image pixels simultaneously.
- Existing view-selection schemes can bias aggregation or ignore pixelwise view selection.
- Prior multiscale patch matching uses variance-based consistency that may be too strong and overlooks errors in details.
3. Overview
The method builds an image pyramid, estimates reliable coarse-scale depths, propagates them upward, and combines geometric consistency with detail restoration at finer scales.
- The input images and calibrated cameras are used to estimate depth maps for all images and fuse them into a 3D point cloud.
- The method constructs an image pyramid with k scales and downsampling factor η, where the finest scale contains the raw images.
- Reliable estimates from low-textured areas are propagated from coarser scales to finer scales while limiting detail loss.
- At the coarsest scale, ACMH first uses photometric consistency and then geometric consistency before depth maps are upsampled.
- At each finer scale, a detail restorer corrects errors induced by upsampling before subsequent estimation.
4. Structured Region Information
ACMH exploits structured region information through adaptive checkerboard sampling and multi-hypothesis joint view selection. It scores structured hypotheses across views, selects suitable views by voting, weights them, and updates estimates by minimum aggregated cost.
- Structured regions are approximated by shared 3D planes, enabling ACMH to sample better propagation hypotheses and select credible views.
- ACMH expands fixed checkerboard neighbors into V-shaped and long-strip sampling regions, then selects eight good hypotheses.
- Each hypothesis is scored across source views in a matching-cost matrix using photometric consistency between reference and source patches.
- Joint view selection votes by column, accepting views with enough low-cost matches and few high-cost matches under iteration-dependent thresholds.
- Selected-view confidence weights make good views more discriminative, while the previous iteration’s most important view retains influence.
- The current depth and normal are updated with the hypothesis having minimum aggregated multiview cost, followed by refinement using random and perturbed hypotheses.
5. Multi-Scale Geometric Consistency
The method combines coarse-scale geometric consistency with finer-scale propagation to improve low-textured depth estimates, then restores details lost during upsampling. Geometric consistency constrains finer-scale optimization, while a photometric-cost difference map identifies and corrects erroneous detail estimates.
- Geometric Consistency Guidance: ACMM uses coarse-scale geometric consistency to obtain reliable low-textured depth estimates and propagate them to finer scales.At each finer scale, upsampled estimates initialize ACMH, while geometric consistency replaces photometric consistency for hypothesis updates.
- Geometric Consistency Guidance: Forward-backward reprojection error measures geometric consistency between reference and source-image depth estimates.The reprojection error is truncated by δ to improve robustness against occlusions, and λ balances geometric and photometric terms.
- Geometric Consistency Guidance: Geometric consistency limits finer-scale hypothesis updates, particularly in low-textured areas, helping preserve reliable estimates propagated from coarser scales.The constrained solution space prevents photometric optimization from disturbing the propagated estimates.
- Detail Restorer: Multi-scale guidance improves low-textured regions but can blur thin structures and boundaries because coarse scales lose image details.A fixed patch window also makes it difficult to balance thin structures against low-textured areas under the local planar assumption.
- Detail Restorer: The detail restorer uses photometric-consistency differences between adjacent scales to detect and replace erroneous propagated estimates.The difference map emphasizes detail errors while suppressing differences from reliable low-textured estimates; estimates are flagged when the difference exceeds ξ.
6. Fusion
The fusion stage converts per-image depth maps into 3D points, retains estimates supported by consistent neighboring views, and averages accepted points and normals into a unified point cloud.
- Fusion: Figure 4 compares absolute-error and photometric-consistency maps for the method without detail restoration, the basic MVS model, and ACMM.Green denotes missing ground truth, red denotes errors larger than 2 cm, and gray denotes errors from 0 to 2 cm.
- Fusion: Fusion casts each image as a reference, converts its depth map to world-coordinate 3D points, and projects them into neighboring views.These projections provide corresponding matches for cross-view consistency checking.
- Fusion: A depth estimate is accepted when at least n ≥ 2 neighboring views satisfy depth, normal-angle, and reprojection constraints.The thresholds are ϵ ≤ 0.01 for relative depth difference, θ ≤ 30° for normal angle, and ψ ≤ 2 for reprojection error.
- Fusion: Accepted consistent estimates are merged by averaging their corresponding 3D points and normal estimates into a unified 3D point.This fusion step produces the complete point cloud from the depth maps.
7. Experiments
Experiments on Strecha and ETH3D evaluate depth maps, point clouds, and runtime. ACMM improves challenging low-texture reconstruction and detail preservation while remaining substantially faster than COLMAP.
- Evaluation setup: The method is evaluated on Strecha and ETH3D using depth-map assessment and point-cloud evaluation.Strecha provides ground-truth depth maps, while ETH3D includes high-resolution training and test datasets.
- Depth-map evaluation: ACMM surpasses ACMH on Strecha and nearly achieves the best depth-map performance, while its advantage is strongest on detailed Fountain imagery.CMPMVS is slightly better on HerzJesu at the 2cm threshold, whereas ACMM is much better than CMPMVS on Fountain at 2cm.
- Depth-map evaluation: ACMM clearly outperforms COLMAP on challenging ETH3D training datasets, especially indoor scenes with poorly textured regions.ACMH is generally competitive and produces more robust low-texture estimates than COLMAP and DWTA, while ACMM further boosts these regions through its multi-scale scheme.
- Point-cloud evaluation: ACMM produces point clouds with higher completeness on indoor low-texture datasets and outperforms other methods overall in F1 score.All methods show similar accuracy; ACMM inherits ACMH’s structured-region property and adds multi-scale processing.
- Point-cloud evaluation: ACMM produces more complete point clouds than competing methods in challenging areas shown in qualitative ETH3D comparisons.The comparison covers high-resolution multi-view test datasets including living and old co.
- Runtime performance: ACMH is around 6× faster than COLMAP, while ACMM is no more than twice ACMH’s runtime and about 3× faster than COLMAP.ACMM incurs extra computation for multi-scale geometric consistency, but performs that processing on downsampled images at coarser scales.
8. Conclusion
The paper proposes ACMH for efficient hypothesis propagation and view selection, then extends it to ACMM with multi-scale geometric consistency for low-texture areas. Experiments show smooth, consistent depth maps and complete dense 3D models while retaining fine details and good efficiency.
- Conclusion: ACMH uses structured region information to propagate good hypotheses quickly and infer pixelwise view selection.Its components are Adaptive Checkerboard sampling and Multi-Hypothesis joint view selection.
- Conclusion: ACMM combines ACMH with multi-scale geometric consistency guidance to improve depth estimation in low-textured areas.A detail restorer helps retain fine details after coarse-scale estimates are propagated to finer scales.
- Conclusion: Experiments demonstrate smooth, consistent depth maps and complete dense 3D models with good efficiency.The reported results support applications of the proposed methods within the evaluated settings.