PaperlayerView exact revision on arXiv

Source-linked AI summary

Point-Based Multi-View Stereo Network

Rui Chen, Songfang Han, Jing Xu, Hao Su

arXiv:1908.04422v1cs.CV

TL;DR

MVS reconstruction needs methods that capture 3D geometry without the costly fixed-resolution processing used by volumetric approaches. Point-MVSNet processes scenes as point clouds through coarse-to-fine depth refinement, and it achieves state-of-the-art learning-based performance on DTU and Tanks and Temples.

  • Problem

    Dense multi-scale 3D CNNs used in learning-based MVS require memory cubic to model resolution, while grid partitioning can introduce quantization artifacts.

  • Method

    Point-MVSNet generates a coarse depth map, converts it to a point cloud, and iteratively refines it using geometry priors and multiview image features.

  • Results

    Point-MVSNet achieves state-of-the-art performance among learning-based methods on DTU and Tanks and Temples, with better completeness and overall quality than previous state-of-the-art methods.

  • Takeaways & Limitations

    The point-based architecture preserves surface continuity, reduces unnecessary computation, and supports foveated depth inference.

Abstract

from arXiv · show

We introduce Point-MVSNet, a novel point-based deep framework for multi-view stereo (MVS). Distinct from existing cost volume approaches, our method directly processes the target scene as point clouds. More specifically, our method predicts the depth in a coarse-to-fine manner. We first generate a coarse depth map, convert it into a point cloud and refine the point cloud iteratively by estimating the residual between the depth of the current iteration and that of the ground truth. Our network leverages 3D geometry priors and 2D texture information jointly and effectively by fusing them into a feature-augmented point cloud, and processes the point cloud to estimate the 3D flow for each point. This point-based architecture allows higher accuracy, more computational efficiency and more flexibility than cost-volume-based counterparts. Experimental results show that our approach achieves a significant improvement in reconstruction quality compared with state-of-the-art methods on the DTU and the Tanks and Temples dataset. Our source code and trained models are available at https://github.com/callmeray/PointMVSNet .

1. Introduction

Point-MVSNet addresses the memory and quantization limitations of volumetric MVS by processing scenes as point clouds and refining them adaptively. It combines coarse-to-fine reconstruction with geometry and image cues, achieving state-of-the-art learning-based benchmark performance.

  • Learning-based MVS improves robustness and reconstruction completeness by using global semantic information, but dense multi-scale 3D CNNs have resolution-dependent memory costs.Octree partitioning reduces the memory problem but retains grid quantization artifacts, according to the introduction.
  • Point-MVSNet directly processes the target scene as a point cloud, a more efficient representation at high 3D resolution.The framework first generates a coarse depth map using a relatively small 3D cost volume before converting it into a point cloud.
  • PointFlow iteratively refines the initial point cloud by predicting depth residuals using geometry priors and dynamically fetched multi-view image appearance cues.The module estimates 3D flow for each point while combining 3D geometry and 2D texture information.
  • Adaptive surface-point sampling preserves surface continuity, processes only valid near-surface information, and densifies reconstruction in regions of interest.These design choices provide reported advantages in accuracy, efficiency, and flexibility over fixed-resolution volumetric methods.
  • Point-MVSNet achieves state-of-the-art performance among learning-based methods on DTU and Tanks and Temples, with better completeness and overall quality than previous state-of-the-art methods.The paper also reports potential applications including foveated depth inference.

2. Related work

Related MVS methods use classical iterative representations, 2D learning, volumetric cost volumes, space partitioning, or direct point-cloud processing. These approaches trade among contextual geometry, memory use, computational efficiency, continuity, and robustness.

  • Multi-view Stereo Reconstruction: Classical MVS methods iteratively update volumes, deformation models, or patches using multi-view photometric consistency and regularization optimization.The paper positions its iterative depth-refinement procedure as sharing this general idea while using learning-based prediction.
  • Learning-based MVS: 2D learning-based stereo methods improve patch representation and matching but are difficult to extend to MVS and lack contextual geometry knowledge in challenging scenes.
  • Learning-based MVS: 3D cost-volume methods explicitly capture scene geometry by warping multiview 2D features into a camera-frustum or scene volume for 3D CNN processing.
  • High-Resolution MVS: Traditional high-resolution MVS methods expand dense patches from confident keypoints, whereas recent learning methods reduce memory consumption through advanced space partitioning.The traditional approaches are described as potentially time-consuming and sensitive to noise and viewpoint changes because they use hand-crafted features.
  • Point-based 3D Learning: Point-based 3D networks process point clouds directly without volumetric grids, concentrating computation on points while preserving spatial continuity.

3. Method

Point-MVSNet combines low-resolution coarse depth prediction with point-cloud-based iterative refinement, integrating 2D image features and 3D geometry to estimate depth residuals.

  • Coarse depth prediction: The method first predicts a coarse depth map, unprojects it into a point cloud, and iteratively refines the depth.Low-resolution cost-volume prediction initializes the point-based refinement process.
  • 2D-3D feature lifting: A three-scale feature pyramid provides multi-view image features that are dynamically fetched for each 3D point.Feature warping accounts for each pyramid level's image resolution, and variance aggregates features across views.
  • 2D-3D feature lifting: Each feature-augmented point combines fetched image features with normalized 3D coordinates before entering PointFlow.Point positions and fetched features are updated after each refinement iteration.
  • PointFlow: PointFlow generates depth hypotheses along the reference camera direction and uses kNN edge convolutions to aggregate local geometric context.The hypotheses gather image information at different depths while directed edges propagate neighborhood structure.
  • PointFlow: The module predicts a depth residual from pointwise features, converts hypothesis probabilities into displacement, and adds the projected residual to refine depth.Three EdgeConv layers, shortcut connections, and a shared MLP produce probabilities over each point's hypotheses.

4. Experiments

Experiments evaluate Point-MVSNet on DTU through benchmark comparisons, qualitative reconstruction, iterative refinement, and component ablations. Results emphasize detailed point clouds, improved completeness and overall quality, and the importance of geometry-aware local aggregation and multiscale features.

  • 4.1. DTU dataset: The DTU dataset contains 124 scenes scanned under seven lighting conditions from 49 or 64 positions.
  • 4.2. Implementation details: Evaluation uses 1280×960 and 1600×1152 input resolutions, with three flow iterations for refinement.
  • 4.3. Benchmarking on DTU dataset: Point-MVSNet produces more detailed point clouds than MVSNet, especially in edgy areas with high-frequency geometric features.
  • 4.5. Ablation study: Iterative refinement makes the generated point cloud denser and captures more geometry details as flow iterations increase.
  • 4.5. Ablation study: Replacing geometry-aware edge convolution with equal neighbor aggregation significantly reduces reconstruction quality, showing the importance of local geometric relationships.
  • 4.5. Ablation study: Using image-space adjacent pixels instead of k-nearest 3D points decreases performance because occlusion can associate nearby pixels with distant objects.
  • 4.5. Ablation study: Removing components of the feature pyramid causes a significant performance drop, demonstrating the value of multiscale context for feature fetching.

4.6. Reliance on initial depth maps

Point-MVSNet iteratively refines coarse depth into denser point clouds and remains robust to initial depth noise within a measured range. Its point-based design also supports region-focused inference and arbitrary point counts.

  • Reliance on initial depth maps: The reconstruction error increases slowly with Gaussian initial-depth noise and remains smaller than MVSNet up to 6mm noise.The experiment evaluates robustness across different noise scales added to the initial depth map.
  • Iterative refinement: PointFlow refines a coarse point cloud by predicting depth residuals relative to the ground truth.The module repeatedly estimates 3D flow using point-cloud geometry priors and multiview image appearance cues.
  • Point-cloud upsampling: The study compares Point-MVSNet with PU-Net as an alternative for upsampling a coarse point cloud.PU-Net is applied twice to generate a point cloud with 16 times more points using EMD and repulsion losses.
  • Foveated depth inference: The method can process an arbitrary number of points, enabling selective depth inference within a region of interest.ROI refinement can be applied using either the input image or the predicted coarse depth map.
  • Foveated depth inference: Foveated inference produces three point-density levels by upsampling and refining only the ROI from the previous stage.Figure 9 distinguishes sparsest, intermediate, and densest regions by color.

5. Conclusion

Point-MVSNet processes scenes directly as point clouds for high-resolution MVS reconstruction instead of constructing high-resolution cost volumes. Experiments report high-quality benchmark reconstructions and support foveated depth inference.

  • 5. Conclusion: Point-MVSNet processes scenes directly as point clouds rather than building high-resolution cost volumes.The architecture reduces unnecessary computation and preserves spatial continuity.
  • 5. Conclusion: The method produces high-quality reconstruction point clouds on benchmarks.The conclusion reports this outcome without specifying a numerical metric in the supplied passage.
  • 5. Conclusion: Point-MVSNet supports foveated depth inference, which the paper describes as greatly reducing computation.The conclusion contrasts this application with cost-volume-based methods.

6. Additional Ablation Study

Additional ablations examine point-hypothesis count and input-view count on DTU. Two point hypotheses yield the best reported reconstruction quality, while quality improves as the number of input views increases.

  • 6.1. Number of point hypotheses: m = 2 achieves the best DTU reconstruction quality for completeness and overall quality among the tested point-hypothesis counts.The comparison evaluates m = 1, 2, and 3, with the model trained using m = 2.
  • 6.2. Number of Views: Reconstruction quality improves as the number of input views increases on the DTU evaluation set.The model is trained with N = 3 but evaluated with N = 2, 3, and 5.

7. Memory, runtime and overhead of kNN

Point-MVSNet varies output resolution and speed by changing refinement iterations, while restricting neighborhood searches avoids naive all-point kNN costs.

  • Memory and runtime: Changing the number of iterations allows Point-MVSNet to predict different depth-map resolutions at different speeds.This provides a runtime-resolution trade-off within the point-based architecture.
  • kNN overhead: Naive kNN over N points has O(N^2) memory complexity, so the method restricts each search to a local k × k × (2m + 1) neighborhood.The restriction uses nearby 2D pixels and hypothetical points rather than searching the whole point cloud.
  • kNN overhead: Distance computations are parallelized with a fixed-weight 3D kernel.This is part of the implementation used to reduce kNN overhead.

8. Post-processing

Point-MVSNet post-processes predictions through photometric filtering, geometric consistency filtering, and depth fusion.

  • Post-processing applies photometric filtering, geometric consistency filtering, and depth fusion in sequence.
  • Photometric filtering uses the most likely depth layer's predicted probability as confidence, with thresholds of 0.5 for coarse and 0.2 for PointFlow outputs.
  • Geometric consistency filtering discards points whose reverse-projected multiview depth discrepancy exceeds 0.12mm.
  • Depth fusion averages retained depth values after photometric and geometric consistency filtering.

9. Reconstruction Results

Point-MVSNet reconstructs dense and accurate point clouds across the DTU evaluation set and the intermediate Tanks and Temples set.

  • Point-MVSNet reconstructs dense and accurate point clouds for all evaluated scenes.
  • The DTU reconstruction results are presented in Figure 10.
  • The Tanks and Temples intermediate-set reconstruction results are presented in Figure 11.
Loading 1908.04422v1…