PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Stereo using Adaptive Thin Volume Representation with Uncertainty Awareness

Shuo Cheng, Zexiang Xu, Shilin Zhu, Zhuwen Li, Li Erran Li, Ravi Ramamoorthi, Hao Su

arXiv:1911.12012v2cs.CVcs.LGcs.RO

TL;DR

Learning-based MVS must balance dense depth sampling for accuracy against high-resolution maps for completeness. UCS-Net addresses this with a three-stage uncertainty-aware cascade using adaptive thin volumes, and it reports superior reconstruction performance across challenging datasets, including a best average F-score of 54.83 among published learning-based methods on Tanks and Temples.

  • Problem

    Learning-based MVS cost volumes require many depth planes for accuracy and high-resolution depth maps for completeness, creating a memory-driven trade-off.

  • Method

    UCS-Net uses a three-stage coarse-to-fine cascade whose later stages construct spatially varying adaptive thin volumes from variance-based uncertainty estimates.

  • Results

    54.83 average F-score was achieved on the Tanks and Temples intermediate dataset, the best among published learning-based methods according to the paper.

  • Takeaways & Limitations

    UCS-Net progressively subpartitions local scene space, enabling high-accuracy and high-completeness reconstruction with small adaptive volumes and few depth planes.

Abstract

from arXiv · show

We present Uncertainty-aware Cascaded Stereo Network (UCS-Net) for 3D reconstruction from multiple RGB images. Multi-view stereo (MVS) aims to reconstruct fine-grained scene geometry from multi-view images. Previous learning-based MVS methods estimate per-view depth using plane sweep volumes with a fixed depth hypothesis at each plane; this generally requires densely sampled planes for desired accuracy, and it is very hard to achieve high-resolution depth. In contrast, we propose adaptive thin volumes (ATVs); in an ATV, the depth hypothesis of each plane is spatially varying, which adapts to the uncertainties of previous per-pixel depth predictions. Our UCS-Net has three stages: the first stage processes a small standard plane sweep volume to predict low-resolution depth; two ATVs are then used in the following stages to refine the depth with higher resolution and higher accuracy. Our ATV consists of only a small number of planes; yet, it efficiently partitions local depth ranges within learned small intervals. In particular, we propose to use variance-based uncertainty estimates to adaptively construct ATVs; this differentiable process introduces reasonable and fine-grained spatial partitioning. Our multi-stage framework progressively subdivides the vast scene space with increasing depth resolution and precision, which enables scene reconstruction with high completeness and accuracy in a coarse-to-fine fashion. We demonstrate that our method achieves superior performance compared with state-of-the-art benchmarks on various challenging datasets.

1. Introduction

UCS-Net addresses the memory-driven trade-off between accurate depth sampling and high-resolution completeness in learning-based MVS. It progressively refines depth using uncertainty-aware adaptive thin volumes (ATVs) in a three-stage coarse-to-fine framework.

  • Motivation: Learning-based MVS uses plane-sweep cost volumes, but limited memory forces a trade-off between more depth planes for accuracy and more pixels for completeness.High-accuracy reconstruction requires many sweeping planes, while high completeness requires high-resolution depth maps.
  • Approach: UCS-Net replaces one large standard volume with multiple small volumes that progressively regress high-quality depth in a coarse-to-fine fashion.The framework targets accurate and complete reconstruction while reducing memory and computation consumption.
  • Approach: The three-stage cascade predicts increasingly larger depth maps, with each later stage refining the previous prediction using higher pixel resolution and finer depth partitioning.The first stage uses a low-resolution standard volume with 64 planes; the following stages use ATVs.
  • Adaptive thin volumes: ATVs use variance-based per-pixel uncertainty intervals to place a curved central plane around prior depth predictions and spatially vary depth hypotheses.This constructs local volumes around the previous prediction rather than sampling the same depth planes everywhere.
  • Adaptive thin volumes: Differentiable uncertainty estimation is trained end to end so ATV intervals are wide enough to cover ground truth depth yet narrow enough for accurate refinement.The resulting multi-stage process progressively subdivides local scene space at finer scales.

2. Related Work

Prior 3D reconstruction methods use volumetric, point-cloud, surface, or implicit representations, each with distinct efficiency, connectivity, smoothness, or scalability limitations. Deep MVS methods improve reconstruction but commonly rely on cost volumes, whereas UCS-Net uses adaptive sampling to refine depth efficiently.

  • Spatial Representation for 3D Reconstruction: Voxel-based representations are memory- and computation-intensive because most voxels do not lie on the surface.Point clouds can produce noisy outliers because they do not efficiently encode connectivity between points.
  • Spatial Representation for 3D Reconstruction: Point-cloud methods may rely on strong shape priors and therefore do not directly extend to large-scale scene reconstruction.Surface methods often produce smoother surfaces than point-cloud approaches.
  • Deep Multi-View Stereo: Traditional MVS relies on photo-consistency but performs poorly in textureless or occluded regions and under complex lighting, motivating deep learning-based alternatives.Deep MVS includes regression-, classification-, recurrent-, and iterative-style architectures.
  • Deep Multi-View Stereo: R-MVSNet uses 512 planes for fine depth-wise partitioning, while UCS-Net uses adaptive sampling with 104 planes in total.The passage reports that UCS-Net achieves significantly better reconstruction than R-MVSNet.

3. Method

UCS-Net uses a three-stage coarse-to-fine cascade that combines multi-scale features, cost volumes, and uncertainty-guided adaptive thin volumes to produce high-resolution depth efficiently.

  • 3. Method: UCS-Net progressively predicts depth maps at increasing resolutions, using multiple cost volumes and separate 3D CNN stages.The first stage operates at low resolution, while later stages refine predictions with adaptive thin volumes.
  • 3.1. Multi-scale feature extractor: A multi-scale 2D CNN extracts feature maps at three resolutions for constructing multi-resolution cost volumes.The feature maps have resolutions W/4 × H/4, W/2 × H/2, and W × H, with 32, 16, and 8 channels respectively.
  • 3.2. Cost volume construction: The first cost volume uses constant depth planes, whereas later volumes use spatially varying hypotheses determined by pixel-wise uncertainty.Warping uses camera geometry and per-plane hypotheses; the first stage samples a predefined depth interval uniformly.
  • 3.3. Depth prediction and probability distribution: Each stage applies a 3D CNN and depth-wise softmax to estimate per-pixel depth probability distributions, then reconstructs depth by their weighted sum.The three stages share the architecture but not weights, and each stage predicts a depth probability volume associated with its hypotheses.
  • 3.4. Uncertainty estimation and ATV: UCS-Net estimates uncertainty from the variance of each per-pixel depth distribution and uses confidence intervals to construct adaptive thin volumes around the previous prediction.For each pixel, the next-stage depth hypotheses are uniformly sampled within the interval, creating spatially varying curved planes.
  • 3.4. Uncertainty estimation and ATV: Differentiable uncertainty estimation lets end-to-end training adjust ATV coverage so intervals are large enough to cover ground truth yet small enough for accurate refinement.The resulting thin volumes use few planes while supporting higher-resolution reconstruction under limited memory.

4. Experiments

Experiments show that UCS-Net reconstructs accurate, complete geometry while using adaptive uncertainty-aware volumes to reduce depth ranges and resource demands. Across DTU and Tanks and Temples, it achieves strong accuracy, completeness, F-scores, and efficiency, with progressive stages improving reconstruction quality.

  • DTU evaluation: On DTU, UCS-Net outperforms state-of-the-art baselines in both reconstruction accuracy and completeness, while improving completeness over Gipuma.Gipuma achieves the best accuracy, but UCS-Net has significantly better completeness and overall scores.
  • DTU evaluation: UCS-Net progressively improves reconstruction: its 64-plane first stage trails MVSNet and R-MVSNet, but later ATV stages surpass them.The second-stage reconstruction is already much better than the previous methods, and the third stage further improves quality.
  • Tanks and Temples evaluation: 54.83 average F-score is achieved on Tanks and Temples, the best among published learning-based methods in the reported comparison.UCS-Net obtains higher F-scores than MVSNet and Point-MVSNet in all nine testing scenes and achieves the best overall performance.
  • Uncertainty estimation: 13.88mm and 3.83mm are the average depth ranges of the first and second ATVs, reduced from the initial 508.8mm plane-sweep range.With only 32 or 8 planes, ATVs obtain about 0.48mm sampling distance, compared with 1.99mm and 0.99mm for uniform sampling in MVSNet and R-MVSNet.
  • Uncertainty estimation: 94.72% and 85.22% of pixels have uncertainty intervals covering ground-truth depth in the first and second ATVs.These coverage ratios support the confidence of the estimated intervals used for adaptive volume construction.
  • Runtime and memory: UCS-Net uses about one quarter of the memory and running time of comparison methods while reconstructing depth at the original image resolution.The full model uses 104 depth planes in total and does not incur higher computation or memory consumption despite higher output resolution.

5. Conclusion

The paper concludes that UCS-Net uses uncertainty-aware adaptive thin volumes within a cascaded coarse-to-fine framework for accurate and complete multi-view reconstruction. Its reported benchmark results support improved reconstruction quality with moderate computation and memory complexity.

  • Conclusion: UCS-Net uses variance-based uncertainty intervals to construct ATVs that progressively partition local space around predicted surfaces.The smaller volumes are intended to surround the actual surface with high probability while refining depth at finer scales.
  • Conclusion: The method achieves qualitatively and quantitatively best performance across comparisons while maintaining moderate computation and memory complexity.The conclusion characterizes UCS-Net as a step toward more reliable and efficient learning-based MVS.

Overview

The appendix analyzes uncertainty-interval distributions and shows that median lengths better represent typical ATV sampling than means because a few intervals are unusually large.

  • Additional experiments: The appendix evaluates uncertainty estimation on the DTU validation set and analyzes interval-length distributions for both ATVs.Figure 6 uses 0.5mm bins and marks mean and median interval lengths.
  • Uncertainty intervals: The two ATV interval-length distributions are unimodal, but small portions of very large intervals make their means differ substantially from their modes.Those outliers cause average lengths to underestimate sampling efficiency for most pixels.
  • Network architecture: The appendix also documents the multi-scale feature extractor and 3D U-Net architectures used for cost-volume construction and processing.The 3D U-Net predicts depth probabilities at each stage, while the feature extractor provides multi-scale features.
  • Uncertainty intervals: Median interval lengths are 12.01mm and 2.71mm, corresponding to 0.38mm and 0.34mm sampling distances with 32 and 8 depth planes.The medians are closer to the histogram peaks and less sensitive to large-value outliers than the means.
  • Uncertainty intervals: Most estimated uncertainty intervals cover ground-truth depth, with average covering ratios of 94.7% and 85.2% for the two ATVs.Failures occur mainly near shape boundaries, oblique surfaces, and highly textureless regions.

7. Network architecture.

The network architecture combines a 2D U-Net feature extractor with three same-architecture 3D U-Nets for stage-wise cost-volume processing.

  • Network architecture: A 2D U-Net extracts multi-scale features for cost-volume construction.Its detailed architecture is provided for the multi-scale feature extractor.
  • Network architecture: Three 3D U-Nets with the same architecture process cost volumes across the network stages.The 3D U-Nets are used for cost-volume processing at each stage.
  • Network architecture: The architecture description covers both the feature-extraction and cost-volume-processing subnetworks in detail.These subnetworks correspond to the multi-scale feature extractor and the stage-wise cost-volume processors.

8. Point cloud reconstruction.

The appendix presents final point-cloud reconstructions on the DTU testing set and the Tanks and Temples dataset, with quantitative results reported in the main paper.

  • Point cloud reconstruction: Final point-cloud reconstruction results are shown for the DTU testing set.The appendix references Figures 8 and 9 for these reconstructions.
  • Point cloud reconstruction: Final point-cloud reconstruction results are also shown for the Tanks and Temples dataset.The appendix references Figure 10 for this dataset.
  • Point cloud reconstruction: Quantitative results for both datasets are reported in Tables 1 and 2 of the main paper.The appendix directs readers to those tables for numerical evaluation.
Loading 1911.12012v2…