PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Stereo using Adaptive Thin Volume Representation with Uncertainty Awareness

Shuo Cheng, Zexiang Xu, Shilin Zhu, Zhuwen Li, Li Erran Li, Ravi Ramamoorthi, Hao Su

arXiv:1911.12012v2cs.CVcs.LGcs.RO

TL;DR

Learning-based MVS faces a trade-off between reconstruction accuracy and completeness under memory limits. UCS-Net uses uncertainty-aware adaptive thin volumes in a coarse-to-fine cascade, and reports better accuracy and completeness than state-of-the-art baseline methods, including the best average F-score among published learning-based methods on Tanks and Temples intermediate.

  • Problem

    Under limited memory, prior plane-sweep MVS methods trade reconstruction accuracy, which needs more planes, against completeness, which needs higher-resolution depth maps.

    Source: §1 · p. 1

  • Method

    UCS-Net progressively refines depth using a standard plane-sweep volume followed by two adaptive thin volumes whose spatially varying hypotheses are constructed from variance-based uncertainty estimates.

    Source: §1 · p. 2, §3.2 · p. 4, §3.4 · p. 4

  • Results

    UCS-Net outperforms state-of-the-art baseline methods in both accuracy and completeness, and achieves the best average F-score (54.83) among published learning-based methods on Tanks and Temples intermediate.

    Source: §4 · p. 6

  • Takeaways & Limitations

    The adaptive volumes progressively subpartition local space at finer scales while keeping the smaller volume likely to surround the actual surface.

    Source: §5 · p. 8

Abstract

from arXiv · show

We present Uncertainty-aware Cascaded Stereo Network (UCS-Net) for 3D reconstruction from multiple RGB images. Multi-view stereo (MVS) aims to reconstruct fine-grained scene geometry from multi-view images. Previous learning-based MVS methods estimate per-view depth using plane sweep volumes with a fixed depth hypothesis at each plane; this generally requires densely sampled planes for desired accuracy, and it is very hard to achieve high-resolution depth. In contrast, we propose adaptive thin volumes (ATVs); in an ATV, the depth hypothesis of each plane is spatially varying, which adapts to the uncertainties of previous per-pixel depth predictions. Our UCS-Net has three stages: the first stage processes a small standard plane sweep volume to predict low-resolution depth; two ATVs are then used in the following stages to refine the depth with higher resolution and higher accuracy. Our ATV consists of only a small number of planes; yet, it efficiently partitions local depth ranges within learned small intervals. In particular, we propose to use variance-based uncertainty estimates to adaptively construct ATVs; this differentiable process introduces reasonable and fine-grained spatial partitioning. Our multi-stage framework progressively subdivides the vast scene space with increasing depth resolution and precision, which enables scene reconstruction with high completeness and accuracy in a coarse-to-fine fashion. We demonstrate that our method achieves superior performance compared with state-of-the-art benchmarks on various challenging datasets.

1. Introduction

Multi-view stereo reconstructs dense 3D geometry from multiple calibrated images, but prior plane-sweep methods face a memory-driven trade-off between accuracy and completeness [58] [21]. UCS-Net uses uncertainty-aware adaptive thin volumes in a three-stage cascade to progressively refine depth toward accurate, complete reconstruction.

Introduction

  • Prior plane-sweep MVS methods [58] [21] trade accuracy against completeness under limited memory, since accuracy needs more planes and completeness needs higher-resolution depth maps.High-accuracy large-scene reconstruction requires many sweeping planes, while high completeness requires reconstructing high-resolution depth maps.Source: §1 · p. 1
  • UCS-Net uses three stages: a low-resolution standard volume with 64 planes, followed by higher-resolution ATVs with 32 and 8 planes.Each later stage refines the prior depth prediction with finer depth partitioning; prior work used 256 or 512 planes [58].Source: §1 · p. 2
  • Each ATV centers on the previous depth map and uses variance-based per-pixel confidence intervals to define local uncertainty ranges for the next stage.This explicitly represents depth uncertainty and embeds it in the next stage’s input volume.Source: §1 · p. 2
  • UCS-Net outperforms state-of-the-art learning-based MVS methods on various datasets.The paper’s stated goal is to achieve highly accurate and complete reconstruction while reducing memory and computation.Source: §1 · p. 2
  • The variance-based uncertainty estimation is differentiable, and end-to-end depth supervision across all three stages trains intervals to balance ground-truth coverage with accurate refinement.The intervals must be large enough to cover ground-truth depth but small enough to enable accurate reconstruction in later stages.Source: §1 · p. 2

2. Related Work

Prior 3D reconstruction representations trade efficiency, connectivity, and detail, while deep MVS methods often rely on costly fixed-depth cost volumes. UCS-Net uses adaptive thin volumes to progressively partition local space for coarse-to-fine depth reconstruction.

Spatial Representation for 3D Reconstruction

  • Voxel volumes are memory- and computation-inefficient, point clouds can be noisy, and reconstructed surfaces often lack high-frequency detail; depth maps provide dense information aligned with a reference view.Prior work also explores meshes, deformable shapes, implicit distance functions, and depth estimation from single or multiple views.Source: §2 · p. 2

Deep Multi-View Stereo (MVS)

  • Deep MVS methods use regression, classification, or recurrent and iterative architectures, but single cost volumes with uniformly sampled depths need many planes and constrain high-resolution reconstruction through memory limits.Traditional MVS relies on photo-consistency and can perform poorly in texture-less or occluded areas and under complex lighting.Source: §2 · p. 2
  • UCS-Net uses adaptive thin volumes for efficient spatial partitioning, with a higher depth-wise sampling rate using 104 planes total versus R-MVSNet’s 512, and achieves significantly better reconstruction than R-MVSNet.Point-MVSNet [5] densifies a coarse reconstruction within a predefined local spatial range, whereas UCS-Net refines depth in learned local space with adaptive thin volumes.Source: §2 · p. 3

3. Method

UCS-Net targets high-accuracy, high-completeness per-view depth reconstruction with a three-stage coarse-to-fine framework. It combines multi-scale features, cost volumes, and uncertainty-adaptive thin volumes to progressively refine depth.

3.1. Multi-scale feature extractor

  • A small 2D U-Net produces three feature scales whose learned upsampling incorporates lower-resolution information into high-resolution features for later depth stages.The feature maps have resolutions W/4 × H/4, W/2 × H/2, and W × H, with 32, 16, and 8 channels, respectively.Source: §3.1 · pp. 3–4

3.2. Cost volume construction

  • The first stage builds a standard plane-sweep volume with uniformly sampled constant depth hypotheses; the next two use spatially varying hypotheses adapted to prior pixel-wise uncertainty.Cost volumes are formed by warping source-view features to the reference view and calculating cross-view feature variance at each plane.Source: §3.2 · p. 4

3.3. Depth prediction and probability distribution

  • At each stage, a 3D U-Net processes the cost volume and applies depth-wise softmax to predict per-pixel depth probabilities; the three stages use the same architecture without shared weights.The depth map is reconstructed as a weighted sum of the hypotheses and their predicted probabilities.Source: §3.3 · p. 4

3.4. Uncertainty estimation and ATV

  • UCS-Net uses the variance of each predicted depth distribution to estimate uncertainty and construct differentiable confidence intervals for the next stage’s adaptive thin volume.Uniformly sampled hypotheses within each pixel’s interval set the ATV’s local thickness, allowing end-to-end learning to adjust the intervals.Source: §3.4 · pp. 4–5

3.5. Coarse-to-fine prediction

  • The three stages predict depth at progressively finer resolutions using 64, 32, and 8 planes, enabling local depth partitioning at high pixel-wise resolution with limited memory.The plane counts correspond to a standard volume and two ATVs at W/4 × H/4, W/2 × H/2, and W × H resolutions.Source: §3.5 · p. 5

3.6. Training details

  • The network is trained on DTU using rendered ground-truth depth at three resolutions, with images sized 640 × 512.The training data uses a training, validation, and testing split.Source: §3.6 · p. 5
  • Training uses an end-to-end three-stage network for 60 epochs with Adam, an initial learning rate of 0.0016, and batch size 16 across eight GTX 1080Ti GPUs.The batch size is two per GPU.Source: Loss function · p. 5

Loss function.

  • The final training loss combines L1 depth-prediction losses at all three resolutions, each compared with rendered ground truth at the matching resolution.Source: Loss function · p. 5

4. Experiments

UCS-Net achieves strong DTU and Tanks and Temples reconstructions while using uncertainty-guided adaptive volumes to refine depth efficiently across stages. Experiments also show high interval coverage and lower runtime and memory use than comparison methods.

Evaluation on the DTU dataset

  • On DTU, UCS-Net achieves significantly better completeness and overall scores than prior methods while outperforming state-of-the-art baselines in both accuracy and completeness.Gipuma achieves the best accuracy among all methods, while UCS-Net's example reconstruction is smoother and has fewer outliers than R-MVSNet's despite comparable completeness.Source: §4 · p. 6, Figure 4 · p. 6

Evaluation on Tanks and Temple dataset

  • 54.83 average F-score made UCS-Net best among published learning-based methods; it beat MVSNet and Point-MVSNet on all nine scenes and achieved best overall performance despite Dense-R-MVSNet leading two.UCS-Net outperformed Dense-R-MVSNet on most testing scenes.Source: §4 · pp. 6–7, Table 2 · p. 6

Evaluation of uncertainty estimation

  • 13.88 mm and 3.83 mm average intervals in the first and second ATVs shrink the 508.8 mm initial range; 32 or 8 planes yield about 0.48 mm sampling.The ATVs cover ground-truth depth in 94.72% and 85.22% of cases, respectively; differentiable variance estimates also provide useful intervals for many multimodal depth distributions.Source: §4 · p. 7, Table 3 · p. 7

Evaluation of multi-stage depth prediction

  • ATV-based stages progressively densify and improve reconstruction, whereas naive bilinear upsampling offers only limited improvement over the low-resolution results.Qualitative comparisons show increasingly dense and accurate point clouds through the stages.Source: §4 · pp. 7–8, Table 4 · p. 7, Figure 5 · p. 8

Comparing runtime performance

  • The full model reconstructs original-resolution depth using 104 planes while requiring only about one-quarter of the memory and runtime of MVSNet and R-MVSNet.The comparison used 640 × 480 inputs and 256 depth planes for MVSNet and R-MVSNet.Source: §4 · p. 8, Table 5 · p. 8

5. Conclusion

UCS-Net introduces adaptive thin volumes (ATVs) that use prediction uncertainty to refine multi-view stereo reconstruction progressively. The authors report best qualitative and quantitative performance against state-of-the-art benchmarks with moderate computation and memory complexity.

5. Conclusion

  • UCS-Net uses variance-based uncertainty intervals from one cascade stage to construct an adaptive thin volume for the next stage.The authors describe this as the first use of prediction uncertainty in a learning-based MVS system.Source: §5 · p. 8
  • ATVs progressively partition local space more finely while keeping the smaller volume likely to surround the actual surface.This supports accurate and complete reconstruction in a coarse-to-fine fashion.Source: §5 · p. 8
  • UCS-Net achieves the best qualitative and quantitative performance against state-of-the-art benchmarks with moderate computation and memory complexity.The authors frame the method as a step toward more reliable and efficient learning-based MVS.Source: §5 · p. 8

Appendix Overview

The appendix adds uncertainty-estimation experiments, detailed views of the network’s sub-networks, and final point-cloud reconstruction results.

Appendix Overview

  • It evaluates uncertainty estimation, details the network’s sub-networks, and presents final point-cloud reconstructions for the DTU testing set and Tanks and Temple dataset.Source: Appendix Overview · p. 9

6. Additional experiments of uncertainty estimation.

On the DTU validation set, the uncertainty analysis shows that ATVs combine efficient depth-wise sampling with reliable ground-truth coverage, while interval-length outliers and difficult regions qualify those results.

6. Additional experiments of uncertainty estimation.

  • 12.01mm and 2.71mm median ATV interval lengths correspond to sampling distances of 0.38mm and 0.34mm (32 and 8 planes), versus MVSNet [58] at 1.99mm (256 planes) and RMVSNet at 0.99mm (512 planes).The authors attribute the high sampling rates with few planes to efficient spatial partitioning by the ATVs.Source: Figure 6 · p. 9, §6 · p. 10
  • The two ATV interval-length distributions are unimodal, but small numbers of very large intervals pull their means away from the peaks and make average lengths underestimate sampling efficiency for most pixels.The median lengths are less sensitive to outliers and more representative of these distributions.Source: Figure 6 · p. 9, §6 · pp. 9–10
  • 94.7% and 85.2% average covering ratios show that most uncertainty intervals cover ground-truth depth; the 3rd-stage ATV also shows reduced uncertainty and more precise predictions.Failures to cover ground truth occur mostly around shape boundaries, oblique surfaces, and highly textureless regions, where the method predicts large uncertainty.Source: §6 · p. 10

7. Network architecture.

The network combines a 2D U-Net for feature extraction with three same-architecture 3D U-Nets for cost-volume processing.

7. Network architecture.

  • The architecture uses one 2D U-Net for feature extraction and three 3D U-Nets with the same architecture to process cost volumes.The 3D U-Nets process the cost volume at each stage.Source: §7 · p. 10
  • The 2D U-Net serves as the multi-scale feature extractor; its details are listed in Table 6.Source: §7 · p. 10
  • The 3D U-Net architecture used for cost-volume processing at each stage is detailed in Table 7.Source: §7 · p. 10

8. Point cloud reconstruction.

The section presents final point-cloud reconstructions on the DTU testing set and Tanks and Temples dataset, with quantitative results referenced in the main paper’s tables.

8. Point cloud reconstruction.

  • Final point-cloud reconstructions for the DTU testing set are shown in Figs. 8 and 9.The passage directs readers to the main paper’s Tab. 1 and Tab. 2 for quantitative results on the datasets.Source: §8 · p. 10
  • Figure 10 shows final point-cloud reconstruction results for the Tanks and Temple dataset.Source: §8 · p. 10
  • Quantitative results for these datasets are provided in Tabs. 1 and 2 of the main paper.Source: §8 · p. 10

Related papers

Loading 1911.12012v2…