Source-linked AI summary
DeepMVS: Learning Multi-view Stereopsis
Po-Han Huang, Kevin Matzen, Johannes Kopf, Narendra Ahuja, Jia-Bin Huang
TL;DR
Multi-view stereo methods can fail in poorly textured and reflective regions, while existing learned approaches constrain camera poses or input counts. DeepMVS predicts disparity maps from plane-sweep volumes using learned feature aggregation, arbitrary unordered views, VGG features, and photorealistic synthetic training. On ETH3D, it outperforms DeMoN and is competitive with COLMAP, particularly in poorly textured regions.
Problem
Conventional MVS often struggles where photometric consistency is unreliable, while learned methods may constrain camera poses or the number of input images.
Method
DeepMVS builds plane-sweep volumes and uses learned feature aggregation, pretrained VGG activations, and max-pooling to predict disparity from an arbitrary number of unordered posed images.
Results
On ETH3D, DeepMVS outperforms DeMoN and achieves competitive performance with COLMAP, often producing correct disparities in poorly textured regions.
Takeaways & Limitations
Learning-based MVS can address weaknesses of conventional algorithms, particularly for poorly textured regions and thin structures.
Takeaways & Limitations
The network suffers from disparity quantization errors, often fails on vegetation, and is slowed by plane-sweep generation and large network structures.
Abstract
from arXiv · showhide
We present DeepMVS, a deep convolutional neural network (ConvNet) for multi-view stereo reconstruction. Taking an arbitrary number of posed images as input, we first produce a set of plane-sweep volumes and use the proposed DeepMVS network to predict high-quality disparity maps. The key contributions that enable these results are (1) supervised pretraining on a photorealistic synthetic dataset, (2) an effective method for aggregating information across a set of unordered images, and (3) integrating multi-layer feature activations from the pre-trained VGG-19 network. We validate the efficacy of DeepMVS using the ETH3D Benchmark. Our results show that DeepMVS compares favorably against state-of-the-art conventional MVS algorithms and other ConvNet based methods, particularly for near-textureless regions and thin structures.
1. Introduction
DeepMVS is a learning-based multi-view stereo method designed to address artifacts in conventional reconstruction, especially in poorly textured regions and thin structures. It combines arbitrary-view aggregation, photorealistic synthetic pretraining, and semantic feature integration.
- Conventional MVS relies on photometric consistency and often produces artifacts in poorly textured regions, thin structures, and reflective or transparent surfaces.
- DeepMVS generates plane-sweep volumes, aggregates features with an encoder-decoder, uses pretrained VGG features, and fuses neighbor information with max-pooling.
- The MVS-SYNTH dataset provides 120 photorealistic synthetic urban sequences with dense training supervision for learning-based MVS.
- On ETH3D, DeepMVS outperforms DeMoN and achieves competitive performance with COLMAP, often recovering correct disparities in poorly textured regions.
- DeepMVS processes an arbitrary number of input images, with disparity estimates invariant to input order.
- The paper identifies semantic features, photorealistic synthetic training, and large-area feature aggregation as contributors to improved performance.
2. Related Work
Related work spans conventional MVS, learned stereo matching, volumetric reconstruction, and synthetic-data training. DeepMVS is positioned as a direct disparity predictor that learns visual cues while accepting an arbitrary number of posed images.
- Conventional MVS relies heavily on photometric consistency and struggles with poorly textured or reflective surfaces, while handcrafted integration of lighting, shadows, and semantics is non-trivial.
- Learned patch-matching methods estimate similarity and then produce disparity maps through post-processing, whereas DeepMVS predicts disparities directly from posed images.
- Several plane-sweep ConvNet methods assume a fixed number of input images, while DeepMVS accepts an arbitrary number.
- Volumetric methods use voxel occupancy grids whose resolution is constrained by GPU memory, leaving high-resolution real-world stereo generalization unclear.
- Synthetic datasets reduce the difficulty and cost of collecting large-scale training data, and MVS-SYNTH supplements missing real-world depth measurements such as sky and reflective surfaces.
3. Learning Multi-view Stereopsis
DeepMVS transforms posed image sequences into plane-sweep volumes and predicts disparity through patch matching, intra-volume processing, and unordered inter-volume aggregation. The architecture combines multi-scale semantic features, arbitrary-image max-pooling, classification-based disparity prediction, and DenseCRF refinement.
- 3. Learning Multi-view Stereopsis: The pipeline preprocesses posed images, generates plane-sweep volumes, estimates disparity, and applies final refinement.Camera poses and calibration may be estimated with SFM; the reference image receives the disparity map.
- 3. Learning Multi-view Stereopsis: Each neighbor view is warped across disparity levels to form a plane-sweep volume whose layers can match the reference when geometry is correct and visible.The volumes use disparity levels from 0 to (D −1)δ, with D = 100 chosen as a memory and quantization compromise.
- 3. Learning Multi-view Stereopsis: The network extracts learned patch features, aggregates all disparity levels with an intra-volume U-Net, and incorporates multi-layer VGG-19 features.The intra-volume output is an 800-channel volume containing disparity information from each neighbor.
- 3. Learning Multi-view Stereopsis: Element-wise max-pooling aggregates neighbor volumes, allowing arbitrary input counts and making predictions invariant to neighbor-image order.Training samples one to four neighbors, while the trained network is applied to arbitrary numbers of neighbor images.
- 3. Learning Multi-view Stereopsis: Disparity prediction uses cross-entropy classification, selecting the disparity level with highest predicted probability before DenseCRF refinement.DenseCRF encourages nearby, similarly colored pixels to receive closer disparity predictions.
4. Experimental Results
DeepMVS is evaluated on ETH3D through comparisons with DeMoN and COLMAP, rephotography tests, and component ablations. Results show strong performance in challenging regions, while quantized disparities and vegetation remain limitations.
- Datasets: MVS-SYNTH contains 120 GTA V urban-scene sequences with 100 high-resolution frames each and complete disparity, intrinsic, and extrinsic ground truth.Its annotations cover sky, reflective surfaces, and thin structures that are often missing from real-world datasets.
- Training: DeepMVS is trained in two stages, first for pretraining and then with the intra-volume aggregation network using DeMoN and MVS-SYNTH data.Each stage uses Adam for 320k iterations, with learning rates of 10^-5 and 10^-6, respectively.
- Comparisons: DeepMVS generally recovers reference images with small holes in rephotography, although disparity quantization causes jagged edges.DeMoN results are often blurry and distorted, while COLMAP produces large holes in challenging regions.
- Comparisons: On ETH3D, DeepMVS produces competitive results against COLMAP and outperforms DeMoN, especially in poorly textured regions such as sky, walls, floors, and desks.COLMAP is more accurate for many pixels with sufficient depth cues, while DeepMVS performs better on a challenging subset.
- Comparisons: DeepMVS remains accurate with few input images, whereas COLMAP tends to produce large geometric errors when the number of views is small.The comparison examines progressive improvement as the number of input images increases.
- Ablation Studies: Adding DenseCRF, MVS-SYNTH, U-Net features, or VGG features improves both error metrics, but MVS-SYNTH mainly benefits poorly annotated regions.Without MVS-SYNTH, predictions for sky and reflective surfaces degrade even though quantitative errors change only slightly.
- Limitations: The network has three reported limitations: disparity quantization, failures on vegetation, and slow computation from plane-sweep generation and large network structures.These limitations affect geometric and photometric accuracy, scene coverage, and processing speed.
5. Conclusions
DeepMVS demonstrates that multi-view stereopsis can be learned with a convolutional neural network, with learning-based approaches overcoming weaknesses of conventional algorithms.
- DeepMVS demonstrates the feasibility of learning multi-view stereopsis with a convolutional neural network.
- Learning-based approaches can overcome weaknesses of conventional multi-view stereo algorithms.