Source-linked AI summary
DPSNet: End-to-end Deep Plane Sweep Stereo
Sunghoon Im, Hae-Gon Jeon, Stephen Lin, In So Kweon
TL;DR
Multiview stereo seeks dense scene depth from images captured under arbitrary camera motion, where textureless and reflective regions challenge conventional matching. DPSNet uses an end-to-end plane-sweep CNN with differentiable cost-volume construction and context-aware aggregation, achieving state-of-the-art reconstruction results across several datasets. Its main scope boundaries are reliance on calibrated camera parameters and the lack of rectified-stereo-specific strategies.
Problem
Multiview stereo must reconstruct scene depth from unstructured images despite unreliable photometric consistency in textureless and reflective regions.
Method
DPSNet builds plane-sweep cost volumes from deep features using differentiable warping, then applies context-aware cost aggregation and depth regression in an end-to-end CNN.
Results
DPSNet achieves state-of-the-art reconstruction results on a variety of datasets, and ablations report appreciable improvements from its technical contributions.
Takeaways & Limitations
The paper demonstrates that traditional multiview stereo schemes can be incorporated into a deep learning framework for dense depth reconstruction.
Takeaways & Limitations
DPSNet requires pre-calibrated intrinsic and extrinsic camera parameters, and its depth accuracy could improve with rectified stereo pair-specific strategies.
Abstract
from arXiv · showhide
Multiview stereo aims to reconstruct scene depth from images acquired by a camera under arbitrary motion. Recent methods address this problem through deep learning, which can utilize semantic cues to deal with challenges such as textureless and reflective regions. In this paper, we present a convolutional neural network called DPSNet (Deep Plane Sweep Network) whose design is inspired by best practices of traditional geometry-based approaches for dense depth reconstruction. Rather than directly estimating depth and/or optical flow correspondence from image pairs as done in many previous deep learning methods, DPSNet takes a plane sweep approach that involves building a cost volume from deep features using the plane sweep algorithm, regularizing the cost volume via a context-aware cost aggregation, and regressing the dense depth map from the cost volume. The cost volume is constructed using a differentiable warping process that allows for end-to-end training of the network. Through the effective incorporation of conventional multiview stereo concepts within a deep learning framework, DPSNet achieves state-of-the-art reconstruction results on a variety of challenging datasets.
1 INTRODUCTION
Multiview stereo must infer depth from unstructured images despite unreliable photometric consistency in textureless and reflective regions. DPSNet brings plane-sweep cost-volume construction and context-aware regularization into an end-to-end CNN, achieving state-of-the-art results.
- Photometric consistency constraints can be unreliable in textureless and reflective scenes, motivating learned multiview depth estimation.
- DPSNet fully models plane-sweep stereo within a CNN by constructing cost volumes through differentiable warping.This makes plane-sweep stereo learnable end to end.
- A context-aware cost aggregation module regularizes cost-volume slices and reduces the effects of unreliable matches.
- DPSNet obtains state-of-the-art results over several standard datasets, while ablations show appreciable accuracy improvements from its technical contributions.
2 RELATED WORK
Prior depth-estimation research spans stereo matching, single-image inference, and multiview stereo. Existing multiview approaches vary in view support, camera-parameter handling, and computational representation.
- Stereo matching: Stereo matching addresses depth estimation from a pair of rectified images captured by a stereo rig.
- Depth from single images: Single-image methods use CNN features to infer scene depth and refine estimates, including refinements based on superpixel conditional random fields.
- Multi-view stereo: Some multiview methods recover camera motion but handle only two views, using components such as optical-flow, depth, motion, and refinement networks.
- Multi-view stereo: Networks handling arbitrary numbers of views generally assume known or conventionally estimated camera parameters, while voxel representations can restrict practical scene resolution through GPU-memory limits.
3 APPROACH
DPSNet combines feature extraction, differentiable plane-sweep cost-volume generation, context-aware cost aggregation, and continuous depth regression in an end-to-end framework. It supports multiple views by averaging pairwise cost volumes and uses learned aggregation to regularize noisy costs while preserving edges.
- DPSNet consists of feature extraction, cost volume generation, cost aggregation, and depth map regression.
- Cost volume generation: Plane sweep stereo back-projects images onto successive virtual planes and measures photo-consistency for each pixel.DPSNet constructs cost volumes from image features using this plane-sweep principle.
- Cost volume generation: At test time, cost volumes from any number of paired images can be averaged, while training uses one paired image with the reference view.This multi-view aggregation reduces image noise through averaging across pairs.
- Cost volume generation: Features are warped into reference coordinates using camera intrinsics and extrinsics, then concatenated across depth labels into a 4D volume.The spatial-transformer warping requires no learnable parameters and supports images with different intrinsics.
- Cost aggregation: The context network refines each cost slice using reference features, and the refined volume is added to the initial volume.Dilated convolutions provide varying receptive fields, and shared weights process all cost slices.
- Depth regression: Continuous depth is regressed by applying softmax to cost labels, computing their probability-weighted sum, and converting the predicted label using L and dmin.The reported configuration sets L=64 and dmin=0.5.
- Training: The training loss supervises depths from both initial and refined cost volumes using a Huber norm, with the initial-depth term weighted by λ=0.7.The learnable parameters cover feature extraction, cost-volume generation, and cost aggregation; plane sweep and depth regression have no learnable parameters.
4 EXPERIMENTS
DPSNet is evaluated against geometry-based, two-view, and CNN-based methods across multiple datasets, with ablations examining its cost-volume design, aggregation, sampling, and input-view count. It achieves strong dense-depth results, including robust performance in textureless and homogeneous regions, while rectified stereo remains a setting where specialized strategies may help.
- Comparison with state-of-the-art methods: DPSNet provides the best performance on nearly all measures in comparisons across MVS, SUN3D, RGBD, and Scenes11.The comparison uses COLMAP, DeMoN, and DeepMVS on two unstructured views with standard depth-quality metrics.
- Comparison with state-of-the-art methods: DPSNet accurately recovers homogeneous regions, object boundaries, keyboard details, and fine structures where competing methods often fail.Differential feature warping penalizes inaccurate reconstructions, while cost aggregation improves textureless-region estimates.
- Comparison with state-of-the-art methods: DPSNet achieves the best overall performance except filtered COLMAP, while retaining 100% completeness compared with filtered COLMAP's 71%.The ETH3D comparison reports completeness, geometry error, and photometry error; unfiltered COLMAP shows a significant drop across error metrics.
- Ablation study: Feature concatenation outperforms absolute-difference cost volumes, and the proposed cost aggregation yields significant improvements over omitting aggregation or using a stacked hourglass.Reference-image features guide aggregation, producing more accurate depth especially in homogeneous regions.
- Ablation study: Inverse-depth sampling produces more accurate depth maps in general, while adding views improves results until performance plateaus at seven or more images.More views reduce cost-volume noise through averaging and provide features from areas unseen in other views.
- Rectified stereo: Fine-tuned DPSNet reaches a KITTI2015 D1-all score of 4.34% for all pixels and 4.05% for non-occluded pixels.Although not designed for rectified stereo, DPSNet produces reasonable results and achieves performance similar to Mayer et al. (2016) in D1-all.
5 DISCUSSION
DPSNet integrates traditional multiview stereo practices into an end-to-end network and achieves state-of-the-art reconstruction across datasets. Its remaining scope includes dependence on pre-calibrated camera parameters and several proposed improvement directions.
- DPSNet incorporates traditional multiview stereo schemes into a deep learning framework for end-to-end reconstruction.Its design uses differentiable plane-sweep cost volumes, multilabel depth classification, and context-aware cost aggregation without post-processing.
- State-of-the-art reconstruction results are achieved on a variety of datasets.
- DPSNet could be improved by integrating semantic instance segmentation into cost aggregation and using viewpoint selection when constructing cost volumes.The current approach averages estimated cost volumes rather than selecting viewpoints.
- DPSNet requires pre-calibrated intrinsic and extrinsic parameters for reconstruction.The authors identify end-to-end camera-pose estimation as an important future challenge for removing this restriction.