PaperlayerView exact revision on arXiv

Source-linked AI summary

StereoDRNet: Dilated Residual Stereo Net

Rohan Chabra, Julian Straub, Chris Sweeney, Richard Newcombe, Henry Fuchs

arXiv:1904.02251v3cs.CV

TL;DR

Stereo matching can be ambiguous on texture-less or repeating and confusingly textured surfaces, while dense and correct depth around object boundaries is needed for spatial maps. StereoDRNet estimates disparity from stereo pairs using Vortex Pooling, 3D dilated cost filtering, and refinement that predicts disparity and occlusion from geometric and photometric errors. The authors report state-of-the-art results on KITTI 2012, comparable results to the best published method on KITTI 2015, and best performance on almost half of ETH dataset evaluation metrics.

  • Problem

    Stereo matching can be ambiguous on texture-less or repeating and confusingly textured surfaces, while dense and correct depth around object boundaries is needed for spatial maps.

    Source: §1 · pp. 1–2

  • Method

    StereoDRNet estimates disparity from stereo pairs using Vortex Pooling, 3D dilated cost filtering, and refinement that predicts disparity and occlusion from geometric and photometric errors.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    The authors report state-of-the-art results on KITTI 2012, comparable results to the best published method on KITTI 2015, and best performance on almost half of ETH dataset evaluation metrics.

    Source: §4.2 · p. 6, §4.3 · p. 6

  • Takeaways & Limitations

    StereoDRNet achieved lower RMSE than PSMNet on “Plants and couch” (2.1 cm versus 2.5 cm) and “Kitchen and bike” (2.1 cm versus 2.8 cm).

    Source: §4.4 · p. 8

    Infrared structured-light depth sensors are unresponsive to dark and highly reflective surfaces.

    Source: §4.4 · p. 7

Abstract

from arXiv · show

We propose a system that uses a convolution neural network (CNN) to estimate depth from a stereo pair followed by volumetric fusion of the predicted depth maps to produce a 3D reconstruction of a scene. Our proposed depth refinement architecture, predicts view-consistent disparity and occlusion maps that helps the fusion system to produce geometrically consistent reconstructions. We utilize 3D dilated convolutions in our proposed cost filtering network that yields better filtering while almost halving the computational cost in comparison to state of the art cost filtering architectures.For feature extraction we use the Vortex Pooling architecture. The proposed method achieves state of the art results in KITTI 2012, KITTI 2015 and ETH 3D stereo benchmarks. Finally, we demonstrate that our system is able to produce high fidelity 3D scene reconstructions that outperforms the state of the art stereo system.

1. Introduction

Passive stereo can provide dense depth in indoor and sunlight environments, but matching remains ambiguous on homogeneous or confusingly textured surfaces. StereoDRNet addresses this with disparity refinement and dilated cost filtering, alongside Vortex Pooling and training-data filtering.

1. Introduction

  • Passive stereo can work indoors and in sunlight without active illumination, and its resolution is typically greater than the sparse patterns of structured-light sensors.The introduction notes stereo depth can support dense maps with accurate object boundaries and corners.Source: §1 · p. 1
  • Stereo matching remains ambiguous on texture-less or repeating and confusingly textured surfaces, motivating learned functions for these regions.Source: §1 · p. 2
  • The refinement network combines geometric error, photometric error, and unrefined disparity to predict refined disparity and an occlusion map for geometrically consistent stereo maps.The maps are intended for direct use by TSDF-based fusion systems such as KinectFusion.Source: §1 · p. 2
  • 3D dilated convolutions across width, height, and disparity channels improve cost filtering with less computation than state-of-the-art cost-filtering architectures.The authors report better results with the structure shown in Fig. 4 and refer to Table 1 for the comparison.Source: §1 · p. 2
  • Vortex Pooling outperforms spatial pyramid pooling in the reported ablation, while exclusion masks help produce sharp edges and fine disparity details.The masks filter non-confident ground-truth regions during fine-tuning.Source: §1 · p. 2
  • StereoDRNet achieves 1.3–2.1 cm RMSE on 3D reconstructions of three scenes prepared using a structured-light system.Source: §1 · p. 2

2. Related Work

Stereo matching has developed from hand-designed cost aggregation and optimization toward CNN-based disparity estimation and cost filtering. StereoDRNet uses Vortex Pooling with PSMNet-style cost filtering and adds geometric-error and occlusion cues to refinement.

2. Related Work

  • Stereo matching is broadly organized around cost metrics, cost aggregation, global or semi-global optimization, and refinement or filtering.Source: §2 · p. 2
  • Earlier global cost filtering used discrete labeling such as Graph Cuts, belief propagation and, or Total Variation denoising,,, and.Source: §2 · p. 2
  • CNN methods progressed from patch-matching with semi-global matching [30] to end-to-end disparity estimation [13], iterative filtering [12], and 3D cost filtering with soft argmax [7].Source: §2 · p. 2
  • PSMNet [2] adds global context through pyramid spatial pooling, while StereoDRNet uses Vortex Pooling and improves its cost filtering approach.Vortex Pooling averages grids of varying dimensions before dilated convolutions to incorporate pixels not used at larger dilation steps.Source: §2 · p. 2
  • StereoDRNet refinement takes geometric error, photometric error, and unrefined disparity to predict refined disparity and an occlusion map; the authors say prior refinement methods used photometric error alone.The paper identifies geometric error and occlusion training as its novel refinement focus.Source: §2 · p. 2

3. Algorithm

StereoDRNet separates stereo disparity estimation into feature extraction, cost-volume filtering, regression, and refinement. Its refinement stage uses image and consistency-error information to predict an occlusion map and residual for the disparity estimate.

3. Algorithm

  • The overall architecture is organized into feature extraction, cost-volume filtering, and refinement rather than a generic encoder-decoder CNN.Source: §3 · p. 2

3.1. Feature Extraction

  • The feature extractor uses a shared-weight Siamese network, downsamples with stride-2 convolutions, then applies Vortex Pooling for broader context.Feature channels are kept at 32 throughout feature extraction.Source: §3.1 · p. 3

3.2. Cost Volume Filtering

  • The cost filter processes both views, using parallel dilated convolutions at rates 1, 2, and 4 to combine information from different receptive fields.A cascade of residual blocks produces three disparity maps, and the filtering process aims to resolve ambiguous local minima in the cost volume.Source: §3.2 · p. 3

3.3. Disparity Regression

  • For each pixel, disparity regression uses a weighted softmax over costs, while Huber loss supervises each predicted disparity map and their weighted total data loss.The cost at pixel i and the maximum disparity define the regression distribution; the weights are assigned per disparity map.Source: §3.3 · p. 3

3.4. Disparity Refinement

  • The refinement network combines left-image and disparity information with photometric and geometric consistency errors to predict an occlusion map and disparity residual.The authors report that feeding the consistency-error maps to the network improved refinement because those errors are meaningful only at non-occluding pixels.Source: §3.4 · pp. 3–4

3.5. Training

  • Training used Adam with β1 = 0.9 and β2 = 0.999, 512x256 crops, and minibatches of 8 on two Nvidia Titan-Xp GPUs.The reported loss weights were w1 = 0.2, w2 = 0.4, w3 = 0.6, λ1 = 1.2, and λ2 = 0.3.Source: §3.5 · p. 4

4. Experiments

The experiments evaluate StereoDRNet on SceneFlow, KITTI 2012/2015, and ETH3D, and test indoor reconstructions formed by fusing its predicted depth maps. The reported results include improved KITTI comparisons, competitive ETH3D metrics, and lower RMSE than PSMNet on two challenging indoor scenes.

4. Experiments

  • The experiments evaluate the architecture on rectified stereo datasets and demonstrate indoor 3D reconstruction by TSDF fusion of predicted depth maps.For indoor training, the network was fine-tuned on 250 stereo pairs collected with the authors’ scanning rig.Source: §4 · p. 4, §4.4 · p. 7

4.1. SceneFlow Dataset

  • SceneFlow training uses over 30,000 stereo pairs, both left and right ground-truth disparities, and an occlusion label for pixels with disparity inconsistency above 1 px.The dataset has around 4,000 evaluation pairs and includes occlusions, thin structures, and large disparities.Source: §4.1 · p. 4
  • On SceneFlow, StereoDRNet’s unrefined architecture outperforms PSMNet while requiring less computation, and co-training occlusion with residual disparity lowers final disparity EPE from 0.93 px to 0.86 px.The ablation reports that passing only photometric error into refinement degrades performance.Source: Table 1 · p. 5, §4.1 · p. 5

4.2. KITTI Datasets

  • StereoDRNet achieves state-of-the-art results on KITTI 2012 and results comparable to the best published method on KITTI 2015, with better results in non-occluded regions.The authors report consistently strong D1-bg performance on KITTI 2015 and comparable performance across all pixels.Source: §4.2 · p. 6, Table 3 · p. 6, Table 4 · p. 6

4.3. ETH3D Dataset

  • On ETH3D, StereoDRNet performs best on almost half of the evaluation metrics, and dense training disparities enabled training and evaluation of its refinement network.The model was pretrained on SceneFlow and fine-tuned on the ETH3D training set.Source: §4.3 · p. 6, Table 5 · p. 6

5. Conclusion

StereoDRNet addresses difficult passive-stereo depth estimation and reports indoor reconstructions approaching state-of-the-art structured-light quality. Its approach combines geometry-focused disparity refinement with global context and dilated residual cost filtering.

5. Conclusion

  • The passive stereo system approaches the quality of state-of-the-art structured-light systems in challenging indoor scene reconstruction.The conclusion attributes recovery of thin structures and sharp object boundaries to perfect synthetic data and careful filtering of real training data.Source: §5 · p. 8
  • Global spatial pooling and dilated residual cost filtering approximate geometry in homogeneous texture, shadows, and specular reflections.The refinement network uses occlusion and view-consistency cues to produce geometrically consistent disparity maps.Source: §5 · p. 8
  • Perfect synthetic data and careful filtering of real training data enabled recovery of thin structures and sharp object boundaries.Source: §5 · p. 8

A. Overview

The supplementary material details the indoor reconstruction experiments and network, and presents additional ablations and benchmark disparity comparisons. The indoor training scene uses rendered stereo views from poses along a camera trajectory.

A. Overview

  • The supplement covers indoor reconstruction training and evaluation, network architecture, refinement effects, dilated-convolution ablations, and KITTI and ETH3D disparity comparisons.Source: A. Overview · p. 9
  • About 200 OpenGL-rendered stereo views from poses along a camera trajectory form the training scene for indoor reconstruction experiments.The figure visualizes the camera trajectory as a blue curve.Source: Figure 11 · p. 9
  • The training-scene 3D reconstruction was built using the method described in [25].Source: Figure 11 · p. 9

B. 3D Reconstruction Experiments

Indoor reconstruction experiments fine-tune SceneFlow-pretrained networks on about 200 stereo views. KinectFusion fuses the depth maps, while its ICP procedure supplies camera views without SfM or external localization.

B. 3D Reconstruction Experiments

C. Network Details

Network details cover Vortex Pooling settings, dilated cost-filtering ablations, and the refinement architecture. The ablation reports improved predictions with increasing dilation rates, with no significant gains above rate 4.

C. Network Details

  • Vortex Pooling uses dilation rates 3, 5, and 15 and average grids of 3×3, 5×5, and 15×15, which improved disparity predictions more than the original segmentation settings.Source: C. Network Details · p. 9
  • Increasing cost-filtering dilation rates improved SceneFlow predictions, but rates above 4 provided no significant gains.Table 6 evaluates StereoDRNet without refinement.Source: C. Network Details · p. 9, Table 6 · p. 9
  • The refinement network outputs refined disparity and occlusion probability.Source: Table 7 · p. 9
  • The refinement network follows StereoNet's basic architecture and uses dilated residual blocks to enlarge the filtering receptive field without compromising resolution.It also uses photometric and geometric error maps as inputs and co-trains occlusion maps, which the authors report provided additional gains.Source: C. Network Details · pp. 9–10

D. Effect of Refinement

The refinement procedure improves disparity error and geometric consistency, with qualitative improvements in object structure and real-scene reconstructions.

D. Effect of Refinement

Related papers

Loading 1904.02251v3…