PaperlayerView exact revision on arXiv

Source-linked AI summary

EdgeStereo: An Effective Multi-Task Learning Network for Stereo Matching and Edge Detection

Xiao Song, Xu Zhao, Liangji Fang, Hanwen Hu

arXiv:1903.01700v2cs.CV

TL;DR

EdgeStereo targets persistent stereo-matching errors in ill-posed and structurally detailed regions. It jointly learns disparity and edge detection using edge-aware interactions and a residual-pyramid decoder, and reports strong benchmark performance with comparable disparity generalization. The paper concludes that the two tasks can promote each other through learned geometric knowledge.

  • Problem

    End-to-end stereo networks remain challenged by ambiguous correspondences in ill-posed regions and by detailed structures, small objects and near boundaries.

  • Method

    EdgeStereo uses shared shallow features, edge feature embedding, edge-aware smoothness loss and a residual pyramid within a unified disparity-and-edge network.

  • Results

    EdgeStereo achieves state-of-the-art performance on FlyingThings3D, KITTI 2012 and KITTI 2015, with comparable generalization for disparity estimation.

  • Takeaways & Limitations

    Stereo matching and edge detection can promote each other through geometric knowledge learned from their multi-task interactions.

  • Takeaways & Limitations

    On Middlebury, EdgeStereo outperforms reported end-to-end comparators but performs worse than the non-end-to-end MC-CNN method.

Abstract

from arXiv · show

Recently, leveraging on the development of end-to-end convolutional neural networks (CNNs), deep stereo matching networks have achieved remarkable performance far exceeding traditional approaches. However, state-of-the-art stereo frameworks still have difficulties at finding correct correspondences in texture-less regions, detailed structures, small objects and near boundaries, which could be alleviated by geometric clues such as edge contours and corresponding constraints. To improve the quality of disparity estimates in these challenging areas, we propose an effective multi-task learning network, EdgeStereo, composed of a disparity estimation branch and an edge detection branch, which enables end-to-end predictions of both disparity map and edge map. To effectively incorporate edge cues, we propose the edge-aware smoothness loss and edge feature embedding for inter-task interactions. It is demonstrated that based on our unified model, edge detection task and stereo matching task can promote each other. In addition, we design a compact module called residual pyramid to replace the commonly-used multi-stage cascaded structures or 3-D convolution based regularization modules in current stereo matching networks. By the time of the paper submission, EdgeStereo achieves state-of-art performance on the FlyingThings3D dataset, KITTI 2012 and KITTI 2015 stereo benchmarks, outperforming other published stereo matching methods by a noteworthy margin. EdgeStereo also achieves comparable generalization performance for disparity estimation because of the incorporation of edge cues.

1 Introduction

EdgeStereo addresses difficult stereo correspondences by combining disparity estimation with edge detection in a unified multi-task network. It uses edge interactions and a compact residual pyramid while reporting strong benchmark performance and improved estimates in challenging regions.

  • Motivation: End-to-end stereo networks still struggle with correspondences in ill-posed regions, detailed structures, small objects and near boundaries.These limitations include local ambiguities in repeated, textureless and reflective regions.
  • Motivation: Edge contours provide geometric clues for depth changes and depth consistency, motivating joint exploitation of stereo and edge information.The paper connects object boundaries with depth changes between foreground and background.
  • Results: Incorporating edge cues refines disparity estimates in reflective regions, sky and image details, while the unified model improves both stereo matching and edge detection.The reported qualitative examples and multi-task results cover challenging regions and cross-task improvement.
  • Method: EdgeStereo jointly predicts disparity and edges through two branches that share shallow features and interact through edge feature embedding and edge-aware smoothness loss.The edge branch supplies fine-grained features and its edge map guides disparity smoothness during training.
  • Method: The residual pyramid is a compact 2-D-convolution decoder that estimates disparities at the smallest scale and residuals at other scales.This design forms an efficient one-stage disparity regression model and replaces more complex regularization structures.
  • Results: EdgeStereo achieves state-of-the-art performance on FlyingThings3D, KITTI 2012 and KITTI 2015 stereo benchmarks.The paper reports a noteworthy margin over other published stereo methods.

2 Related Work

Prior work evolved from hand-engineered stereo pipelines to end-to-end CNNs, while EdgeStereo extends this direction through joint stereo matching and edge detection. Its design targets efficient disparity estimation and cross-task use of edge information.

  • CNN-based Stereo Matching: CNN-based stereo methods replaced or supplemented hand-crafted matching, aggregation, optimization, and refinement components with learned representations and end-to-end pipelines.
  • End-to-end Stereo Matching: End-to-end stereo networks commonly use cascaded 2-D encoder-decoders or 3-D convolutional regularization, but these designs can be computationally burdensome or over-parameterized.
  • Edge Detection and Multi-task Learning: Unlike multi-stage or 3-D-convolution regularization, EdgeStereo uses a residual pyramid for efficient one-stage full-size disparity prediction.
  • Unsupervised Stereo Matching: Prior unsupervised stereo methods avoid ground-truth disparities but remain less comparable with supervised methods and use less robust image-gradient smoothness terms.
  • Edge Detection and Multi-task Learning: EdgeStereo introduces edge detection into stereo matching to provide semantic and high-frequency representations plus geometric constraints.

3 Approach

EdgeStereo unifies disparity estimation and edge detection in a shared-backbone multi-task network. It combines edge feature embedding, edge-aware smoothness constraints, and a residual pyramid to refine disparities efficiently from coarse to fine.

  • Basic Network Architecture: EdgeStereo maps a stereo pair to a disparity map and a reference-image edge map through differentiable modules informed by stereo geometry.
  • Basic Network Architecture: The two branches share shallow backbone features, while the disparity branch combines extraction, matching, and regularization in one network.
  • Basic Network Architecture: Edge feature embedding concatenates left-image descriptors, cost volume, and transformed edge features before disparity regularization.
  • Residual Pyramid: The residual pyramid directly estimates disparity at the smallest scale, then upsamples and refines it with residual signals at successively larger scales.
  • Incorporation of Edge Cues: Edge cues supplement fine-grained representations and impose object-level constraints because disparity boundaries should align with scene edge contours.
  • Residual Pyramid: At each scale, warped right-image features and correlation-based consistency information are processed to produce residual signals supervised by disparity and edge-aware losses.
  • Multi-task Effects: Experiments report that edge cues improve disparity estimation in detailed structures, reflective regions, and near boundaries, while stereo training also improves edge predictions without edge annotations.

4 Experiments

The experiments evaluate EdgeStereo through ablations, cross-task promotion tests, state-of-the-art stereo benchmark comparisons, and generalization experiments.

  • Experimental Program: Ablation studies verify EdgeStereo’s design choices and test whether stereo matching and edge detection promote each other in the unified model.
  • Experimental Program: EdgeStereo is compared with state-of-the-art stereo matching methods on FlyingThings3D, KITTI 2012, and KITTI 2015 benchmarks.
  • Experimental Program: The experiments also evaluate EdgeStereo’s generalization performance for disparity estimation.

4.1 Datasets and Evaluation Metrics

EdgeStereo uses multiple stereo and edge datasets spanning synthetic, real-world, urban, indoor, and natural-scene settings. Evaluation uses disparity-error and edge-detection F-measure metrics.

  • Datasets: Five stereo datasets support training and testing: FlyingThings3D, KITTI2012, KITTI2015, CityScapes, and Middlebury 2014.
  • Datasets: Two edge datasets are used: Multicue for training and BSDS500 only for testing.
  • Evaluation Metrics: Stereo evaluation reports EPE and the percentage of pixels with EPE larger than t pixels, while edge evaluation uses ODS and OIS F-measures.

4.2 Implementation Details

EdgeStereo uses staged training and dataset-specific finetuning, with shared implementation settings for edge and disparity learning.

  • Training setup: The edge branch is pretrained on fused Multicue and PASCAL VOC Context data before stereo training.The first stage uses SGD with minibatches of 16 images and a learning rate reduced every 10K iterations.
  • Training setup: The second stage fixes the edge branch while pretraining EdgeStereo on FlyingThings3D or combined FlyingThings3D and CityScapes data.This stage uses stereo pairs, poly learning-rate scheduling, and SGD.
  • Training setup: The third stage jointly pretrains both branches on the same stereo data, changing the base learning rate to 0.002.Other hyperparameters remain as in the second stage.
  • Dataset finetuning: Middlebury finetuning uses 35 image pairs for training and 15 for validation, with a 10K-iteration schedule.The pretrained model comes from FlyingThings3D, and all 50 pairs are used for submission finetuning.
  • Evaluation: Testing runs on one GTX 1080Ti GPU with dataset-specific input sizes, including 961×545 for FlyingThings3D and 1281×385 for KITTI.The KITTI submission input is enlarged to 1313×393.
  • Ablation baseline: Ablations use the disparity estimation sub-network without edge cues as the baseline model.The baseline structure is inferred from Table 1.

4.3 Ablation Studies

Ablations show that edge cues, the residual pyramid, and a tuned smoothness-loss intensity improve disparity estimation, while RP 4 is selected for the final model.

  • Edge cues: The 3-pixel error on KITTI 2012 falls from 2.844% to 2.555% with edge-aware smoothness and to 2.385% after edge-feature embedding.These results directly compare the baseline with successive edge-cue components.
  • Evaluation: Table 2 evaluates 3-pixel error and EPE on FlyingThings3D and KITTI 2012 and 2015 validation sets.FlyingThings3D evaluation is restricted to disparities below 192.
  • Smoothness regularization: The proposed edge-aware smoothness loss achieves the best performance on three datasets among the compared image-gradient regularizers.It also guides multi-task learning and helps refine edge predictions.
  • Residual pyramid: Residual pyramids improve disparity estimation, and their coarse-to-fine residual learning outperforms an additional cascaded refinement network.The comparison includes RP 2, RP 4, RP 8, no RP, and cascade variants.
  • Residual pyramid: RP 4 is selected for the final model, yielding a 2.289% error rate on KITTI 2012 and 0.740 EPE on FlyingThings3D.These are the reported best RP 4 results for the two stated evaluations.
  • Smoothness-loss intensity: The edge-aware smoothness intensity is best at β = 2, while β = 0 oversmooths disparities and large β values amplify edge-estimate noise.The β = 2 setting yields a 2.289% error rate on KITTI 2012.

4.4 Effectiveness of Multi-task Learning

Multi-task learning improves both edge predictions and disparity estimates near boundaries, including on datasets without edge annotations used in training.

  • Edge detection: Multi-task learning improves ODS and OIS F-measures across the BSDS500 training, validation, and test sets.BSDS500 is not used for pretraining, and FlyingThings3D provides no ground-truth edge annotations during training.
  • Edge detection: On KITTI 2015 training images, multi-task learning refines edge predictions and highlights details despite no KITTI 2015 training.The comparison is between the Multicue-pretrained edge sub-network and EdgeStereo after multi-task learning.
  • Cross-task interaction: The experiments support mutual improvement between stereo matching and edge detection under the edge-aware smoothness loss.This conclusion follows the reported quantitative and qualitative improvements in both tasks.
  • Disparity estimation: Incorporating edge cues reduces near-boundary 3-pixel errors by 14.1% on KITTI 2012 and 8.7% on KITTI 2015.The comparison is against the baseline model using predicted edges as boundaries for evaluation.

4.5 Comparison with Other Stereo Matching Methods

EdgeStereo performs strongly across stereo benchmarks and challenging regions, while its Middlebury result exposes a small-data limitation for the end-to-end model.

  • KITTI 2012: EdgeStereo is reported as outperforming all published stereo methods on the KITTI 2012 benchmark across all evaluation metrics.The evaluation includes erroneous-pixel percentages and EPE in non-occluded and all regions.
  • KITTI 2012: On KITTI 2012 reflective regions, EdgeStereo surpasses the baseline and other methods by a noteworthy margin.The comparison includes SegStereo and Displets, which use semantic or object knowledge to address ambiguities.
  • KITTI 2015: On the KITTI 2015 benchmark, EdgeStereo achieves the best performance against the baseline and published methods while being more efficient than 3-D convolutional and cascaded structures.The reported comparison uses D1-bg, D1-fg, and D1-all metrics.
  • Middlebury: On the half-size Middlebury validation set, EdgeStereo improves over the baseline from 12.473% to 11.540% in 2-pixel error and from 1.225 to 1.139 EPE.These results demonstrate the reported effect of incorporating edge cues for disparity learning.
  • Middlebury: EdgeStereo outperforms iResNet and PSMNet on Middlebury but performs worse than MC-CNN.The authors attribute the limitation to Middlebury’s small training set relative to EdgeStereo’s capacity and favor larger datasets for evaluation.

4.6 Generalization Performance

The section evaluates EdgeStereo’s generalization across KITTI, Middlebury, and indoor edgeless scenarios using synthetic or collaborative pretraining. EdgeStereo shows comparable cross-domain performance and remains effective without edge-rich scenes.

  • EdgeStereo achieves comparable generalization performance with DispNet and SegStereo and significantly outperforms PSMNet on KITTI training sets.It performs slightly worse than CRL and iResNet, which the authors relate to model capacity and domain differences between FlyingThings3D and KITTI.
  • The experiments compare qualitative disparity estimates and error maps on KITTI test sets and disparity estimations on the Middlebury validation set.
  • Table 12 evaluates FlyingThings3D-pretrained models on KITTI 2012 and 2015, while collaboratively pretrained models are evaluated on KITTI 2012 and Middlebury 2014.
  • After incorporating edge cues, EdgeStereo obviously outperforms the baseline when pretrained and evaluated across different domains.
  • 2.252% 3-pixel error is achieved on the challenging full-resolution Middlebury validation set in indoor edgeless scenarios.The collaboratively pretrained model also outperforms the baseline there, indicating that the edge branch does not harm performance in edgeless scenes.

4.7 Comparison with the Previous Version

The authors revise EdgeStereo’s architecture relative to their ACCV version by strengthening its backbone, residual pyramid, and multi-task interaction mechanism. The revised architecture outperforms the previous version across all reported test sets, including reflective regions.

  • The backbone changes from ImageNet-pretrained VGG16 to ResNet50, with a ResNet-like encoder for better intermediate representations.
  • The residual pyramid is redesigned to use multi-scale features for binocular warping and cost-volume generation.
  • The multi-task learning mechanism is refined, including the edge-aware smoothness component.
  • The new architecture outperforms the original version on all test sets and in challenging reflective regions.

5 Conclusion and Future Work

EdgeStereo jointly performs stereo matching and edge detection using edge feature embedding and edge-aware smoothness loss. The reported results show improved difficult-region disparity estimates, strong benchmark performance, mutual task benefits, and comparable generalization, while future work targets broader multi-task extensions.

  • EdgeStereo is a multi-task network for stereo matching and edge detection that incorporates edge cues through edge feature embedding and edge-aware smoothness loss.
  • Disparity estimates improve significantly in texture-less regions, large occlusions, detailed structures, and near boundaries after incorporating edge cues.
  • Edge predictions improve after multi-task learning even without ground-truth edge annotations, while disparity estimation retains comparable generalization capability.
  • Future work includes applying the mechanism to optical flow and multi-view reconstruction, modeling stereo geometry for edge detection, and adding segmentation tasks.
Loading 1903.01700v2…