Source-linked AI summary
EdgeStereo: A Context Integrated Residual Pyramid Network for Stereo Matching
Xiao Song, Xu Zhao, Hanwen Hu, Liangji Fang
TL;DR
Stereo networks remain limited in non-textured regions, boundaries, and tiny details. EdgeStereo addresses these gaps with a context-integrated residual disparity network and an edge sub-network, achieving state-of-the-art performance on Scene Flow and KITTI stereo benchmarks while improving both tasks.
Problem
End-to-end stereo methods still struggle with local ambiguities, computationally expensive structures, and inaccurate predictions at thin structures or boundaries.
Method
EdgeStereo jointly predicts disparity and edges using a multi-scale context pyramid, one-stage residual pyramid, embedded edge features, and edge-aware smoothness regularization.
Results
EdgeStereo achieves state-of-the-art performance on Scene Flow and KITTI Stereo 2012 and 2015 benchmarks, with both disparity estimation and edge detection improved after multi-task learning.
Takeaways & Limitations
Context cues support ill-posed-region matching, while edge cues preserve subtle disparity details within the unified model.
Takeaways & Limitations
Stereo-dataset evaluation cannot quantify edge-task improvement because those datasets do not provide ground-truth edge maps.
Abstract
from arXiv · showhide
Recent convolutional neural networks, especially end-to-end disparity estimation models, achieve remarkable performance on stereo matching task. However, existed methods, even with the complicated cascade structure, may fail in the regions of non-textures, boundaries and tiny details. Focus on these problems, we propose a multi-task network EdgeStereo that is composed of a backbone disparity network and an edge sub-network. Given a binocular image pair, our model enables end-to-end prediction of both disparity map and edge map. Basically, we design a context pyramid to encode multi-scale context information in disparity branch, followed by a compact residual pyramid for cascaded refinement. To further preserve subtle details, our EdgeStereo model integrates edge cues by feature embedding and edge-aware smoothness loss regularization. Comparative results demonstrates that stereo matching and edge detection can help each other in the unified model. Furthermore, our method achieves state-of-art performance on both KITTI Stereo and Scene Flow benchmarks, which proves the effectiveness of our design.
1 Introduction
EdgeStereo addresses persistent stereo-matching weaknesses in ill-posed regions, boundaries, and thin details by combining context and edge cues in a multi-task network. Its context and residual pyramids support disparity estimation, while joint edge learning and multi-phase training produce strong benchmark performance.
- Motivation: End-to-end stereo networks still struggle with local ambiguities in ill-posed regions, expensive cascade or 3D-convolution structures, and inaccurate thin or boundary predictions.These limitations motivate broader context modeling and detail-preserving refinement.
- Evidence: Edge cues improve estimates in detailed regions, while context cues improve disparities in ill-posed areas such as shadowed roads.Figure 1 provides qualitative comparisons against variants without the context pyramid or edge branch.
- Proposed approach: EdgeStereo jointly learns disparity estimation and edge detection, using edge cues and edge-aware smoothness loss to guide disparity learning.The multi-task labels need not come from a single dataset because training is performed in multiple phases.
- Proposed approach: The context pyramid encodes multi-scale information for ill-posed regions, while the one-stage residual pyramid replaces more complicated cascade refinement structures.The disparity branch aggregates contextual cues into a hierarchical scene prior before residual refinement.
- Results: EdgeStereo achieves state-of-the-art performance on Scene Flow and the KITTI Stereo 2012 and 2015 benchmarks.The model is trained first on Scene Flow and then fine-tuned on KITTI datasets.
2 Related Work
Related work replaces traditional stereo stages with learned components, incorporates contextual information, and explores multi-task vision models. EdgeStereo differs by using multi-phase training to combine disparity and edge learning without requiring joint labels in one dataset.
- Deep stereo methods: Traditional stereo pipelines have been progressively replaced by learned modules for matching costs, global disparity prediction, and disparity refinement.These approaches retain a decomposition of the classical stereo process while substituting networks for hand-crafted components.
- Deep stereo methods: End-to-end methods combine stereo-processing steps for joint optimization, using encoder-decoder, cascade-refinement, or context-volume architectures.DispNet, its residual refinement extension, and GC-Net represent distinct architectural directions.
- Multi-task vision: Earlier multi-task stereo methods jointly addressed stereo with object segmentation, foreground recognition, or other scene tasks, but hand-crafted designs were not robust.These methods established the value of combining related visual tasks while leaving robustness concerns.
- Edge detection: Fully convolutional edge detectors can be incorporated into disparity-estimation networks to provide features that preserve disparity details.HED and richer-convolutional-feature variants provide the relevant edge-detection precedent.
- Multi-task vision: Unlike iterative multi-task training in SegFlow, EdgeStereo uses multi-phase training so disparity can exploit stable boundary information from a pretrained edge branch.It also avoids requiring disparity and edge labels from the same dataset.
3 Approach
EdgeStereo combines a context-integrated disparity network with an edge sub-network. Its context pyramid captures multi-scale scene information, while a one-stage residual pyramid and edge guidance refine disparities and preserve details.
- Basic Architecture: EdgeStereo shares shallow backbone computation between disparity and edge branches for unified multi-task processing.The edge branch supplies edge features and edge maps to the disparity branch.
- Basic Architecture: Mixed image, correspondence, and edge features are fused before the context pyramid builds a hierarchical scene prior.Reduced image features, correlation cost volumes, and edge features are concatenated and fused by a 1 × 1 convolution.
- Context Pyramid: The context pyramid uses multiple receptive fields to encode contextual cues for correspondence inference in ill-posed regions.Its branches capture context at different scales, and their outputs are concatenated with the input representation.
- Residual Pyramid: The residual pyramid predicts disparity at the smallest scale, then estimates residuals at larger scales to produce the full-resolution map.This one-stage design replaces more expensive cascade refinement structures and follows an easy-to-tough multi-scale strategy.
- Cooperation of Edge Cues: Edge features are embedded at context and residual-pyramid scales to restore low-level representations lost through convolution and down-sampling.The embedding supports accurate disparity and residual prediction, especially for subtle details.
- Cooperation of Edge Cues: An edge-aware smoothness loss encourages smooth disparities in non-edge regions while permitting discontinuities at object contours.The regularization weights disparity gradients using edge-map gradients rather than intensity variation.
- Multi-phase Training Strategy: A three-phase training strategy separates edge-network pretraining from disparity supervision and avoids requiring paired disparity and edge labels.The backbone is fixed across phases; the edge branch is pretrained, then multi-scale disparity losses are optimized.
4 Experiments
The experiments evaluate EdgeStereo components on Scene Flow and compare the method with state-of-the-art stereo methods on KITTI benchmarks. The implementation uses a VGG-16 backbone, multi-scale residual prediction, and an HEDβ edge sub-network.
- Evaluation Setup: Experiments evaluate EdgeStereo components on Scene Flow and compare it with state-of-the-art stereo matching methods on KITTI benchmarks.The study also examines whether multi-task learning improves edge maps.
- Backbone: The model uses VGG-16, sharing backbone layers from conv1_1 through conv3_3 between the disparity and edge tasks.Shared features have one-quarter the spatial size of the raw images.
- Disparity Network: The encoder downsamples by a factor of 64 and supplies seven output scales to the residual pyramid.Each scale uses four 3 × 3 convolution layers before regressing disparity or residual maps.
- Edge Network: HEDβ modifies HED with five side branches that produce edge probability maps from progressively deeper VGG-16 features.Each side branch uses two 3 × 3 convolutions, upsampling, and a 1 × 1 output convolution.
- Context Pyramid: The context-pyramid variants use convolution, pooling, or dilation to obtain representations with different receptive fields.The pooling variant applies differently sized average-pooling kernels, 1 × 1 convolutions, and upsampling.
4.2 Datasets and Evaluation Metrics
The evaluation uses synthetic Scene Flow and real-world KITTI stereo datasets, with BSDS500 and PASCAL VOC Context supporting edge-subnetwork pretraining. Stereo quality is measured using EPE and t-pixel error.
- Datasets: Scene Flow contains 35,454 training and 4,370 test image pairs with dense ground-truth disparities.
- Datasets: KITTI 2012 and KITTI 2015 provide sparse ground-truth disparities across 194/195 and 200/200 training/test pairs, respectively.
- Datasets: BSDS500, combined with PASCAL VOC Context, is used to pretrain the edge sub-network.
- Evaluation Metrics: EPE measures average Euclidean disparity error, while t-pixel error measures the percentage of pixels whose error exceeds t pixels.
4.4 Ablation Studies
Ablations on Scene Flow show that stronger feature extraction and multi-scale context reduce disparity error. Hybrid feature fusion and the selected context pyramid each improve the 3-pixel error over simpler alternatives.
- Feature Extraction: Replacing DispFulNet unary feature extraction with a shallow VGG-16 backbone reduces 3-pixel error from 8.61% to 6.83%.
- Feature Extraction: Adding a 1 × 1 convolution over unary features and cost volumes further reduces 3-pixel error to 6.70%.The resulting hybrid feature fuses local semantic features with matching-cost distributions.
- Context Pyramid: Using context pyramid P-2 4 8 16 reduces 3-pixel error from 6.70% without a context pyramid to 5.95%.The authors associate this improvement with multi-scale context cues for dense disparity estimation.
Encoder-Decoder (Residual Pyramid).
The residual pyramid replaces direct disparity regression or more elaborate cascade refinement with multi-scale residual learning. In the final CP-RPN configuration, this design achieves lower error than the baseline.
- Encoder-Decoder (Residual Pyramid).: Replacing the DispFulNet encoder-decoder with a residual pyramid reduces 3-pixel error from 6.70% to 5.96%.The experiment uses the same encoder and attributes the improvement to multi-scale residual learning.
- Encoder-Decoder (Residual Pyramid).: CP-RPN achieves 5.33% 3-pixel error and 1.15 EPE, outperforming the baseline by 3.28% and 0.60.CP-RPN combines hybrid feature extraction, context pyramid P-2 4 8 16, and the residual-pyramid encoder-decoder.
- Encoder-Decoder (Residual Pyramid).: Pooling context pyramid P-2 4 8 16 performs best among the tested context pyramids, while larger dilation rates can harm context extraction.The tested D-12 9 6 3 and D-24 18 12 6 configurations obtain 5.52% and 5.88% 3-pixel error, respectively.
Context Pyramid Comparisons.
Compared with multi-stage refinement networks, CP-RPN delivers comparable or better stereo accuracy with lower computational cost. Edge-subnetwork cooperation further improves stereo models and preserves subtle details.
- Comparisons with Multi-stage Refinement: CP-RPN outperforms the three-stage DRR refinement model by 0.15% while running 2.3 times faster.
- Comparisons with Multi-stage Refinement: Against the two-stage CRL structure, CP-RPN has almost equal performance but uses fewer parameters and runs faster.
- Benefits from Edge Cues: Cooperating the edge sub-network with different disparity networks improves all tested stereo matching models and reduces error by preserving subtle details.
4.5 Comparisons with Other Stereo Methods
EdgeStereo is evaluated against traditional and end-to-end stereo methods on Scene Flow and KITTI benchmarks, achieving strong performance in challenging image regions.
- EdgeStereo achieves the best performance on Scene Flow across two evaluation metrics.
- On KITTI 2012, EdgeStereo handles large occlusions, texture-less regions, and reflective areas by leveraging context and edge cues.
- EdgeStereo achieves state-of-the-art performance on KITTI 2015, while its one-stage structure is faster than most stereo models.
- On KITTI, incorporating edge cues improves disparity accuracy in thin structures, near boundaries, and upper image regions.
4.6 Better Edge Map
The paper evaluates whether multi-task learning improves edge maps using visual demonstrations and quantitative BSDS500 results, since stereo datasets lack ground-truth edge labels.
- Stereo datasets cannot directly evaluate edge-task improvement after multi-task learning because they provide no ground-truth edge maps.
- EdgeStereo produces edge maps with finer details than HEDβ without multi-task learning in Scene Flow visual demonstrations.
- ODS F-measure is 0.795 for HEDβ after multi-task learning, compared with 0.790 for original HEDβ and 0.788 for baseline HED on BSDS500.
5 Conclusion
The conclusion presents EdgeStereo as a multi-task architecture that integrates edge cues with context and residual pyramids, achieving state-of-the-art results on Scene Flow and KITTI benchmarks.
- EdgeStereo incorporates edge cues into the disparity estimation pipeline within a unified multi-task architecture.
- The context pyramid and residual pyramid target challenging scenarios while providing an effective one-stage structure.
- KITTI 2012 results are ranked by 3-pixel error in non-occluded regions, while KITTI 2015 results use D1-all error over all pixels.
- Qualitative KITTI results are organized with input images, disparity predictions, and error maps across KITTI 2012 and 2015 test sets.