Source-linked AI summary
SegStereo: Exploiting Semantic Information for Disparity Estimation
Guorun Yang, Hengshuang Zhao, Jianping Shi, Zhidong Deng, Jiaya Jia
TL;DR
Disparity estimation struggles with local ambiguity when stereo regions lack sufficient matching clues. SegStereo incorporates semantic features into the disparity pipeline and uses semantic softmax loss alongside standard training objectives, achieving state-of-the-art KITTI results with effectiveness or adaptability reported on CityScapes and FlyingThings3D.
Problem
Deep learning disparity estimators still struggle with local ambiguity because ambiguous regions provide insufficient matching clues for correct prediction.
Method
SegStereo embeds segmentation features into the disparity branch and regularizes semantic consistency with semantic softmax loss for unsupervised and supervised training.
Results
SegStereo achieves state-of-the-art performance on KITTI Stereo and shows effectiveness or adaptability on CityScapes and FlyingThings3D.
Takeaways & Limitations
Semantic-guided disparity estimation produces more reliable results, especially in ambiguous areas, across unsupervised and supervised settings.
Abstract
from arXiv · showhide
Disparity estimation for binocular stereo images finds a wide range of applications. Traditional algorithms may fail on featureless regions, which could be handled by high-level clues such as semantic segments. In this paper, we suggest that appropriate incorporation of semantic cues can greatly rectify prediction in commonly-used disparity estimation frameworks. Our method conducts semantic feature embedding and regularizes semantic cues as the loss term to improve learning disparity. Our unified model SegStereo employs semantic features from segmentation and introduces semantic softmax loss, which helps improve the prediction accuracy of disparity maps. The semantic cues work well in both unsupervised and supervised manners. SegStereo achieves state-of-the-art results on KITTI Stereo benchmark and produces decent prediction on both CityScapes and FlyingThings3D datasets.
1 Introduction
SegStereo addresses local ambiguity in disparity estimation by incorporating semantic information into a unified stereo framework. Semantic feature embedding and semantic loss support both unsupervised and supervised training, with reported benchmark effectiveness across multiple datasets.
- Disparity estimation finds corresponding pixels in stereo images to infer object distance and supports depth prediction, scene understanding, and autonomous driving.
- Deep learning methods still struggle with local ambiguity because ambiguous regions lack sufficient matching clues for correct disparity convergence.
- Semantic cues provide high-level information about foreground, background, familiar object scale, and semantic consistency that can help address ambiguous regions.
- SegStereo integrates a segmentation sub-network with a disparity network, connecting semantic features to the disparity branch through semantic feature embedding.The model uses a ResNet encoder with correlation operations, deconvolutional decoding, and fully convolutional end-to-end training.
- Semantic softmax loss is combined with photometric consistency for unsupervised learning and disparity regression loss for supervised learning.The semantic feature embedding and softmax loss introduce semantic-consistency constraints during training.
- SegStereo achieves state-of-the-art results on KITTI Stereo datasets, while results on CityScapes and FlyingThings3D further demonstrate effectiveness or adaptability.
2 Related work
Related work spans supervised and unsupervised stereo matching, semantic-guided vision algorithms, and architectures that combine task-specific branches. SegStereo differs by embedding semantic information and propagating semantic supervision into disparity estimation.
- Supervised Stereo Matching: Supervised stereo methods evolved from hand-crafted local descriptors and global optimization to CNN matching costs, multi-scale embeddings, and cost-volume architectures.
- Supervised Stereo Matching: Fully convolutional and correlation-based networks enabled end-to-end disparity-map learning from stereo feature representations.
- Unsupervised Stereo Matching: Unsupervised stereo methods reduce dependence on expensive ground-truth labels by learning from photometric, projection, or alignment-based objectives.
- Semantic-Guided Algorithms: Semantic-guided algorithms use scene parsing, object knowledge, instance cues, or semantic masks to improve stereo, optical-flow, or scene-flow estimation.
- Semantic-Guided Algorithms: SegFlow jointly learns segmentation and optical flow with concatenated branch features, whereas SegStereo focuses on disparity and warps semantic supervision into the disparity branch.
- SegStereo Framework: SegStereo's framework extracts stereo features, computes a correlation cost volume, embeds left segmentation features into disparity, and uses warped right-view semantic prediction with softmax regularization.
3 Our Method
SegStereo incorporates semantic cues into disparity estimation through feature embedding and semantic loss regularization, targeting local ambiguities and flat regions. The framework supports both unsupervised and supervised training.
- 3.1 Basic Network Architecture: SegStereo combines a ResNet-50/correlation disparity backbone with a segmentation branch whose semantic features are embedded into disparity processing.Left disparity features, correlated stereo features, and left semantic features are concatenated before decoding a full-size disparity map.
- 3.2 Semantic Feature Embedding: Semantic feature embedding fuses low-level disparity representations with high-level recognition information, providing object-level priors on flat regions.The segmentation branch shares shallow computation with the disparity network and is described as efficient and representationally effective.
- 3.3 Semantic Loss Regularization: Semantic loss regularization warps right semantic features into the left view and compares classified warped maps with left segmentation labels.The semantic loss is propagated back through the semantic classifier and feature-warping layer.
- 3.3 Semantic Loss Regularization: Semantic loss adds object-aware constraints alongside photometric or regression loss, and experiments report improved resolution of local disparity ambiguities.The reported benefit is especially emphasized for unsupervised learning.
- 3.4 Objective Function: In unsupervised training, disparity is learned from photometric consistency between a warped right image and the original left image, with masking for invalid correspondences.A smoothness penalty on disparity gradients is also used to reduce local incoherence.
- 3.4 Objective Function: In supervised training, an ℓ1 disparity regression loss is combined with smoothness and semantic losses when ground-truth disparity is available.The supervised objective uses valid pixels from the ground-truth disparity map and includes a regression-term weight.
4 Experimental Results
Experiments evaluate SegStereo across unsupervised and supervised settings on KITTI, CityScapes, and FlyingThings3D. Semantic feature embedding and softmax regularization reduce disparity errors, while the model achieves strong benchmark performance and transfers across datasets.
- Datasets and Evaluation Metrics: The experiments evaluate unsupervised and supervised models on KITTI Stereo 2015, with additional evaluations on CityScapes and FlyingThings3D.Evaluation uses EPE and D1, reporting both non-occluded and all-pixel errors.
- Implementation Details: SegStereo uses PSPNet-50 semantic features, 1D correlation, feature concatenation, residual encoder blocks, and a deconvolutional decoder to output full-size disparity maps.The segmentation network's shallow layers and weights are fixed during SegStereo training; correlated and transformed features form the hybrid feature map.
- Unsupervised Learning: 20% lower EPE and 15% lower D1 result from semantic feature embedding with photometric loss alone.With photometric and smoothness losses, the corresponding improvements are 12% for EPE and 13% for D1.
- Unsupervised Learning: Adding softmax loss after photometric and smoothness losses reduces EPE from 2.17 to 1.89 and D1 from 10.53 to 10.03 on all pixels.The softmax-loss gain mainly appears on large objects such as roads and cars, improving locally ambiguous regions.
- Unsupervised Learning: Fine-tuning on 200 KITTI 2015 training images further reduces prediction errors, and SegStereo outperforms two other unsupervised methods on the benchmark.The fine-tuning process uses photometric, smoothness, and softmax losses.
- Cross-Dataset Results: SegStereo produces better global structures and object details than SGM on CityScapes, achieves state-of-the-art results on KITTI 2015, and outperforms ResNetCorr on FlyingThings3D.The FlyingThings3D comparison reports state-of-the-art performance and indicates effectiveness across different datasets.
5 Conclusion
SegStereo integrates semantic feature maps and semantic softmax loss into disparity prediction, supporting both unsupervised and supervised training. It achieves state-of-the-art performance on KITTI Stereo and shows adaptability on CityScapes and FlyingThings3D.
- SegStereo integrates semantic feature maps into the disparity prediction pipeline.
- A combined softmax and photometric or disparity regression loss enables both unsupervised and supervised training.
- Semantic guidance produces more reliable disparity results, especially in ambiguous areas.
- SegStereo achieves state-of-the-art performance on the KITTI Stereo benchmark.
- Results on CityScapes and FlyingThings3D further demonstrate the method's adaptability.