Source-linked AI summary
Domain-invariant Stereo Matching Networks
Feihu Zhang, Xiaojuan Qi, Ruigang Yang, Victor Prisacariu, Benjamin Wah, Philip Torr
TL;DR
Stereo matching networks often fail on unseen environments because domain differences destabilize their learned features. DSMNet uses domain normalization and a trainable non-local graph-based filter to improve domain-invariant representations, and it achieves strong cross-domain performance when trained only on synthetic data. Its real datasets are used only for testing because their quantity or ground-truth labels are limited.
Problem
Stereo matching networks struggle to generalize to unseen data because domain differences destabilize features and produce wrong matching results.
Method
DSMNet combines domain normalization with a learnable non-local graph-based filter to regulate feature distributions and capture robust structural and geometric representations.
Results
Without fine-tuning or adaptation, DSMNet far outperforms almost all state-of-the-art models trained on the same synthetic dataset and surpasses some fine-tuned supervised models.
Takeaways & Limitations
DSMNet provides a domain-invariant stereo matching network for cross-domain disparity estimation without further fine-tuning or adaptation.
Takeaways & Limitations
Real datasets are used only as test sets because their quantity or ground-truth labels are insufficient for training deep learning models.
Abstract
from arXiv · showhide
State-of-the-art stereo matching networks have difficulties in generalizing to new unseen environments due to significant domain differences, such as color, illumination, contrast, and texture. In this paper, we aim at designing a domain-invariant stereo matching network (DSMNet) that generalizes well to unseen scenes. To achieve this goal, we propose i) a novel "domain normalization" approach that regularizes the distribution of learned representations to allow them to be invariant to domain differences, and ii) a trainable non-local graph-based filter for extracting robust structural and geometric representations that can further enhance domain-invariant generalizations. When trained on synthetic data and generalized to real test sets, our model performs significantly better than all state-of-the-art models. It even outperforms some deep learning models (e.g. MC-CNN) fine-tuned with test-domain data.
1. Introduction
Stereo matching networks struggle to generalize from synthetic or specific training domains to unseen real scenes because domain differences destabilize learned features. DSMNet addresses this with domain normalization and non-local filtering, achieving strong cross-domain results without adaptation.
- Stereo reconstruction estimates 3D geometry by computing disparities between matching pixels in stereo image pairs.
- State-of-the-art networks generalize poorly to unseen data because color, illumination, contrast, and texture differ across domains.
- Domain differences make learned features unstable, distorted, and noisy, producing incorrect matching results without retraining or adaptation.
- DSMNet combines domain normalization with a learnable non-local graph-based filter to regulate feature distributions and capture robust structural and geometric representations.
- Without fine-tuning or adaptation on real test datasets, DSMNet outperforms almost all state-of-the-art models trained on the same synthetic data.
- Figure 1 shows DSMNet producing less artifact-ridden features and accurate disparity estimates on novel KITTI scenes compared with GANet.
2. Related Work
Prior stereo methods use patch similarities, correlations, or cost volumes, but generally require substantial labeled data and remain weak under domain shifts. Related domain-generalization and normalization methods motivate DSMNet’s focus on domain-invariant stereo features.
- Deep stereo methods include patch-wise similarity models, correlation-based networks, and cost-volume architectures.
- These models commonly require many training samples with ground-truth disparity and generalize poorly from one domain to new scenes without fine-tuning or retraining.
- Self-supervised methods based on image reconstruction struggle with occlusions and reflective regions and do not generalize well to new domains.
- Domain Adaptation: Domain adaptation methods require many target-domain stereo images, whereas domain generalization assumes no target information for adaptation or fine-tuning.
- Domain Generalization: Prior domain-generalization work learns invariant features through source-domain strategies, meta-learning, adversarial networks, or normalization improvements.
- Stereo matching has received relatively little attention for improving end-to-end domain-generalization ability, motivating domain-invariant stereo networks.
3. Proposed DSMNet
DSMNet combines domain normalization with graph-based non-local filtering to build stereo representations that are less sensitive to domain shifts and local artifacts. Its architecture applies these components throughout feature extraction and cost aggregation while retaining efficient graph propagation.
- Network Architecture: The architecture uses GANet as its baseline, removes a domain-dependent local guided aggregation layer, and inserts domain normalization and non-local filters across the network.Domain normalization follows each convolutional layer in feature extraction and guidance, while non-local filters operate in feature extraction and cost aggregation.
- Domain Normalization: Instance normalization reduces image-level style variation but does not regulate the pixel-wise norm distribution needed for stereo correspondence.Stereo matching depends on consistent scaling of each pixel’s C-channel feature vector when computing matching costs and similarities.
- Domain Normalization: Domain normalization regulates feature distributions across spatial and channel dimensions to reduce sensitivity to dataset-specific appearance differences.It combines spatial normalization with channel-wise L2 normalization, followed by trainable per-channel scale and shift parameters.
- Non-local Graph-based Filtering: The non-local filter constructs an 8-connected image graph, splits it into two reverse directed graphs, and aggregates information along feasible paths.Path weights are products of learned edge weights, while the total weight between nodes sums contributions from all valid paths.
- Non-local Graph-based Filtering: Unlike local filters and spatially unaware attention mechanisms, the graph-based filter propagates long-range information while preserving spatial and structural context.This design is intended to reduce dependence on domain-sensitive local patterns and avoid oversmoothing depth edges and thin structures.
- Non-local Graph-based Filtering: The aggregation process is implemented iteratively over both directed graphs with linear time complexity O(n), and its weight transformation improves training robustness while reducing computational cost.The updated representation from G1 is used as input for aggregation with G2, and backpropagation reverses the propagation equation.
4. Experimental Results
Experiments evaluate DSMNet trained on synthetic data against real datasets, ablations, alternative strategies, and fine-tuned models. Results show stronger cross-domain accuracy, component benefits, and efficient integration into other stereo networks.
- 4.1. Datasets: DSMNet is trained only on synthetic data and evaluated on real datasets used as test sets for cross-domain generalization.The evaluation includes KITTI, Middlebury, Cityscapes, and ETH 3D datasets; existing real datasets are reserved for testing because of limited quantity or imperfect labels.
- 4.2. Ablation Study: The full DSMNet improves accuracy over the baseline by 3% on KITTI and 8% on Middlebury.Domain normalization contributes about 1.5% and non-local filter layers contribute another 1.4% on KITTI.
- 4.3. Component Analysis and Comparisons: DSMNet’s domain normalization outperforms batch, instance, and adaptive batch-instance normalization for domain-invariant stereo matching.The comparison keeps the remaining training and evaluation settings fixed while replacing only the normalization method.
- 4.3. Component Analysis and Comparisons: The graph-based non-local filter outperforms alternative propagation, denoising, and attention strategies for robust domain-invariant matching.The paper attributes this advantage to structural and geometric context with spatial constraints, avoiding over-smoothing of depth edges.
- 4.4. Cross-Domain Evaluations: DSMNet reduces error rates by 3–30% against state-of-the-art models across four real datasets when all models are trained on synthetic data.It also outperforms traditional methods such as SGM, costfilter, and patchmatch.
- 4.5. Fine-tuning: DSMNet without fine-tuning outperforms several supervised models trained or fine-tuned on KITTI and produces more accurate object boundaries than other fine-tuned state-of-the-art models.With fine-tuning, DSMNet outperforms most recent models; adding the graph-based filter to GANet produces 1.77% accuracy on KITTI 2015.
5. Conclusion
DSMNet combines domain normalization with non-local graph-based filtering to improve cross-domain stereo matching. Evaluation on four real datasets shows superior accuracy over other state-of-the-art stereo matching networks.
- DSMNet uses domain normalization to address feature distribution shifts and non-local graph filtering to capture robust structural and geometric features.Both layers are end-to-end trainable components of the proposed network.
- Evaluation on four real datasets demonstrates superior cross-domain accuracy compared with other state-of-the-art stereo matching networks.
A. Proof of Footnote 1
The proof establishes the graph-based propagation equivalence through path-based expansion and mathematical induction. It also describes inverse iterative backpropagation for the filtering operation.
- Graph paths reaching node p pass through neighboring nodes q, allowing W(q,p) to be expanded through neighborhood relations.
- Mathematical induction over the ordered nodes p0,p1...pn...pN proves the stated equivalence, beginning at n = 0 and extending to n = t + 1.
- Backpropagation for ωe and C(p) is computed inversely, with temporary gradient variables calculated iteratively.The inverse propagation proceeds in the order pN,pN−1,...p0.
C. Details of the Architecture
The DSMNet architecture uses multiple non-local filtering layers and replaces Batch Normalization with Domain Normalization in specified feature-extraction pathways. The filtering remains efficient and adds no new parameters.
- DSMNet contains seven non-local filtering layers for feature extraction and cost aggregation.
- Domain Normalization replaces Batch Normalization after each 2D convolutional layer in the feature extraction and guidance networks.
- Inference time increases by about 5% compared with the baseline, while the proposed layers introduce no new parameters.
- Table 6 reports elapsed time and parameter count for the efficiency comparison.
E. Carla Dataset
The Carla supplementary dataset expands synthetic stereo data for outdoor driving scenes with varied resolutions, focal lengths, and camera baselines. Figure 9 provides an example of Carla stereo data.
- Carla is used to generate stereo pairs for outdoor driving scenes because Sceneflow contains only about 7,000 stereo pairs for diving scenes.
- The supplementary dataset includes image resolutions of 720 × 1080 and 1080 × 1920, three focal lengths, and six camera baselines ranging from 0.2-1.5m.
- Figure 9 shows an example of the Carla stereo data.
F. More Results
DSMNet captures non-local structure and shape information rather than mainly local patterns, producing feature maps without artifacts under domain shifts.
- DSMNet mainly captures non-local structure and shape information, which are robust for cross-domain generalization.
- State-of-the-art models mainly learn local patterns that produce artifacts such as noise under domain shifts.
- DSMNet’s feature maps contain no artifacts in the reported comparison.
F.2. Disparity Results on Different Datasets
The experiments compare models trained on synthetic data and tested on multiple real datasets. DSMNet preserves thin structures while capturing structural and geometric context without the oversmoothing associated with other non-local strategies.
- All compared models are trained on the synthetic dataset and tested on real KITTI, Middlebury, ETH3D, and Cityscapes datasets.
- DSMNet’s graph-based filtering captures structural and geometric context for robust domain-invariant stereo matching.
- Non-local neural network denoising and non-local attention can oversmooth depth edges and thin structures because they lack spatial constraints.
- Cross-domain feature comparisons show local-pattern features from baseline models develop artifacts such as noise under domain shifts.
- DSMNet produces accurate disparity estimation on new datasets without fine-tuning.
- DSMNet preserves thin structures in disparity maps that non-local attention and non-local denoising easily erode.