Source-linked AI summary
Real-time self-adaptive deep stereo
Alessio Tonioni, Fabio Tosi, Matteo Poggi, Stefano Mattoccia, Luigi Di Stefano
TL;DR
Deep stereo networks lose accuracy under domain shift, while supervised target-domain tuning is costly and full online adaptation is computationally demanding. The paper proposes unsupervised continuous adaptation with MADNet and MAD, which independently updates network modules to retain real-time operation. Together, they achieve self-adaptation to new environments at 25 FPS and are reported to provide competitive accuracy.
Problem
Deep stereo accuracy decreases under domain shift, while effective target-domain tuning depends on scarce annotations or limited training samples.
Method
The paper combines a lightweight modular stereo architecture, MADNet, with MAD, which independently trains sub-portions online using unsupervised losses.
Results
MADNet with MAD adapts to unseen environments without supervision at approximately 25 FPS while maintaining competitive accuracy.
Takeaways & Limitations
The system supports real-time self-adaptive deep stereo for deployment across heterogeneous environments.
Takeaways & Limitations
Challenging scenes with differently viewed content can cause all evaluated models to fail to produce good predictions.
Abstract
from arXiv · showhide
Deep convolutional neural networks trained end-to-end are the state-of-the-art methods to regress dense disparity maps from stereo pairs. These models, however, suffer from a notable decrease in accuracy when exposed to scenarios significantly different from the training set, e.g., real vs synthetic images, etc.). We argue that it is extremely unlikely to gather enough samples to achieve effective training/tuning in any target domain, thus making this setup impractical for many applications. Instead, we propose to perform unsupervised and continuous online adaptation of a deep stereo network, which allows for preserving its accuracy in any environment. However, this strategy is extremely computationally demanding and thus prevents real-time inference. We address this issue introducing a new lightweight, yet effective, deep stereo architecture, Modularly ADaptive Network (MADNet) and developing a Modular ADaptation (MAD) algorithm, which independently trains sub-portions of the network. By deploying MADNet together with MAD we introduce the first real-time self-adaptive deep stereo system enabling competitive performance on heterogeneous datasets.
1. Introduction
Deep stereo networks lose accuracy under domain shift, while labeled target-domain data and full online adaptation are costly. The paper introduces MADNet and MAD to enable fast, unsupervised adaptation with competitive accuracy.
- Domain shift reduces deep stereo accuracy when training and testing scenarios differ, especially between synthetic training images and real deployment data.
- Continuous online adaptation uses unsupervised losses on incoming stereo pairs but greatly reduces inference speed when updating the whole network.
- MADNet is a lightweight stereo architecture with accuracy comparable to DispNetC, about one-tenth its parameters, and disparity inference around 40 FPS.
- MAD independently trains network sub-portions, enabling unsupervised adaptation to unseen environments at approximately 25 FPS.
- Online adaptation improves KITTI disparity maps in as few as 150 frames, with reported latencies of about 10 seconds for full adaptation and 6 seconds for MAD.
- The MADNet–MAD combination is presented as the first real-time, self-adapting deep stereo system.
2. Related work
Stereo research progressed from conventional pipelines and learned matching costs toward end-to-end CNNs and unsupervised image-reconstruction losses. These approaches improve learning without labels but remain connected to generalization and adaptation challenges.
- Early stereo learning estimated confidence measures, while CNN matching costs later replaced conventional costs within SGM pipelines.
- End-to-end deep learning eventually combined matching-cost computation and disparity selection in a unified model.
- External depth measurements such as Lidar can reduce domain-shift effects in end-to-end CNNs.
- Unsupervised depth estimation commonly warps views from stereo pairs or image sequences and minimizes image-reconstruction error; the principle also applies to stereo.
3. Online Domain Adaptation
The paper adapts stereo networks online from incoming frames without ground-truth labels, addressing the computational cost of full per-frame training with modular architecture and adaptation.
- Online Domain Adaptation: Online adaptation continuously fine-tunes a stereo model on each incoming pair using an unsupervised loss rather than ground-truth annotations.
- MADNet - Modularly ADaptive Network: A full train iteration for every frame can reduce inference rate to roughly one third, motivating the lightweight MADNet architecture.
- MADNet - Modularly ADaptive Network: MADNet uses pyramidal feature towers and independent modules to maximize speed and support modular adaptation across resolutions.
- MADNet - Modularly ADaptive Network: At the lowest resolution, correlated left-right features feed a disparity decoder, while higher-resolution decoders refine upsampled disparities using warped features and correlations.
- MADNet - Modularly ADaptive Network: A dilated-convolution refinement module operates at quarter resolution before bilinear upsampling produces the full-resolution disparity map.
- MADNet - Modularly ADaptive Network: MADNet reduces memory and computation through decimated resolutions, small correlation windows, and residual-displacement searches.
- MAD - Modular ADaptation: MAD partitions the network into non-overlapping modules, independently updating one module using its intermediate disparity prediction and loss.
- MAD - Modular ADaptation: Interleaving module optimization approximates full back-propagation over time, while single-module updates provide faster adaptation benefits through the cascade architecture.
4. Experimental results
Experiments on heterogeneous KITTI sequences show that online adaptation improves stereo accuracy, while MADNet and MAD make adaptation substantially faster and suitable for real-time deployment. Longer sequences further narrow or surpass the gap to offline ground-truth fine-tuning.
- Evaluation protocol: 43k KITTI images across Road, Residential, Campus, and City evaluate online adaptation under heterogeneous driving conditions without target-domain fine-tuning data.The sequences contain 5674, 28067, 1149, and 8027 frames respectively.
- Online adaptation: Full online adaptation consistently reduces DispNetC outliers across environments, nearly halving D1-all on Campus, Residential, and City and approaching one-third on Road.Average EPE also improves substantially, reaching nearly 40% relative improvement on Road.
- Online adaptation: 14.26 FPS versus 5.22 FPS: MADNet with full adaptation runs nearly three times faster than fully adapted DispNetC while exceeding its accuracy in all sequences except Campus.MADNet also achieves better accuracy than DispNetC under comparable ground-truth fine-tuning while running about 2.5 times faster.
- Modular adaptation: 25 FPS: MAD adaptation significantly improves MADNet over synthetic-only training while adding relatively little computational overhead.The method trades some short-term accuracy for speed, with sufficiently long sequences recovering comparable accuracy.
- Adaptation over time: 900 frames for full adaptation and about 1600 frames for MAD: online adaptation reaches performance nearly equivalent to offline ground-truth fine-tuning after sufficient observations.The corresponding delays are about one minute and 64 seconds, respectively.
- Long-term adaptation: Approximately 43000-frame sequences spanning multiple environments allow full online adaptation to achieve lower average errors than corresponding offline ground-truth fine-tuned models.The long-term evaluation concatenates Campus, City, Residential, and Road without resetting the network.
- Embedded deployment: 0.26s versus 0.76–0.96s per forward pass: MADNet is faster than StereoNet on an NVIDIA Jetson TX2 while also being more accurate.This supports MADNet’s suitability for low-power embedded deployment.
5. Conclusions and future work
The paper concludes that online unsupervised adaptation addresses domain shift in deep stereo, while MADNet and MAD enable efficient real-time deployment. It reports 25 FPS adaptation with improved accuracy over alternatives and identifies extending MAD as future work.
- Online unsupervised fine-tuning can successfully address domain adaptation for end-to-end disparity regression networks.
- MADNet and MAD adapt to new environments at 25 FPS while yielding better accuracy than alternatives such as DispNetC.
- Future work will test MAD on other end-to-end stereo systems and investigate selecting which network portions to update online.
Supplementary material for “Real-time self-adaptive deep stereo”
The supplementary material identifies the paper’s authors and their affiliation with the University of Bologna’s Department of Computer Science and Engineering.
- The paper is authored by Alessio Tonioni, Fabio Tosi, Matteo Poggi, Stefano Mattoccia, and Luigi di Stefano.
- The authors are affiliated with the Department of Computer Science and Engineering at the University of Bologna, Italy.
1. Detailed MADNet structure
MADNet uses shared-weight feature extractors for the stereo pair and builds multiresolution disparity modules that combine cost volumes with upsampled lower-resolution estimates. A residual refinement module produces the final full-resolution disparity.
- Feature extraction: MADNet uses two parameter-sharing pyramidal feature extractors to independently process the left and right frames.
- Disparity estimation: The architecture builds one disparity estimation module for each of six resolutions.
- Disparity estimation: Each module computes a correlation cost volume between same-resolution left and right features, optionally using warping to align features.
- Disparity estimation: The module concatenates the cost volume with an upscaled lower-resolution disparity to refine the estimate at the current resolution.
- Residual refinement: At the highest resolution, atrous convolutions and residual connections refine disparity before bilinear interpolation recovers full resolution.
2. Implementation and training details for MADNet and MAD
MADNet is pretrained on FlyingThings3D, partitioned into independently trainable resolution-based modules, and adapted online with photometric consistency losses. The supplementary material also specifies the MAD algorithm and implementation resources.
- Implementation resources: The supplementary material includes implementation details, convolutional extractor specifications, and a project repository link.
- Pre-Training: MADNet is initially trained for 1200000 iterations on FlyingThings3D using Adam with a learning rate of 0.0001.
- Architecture: The implementation describes a six-resolution stereo estimation architecture using bilinear upsampling and correlation-based cost volumes.
- Adaptation: The network is partitioned into five independently trainable portions by grouping layers operating at the same resolutions.
- Online loss: The adaptation loss combines photometric consistency, SSIM on 3 × 3 patches, and L1 distance with weights 0.85 and 0.15.
- MAD algorithm: Algorithm 1 repeatedly reads frames, computes predictions and losses, samples a module, and updates its adaptation state online.
3. Detailed algorithm for one online adaptation step using MAD
MAD performs online adaptation by evaluating each incoming stereo pair, selecting a network module, updating it with an unsupervised loss, and adjusting future sampling probabilities.
- MAD initializes one histogram bin per module to track sampling probabilities during online adaptation.
- For each stereo pair, the network predicts disparity and computes a loss from full-resolution disparity and potentially the input frames.The loss can use a reprojection error between the left and right frames.
- MAD selects a module and performs one optimization step using the loss computed on its lower-scale disparity prediction.
- The algorithm estimates the expected loss by linear interpolation, compares it with the measured loss, and updates module sampling probabilities according to effectiveness.The expected loss is Lexp = 2 · Lt−1 − Lt−2.
- MAD uses independently sampled network portions to support fast online adaptation.Figure 4 reports sampling frequencies for the independent portions of MADNet.
4. MAD sampling policy
MAD’s reward-punishment sampling policy concentrates training on network portions that improve adaptation, especially lower-resolution modules.
- MAD tracks how often each network portion is sampled by adapting MADNet five times on the full KITTI raw dataset.
- MAD favors fine-tuning lower-resolution modules, particularly the 1/16 portion of the network.
- The preference for lower-resolution modules is consistent with MADNet’s coarse-to-fine architecture, which starts with low-resolution disparity and iteratively refines it.
5. Qualitative comparison between fast networks
MADNet is qualitatively compared with DispNetC and StereoNet on KITTI 2015 disparity predictions, with better preservation of thin structures than StereoNet.
- MADNet’s KITTI 2015 disparity maps are compared qualitatively with predictions from DispNetC and StereoNet.Figure 5 presents the reference image followed by disparity maps from the three architectures.
- MADNet better maintains thin structures than StereoNet in the qualitative comparison.
6. Qualitative Results on Online Adaptation
Online adaptation improves MADNet predictions in outdoor KITTI and indoor Wean Hall sequences. Full-network adaptation improves sooner, while MAD gradually closes the gap and reaches comparable performance on longer sequences.
- Online adaptation is visualized at 40FPS inference alongside Full and MAD adaptation configurations.The supplementary video uses bright colors for high disparity and dark colors for low disparity.
- In the KITTI residential sequence, full-network adaptation resolves most mistakes in about 150 frames, whereas MAD requires about 400 frames before comparable improvement.These correspond to roughly 10 seconds at 15FPS and 16 seconds at 25FPS, respectively.
- Over the final approximately 500 KITTI frames, MAD and full-network adaptation converge to comparable performance.
- In the indoor Wean Hall sequence, both adaptation strategies substantially improve disparity quality, with full adaptation reaching good performance sooner than MAD.
- Figure 6 shows Full Adaptation resolving most KITTI disparity mistakes by 100 frames, while MAD initially reduces their magnitude only slightly.
- MAD improves sharply around iteration 500 and converges with Full Adaptation to similar predictions by frame 2000.
- On Wean Hall, Full Adaptation has no macro mistakes by frame 500, while MAD reaches comparable performance after about 1500 frames.
- Reflections from neon lights can make the reprojection loss produce unhelpful gradients, and all three configurations fail in those challenging scenes.The authors plan to address this with multiple unsupervised losses.