Source-linked AI summary
Unsupervised Adversarial Depth Estimation using Cycled Generative Networks
Andrea Pilzer, Dan Xu, Mihai Marian Puscas, Elisa Ricci, Nicu Sebe
TL;DR
Supervised depth estimation achieves strong performance but depends on costly ground-truth annotations. The paper proposes an adversarial cycled generative network that predicts stereo disparity through closed-loop view synthesis, with experiments on KITTI and Cityscapes demonstrating effectiveness and competitive state-of-the-art results.
Problem
Supervised deep depth estimation requires costly ground-truth depth annotations for training.
Method
Two generative sub-networks jointly use adversarial learning and closed-loop view synthesis to predict forward and reverse disparity maps from calibrated stereo images.
Results
Experiments on KITTI and Cityscapes demonstrate the effectiveness of the adversarial image synthesis and cycled generative network structure.
Takeaways & Limitations
The approach shows that stereo depth estimation can be tackled through unsupervised adversarial learning of image synthesis.
Takeaways & Limitations
Future work includes attention for feature representations and graphical models for more structured disparity predictions.
Abstract
from arXiv · showhide
While recent deep monocular depth estimation approaches based on supervised regression have achieved remarkable performance, costly ground truth annotations are required during training. To cope with this issue, in this paper we present a novel unsupervised deep learning approach for predicting depth maps and show that the depth estimation task can be effectively tackled within an adversarial learning framework. Specifically, we propose a deep generative network that learns to predict the correspondence field i.e. the disparity map between two image views in a calibrated stereo camera setting. The proposed architecture consists of two generative sub-networks jointly trained with adversarial learning for reconstructing the disparity map and organized in a cycle such as to provide mutual constraints and supervision to each other. Extensive experiments on the publicly available datasets KITTI and Cityscapes demonstrate the effectiveness of the proposed model and competitive results with state of the art methods. The code and trained model are available on https://github.com/andrea-pilzer/unsup-stereo-depthGAN.
1. Introduction
Depth estimation has advanced with deep learning, but supervised approaches require costly ground-truth depth data. This paper proposes an unsupervised adversarial approach using cycled generative networks to predict disparity from calibrated stereo images.
- Supervised CNN-based depth estimation requires costly ground-truth depth data, especially because deep architectures typically need large training datasets.
- Unsupervised methods learn the disparity map between rectified views from a calibrated stereo camera and derive depth using camera parameters.
- The proposed network uses two generative sub-networks to predict forward and reverse disparities and synthesize views in a closed loop.
- Adversarial learning jointly optimizes the generators, while mutual cycle constraints provide supervision from both image views.
- The final disparity map combines the outputs of the two generators, and experiments evaluate the approach on KITTI and Cityscapes.
2. Related Work
Related work spans supervised depth estimation, unsupervised stereo and image-synthesis approaches, and adversarial generation models. The proposed work extends unsupervised stereo depth estimation by applying adversarial learning and cycled view synthesis.
- Supervised Depth Estimation: Supervised depth estimation has benefited from large annotated datasets and deep neural networks for monocular and stereo settings.
- Unsupervised Depth Estimation: Unsupervised depth estimation avoids costly ground-truth annotations by learning disparity directly or indirectly through image synthesis.
- Unsupervised Depth Estimation: Prior unsupervised methods use stereo matching, image synthesis, left-right consistency, or jointly learned camera pose for depth estimation.
- GANs: GAN research developed adversarial models for image generation, including CycleGAN and DualGAN, while comparatively few works considered depth estimation.
- GANs: The proposed comparison distinguishes direct disparity prediction from indirect adversarial image synthesis and cycled generative networks.
3. The Proposed Approach
The proposed approach estimates stereo disparity through adversarial image synthesis, using two generators organized in a cycle to constrain one another. Reconstruction, adversarial, and consistency objectives train the model, whose final disparity averages outputs from both directions.
- Unsupervised Adversarial Depth Estimation: The model indirectly predicts disparity by synthesizing one calibrated stereo view from the other with a warping operation.A generator predicts disparity, which is used to warp the input image into the opposite view; an L1 reconstruction loss supplies supervision.
- Unsupervised Adversarial Depth Estimation: Two generative sub-networks estimate disparity in opposite directions and fuse complementary predictions through concatenation and 1×1 convolution.The generators use convolutional encoder-decoder structures, and their distinct disparity maps are combined into an enhanced disparity map.
- Unsupervised Adversarial Depth Estimation: Adversarial learning improves synthesized-view generation by training discriminators to distinguish reconstructed images from real stereo views.The optimization combines reconstruction and adversarial losses, with weighted terms used to balance their magnitudes.
- Cycled Generative Networks for Adversarial Depth Estimation: The cycled architecture reconstructs both views in a closed loop and adds consistency supervision between disparity maps from the two half-cycles.Because the maps correspond to different views and are not aligned, warping is used before applying the L1 consistency loss.
- Cycled Generative Networks for Adversarial Depth Estimation: The full objective combines reconstruction losses, adversarial losses, and half-cycle consistency, while inference averages the two output disparity maps.The final disparity map is obtained by treating the two half-cycles with equal importance.
- Network Implementation: The generators use ResNet-50 encoders, five deconvolutional ReLU upsampling stages, and skip connections for feature aggregation.The implementation also uses shared encoder options and convolutional discriminators for the two image views.
4. Experimental Results
Experiments on KITTI and Cityscapes evaluate training setup, ablations, qualitative outputs, runtime, and comparisons with state-of-the-art methods. The results support stereo inputs, adversarial learning, and the full cycled architecture, while Cityscapes evaluation uses the KITTI protocol because no official depth protocol exists.
- Experimental Setup: Experiments use KITTI and Cityscapes, with KITTI Eigen-split training/testing and image preprocessing resized to 512×256.KITTI contains 22,600 training image pairs and 697 test pairs; Cityscapes images are cropped and resized before training.
- Experimental Setup: The study evaluates standard depth-estimation metrics, including Abs Rel, squared relative error, and root mean squared error.The evaluation follows metrics used in prior work and compares estimated depth with ground-truth depth over test pixels.
- Ablation Study: Half-Cycle Stereo outperforms Half-Cycle Mono on seven of eight KITTI metrics, indicating gains from stereo images and complementary disparity maps.The comparison is part of the KITTI ablation study.
- Ablation Study: Around 1.7 points on Abs Rel separate Half-Cycle + D from Half-Cycle Stereo, while adversarial training also yields a more stable convergence trend.The authors associate this improvement with better image synthesis and disparity prediction, while describing the optimization explanation as probable.
- Ablation Study: Full-Cycle + D improves Abs Rel by around 2 points and accuracy a1 by around 1.9 points over Half-Cycle + D, with better results across all metrics.The authors attribute the improvement to stronger mutual constraint and supervision from the cycled design.
- Ablation Study: Full-Cycle + D + SE performs better than Full-Cycle + D, while Cityscapes shows similar trends but slightly worse first-two error metrics for Full-Cycle + D.Cityscapes results are evaluated using the KITTI protocol and directly on disparity maps because no official depth-estimation protocol exists.
- State-of-the-Art Comparison: On KITTI, the fully unsupervised model is competitive with the best supervised method and close to the strongest unsupervised competitor, while outperforming AdaDepth.The comparison uses Eigen test split results with Garg image cropping and depth predictions capped at the common threshold of 80 meters.
- Efficiency: Inference takes around 0.140 seconds per 512×256 image, described by the authors as near-realtime processing.Training on KITTI takes around 45 hours on a single Tesla K80 GPU.
5. Conclusion
The paper presents unsupervised adversarial depth estimation through a cycled generative network that predicts disparity from calibrated stereo views. Experiments on KITTI and Cityscapes demonstrate effectiveness and competitive KITTI performance, while future work targets richer feature guidance and structured disparity prediction.
- 5. Conclusion: The approach uses adversarial learning in a cycled generative network to estimate depth without supervised depth labels.Two symmetric generators predict disparity between stereo views and form a closed cycle that constrains and supervises both generators.
- 5. Conclusion: Two symmetric generative sub-networks synthesize different stereo views and are merged into a closed cycle for mutual optimization.The cycle provides strong constraints and supervision during training.
- 5. Conclusion: Experiments on KITTI and Cityscapes demonstrate the model’s effectiveness, with competitive performance against state-of-the-art methods on KITTI.The conclusion reports results across two publicly available datasets and specifically highlights KITTI competitiveness.
- 5. Conclusion: Future work will use attention to guide generator representations and graphical models to improve structured disparity predictions.These directions aim to produce predictions with better scene structures.