Source-linked AI summary
AdaDepth: Unsupervised Content Congruent Adaptation for Depth Estimation
Jogendra Nath Kundu, Phani Krishna Uppala, Anuj Pahuja, R. Venkatesh Babu
TL;DR
Monocular depth estimation is limited by costly, noisy ground truth and poor transfer from synthetic training data to natural scenes, while prior adversarial adaptation largely targets classification. AdaDepth uses unsupervised adversarial adaptation with explicit content consistency on adapted representations, delivering competitive unsupervised results and state-of-the-art semi-supervised performance.
Problem
Monocular depth estimation lacks abundant clean ground truth, and synthetic-trained models generalize poorly to natural scenes because of domain shift.
Method
AdaDepth performs unsupervised adversarial adaptation on structured encoder representations while enforcing content consistency through feature reconstruction regularization.
Results
AdaDepth-U delivers comparable results to previous state-of-the-art methods on NYUD and KITTI, while AdaDepth-S outperforms previous fully supervised methods.
Takeaways & Limitations
AdaDepth demonstrates that synthetic RGB-D representations can be adapted to natural scenes while maintaining spatial content during adversarial learning.
Takeaways & Limitations
Training ADDA without regularization was unstable and failed to converge, while the Residual Transfer Framework made hyperparameter selection difficult.
Abstract
from arXiv · showhide
Supervised deep learning methods have shown promising results for the task of monocular depth estimation; but acquiring ground truth is costly, and prone to noise as well as inaccuracies. While synthetic datasets have been used to circumvent above problems, the resultant models do not generalize well to natural scenes due to the inherent domain shift. Recent adversarial approaches for domain adaption have performed well in mitigating the differences between the source and target domains. But these methods are mostly limited to a classification setup and do not scale well for fully-convolutional architectures. In this work, we propose AdaDepth - an unsupervised domain adaptation strategy for the pixel-wise regression task of monocular depth estimation. The proposed approach is devoid of above limitations through a) adversarial learning and b) explicit imposition of content consistency on the adapted target representation. Our unsupervised approach performs competitively with other established approaches on depth estimation tasks and achieves state-of-the-art results in a semi-supervised setting.
1. Introduction
AdaDepth targets monocular depth estimation under scarce, noisy ground truth and the domain gap between synthetic training data and natural scenes. It combines adversarial adaptation with content consistency to adapt synthetic representations to real scenes.
- Motivation: Synthetic-trained depth models often perform sub-optimally on real scenes because the source and test distributions differ.The paper frames this as a domain-generalization problem when synthetic images do not match natural-scene distributions.
- Motivation: Depth sensors produce noisy artifacts, while manual correction is laborious and prone to pixel-level annotation errors.These limitations make clean ground-truth depth scarce for supervised training.
- Approach: AdaDepth adapts representations learned from synthetic RGB-D pairs to real natural scenes.The setup is designed to reduce dependence on scarce clean ground-truth data.
- Approach: The method explores unsupervised adversarial domain adaptation for the spatially structured regression task of depth estimation.The paper identifies this as distinct from the predominantly classification-focused domain-adaptation literature.
- Contributions: AdaDepth addresses adversarial mode collapse by enforcing content consistency on the adapted representation through feature reconstruction regularization.Its contributions include adaptation at high-dimensional structured encoder representations and evaluation on NYU Depth and KITTI.
2. Related work
Prior work spans supervised and geometry-based depth estimation, synthetic-scene transfer, and domain adaptation using discrepancy or adversarial objectives. AdaDepth applies adversarial domain adaptation at intermediate feature representations to pixel-wise depth regression.
- Supervised Monocular Depth Estimation: Supervised monocular depth estimation progressed from hand-crafted and probabilistic models to CNNs, multi-scale features, semantic cues, and graphical refinements.The cited work includes semantic segmentation, surface normals, CRFs, and continuous depth prediction.
- Unsupervised/Semi-supervised Depth Estimation: Unsupervised and semi-supervised depth methods use geometric cues such as stereo image alignment, left-right consistency, and sparse ground-truth depth.These approaches reduce reliance on dense depth annotation by exploiting paired views or sparse supervision.
- Transfer learning using Synthetic Scenes: Synthetic datasets have supported pose, optical-flow, and semantic-segmentation tasks, but models trained on them do not directly generalize to natural RGB scenes.Physically based rendering was proposed to reduce the real-synthetic gap through improved lighting.
- Domain adaptation: Deep domain-adaptation methods commonly combine classification and discrepancy losses, including Maximum Mean Discrepancy and correlation alignment.These methods match distribution statistics or reduce representation discrepancies between source and target domains.
- Domain adaptation: AdaDepth uses adversarial learning at an intermediate feature level for the more challenging pixel-wise regression task of depth estimation.This contrasts with prior approaches that typically adapt class outputs through fully connected layers.
3. Approach
AdaDepth adapts a synthetic-trained monocular depth model to real images by aligning latent and prediction representations adversarially while preserving target content through consistency regularization. It updates target-domain representations using shared encoder-decoder structures and alternative regularization frameworks.
- Adversarial Objectives: AdaDepth models synthetic and real images as source and target distributions without access to real depth labels.The base model separates image-to-latent transformation from latent-to-depth transformation, with target inference affected by domain shift.
- Adversarial Objectives: The method adapts the target encoder representation while leveraging similarities in objectness information and relative object placement across domains.The target transformation is adapted instead of both encoder and decoder transformations.
- Adversarial Objectives: Two discriminators align source and target latent representations and distinguish source depth maps from target predictions at the prediction level.The feature discriminator operates on latent representations, while the depth discriminator operates on predicted outputs.
- 3.2.1 Domain Consistency Regularization (DCR): Adversarial adaptation alone can cause mode collapse, producing predictions inconsistent with the corresponding input image.AdaDepth therefore adds content-congruent regularization to preserve information during adaptation.
- 3.2.2 Residual Transfer Framework (RTF): The Residual Transfer Framework models target features as Ms + ∆M, constraining a trainable residual branch to keep perturbations small.Only the residual branch parameters are trainable during this adaptation process.
4. Experiments
Experiments evaluate AdaDepth-U and AdaDepth-S on NYU Depth v2 and KITTI, using synthetic pretraining, convolutional feature adaptation, and limited labeled target data for the semi-supervised variant.
- Datasets and settings: AdaDepth-U and AdaDepth-S are evaluated on NYU Depth v2 and KITTI datasets.AdaDepth-U uses no target ground-truth depth, while AdaDepth-S uses limited labeled target samples.
- Base network: The base network uses ResNet-50 followed by upprojection layers and is trained on synthetic images with BerHu loss.Pre-training uses 21,260 synthetic RGB-D pairs from Virtual KITTI.
- Adaptation network: Adaptation operates on a spatial convolutional latent representation with a standard convolutional discriminator.The latent representation has spatial size 8×10 with 2048 activation channels.
- Reported comparisons: On NYU Depth v2, AdaDepth-U performs competitively while AdaDepth-S outperforms the compared methods, which use full ground-truth supervision.The comparison is reported in Table 3.
- Adaptation network: The adaptation stage updates the discriminator and generator with Momentum optimization and λ = 10.The feature-reconstruction branch is updated before adversarial training and then jointly with the discriminators and target predictor.
- Semi-supervised training: AdaDepth-S initializes from AdaDepth-U and alternates labeled and unlabeled target batches to reduce over-fitting.For labeled batches, BerHu loss between predicted and ground-truth depth replaces the content loss.
5. Results
Results show benefits from content-consistency design choices, competitive unsupervised performance on NYU Depth v2 and KITTI, and reasonable generalization to Make3D; ADDA training was unstable without regularization.
- Content consistency: Adversarial domain adaptation improves performance over direct inference from the synthetic-trained network on target samples.Feature Consistency Framework performs better than the other evaluated regularization methods.
- Comparison with existing approaches: AdaDepth-S outperforms existing state-of-the-art methods on KITTI in 4 out of 7 metrics, whereas AdaDepth-U shows comparable metrics.With the exception of [52], compared methods use depth or pose ground-truth supervision.
- Content consistency: Feature Consistency Framework preserves content by learning an inverse mapping during unpaired adaptation, while DY preserves the continuous-valued depth-like structure.The combination of DF and DY addresses both marginal and conditional domain discrepancies.
- Comparison with existing approaches: AdaDepth-U delivers comparable results to previous state-of-the-art methods on NYU Depth v2 and KITTI, while AdaDepth-S outperforms prior fully supervised methods.The comparison is reported for both natural-scene datasets.
- Comparison with existing approaches: ADDA training was unstable without regularization and did not converge, while DAN results are reported on NYU Depth v2.These observations arise from the paper’s comparison with alternative adaptation setups.
- Generalization to Make3D: On Make3D, models adapted from Virtual KITTI generalize and perform reasonably well against previous methods despite semantic and physical dataset differences.Both unsupervised and semi-supervised adaptation are evaluated, with errors computed for depths below 70 meters.
6. Conclusion
AdaDepth adapts depth predictions from synthetic RGB-D pairs to natural scenes while preserving spatial content during adversarial learning. Benchmark results support its effectiveness and motivate future multi-task and geometric-cue extensions.
- AdaDepth adapts depth predictions from synthetic RGB-D pairs to natural scenes through unsupervised domain adaptation.
- The Feature Consistency Framework maintains spatial content information during adversarial learning.
- Benchmark results illustrate AdaDepth’s effectiveness on challenging datasets.
- Future work considers multi-task learning and exploiting global geometric cues from synthetic images.