Source-linked AI summary
T2Net: Synthetic-to-Realistic Translation for Solving Single-Image Depth Estimation Tasks
Chuanxia Zheng, Tat-Jen Cham, Jianfei Cai
TL;DR
Single-image depth estimation is constrained by the cost and availability of real image-depth or stereo pairs. T2Net trains an image translator and depth predictor using synthetic pairs and unpaired real images, and reports strong NYUDv2 and KITTI results despite lacking real paired training data, including performance beyond earlier real-data methods.
Problem
Single-image depth estimation commonly relies on large paired image-depth datasets or stereo pairs that are costly or difficult to acquire, while real images may lack ground-truth depth.
Method
T2Net jointly trains a wide-spectrum synthetic-to-realistic translator and a depth predictor using adversarial loss for synthetic inputs, reconstruction loss for real inputs, task loss, and feature GAN loss.
Results
T2Net produces good results on NYUDv2 and KITTI without real image-depth pairs or stereo pairs, and can outperform earlier deep-learning methods trained with real paired data or stereo pairs.
Takeaways & Limitations
End-to-end training couples realistic translation with depth prediction while preserving geometric content, providing a viable synthetic-and-unpaired-real training framework.
Takeaways & Limitations
The training setup assumes access to individual real images without stereo pairing or corresponding ground-truth depth maps, plus renderable synthetic 3D models with depth maps.
Abstract
from arXiv · showhide
Current methods for single-image depth estimation use training datasets with real image-depth pairs or stereo pairs, which are not easy to acquire. We propose a framework, trained on synthetic image-depth pairs and unpaired real images, that comprises an image translation network for enhancing realism of input images, followed by a depth prediction network. A key idea is having the first network act as a wide-spectrum input translator, taking in either synthetic or real images, and ideally producing minimally modified realistic images. This is done via a reconstruction loss when the training input is real, and GAN loss when synthetic, removing the need for heuristic self-regularization. The second network is trained on a task loss for synthetic image-depth pairs, with extra GAN loss to unify real and synthetic feature distributions. Importantly, the framework can be trained end-to-end, leading to good results, even surpassing early deep-learning methods that use real paired data.
1 Introduction
The paper addresses single-image depth estimation without costly real paired data by combining synthetic image-depth pairs with unpaired real images. Its end-to-end T2Net framework translates inputs toward realism while preserving depth-relevant content, achieving strong results without real pairs or stereo pairs.
- Single-image depth estimation requires extensive paired training data, whose acquisition is costly and difficult to generalize across datasets and environments.Specialized depth-sensing equipment makes scene-specific dataset construction high-effort and high-cost.
- The paper considers synthetic image-depth pairs alongside abundant unpaired real images as an alternative training scenario for estimating depth on real images.Synthetic scenes provide rendered images with corresponding depth maps, while real imagery lacks ground-truth depth.
- A naive synthetic-only estimator and a vanilla GAN translation pipeline are limited by synthetic-real differences and by insufficient preservation of physically correct depth relationships.The vanilla GAN encourages realistic image distributions but does not explicitly preserve correspondence with the source depth map.
- T2Net uses a wide-spectrum translator that applies reconstruction loss to real inputs and adversarial loss to synthetic inputs, reducing the need for ad hoc synthetic-image regularization.The translator is intended to modify realistic images less than obviously synthetic images while producing realistic outputs.
- The jointly trained architecture combines image translation and depth prediction, requires neither real image-depth pairs nor stereo pairs, and performs substantially better than synthetic-only approaches while surpassing some earlier real-data methods.The reported qualitative and quantitative comparisons include methods trained with real image-depth pairs or stereo pairs.
2 Related Work
Related work spans supervised and stereo-based depth estimation, unpaired image translation, and synthetic-to-real rendering. T2Net differs by treating translation as an intermediate step whose preservation of 3D shape semantics is central to depth estimation.
- Supervised deep-learning depth estimators learn image-to-depth mappings from paired data, while unsupervised methods reduce pair requirements but depend on stereo imagery and accurate camera rigs.Stereo dependencies make cross-dataset generalization difficult when baselines and relative poses differ.
- Unpaired image-to-image translation methods have pursued style transfer, shared representations, variational models, adversarial training, and cycle consistency.These approaches generally focus on producing meaningful or visually pleasing translated images without paired data.
- Unlike visually oriented translation work, T2Net treats image translation as an intermediate step for depth estimation, making fidelity of 3D shape semantics the overriding concern.The translation must preserve geometric content relevant to the downstream task rather than only match target-domain appearance.
- SimGAN-style self-regularization constrains generated images to resemble their sources, but the paper characterizes this loss as ad hoc and potentially too limiting for larger domain shifts.T2Net instead uses a principled reconstruction loss for real inputs.
3 Method
T2Net jointly trains a synthetic-to-realistic image translator and a depth predictor using synthetic image-depth pairs alongside unpaired real images. Its wide-spectrum translator uses adversarial training for synthetic inputs and reconstruction for real inputs, while feature alignment and task losses support depth prediction across domains.
- Training data: The framework assumes unpaired real images and synthetic image-depth pairs, avoiding the need for real image-depth or stereo pairs during training.Synthetic images are rendered from 3D models with corresponding depth maps, while real images lack paired depth annotations.
- Translation network: The wide-spectrum translator accepts both synthetic and real images and aims to apply the minimum realism-enhancing change needed for each input.Realistic inputs should be modified less than obviously synthetic inputs, preserving the shape semantics needed for depth prediction.
- Architecture: T2Net combines an image translation network with a task prediction network in a shared, end-to-end trainable architecture.The translation and depth-prediction networks are optimized jointly with shared weights across synthetic and real input pipelines.
- Translation network: Synthetic inputs receive adversarial translation and real inputs receive reconstruction, replacing heuristic self-regularization with a forward reconstruction constraint.The reconstruction loss is unidirectional rather than cyclical, and vanilla GAN training alone is described as insufficient for preserving scene geometry.
- Task prediction: Translated synthetic images remain paired with their original depth maps, enabling the task network to learn depth through a synthetic task loss.The task network directly measures per-pixel differences between predicted and synthetic ground-truth depth maps.
- Task prediction: Additional losses regularize predictions on real images and align feature distributions between real and translated-synthetic inputs.The framework uses an edge-aware local smoothness loss for real-image depth predictions and a feature-level GAN loss for encoder representations.
- Network design: The implementation uses a residual translation network, PatchGAN image discriminators, and a multi-scale depth predictor with dilation convolutions and weighted skip connections.These design choices target local realism discrimination, fine-grained detail preservation, and multi-scale scene understanding.
4 Experimental Results
Experiments on indoor and outdoor datasets evaluate the framework’s implementation, translation quality, depth accuracy, generalization, and ablations. Results show improved realism while preserving geometry, competitive depth estimation without real paired data, and benefits from end-to-end training and reconstruction loss.
- NYUDv2 Dataset: Translated SUNCG images become more similar to real NYUDv2 images in low-level appearance, including colors, textures, illumination, and shadows.Figure 3 compares synthetic renderings, GS→R outputs, and unrelated real NYUDv2 images.
- NYUDv2 Dataset: The full T2Net model surpasses the train-set-mean and synthetic-only baselines and exceeds an earlier supervised method on two of three NYUDv2 metrics without real paired data.It remains below the latest fully supervised methods using real image-depth pairs.
- KITTI Dataset: Outdoor translation substantially increases resemblance to real KITTI images while preserving synthetic scene geometry and replicating real colors, textures, and shadows.The qualitative results compare synthetic vKITTI, translated images, and real KITTI images.
- KITTI Dataset: 9.1% absolute improvement is achieved on KITTI over the all-synthetic model, and T2Net outperforms methods trained on real paired data and real stereo data.The best outdoor performance occurs without Dfeat, attributed to translated images being closer to real KITTI.
- Generalization: On Make3D, T2Net trained only with vKITTI pairs and unpaired real KITTI images achieves RMSE(m) 8.935, Log-10 0.574, Abs Rel 0.508, and Sqr Rel 6.589.Qualitative results are reasonable in most situations, with some failure cases.
- Ablation Study: End-to-end training outperforms separated training because task loss incentivizes translation to preserve synthetic geometric content.CycleGAN produced visually realistic images but distorted scene geometry, with performance close to separated training.
- Ablation Study: Removing real-image reconstruction loss makes high-resolution translation harder and causes accuracy to drop after early training, while L1 target reconstruction fails under large domain shifts.Without reconstruction, the best results occur at epoch 3; adversarial and self-domain reconstruction losses can oppose one another.
5 Conclusion and Future Work
T2Net performs single-image depth estimation using synthetic image-depth pairs and unpaired real images, without real paired or stereo training data. It achieves good NYUDv2 and KITTI results and surpasses early deep-learning methods trained on real paired data.
- T2Net requires only synthetic image-depth pairs and unpaired real images for training.
- The system combines an image translation network with a depth prediction network.Translation uses adversarial loss for synthetic inputs and target-domain reconstruction loss for real inputs; depth prediction combines task and feature GAN losses.
- T2Net can be trained end-to-end without real image-depth pairs or stereo pairs.
- T2Net produces good results on NYUDv2 and KITTI despite lacking real paired training data, and outperforms early deep-learning methods trained on real paired data.