Source-linked AI summary
ATGV-Net: Accurate Depth Super-Resolution
Gernot Riegler, Matthias Rüther, Horst Bischof
TL;DR
Consumer depth sensors provide dense measurements but suffer from noise and low lateral resolution, motivating single-map depth super-resolution without an auxiliary image. ATGV-Net combines a deep convolutional network with anisotropic TGV variational refinement, trained end-to-end by unrolling optimization, and achieves state-of-the-art results across four benchmarks while using large-scale synthetic training data.
Problem
Consumer depth sensors produce noisy, incomplete, and low-resolution measurements, while intensity guidance is not always available.
Method
ATGV-Net combines a deep network predicting depth and discontinuities with anisotropic TGV variational refinement, trained end-to-end by unrolling optimization.
Results
ATGV-Net improves upon state-of-the-art results on four benchmark datasets and can be trained entirely on large quantities of synthetic data.
Takeaways & Limitations
The method is especially useful when low-resolution depth contains noise, as with most consumer depth sensors.
Takeaways & Limitations
Training assumes accurate high-resolution ground-truth depth maps for each input sample.
Abstract
from arXiv · showhide
In this work we present a novel approach for single depth map super-resolution. Modern consumer depth sensors, especially Time-of-Flight sensors, produce dense depth measurements, but are affected by noise and have a low lateral resolution. We propose a method that combines the benefits of recent advances in machine learning based single image super-resolution, i.e. deep convolutional networks, with a variational method to recover accurate high-resolution depth maps. In particular, we integrate a variational method that models the piecewise affine structures apparent in depth data via an anisotropic total generalized variation regularization term on top of a deep network. We call our method ATGV-Net and train it end-to-end by unrolling the optimization procedure of the variational method. To train deep networks, a large corpus of training data with accurate ground-truth is required. We demonstrate that it is feasible to train our method solely on synthetic data that we generate in large quantities for this task. Our evaluations show that we achieve state-of-the-art results on three different benchmarks, as well as on a challenging Time-of-Flight dataset, all without utilizing an additional intensity image as guidance.
1 Introduction
Depth sensors provide useful measurements but remain noisy, incomplete, and low-resolution. ATGV-Net combines deep convolutional prediction with anisotropic variational refinement to recover high-resolution depth from a single low-resolution map without requiring an intensity guide.
- Consumer depth sensors are constrained by noise, quantization, missing values, and low spatial resolution.
- Intensity-guided methods can be problematic when no additional camera exists or when noisy depth must be projected onto the guidance image.
- Depth data contain textureless piecewise-affine regions and sharp discontinuities, making generic patch-based image super-resolution suboptimal.
- ATGV-Net combines a deep convolutional network with a variational model to refine a high-resolution estimate from one low-resolution depth map.
- The network predicts both high-resolution depth and discontinuity locations, which weight anisotropic TGV regularization during variational refinement.
- The joint model is trained end-to-end by unrolling primal-dual optimization and improves upon state-of-the-art results on four benchmark datasets.
2 Related Work
Single-image super-resolution has progressed from patch-based and sparse-coding methods toward deep learning, while depth super-resolution often relies on intensity guidance. ATGV-Net instead integrates an exact unrolled variational optimizer with a deep network for depth-only reconstruction.
- Depth Super-Resolution: Single-image super-resolution methods use one low-resolution image rather than a series of aligned images.
- Depth Super-Resolution: Patch-based approaches search for related patches across scales or among 3D point neighborhoods to estimate high-resolution depth.
- Depth Super-Resolution: Machine-learning approaches include sparse coding, regression forests, and deep convolutional networks for predicting high-resolution outputs.
- Depth Super-Resolution: Depth super-resolution has commonly used a high-resolution intensity image because large accurate depth datasets were historically scarce.
- Deep Network Integration of Energy Minimization Methods: Energy-minimization models combine unary data terms with pairwise terms that encode dependencies between neighboring pixels.
- Deep Network Integration of Energy Minimization Methods: Earlier differentiable variational approaches required smooth energy functionals, whereas ATGV-Net unrolls every step of an exact optimization scheme.
- Deep Network Integration of Energy Minimization Methods: Unrolling permits stronger pairwise regularization and more robust training of deeper networks than implicit differentiation approaches.
3 ATGV-Net
ATGV-Net combines a deep convolutional network with an unrolled variational optimization procedure to super-resolve a single noisy depth map. The model uses learned depth estimates and discontinuities within an anisotropic TGV framework and trains the complete pipeline end-to-end.
- ATGV-Net takes a single low-resolution, potentially noisy depth map and computes a high-resolution output after bilinear upsampling.Training pairs contain mid-resolution inputs and accurate high-resolution ground truth.
- The model formulates training as a bi-level problem in which learned weights make the lower-level energy minimizer achieve low loss on training samples.The loss is evaluated on the variational solution u* and uses the Euclidean norm in the described formulation.
- Network Architecture: The network predicts a high-resolution depth estimate and depth-discontinuity weights from the mid-resolution input, while a learned parameter controls the data–regularization trade-off.The deep network realizes g and h; h is used to weight the pairwise regularization term.
- Unrolling the Optimization: Unrolling fixed primal-dual optimization iterations into network operations removes restrictive energy-function assumptions and enables joint back-propagation training.The unrolled procedure trains optimization parameters together with the deep-network parametrization.
- Variational Model: The variational model uses second-order TGV regularization to favor piecewise affine depth surfaces, with an anisotropic tensor enforcing less smoothness across discontinuities and more in homogeneous regions.The energy combines a pairwise regularization term with an ℓ2 data term, and the tensor is based on the Nagel-Enkelmann operator.
- Network Architecture: The architecture uses 10 convolutional layers, and its primal, dual, gradient, transpose-gradient, projection, and over-relaxation operations are implemented as network layers or fixed operations.The displayed architecture uses 3×3 filters with 64 hidden feature maps; gradients use fixed convolutional filters.
4 Evaluation
The evaluation uses large-scale synthetic training data and tests ATGV-Net across clean, noisy, and real Time-of-Flight depth-super-resolution benchmarks. Results compare interpolation, guidance-based methods, the CNN alone, and increasingly integrated variational models.
- Evaluation Scope: The evaluation includes noise-free Middlebury, Laserscan, noisy Middlebury, and real ToFMark experiments.The noisy Middlebury setting adds depth-dependent Gaussian noise to simulate Time-of-Flight acquisition, while ToFMark uses 120 × 160 inputs and 610 × 810 target resolution.
- 4.1 Training Data: More than 40,000 high-quality synthetic depth maps are generated with a GPU ray-caster for training.Scenes contain randomly placed cuboids and spheres with varied scales and rotations; low-resolution inputs are created by resampling and optionally adding depth-dependent noise.
- 4.2 Clean Middlebury & Laserscan: On noise-free Middlebury and Laserscan data, the deep network significantly improves RMSE over competing methods across datasets and upsampling factors.The CNN also outperforms methods using an additional guidance image, while the variational model further improves performance and end-to-end training gives the best overall results.
- 4.3 Noisy Middlebury: End-to-end training substantially improves over the CNN alone on noisy Middlebury data, reducing noise in homogeneous regions while preserving fine structures near discontinuities.The CNN alone already outperforms guidance-based state-of-the-art methods for most images and upsampling factors; CNN + ATGV-L2 provides only a slight improvement.
- 4.4 ToFMark: On real ToFMark data, ATGV-Net is at least on par with state-of-the-art methods that use an additional intensity image as guidance.The benchmark contains three ToF depth maps with accurate structured-light ground truth, evaluated using RMSE in millimeters.
5 Conclusion
The method combines a deep convolutional network with a variational model, using unrolled optimization for end-to-end training and achieving state-of-the-art results on four benchmarks. The authors also identify noisy depth inputs as an especially relevant setting and propose extending the model to larger missing regions.
- Method: ATGV-Net combines a deep convolutional network with a variational model to super-resolve single depth maps.The network computes the high-resolution depth map and depth discontinuities, while the variational model further refines the output.
- Training: Unrolling the variational optimization procedure enables end-to-end optimization of the joint model.
- Results: Training solely on massive synthetic depth data produced state-of-the-art results on four different benchmarks.
- Scope: The model is especially useful when low-resolution depth maps contain noise, as in most consumer depth sensors.
- Future work: Future work will address larger areas of missing pixels by setting w_λ = 0 where depth measurements are missing.
- Acknowledgment: The work was supported by Infineon Technologies Austria AG and the Austrian Research Promotion Agency under the FIT-IT Bridge program.
6 Supplemental Material
The supplemental material explains how a primal-dual iteration is realized as deep-network layers and adds quantitative and qualitative evaluations across datasets and upsampling factors. It reports MAE results on noisy Middlebury and ToFMark data and presents comparisons with guidance-image methods.
- Primal-dual network: A feed-forward graph represents one primal-dual iteration through layers of a deep network.The graph distinguishes network inputs, previous-iteration values, current outputs, and computation layers.
- Qualitative results: Qualitative evaluations cover four benchmark datasets and multiple upsampling factors with corresponding error images.The supplemental figures include noise-free and noisy Middlebury samples, Laserscan, and ToFMark examples.
- Quantitative results: Additional quantitative results report mean absolute error on noisy Middlebury and ToFMark datasets.Table 4 uses MAE in pixel disparity, while Table 5 uses MAE in millimeters.
- Quantitative results: On noisy Middlebury and ToFMark data, ATGV-Net obtains reasonable MAE results despite comparisons against methods using an additional guidance image.The method was not trained on the MAE metric described for these comparisons.
- Qualitative results: The qualitative figures compare bicubic or bilinear upsampling and state-of-the-art approaches with the network-only and ATGV-Net outputs.For ToFMark, the figures show Books, Devil, and Shark; for Laserscan, they show Scan42.