Source-linked AI summary
Towards Fast and Accurate Real-World Depth Super-Resolution: Benchmark Dataset and Baseline
Lingzhi He, Hongguang Zhu, Feng Li, Huihui Bai, Runmin Cong, Chunjie Zhang, Chunyu Lin, Meiqin Liu, Yao Zhao
TL;DR
Low-resolution depth sensors and the lack of real-world paired LR–HR data limit depth-map super-resolution research. The paper builds RGB-D-D and introduces FDSR, an RGB high-frequency-guided baseline. It reports strong speed and accuracy on public and RGB-D-D datasets, with clearer boundaries and partly corrected depth errors on real-world maps.
Problem
Depth sensors provide low-resolution maps, while downsampling-based training pairs do not comprehensively reproduce complex real-world LR–HR correspondences.
Source: §1 · p. 1Method
The paper constructs RGB-D-D from paired mobile-phone LR and Lucid Helios HR depth maps and proposes FDSR, which uses RGB high-frequency guidance in a multi-scale structure.
Source: §1 · p. 2Results
FDSR achieves superior speed and accuracy on public datasets and RGB-D-D, with more accurate real-world outputs, clearer boundaries, and some depth-value error correction.
Source: §1 · p. 2Takeaways & Limitations
RGB-D-D provides real-scene, real-correspondence data intended to support depth SR research and other depth-related tasks.
Source: §1 · p. 2Inevitable errors in filled depth maps can cause their quality to degrade.
Source: §3.2 · p. 4
Abstract
from arXiv · showhide
Depth maps obtained by commercial depth sensors are always in low-resolution, making it difficult to be used in various computer vision tasks. Thus, depth map super-resolution (SR) is a practical and valuable task, which upscales the depth map into high-resolution (HR) space. However, limited by the lack of real-world paired low-resolution (LR) and HR depth maps, most existing methods use downsampling to obtain paired training samples. To this end, we first construct a large-scale dataset named "RGB-D-D", which can greatly promote the study of depth map SR and even more depth-related real-world tasks. The "D-D" in our dataset represents the paired LR and HR depth maps captured from mobile phone and Lucid Helios respectively ranging from indoor scenes to challenging outdoor scenes. Besides, we provide a fast depth map super-resolution (FDSR) baseline, in which the high-frequency component adaptively decomposed from RGB image to guide the depth map SR. Extensive experiments on existing public datasets demonstrate the effectiveness and efficiency of our network compared with the state-of-the-art methods. Moreover, for the real-world LR depth maps, our algorithm can produce more accurate HR depth maps with clearer boundaries and to some extent correct the depth value errors.
1. Introduction
The paper addresses the mismatch between low-resolution mobile depth maps and the high-resolution maps needed for practical applications, and the limits of training from downsampled data. It introduces RGB-D-D, a real-world paired dataset, and FDSR, a fast baseline guided by RGB high-frequency features.
1. Introduction
- Existing SR training commonly pairs LR and HR maps by downsampling, which does not comprehensively simulate complex real-world correspondences.Real sensor maps can contain noise and depth holes.Source: §1 · p. 1
- RGB-D-D contains 4,811 paired LR and HR depth samples captured by mobile-phone and Lucid Helios sensors across indoor and challenging outdoor scenes.It offers both downsampled HR ground truth and raw LR sensor data as SR inputs.Source: §1 · p. 2
- FDSR uses a high-frequency-guided multi-scale structure to provide RGB frequency guidance and contextual information while improving efficiency and retaining reconstruction performance.The RGB high-frequency features guide early stages of depth reconstruction, while low-frequency components are suppressed to reduce parameters.Source: §1 · p. 2
- FDSR achieves superior speed and accuracy on public datasets and RGB-D-D, and produces real-world results with clearer boundaries and, to some extent, corrected depth-value errors.The authors describe its real-world reconstructions as more accurate.Source: §1 · p. 2
2. Related Work
Prior depth SR datasets and methods offer useful data and approaches, but the paper identifies gaps in real LR–HR correspondence and computational efficiency. It positions RGB-D-D and FDSR as responses to those gaps.
Benchmark Datasets
- Existing real-scene datasets provide few high-quality noisefree maps or focus on indoor scenes, but do not establish real LR–HR correspondences for depth SR.RGB-D-D is presented as the first dataset to combine real scenes with real correspondences.Source: §2 · p. 2
Algorithm Models
- Depth SR methods are categorized as non-RGB-guided or RGB-guided; RGB-guided methods have become mainstream, while non-RGB-guided methods may underuse color information.Unsupervised approaches generally model SR as an optimization problem.Source: §2 · p. 2
- Prior learning-based RGB-guided methods extract or derive features from RGB images, while efforts focused on accuracy may incur high computational complexity.The authors contrast their efficiency–accuracy goal with methods that use real-scene LR maps only for evaluation rather than training on real paired correspondences.Source: §2 · p. 3
3. RGB-D-D Dataset
RGB-D-D pairs mobile-phone color and LR depth data with industrial-ToF HR depth maps across varied scenes. Its collection, alignment, and ground-truth processing aim to provide real correspondences and dense, high-quality depth maps, with some residual errors in filled maps.
3. RGB-D-D Dataset
- RGB-D-D comprises 4,811 pairs of mobile-phone color images and real-world LR depth maps with HR depth maps captured by an industrial ToF camera.The collection is described as the first real-world depth SR dataset.Source: §3 · p. 3
3.2. Dataset Statistic
- The four scene categories—portraits, models, plants, and lights—cover different depth SR settings, including portrait blur, object modeling, dense foliage, and complex illumination.The lights category includes strong indoor sources and outdoor light, which affect low-power sensor depth quality.Source: §3.2 · p. 4
3.1. Dataset Collection
- The dataset aligns phone LR depth and color at 192 × 144 with projected Helios depth at 512 × 384 to preserve real LR–HR correspondences.Collection processing calibrates the phone and Helios cameras, projects Helios point clouds, crops HR depth maps, and fills holes using color boundaries; user evaluation and manual scribbles also help select or repair ground truth.Source: §3.1 · pp. 3–4, §3.2 · p. 4
3.2. Dataset Statistic
- Most RGB-D-D raw HR depth maps have return density above 90%, and the authors report fewer depth-value errors and better boundaries than NYU v2.They also note that inevitable errors in filled depth maps can degrade preprocessing quality.Source: §3.2 · p. 4
4. Proposed Framework
FDSR reconstructs a high-resolution depth map from an upscaled low-resolution depth map and RGB-derived high-frequency guidance. Its multi-scale reconstruction branch combines that guidance with contextual features, and training minimizes pixelwise L1 error.
4.1. Problem Formulation
- FDSR learns a residual mapping from the upscaled depth map and RGB guidance to the high-resolution depth map.The RGB image is embedded in a high-frequency extractor, and the network learns the residual between the upscaled input and HR output.Source: §4.1 · p. 4
4.2. Network Architecture
- The high-frequency guidance branch adaptively highlights RGB high-frequency features, while the multi-scale reconstruction branch progressively combines them with contextual features to recover depth details.The reconstruction branch uses multi-scale dilated blocks with different receptive fields; only extracted high-frequency features are fused as guidance.Source: §4.2 · pp. 4–5
- FDSR reduces computation by using adaptive high-frequency extraction and by converting inputs to grayscale, splitting them into blocks, and applying pixel shuffle to restore the output size.The paper says these operations are intended to make the network faster for mobile devices and embedded systems.Source: §4.2 · p. 5
4.3. Loss Function
- Training minimizes the L1 norm between the super-resolved depth map and ground truth over all pixels.The loss is defined over the set of image pixels.Source: §4.3 · pp. 5–6
5. Experiments
On RGB-D-D, real-world-trained FDSR improves visual results and depth-value accuracy where methods trained with conventional downsampling perform poorly. Experiments also report strong NYU v2 performance, gains from RGB-D-D training, and benefits from both ablated components.
5.1. Datasets and Implementation Details
- NYU v2 experiments use 1,000 training and 449 test pairs; RGB-D-D splits portraits, plants, and models into training and test sets.FDSR is implemented in PyTorch, with a MindSpore implementation also provided.Source: §5.1 · p. 6
5.2. Experiments on NYU v2 Dataset
- FDSR achieves the best reported NYU v2 performance at ×4, ×8, and ×16 scaling, with fewer value and edge errors.The reported errors assess global depth accuracy and local edge accuracy.Source: §5.2 · p. 6, Table 2 · p. 6
- On a 640 × 480 input, FDSR has efficiency comparable to DJFR and FDKN while outperforming both in performance.NYU v2 visual results are described as having finer boundaries and details without texture-copy artifacts or extra noise.Source: §5.2 · pp. 6–7
5.3. Experiments on RGB-D-D Dataset
- RGB-D-D training improves the reported results, particularly at object boundaries and in depth-value accuracy.The downsampling-trained RGB-D-D model is FDSR+; the associated quantitative results are identified in Tables 3 and 5.Source: §5.3 · pp. 7–8, Table 3 · p. 7, Table 5 · p. 8
- Real-world-trained FDSR++ substantially improves visual quality and value accuracy, while evaluated methods generally perform poorly on real-world depth SR.The authors attribute the gap to conventional downsampling failing to model real LR–HR correspondences.Source: Figure 5 · p. 7, §5.3 · p. 8, Table 4 · p. 8
5.4. Ablation Study
- Ablations report that both HFL and HFGB improve FDSR performance.The reported RMSE evaluation is identified in Table 6.Source: §5.4 · p. 8, Table 6 · p. 8
6. Conclusion
The paper introduces RGB-D-D as a real-world depth map SR benchmark and presents a fast baseline that performs competitively on public and proposed datasets.
6. Conclusion
- RGB-D-D provides paired LR and HR depth maps across multiple scenarios for real-world depth map SR research.The paper describes it as the first benchmark dataset satisfying both real-scene and real-correspondence requirements.Source: §6 · p. 8
- The baseline adaptively focuses on high-frequency guidance components while suppressing low-frequency components.Source: §6 · p. 8
- The baseline achieves competitive performance on public and proposed datasets and can cope with real-world depth map SR.Source: §6 · p. 8