Source-linked AI summary
Discrete Cosine Transform Network for Guided Depth Map Super-Resolution
Zixiang Zhao, Jiangshe Zhang, Shuang Xu, Zudi Lin, Hanspeter Pfister
TL;DR
GDSR seeks to reconstruct high-resolution depth from degraded low-resolution depth using same-scene RGB guidance, while addressing depth-specific structure and cross-modal transfer challenges. DCTNet combines DCT-based feature reconstruction, semi-coupled shared/private feature extraction, and adaptive edge attention, and achieves state-of-the-art performance across four datasets with relatively few parameters. The authors note that its components make the formulation more complex than end-to-end regression networks.
Problem
GDSR must reconstruct HR depth from noisy or low-resolution depth using RGB guidance, while handling depth-specific structures, cross-modal feature extraction, texture over-transfer, and unclear network mechanisms.
Method
DCTNet combines semi-coupled shared/private feature extraction, RGB edge attention, and a DCT module that solves a channel-wise optimization problem for HR depth features.
Results
DCTNet achieves state-of-the-art GDSR performance across four RGBD datasets with relatively few parameters.
Takeaways & Limitations
DCTNet integrates interpretable optimization-inspired processing with multimodal feature extraction and adaptive edge guidance for GDSR.
Takeaways & Limitations
Its component-based formulation is more complex than end-to-end deep networks, and future work targets simplification plus low illumination and blurry RGB boundaries.
Abstract
from arXiv · showhide
Guided depth super-resolution (GDSR) is an essential topic in multi-modal image processing, which reconstructs high-resolution (HR) depth maps from low-resolution ones collected with suboptimal conditions with the help of HR RGB images of the same scene. To solve the challenges in interpreting the working mechanism, extracting cross-modal features and RGB texture over-transferred, we propose a novel Discrete Cosine Transform Network (DCTNet) to alleviate the problems from three aspects. First, the Discrete Cosine Transform (DCT) module reconstructs the multi-channel HR depth features by using DCT to solve the channel-wise optimization problem derived from the image domain. Second, we introduce a semi-coupled feature extraction module that uses shared convolutional kernels to extract common information and private kernels to extract modality-specific information. Third, we employ an edge attention mechanism to highlight the contours informative for guided upsampling. Extensive quantitative and qualitative evaluations demonstrate the effectiveness of our DCTNet, which outperforms previous state-of-the-art methods with a relatively small number of parameters. The code is available at \url{https://github.com/Zhaozixiang1228/GDSR-DCTNet}.
1. Introduction
GDSR reconstructs high-resolution depth from low-resolution depth using same-scene RGB guidance, but must handle depth-specific structures, cross-modal mismatch, texture over-transfer, and limited interpretability. DCTNet addresses these challenges with semi-coupled feature extraction, guided edge attention, and a DCT-based module, achieving strong performance with relatively few parameters.
- Motivation: GDSR uses HR RGB images to reconstruct HR depth from noisy or low-resolution depth acquired under suboptimal imaging conditions.The approach relies on statistical co-occurrences between RGB texture edges and depth discontinuities.
- Motivation: Depth super-resolution differs from image super-resolution because it must recover textureless piecewise-affine regions, sharp discontinuities, and noise-tolerant outputs.These characteristics make direct adoption of image-SR methods inappropriate without evaluating depth-specific behavior.
- Method overview: DCTNet combines semi-coupled feature extraction, guided edge spatial attention, a discrete cosine transform module, and depth reconstruction.The workflow extracts shared and private modality features, highlights useful RGB edges, computes HR depth features, and reconstructs the SR depth map.
- Method overview: The DCT module uses DCT to solve an optimization model in the multi-channel feature domain and acquire HR depth features guided by RGB features.This design is intended to improve interpretability beyond a purely empirically designed deep-learning architecture.
- Method overview: GESA highlights RGB edges associated with depth discontinuities to adaptively reduce transfer of irrelevant texture details.The mechanism uses enhanced spatial attention to activate intensity edges useful for guided depth upsampling.
- Contributions and results: DCTNet achieves state-of-the-art GDSR performance across four RGBD datasets with relatively few parameters.Evaluations cover NYU v2, Middlebury, Lu, and RGBDD.
2. Related Work
Prior GDSR research includes filter-based, optimization-based, learning-based, and deep-learning or hybrid approaches. DCTNet connects optimization and learning by solving HR feature reconstruction with DCT, learning edge weights adaptively, and using shared/private convolutional features without explicit dictionary learning.
- Existing approaches: Existing GDSR methods span filter-based, optimization-based, and learning-based categories, with later work combining deep learning with filtering or optimization.These approaches address edge preservation, data priors, cross-modal dependencies, and LR-to-HR mapping in different ways.
- Comparison with existing approaches: DCTNet uses DCT inside a deep-learning framework to solve an optimization problem for multi-channel HR depth features, with learnable parameters improving flexibility.The paper identifies this as the first use of DCT for degraded depth restoration to its knowledge.
- Comparison with existing approaches: Unlike explicit coupled dictionary learning, DCTNet extracts shared and private features by controlling parameter sharing across convolutional kernels.Its ESA module also learns RGB edge-attention weights adaptively rather than relying on fixed local or global transfer rules.
3. Method
DCTNet combines semi-coupled feature extraction, guided edge attention, DCT-based guided upsampling, and depth reconstruction for GDSR. Its DCT module provides learning-free feature acquisition with learnable channel-wise parameters, while the architecture targets cross-modal feature extraction and texture transfer.
- Overall architecture: DCTNet processes depth and RGB features through SCFE, GESA, DCT, and DR modules to produce an SR depth map.SCFE extracts features, GESA obtains useful RGB edge weights, DCT acquires HR depth features, and DR predicts the HR depth map.
- 3.2.1 Semi-coupled feature extraction: Shared kernels extract cross-modal information while private kernels preserve modality-specific depth and RGB features.The semi-coupled design applies shared convolutional kernels to both modalities and separate private kernels to their unique information.
- 3.2.2 Guided edge spatial attention: GESA uses lightweight spatial attention to highlight RGB edges useful for upsampling and reduce transfer of irrelevant textures.The module replaces manually specified edge-weight extraction with data-driven attention weights.
- 3.2.3 Discrete cosine transform: DCT reconstructs multi-channel HR depth features by solving a guided upsampling optimization problem in the feature domain.The module uses RGB features and guided edge attention weights, and its channel-wise parameters are learnable during training.
- 3.2.3 Discrete cosine transform: DCT feature acquisition is learning-free apart from learnable channel parameters, reducing learnable weights and improving interpretability.The channel-wise parameters are updated during training, allowing the optimization-based component to remain flexible.
- 3.2.4 Depth reconstruction: The DCT feature map is passed to a depth reconstruction module that predicts the HR depth map using an ℓ2 training objective.The reconstruction function is defined as Ĥ = R(ΦH), while the loss compares predictions with ground-truth HR depth maps.
4. Experiment
DCTNet is evaluated on four GDSR benchmarks using quantitative, qualitative, parameter-efficiency, and ablation studies. Across these evaluations, it achieves strong multi-scale performance with relatively few parameters, while each major architectural component contributes to performance or flexibility.
- Validation experiments: Highlighted GESA weights emphasize object contours while smoothing internal texture, helping alleviate RGB texture over-transfer.The attention visualization is presented in Figure 3.
- Validation experiments: Learnable channel-wise DCT parameters adapt the balance between fidelity and regularization during training, improving flexibility over manually fixed parameters.The changing parameter curves are shown in Figure 3, while the adaptive role is described in the validation analysis.
- Comparison with the state-of-the-arts: Qualitative error maps show DCTNet predictions with lower errors and closer visual agreement to ground truth on NYU v2 and Middlebury examples.The examples cover 8× super-resolution on NYU v2 and 16× super-resolution on Middlebury.
- Comparison with the state-of-the-arts: DCTNet achieves the best or second-best performance across multiple datasets and ×4, ×8, and ×16 super-resolution scales.The comparison uses RMSE on four test sets, and the quantitative results are reported in Table 2.
- Ablation Studies: Ablations show that semi-coupled filters, the DCT module, learnable parameters, and residual connections each support the model’s performance or flexibility.Independent or fully shared filters perform worse than semi-coupled filters; removing DCT increases parameters and reduces quality, fixing λ restricts flexibility, and removing residual connections causes significant degradation.
5. Conclusion
DCTNet combines discrete cosine transform, semi-coupled convolutional feature extraction, and adaptive edge attention to address key GDSR challenges. The authors suggest its components may also benefit other multi-modal image-processing tasks.
- DCTNet integrates discrete cosine transform, semi-coupled convolutional feature extraction, and adaptive edge attention for guided depth super-resolution.
- The design targets RGB texture over-transfer, ineffective cross-modal feature extraction, and unclear working mechanisms in existing methods.
- The authors propose that all or some DCTNet components could benefit future multi-modal image-processing tasks.