PaperlayerView exact revision on arXiv

Source-linked AI summary

WAVE: Reversing the Guidance Hierarchy for Coarse-to-Fine Guided Depth Super-Resolution

Tayyab Nasir, Daochang Liu, Ajmal Mian

arXiv:2608.25302v1cs.CVcs.LG

TL;DR

GDSR guidance pipelines inherit a fine-to-coarse bias that can introduce misleading RGB cues and blur depth boundaries. WAVE reverses this hierarchy with explicit wavelet and semantic control, and it performs especially strongly at high upsampling factors. The paper also identifies extending coarse-to-fine reconstruction to blind or real-world degradations as a future direction.

  • Problem

    Fine-to-coarse CNN and semantic-token hierarchies introduce low-level RGB cues before global structure, risking artifacts and blurred depth boundaries.

  • Method

    WAVE applies ML-DWT to RGB guidance, consumes wavelet sub-bands and DINO tokens in reverse order, and separates structure, detail, semantic gating, and invertible fusion.

  • Results

    WAVE matches or outperforms existing methods across benchmarks, with its largest gains at high upsampling factors; at 32×, RMSE is 3.77 versus SPFNet’s 3.97 on RGB-D-D.

  • Takeaways & Limitations

    Explicit coarse-to-fine control and source-level filtering are associated with stronger performance when low-resolution depth retains the least structure.

  • Takeaways & Limitations

    Extending the coarse-to-fine schedule to blind or real-world degradations remains a future direction.

Abstract

from arXiv · show

Guided depth super-resolution (GDSR) typically extracts RGB guidance features through convolutional hierarchies, inheriting their fine-to-coarse bias. Thus, low-level spatial cues surface in early layers, leaving the deeper layers to suppress those that do not correspond to true depth boundaries, which risks artifacts and blurred edges. The same fine-to-coarse bias persists in semantics-based methods that consume low-level tokens early and global tokens late. We present WAVE, which introduces a multi-level discrete wavelet transform (ML-DWT) as an explicit and interpretable feature-control mechanism, enabling a coarse-to-fine reconstruction by consuming sub-bands and semantic tokens in reverse of their generation order. WAVE further exploits these sub-bands to treat high- and low-frequency content separately, filtering at its source the misleading RGB color and texture cues that often lead to blurred boundaries and artifacts, offering an intuitive alternative to the suppression learned implicitly by an opaque network. WAVE separates structure and detail reconstruction into dedicated modules that: i) model interactions within and across wavelet sub-bands, depth features, and semantic priors, ii) apply semantic gating to the high-frequency bands, and iii) fuse modalities through an invertible coupling mechanism that prevents collapse onto a single modality. Extensive experiments across multiple benchmarks demonstrate that WAVE matches or outperforms existing methods, with the largest gains at high upsampling factors, where low-resolution depth contains the least structure.

Introduction

WAVE addresses fine-to-coarse guidance in GDSR by reversing the order of wavelet and semantic features, filtering misleading RGB cues explicitly, and reconstructing depth from global structure to fine detail. Experiments show its strongest gains at extreme upsampling factors.

  • Motivation: GDSR combines low-resolution depth with higher-resolution RGB, whose misleading cues can produce artifacts, false depth discontinuities, and blurred boundaries.The RGB and depth modalities provide complementary information, but captured depth has lower spatial resolution than RGB.
  • Motivation: CNN-based GDSR processes guidance fine-to-coarse, introducing high-frequency color and texture cues early and suppressing them only after global structure emerges.Semantic-token methods retain a similar order by introducing deeper global tokens later.
  • WAVE: WAVE consumes ML-DWT pyramid features and layer-wise DINO tokens in reverse generation order, reconstructing global structure first and detail last.This provides an explicit coarse-to-fine control mechanism rather than relying on implicit CNN feature hierarchies.
  • WAVE: WAVE processes structural, edge, and texture wavelet sub-bands independently and uses semantic gating to regulate high-frequency RGB content.Its interaction modules connect wavelet sub-bands, semantic priors, and depth features while targeting guide-induced noise at its source.
  • WAVE: WAVE uses invertible coupling for cross-modal fusion, preserving recoverability of both modalities and discouraging collapse onto a single modality.The semantic branch is adapted through a low-rank residual on frozen DINO output tokens rather than backbone fine-tuning.
  • Results: At 32×, WAVE reduces RMSE versus SPFNet from 3.97 to 3.77 on RGB-D-D and from 8.06 to 7.90 on NYU_v2.Across benchmarks, the largest gains occur at extreme upsampling, where low-resolution depth retains the least structure.

Methodology

WAVE imposes coarse-to-fine GDSR by reversing the consumption order of multilevel wavelet sub-bands and semantic tokens. Its HUMMA blocks separately reconstruct structure and detail, then combine depth, RGB, and semantic information through controlled attention, gating, and invertible fusion.

  • Wavelet Guidance: A four-level Haar ML-DWT separates RGB guidance into LL structural bands and LH, HL, HH edge and texture bands at multiple abstraction levels.Recursive decomposition halves spatial resolution at each level, providing explicit control over which RGB features enter reconstruction and at what scale.
  • Semantic Guidance: DINOv3 tokens remain frozen and are adapted by SToRA, a zero-initialized low-rank residual module that uses 2Cr parameters per adapter.The adapter preserves pretrained representations while gradually introducing task-specific token adjustments.
  • Coarse-to-Fine Reconstruction: WAVE reverses the generation order of ML-DWT features and DINO tokens to reconstruct depth from coarse structure toward fine detail.Four HUMMA blocks progressively process matching pyramid levels while upsampling depth features by 2× at each block.
  • Detail Reconstruction: The detail branch processes horizontal and vertical bands through an edge path and all three high-frequency bands through a multiscale texture path.These paths learn band-specific representations before merging them into a high-frequency detail feature map.
  • Detail Reconstruction: Projected SToRA tokens query global summaries of edge and texture features, while zero-initialized gates introduce semantic reweighting gradually.This semantic gating aligns high-frequency responses with true geometric boundaries and supports detail sharpening.
  • Structure Reconstruction: The structure branch combines low-frequency depth features, LL bands, and semantic tokens, using depth-query attention to retrieve structure-relevant semantics.An IRN-style invertible coupling then fuses semantically refined depth with LL information while discouraging either modality from being discarded.

Experiments

WAVE is evaluated under two benchmarking protocols across multiple datasets and upsampling factors. It remains competitive broadly, with its strongest advantages at higher scales and on out-of-domain data.

  • Protocols: WAVE is evaluated under HYPERSIM and NYU_v2 training protocols using established prior-work settings.The protocols differ in training data and evaluation sets for fair comparison.
  • Results: WAVE shows competitive performance across all reported experimental settings.
  • Results: WAVE’s margins are largest at higher upsampling factors, where the low-resolution input retains the least structure.
  • Results: On in-domain NYU_v2 at lower scales, leading methods fall within roughly 0.1-0.2 RMSE, leaving WAVE competitive rather than dominant.
  • Qualitative comparison: Qualitatively, WAVE produces sharper object boundaries, fewer texture-copying artifacts, and less boundary bleeding than competing methods.

Ablation

The ablation tests whether WAVE’s coarse-to-fine ordering itself contributes to performance, using fixed settings and an out-of-distribution evaluation.

  • Ablation: Ablations reorder ML-DWT subbands and layer-wise DINOv3 tokens into a fine-to-coarse schedule for comparison with WAVE.All variants use NYU_v2 training at 16× and are evaluated on the unseen TOFDSR dataset.
  • Ablation: The reordered variant shows a significant RMSE increase, supporting the importance of coarse-to-fine reconstruction beyond merely adding wavelet and semantic hierarchies.

Conclusion

WAVE reconstructs depth from global structure to fine detail by reversing the ordering of wavelet subbands and hierarchical semantic tokens. It performs best at high upsampling factors, while remaining less dominant in saturated low-scale settings and relying on a frozen foundation model.

  • Conclusion: WAVE counters conventional fine-to-coarse pipelines with a coarse-to-fine architecture.
  • Conclusion: WAVE decomposes RGB with ML-DWT, processes structural, edge, and texture components through dedicated branches, and uses semantics to gate high-frequency content.
  • Conclusion: WAVE uses invertible coupling for multimodal fusion to avoid discarding either modality.
  • Conclusion: WAVE’s largest gains occur at high upsampling factors, where low-resolution depth retains the least structure.
  • Conclusion: WAVE’s gains are smaller in saturated, in-domain low-scale settings and depend on a frozen foundation model for semantic priors.
  • Conclusion: At 16× on TOFDSR and RGBD-D, WAVE recovers sharper boundaries and cleaner surfaces with fewer texture-copying artifacts, closest to ground truth.

Literature Review

GDSR research has increasingly addressed misleading RGB guidance through selective fusion, semantic information, and other auxiliary cues. WAVE instead decomposes the guide into explicit frequency bands and reverses the hierarchy’s processing order.

  • Prior work: GDSR methods increasingly use transformers, diffusion models, state-space models, knowledge distillation, semantics, gradients, Fourier features, and multi-task supervision.
  • Prior work: Selective feature integration mitigates guidance-induced noise through attention-based fusion, constrained fusion, and auxiliary or multi-task supervision.
  • WAVE: WAVE decomposes RGB into explicit frequency subbands before fusion and consumes them with semantic tokens in reverse coarse-to-fine order.

RGB Shift Robustness Tests

WAVE remains robust when the RGB guide is horizontally or vertically misaligned with the depth input, maintaining the best or tied-best RMSE across four test sets at 16×.

  • WAVE remains the lowest-RMSE or tied-best method across all four test sets under RGB-guide shifts of 1–8 pixels at 16×.The depth input, ground truth, and evaluation mask remain fixed, isolating reliance on pixel-accurate RGB-depth registration.

Replacing DINOv3 with SAM-2 and ResNet

WAVE’s gains persist when semantic refinement or gating is removed, and the coarse-to-fine schedule is retained across alternative frozen semantic encoders.

  • Removing semantic refinement or gating degrades WAVE’s performance.
  • Removing semantics entirely shows that WAVE’s gains do not rely solely on its semantic encoder.
  • DINOv3 is replaced with SAM-2 and ResNet-50 while extracting four increasingly abstract layers in the same coarse-to-fine order.Table 6 evaluates these substitutions under the NYU_v2 training protocol at 16× across five benchmarks using RMSE in centimeters.

Complexity Analysis

The complexity analysis reports trainable and frozen parameters, FLOPs, and peak GPU memory for single-batch inference at benchmark input resolutions.

  • Table 7 separates trainable and non-trainable parameters for each model.
  • Complexity is measured during a single-batch inference pass on one NVIDIA RTX 4090 GPU.
  • The comparison reports FLOPs and peak GPU memory allocation at the corresponding benchmark input resolution.
  • Table 7 compares complexity at 8× and 16× using 448 × 448 input resolution, with parameters in M, FLOPs in G, and memory in GB.

Qualitative Comparisons

Across qualitative comparisons at 8× and 32×, WAVE produces sharper boundaries, preserves thin structures and object openings, and remains closest to ground truth.

  • At the extreme 32× factor, WAVE produces straighter object boundaries and cleaner surfaces when low-resolution input retains the least structure.
  • On TOFDSR, WAVE recovers the stool’s thin legs and crossbars and preserves the gap beneath the seat.
  • At 32×, WAVE preserves object openings, creates straighter boundaries, and avoids false discontinuities and background artifacts compared with C2PD.The comparison covers NYU_v2, RGBD-D, TOFDSR, and DIML; C2PD is the only competing method with publicly available 32× weights.
  • WAVE yields more prominent depth boundaries than the blurred boundaries visible in competing methods.
  • At 8× on TOFDSR and RGB-D-D, WAVE recovers thin structures and object openings with sharper, non-bleeding boundaries.
Loading 2608.25302v1…