PaperlayerView exact revision on arXiv

Source-linked AI summary

PatchFusion: An End-to-End Tile-Based Framework for High-Resolution Monocular Metric Depth Estimation

Zhenyu Li, Shariq Farooq Bhat, Peter Wonka

arXiv:2312.02284v1cs.CV

TL;DR

High-resolution monocular metric depth estimation is constrained by low-resolution models and limitations in existing refinement strategies. PatchFusion uses guided tile fusion, global-to-local context, and consistency-aware training and inference to produce high-resolution depth maps. It improves RMSE by 17.3% on UnrealStereo4K and 29.4% on MVS-Synth, while supporting zero-shot transfer.

  • Problem

    High-resolution monocular metric depth estimation is difficult because prevailing models are bottlenecked by low-resolution backbones and existing strategies can propagate errors or lose high-frequency details.

  • Method

    PatchFusion is an end-to-end tile-based framework combining high-level feature-guided fusion, a Global-to-Local module, and consistency-aware training and inference.

  • Results

    17.3% and 29.4% RMSE improvements over the baseline are reported on UnrealStereo4K and MVS-Synth, respectively, with satisfactory zero-shot transfer.

  • Takeaways & Limitations

    PatchFusion generates detailed high-resolution metric depth maps using forward passes without additional pre-processing or post-processing.

  • Takeaways & Limitations

    Increasing the number of randomly selected patches improves prediction but proportionally increases processing time, and high-resolution real-world training data remains limited.

Abstract

from arXiv · show

Single image depth estimation is a foundational task in computer vision and generative modeling. However, prevailing depth estimation models grapple with accommodating the increasing resolutions commonplace in today's consumer cameras and devices. Existing high-resolution strategies show promise, but they often face limitations, ranging from error propagation to the loss of high-frequency details. We present PatchFusion, a novel tile-based framework with three key components to improve the current state of the art: (1) A patch-wise fusion network that fuses a globally-consistent coarse prediction with finer, inconsistent tiled predictions via high-level feature guidance, (2) A Global-to-Local (G2L) module that adds vital context to the fusion network, discarding the need for patch selection heuristics, and (3) A Consistency-Aware Training (CAT) and Inference (CAI) approach, emphasizing patch overlap consistency and thereby eradicating the necessity for post-processing. Experiments on UnrealStereo4K, MVS-Synth, and Middleburry 2014 demonstrate that our framework can generate high-resolution depth maps with intricate details. PatchFusion is independent of the base model for depth estimation. Notably, our framework built on top of SOTA ZoeDepth brings improvements for a total of 17.3% and 29.4% in terms of the root mean squared error (RMSE) on UnrealStereo4K and MVS-Synth, respectively.

1. Introduction

PatchFusion addresses the mismatch between low-resolution monocular depth models and modern high-resolution imagery by refining tile-based estimation with guided fusion, global context, and consistency-aware processing.

  • 1. Introduction: Existing high-resolution strategies remain limited because guided depth super-resolution can propagate errors, while low-resolution processing can lose fine details.The resolution gap is especially pronounced relative to modern consumer-camera and display resolutions.
  • 1. Introduction: BoostingDepth exhibits scale inconsistencies and requires post-processing, while its fusion network lacks guidance and holistic image context.These shortcomings motivate PatchFusion’s end-to-end design and removal of heuristic patch selection.
  • 1. Introduction: PatchFusion combines high-level feature guidance, the Global-to-Local module, and consistency-aware training and inference to improve tile-based depth estimation.These components target fusion training, contextual awareness, and patch-overlap consistency.
  • 1. Introduction: PatchFusion is designed as an end-to-end tile-based metric depth framework without pre-processing, patch-selection heuristics, or post-processing.The approach is presented as a refinement of prior tile-based depth estimation.
  • 1. Introduction: 17.3% and 29.4% improvements over current SOTA are reported on UnrealStereo4K and MVS-Synth, respectively.The framework is evaluated on UnrealStereo4K, MVS-Synth, and Middleburry 2014.

2. Related Work

Prior high-resolution depth methods use transformer backbones, super-resolution, implicit functions, or tile-based blending, but tile-based outputs still require careful fusion for global scale and local detail.

  • 2. Related Work: Current state-of-the-art depth frameworks achieve strong performance but generally process low-resolution inputs, limiting direct use on modern high-resolution imagery.ZoeDepth, for example, is described as inferring 384×512 inputs.
  • 2. Related Work: Guided depth super-resolution reconstructs high-resolution depth from low-resolution observations and color references, but its training setup and error propagation limit monocular deployment.The supplied discussion identifies direct downsampling-based training samples as a notable limitation.
  • 2. Related Work: Implicit-function approaches estimate disparity continuously across image locations, yet input downsampling can still remove intricate high-frequency details.SMD-Net is cited as an example of this strategy [37].
  • 2. Related Work: Tile-based methods blend coarse whole-image maps with fine patch-wise maps to preserve global scale while retaining local detail.PatchFusion reports seamless stitching without post-optimization.
  • 2. Related Work: Earlier tile-based blending uses linear polynomial or deformable-depth-field corrections, whereas PatchFusion aims for end-to-end stitching without post-optimization.The comparison concerns how coarse and fine depth predictions are fused.

3. Method

PatchFusion estimates high-resolution metric depth by combining globally consistent coarse predictions with detailed patch-wise predictions, then fusing them using global context and overlap consistency. Its end-to-end pipeline avoids heuristic patch selection and post-processing during inference.

  • Overall Framework: PatchFusion divides high-resolution estimation into global coarse prediction, local fine-depth prediction, and fusion to balance global consistency with high-frequency detail.Downsampling enables efficient coarse estimation but loses fine details, while patches recover boundary and intricate-structure detail but lack global context.
  • Overall Framework: Patch-wise predictions can suffer scale shifts and fluctuations across patches because each patch lacks global context, producing visible boundary artifacts.The framework addresses this inconsistency by transferring global information from coarse depth into patch-wise predictions.
  • Global-to-Local Module: The Global-to-Local module preserves global context for cropped regions by applying windowed and shifted-window self-attention to multi-layer coarse features.Directly cropping global guidance features can lose scale-inference information; G2L is designed to retain it while limiting memory demands.
  • Guided Fusion Network: The Guided Fusion Network combines cropped images, coarse depth, and fine depth, injecting scale-aware guidance through fusion blocks and decoder features before final depth prediction.Its U-Net-like design uses image and depth values in the encoder while incorporating network guidance through skip connections and decoder layers.
  • Consistency-Aware Training and Inference: Consistency-Aware Training enforces agreement between intermediate features and depth predictions in overlapping regions from independently processed patches.Two shifted crops are processed separately, and an L2 loss penalizes discrepancies over their overlap; the method emphasizes feature consistency because it yields better results.
  • Consistency-Aware Training and Inference: Inference processes the image with non-overlapping or additional shifted and random patches, then stitches the resulting depth maps without heuristic patch selection or post-processing.The standard pipeline uses P = 16 non-overlapping patches, while expanded variants add 33 shifted patches or R = 128 random patches.

4. Experimental Results

PatchFusion is evaluated on synthetic high-resolution datasets, zero-shot transfer to Middlebury 2014, qualitative depth quality, text-to-image generation, and targeted ablations. It improves benchmark accuracy and boundary detail while exposing computational and real-world-data limitations.

  • Main Results: 17.3% lower RMSE and 14.7% lower REL than ZoeDepth on UnrealStereo4K, with the lowest Soft Edge Error and sharper boundary details.The comparison uses the UnrealStereo4K test set, and qualitative results emphasize intricate boundary details.
  • Main Results: PatchFusion transfers zero-shot to Middlebury 2014 and produces higher-quality text-to-image outputs with more coherent structures, textures, and details than ControlNet’s baseline depth model.The comparison uses high-resolution real-world stereo images and aligns depth resolutions by rescaling before generation.
  • Ablation Studies and Discussion: Feature guidance improves REL by 6.0% and SEE by 3.8%, while G2L reduces consistency error by 26.0% and REL by 2.1%.The ablation adds guidance and G2L incrementally to a baseline encoder-decoder fusion network.
  • Ablation Studies and Discussion: Consistency-aware training improves REL and CE, while applying consistency to predictions and intermediate features balances standard accuracy, boundary quality, and patch coherence; CAI further boosts REL and CE.The study notes a slight trade-off in CE or SEE for some individual consistency configurations before the combined design and local ensemble improve the balance.
  • Limitations and Future Work: Increasing randomly selected patches improves depth prediction but proportionally increases processing time, while limited high-resolution real-world training data constrains synthetic-to-real transfer and scale quality.The authors propose efficient patch selection and larger real-world high-resolution datasets as future work.

5. Conclusions

PatchFusion is an end-to-end tile-based framework for high-resolution monocular metric depth estimation that uses consistency-aware training and inference. It improves RMSE over the baseline on UnrealStereo4K and MVS-Synth while showing satisfactory zero-shot transfer.

  • 17.3% and 29.4% RMSE improvements over the baseline are achieved on UnrealStereo4K and MVS-Synth, respectively.The framework also demonstrates satisfactory zero-shot transfer.
  • PatchFusion combines a tile-based network architecture with consistency-aware training and inference for high-resolution monocular metric depth estimation.Its outputs require only forward passes through networks, without additional pre-processing or post-processing.
  • On MVS-Synth, qualitative results compare PatchFusion against BoostingDepth and highlight improved estimation quality near boundaries.The comparison also attributes limitations to error propagation and loss of high-frequency details in competing approaches.

A. Evaluation Metrics

The evaluation follows a standard monocular metric depth estimation protocol using RMSE, SILog, and threshold accuracy. Depth predictions are evaluated at ground-truth resolution with a capped depth range for UnrealStereo4K.

  • RMSE, SILog, and accuracy under the threshold δ_i < 1.25^i are used to evaluate predicted depth against ground truth.The metrics are computed over M image pixels using ground-truth and predicted depths d_i and d̂_i.
  • UnrealStereo4K evaluation caps depth values to the range 1e−3m to 80m and evaluates final outputs at ground-truth resolution.

B. More Qualitative Results

Qualitative comparisons evaluate PatchFusion against guided super-resolution, implicit-function, and tile-based baselines. PatchFusion shows better boundary estimation and scale consistency than BoostingDepth.

  • The qualitative comparison spans three strategies: DGSR, ZoeDepth [3] with an implicit function, and BoostingDepth [27] fine-tuned on the target dataset.
  • Table 3 reports models trained on UnrealStereo4K and tested on Middleburry 2014 without fine-tuning, marking best and second-best results.
  • PatchFusion achieves better estimation quality at boundaries and better scale consistency than the SOTA tile-based framework BoostingDepth.The comparison includes DGSR and an implicit-function variant of ZoeDepth [3].

C. Zero-Shot Transfer on Middlebury 2014

PatchFusion demonstrates zero-shot transfer from UnrealStereo4K to Middlebury 2014 without fine-tuning. Its qualitative results show clearer object boundaries than BoostingDepth, despite BoostingDepth being trained on the target dataset.

  • Figure 8 reports better details and scale consistency than BoostingDepth [27] on UnrealStereo4K qualitative results.
  • Figure 9 compares PatchFusion with BoostingDepth [27], whose predictions are aligned to ground-truth scales using post-process optimization for visualization.
  • Adding an incremental number of randomly selected patches systematically refines depth estimations beyond the baseline model.These outcomes are reported in Table 3.
  • PatchFusion produces enhanced Middlebury 2014 results with clearer object boundaries despite not being trained on that dataset.BoostingDepth, used for comparison, includes training on Middlebury 2014.
  • PatchFusion uses 16 non-overlapping patches in PatchFusionP=16, adds 33 shifted patches in PatchFusionP=49, and randomly places 128 extra patches in PatchFusionR=128.

D. Inference Setup

PatchFusion supports multiple tile layouts for high-resolution depth inference, progressing from a full non-overlapping grid to shifted and randomly sampled patches.

  • D. Inference Setup: PatchFusionP=16 slices the image into 16 non-overlapping patches covering the entire image before stitching their depth maps.
  • D. Inference Setup: PatchFusionP=49 adds 33 shifted, tidily arranged patches to the standard 16-patch pipeline.
  • D. Inference Setup: PatchFusionR further augments inference with extra randomly sampled patches.

E. Qualitative Comparison of CAI

Consistency-Aware Inference addresses discontinuities between overlapping patch predictions by averaging their contributions as inference proceeds.

  • E. Qualitative Comparison of CAI: CAI computes each pixel from the running average of predictions contributed by overlapping patches.
  • E. Qualitative Comparison of CAI: The CAI running-average strategy improves continuity, producing more seamless and integrated depth estimations across patches.
  • E. Qualitative Comparison of CAI: Without CAI, newer patch estimations overwrite earlier ones and create artifacts at patch boundaries.
Loading 2312.02284v1…