PaperlayerView exact revision on arXiv

Source-linked AI summary

NeW CRFs: Neural Window Fully-connected CRFs for Monocular Depth Estimation

Weihao Yuan, Xiaodong Gu, Zuozhuo Dai, Siyu Zhu, Ping Tan

arXiv:2203.01502v2cs.CV

TL;DR

Single-image depth estimation is ambiguous and ill-posed, while existing neural and CRF approaches face complexity or limited connectivity. NeW CRFs performs fully-connected CRF optimization within windows, uses attention in a transformer encoder-decoder network, and reports improved results across KITTI, NYUv2, and MatterPort3D.

  • Problem

    Single-image depth estimation is inherently ambiguous, and existing CRF-based methods are limited to neighborhood connections while fully-connected CRFs are expensive.

  • Method

    The method performs fully-connected CRF optimization within input windows, uses multi-head attention for pairwise potentials, and embeds the module as a decoder with a vision transformer encoder.

  • Results

    The method significantly outperforms previous methods across KITTI, NYUv2, and MatterPort3D, including new state-of-the-art performance on all three datasets.

  • Takeaways & Limitations

    Windowed neural FC-CRFs provide a feasible way to incorporate fully-connected graph optimization into monocular depth estimation.

  • Takeaways & Limitations

    Window FC-CRFs omit connections between different windows, requiring window shifting to address their isolation.

Abstract

from arXiv · show

Estimating the accurate depth from a single image is challenging since it is inherently ambiguous and ill-posed. While recent works design increasingly complicated and powerful networks to directly regress the depth map, we take the path of CRFs optimization. Due to the expensive computation, CRFs are usually performed between neighborhoods rather than the whole graph. To leverage the potential of fully-connected CRFs, we split the input into windows and perform the FC-CRFs optimization within each window, which reduces the computation complexity and makes FC-CRFs feasible. To better capture the relationships between nodes in the graph, we exploit the multi-head attention mechanism to compute a multi-head potential function, which is fed to the networks to output an optimized depth map. Then we build a bottom-up-top-down structure, where this neural window FC-CRFs module serves as the decoder, and a vision transformer serves as the encoder. The experiments demonstrate that our method significantly improves the performance across all metrics on both the KITTI and NYUv2 datasets, compared to previous methods. Furthermore, the proposed method can be directly applied to panorama images and outperforms all previous panorama methods on the MatterPort3D dataset. Project page: https://weihaosky.github.io/newcrfs.

1. Introduction

Monocular depth estimation is ill-posed, motivating methods that combine neural prediction with CRF-based optimization. NeW CRFs makes fully-connected optimization feasible through windows and integrates it into a transformer-based network, achieving strong results across datasets.

  • Single-image depth estimation is inherently ambiguous because infinitely many 3D scenes can project to the same 2D image.
  • Existing approaches either directly regress depth with increasingly complex networks or use CRFs that are limited to neighborhood connections.
  • NeW CRFs partitions the input into sub-windows, reducing computation complexity and making fully-connected CRF optimization feasible.
  • Multi-head attention computes pairwise potentials, while the neural window FC-CRFs module serves as a decoder with a vision transformer encoder.
  • 10.3% and 9.8% decreases in KITTI Abs-Rel and RMS errors, respectively, accompany further reductions on NYUv2 and state-of-the-art results on MatterPort3D.

2. Related Work

Prior monocular depth methods use hand-designed graphical models, direct neural regression, or neural CRF refinement. NeW CRFs instead makes fully-connected graph optimization feasible within windows and trains the resulting decoder end-to-end.

  • Traditional Monocular Depth Estimation: Traditional MRF and CRF methods incorporate multiscale image cues and relationships between pixel depths, but struggle with accurate high-resolution dense prediction.
  • Neural Network Based Methods: Neural methods commonly either directly regress continuous depth maps or combine coarse predictions with CRF-based refinement.
  • Neural CRFs for Monocular Depth: Fully-connected CRFs connect each node to every other graph node, unlike regular CRFs, which connect nodes only to neighbors.
  • Neural CRFs for Monocular Depth: NeW CRFs divides the graph into windows, uses multi-head attention for pairwise relationships, and embeds the module as an end-to-end trainable decoder.

3. Neural Window Fully-connected CRFs

The paper makes fully-connected CRFs practical for monocular depth estimation by restricting optimization to windows and replacing hand-designed potentials with neural, attention-based potentials. These modules are integrated into a multilevel encoder–decoder that combines local window reasoning with shifted windows and global feature aggregation.

  • 3.1. Fully-connected Conditional Random Fields: Fully-connected CRFs connect each node to every other graph node, increasing receptive field but incurring resource-intensive pairwise computation.The paper notes that distant pixels may not determine a pixel’s depth, motivating a restricted connectivity pattern.
  • 3.2. Window Fully-connected CRFs: Window FC-CRFs divide an image into patch-based windows and fully connect patches only within each window, reducing pairwise computation complexity.Each patch is treated as one graph node, while patches in different windows are not connected.
  • 3.2. Window Fully-connected CRFs: Shifted windows provide information connections between neighboring regions that are isolated in the non-overlapping partition.The method computes energy for original and shifted windows successively.
  • 3.3. Neural Window FC-CRFs: Neural potentials replace hand-designed CRF functions, with unary potentials computed from image features and pairwise potentials learned from features, node values, and pairwise weights.The pairwise formulation computes nodewise sums using network-produced weighting functions.
  • 3.3. Neural Window FC-CRFs: Multi-head attention forms pairwise potential weights from query–key scores and relative position information, then applies them to previous predictions for message passing.The dot product between Q and K scores relationships among nodes, while the SoftMax output supplies message-passing weights.
  • 3.4. Network Structure: The network uses a Swin Transformer encoder and a multilevel CRF decoder, with a PPM head producing an initial prediction before successive CRF refinements.Four feature levels are processed, with rearrange upscaling between levels and global information aggregated at the top level.

4. Experiments

Experiments evaluate the method on outdoor, indoor, and panorama depth datasets, with comparisons, ablations, and implementation details. The method improves reported performance across KITTI, NYUv2, and MatterPort3D, while module studies clarify contributions and trade-offs.

  • Datasets: The experiments cover KITTI outdoor scenes, NYUv2 indoor scenes, and MatterPort3D panorama images.KITTI uses established splits, NYUv2 follows its official training/testing split, and MatterPort3D uses 7829 training images plus merged validation and testing sets.
  • KITTI evaluation: The method ranks first on the KITTI online benchmark and markedly reduces the main SILog ranking error.The official-split testing results come from the online benchmark, whose ground-truth test maps are withheld.
  • NYUv2 evaluation: NYUv2 performance improves across all metrics without additional data, with Abs Rel below 0.1 and δ < 1.252 reaching 99%.Qualitative results show better depth estimates in repeated textures, messy environments, and poor lighting.
  • MatterPort3D evaluation: 14.8% and 20.0%: MatterPort3D Abs Rel and Abs errors decrease when the perspective-image network is applied directly to panoramas.Pretraining on 50K additional real-world images further improves panorama depth estimation.

5. Conclusion

The paper concludes that neural window fully-connected CRFs make structured depth optimization feasible and achieve state-of-the-art results across three datasets.

  • 5. Conclusion: The neural window FC-CRFs module performs within sub-windows, uses multi-head attention for pairwise potentials, and serves as a decoder with a transformer encoder.The resulting bottom-up-top-down network predicts depth maps and is reported to outperform previous methods on KITTI, NYUv2, and MatterPort3D.

Estimation

The supplied passage identifies the paper's authors and their affiliation with Alibaba Group.

  • The listed authors are Weihao Yuan, Xiaodong Gu, Zuozhuo Dai, Siyu Zhu, and Ping Tan.
  • The authors are affiliated with Alibaba Group.
  • The passage provides author and affiliation information rather than estimation methods or results.

A. Network Structure

The network structure combines a transformer encoder with global-information aggregation and multi-level windowed graph processing using different attention-head counts.

  • A. Network Structure: The encoder’s top-level output feeds a PPM head, while each level builds patch graphs split into windows.The four levels use 32, 16, 8, and 4 attention heads from top to bottom.

B. Efficiency Experiments

Window FC-CRFs trade accuracy gains against rapidly increasing computation as window size grows, while remaining substantially more efficient than fully connected alternatives.

  • B. Efficiency Experiments: Window size 2 adds 26 ms and 0.1 G beyond feature extraction, establishing the efficiency baseline for window FC-CRFs.The reported differences are 121 − 95 = 26 ms and 2.7 − 2.6 = 0.1 G.
  • B. Efficiency Experiments: Accuracy improvements taper as window size increases, with the gain nearly saturated at 0.055 when N = 7.
  • B. Efficiency Experiments: Window FC-CRFs interpolate between traditional CRFs at N = 2 and full-image FC-CRFs at 352 × 1216.
  • B. Efficiency Experiments: Even with large windows, window FC-CRFs use less memory and computation than fully connected CRFs.

C. More Qualitative Results

Qualitative comparisons show that the method recovers finer depth details than prior methods, especially in challenging visual environments, using a multi-head pairwise potential.

  • C. More Qualitative Results: The method estimates better depth and recovers more details than BTS and Adabins on NYUv2 test images.
  • C. More Qualitative Results: The qualitative results use NYUv2 test-set comparisons among BTS, DPT, and the proposed model, with efficiency timing reported separately on RTX 2080 Ti.
  • C. More Qualitative Results: The top-level pairwise potential uses 32 heads, each with 32 channels, while the image graph is split into N × N windows.
  • C. More Qualitative Results: The qualitative advantage is most visible in repeated textures, messy environments, and poor lighting.

D. Point Cloud Visualization

Point-cloud visualizations indicate that estimated depths reconstruct indoor 3D structure reasonably on NYUv2 and preserve geometric regularity on unseen panoramas.

  • D. Point Cloud Visualization: NYUv2 point clouds recover the structures of the 3D world reasonably from the estimated depth maps.
  • D. Point Cloud Visualization: On unseen real-world indoor panoramas, complete room structures are reconstructed, including flat surfaces, straight lines, and right angles.
Loading 2203.01502v2…