NeW CRFs: Neural Window Fully-connected CRFs for Monocular Depth Estimation

Introduction

• Monocular depth is ill-posed, and methods relying on increasingly complex direct-regression networks face a difficult fitting problem without multi-view geometric constraints.Traditional MRF/CRF methods use observation cues and prior predictions to guide depth estimation, but are generally limited to neighborhood CRFs.Source: §1 · p. 1
• Splitting images into windows makes fully connected CRF optimization feasible, and multi-head attention computes pairwise potentials for the neural CRF module.Fully connected CRFs capture relationships between any graph nodes, but their computational cost had limited prior methods to neighborhood CRFs.Source: §1 · p. 1
• A vision transformer serves as encoder and window FC-CRFs as decoder, with window shifts and globally pooled features addressing window isolation and missing global information.The method is organized as a bottom-up-top-down network.Source: §1 · p. 2
• KITTI Abs-Rel and RMS errors fell by 10.3% and 9.8%, while NYUv2 errors fell by 7.8% and 8.2%; the method ranks first on KITTI and sets a new MatterPort3D state of the art.The introduction reports these reductions and benchmark standings for the proposed method.Source: §1 · p. 2

2.1. Traditional Monocular Depth Estimation

• Saxena et al. used multiscale local and global features in discriminative MRFs, while later MRF/CRF methods still struggled to estimate accurate high-resolution dense depth maps.These methods inferred depth from monocular cues including color, pixel position, occlusion, object size, haze, and defocus.Source: §2.1 · p. 2

2.2. Neural Networks Based Monocular Depth

• Neural depth methods either directly regress continuous maps [12] [17] [28] [39] or discretize depth for classification or ordinal regression, but remain difficult-fitting approaches with increasingly complex networks.Other approaches use sparse depth [10] or segmentation information [15] [24] [27] [44] to help train depth networks.Source: §2.2 · p. 2

2.3. Neural CRFs for Monocular Depth

• Windowing makes fully connected CRFs feasible by connecting each node within its window, while multi-head attention models pairwise relations in an end-to-end decoder.Prior neural CRFs either refined network predictions separately or integrated CRFs end-to-end [20] [38], but avoided fully connected CRFs because of computational complexity.Source: §2.3 · pp. 2–3, Figure 2 · p. 3

3.1. Fully-connected Conditional Random Fields

• Fully connected CRFs model each pixel’s depth using unary predictions and pairwise connections to all other nodes, capturing long-range relationships that regular CRFs omit.The approach uses this global connectivity to help depth prediction, although its cost motivates restricting connections to windows.Source: §3.1 · p. 3

3.2. Window Fully-connected CRFs

• Window FC-CRFs connect patches only within each window, reducing pairwise computation; shifted windows then connect otherwise isolated neighboring regions.Each window contains N × N patches, and optimization is performed successively on regular and shifted windows.Source: §3.2 · pp. 3–4

3.3. Neural Window FC-CRFs

• Neural window FC-CRFs learn unary and pairwise potentials, using feature-derived query-key dot products and relative position embeddings to weight message passing between predicted values.Multiple attention heads enhance relationship capture; the optimization network uses two fully connected layers to produce the refined depth map.Source: §3.3 · pp. 4–5, §3.4 · p. 5

3.4. Network Structure

• The network uses four levels of neural window FC-CRF optimization as a decoder, with a Swin Transformer encoder extracting image features.A pyramid pooling module aggregates whole-image information for the initial prediction; shuffle-based rearrangement upscales between levels, and training uses Scale-Invariant Logarithmic loss.Source: §3.4 · p. 5

4.1. Implementation Details

• The network is trained end-to-end for 20 epochs with Adam, batch size 8, and learning rate decreasing from 1 × 10−4 to 1 × 10−5.Implementation uses PyTorch on Nvidia GTX 2080 Ti GPUs; output depth maps are produced at one-quarter resolution and resized to full resolution.Source: §4.1 · p. 6

4.2. Datasets

• Experiments use KITTI, NYUv2, and MatterPort3D dataset splits; MatterPort3D images are resized to 1024 × 512 for training and evaluation.KITTI uses the Eigen split (23,488 training pairs; 697 test images) and official split (42,949 training pairs; 1,000 validation images; 500 test images). NYUv2 uses 249 training scenes and 654 test images from 215 scenes. MatterPort3D uses 7,829 training images and a merged set of 957 validation and 2,014 testing images.Source: §4.2 · p. 6

4.3. Evaluations

• On NYUv2, without additional data, the method improves all metrics, with Abs Rel below 0.1 and δ < 1.252 accuracy reaching 99%.Qualitative results show better estimates in difficult regions, including repeated textures, messy environments, and poor lighting.Source: Table 3 · p. 7, §4.3 · p. 7, Figure 5 · p. 7
• On KITTI, Eigen-split Abs-Rel, Sq Rel, RMSE, and RMSElog errors fall 10.3%, 18.4%, 9.8%, and 10.2%; on the official split, the method ranks first.The Eigen-split method also outperforms prior methods trained with additional data, despite using none; on the official split it markedly reduces SILog error.Source: §4.3 · p. 7
• On MatterPort3D, the method outperforms prior panorama methods, reducing Abs Rel by 14.8% and Abs by 20.0%.Pretraining on 50K additional images before MatterPort3D fine-tuning further improves performance.Source: Table 4 · p. 8, §4.3 · p. 8

4.4. Abalation Study

• Replacing the baseline convolutional decoder with neural window FC-CRFs reduces Abs Rel from 0.069 to 0.055, then to 0.054 with shift.The baseline uses the same encoder but omits the PPM head and rearrange upscale module.Source: Table 5 · p. 8, §4.4 · p. 8
• Ablations show rearrange upscale sharpens edges and reduces parameters, PPM adds global context, and fewer energy heads slightly reduce performance.The PPM head helps in regions difficult to estimate from local information alone, such as complex textures and white walls.Source: §4.4 · p. 8

• The method significantly outperforms previous methods and sets new state-of-the-art performance on KITTI, NYUv2, and MatterPort3D.Source: §5 · p. 8
• The module addresses the expensive computation of fully-connected CRFs by calculating pairwise potentials within input sub-windows.Source: §5 · p. 8
• Multi-head attention computes a neural potential function, and the CRF module serves as decoder in a bottom-up-top-down structure with a transformer encoder.Source: §5 · p. 8

• The encoder’s top-level output feeds a PPM head that aggregates local and global information.Source: A. Network Structure · p. 11
• At each level, the network builds a graph with patches as nodes and splits it into k windows of size N × N.Source: A. Network Structure · p. 11
• From the top level to the bottom level, the numbers of heads are 32, 16, 8, and 4.Source: A. Network Structure · p. 11

B. Efficiency Experiments

• At N = 7, accuracy improvement is almost saturated at 0.055, while cost grows exponentially to 250 ms and 5.3 G at N = 48.The N = 2 window costs 121 − 95 = 26 ms and 2.7 − 2.6 = 0.1 G, excluding feature extraction.Source: B. Efficiency Experiments · p. 11
• The 7 × 7 window FC-CRFs costs 29 ms without feature extraction, versus 59.1 s for depth-estimation CRFs and 0.5 s for segmentation FC-CRFs.Source: B. Efficiency Experiments · p. 11
• Window size 2 is equivalent to traditional CRFs, while a 352 × 1216 window is equivalent to fully connected CRFs; even N = 48 uses less memory and computation than FC-CRFs.Source: B. Efficiency Experiments · p. 11

C. More Qualitative Results

• On the NYUv2 test set, the authors’ method estimates better depth and recovers more details than BTS and Adabins, especially in repeated-texture, messy-environment, and bad-light regions.Source: C. More Qualitative Results · pp. 11–12

D. Point Cloud Visualization

• Figure 3 shows NYUv2 test-set point clouds generated by projecting image pixels with estimated depth, with 3D-world structures reasonably recovered.Source: D. Point Cloud Visualization · p. 12
• Figure 4 shows complete room reconstruction from unseen real-world indoor panoramas, with flat floors, ceilings, and walls, straight lines, and right angles, demonstrating generalization.Source: D. Point Cloud Visualization · p. 12