Source-linked AI summary
Guiding Monocular Depth Estimation Using Depth-Attention Volume
Lam Huynh, Phong Nguyen-Ha, Jiri Matas, Esa Rahtu, Janne Heikkila
TL;DR
Monocular depth estimation must infer scene depth from a single image using learned priors, but conventional approaches require large datasets and bulky networks. This paper guides estimation toward planar structures with a non-local coplanarity constraint implemented through depth-attention volume (DAV), achieving state-of-the-art results on NYU-Depth-v2 and ScanNet with 2-8 times fewer parameters than competing methods.
Problem
Monocular depth estimation requires learning priors from large datasets to infer depth from a single image, leading to large network architectures.
Method
The method incorporates a non-local coplanarity constraint through depth-attention volume (DAV) to guide depth estimation toward planar structures.
Results
State-of-the-art depth estimation results were achieved on NYU-Depth-v2 and ScanNet using 2-8 times fewer parameters than competing methods.
Takeaways & Limitations
DAV enables more efficient learning of depth priors while favoring planar structures common in indoor environments.
Takeaways & Limitations
The approach assumes scenes contain multiple non-parallel planes and that same-plane depth values are linearly dependent, particularly in indoor environments.
Abstract
from arXiv · showhide
Recovering the scene depth from a single image is an ill-posed problem that requires additional priors, often referred to as monocular depth cues, to disambiguate different 3D interpretations. In recent works, those priors have been learned in an end-to-end manner from large datasets by using deep neural networks. In this paper, we propose guiding depth estimation to favor planar structures that are ubiquitous especially in indoor environments. This is achieved by incorporating a non-local coplanarity constraint to the network with a novel attention mechanism called depth-attention volume (DAV). Experiments on two popular indoor datasets, namely NYU-Depth-v2 and ScanNet, show that our method achieves state-of-the-art depth estimation results while using only a fraction of the number of parameters needed by the competing methods.
1 Introduction
Monocular depth estimation must infer 3D structure from a single image, but brute-force learning of monocular cues demands large datasets and bulky networks. The paper guides estimation with coplanarity through depth-attention volume, achieving state-of-the-art results with fewer parameters.
- Monocular depth estimation infers depth from pixel values, making it a compelling alternative to multi-view geometry for applications such as 3D modeling, augmented reality, and autonomous vehicles.
- Brute-force learning of monocular cues requires huge training data and large architectures, while computational and memory demands limit practical deployment.
- The method guides learning with a coplanarity constraint for points on planar surfaces, especially relevant to indoor scenes containing walls, floors, ceilings, and tables.
- Depth-attention volume aggregates non-local information from coplanar structures and learns both fronto-parallel and non-fronto-parallel constraints end to end.
- The proposed model achieves state-of-the-art depth estimation on NYU-Depth-v2 and ScanNet while using considerably fewer parameters than previous methods.
- Depth-attention maps visualize how the coplanarity prior supports recovery of depth in a textureless white wall.
2 Related work
Prior monocular depth methods learned depth cues with deep networks, while plane-based methods imposed geometric structure explicitly. This paper instead uses DAV to encode non-local coplanarity implicitly and guide depth toward planar surfaces.
- Learning-based monocular depth estimation: Deep monocular depth estimation commonly uses large encoders and feature-fusion strategies to learn depth cues from images.These approaches include VGG-16, ResNet variants, up-projection, multi-scale fusion, and adaptive dense feature fusion.
- Plane-based approaches: Plane-based methods impose geometric constraints through explicit plane parameters, planar segmentation, or virtual planes.Prior work differs in whether it estimates plane parameters and segmentation masks or uses a large virtual plane for robustness to noisy ground truth.
- Attention mechanism: Attention mechanisms have been applied to vision tasks and monocular depth estimation through channel-wise, spatial-wise, and multi-scale designs.Earlier depth work used spatial attention to guide a Conditional Random Fields model.
- Attention mechanism: DAV encodes non-local geometric dependencies and guides depth estimation toward values associated with planar surfaces without explicitly segmenting planes.The network learns the coplanarity constraint implicitly rather than predicting plane masks or parameters.
3 Proposed Method
The proposed method builds depth-attention volumes from planar relationships, predicts and applies them in a non-local module, and trains the network with attention and depth losses. Its architecture combines an encoder, DAV module, and decoder while using losses for depth, edges, and surface normals.
- 3.1 Depth-attention volume: DAV measures how well one image point can predict another point’s depth, using planar and fronto-parallel relationships.The method assumes points on the same plane have linearly dependent depths and constructs attention volumes from prominent planes.
- 3.1 Depth-attention volume: The method combines first-order plane volumes and a zero-order fronto-parallel volume by taking the maximum attention value for each point pair.The resulting DAV is symmetric and represented as a 4-D tensor, with subsampling used to limit memory requirements.
- 3.2 Network Architecture: The network passes the image through an encoder, non-local depth-attention module, and decoder to produce the estimated depth map.The encoder uses DRN-D-22, while the decoder upsamples features through bilinear interpolation and convolutional layers.
- 3.2 Network Architecture: The non-local module adds DAV-based information to encoder features while predicting and learning the DAV from ground-truth data.Its feature-processing design includes embeddings, DAV-weighted feature multiplication, and a residual connection.
- 3.3 Loss Function: Training uses attention loss for predicted-versus-ground-truth DAVs and depth loss combining logarithmic depth, gradient, and surface-normal terms.The logarithmic loss measures depth error in log space, the gradient loss penalizes abrupt edge changes, and the normal loss emphasizes small details.
4 Experiments
The method is evaluated on NYU-Depth-v2, ScanNet, iBims-1, and cross-dataset SUN-RGBD settings against recent monocular depth-estimation methods. It achieves strong benchmark performance, with fewer parameters than competing approaches and evidence that the attention loss and planar guidance contribute to performance.
- Evaluation setup: The evaluation uses NYU-Depth-v2 and ScanNet, with standard depth metrics and additional iBims-1 and cross-dataset SUN-RGBD assessments.NYU-Depth-v2 uses REL, RMS, and thresholded accuracy; ScanNet and SUN-RGBD use inverse-depth metrics; iBims-1 includes plane-related metrics.
- NYU-Depth-v2: The proposed model achieves the best NYU-Depth-v2 performance while using the fewest parameters; leading baselines use 4.5, 6.2, and 8.3 times more parameters.The compared baselines are Yin et al., Hu et al., and Chen et al., respectively.
- Qualitative comparison: Qualitative NYU-Depth-v2 results show closer agreement with ground truth than state-of-the-art methods, while baselines trade detail, smoothness, or continuity differently.Eigen and Fergus perform well on uniform regions but miss detail; Laina et al. produce overly smoothed maps; Fu et al. produce discontinuities inside objects or smooth areas.
- ScanNet: On ScanNet, the proposed model outperforms Ren et al. with a clear margin in REL and is substantially better than ROB challenge methods, although the latter comparison is not strictly fair.The ROB comparison uses different test splits, while all methods use the same ROB training split.
- Ablation: Removing or later dropping the attention loss degrades performance, indicating that this loss term is important for the model.The comparison trains models with and without the attention loss and continues training after removing it.
- Cross-dataset evaluation: When trained on NYU-Depth-v2 and tested on SUN-RGBD without fine-tuning, the model performs favorably relative to the evaluated baselines.The cross-dataset experiment assesses generalization across indoor datasets.
5 Conclusions
The paper proposes DAV, which incorporates a non-local coplanarity constraint to favor planar structures in monocular depth estimation. The method achieves state-of-the-art performance on two indoor benchmarks with substantially fewer parameters and also demonstrates cross-dataset generalization.
- Contribution: The method incorporates a non-local coplanarity constraint through depth-attention volume to favor planar structures common in indoor environments.The proposed attention mechanism guides depth estimation toward planar scene structure.
- Generalization: Cross-dataset experiments further demonstrate the method’s generalization ability.The conclusion reports this as an additional property of the proposed solution.
Depth-Attention Volume
Additional qualitative analyses examine the effect of the loss configuration, reconstructed geometry, and depth predictions from the proposed model. The full loss improves boundaries and detailed areas, while reconstructed planar regions remain close to ground truth.
- Loss analysis: The full loss significantly improves depth-map quality at boundaries and detailed areas compared with the model trained without the attention term.The comparison is shown in Figures 1 and 2.
- Point-cloud reconstruction: Reconstructed point clouds from NYU-Depth-v2 show virtually linear and ground-truth-like results near walls, floors, and ceilings.The point clouds are reconstructed from three arbitrary test-set views.
B Additional qualitative results of cross-dataset evaluation on SUN-RGBD
Cross-dataset qualitative results evaluate a model trained on NYU-Depth-v2 and tested on SUN-RGBD without fine-tuning. The model retains scene geometry, attends to planar regions, and produces point clouds relatively close to ground truth.
- Cross-dataset setup: The cross-dataset analysis uses a pretrained NYU-Depth-v2 model to estimate SUN-RGBD depth without fine-tuning.Predicted depth maps, reconstructed point clouds, and attention maps are used to examine generalization.
- Depth predictions: SUN-RGBD depth maps retain the scene geometry even in difficult examples, despite the model not being trained on SUN-RGBD.The model provides reasonable depth maps for the displayed SUN-RGBD examples.
- Attention behavior: Attention maps show that the network focuses on planar areas, shifting from table surfaces to background walls at different query points.Warm attention indicates concentration on the relevant planar surface.
- Point-cloud reconstruction: Reprojected SUN-RGBD point clouds are relatively close to ground truth even though they are produced by a model trained on a different dataset.The point-cloud views are shown from different camera poses.
C.1 Planarity error
The evaluation compares the model with officially provided iBims-1 methods and examines embedding-space choices for the depth-attention module.
- The model outperforms recent iBims-1 methods on most metrics, including plane-related metrics.The comparison is reported in Table 1.
- Cross-modulation in two embedding spaces performs better than a single embedding with twice as many features.This finding is reported empirically in Table 2.
D Network architecture
The network combines an encoder, a non-local depth-attention module, and a decoder, with staged training and a straightforward upsampling-based decoding path.
- Network structure: The network consists of an encoder, a non-local depth-attention module, and a decoder.The encoder is based on the DRN-D-22 dilated residual network, with its detailed structure given in Table 3.
- Training: The first training phase initializes the encoder with ImageNet-pretrained weights, while the remaining two stages follow the main-paper procedure.The experiments report improved accuracy and faster convergence with pretraining.
- Depth-attention module: The non-local depth-attention module uses green, blue, and orange embedding spaces with element-wise products, sums, and outer products.The module's internal structure is detailed in Table 4, with tensor reshaping and permutation used where required.
- Decoder: The decoder uses two bilinear upsamplings followed by 2D convolutional layers and batch normalization to produce the final depth map.The upsampled features are refined with two additional 2D convolutional layers; Table 5 gives the detailed structure.
E Definitions of the evaluation metrics
The evaluation uses depth, inverse-depth, threshold, plane, and boundary metrics, with metric direction explicitly distinguishing better lower from better higher values.
- Validity and depth metrics: Valid pixels have predicted and ground-truth depths in [0.0, 10.0] and are used to calculate errors.NYU-Depth-v2 evaluation uses these valid pixels for the listed metrics.
- Validity and depth metrics: REL, RMSE, and thresholded accuracy δ_i evaluate depth prediction, with lower REL and RMSE and higher δ1, δ2, and δ3 indicating better results.REL is mean absolute relative error, RMSE is root mean square error, and δ_i is thresholded accuracy.
- Additional depth metrics: Additional depth metrics include sqREL, iMAE, iRMSE, and SI for evaluation on ScanNet and SUN-RGBD.These comprise mean relative square error, inverse-depth absolute and root mean square errors, and scale-invariant mean square error.
- Plane metrics: Plane metrics measure predicted-plane flatness through point-to-plane distance variation and orientation through angles between predicted and ground-truth normals.The corresponding metrics are denoted ϵplan and ϵorie.
- Boundary and directed errors: Boundary metrics assess depth-edge accuracy and completeness by combining predicted or ground-truth edge maps with distance maps.The metrics are denoted ϵacc and ϵcomp.
- Boundary and directed errors: Directed depth errors measure the proportions of predicted 3D points on, in front of, or behind a reference plane located 3 meters away.These metrics are denoted ϵ0, ϵ−, and ϵ+.