Source-linked AI summary
From 2D to 3D: Re-thinking Benchmarking of Monocular Depth Prediction
Evin Pınar Örnek, Shristi Mudgal, Johanna Wald, Yida Wang, Nassir Navab, Federico Tombari
TL;DR
Monocular depth prediction is often benchmarked with 2D metrics that do not adequately assess the 3D geometry needed by depth-based applications. The paper proposes 3D evaluation metrics and the RIO-D3D indoor benchmark; its comparisons show that similar 2D scores can accompany different structural quality, with 3D metrics reflecting those differences.
Problem
Common 2D depth metrics do not sufficiently represent predicted 3D geometry, although depth-based applications rely on scene and object structure.
Source: §I · pp. 1–2Method
The paper proposes metrics that evaluate depth predictions in 3D and RIO-D3D, an indoor benchmark with high-quality 3D ground truth and geometric and semantic annotations.
Source: §I · p. 2, §III · p. 4, §IV · p. 4Results
Methods with similar 2D scores can differ in structural quality: Eigen and FCRN both score 0.21 absrel, while FCRN is better on 3D metrics, and BTS has the best F-score.
Source: B. Monocular Depth Prediction… · p. 6Takeaways & Limitations
The experiments support using 3D metrics to assess predicted geometry, with F-score identified as the most promising indicator.
Source: §VI · p. 7
Abstract
from arXiv · showhide
There have been numerous recently proposed methods for monocular depth prediction (MDP) coupled with the equally rapid evolution of benchmarking tools. However, we argue that MDP is currently witnessing benchmark over-fitting and relying on metrics that are only partially helpful to gauge the usefulness of the predictions for 3D applications. This limits the design and development of novel methods that are truly aware of - and improving towards estimating - the 3D structure of the scene rather than optimizing 2D-based distances. In this work, we aim to bring structural awareness to MDP, an inherently 3D task, by exhibiting the limits of evaluation metrics towards assessing the quality of the 3D geometry. We propose a set of metrics well suited to evaluate the 3D geometry of MDP approaches and a novel indoor benchmark, RIO-D3D, crucial for the proposed evaluation methodology. Our benchmark is based on a real-world dataset featuring high-quality rendered depth maps obtained from RGB-D reconstructions. We further demonstrate this to help benchmark the closely-tied task of 3D scene completion.
I. INTRODUCTION
Monocular depth prediction is increasingly shaped by optimizing a small set of 2D metrics that do not adequately reflect reconstructed 3D geometry, while some datasets also rely on noisy sensor depth. The paper proposes a more 3D-driven evaluation and benchmark, including RIO-D3D with complete, denoised 3D ground truth and semantic information.
I. INTRODUCTION
- Common 2D metrics assess depth globally on the image plane, so predictions that score well may still yield incorrect 3D structures, and sensor-noisy ground truth can bias training and evaluation.The authors connect structural accuracy to applications such as obstacle avoidance, grasping objects in clutter, and augmenting shapes with 3D information.Source: §I · pp. 1–2
- The work aims to shift MDP development, evaluation, and benchmarking toward 3D structure rather than a narrow set of 2D metrics.RIO-D3D is also intended to support evaluation of semantic scene completion.Source: §I · p. 2
- The paper proposes a 3D-oriented evaluation protocol and RIO-D3D, an indoor benchmark with highly accurate, complete 3D ground truth and normal, occlusion, and semantic annotations.The authors also compare state-of-the-art depth prediction and scene-completion methods using the proposed criteria.Source: §I · p. 2
II. RELATED WORK
Related work spans hand-crafted and deep-learning monocular depth prediction, methods targeting structural or boundary accuracy, and datasets with differing depth, semantic, and 3D-scene coverage. Standard evaluation is predominantly pixel-based, while prior boundary and plane metrics do not evaluate depth maps in terms of 3D structures.
Monocular depth prediction
- MDP methods progressed from hand-crafted depth-prior features to deep networks, ordinal formulations, and approaches addressing depth imbalance, boundaries, and multi-task prediction.Examples include ordinal regression, reversed Huber loss, and joint depth-and-normal or depth-and-semantic learning,,.Source: §II · p. 2
3D-aware depth prediction
- Several 3D-aware MDP approaches use normals, planar constraints, or 3D surface-patch consistency, but the reviewed methods remain limited by training-data quality and evaluation metrics.The 3D training loss of Yin et al. back-projects depth and enforces consistency on 3D surface patches.Source: §II · p. 2
Datasets and metrics
- Indoor datasets vary in resolution, depth quality, semantic coverage, and availability of complete 3D scenes; ScanNet lacks reconstructed depth maps, while ETH and DIODE lack semantics and full scenes including occluded regions.The passage also notes that DIODE is relatively small.Source: §II · pp. 2–3
- Common evaluation averages pixelwise depth comparisons using absolute and squared relative distance, root mean square error, logged root mean square error, and threshold accuracy.Koch et al. also proposed occlusion-boundary and plane-accuracy metrics, but the paper states that these do not evaluate depth maps in terms of 3D structures.Source: §II · p. 3
III. EVALUATION METHODOLOGY
The section reframes monocular depth prediction as single-view 3D reconstruction and applies 3D metrics to evaluate predicted geometry. It describes complementary measures of point-cloud matching, completeness, and overlap.
III. EVALUATION METHODOLOGY
- MDP is recast as single-view 3D reconstruction, with 3D metrics proposed to evaluate depth predictions rather than only 2D distances.The section presents this as a way to address missing 3D structure in evaluation.Source: §III · p. 3
- Earth Mover’s Distance matches equal-sized ground-truth and reconstructed point clouds through a bijection, but exact matching is too expensive and is approximated.The bijection maps points from the ground truth to the reconstruction.Source: §III · p. 3
- Chamfer Distance uses nearest-neighbor matches in both directions, but is sensitive to outliers and can find incorrect correspondences.Source: §III · p. 3
- Completeness measures ground-truth-to-reconstruction distances, while the 3D Point Cloud F-score summarizes precision and recall for surface quality.Completeness indicates error on retrieved points; F-score is the harmonic mean of precision and recall.Source: §III · p. 3
- The proposed non-voxelized point-cloud IoU expresses overlap through precision and recall, avoiding voxelization and its loss of surface quality.The 3D metrics are compared on the proposed dataset.Source: §III · p. 4
IV. EVALUATION DATASET: RIO-D3D
RIO-D3D is an indoor benchmark built on 3RScan reconstructions, with rendered depth maps and extensive geometric and semantic annotations. Its real-image evaluation subset contains 34K images.
IV. EVALUATION DATASET: RIO-D3D
- RIO-D3D combines calibrated RGB-D sequences and high-quality 3D ground truth with instance-level semantics, normals, and geometric and occlusion boundaries.It builds on 3RScan, which contains 1,482 RGB-D scans from 478 environments.Source: §IV · p. 4
- Dense 3D reconstructions reduce missing surfaces and sensor noise, enabling rendered depth maps with higher coverage and richer structural information.Depth maps are rendered using the RGB frames, camera poses, and intrinsic data.Source: §IV · p. 4
- The dataset adds normal maps, occlusion boundaries, semantic labels, and instance labels across the dataset.Normal maps and occlusion boundaries are generated using a point-cloud-based approach.Source: §IV · p. 4
- The real-image benchmark contains 34K images split 8:1:1 into training, validation, and test sets.The subset was selected for content richness and minimal redundancy.Source: §IV · p. 4
- Figure 3 shows depth distribution in millimeters by pixel count and dataset coverage.Source: Figure 3 · p. 5
V. EXPERIMENTS
The experiments first use two toy setups to motivate a new evaluation protocol, then report qualitative and quantitative results for monocular depth prediction and scene completion on RIO-D3D.
V. EXPERIMENTS
- Two toy experiments establish the need for a new evaluation protocol before the benchmark results are presented.Source: §V · p. 5
- The experiments report qualitative and quantitative results for monocular depth prediction on RIO-D3D.Source: §V · p. 5
- The benchmark results also cover 3D scene completion using RIO-D3D and the proposed evaluation methodology.Source: §V · p. 5
A. What do deep learning based MDP networks learn?
Toy experiments test whether 2D-based evaluation rewards predictions that lack scene structure. A median-depth plane receives reasonable 2D scores, while retrieval from training images outperforms regression methods quantitatively.
A. What do deep learning based MDP networks learn?
- The median-plane test uses handcrafted depth maps whose pixels all take the ground-truth map's median depth, while Oracle Nearest Neighbour retrieves the training depth map with the lowest absrel for each validation image.Source: A. What do deep learning based MDP… · p. 5
- The median-plane prediction scores 0.211 absrel, on par with model predictions, but has a CD of 0.677 and F-score of 0.369.Oracle Nearest Neighbour retrieval is quantitatively superior to the prediction methods, though its specific scores are not stated here.Source: A. What do deep learning based MDP… · p. 5
- The authors conclude that the analysis does not show true improvement in predicted structure and geometry, motivating a rethink of MDP benchmarking.Source: A. What do deep learning based MDP… · p. 5
B. Monocular Depth Prediction Benchmarking
On RIO-D3D, 3D metrics distinguish structural quality that standard 2D metrics often miss, including across methods with similar absrel scores. The authors recommend F-score as a promising 3D indicator for MDP.
B. Monocular Depth Prediction Benchmarking
- The benchmark evaluates Eigen, FCRN, BTS, and VNL; despite good metric scores, BTS point clouds contain outliers and fail to capture the 3D object surface.Source: B. Monocular Depth Prediction… · p. 5
- Eigen and FCRN both score 0.21 absrel, but FCRN performs better on 3D metrics; BTS is structurally superior and has the best F-score, while VNL has 0.25 absrel and 0.76 CD versus FCRN's 0.21 and 0.49.The qualitative predictions are shown alongside ground truth in Figure 5.Source: B. Monocular Depth Prediction… · p. 6, Figure 5 · p. 6
- Standard 2D scores can conflict with per-sample structure: examples with absrel near the test average have poor F-scores and poor 3D structure.Figure 6's error-map colors distinguish ground truth, prediction, and their intersection.Source: B. Monocular Depth Prediction… · p. 6, Figure 6 · p. 6
- For a small range of absrel values, 3D scores vary more and better reflect structural quality; F-score worsens quickly as structural overlap falls.Per-pixel averaged depth error may remain small even when depth is highly inaccurate.Source: B. Monocular Depth Prediction… · p. 6
- F-score is recommended as the most promising proposed indicator because it balances precision and recall and measures correctly reconstructed 3D points.The authors describe it as more reliable than CD, EMD, and completion, and more robust than IoU.Source: B. Monocular Depth Prediction… · p. 7
C. Scene Completion Benchmarking
RIO-D3D extends its depth benchmark to 3D scene completion using partial depth maps and full 3D scenes. On this benchmark, voxel-based approaches perform better than point-cloud methods on CD, EMD, and qualitative results.
C. Scene Completion Benchmarking
- RIO-D3D supports scene completion benchmarking with partial depth-map inputs and completed scenes extracted from full 3D meshes.The paper compares state-of-the-art scene-completion methods on this benchmark.Source: C. Scene Completion Benchmarking · p. 7
- Voxel-based approaches (ForkNet) work better than point-cloud methods in CD, EMD, and qualitative results.Point-cloud methods GRNet and MSN produce noisy results.Source: C. Scene Completion Benchmarking · p. 7
- Point-cloud methods produce noisy results, while RIO-D3D scenes vary more in geometry and depth than existing scene-completion datasets.The methods focus on completion tasks where samples do not vary significantly in geometry and poses.Source: C. Scene Completion Benchmarking · p. 7
VI. CONCLUSION
The conclusion argues for evaluating monocular depth prediction by 3D structural accuracy rather than global pixel-wise error. It identifies F-score as the most promising indicator and presents complete 3D structures and high-quality depth maps as part of the proposed benchmark.
VI. CONCLUSION
- F-score is the most promising indicator among the experiments reported.Source: §VI · p. 7
- The paper points to 3D metrics and a benchmark with complete 3D structures and high-quality depth maps for evaluating depth-map structure.Source: §VI · p. 7
- The authors hope future MDP research will move beyond retrieving planes and images.Source: §VI · p. 8