Source-linked AI summary
Evaluation of CNN-based Single-Image Depth Estimation Methods
Tobias Koch, Lukas Liebel, Friedrich Fraundorfer, Marco Körner
TL;DR
Evaluation of single-image depth estimation is limited because common metrics do not directly assess important structural properties of predicted depth maps. The paper introduces interpretable quality measures and a high-quality RGB-D dataset, then uses them to compare state-of-the-art methods and reveal differences beyond global metrics.
Problem
Common depth metrics do not directly assess planar reconstruction, plane orientation, or the precise localization of depth discontinuities.
Method
The paper introduces complementary quality measures for edges, planar regions, depth consistency, and absolute distance accuracy, supported by a high-quality RGB-D dataset.
Results
The protocol revealed subtle differences between state-of-the-art methods across edge, planarity, consistency, and distance-accuracy properties.
Takeaways & Limitations
The proposed criteria enable more meaningful analysis and comparison of single-image depth maps than commonly used global metrics alone.
Takeaways & Limitations
Existing benchmark datasets can have low-resolution or sparse depth maps, limited scene variety, redundancy, or setups not designed for single-image depth estimation.
Abstract
from arXiv · showhide
While an increasing interest in deep models for single-image depth estimation methods can be observed, established schemes for their evaluation are still limited. We propose a set of novel quality criteria, allowing for a more detailed analysis by focusing on specific characteristics of depth maps. In particular, we address the preservation of edges and planar regions, depth consistency, and absolute distance accuracy. In order to employ these metrics to evaluate and compare state-of-the-art single-image depth estimation approaches, we provide a new high-quality RGB-D dataset. We used a DSLR camera together with a laser scanner to acquire high-resolution images and highly accurate depth maps. Experimental results show the validity of our proposed evaluation protocol.
1 Introduction
Single-image depth estimation methods can infer geometry from monocular images, but standard evaluation often misses important local defects. The paper addresses this gap with a high-quality dataset and interpretable measures targeting those defects.
- CNNs infer geometric information from monocular RGB or intensity images for applications including autonomous driving, robot navigation, and augmented reality.
- Standard evaluation mainly reports global depth-residual statistics, providing limited insight into planar surfaces and geometric discontinuities.
- Example predictions can contain artifacts, smoothing, missing objects, and inaccuracies in textured image regions despite appearing reasonable overall.
- The paper contributes a dataset with high-resolution RGB images and highly accurate laser-scan depth maps, alongside new interpretable error metrics and state-of-the-art evaluations.
2 Related Work
Related work covers learning-based single-image depth prediction and existing RGB-D benchmarks. Existing datasets provide useful training or evaluation data but have limitations in resolution, density, scene variety, or suitability for single-image methods.
- The related work reviews recent learning-based methods for predicting depth from a single image and datasets used to train and evaluate them.
- Single-view depth estimation differs from stereo reconstruction, which commonly triangulates 3D points from corresponding observations in multiple cameras.
- Existing multi-view stereo benchmarks provide high-resolution images and accurate laser-scanner ground truth but are not designed for single-image depth estimation.
- Make3D, KITTI, and NYU Depth v2 have limitations including low-resolution or sparse depth maps, limited scene variety, and redundant or restricted scene coverage.
3 Error Metrics
The paper supplements established global depth errors with interpretable measures targeting absolute-depth ranges, planar surfaces, depth boundaries, and directional consistency. These metrics use annotated planar regions, edge comparisons, and reference depth planes to analyze specific geometric properties.
- Limitations of established metrics: Standard global statistics can miss planar-surface accuracy, plane orientation, and precise depth-discontinuity localization.They summarize overall residuals but do not directly assess these salient geometric characteristics.
- Proposed quality measures: New quality measures evaluate predicted depth maps across absolute scene-depth ranges, planar reconstruction, depth-discontinuity representation and localization, and whole-image consistency.The measures are intended to complement, rather than replace, traditional error metrics.
- Planarity: Planarity errors compare predicted and ground-truth 3D planes fitted to masked planar regions for flatness and orientation.Planarity error uses averaged point-to-plane distances, while orientation error is the 3D angle between plane normal vectors; predictions are scaled to ground truth.
- Depth boundary errors: Depth boundary errors compare predicted and ground-truth binary depth edges using truncated chamfer distances, measuring both boundary accuracy and completeness.Distances beyond threshold θ are ignored; completeness accounts for edges missing from the predicted depth image.
- Directed depth errors: Directed depth errors measure the proportions of predicted pixels that are too far or too close relative to a reference depth plane.A reference plane at a specified distance masks predicted pixels in front of or behind it for correctness assessment.
4 Dataset
The authors compile the IBims-1 dataset to support direct evaluation of single-image depth estimation using high-quality RGB imagery, accurate laser-scanned depth, and annotations targeting planes, edges, and challenging scenes.
- Dataset construction: IBims-1 was compiled because the proposed metrics require extended ground truth unavailable in standard datasets.The dataset was designed specifically for direct evaluation of depth maps produced by single-image depth estimation methods.
- Acquisition: Camera calibration, 2D–3D registration, and a virtual depth sensor were used to derive matching depth maps from the laser point clouds.Undistortion and EPnP-based pose estimation aligned the camera imagery with the laser-scanner coordinate system.
- Acquisition: A Leica HDS7000 laser scanner captured scans with 3 mm point spacing and 0.4 mm RMS at 10 m, while an additional camera captured optical imagery.The scanner supplied dense, low-noise point clouds; the camera supplied RGB images because the scanner did not provide them.
- Dataset composition: The main dataset contains 54 indoor scenes and provides high-quality and NYU-v2-like RGB-D versions.Scenes include offices, lecture and living rooms, computer labs, a factory room, corridors, and potted plants; the final dataset was planned to contain 100 image pairs.
- Annotations: IBims-1 includes manually created semantic masks, plane annotations, and distinct edges detected automatically and manually selected for DBE evaluation.Figure 3 illustrates masks for walls, floors, tables, transparent objects, invalid pixels, and distinct edges.
- Auxiliary data: Four auxiliary image sets cover outdoor scenes, misleading printed patterns, geometric and radiometric augmentations, and viewpoint changes for multi-view stereo evaluation.The auxiliary data include cars, buildings, far ranges, printed NYU-v2 and Pattern samples, augmented images, and handheld views.
5 Evaluation
The evaluation compares state-of-the-art single-image depth estimation methods on IBims-1 and NYU-v2 using established and proposed metrics. Additional experiments examine robustness to geometric and color transformations and planarity on textured vertical surfaces.
- 5 Evaluation: State-of-the-art single-image depth estimation methods were evaluated on both the IBims-1 reference test dataset and the commonly used NYU-v2 dataset.The evaluation used established metrics together with the proposed metrics.
- 5 Evaluation: Additional experiments investigated robustness to geometric and color transformations and the planarity of textured vertical surfaces.
- 5 Evaluation: The experiments compared several state-of-the-art methods using the proposed evaluation protocol.
5.1 Evaluation Using Proposed Metrics
The proposed metrics evaluate SIDE methods across distance ranges, planar structures, depth boundaries, and directed depth errors on NYU-v2 and IBims-1. Results expose dataset-range effects, orientation differences, edge oversmoothing, and depth-direction biases.
- Evaluation protocol: Evaluation on NYU-v2 and IBims-1 uses the paper’s proposed metrics, with results reported graphically and in Table 3.The protocol covers distance-related errors, planarity, depth-boundary accuracy, and directed depth error.
- Distance-related assessment: 1 m depth intervals reveal comparable relative and RMS error trends across datasets within their shared depth range.The comparable trends support the reported explanation that lower overall scores arise mainly from depths beyond 10 m.
- Planarity: Planarity errors differ across object classes: floors have substantially more accurate orientations than walls, whose absolute orientation errors remain high.Li et al. performs much better on horizontal-plane orientations but poorly on vertical surfaces.
- Depth-boundary accuracy: Liu et al. and Li et al. best preserve true depth boundaries, whereas smoother predictions lose sharp transitions and produce larger completeness errors.The comparison uses the accuracy and completeness depth-boundary errors, illustrated in Figure 7.
- Directed depth error: At a 3 m reference plane, correctly estimated depths approach 90% on NYU-v2 and 80% on IBims-1, while predictions tend toward distances that are too short.The directed depth error distinguishes correct-side predictions from overestimated and underestimated depths.
5.2 Further Analyses
Additional experiments test SIDE robustness to image transformations, noise, and planar images with printed patterns. The methods show transformation-dependent behavior and fail to identify the actual planarity in the auxiliary examples.
- Data augmentation: Augmentation experiments apply horizontal and vertical flips, color transformations, and Gaussian and salt-and-pepper noise to assess robustness.The augmented images are evaluated using global relative error for selected methods.
- Data augmentation: Vertical flipping increases error by up to 60%, whereas horizontal flipping has little effect.The contrasting response is reported as evidence of different behavior under geometric transformations.
- Data augmentation: Slight overexposure notably affects results, while underexposure is less problematic and histogram stretching has no observed influence.The reported effects concern the global relative error metric.
- Data augmentation: The methods appear robust to color changes: grayscale inputs produce the same error as the reference inputs.This result is reported for the augmentation parameter s = 0.
- Textured planar surfaces: On printed patterns and NYU-v2 images placed on planar surfaces, all tested methods estimate depth incorrectly and fail to identify the actual planarity.Image gradients appear to provide a strong but misleading cue in the depicted example.
6 Conclusions
The paper introduces quality criteria and a high-quality RGB-D dataset for evaluating SIDE methods beyond standard global statistics. Experiments reveal persistent weaknesses in planar-surface accuracy and edge sharpness.
- Contributions: The paper presents new quality criteria for evaluating SIDE methods and a high-quality dataset supporting their use.The dataset provides high-resolution images, accurate depth maps, and new scene content for evaluation.
- Evaluation scope: The protocol assesses edge preservation, planar-region quality, depth consistency, and absolute distance accuracy.These properties enable comparisons that are more specific than commonly used global metrics.
- Evaluation scope: The proposed criteria reveal subtle differences between SIDE methods that common global metrics may not expose.The conclusion attributes this capability to the broader set of meaningful depth-map properties evaluated.
- Findings: Experiments show that planar-surface predictions often lack accuracy and that many methods produce oversmoothed depth edges.The dataset is intended to support future development using high-resolution images and scenes with extended depths.
Supplementary Material for Evaluation of CNN-based Single-Image Depth
The supplementary document provides additional information and visualizations for the paper’s sections, including dataset samples, masks, acquisition setup, and evaluation results.
- Supplementary material: The supplementary material adds information on paper sections and presents further visualizations related to the IBims-1 dataset and evaluation.Examples include sample images, provided masks, the data-acquisition setup, and qualitative evaluation results.
- Organization: Its section numbering and naming follow those used in the main paper.This correspondence is stated as a guide to the supplementary document.
- Contents: The supplementary material includes dataset samples, masks, acquisition details, and qualitative evaluation visualizations.These materials provide additional documentation beyond the graphical results shown in the paper.
4. Dataset
IBims-1 combines high-quality RGB images with accurate laser-scanned depth maps from indoor scenarios, supplemented by annotations and acquisition details for evaluation.
- Common RGB-D products were unsuitable as reference data because their resolution and depth accuracy were insufficient.
- IBims-1 provides sample RGB images and corresponding depth maps captured with different sensors.
- Additional dataset materials include RGB-depth examples and edge-mask examples used in evaluation procedures.
- Dataset images include binary masks identifying unreliable or invalid depth pixels, including transparent-object regions.
- The acquisition setup used a DSLR camera and laser scanner mounted on a survey tripod.
- A custom panoramic tripod aligned the camera’s optical center with the scanner-coordinate origin to avoid gaps in depth maps.
5. Evaluation
The evaluation combines standard global errors with targeted analyses of planar regions, perturbation robustness, and responses to textured planar patterns. Results include quantitative tables, visualizations, and point-cloud inspections.
- Global evaluation: Standard global metrics are reported for NYU-v2 and IBims-1, with lower rel, log10, and RMS values indicating better results.
- Planarity evaluation: Planarity metrics separately analyze horizontal and vertical planes, with fitted planes and 3D point clouds visualized for sample images.
- Data augmentation: All examined methods tolerated up to 10% image noise before result quality decreased notably.
- Data augmentation: The AlexNet version of Eigen and Fergus appeared more noise-robust than its VGG version, while VGG was less sensitive to blurred inputs.
- Textured planar surfaces: On printed planar patterns, all networks responded to the patterns, but Laina et al. produced a less severe constant offset response.
- Textured planar surfaces: Input edges influenced predictions, while the superpixel-based method produced visible artifacts in the pattern experiments.
- Data augmentation: Increasing Gaussian blur was evaluated across six kernel levels from 0.1 to 10.0 px for multiple SIDE methods.
- Data augmentation: Increasing Gaussian and salt-and-pepper noise levels were evaluated through qualitative depth predictions across several SIDE methods.