PaperlayerView exact revision on arXiv

Source-linked AI summary

Deconstructing Self-Supervised Monocular Reconstruction: The Design Decisions that Matter

Jaime Spencer, Chris Russell, Simon Hadfield, Richard Bowden

arXiv:2208.01489v4cs.CVcs.CGcs.LG

TL;DR

Self-supervised monocular depth research lacks informative, reliable benchmarks, complicating comparisons across established design choices. This paper reimplements and evaluates 16 recent methods in a modular benchmark, finding that many contributions do not improve over a legacy stereo baseline and that backbone modernization alone can yield 25% gains.

  • Problem

    Erroneous evaluation procedures, including inaccurate ground-truth depth maps in the commonly used KITTI Eigen split, leave the field without informative benchmarks for comparing monocular depth methods.

    Source: §3 · p. 5

  • Method

    The authors reimplement 16 recent methods in a modular framework for fair, comparable evaluation and introduce SYNS-Patches, a dense-depth dataset spanning urban and natural scenes.

    Source: §1 · pp. 2–3

  • Results

    On a level playing field, many contributions do not improve over the legacy stereo baseline, while modernizing legacy backbones alone yields 25% performance gains.

    Source: §1 · p. 2, §6 · p. 15

  • Takeaways & Limitations

    SYNS-Patches extends evaluation beyond automotive scenes, and its dense ground truth supports informative metrics in complex regions such as depth boundaries.

    Source: §1 · pp. 2–3

    Video-trained methods require a relative-pose network during training for view synthesis and photometric loss, but it can be discarded afterward.

    Source: §4 · p. 7

Abstract

from arXiv · show

This paper presents an open and comprehensive framework to systematically evaluate state-of-the-art contributions to self-supervised monocular depth estimation. This includes pretraining, backbone, architectural design choices and loss functions. Many papers in this field claim novelty in either architecture design or loss formulation. However, simply updating the backbone of historical systems results in relative improvements of 25%, allowing them to outperform the majority of existing systems. A systematic evaluation of papers in this field was not straightforward. The need to compare like-with-like in previous papers means that longstanding errors in the evaluation protocol are ubiquitous in the field. It is likely that many papers were not only optimized for particular datasets, but also for errors in the data and evaluation criteria. To aid future research in this area, we release a modular codebase (https://github.com/jspenmar/monodepth_benchmark), allowing for easy evaluation of alternate design decisions against corrected data and evaluation criteria. We re-implement, validate and re-evaluate 16 state-of-the-art contributions and introduce a new dataset (SYNS-Patches) containing dense outdoor depth maps in a variety of both natural and urban scenes. This allows for the computation of informative metrics in complex regions such as depth boundaries.

1 Introduction

This paper re-evaluates self-supervised monocular depth methods under comparable design and evaluation choices, finding that backbone modernization can produce 25% gains and let legacy formulations outperform many recent methods. It also provides a modular codebase, corrected benchmark evaluation, and SYNS-Patches, a dataset spanning urban and natural scenes.

1 Introduction

  • The paper re-implements popular methods in a modular framework to study component impacts and train approaches fairly and comparably.The codebase is intended to enable direct like-with-like comparisons and better research practices.Source: §1 · p. 2
  • 25% performance gains result from modernizing the backbone, and legacy formulations can outperform many recent methods under common design choices.The paper reports that improvements from individual contributions are often smaller than those claimed in original publications.Source: §1 · p. 2, Figure 1 · p. 2
  • The re-evaluation uses corrected ground truth on the existing Kitti benchmark to address errors propagated for compatibility with earlier results.The paper identifies inaccurate ground-truth depth, particularly around object boundaries, as a key benchmark problem.Source: §1 · p. 2
  • SYNS-Patches contains urban and natural scenes, with dense ground truth supporting informative metrics in complex regions such as depth boundaries.The dataset is intended to assess generality beyond the automotive domain emphasized by most papers.Source: §1 · pp. 2–3

2 Related Work

Prior monocular-depth methods use different training and evaluation choices, complicating direct comparisons. The literature develops stereo and monocular view-synthesis approaches, while benchmark ground-truth errors and dataset depth distributions shape evaluation.

2.1 Stereo

  • Stereo methods progress from view-synthesis reconstruction and smoothness losses to differentiable synthesis, stereo consistency, and specialized architectures.Later approaches include sub-pixel convolution, stereo blending, and disparity-volume or occlusion-aware designs.Source: §2.1 · p. 3
  • Other stereo-related approaches add proxy depth supervision or stereo-matching refinement, using sources such as sparse LiDAR, SLAM, and hand-crafted matching.DepthHints applies proxy regression only where the hint yields a lower photometric reconstruction loss.Source: §2.1 · p. 3

2.2 Monocular

  • Monocular methods develop from VO and uncertainty handling to photometric robustness, geometric consistency, uncertainty estimation, and higher-resolution architectures.The approaches described include learned feature losses, geometric constraints, discrete disparity volumes, and attention-based encoder-decoder designs.Source: §2.2 · p. 4
  • Methods differ in epochs, pretraining data, backbones, post-processing, and resolution, so the paper evaluates recent contributions comparably and studies less-publicized component changes.The authors ask how much reported improvement comes from core contributions versus these other design choices.Source: §2.2 · p. 4
  • The original Kitti ground truth used by prior benchmarks is inaccurate at object boundaries, and SYNS-Patches includes more varied depths due to indoor scenes.Evaluation clamps maximum depth to 100 meters across the datasets.Source: Figure 2 · p. 5, Figure 3 · p. 5

3 Benchmark Datasets

The benchmark addresses persistent errors in the common Kitti Eigen evaluation and supplements corrected Kitti evaluation with SYNS-Patches, a dense dataset spanning varied scenes.

3 Benchmark Datasets

  • Errors in the Kitti Eigen split include inaccurate ground truth, viewpoint and occlusion artifacts, omitted camera-frame transformation, and incorrectly computed Squared Relative error.These errors persisted as researchers copied results to preserve like-with-like comparisons.Source: §3 · p. 5

3.1 Kitti Eigen-Benchmark

  • The Kitti Eigen-Benchmark corrects ground truth using accumulated LiDAR and occlusion checks, while evaluation avoids saturated or incorrect image metrics and also reports pointcloud metrics.The corrected split contains 652 of the original 697 images.Source: §3.1 · pp. 5–6

3.2 SYNS-Patches

  • SYNS-Patches extends evaluation beyond automotive scenes with dense depth maps that support metrics focused on depth boundaries.Its 1,175-image test set has ground-truth depth for 78.30% of pixels; evaluation also includes boundary-only and edge-based metrics.Source: §3.2 · p. 6

4 The Design Decisions That Matter

This section evaluates practical framework choices through a shared implementation and updated evaluation protocol. Its results highlight backbone selection and the ground-truth dependence of regularization performance.

4 The Design Decisions That Matter

  • The study targets framework components rarely claimed as contributions; evaluation of the monocular-training pose network is outside its scope.The pose network is used during training for view synthesis and photometric loss, then discarded for depth evaluation.Source: §4 · p. 7

4.1 Implementation details

  • A 700-image KEZ validation split with corrected ground truth is used for ablations to reduce test-set overfitting.Models use a pretrained ConvNeXt-T depth backbone and are trained for 30 epochs; variants use three random seeds and report mean performance.Source: §4.1 · p. 7

4.2 Backbone Architecture & Pretraining

  • ∼20% improvement over ResNet-18 from scratch is followed by an additional 15% from ConvNeXt or HRNet; the remaining comparisons use ConvNeXt-B.Figure 5 reports relative improvements in F-Score, MAE, LogSI and AbsRel.Source: Figure 5 · p. 8, §4.2 · p. 9
  • ConvNeXt variants outperform other backbones, HRNet-W64 follows, and HRNet-W18 is the strongest option among backbones under 20 MParams.Pretrained weights improve ResNet variants; self- or weakly supervised weights further improve ResNeXt variants.Source: §4.2 · p. 8, Table 2 · p. 8

4.3 Depth Regularization

  • On corrected ground truth, omitting regularization performs best, while smoothness regularization slightly improves results on the outdated Eigen split.Adding occlusion regularization alongside smoothness further decreases performance on corrected ground truth; the authors attribute the Eigen result to oversmoothing inaccurate boundaries.Source: §4.3 · p. 9, Table 3 · p. 9
  • The State-of-the-Art summary lists evaluated settings, which differ from original implementations because changes were made for comparability and performance.Source: Table 4 · p. 10

5 Results

The results compare updated baselines and recent methods under consistent evaluation, then test transfer to SYNS-Patches. Across these evaluations, modern design choices improve some metrics, while monocular methods and predictions beyond the automotive domain remain limited.

5.1 Evaluation details

  • Evaluation uses corrected Kitti ground truth, a 100-meter depth cap, and no border cropping or stereo-blending post-processing.Stereo-supervised methods use fixed scaling; monocular methods use per-image median scaling.Source: §5.1 · pp. 10–11

5.2 Kitti Eigen

  • The updated baselines outperform their original published counterparts on Kitti Eigen; δ < 1.25 accuracy improves by a raw 10% for both mono and stereo baselines.AbsRel error decreases by 5%.Source: Table 5 · p. 11, §5.2 · p. 12

5.3 Kitti Eigen-Benchmark

  • On Kitti Eigen-Benchmark, Garg et al. (2016) remains among the top performers across metrics, while recent contributions improve image-based metrics but not 3-D pointcloud reconstruction.Monocular approaches underperform stereo methods despite median scaling; adding Monodepth2 (Godard et al., 2019) contributions significantly improves over SfM-Learner.Source: §5.3 · p. 12, Table 6 · p. 13

5.4 SYNS-Patches

  • On SYNS-Patches, performance drops outside the automotive domain; Garg et al. (2016) remains among the top 3-D pointcloud performers, while predicted edges are accurate but incomplete.Edge-Acc is <5 px, but Edge-Comp is >25 px.Source: Table 7 · p. 14, §5.4 · pp. 14–15
  • On SYNS-Patches, Monodepth2 and successor methods improve thin-structure predictions, but natural and woodland scenes remain challenging across methods.The evaluation uses Kitti-trained models without retraining or fine-tuning.Source: §5.4 · pp. 14–15, Figure 8 · p. 15

6 Conclusion

The paper concludes that consistent re-evaluation changes the picture of reported progress in self-supervised monocular depth estimation. It identifies generalization and depth discontinuities as key remaining challenges.

6 Conclusion

  • Under comparable benchmarking, many contributions do not improve on the legacy stereo baseline, and reported gains are often much smaller than originally claimed.Unclaimed design changes can produce improvements equal to or greater than claimed contributions.Source: §6 · p. 15
  • The paper identifies generalization beyond automotive data and accuracy on thin objects and depth discontinuities as future research directions.It points to indoor, natural, and synthetic data as possible additional sources and makes its codebase publicly available.Source: §6 · p. 15

A Monocular Depth Overview

The paper limits its benchmark to methods that estimate depth, optionally with visual odometry.

A Monocular Depth Overview

  • The benchmark excludes methods that additionally learn tasks such as semantic segmentation, optical flow, or surface normals.A field review and practical implementation details are provided in supplementary material.Source: A Monocular Depth Overview · p. 20

A.1 DepthNet

DepthNet predicts dense depth from a single image using an encoder-decoder network. The section also describes optional output and training components used by different frameworks.

A.1 DepthNet

  • DepthNet is a fully convolutional encoder-decoder with skip connections that produces multi-scale dense depth predictions.Any multi-scale feature-producing architecture can serve as the encoder; the paper reports ConvNeXt as best overall in its ablation.Source: §A.1 · p. 20
  • The DispNet-inspired decoder uses five upsampling stages with skip connections and intermediate depth predictions for training supervision.In practice it predicts inverse depth (disparity) via a convolution and sigmoid activation.Source: §A.1 · p. 20
  • Virtual stereo adds left- and right-view disparity outputs so training can use either stereo viewpoint without an additional decoder.The selected virtual disparity corresponds to the opposite view from the input image.Source: §A.1 · p. 20
  • Monocular training can produce artifacts for independently moving objects, including infinite-depth predictions when their motion resembles the camera's.Predictive masks downweight unreliable photometric regions; recent implementations predict masks for support images using an additional DepthNet decoder.Source: §A.1 · pp. 20–21

A.2 PoseNet

PoseNet supports interchangeable pretrained encoders, while the pose network is typically kept as a pretrained ResNet-18 for efficiency. Its decoder regresses a six-dimensional motion representation that is converted into a camera transform.

A.2 PoseNet

  • PoseNet is typically restricted to a pretrained ResNet-18 for efficiency, although its encoder can be replaced with another pretrained architecture.The encoder uses pretrained models from Wightman (2019).Source: §A.2 · p. 21
  • For each support frame, VO is predicted independently from the target and support images concatenated channel-wise, with image order swapped when needed to enforce forward motion.The first convolution is modified to accommodate the concatenated input while duplicating and scaling pretrained weights.Source: §A.2 · p. 21
  • The decoder regresses translation and axis-angle rotation as a six-dimensional output, then converts the rotation vector into a 3 × 3 transform matrix.Its encoder features pass through a 1 × 1 convolution and two Conv-ReLU blocks; predictions are scaled by 0.01 for stability and convergence.Source: §A.2 · p. 21

A.3 View Synthesis

View synthesis uses predicted scene depth and camera locations to reproject target-image points into support frames. Bilinear sampling at the resulting correspondences produces support views aligned with the target.

A.3 View Synthesis

  • Given scene depth and camera locations, target-image points can be reprojected onto available support frames to enable view synthesis.This is described as the core component of self-supervised monocular depth approaches.Source: §A.3 · p. 21
  • Reprojection uses shared camera intrinsics, predicted target-point depth, and predicted camera motion or a known stereo baseline to determine corresponding support-frame pixels.The correspondences are used to synthesize a support frame aligned to the target image.Source: §A.3 · p. 21
  • Bilinear sampling at the reprojected pixels produces the synthesized support image aligned with the target.The sampling operation is associated with Spatial Transformer Networks (Jaderberg et al., 2015).Source: §A.3 · p. 21

A.4 Losses

The loss designs combine photometric comparison of target images with synthesized views and, optionally, feature-based reconstruction, proxy depth regression, and virtual-stereo disparity consistency. They also use smoothness and explainability-mask regularization to shape disparity and limit the effects of ambiguous or potentially uninformative correspondences.

A.4 Losses

  • Photometric error compares the target frame with a synthesized support view, typically combining L1 and SSIM losses for greater robustness.The SSIM term is weighted alongside L1 in the photometric loss.Source: §A.4 · pp. 21–22
  • An explainability mask downweights photometric loss, while an uncertainty mask weights it by exp(-M_U) and adds the uncertainty value.The masks are applied elementwise to the photometric loss.Source: §A.4 · p. 22
  • Reconstruction loss averages photometric errors across support frames and pixels, while Monodepth2 (Godard et al., 2019) selects the minimum support-frame error per pixel to avoid occlusion-related errors.The minimum is intended to preserve correct pixel-wise correspondences rather than average in incorrect errors caused by occlusions.Source: §A.4 · p. 22
  • Monodepth2 (Godard et al., 2019) also masks static pixels when the original support frame has lower photometric error than the synthesized view.The comparison is made using the minimum photometric error across support frames.Source: §A.4 · p. 22
  • Feature-based reconstruction can address ambiguous image correspondences in low-light scenes, using reprojection correspondences and low-level features from the depth encoder instead of an additional pretrained dense feature network.The feature loss can replace the L1 + SSIM photometric loss with L2 distance between feature embeddings.Source: §A.4 · p. 22
  • Proxy depth regression supplements self-supervised losses to prevent local minima, using L1 for disparity and berHu for depth with a per-batch threshold δ = 0.2 max L1.The margin threshold is set adaptively per batch; D denotes proxy ground-truth depth and the network prediction is compared against it.Source: §A.4 · pp. 22–23
  • DepthHints (Watson et al., 2019) recommends berHu for ground-truth LiDAR regression but Log-L1, log(1 + L1), for hand-crafted stereo disparities.Source: §A.4 · p. 23
  • Virtual stereo consistency follows Monodepth (Godard et al., 2017), applying view synthesis to disparities and L1-comparing target disparities with warped virtual-stereo predictions.Source: §A.4 · p. 23

A.5 Regularization

  • Garg et al. (2016) penalize all gradients of mean-normalized disparity, while Godard et al. (2017) weight gradients by image edges to permit disparity discontinuities.The gradients are computed independently in the x and y directions; edge-aware weighting uses the exponential of the negative image-gradient magnitude.Source: §A.5 · p. 23
  • Rui et al. (2018) use edge-aware second-order smoothness to encourage gradual gradient changes, and the benchmark optionally applies Gaussian smoothing before computing gradients.Source: §A.5 · p. 23
  • Rui et al. (2018) add an occlusion regularizer that penalizes summed disparities to sharpen occlusion boundaries while favoring background disparities.Source: §A.5 · p. 23
  • Explainability masks downweight photometric loss where correspondences may be incorrect or uninformative, including independently moving objects; binary cross-entropy regularization pushes mask values toward 1 (Zhou et al., 2017).Source: §A.5 · p. 24

B Benchmark Datasets

The supplementary benchmark-datasets section provides dataset-specific details, including KITTI split metrics and the SYNS-Patches creation procedure.

B Benchmark Datasets

B.1 Kitti Eigen

The KE split defines image-depth error metrics, but the authors recommend avoiding this evaluation because its ground truth is inaccurate. The section also identifies an error in the SqRel computation.

B.1 Kitti Eigen

  • The authors recommend avoiding KE evaluation because its ground-truth depth is inaccurate.They provide its metrics for comparison with previous publications during the transition to the proposed benchmark.Source: §B.1 · p. 24
  • AbsRel measures mean relative error (%); RMSE measures root mean square error in meters; and LogRMSE measures root mean square log error in log-meters.Source: §B.1 · p. 24, §B.2 · p. 24
  • SqRel is described as mean relative square error, but its stated computation omits the squared denominator.The authors explicitly flag this as an incorrect computation.Source: §B.1 · p. 24

B.2 Kitti Eigen-Benchmark

The benchmark replaces the inaccurate KE ground truth with corrected ground truth and adds 3-D reconstruction and depth-boundary metrics. SYNS-Patches supplies diverse scenes and dense depth maps for evaluating generalization and complex regions.

B.2 Kitti Eigen-Benchmark

  • The proposed replacement for KE uses corrected ground truth from Uhrig et al. (2018), alongside image-based and pointcloud-based evaluation metrics.The 3-D metrics assess reconstructed pointcloud fidelity; the section also specifies a 10 cm threshold for a correctly reconstructed point.Source: §B.2 · pp. 24–26
  • KE metrics include depth errors such as LogMAE and LogRMSE, plus scale-invariant log error, which rewards predictions consistently wrong in the same direction.The section also lists AbsRel, SqRel, inverse-depth errors, and threshold accuracy.Source: §B.2 · p. 25
  • Chamfer distance, precision, and recall assess pointcloud agreement; F-Score and IoU summarize precision-recall and volumetric reconstruction quality.The ground-truth and predicted pointclouds comprise 3-D points; precision and recall use a distance threshold.Source: §B.2 · pp. 25–27

B.3 SYNS-Patches

  • SYNS-Patches samples 18 eye-level views per scene from panoramic data and retains 1,175 images after removing artifacts and empty or uniform depth maps.The original scanning process can introduce distortions from moving objects and mismatches between independently captured images and depth scans.Source: §B.3 · p. 27
  • SYNS-Patches derives depth boundaries from Canny edges, masks invalid-depth artifacts, and finds log-depth with smoothing sigma 1 gives the best qualitative results.Figure 9 compares boundary extraction across raw, log, and inverse depth maps with increasing Gaussian blur.Source: Figure 9 · p. 26, §B.3 · p. 27
  • SYNS-Patches combines image- and pointcloud-based metrics with optional boundary-only evaluation and edge accuracy and completeness measures.The edge measures use distances to the nearest ground-truth or predicted edge, with the Euclidean Distance Transform truncated at 10.Source: §B.3 · p. 27
Loading 2208.01489v4…