Source-linked AI summary
The Third Monocular Depth Estimation Challenge
Jaime Spencer, Fabio Tosi, Matteo Poggi, Ripudaman Singh Arora, Chris Russell, Simon Hadfield, Richard Bowden, GuangYuan Zhou, ZhengXin Li, Qiang Rao, YiPing Bao, Xiao Liu, Dohyeong Kim, Jinseong Kim, Myunghyun Kim, Mykola Lavreniuk, Rui Li, Qing Mao, Jiang Wu, Yu Zhu, Jinqiu Sun, Yanning Zhang, Suraj Patni, Aradhye Agarwal, Chetan Arora, Pihai Sun, Kui Jiang, Gang Wu, Jian Liu, Xianming Liu, Junjun Jiang, Xidan Zhang, Jianing Wei, Fangjun Wang, Zhiming Tan, Jiabao Wang, Albert Luginov, Muhammad Shahzad, Seyed Hosseini, Aleksander Trajcevski, James H. Elder
TL;DR
MDEC addresses the challenge of evaluating monocular depth estimation models on diverse environments where they cannot train or fine-tune. The paper reports the third edition's results on SYNS-Patches, where submissions could use any supervision. Participation and F-Score performance improved, while methods still struggled with fine structures and depth discontinuities.
Problem
MDE models have often been tailored to defined domains, leaving their ability to generalize to unseen environments under-evaluated despite the importance of diverse real-world settings.
Source: §1 · p. 1Method
Results
19 submissions beat the baseline on at least one image- or pointcloud-based metric, and PICO-MR achieved an F-score of 23.72 versus DJI&ZJU's 17.51.
Source: §1 · p. 2, §5.1 · p. 6Takeaways & Limitations
The authors report that foundational models such as Depth Anything had a higher impact on submitted results than the specific supervision type, while fine structures and discontinuities remain difficult.
Source: §6 · p. 8Methods struggle to estimate depth for non-Lambertian surfaces such as glass and transparent objects, which the authors link to a lack of accurate training supervision for those surfaces.
Source: §5.2 · p. 8
Abstract
from arXiv · showhide
This paper discusses the results of the third edition of the Monocular Depth Estimation Challenge (MDEC). The challenge focuses on zero-shot generalization to the challenging SYNS-Patches dataset, featuring complex scenes in natural and indoor settings. As with the previous edition, methods can use any form of supervision, i.e. supervised or self-supervised. The challenge received a total of 19 submissions outperforming the baseline on the test set: 10 among them submitted a report describing their approach, highlighting a diffused use of foundational models such as Depth Anything at the core of their method. The challenge winners drastically improved 3D F-Score performance, from 17.51% to 23.72%.
1. Introduction
Monocular depth estimation predicts scene-point distances from image pixels, but the lack of multi-image geometric priors makes it ill-posed. The third MDEC evaluates generalization to diverse unseen environments and drew more baseline-beating submissions than the previous edition.
1. Introduction
- Monocular depth estimation predicts the distance from the camera to scene points represented by image pixels, despite lacking geometric priors typically available from multiple images.The paper describes the problem as highly ill-posed.Source: §1 · p. 1
- MDEC evaluates whether monocular depth models generalize to unseen natural, agricultural, urban, and indoor environments without training or fine-tuning on its splits.The benchmark provides validation and test splits.Source: §1 · p. 1
- The third edition allowed any form of supervision, following the second edition's expansion from self-supervised methods to supervised methods.The first edition focused on self-supervised approaches; the second established methods that beat the baseline on both image-based and pointcloud metrics.Source: §1 · p. 1
- SYNS-Patches was selected for its varied scene types and dense, high-quality LiDAR ground truth, which the authors say can support a benchmark reflecting model capabilities with potentially fewer biases.Its scenes include urban, residential, industrial, agricultural, natural, and indoor settings.Source: §1 · pp. 1–2
- 19 submissions beat the baseline in at least one image- or pointcloud-based metric, up from 8 teams in the second edition.Ten of this year's submissions included reports, and seven of those outperformed the previous edition's winning team.Source: §1 · p. 2
2. Related Work
The related-work review covers supervised and self-supervised monocular depth estimation, methods for generalizing to unknown settings, and challenges involving adverse weather and transparent or specular surfaces.
2. Related Work
- Supervised MDE uses ground-truth depth, while self-supervised methods were developed to reduce dependence on costly ground-truth annotations.The review describes architectural, loss-function, and supervision advances across both approaches.Source: §2 · p. 2
Generalization and “In-the-Wild” MDE
- “In-the-wild” MDE targets methods that generalize across unknown settings, with strategies including varied training data, self-supervision, vision transformers, and camera-intrinsics inputs.The review notes that annotation accuracy can vary and acquiring new data sources remains challenging.Source: §2 · p. 2
Adverse Weather and Transparent/Specular Surfaces
- Existing MDE networks have struggled with adverse weather and transparent or mirror surfaces, prompting work on weather robustness and transparent-object datasets and methods.The review identifies Costanzino et al. [21] as the only work it describes as dedicated to transparent or mirror surfaces.Source: §2 · pp. 2–3
2. Related Work
- SYNS-Patches spans indoor depths up to 20m and outdoor depths up to 120m; natural and agricultural scenes contain more 20–80m depths, while urban scenes focus on 20–40m.The figure also shows image counts per category for the validation and test splits.Source: Figure 1 · p. 3
3. The Monocular Depth Estimation Challenge
The challenge tests monocular depth estimation on SYNS-Patches, with any supervision allowed and test ground truth withheld. Its evaluation combines image-, pointcloud-, and edge-based metrics, ranking submissions by reconstruction F-Score.
3. The Monocular Depth Estimation Challenge
- The development phase lasted four weeks on the SYNS-Patches validation split with an anonymized public leaderboard, while the three-week final phase used a private leaderboard and withheld ground truth.The authors say these final-phase choices discouraged test-set overfitting through repeated evaluations.Source: §3 · p. 3
- The challenge allowed any supervision and reported only submissions that beat the baseline on at least one image- or pointcloud-based metric on the overall dataset.Participants submitted up-to-scale disparity predictions; evaluation aligned predictions to ground truth using median depth and applied a 100-meter maximum depth threshold.Source: §3 · p. 3
- SYNS-Patches provides dense LiDAR ground truth with average coverage of 78.20%, and depth boundaries are derived using Canny edge detection on log-depth maps.Images were manually checked to remove dynamic-object artifacts.Source: §3 · p. 3, Figure 2 · p. 4
- Reconstruction F-Score is the leaderboard ranking metric, alongside image-based, pointcloud-based, and depth-boundary metrics.Image-based metrics compare predicted and ground-truth depth pixelwise; pointcloud metrics assess reconstructed pointclouds, while edge metrics are computed at depth boundaries.Source: §3 · pp. 3–4
4. Challenge Submissions
The paper describes submissions using varied architectures and supervision, including frozen or fine-tuned foundation models, self-supervised methods, and geometric depth propagation. The supplied submission descriptions cover the baseline and several teams' approaches.
4. Challenge Submissions
- The submission overview labels methods by supervision, including ground truth, proxy ground truth, Depth Anything pretraining, and monocular or stereo photometric support.Teams are numbered according to their rankings.Source: §4 · p. 4
Baseline – S
- The baseline uses a ConvNeXt-B encoder and Monodepth decoder, trained self-supervised with stereo photometric loss and edge-aware disparity smoothness.It was trained for 30 epochs on Kitti Eigen-Zhou at 192 × 640 resolution.Source: Baseline – S · p. 4
- PICO-MR fine-tunes its model on CityScapes with supervised proxy labels derived from stereo images and a loss combining SILog, SSIL, gradient, and RPNL terms.The described training uses 384×768 inputs and stops early at four epochs.Source: Baseline – S · p. 4
- RGA-Robot uses frozen Depth Anything for relative depth and trains an auxiliary NAFNet to convert predictions to metric depth with self-supervised image-reconstruction and smoothness losses.Training is conducted on Kitti Eigen-Zhou for four epochs, with only NAFNet trained.Source: Baseline – S · p. 4
Team 7: visioniitd – D
- ECoDepth conditions Stable Diffusion's UNet using Comprehensive Image Detail Embeddings and is trained supervised on NYUv2, KITTI, and virtual KITTI v2.Its depth regressor receives resized hierarchical decoder features; the reported training uses SILog loss with variance focus λ = 0.85.Source: Team 7: visioniitd – D · p. 5
4. Challenge Submissions
- The listed Depth Anything-based submissions include CLIP-based routing between indoor and outdoor models and a separately fine-tuned model trained on multiple indoor and outdoor datasets.HIT-AIIA fine-tunes separate Depth Anything instances on NYUv2 and KITTI; FRDC-SH fine-tunes on NYU-v2, 7Scenes, SUNRGBD, DIODE, KITTI, DDAD, and Argoverse.Source: Team 7: visioniitd – D · p. 5
Team 15: hyc123 – D
- hyc123 uses a Swin encoder with channel-wise self-attention decoder modules and ground-truth depth supervision; ReadingLS uses a 6.4M-parameter SwiftDepth model with self-supervision and offline distillation.ReadingLS trains on Kitti Eigen-Zhou and selected outdoor YouTube videos, and its teacher model was not trained on those datasets or SYNS-Patches.Source: Team 15: hyc123 – D · pp. 5–6
Team 19: Elder Lab – D
- Elder Lab estimates ground-plane depth from image geometry and semantic class, propagates it through scene segments, then completes the map using heat equations.Three kernel-regression models are trained on CityScapes and NYUv2.Source: Team 19: Elder Lab – D · p. 6
5. Results
Submissions were evaluated on SYNS-Patches’ test split with different supervision and prediction types, aligned by median depth scaling. PICO-MR achieved the highest reported F-score, while qualitative results show persistent challenges at thin structures, depth boundaries, and transparent surfaces.
5.1. Quantitative Results
- Leading teams widely adopted pre-trained Depth Anything, with Table 1 reporting results across the full SYNS-Patches test split.PICO-MR, RGA-Robot, EVP++, 3DCreators, HIT-AIIA, FRDC-SH, and ReadingLS used it as a backbone.Source: §5.1 · p. 6, Table 1 · p. 7
- 23.72 F-score put PICO-MR 72.9% above the baseline and 35.5% above prior winner DJI&ZJU’s 17.51 F-score.PICO-MR fine-tuned Depth Anything on Cityscapes for a few epochs using SILog, SSIL, Gradient, and Random Proposal Normalization losses.Source: §5.1 · p. 6, Table 1 · p. 7
- RGA-Robot achieved 22.79 F-score, 66.1% above baseline, and the best F-Edges result.Its method kept Depth Anything frozen and added NAFNet to convert relative predictions to metric depth, alongside self-supervised losses.Source: §5.1 · p. 6
- EVP++ achieved 20.87 F-score, 52.1% above baseline, and the lowest MAE, RMSE, and AbsRel.Its approach trained Depth Anything on indoor and outdoor data with parameters tailored to each scenario.Source: §5.1 · p. 6
- Other submissions also improved on baseline metrics: 3DCreators reached 20.42 F-score, while HIT-AIIA and FRDC-SH reached 17.83 and 17.81, respectively.Remaining teams beat baseline on at least one metric but did not surpass the previous edition’s winner; Elder Lab had 3.22 Acc-Edge against baseline despite a lower F-score.Source: §5.1 · p. 7
5.2. Qualitative Results
- Thin structures remain difficult for top methods, although visioniitd’s ECoDepth predictions recover them well; depth discontinuities still show halos.The qualitative comparison also suggests smooth regions may be harder to estimate than thin structures.Source: §5.2 · p. 7, Figure 3 · p. 8
- Qualitative anomalies include salt-and-pepper noise from hyc123, over-smoothing from Elder Lab, and grid-like artifacts from PICO-MR and RGA-Robot.The grid-like artifacts appear particularly in regions where the networks seem uncertain about depth estimates.Source: §5.2 · p. 7
- PICO-MR, EVP++, HIT-AIIA, and FRDC-SH estimate indoor scene structure strongly, but all methods still struggle with non-Lambertian surfaces and smooth depth boundaries.The paper links the transparent-surface limitation to a lack of accurate training supervision for those surfaces.Source: §5.2 · p. 8
6. Conclusions & Future Work
Across MDEC editions, performance improved markedly, with foundational models such as Depth Anything accompanying a major boost in submissions. Methods still struggle with fine structures and discontinuities, and the authors invite participation in future challenges.
6. Conclusions & Future Work
- Performance improved drastically across MDEC editions, highlighting real-world generalization as an active research area.Source: §6 · p. 8
- Foundational models such as Depth Anything [111] had a much higher impact on results than the supervision type, but methods still struggle with fine structures and discontinuities.These difficulties persist despite the massive amount of data used to train Depth Anything.Source: §6 · p. 8
- The authors invite researchers and practitioners to participate in future editions of MDEC.Source: §6 · p. 8