Source-linked AI summary
MonoViT: Self-Supervised Monocular Depth Estimation with a Vision Transformer
Chaoqiang Zhao, Youmin Zhang, Matteo Poggi, Fabio Tosi, Xianda Guo, Zheng Zhu, Guan Huang, Yang Tang, Stefano Mattoccia
TL;DR
Self-supervised monocular depth estimation avoids hard-to-source depth labels, but CNNs’ limited receptive fields restrict reasoning to local context. MonoViT combines convolutions with Transformer blocks for joint local and global modeling, achieving superior KITTI accuracy and generalization on Make3D and DrivingStereo.
Problem
CNN-based self-supervised monocular depth estimation is constrained by limited receptive fields that restrict reasoning to local context.
Method
MonoViT combines convolutional layers with Transformer blocks to model local objects and global relationships within the same image.
Results
MonoViT vastly and consistently outperforms state-of-the-art architectures on KITTI and achieves better generalization on Make3D and DrivingStereo.
Takeaways & Limitations
Joint local-global interaction preserves small foreground objects even at the lowest resolution.
Takeaways & Limitations
View-reconstruction losses remain limited by occlusions, dynamic objects, and photometric changes.
Abstract
from arXiv · showhide
Self-supervised monocular depth estimation is an attractive solution that does not require hard-to-source depth labels for training. Convolutional neural networks (CNNs) have recently achieved great success in this task. However, their limited receptive field constrains existing network architectures to reason only locally, dampening the effectiveness of the self-supervised paradigm. In the light of the recent successes achieved by Vision Transformers (ViTs), we propose MonoViT, a brand-new framework combining the global reasoning enabled by ViT models with the flexibility of self-supervised monocular depth estimation. By combining plain convolutions with Transformer blocks, our model can reason locally and globally, yielding depth prediction at a higher level of detail and accuracy, allowing MonoViT to achieve state-of-the-art performance on the established KITTI dataset. Moreover, MonoViT proves its superior generalization capacities on other datasets such as Make3D and DrivingStereo.
1 East China University of Science and Technology 2 University of Bologna 3 PhiGent Robotics
Figure 1 contrasts limited-receptive-field methods with MonoViT, which achieves superior depth-estimation results and preserves finer details.
- MonoViT achieves superior depth-estimation results compared with existing solutions such as HR-Depth.The comparison highlights finer details, including the car and cyclist.
1. Introduction
Monocular depth estimation avoids costly depth labels and supports flexible deployment, but CNNs’ limited receptive fields constrain global reasoning. MonoViT combines convolutional and Transformer components to address this limitation and reports superior accuracy and generalization across datasets.
- Estimating depth from a single image enables monocular depth-sensing deployment without multiple cameras.Standard images also offer higher resolution, lower hardware costs, and potentially unconstrained working range.
- Self-supervised methods replace costly depth-label supervision with image-reprojection signals from constrained monocular videos.View-reconstruction losses nevertheless suffer from occlusions, dynamic objects, and photometric changes.
- CNNs’ reduced receptive fields limit global context modeling, creating a bottleneck for dense estimation accuracy and generalization.The introduction identifies this as a shared shortcoming of existing self-supervised models.
- MonoViT combines convolutional layers with Transformer blocks to model local objects and global relationships within the same image.The paper presents this design as addressing the limited perceptive fields of CNN encoders and producing finer-grained predictions.
- MonoViT reports constantly superior accuracy on KITTI and superior generalization on Make3D and DrivingStereo.The comparisons use the standard KITTI split and evaluate cross-dataset generalization on Make3D and DrivingStereo.
2. Related works
Related work covers self-supervised monocular depth estimation, architectural advances, and the emergence of Transformers for global-context modeling. Existing Transformer depth-estimation methods discussed here focus on supervised settings, while the figure material motivates attention to small foreground objects.
- Monocular Depth Estimation: Self-supervised monocular depth estimation frames depth prediction as view synthesis between adjacent views and avoids ground-truth depth annotations.Training can use stereo images, monocular videos, or both, with photometric differences providing supervision.
- Monocular Depth Estimation: Unlabeled video is a flexible alternative to stereo but requires learning camera poses alongside depth.The related work also describes additional losses and constraints used to advance this research line.
- Transformers in Depth Estimation: Figure 2 compares RGB input, multiscale disparity predictions, and attention maps across methods for a difficult small car.The comparison examines whether each method preserves the car across resolutions.
- Network architectures: Backbone and decoder architectures substantially affect monocular depth performance, motivating ResNet, PackNet, attention modules, and multiscale feature fusion.The surveyed approaches target feature compression, decoding, and feature integration.
- Transformers in Depth Estimation: Vision Transformers gained attention for modeling global context across image classification, detection, and segmentation tasks.The related-work discussion connects this capability to Transformer-based monocular depth estimation.
- Transformers in Depth Estimation: The cited Transformer-based monocular depth methods focus on supervised estimation rather than self-supervised training.This distinguishes the prior work discussed from MonoViT’s target setting.
3. Proposed framework
MonoViT combines convolutional layers and Transformer blocks in a self-supervised depth-and-pose framework, jointly modeling local detail and global context. Its encoder-decoder design predicts dense inverse depth from monocular videos using reprojection and smoothness losses.
- Framework: MonoViT uses a DepthNet and a lightweight PoseNet, trained through image reconstruction losses for depth prediction and adjacent-frame pose estimation.DepthNet predicts each input image's depth, while PoseNet estimates a 6 DoF relative pose from concatenated adjacent frames.
- Depth encoder: The depth encoder combines convolutional paths with Transformer blocks to model local object information and global relationships within an image.Joint CNN & Transformer Layers use multi-scale convolutional patch embeddings, factorized self-attention, and feature fusion between local and global representations.
- Depth encoder: Integrating convolutions with Transformers helps preserve small foreground objects at low resolution by capturing pixel interactions and relative object positions.The architecture is intended to avoid collapsing small foreground objects into texture-like background representations.
- Depth decoder: The depth decoder uses cross-layer and cross-scale connections, spatial and channel attention, and four convolutional heads to progressively predict disparity at multiple resolutions.Multi-scale features are decoded while attention enhances feature fusion; disparity is inverse depth.
- Self-supervised learning: Training reconstructs a target image from a nearby source view using predicted depth, camera intrinsics, and pose, combining reprojection and edge-aware smoothness losses across scales.The reprojection objective uses structural similarity and intensity differences, while an auto-mask filters static frames and camera-motion-consistent objects.
4. Experiments
MonoViT is evaluated on KITTI and cross-dataset benchmarks using standard protocols, where it consistently surpasses existing self-supervised methods and generalizes across environments and weather conditions. Ablations further examine the contributions of its CNN–Transformer design and deployment efficiency.
- Experimental protocol: Evaluation uses seven standard depth metrics on the KITTI Eigen split, with raw LiDAR and improved ground-truth test sets.The KITTI data comprise 39,810 training monocular triplets and 4,424 validation triplets, with 697 raw-LiDAR and 652 improved-label test images.
- KITTI evaluation: MonoViT significantly outperforms existing state-of-the-art methods across input resolutions, training settings, and all reported KITTI metrics.The comparison includes monocular and binocular video training, with additional evaluation using improved ground-truth labels.
- Cross-dataset generalization: MonoViT consistently outperforms CNN competitors on DrivingStereo across foggy, cloudy, rainy, and sunny conditions without retraining or fine-tuning.The reported performance margin is higher than those observed on KITTI and Make3D.
- Qualitative results: Qualitative comparisons show finer-grained foreground and background depth estimates on KITTI, Make3D, and DrivingStereo.The paper attributes this behavior to MonoViT’s global receptive field and its ability to model object and scene relationships.
- Ablation study: Ablation results indicate that the MPViT backbone and its CNN path, Transformer path, and attention blocks each contribute to the architecture’s performance.The study compares MPViT variants with SwinT-tiny and PVT while reporting parameter counts and FPS.
5. Conclusion
MonoViT combines convolutions and Transformer blocks for self-supervised monocular depth estimation, modeling local and global image context jointly. It outperforms state-of-the-art methods on KITTI and generalizes better on Make3D and DrivingStereo.
- MonoViT combines convolutions and Transformer blocks to model local and global image context jointly.
- MonoViT vastly and consistently outperforms state-of-the-art methods on the KITTI dataset.
- MonoViT achieves better generalization performance than state-of-the-art self-supervised depth-estimation architectures on Make3D and DrivingStereo.