Source-linked AI summary
Lite-Mono: A Lightweight CNN and Transformer Architecture for Self-Supervised Monocular Depth Estimation
Ning Zhang, Francesco Nex, George Vosselman, Norman Kerle
TL;DR
Self-supervised monocular depth estimation needs lightweight models because existing high-performing architectures often rely on heavier backbones. Lite-Mono addresses this with a hybrid CNN-Transformer design using CDC and LGFI modules, and reports strong accuracy with reduced complexity and good inference speed.
Problem
Self-supervised monocular depth estimation seeks effective training without ground-truth depth, but heavier backbones and Transformer-based designs can increase model size or computational cost.
Method
Lite-Mono combines CDC modules for multi-scale local features with LGFI modules that encode long-range global information using channel-wise cross-covariance attention.
Results
Lite-Mono achieves state-of-the-art KITTI accuracy with the least trainable parameters, generalizes to Make3D, and shows a good trade-off between model complexity and inference speed.
Takeaways & Limitations
A lightweight hybrid CNN-Transformer architecture can model multi-scale local and long-range global contexts while maintaining a favorable complexity-speed trade-off.
Abstract
from arXiv · showhide
Self-supervised monocular depth estimation that does not require ground truth for training has attracted attention in recent years. It is of high interest to design lightweight but effective models so that they can be deployed on edge devices. Many existing architectures benefit from using heavier backbones at the expense of model sizes. This paper achieves comparable results with a lightweight architecture. Specifically, the efficient combination of CNNs and Transformers is investigated, and a hybrid architecture called Lite-Mono is presented. A Consecutive Dilated Convolutions (CDC) module and a Local-Global Features Interaction (LGFI) module are proposed. The former is used to extract rich multi-scale local features, and the latter takes advantage of the self-attention mechanism to encode long-range global information into the features. Experiments demonstrate that Lite-Mono outperforms Monodepth2 by a large margin in accuracy, with about 80% fewer trainable parameters.
1. Introduction
Lite-Mono targets self-supervised monocular depth estimation with a lightweight hybrid CNN-Transformer architecture. It combines multi-scale local feature extraction with global-context modeling and reports strong accuracy, efficiency, and inference-speed results.
- Self-supervised monocular training is preferred because collecting stereo data requires complicated configurations and data processing.
- CNNs struggle to capture long-range global information, while deeper or more complicated architectures increase model size and Transformer attention can be computationally expensive.
- Lite-Mono combines a lightweight CNN-Transformer architecture with CDC modules for enhanced multi-scale local features and LGFI modules for global-context encoding.
- Lite-Mono is proposed as a lightweight architecture whose effectiveness is demonstrated with respect to model size and FLOPs.
- Lite-Mono achieves state-of-the-art accuracy on KITTI with the least trainable parameters, while its generalization is validated on Make3D.
- Inference tests on NVIDIA TITAN Xp and Jetson Xavier demonstrate a trade-off between model complexity and inference speed.
2. Related work
Deep-learning monocular depth estimation includes supervised and self-supervised approaches, with architecture design increasingly used to improve feature extraction and accuracy. Related methods explore multi-scale features, attention, and CNN-Transformer encoders, while computational cost remains a concern.
- Single-image depth estimation is ill-posed because one 2D image may correspond to multiple 3D scenes at different scales.
- Supervised depth estimation: Supervised methods use ground-truth depth maps to learn the relationship between RGB values and depth, often combining global coarse and local fine predictions.
- Self-supervised depth estimation: Self-supervised methods avoid requiring ground truth by deriving supervision from stereo pairs or monocular video frames.
- Network architectures: Prior architecture work uses channel-wise attention, feature fusion, and feature modulation to capture long-range information, enhance local features, or learn multi-scale features.
- Advanced architectures: MonoViT achieves state-of-the-art accuracy with MPViT, but its multiple parallel blocks slow inference and its architecture has a speed gap relative to CNN-only methods.
3. The proposed framework: Lite-Mono
Lite-Mono combines a lightweight CNN-Transformer encoder with a conventional depth-decoder and self-supervised image-reconstruction objective. CDC modules capture multi-scale local context, while LGFI modules encode global information with reduced attention complexity.
- Overall architecture: Lite-Mono uses an encoder-decoder DepthNet and PoseNet to estimate multi-scale inverse depth, camera motion, and reconstructed target images.The PoseNet uses a pre-trained ResNet18 pose encoder and predicts 6-DoF relative pose between adjacent images.
- Depth encoder: LGFI modules encode global context with cross-covariance attention computed across feature channels rather than spatial positions.This reduces memory complexity from O(hN^2+Nd) to O(d^2/h+Nd) and time complexity from O(N^2d) to O(Nd^2/h).
- Depth encoder: Table 1 defines four depth-encoder variants by CDC kernel output channels, repetition count, and dilation rate.The notation [3 × 3, C] × N denotes a 3 × 3 CDC block producing C channels and repeated N times.
- Depth encoder: The four-stage depth encoder downsamples features while concatenating pooled input images and previous-stage outputs to preserve spatial information and cross-stage correlations.Feature maps progress through dimensions including H/2 × W/2 × C1, H/4 × W/4 × C2, and H/16 × W/16 × C4.
- Depth encoder: CDC modules extract multi-scale local features through consecutive dilated convolutions inserted across encoder stages.Dilated convolutions enlarge the receptive field while preserving output size, and consecutive stacking expands the observed area without extra training parameters.
- Depth decoder: The depth decoder uses bilinear up-sampling and convolutional feature concatenation to predict inverse-depth maps at full, 1/2, and 1/4 resolutions.It is adapted from prior work rather than using a more complicated up-sampling method or additional attention modules.
- Self-supervised learning: Self-supervised training minimizes image-reconstruction and edge-aware smoothness losses instead of using ground-truth depth.The reconstruction loss uses synthesized target images generated from source images, estimated pose, predicted depth, and camera intrinsics; minimum photometric loss handles out-of-view and occluded pixels, while a binary mask removes moving pixels.
4. Experiments
Experiments evaluate Lite-Mono on KITTI and Make3D, then examine its efficiency, qualitative behavior, and architectural components. Results show strong accuracy, generalization, speed, and ablation benefits from the proposed design choices.
- KITTI results: Lite-Mono is compared with representative methods on KITTI using the Eigen split and standardized evaluation metrics.The benchmark uses resized 640 × 192 inputs and reports seven commonly used depth-estimation metrics.
- KITTI results: Lite-Mono beats all compared methods except MonoViT-tiny and is the smallest model at 3.1M parameters.It greatly exceeds Monodepth2 with a ResNet18 backbone, whose model size is about five times larger, and outperforms the ResNet50 version and MonoFormer across the reported metrics.
- Make3D results: Lite-Mono performs best on Make3D when trained on KITTI and inferred directly without fine-tuning.Qualitative results indicate that its feature-extraction modules model local and global contexts and perceive objects with different sizes.
- Complexity and speed evaluation: Lite-Mono achieves a favorable balance between model complexity and inference speed on NVIDIA TITAN Xp and Jetson Xavier.Lite-Mono-tiny is faster and more accurate than Monodepth2, while the proposed models infer quickly on Jetson Xavier for edge-device use.
- Ablation study on model architectures: Removing LGFI blocks decreases accuracy while reducing model size by 0.4M, supporting their role in encoding long-range global contexts.LGFI compensates for CNNs’ local receptive fields through global-context encoding.
- Ablation study on model architectures: Removing dilated convolutions, pooled concatenations, or cross-stage connections also decreases accuracy, while CDC adds multi-scale locality without extra trainable parameters.The CDC ablation produces a larger accuracy drop than removing LGFI blocks, and cross-stage connections promote feature propagation and information fusion.
5. Conclusions
Lite-Mono is a lightweight self-supervised monocular depth estimation architecture combining CNNs and Transformers to model multi-scale local features and long-range global contexts. Experiments show strong KITTI performance, Make3D generalization, and a favorable complexity–inference-speed trade-off.
- Lite-Mono combines CNNs and Transformers to model multi-scale enhanced local features and long-range global contexts.
- Optimized dilation rates in CDC blocks and LGFI modules help Lite-Mono perceive objects at different scales, including challenging nearby moving objects.
- Lite-Mono demonstrates superior performance on the KITTI dataset and validated generalization ability on Make3D.
- The model achieves a good trade-off between model complexity and inference speed.