Source-linked AI summary
The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth
Jamie Watson, Oisin Mac Aodha, Victor Prisacariu, Gabriel Brostow, Michael Firman
TL;DR
Many self-supervised monocular depth networks ignore video frames available at test time, while existing sequence-aware methods either use expensive refinement or indirectly exploit geometry. ManyDepth introduces a self-supervised adaptive cost-volume system that handles unreliable matches and static cameras, achieving state-of-the-art results on KITTI and Cityscapes with efficient inference.
Problem
Self-supervised monocular depth networks typically do not exploit sequence information available at test time, while existing sequence-aware methods have computational or geometric-reasoning limitations.
Method
ManyDepth combines self-supervised reprojection training with an adaptive multi-view cost volume, reliability handling for cost-volume overfitting, and augmentation for static cameras and single-frame inputs.
Results
ManyDepth achieves state-of-the-art results on KITTI and Cityscapes and outperforms previously published self-supervised methods on most metrics.
Takeaways & Limitations
ManyDepth provides one online model that predicts depth from a single image or multiple images, combining multi-frame and monocular capabilities while improving robustness to moving objects and static cameras.
Takeaways & Limitations
The baseline cost-volume model can produce severe artefacts, including large holes on moving objects, when the cost volume is unreliable.
Abstract
from arXiv · showhide
Self-supervised monocular depth estimation networks are trained to predict scene depth using nearby frames as a supervision signal during training. However, for many applications, sequence information in the form of video frames is also available at test time. The vast majority of monocular networks do not make use of this extra signal, thus ignoring valuable information that could be used to improve the predicted depth. Those that do, either use computationally expensive test-time refinement techniques or off-the-shelf recurrent networks, which only indirectly make use of the geometric information that is inherently available. We propose ManyDepth, an adaptive approach to dense depth estimation that can make use of sequence information at test time, when it is available. Taking inspiration from multi-view stereo, we propose a deep end-to-end cost volume based approach that is trained using self-supervision only. We present a novel consistency loss that encourages the network to ignore the cost volume when it is deemed unreliable, e.g. in the case of moving objects, and an augmentation scheme to cope with static cameras. Our detailed experiments on both KITTI and Cityscapes show that we outperform all published self-supervised baselines, including those that use single or multiple frames at test time.
1. Introduction
Self-supervised monocular depth methods often ignore additional frames available at test time, despite a performance gap with specialist hardware and deep multi-view methods. ManyDepth addresses this by combining monocular and multi-view estimation with adaptive cost volumes and achieves state-of-the-art results on KITTI and Cityscapes.
- Self-supervised monocular depth estimation avoids ground-truth depth capture but remains behind specialist hardware and deep multi-view methods at test time.
- ManyDepth uses additional frames at training and test time to self-supervise a multi-frame depth estimation system.
- ManyDepth combines monocular and multi-view depth estimation while addressing moving objects, static scenes, and scale ambiguity.
- Adaptive cost volumes learn their extents from data rather than setting them as parameters.
- ManyDepth outperforms existing single- and multi-frame approaches on KITTI and Cityscapes.
2. Related work
Related work spans self-supervised monocular depth, temporal test-time methods, and multi-view stereo. ManyDepth targets limitations involving inference efficiency, variable frame availability, camera motion, poses, and moving objects while retaining strong benchmark performance.
- Self-supervised monocular methods train from image-reconstruction losses using stereo pairs or monocular video instead of ground-truth depth.
- Temporal depth methods include test-time refinement and recurrent networks, trading accuracy or direct geometric reasoning against computation and training complexity.
- Unlike supervised multi-view methods, ManyDepth requires no ground-truth supervision and is robust to object motion.
- ManyDepth often outperforms test-time refinement in accuracy while retaining recurrent methods’ inference efficiency.
- Existing multi-view approaches commonly require multiple test images, non-static cameras, camera poses, or scenes without moving objects.
3. Problem setup
The problem setup predicts a pixel-aligned depth map from a current image and optionally preceding video frames. The model can operate with one or multiple test-time frames, learns relative poses, avoids semantic masks, and assumes known camera intrinsics.
- Single-image depth estimation maps the input image I_t to a pixel-aligned depth map D_t using a depth network θ_depth.
- ManyDepth accepts N previous temporal video frames in addition to the current frame.
- The model supports single-frame test inputs, uses only previous frames online, and exploits previous and future frames for training supervision without stereo supervision.
- A differentiable pose network predicts relative poses during training and testing, while semantic models are not used to mask moving objects.
- The setup assumes known fixed camera intrinsics K.
4. Method
ManyDepth combines self-supervised reprojection training with a multi-view cost volume, then adds adaptive depth ranges, reliability-aware consistency training, and augmentation for missing baselines. The model uses neighboring frames when available while addressing scale ambiguity, moving objects, static cameras, and single-frame inputs.
- Core approach: ManyDepth combines self-supervised reprojection-based training with a multi-view cost volume and introduces adaptive volumes, consistency regularization, and augmentation.These innovations target scale ambiguity, unreliable matching, static cameras, and single-frame inputs.
- Self-supervised training: The reprojection loss synthesizes the target view from temporally close neighboring frames using estimated depth and relative pose, selecting the best source image per pixel.Training uses future frames for reprojection, although the cost volume uses preceding frames for online operation.
- Building a cost volume: The cost volume measures feature compatibility across linearly spaced hypothesized depths by warping source-frame features into the target viewpoint.Absolute feature differences are averaged across source images, concatenated with target features, and decoded into depth.
- Adaptive cost volumes: Adaptive cost-volume bounds learn dmin and dmax from batch predictions, updating exponential moving averages that are saved with the model and fixed at test time.This addresses the depth scale ambiguity of monocular self-supervision and avoids test-time coarse-to-fine adaptation.
- Addressing cost volume overfitting: A consistency network supplies depth targets only where the cost volume is unreliable, using a mask based on disagreement between its prediction and the cost-volume argmin.The separate network is discarded after training, and gradients to its predictions are blocked.
- Static cameras and start-of-sequences: Training with randomly zeroed cost volumes lets the model produce plausible depths when no preceding frame or camera baseline is available.This handles single images and sequence starts, while the consistency strategy addresses severe moving-object artefacts.
5. Implementation details
The implementation uses fixed-resolution self-supervised training with adjacent frames for cost-volume construction, while retaining a monocular depth backbone and pose network.
- Models use 640×192 input and output resolution, with N = 1 constructing the cost volume from It and It−1.Training self-supervision uses It−1, It, and It+1.
- Training applies color and horizontal-flip augmentations using the settings from [24].The models are trained with Adam for 20 epochs, initially using a learning rate of 10−4.
- The depth feature extractor uses the first five ResNet18 layers, aggregates features into a cost volume, and concatenates it with input image features.The remaining ResNet18 convolutional layers and the depth decoder from [24] complete the depth network.
- The pose network, depth-network skip connections, and consistency network follow the architectures used in [24].The consistency network uses the standard [24] architecture without modifications.
6. Experiments
Experiments on KITTI and Cityscapes compare ManyDepth with single- and multi-frame self-supervised methods, refinement systems, and ablated variants. The results emphasize broad accuracy, qualitative robustness, and substantially lower inference cost than test-time optimization.
- 6.1. KITTI results: ManyDepth outperforms previously published self-supervised methods without semantic supervision on most KITTI metrics.The comparisons separate resolution and expensive multi-pass test-time refinement settings.
- 6.1. KITTI results: ManyDepth outperforms previous methods across most KITTI metrics whether or not baselines use multiple test-time frames.Table 2 separates medium/low- and high-resolution results, with and without test-time refinement.
- 6.1. KITTI results: 50 test-time refinement steps further improve ManyDepth and outperform other test-time refinement methods.The refinement updates depth and pose encoder weights using sequential image pairs.
- 6.1. KITTI results: Qualitative error maps expose large hidden mistakes from single-image methods, especially in ambiguous regions such as a dark freeway embankment.A multi-frame method that does not explicitly use geometry improves over a single-frame baseline but has higher error than ManyDepth.
- 6.2. KITTI ablation: ManyDepth is more computationally efficient than test-time refinement methods that perform multiple forward-backward passes.Figure 5 compares absolute relative error against MACs and identifies refinement methods as too demanding for real-time applications.
- 6.2. KITTI ablation: The augmentation scheme significantly improves start-of-sequence and static-camera evaluation, while standard moving-camera performance changes negligibly.Static-camera evaluation uses identical input frames, whereas start-of-sequence evaluation uses only It.
- 6.3. Cityscapes results: ManyDepth consistently outperforms competing methods on Cityscapes, including methods using semantic supervision.Table 3 reports results from models trained and tested on Cityscapes.
7. Conclusion
The paper presents a fully self-supervised online method that predicts depth from one image or multiple images when available. It reports state-of-the-art KITTI and Cityscapes results with greater robustness and inference efficiency than relevant alternatives.
- ManyDepth predicts depth from a single image or multiple images when available using a fully self-supervised online method.
- The method is more robust to moving objects and static cameras than a naive cost-volume integration.
- ManyDepth achieves state-of-the-art results on both KITTI and Cityscapes.
- ManyDepth is significantly more efficient during inference than test-time refinement methods, despite close depth accuracy among those methods.
A. Additional Qualitative Results
ManyDepth transfers from KITTI to YouTube videos while using multi-frame cost-volume processing to recover details that a monocular baseline misses.
- Qualitative transfer: ManyDepth recovers details such as signs, cars, pillars, and other structures in YouTube videos using a model trained on KITTI.The comparison includes an equivalent KITTI-trained Monodepth2 model, which misses some details because it cannot use multiple test-time frames.
- Architecture: The architecture extracts features from each input frame, aggregates them into a 96-bin cost volume, and combines that volume with features from the target frame.A decoder then produces full-resolution depth using residual layers, upsampling, and skip connections.
- Architecture: The model uses the same pose network as Monodepth2, while its consistency network follows the standard Monodepth2 architecture without modifications.
D.1. KITTI benchmark scores
The supplementary KITTI materials provide additional evaluation context, ablation references, and comparisons involving multi-frame and self-supervised depth methods.
- Evaluation setup: KITTI results are also evaluated using improved dense ground truth, but only a subset of baselines is included because many do not report or provide predictions for this setting.
- Training setup: Without ImageNet pretraining, ManyDepth scores are slightly worse than with pretraining but remain competitive.
- Ablations: The KITTI ablation reports all seven metrics on the Eigen split, with experiment details given in the main paper.
- Masking comparison: The motion masking method of [24] is competitive on KITTI, while the proposed masking scheme is significantly better on Cityscapes with more moving objects.
E.1. Alternative crop evaluation schemes
Cityscapes evaluation uses two alternative cropping schemes, and the second removes the image region containing the lower image car.
- Comparison: The supplementary table reports Cityscapes results under both cropping schemes, and the paper states that ManyDepth beats all competing models.
- Scheme A: Cropping scheme A evaluates the central 50% vertically, then removes 192 pixels from each side, leaving a 1664 × 512 region.
- Scheme B: Cropping scheme B evaluates only the top 75% of the image, ignoring the bottom 25% and leaving a 2048 × 768 region.This effectively crops out the car at the bottom and suits methods trained with the same cropping scheme.
F. Note on Cityscapes training of Monodepth2
The supplementary material documents evaluation conventions, ablations, computational accounting, and training choices for Cityscapes and related comparisons.
- Monodepth2 training: Monodepth2 was retrained for five epochs with batch size 12 to mirror the consistency-network procedure and account for Cityscapes’ larger training set.Training for the full 20 epochs produced significantly worse scores.
- Masking ablation: Cityscapes masking ablations show larger improvements from ManyDepth’s motion masking than KITTI ablations because Cityscapes contains more moving objects.
- Augmentations: The augmentation experiment compares models trained with and without novel augmentations under start-of-sequence and static-camera scenarios.
- Computation: The architecture table defines feature extraction on each input frame followed by cost-volume combination, while MAC accounting includes both input-frame feature extraction and cost-volume construction.
- Evaluation: The Cityscapes results table uses two alternative crop schemes, and ManyDepth beats all competing models under those reported evaluations.