Source-linked AI summary
Depth from Videos in the Wild: Unsupervised Monocular Depth Learning from Unknown Cameras
Ariel Gordon, Hanhan Li, Rico Jonschkowski, Anelia Angelova
TL;DR
Unsupervised monocular depth learning must handle moving objects, occlusions, and unknown camera parameters while relying only on video consistency. The paper jointly learns depth, egomotion, object motion, and camera intrinsics through differentiable geometric consistency, achieving state-of-the-art benchmark results and qualitative learning from YouTube videos. Its intrinsic estimates remain constrained by a coupling with predicted motion.
Problem
Unsupervised monocular depth learning is limited by moving objects, occlusions, and the requirement in prior work to provide camera intrinsic parameters.
Method
The method jointly predicts depth, camera and object motion, and camera intrinsics from monocular videos using neighboring-frame consistency, geometric occlusion handling, and rough masks for potentially mobile pixels.
Results
The method establishes state-of-the-art depth and odometry results on multiple benchmark datasets and qualitatively learns depth and camera intrinsics from YouTube videos with unknown, differing cameras.
Takeaways & Limitations
Learning camera intrinsics without ground-truth parameters enables depth learning from raw videos captured by unknown cameras.
Takeaways & Limitations
The consistency loss constrains combinations such as Kt and K^-1RK, so camera intrinsics and motion predicted by the same network can remain individually incorrect.
Abstract
from arXiv · showhide
We present a novel method for simultaneous learning of depth, egomotion, object motion, and camera intrinsics from monocular videos, using only consistency across neighboring video frames as supervision signal. Similarly to prior work, our method learns by applying differentiable warping to frames and comparing the result to adjacent ones, but it provides several improvements: We address occlusions geometrically and differentiably, directly using the depth maps as predicted during training. We introduce randomized layer normalization, a novel powerful regularizer, and we account for object motion relative to the scene. To the best of our knowledge, our work is the first to learn the camera intrinsic parameters, including lens distortion, from video in an unsupervised manner, thereby allowing us to extract accurate depth and motion from arbitrary videos of unknown origin at scale. We evaluate our results on the Cityscapes, KITTI and EuRoC datasets, establishing new state of the art on depth prediction and odometry, and demonstrate qualitatively that depth prediction can be learned from a collection of YouTube videos.
1 Introduction
The paper targets unsupervised depth and motion learning from monocular videos despite sparse correspondences, moving objects, occlusions, and unknown camera parameters. It introduces methods for learning camera intrinsics from video and applying the approach to diverse unlabeled videos.
- Estimating 3D structure and camera motion from video is a key computer-vision problem, but correspondences cover only subsets of pixels, leaving depth underdetermined.Traditional methods solve for structure and motion from repeated scene points across consecutive frames.
- Unsupervised video methods learn from raw frames using consistency losses, enabling depth prediction from a single image and egomotion estimation from image sequences.
- Object motion, occlusions, and unavailable camera intrinsics are major obstacles for prior unsupervised monocular depth-learning methods.
- The method learns camera intrinsics, including lens distortion, directly from video and addresses occlusions geometrically using predicted depth.
- A rough mask covering potentially moving pixels replaces instance segmentation and tracking, with unions of rectangular bounding boxes sufficient for the proposed treatment.
- The approach achieves state-of-the-art results on multiple benchmark datasets and learns depth and intrinsics from YouTube videos captured by cameras with unknown, different intrinsics.
2 Related work
Related work spans supervised and unsupervised depth learning, learning from diverse internet imagery, geometric occlusion handling, and camera-intrinsics prediction. The paper positions its contribution as unsupervised intrinsic learning directly from video, jointly with depth, egomotion, and object motion.
- Supervised depth methods commonly learn dense predictions from RGB images using sensor measurements such as LiDAR.
- Unsupervised depth learning obtains supervision from monocular or binocular video itself and jointly predicts depth and ego-motion with differentiable models.
- Depth learning from internet imagery is difficult because input sources are diverse and camera parameters are unknown.
- Prior occlusion treatments included explainability masks, minimum reprojection losses, and optical flow, while this work proposes a direct geometric differentiable loss.
- Camera-intrinsics prediction had mostly used strong supervision, whereas this approach learns intrinsics directly from video jointly with depth, ego-motion, and object motion.
3 Preliminaries
The method connects adjacent frames through depth and camera geometry, then uses differentiable warping and photometric consistency to train predictions of scene structure and motion.
- The core geometric equation ties adjacent video frames through a depth map and the camera intrinsic matrix.
- K denotes the intrinsic matrix, while p and p′ are homogeneous pixel coordinates transformed by rotation R and translation t.
- The depths z and z′ and parameters fx, fy, x0, y0 represent the depths and camera intrinsics used in the transformation.
- Predicted z, R, and t warp one video frame onto another, and differences from the actual frame form the main training loss.
4 Method
The method jointly learns depth, egomotion, object motion, and camera intrinsics from monocular frame consistency. It regularizes motion, handles occlusions geometrically, and uses two networks for depth and motion-related predictions.
- A motion-prediction network estimates camera motion, per-pixel object motion relative to the background, and camera intrinsics, while a second network predicts depth.The intrinsics include focal lengths, offsets, and distortion; consistency across neighboring frames trains both networks.
- 4.1 Learning the intrinsics: Frame-consistency losses constrain depth, rotation, translation, and intrinsics, but translation can remain ambiguous with incorrectly paired intrinsic and translation estimates.The loss depends on K through Kt and K^-1RK, so an incorrect K can be offset by a corresponding incorrect t.
- 4.2 Learning object motion: The method regularizes motion by enforcing image-wide rotation constancy and allowing translation deviations only inside a possibly mobile mask, with L1 smoothing on translation.A union of bounding boxes is sufficient for the mask, avoiding the need for instance segmentation and tracking.
- 4.3 Occlusion-aware consistency: Occlusion-aware losses transform source pixels into the target view and retain only points whose transformed depth lies in front of the target depth, applying the procedure symmetrically.This selects the visible branch when warped depth becomes multivalued and is applied to photometric, geometric, and motion-consistency losses.
- 4.4 Networks, losses and regularizations: The networks use a ResNet-18 UNet for depth and a FlowNet-inspired UNet for global and residual motion prediction.The depth network uses a softplus activation, while the motion network predicts global rotation, global translation, intrinsics, and a spatial residual translation field.
- 4.4 Networks, losses and regularizations: Randomized layer normalization improves evaluation metrics over batch normalization, while SSIM is a crucial component of the training loss.The best reported results used multiplicative noise with layer normalization.
5 Experiments
Experiments evaluate depth, odometry, and camera-intrinsic recovery across KITTI, Cityscapes, EuRoC, and YouTube8M. Results report strong benchmark performance, benefits from learned intrinsics and pooled datasets, and ablation evidence for the proposed components.
- Evaluation setup: The method is evaluated on depth prediction, odometry estimation, and camera-intrinsic recovery across diverse datasets.The evaluation includes KITTI, Cityscapes, EuRoC, and YouTube8M videos.
- EuRoC and unknown cameras: The method learns depth and camera intrinsics from YouTube videos captured by multiple cameras with unknown and generally different intrinsics.The collected YouTube8M videos contain varying fields of view and lens distortion.
- KITTI: On KITTI, the method obtains the best state-of-the-art result, with learned intrinsics consistently improving performance over given intrinsics.The model is trained on KITTI and evaluated on the KITTI Eigen partition.
- Cityscapes: On Cityscapes, the approach outperforms previous methods and benefits from learned intrinsics on this challenging benchmark with many dynamic objects.Cityscapes models are trained and tested on the Cityscapes benchmark using the established disparity-based evaluation protocol.
- Cityscapes + KITTI: Jointly training on Cityscapes and KITTI improves depth results on both datasets beyond the best models trained separately.Learning intrinsics enables pooling datasets despite unknown cameras, supporting data sources of potentially unlimited size.
- Ablations: Ablations show that object-motion estimation, randomized layer normalization, and occlusion handling each improve depth quality, especially on Cityscapes.Bounding-box unions perform as well as segmentation masks for identifying possibly mobile regions.
- EuRoC and unknown cameras: EuRoC depth maps are learned from monocular videos with learned intrinsics and no other input information.The dataset’s significant lens distortion provides a test of learning lens parameters.
- Odometry: Egomotion evaluation on KITTI sequences 09 and 10 shows improvements on both Absolute Trajectory Error and average relative translational drift.The translational-drift metric particularly indicates improved localization relative to prior art.
6 Conclusions
The work tackles major challenges in unsupervised depth and visual odometry, enabling depth learning from raw videos captured by unknown cameras.
- The method addresses occlusions geometrically, accounts for object motion, and introduces a novel regularizer for unsupervised depth and visual odometry.
- Learning camera intrinsics, including lens distortion, from video enables depth estimation from raw videos captured by unknown cameras.
A.1 Accuracy of camera intrinsics - derivation
Incorrect camera intrinsics can be compensated by an incorrect translation, leaving the product of intrinsics and translation correct under the training loss.
- If predicted intrinsics K̃ are incorrect, an incorrect translation t̃ can still make K̃t̃ correct.
A.2 If R̸ = I, ˜K = K: Proof
For non-identity rotations, the proof shows that the predicted intrinsic matrix must equal the correct intrinsic matrix under the stated assumptions.
- The derivation isolates R̃ and imposes the rotation constraint R̃R̃^T = I.
- This yields AR = RA, where A = K^-1K̃K̃^TK^T^-1.
- The proof assumes R ≠ I because identity rotation provides no supervision signal for the camera intrinsics.
- Using eigenvalue and eigenvector properties, the argument concludes A is the identity and therefore K̃ = K.
A.3 Tolerances
Camera-intrinsic supervision is absent for identity rotations and weakens as rotation approaches identity, while the appendix establishes practical pixel-error tolerances and reports benchmark comparisons.
- A.3 Tolerances: Identity rotation provides no supervision for K, whereas non-identity rotation provides complete supervision in the mathematical analysis.
- A.3 Tolerances: Rotations by 180 degrees or reflections are excluded as irrelevant to transforming a frame into an adjacent one.
- A.3 Tolerances: The supervision on K becomes weaker as the rotation R approaches the identity matrix.
- A.3 Tolerances: For rotation around y without translation, the transformation equation simplifies to a specialized form.
- A.3 Tolerances: At the image center, focal-length error can be fully compensated by predicting an adjusted rotation component.
- A.3 Tolerances: The derivation replaces focal length and rotation with their predicted values before eliminating the remaining rotation component.
- A.3 Tolerances: A first-order expansion expresses the transformed pixel error in terms of focal-length and rotation-estimation errors.
- A.3 Tolerances: The worst-case transformed-pixel error occurs at the image boundary, where px = w/2.
A.4 Full tables of metrics for depth estimation
This section supplements the reported depth-estimation metrics with complete tables and documents the EuRoC depth-map construction and KITTI odometry evaluations.
- Full metric coverage: The appendix provides the remaining depth-estimation metrics and full metric tables corresponding to earlier results.Tables A1–A3 contain the complete values for the metrics previously reported in Tables 1–2 and Figure 5.
- EuRoC depth-map construction: EuRoC depth maps are rendered by projecting merged Vicon Room 2 point clouds onto RGB-camera views using camera geometry.Occlusion handling keeps the closer point when projected points overlap substantially, and a projective-space grid makes the rendered maps more uniform.
- KITTI odometry evaluation: KITTI Sequence 10 results compare models with given intrinsics, learned intrinsics, and test-time intrinsic correction against ground truth and Struct2depth.Figure A2 displays the predicted trajectories and the appendix extends the main odometry metrics with additional tables.
- KITTI odometry evaluation: The appendix reports KITTI odometry using absolute trajectory error plus relative translation and rotation errors on sequences 09 and 10.Table A4 states that the method outperforms prior art on Sequence 10 and in average error on Sequence 09; Sequence 09 has larger error variance, potentially because egomotion uses two frames.