Source-linked AI summary
Self-supervised Learning with Geometric Constraints in Monocular Video: Connecting Flow, Depth, and Camera
Yuhua Chen, Cordelia Schmid, Cristian Sminchisescu
TL;DR
Estimating the 3D geometry of dynamic scenes from moving-camera video encompasses depth, optical flow, and odometry, while existing pipelines can produce sparse reconstructions and struggle with occlusion and degenerate geometry. GLNet is a self-supervised framework that jointly learns and refines depth, flow, camera pose, and intrinsics under photometric and geometric constraints, achieving gains on KITTI and Cityscapes and supporting uncalibrated video and transfer learning.
Problem
Estimating 3D geometry in dynamic scenes captured by a moving camera spans depth, optical flow, and odometry, while existing bundle-adjustment pipelines can be sparse and struggle with occlusion and degenerate geometry.
Method
GLNet combines self-supervised photometric and geometric losses with jointly refined depth, pose, flow, and camera intrinsics, extending bundle adjustment through online parameter and output finetuning.
Results
GLNet achieves performance gains over existing methods on KITTI and Cityscapes, improves depth and pose estimation, learns from uncalibrated video, and shows good transfer-learning performance.
Takeaways & Limitations
The framework supports dense multi-task geometric learning from monocular video, including uncalibrated video and cross-domain transfer learning.
Takeaways & Limitations
Existing real-time structure-from-motion pipelines remain affected by partial occlusion and degenerate scene geometry or motion.
Abstract
from arXiv · showhide
We present GLNet, a self-supervised framework for learning depth, optical flow, camera pose and intrinsic parameters from monocular video - addressing the difficulty of acquiring realistic ground-truth for such tasks. We propose three contributions: 1) we design new loss functions that capture multiple geometric constraints (eg. epipolar geometry) as well as an adaptive photometric loss that supports multiple moving objects, rigid and non-rigid, 2) we extend the model such that it predicts camera intrinsics, making it applicable to uncalibrated video, and 3) we propose several online refinement strategies that rely on the symmetry of our self-supervised loss in training and testing, in particular optimizing model parameters and/or the output of different tasks, thus leveraging their mutual interactions. The idea of jointly optimizing the system output, under all geometric and photometric constraints can be viewed as a dense generalization of classical bundle adjustment. We demonstrate the effectiveness of our method on KITTI and Cityscapes, where we outperform previous self-supervised approaches on multiple tasks. We also show good generalization for transfer learning in YouTube videos.
1. Introduction
GLNet combines self-supervised dense reconstruction with explicit geometric constraints and joint refinement of depth, motion, and intrinsics. It is evaluated on KITTI and Cityscapes, with reported gains, transfer performance, and learning from uncalibrated video.
- Classical structure-from-motion systems enforce exact geometric relations but produce sparse reconstructions and struggle with occlusion and degenerate geometry.
- Supervised deep models provide dense estimates but can be geometrically inconsistent, data-intensive, domain-sensitive, and difficult to ground in realistic measurements.
- Photometric self-supervision avoids labeled data but lacks fundamental constraints linking rigid and apparent motion across multiple views.
- GLNet integrates dense self-supervision with exact relations such as epipolar geometry and jointly refines depth, pose, and camera intrinsics.
- The framework is evaluated on KITTI and Cityscapes, showing gains over prior methods, transfer-learning performance, and robust learning from uncalibrated video.
2. Related Work
Prior work spans classical geometric reconstruction, supervised task-specific learning, and self-supervised photometric approaches. These lines of research trade off sparsity, labeling cost, domain transfer, and the completeness of geometric reasoning.
- Classical feature matching and geometric verification produce strong reconstructions but remain sparse and error-prone in textureless or occluded regions.
- Supervised methods address monocular depth, optical flow, and pose estimation, while joint learning combines multiple visual cues.
- Ground-truth training depends on expensive sensors such as LIDAR or synthetic data that can introduce domain shift.
- Monocular-video methods learn depth and ego-motion, with later extensions incorporating object motion, alignment losses, SfM supervision, flow, edges, and multiple rigid motions.
3. Methodology
GLNet jointly learns monocular depth, optical flow, camera pose, and intrinsics through adaptive photometric and geometric constraints. The same self-supervised objective supports online refinement of model parameters or task outputs during inference.
- Framework: GLNet predicts depth, optical flow, camera pose, and camera intrinsics through coupled predictors operating on monocular video.Depth is estimated from one image, while camera and flow use adjacent frames; intrinsics are predicted when unknown.
- Optimization objectives: The objective combines adaptive photometric consistency with geometric losses linking rigid and apparent motion across tasks.The framework includes multi-view 3D structure consistency and an epipolar constraint over dense optical-flow correspondences.
- Adaptive photometric loss: Adaptive photometric loss selects the lower-error explanation between rigid motion and optical flow for each pixel, supporting static, rigidly moving, and non-rigid structures.This channels updates toward the displacement that best explains each pixel’s appearance change.
- Multi-view 3D structure consistency: Multi-view structure consistency penalizes discrepancies between predicted 3D coordinates of corresponding points after transforming them into a common camera coordinate system.The loss updates DepthNet and CameraNet and can generalize immediately to multiple views.
- Epipolar constraint: The epipolar loss enforces a global geometric constraint on dense flow correspondences and updates camera intrinsics, pose, and optical flow.This couples optical flow with the camera displacement and intrinsic parameters through the epipolar relationship.
- Online optimization: The same self-supervised losses are optimized during training and inference, enabling parameter finetuning or direct output refinement while preserving task dependencies.Output regularization penalizes large deviations from the learned model’s original prediction to reduce overfitting to a single image pair.
4. Experiments
GLNet is evaluated on KITTI and Cityscapes for depth, optical flow, and pose estimation, including transfer to new environments and uncalibrated video. Experiments show gains from geometric losses and online refinement, with output refinement offering a faster alternative to parameter refinement.
- Experimental setup: Experiments use KITTI and Cityscapes, including cross-dataset transfer, with KITTI labels supporting evaluation of geometric understanding tasks.Training uses three-frame snippets, standard augmentation, and a nonadaptive photometric-loss baseline.
- Depth estimation: GLNet achieves significant depth-estimation gains over the baseline and competing methods with ground-truth camera intrinsics.Qualitative results show sharper depth maps, and GLNet without refinement still outperforms competing methods without refinement.
- Depth estimation: Predicted intrinsics match the calibrated depth-estimation setting, while Cityscapes-to-KITTI transfer improves with the proposed components.The authors attribute the transfer result partly to geometric constraints remaining valid across domains.
- Depth estimation: YouTube experiments show robust depth estimation from uncalibrated video across varied objects, structures, and scenes.Online refinement improves erroneous feed-forward predictions in a new environment; PFT appears slightly sharper than OFT visually.
- Loss ablations: The adaptive photometric loss improves over the standard loss, while geometric losses provide further gains, especially with online refinement.Multi-view structure consistency is reported as most effective for depth estimation because it integrates information over multiple views.
- Online refinement: OFT prevents overfitting through regularization and reaches improvement comparable to PFT while running about one order of magnitude faster.OFT takes about 2 seconds for 50 iterations versus around 40 seconds for PFT; combining both achieves the best overall performance.
- Optical flow: For optical flow, the epipolar constraint yields a considerable gain over baseline, online refinement improves results further, and geometric constraints enhance rigid-region flow.Uncalibrated flow performance is very close to calibrated performance, although epipolar corrections are calibration-sensitive.
- Pose estimation: GLNet improves pose estimation over existing methods and the baseline on the official KITTI visual odometry benchmark.Sequences 00–08 are used for training and 09–10 for testing.
5. Conclusions
GLNet jointly learns depth, optical flow, camera pose, and intrinsic parameters from monocular video using geometric and photometric constraints. Its refinement strategies and uncalibrated-video capability support strong performance across domains.
- Contributions: GLNet jointly learns depth, optical flow, camera pose, and intrinsic parameters from monocular video.The framework combines photometric and geometric losses, including epipolar and temporal structure-consistency constraints.
- Contributions: Online parameter and output finetuning generalizes bundle adjustment and provides adaptation speed-ups of up to one order of magnitude.These methods reduce the asymmetry between training and testing.
- Contributions: GLNet predicts camera intrinsics for uncalibrated video and exhibits consistent performance across different training and testing domains.Camera pose evaluation is reported using absolute trajectory error on the KITTI odometry dataset.