Source-linked AI summary
GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose
Zhichao Yin, Jianping Shi
TL;DR
GeoNet targets jointly estimating monocular depth, optical flow, and camera motion without costly supervision. It couples these predictions through 3D scene geometry, separates rigid and dynamic motion, and adds adaptive geometric consistency. On KITTI, it outperforms previous unsupervised methods and reaches comparable results to supervised ones across all three tasks.
Problem
GeoNet addresses the need to estimate depth, optical flow, and camera motion without large, costly ground-truth datasets or task-isolated models.
Method
GeoNet jointly learns monocular depth, optical flow, and camera motion from video using scene geometry, fused motion, view synthesis, and adaptive geometric consistency.
Results
GeoNet outperforms previous unsupervised methods and achieves results comparable with supervised ones across depth, optical flow, and camera-motion tasks on KITTI.
Takeaways & Limitations
The framework demonstrates that geometric relationships can support unsupervised learning across previously isolated low-level vision tasks.
Takeaways & Limitations
GeoNet’s residual-flow predictions struggle with large displacement because warping-loss gradients are local; avoiding this issue is left for future work.
Abstract
from arXiv · showhide
We propose GeoNet, a jointly unsupervised learning framework for monocular depth, optical flow and ego-motion estimation from videos. The three components are coupled by the nature of 3D scene geometry, jointly learned by our framework in an end-to-end manner. Specifically, geometric relationships are extracted over the predictions of individual modules and then combined as an image reconstruction loss, reasoning about static and dynamic scene parts separately. Furthermore, we propose an adaptive geometric consistency loss to increase robustness towards outliers and non-Lambertian regions, which resolves occlusions and texture ambiguities effectively. Experimentation on the KITTI driving dataset reveals that our scheme achieves state-of-the-art results in all of the three tasks, performing better than previously unsupervised methods and comparably with supervised ones.
1. Introduction
GeoNet addresses unsupervised monocular depth, optical flow, and camera-motion estimation by exploiting shared 3D scene geometry. Its staged architecture and adaptive consistency loss separately model rigid and dynamic motion while handling occlusions and outliers.
- Motivation: Deep geometry models reduce reliance on traditional feature correspondences but usually require costly ground-truth data and target individual tasks.Prior joint approaches may also require laser-scanned depth, stereo cameras, or lack explicit handling of non-rigidity and occlusions.
- Contribution: GeoNet jointly estimates monocular depth, optical flow, and camera motion from video without supervision.The framework is built around geometric relationships among these tasks and the structure of 3D scenes.
- Architecture: A cascaded two-stage architecture learns rigid scene flow and object motion separately, progressively refining the global motion field.The fused motion field guides view synthesis, providing regularization for unsupervised learning.
- Geometric consistency: Adaptive geometric consistency filters outliers and occlusions, enforcing coherence in non-occluded regions while smoothing erroneous predictions in occluded regions.It is designed to address occlusion handling and photo-inconsistency issues beyond pure view synthesis.
- Evaluation: GeoNet outperforms prior unsupervised methods and achieves results comparable with supervised ones across depth, optical flow, and camera-motion evaluation on KITTI.The authors report comprehensive evaluation on the KITTI dataset for all three tasks.
2. Related Work
Related work spans traditional structure-from-motion, supervised deep geometry models, and unsupervised view-synthesis methods. These approaches improve geometry understanding but remain constrained by feature matching, supervision, equipment requirements, or incomplete geometric consistency.
- Traditional Scene Geometry Understanding: Traditional structure-from-motion jointly reconstructs scene structure and camera motion but depends heavily on accurate feature matching.Low texture, stereo ambiguities, and occlusions are common failure sources.
- Scene Flow Estimation: Scene-flow methods jointly reason about geometry, rigid motion, and segmentation, but discrete optimization can be too slow for practical use.Other approaches model rigid regularities or segment moving objects from the rigid scene.
- Supervised Deep Models for Geometry Understanding: Supervised deep models have advanced depth, optical-flow, and pose estimation, using architectures such as multiscale networks, correlation layers, and 3D convolutions.These methods exploit learned representations and geometric or sensory constraints across distinct tasks.
- Unsupervised Learning of Geometry Understanding: Unsupervised geometry-learning methods commonly use view synthesis and geometric inference to avoid expensive ground-truth data.Prior work extends reconstruction with smoothness or consistency losses, but approaches differ in how they exploit cross-prediction geometry and handle pixels.
3. Method
GeoNet decomposes scene motion into rigid geometry and residual non-rigid motion, jointly learning depth, camera pose, and optical flow through view synthesis. Geometric consistency and robust photometric objectives address outliers, occlusions, and dynamic regions.
- Overview of GeoNet: GeoNet separates static-scene estimation from dynamic-object motion by combining a rigid structure reconstructor with a non-rigid motion localizer.The two components estimate rigid flow from scene geometry and compensate it with residual motion.
- Geometric Consistency Enforcement: GeoNet applies bidirectional geometric consistency checks to filter possible outliers and occlusions, enforcing coherence in valid regions while smoothing erroneous regions.The adaptive consistency mechanism complements view synthesis because occlusions and non-Lambertian surfaces can violate photometric assumptions.
- Rigid Structure Reconstructor: DepthNet predicts depth from individual views, PoseNet regresses relative 6DoF camera poses from frame sequences, and their outputs produce rigid flow.Rigid flow is derived from predicted depth, camera motion, and camera intrinsics, then used for differentiable inverse warping.
- Rigid Structure Reconstructor: The rigid-stage view synthesis objective compares inverse-warped source images with target frames and is supplemented by robust photometric similarity and edge-aware depth smoothness.The smoothness term is weighted by image gradients to filter erroneous predictions while preserving sharp details.
- Non-rigid Motion Localizer: ResFlowNet predicts residual non-rigid flow after rigid reconstruction, focusing on dynamic objects while refining errors caused by difficult illumination and thin structures.The residual signal is combined with rigid flow to obtain the final flow prediction, with corresponding flow warping and smoothness losses.
4. Experiments
Experiments evaluate GeoNet's depth, optical-flow, and camera-pose components on KITTI using quantitative and qualitative comparisons. Results show strong performance across tasks, while optical-flow analysis identifies limitations of cascaded warping-loss optimization.
- 4. Experiments: GeoNet evaluates monocular depth, optical flow, and camera pose estimation on KITTI using task-specific training and testing splits.The experiments report qualitative and quantitative results for all three tasks.
- 4.1. Implementation Details: GeoNet uses DepthNet, PoseNet, and ResFlowNet to reconstruct rigid scene structure and localize non-rigid motion.DepthNet and PoseNet form the rigid structure reconstructor, while ResFlowNet estimates non-rigid motion.
- 4.2. Monocular Depth Estimation: On KITTI monocular depth, GeoNet significantly outperforms supervised and previously unsupervised methods in the reported comparison.The KITTI-only VGG variant also uses the same architecture as the cited unsupervised baseline without batch normalization, isolating the loss-function comparison.
- 4.3. Optical Flow Estimation: GeoNet achieves the lowest overall-region EPE and comparable non-occluded-region performance against unsupervised optical-flow baselines.The geometric consistency loss improves a direct-flow variant, while GeoNet's geometry-based architecture further improves overall-region results.
- 4.3. Optical Flow Estimation: Naively enforcing consistency loss deteriorates accuracy, whereas GeoNet is especially effective at correcting small residual-flow errors near rigid flow.For larger residual displacements, the predicted residual flow can prematurely converge because warping-loss gradients are local pixel-intensity differences amplified by the cascaded architecture.
- 4.4. Camera Pose Estimation: GeoNet outperforms all competing baselines in camera-pose estimation but can confuse the reference system when large dynamic objects appear nearby.This failure case is reported alongside the camera-pose comparison.
5. Conclusion
GeoNet jointly learns low-level vision tasks by exploiting geometric relationships among previously isolated tasks, with results indicating that this can reduce reliance on costly groundtruth data. Future work targets warping-loss gradient locality and adding semantic information.
- GeoNet jointly unsupervisedly learns multiple low-level vision tasks by exploiting geometric relationships among previously isolated tasks.
- The reported results indicate that these tasks may be learned without costly collected groundtruth data, including comparisons with supervised baselines.
- Future work will address the gradient locality issue of warping-based loss and test whether semantic information improves GeoNet.