Source-linked AI summary
DF-Net: Unsupervised Joint Learning of Depth and Flow using Cross-Task Consistency
Yuliang Zou, Zelun Luo, Jia-Bin Huang
TL;DR
Existing depth and flow methods face limited dense supervision and often treat correlated tasks separately. DF-Net jointly trains both models with a cross-task geometric consistency loss on unlabeled videos, and reports favorable or competitive performance against unsupervised methods.
Problem
Dense pixelwise supervision for depth and optical flow is costly to collect, while existing efforts typically study these correlated tasks in isolation.
Method
DF-Net jointly trains single-view depth and optical flow networks using unlabeled videos and a loss enforcing consistency between synthesized rigid flow and directly estimated flow.
Results
Joint training improves both models, with depth comparing favorably against unsupervised methods and flow achieving competitive performance with state-of-the-art approaches.
Takeaways & Limitations
The trained depth and flow models can be applied independently at test time, while geometric constraints provide additional supervisory signals during training.
Takeaways & Limitations
The approach remains constrained by errors in depth or camera-pose estimation, which can propagate into synthesized flow predictions.
Abstract
from arXiv · showhide
We present an unsupervised learning framework for simultaneously training single-view depth prediction and optical flow estimation models using unlabeled video sequences. Existing unsupervised methods often exploit brightness constancy and spatial smoothness priors to train depth or flow models. In this paper, we propose to leverage geometric consistency as additional supervisory signals. Our core idea is that for rigid regions we can use the predicted scene depth and camera motion to synthesize 2D optical flow by backprojecting the induced 3D scene flow. The discrepancy between the rigid flow (from depth prediction and camera motion) and the estimated flow (from optical flow model) allows us to impose a cross-task consistency loss. While all the networks are jointly optimized during training, they can be applied independently at test time. Extensive experiments demonstrate that our depth and flow models compare favorably with state-of-the-art unsupervised methods.
1 Introduction
Depth prediction and optical flow estimation recover correlated scene information, yet are typically learned separately and often depend on difficult dense annotations or unstable photometric assumptions. DF-Net jointly trains both models from unlabeled videos using geometric cross-task consistency, improving performance on benchmark datasets.
- Depth and flow describe correlated scene structure and motion, but existing efforts typically study them in isolation.
- Dense pixelwise depth and flow annotations are difficult and costly to collect, limiting supervised training data.
- Brightness constancy and spatial smoothness assumptions can fail at motion boundaries, making unsupervised training unstable.
- Geometry-based flow synthesis assumes accurate depth and camera pose, because errors in either prediction produce inaccurate flow.
- DF-Net jointly learns depth and flow from unlabeled videos with a cross-task consistency loss using predictions from depth, pose, and optical flow.
- Extensive experiments show that joint training improves both models, which compare favorably with state-of-the-art unsupervised methods and competitively with supervised methods.
2 Related Work
Prior work reduces annotation dependence with photometric and smoothness losses or exploits geometry, but ambiguity, occlusion, and error propagation remain challenges. DF-Net couples depth and flow through geometric consistency without task-specific ground-truth supervision, while retaining independent test-time use.
- Supervised learning of depth and flow: Fully supervised depth and flow methods depend on costly dense annotations, with optical-flow annotations especially difficult to acquire in diverse real-world scenes.
- Self-supervised learning of depth and flow: Self-supervised methods use brightness constancy and spatial smoothness with differentiable frame warping, but textureless regions and occlusion boundaries create ambiguity and instability.
- Methods exploiting geometry cues: Geometry-based methods synthesize rigid-region flow from estimated depth and camera poses, but errors in either estimate propagate into flow predictions.
- Methods exploiting geometry cues: DF-Net uses consistency between network-estimated flow and synthesized rigid flow so depth and flow can learn from each other during coupled training.
- Structure from motion: Structure-from-motion methods can recover semidense depth and camera pose, whereas this work learns dense single-view structure and a flow model jointly.
- Multi-task learning: Unlike multi-task methods requiring direct ground-truth supervision for each task, DF-Net uses meta-supervision to couple training and applies the models independently at test time.
3 Unsupervised Joint Learning of Depth and Flow
DF-Net jointly trains depth and optical-flow models from unlabeled videos by combining standard unsupervised losses with geometric cross-task consistency. It synthesizes rigid flow from predicted depth and camera pose, compares it with estimated flow, and applies consistency constraints on valid regions.
- Flow synthesis using depth and pose predictions: Given consecutive frames, the method predicts depths, forward-backward flows, and a 6D camera pose before synthesizing rigid image-plane flow.Rigid flow is obtained by backprojecting scene motion induced by depth and camera motion.
- Overall objective: The objective combines photometric, smoothness, forward-backward, and cross-task consistency losses, applied symmetrically to depth and flow networks.The four terms are optimized in both forward and backward directions.
- Brightness constancy and spatial smoothness priors: Photometric supervision warps one frame using predicted or rigid flow, while a ternary census loss improves robustness to complex illumination changes.Differentiable bilinear interpolation is used when projected coordinates fall between pixels.
- Brightness constancy and spatial smoothness priors: Spatial smoothness regularizes disparity or flow where brightness constancy is uninformative, especially in low-texture or homogeneous regions.The method adopts a prior spatial smoothness loss for these regions.
- Forward-backward consistency: Forward-backward consistency identifies invalid pixels caused by occlusion or dis-occlusion and excludes them from photometric supervision.The same consistency principle also regularizes predictions in valid regions for both flow and depth.
- Cross-task consistency: For non-occluded static regions, the cross-task loss minimizes endpoint distance between rigid flow from depth and pose and flow predicted by the optical-flow network.Valid masks restrict this constraint to pixels where the comparison is appropriate, allowing both models to be updated jointly.
4 Experimental Results
Experiments evaluate the jointly trained depth, pose, and flow models on KITTI and Make3D, using unlabeled video training and standard depth and flow benchmarks. The method achieves strong depth and flow results, while consistency-loss ablations show the value of the proposed training signals.
- Experimental setup: Joint training uses unlabeled KITTI raw video clips, with separate pre-training on KITTI or CityScapes for depth and pose and SYNTHIA for flow.The joint-training setup uses no depth or flow labels.
- Experimental setup: Evaluation covers KITTI raw and Make3D for depth, and KITTI flow 2012 and 2015 for optical flow.KITTI flow ground truth covers about 50% of pixels because it is obtained from a 3D laser scanner.
- Depth estimation: The depth model achieves state-of-the-art performance among monocular-video methods on KITTI raw, but trails methods using stereo or ground-truth depth supervision.The authors attribute this gap to errors from the pose network and identify calibrated stereo videos as future work.
- Ablation and qualitative results: Removing either forward-backward or cross-task consistency causes significant performance degradation, highlighting the importance of the consistency losses.Qualitative KITTI results also show clearer contours and better thin-structure capture than the cited depth baselines.
- Depth estimation: The depth model achieves state-of-the-art performance against unsupervised models on Make3D and remains competitive with supervised models without Make3D fine-tuning.The evaluated model is trained on Cityscapes plus KITTI, with errors computed only below 70 meters.
- Optical flow estimation: The flow model achieves state-of-the-art performance on KITTI flow 2012 and 2015, and optional label fine-tuning yields competitive performance with similarly structured supervised models.The authors suggest unsupervised pre-training for domains where ground-truth flow data are scarce.
- Pose estimation: The pose network is competitive with state-of-the-art visual-SLAM and unsupervised-learning methods, and the authors expect better pose estimation to improve depth and flow.The pose network follows the evaluation protocol of prior work and uses a five-frame input.
5 Conclusions
The paper presents an unsupervised framework that jointly trains single-view depth and optical-flow models from unlabeled videos through cross-task geometric consistency. Experiments show favorable depth and competitive flow performance, while the trained models remain independently applicable at test time.
- Conclusions: The framework jointly trains depth and optical-flow networks from unlabeled video sequences using cross-task consistency.The consistency couples training through geometric constraints between the tasks.
- Conclusions: At test time, the trained depth and flow models can be applied independently.Joint optimization is used during training, but inference does not require both models together.
- Conclusions: The depth model compares favorably with unsupervised models on KITTI and Make3D, while the flow model achieves competitive performance with state-of-the-art approaches.The conclusion frames geometric constraints as a direction for advancing multiple dense-prediction tasks with unlabeled data.