Source-linked AI summary
Towards Better Generalization: Joint Depth-Pose Learning without PoseNet
Wang Zhao, Shaohui Liu, Yezhi Shu, Yong-Jin Liu
TL;DR
The paper addresses scale inconsistency in self-supervised joint depth-pose learning, where assuming consistent depth and pose scales limits learning and generalization. It replaces PoseNet-based pose prediction with optical-flow geometry and two-view triangulation, aligning predicted depth to the triangulated structure. Experiments report improved accuracy and generalization across challenging scenarios and benchmarks, while identifying limits under large motions, pure rotation, and changing camera intrinsics.
Problem
Self-supervised depth-pose methods assume consistent depth and pose scales across samples, making learning difficult and degrading performance in long-sequence odometry and indoor environments.
Method
The system solves relative pose from optical-flow correspondences, triangulates an up-to-scale 3D structure, and explicitly aligns predicted depth to that structure for end-to-end training.
Results
The method improves accuracy and generalization over existing methods, with reported gains across KITTI Odometry, NYU v2, indoor, long-sequence, and unseen-motion scenarios while maintaining strong KITTI depth and flow performance.
Takeaways & Limitations
Direct optical-flow-based pose recovery and explicit depth-pose scale alignment are reported as key designs for more robust self-supervised learning under challenging scenarios.
Takeaways & Limitations
The system cannot handle multi-view motions beyond the optical-flow cost volume or pure rotation online, and changing camera intrinsics require additional assumptions or network design.
Abstract
from arXiv · showhide
In this work, we tackle the essential problem of scale inconsistency for self-supervised joint depth-pose learning. Most existing methods assume that a consistent scale of depth and pose can be learned across all input samples, which makes the learning problem harder, resulting in degraded performance and limited generalization in indoor environments and long-sequence visual odometry application. To address this issue, we propose a novel system that explicitly disentangles scale from the network estimation. Instead of relying on PoseNet architecture, our method recovers relative pose by directly solving fundamental matrix from dense optical flow correspondence and makes use of a two-view triangulation module to recover an up-to-scale 3D structure. Then, we align the scale of the depth prediction with the triangulated point cloud and use the transformed depth map for depth error computation and dense reprojection check. Our whole system can be jointly trained end-to-end. Extensive experiments show that our system not only reaches state-of-the-art performance on KITTI depth and flow estimation, but also significantly improves the generalization ability of existing self-supervised depth-pose learning methods under a variety of challenging scenarios, and achieves state-of-the-art results among self-supervised learning-based methods on KITTI Odometry and NYUv2 dataset. Furthermore, we present some interesting findings on the limitation of PoseNet-based relative pose estimation methods in terms of generalization ability. Code is available at https://github.com/B1ueber2y/TrianFlow.
1. Introduction
Self-supervised joint depth-pose learning is hindered by scale inconsistency between monocular depth and relative pose. The proposed system explicitly separates scale estimation, using optical-flow geometry and triangulation to improve supervision and generalization.
- Self-supervised methods jointly constrain monocular depth and ego-motion through depth reprojection and image reconstruction errors.
- Assuming consistent depth and pose scales across samples makes learning difficult and degrades performance in long-sequence visual odometry and indoor environments.
- The system recovers relative pose from optical-flow correspondences and fundamental-matrix solving instead of a CNN-based PoseNet.
- Two-view triangulation provides an up-to-scale 3D structure used to align predicted depth for depth-error computation and reprojection consistency.
- Experiments report improved robustness in long sequences, unseen camera motions, and indoor environments, with gains on NYU v2 and KITTI Odometry while maintaining state-of-the-art KITTI depth and flow performance.
2. Related Work
Prior work combines learned monocular depth and motion estimation with geometric constraints, while related two-view methods estimate correspondences and recover geometry. This paper combines dense optical flow, fundamental-matrix estimation, and triangulation to avoid scale inconsistency in joint learning.
- Monocular Depth Estimation: Self-supervised monocular-depth methods use photometric reprojection or SfM-derived supervision instead of requiring dense ground-truth depth.
- Self-Supervised Depth-Pose Learning: Joint depth-pose learning trains depth and camera-ego-motion networks using photometric error and additional geometric or optimization constraints.
- Self-Supervised Depth-Pose Learning: The paper differs from geometry-consistency approaches by directly solving relative pose from optical-flow correspondences to avoid scale inconsistency by design.
- Two-view Geometry: Learning-based two-view methods estimate dense correspondences and then combine them with fundamental-matrix estimation or pose networks.
- Two-view Geometry: The system uses optical flow, fundamental-matrix recovery, cheirality checking, and triangulation to align predicted depths and jointly supervise depth and flow.
3. Method
The method disentangles depth-pose scale by estimating pose from optical-flow correspondences, triangulating an up-to-scale structure, and aligning predicted depth to it. Scale-adapted depth then supports geometric supervision and reprojection during end-to-end training and inference.
- Existing self-supervised depth-pose methods assume depth and relative-pose predictions maintain a consistent scale across images.
- The system estimates relative pose by solving the fundamental matrix from reliable optical-flow correspondences instead of using CNN-based pose estimation.Correspondences are filtered using occlusion and forward-backward consistency, sampled from the top-scoring regions, and solved with RANSAC.
- It selects rigid, non-occluded, accurate matches for differentiable two-view triangulation, producing an up-to-scale 3D structure for depth supervision.Epipolar-line distances, occlusion masks, and flow consistency identify suitable correspondences; negative or out-of-bound triangulated depths are filtered.
- A single scale transformation aligns monocular depth with triangulated depth, and the minimized alignment error provides the depth loss.The transformed depth uses the relative-pose scale, explicitly removing scale inconsistency from the joint estimation.
- The aligned depth supports dense reprojection losses measuring rigid-flow and depth-map consistency, while optical-flow, triangulation, reprojection, and smoothness losses jointly train the system.At inference, the same fundamental-matrix strategy is used, with PnP applied when optical-flow magnitude is too small.
4. Experiments
Experiments evaluate the system on KITTI depth and flow, KITTI Odometry, NYUv2, and TUM-RGBD, emphasizing accuracy, long-sequence robustness, and cross-domain generalization. The results report strong KITTI performance and improved behavior in challenging motion and indoor settings.
- Experimental setup: The system is evaluated on KITTI depth and optical flow, KITTI Odometry, NYUv2, and TUM-RGBD datasets.KITTI uses Eigen’s split for depth and KITTI 2015 for flow; KITTI Odometry uses sequences 00–08 for training and 09–10 for testing.
- KITTI results: On KITTI monocular depth estimation, the method achieves comparable or better performance than state-of-the-art depth-pose learning methods.The reported gain is attributed to scale disentanglement and supervision from two-view triangulation.
- KITTI results: Joint depth-pose learning improves the optical flow module, which outperforms most previous unsupervised flow and joint learning methods.Results are reported using EPE on non-occluded and overall regions and Fl score on KITTI 2015.
- Long-sequence generalization: On KITTI Odometry, the method significantly improves over existing depth-pose learning systems and achieves results comparable with conventional SLAM without local bundle adjustment or mapping at inference.The improvement is linked to mitigating scale drift in depth-pose learning.
- Indoor generalization: On NYUv2 and TUM-RGBD, the method improves generalization under textureless regions, complex or abrupt motion, moving objects, and different lighting conditions.PoseNet-like baselines fail to generalize in these settings, while the proposed system produces state-of-the-art NYUv2 results and reasonably well TUM-RGBD trajectories.
- Discussion: The method maintains on-par or better KITTI performance while improving robustness and generalization through direct optical-flow-based ego-motion estimation and explicit scale alignment.The discussion identifies these two designs as the sources of the reported generalization gains and notes limitations for large motion and pure rotation.
5. Conclusion
The conclusion presents a system that addresses scale inconsistency in self-supervised joint depth-pose learning through optical-flow-based pose recovery and triangulation-based scale alignment. Experiments report improved accuracy and generalization over existing methods.
- Conclusion: The proposed system tackles scale inconsistency by recovering relative pose from optical flow and explicitly aligning depth and pose scales through triangulation.These are the two central design components identified in the conclusion.
- Conclusion: Experiments demonstrate significant improvement in both accuracy and generalization ability over existing methods.The paper identifies handling failure cases, general correspondence prediction, and integration with back-end optimization as future directions.
Appendix
The appendix lists supplemental discussions, derivations, implementation details, additional experiments, numerical results, and visualizations supporting the main paper.
- Supplement contents: The supplement provides a detailed discussion of the system’s scale-invariant design.This material appears in Section A.
- Supplement contents: The supplement derives the differentiable triangulation module and gives additional details about PoseNet and PoseNet-Flow.These materials are provided in Sections B and C.
- Supplement contents: Additional experiments cover PoseNet-Flow visual odometry and sampled KITTI Odometry comparisons.These results are provided in Sections D and F.
- Supplement contents: The supplement reports implementation details, TUM-RGBD numerical results, and additional visualizations across datasets.These materials are provided in Sections E, G, and H.
A. Discussion on Scale-Invariant Design
The method addresses scale inconsistency by separating relative-pose recovery from scale estimation and explicitly aligning predicted depth with triangulated structure.
- Joint depth-pose learning assumes predicted depth and relative pose share a consistent scale across input samples.
- The system introduces two coupled designs: flow-based fundamental-matrix pose recovery and explicit depth alignment to triangulated point clouds.
- Optical flow correspondences recover relative pose up to arbitrary scale without requiring the pose network to learn scale priors.
- Depth is aligned to the triangulated structure before error computation, allowing the predicted depth scale to match the estimated pose scale.
- The design is robust when camera intrinsics are known and fixed; varying intrinsics require the monocular depth network to infer calibration from image structure.
B. Derivation of Triangulation Module
The triangulation module reconstructs an up-to-scale 3D point from two camera rays, while the baseline systems use CNN-based pose estimation for comparison.
- Triangulation: Mid-point triangulation represents each correspondence as two camera rays and minimizes their squared distances to a shared 3D point.
- Triangulation: Ray origins derive from camera extrinsics, while ray directions use camera intrinsics and pixel coordinates.
- Triangulation: Setting the objective derivative to zero yields linear equations for the ray depths λ1 and λ2, after which the triangulated point is computed.
- Triangulation: The triangulation solution is naturally differentiable, supporting end-to-end training of the joint system.
- Baselines: PoseNet predicts relative pose from image pairs, whereas PoseNet-Flow feeds generated optical flow to PoseNet and uses two-stage training.
D. Additional Results and Discussion for PoseNet-Flow
PoseNet-Flow improves over PoseNet in indoor and standard KITTI settings but remains vulnerable to unseen camera motions, unlike the proposed method.
- PoseNet fails to generate plausible NYUv2 predictions under complex camera motions and large textureless regions.
- PoseNet-Flow improves interpretability and learning difficulty by using optical flow for pose regression.
- PoseNet-Flow produces feasible results on NYUv2 and standard KITTI but tends to fail on unseen ego-motions.
- The proposed method works across these challenging scenarios, showing improved robustness and generalization ability.
E. Implementation Details
The implementation combines multi-scale depth and flow networks with correspondence filtering, fundamental-matrix pose recovery, triangulation, and visual-odometry evaluation.
- Network architectures: The depth network uses a ResNet-18 encoder and DispNet decoder, converting disparity to depth with D = 1/(ad+b).
- Optical flow: The flow network uses PWCNet to predict forward and backward flow, with occlusion handling and masks for reliable correspondence sampling.
- Pose estimation: Relative pose estimation samples high-scoring flow correspondences, solves the fundamental matrix with normalized 8-point RANSAC, and selects the physically valid decomposition.
- Geometric filtering: Small ray angles and negative triangulation depths near epipoles are represented as white regions in the angle mask.
- Geometric filtering: Occlusion masks and inlier maps filter occlusions and non-rigid regions, respectively.
- Evaluation: Visual-odometry evaluations report average translation and rotation errors on sampled sequences 09 and 10 with strides 2 and 4.
F. Additional Comparison on sampled KITTI Odometry dataset
The sampled KITTI Odometry comparison evaluates robustness without additional training. The system generalizes better than baselines, but very fast motion exposes optical-flow correspondence as a bottleneck.
- The proposed system shows improved robustness and generalization ability compared with the evaluated baselines on sampled sequences.
- With stride 4 or faster camera motion, inaccurate optical correspondences make optical-flow estimation the bottleneck and degrade performance.
G. Numerical Results of TUM-RGBD dataset
On TUM-RGBD and cross-dataset indoor transfer, the method produces reasonable trajectories and transfers better than PoseNet, while remaining sensitive to scene texture and retaining room for improvement.
- The method produces reasonable TUM-RGBD trajectories under challenging scenarios, whereas the PoseNet baseline fails to generalize.The comparison is reported for selected TUM-RGBD sequences.
- ORB-SLAM2 performs better on richly textured scenes, while the proposed system is less affected by large textureless regions.The passage contrasts fr2/desk and fr3/str tex far with fr3/cabinet and fr3/str ntex far.
- AbsRel 0.276 versus 0.324 shows better NYUv2 transfer for the proposed model than the PoseNet baseline after TUM-RGBD training.The reported transfer ability remains limited and has substantial room for future improvement.
- Figures 14 and 15 provide additional qualitative KITTI and NYUv2 results, including baseline, ours, and ground-truth visualizations.