Source-linked AI summary
Feature-metric Loss for Self-supervised Learning of Depth and Egomotion
Chang Shu, Kun Yu, Zhixiang Duan, Kuiyuan Yang
TL;DR
Self-supervised depth and egomotion estimation is limited by photometric-loss landscapes that become ambiguous or poorly conditioned in textureless and less discriminative regions. The paper learns feature representations with derivative-based regularization to define feature-metric loss, improving KITTI depth estimation from 0.885 to 0.925 measured by δ1 and substantially improving visual odometry.
Problem
Photometric loss can create plateaus in textureless regions and multiple local minima for less discriminative pixels, limiting self-supervised depth and egomotion estimation.
Method
The method learns pixel features with FeatureNet and regularizes them using discriminative and convergent losses to shape feature-metric loss landscapes.
Results
0.885 to 0.925 measured by δ1 on KITTI, with better egomotion estimation and more accurate visual odometry than previous methods.
Takeaways & Limitations
Feature-metric loss achieves state-of-the-art depth estimation comparable to supervised methods and significantly outperforms previous state-of-the-art methods for visual odometry.
Takeaways & Limitations
Rotation estimation remains below traditional SLAM, which can optimize subtler rotation errors over long sequences; the reconstruction may also be insensitive to rotation variation.
Abstract
from arXiv · showhide
Photometric loss is widely used for self-supervised depth and egomotion estimation. However, the loss landscapes induced by photometric differences are often problematic for optimization, caused by plateau landscapes for pixels in textureless regions or multiple local minima for less discriminative pixels. In this work, feature-metric loss is proposed and defined on feature representation, where the feature representation is also learned in a self-supervised manner and regularized by both first-order and second-order derivatives to constrain the loss landscapes to form proper convergence basins. Comprehensive experiments and detailed analysis via visualization demonstrate the effectiveness of the proposed feature-metric loss. In particular, our method improves state-of-the-art methods on KITTI from 0.885 to 0.925 measured by $δ_1$ for depth estimation, and significantly outperforms previous method for visual odometry.
1 Introduction
Monocular depth and egomotion estimation is valuable but difficult, and photometric supervision leaves self-supervised methods substantially behind supervised approaches. The paper addresses this limitation with learned feature representations and reports improved depth and egomotion results.
- Monocular depth and egomotion estimation supports augmented reality, robotics navigation, and autonomous driving but remains challenging for conventional and deep-learning methods.
- Self-supervised methods remain substantially inferior to supervised ones because photometric loss can be ambiguous in textureless regions and produce over-smoothed boundaries.Smoothness losses propagate depth from discriminative to textureless regions, but their propagation is limited and can over-smooth boundaries.
- Feature-metric loss learns pixel features that remain discriminative in textureless regions through a single-view auto-encoding pathway.
- FeatureNet-generated features define the loss used by DepthNet and PoseNet, while the full architecture combines single-view and cross-view reconstruction.
- 0.885 to 0.925 measured by δ1 on KITTI, while feature-metric loss outperforms first-order and second-order smoothness baselines and improves egomotion and visual odometry.
2 Related Work
Related work addresses self-supervised monocular depth and egomotion through photometric supervision, motion handling, geometric priors, and learned reconstruction features. This paper differs by learning features directly from the geometry problem within the same self-supervised framework.
- SfMLearner established photometric supervision of DepthNet and PoseNet for monocular depth and egomotion estimation.
- Optical flow and segmentation masks are used in prior methods to handle moving objects that violate static-scene assumptions.
- Prior work strengthens self-supervision with depth-normal consistency, point-cloud 3D consistency, or binocular videos addressing scale ambiguity and scene dynamics.
- Deep feature reconstruction uses pretrained features, whereas this method explicitly learns deep features from the geometry problem under the same self-supervised framework.
3 Method
The method replaces problematic photometric supervision with learned feature-metric reconstruction, using regularized feature representations to improve optimization for monocular depth and egomotion.
- Cross-view reconstruction: Photometric loss can give small errors for incorrect depth or pose in textureless regions and produce inconsistent gradient directions that hinder convergence.Near-zero image gradients can yield near-zero optimization gradients for both depth and egomotion.
- Cross-view reconstruction: Feature-metric loss defines cross-view reconstruction on learned pixel features rather than directly on image intensities.The learned representation is intended to remain discriminative in textureless regions.
- Single-view reconstruction: FeatureNet learns features through single-view auto-encoding, combining image reconstruction with discriminative and convergent regularizers.The convergent loss penalizes second-order feature gradients to encourage smoother gradients and larger convergence radii.
- Overall pipeline: The framework combines DepthNet and PoseNet for cross-view geometry with FeatureNet for feature learning and feature-based reconstruction.DepthNet predicts target-frame depth, PoseNet predicts relative pose, and FeatureNet supplies features for the cross-view loss.
- Overall pipeline: The proposed feature-metric loss is combined with photometric loss and uses two source views to address occlusion.For monocular videos, the source views are the previous and subsequent frames; calibrated binocular training adds the opposite stereo frame.
- Overall pipeline: Qualitative comparisons report better results in low-texture regions and finer silhouettes and poles than Monodepth2.The reported examples include walls, billboards, human silhouettes, and poles.
4 Experiments
Experiments on KITTI evaluate feature-metric loss for monocular depth and visual odometry, including comparisons, online refinement, and ablations. The method improves low-texture predictions and produces smoother, more optimizable feature landscapes than photometric or smoothness-loss alternatives.
- Evaluation setup: The evaluation uses KITTI 2015 for depth and the official KITTI odometry split, with models trained on sequences 00–08 and tested on sequences 09–10 for odometry.Depth evaluation uses 697 testing samples; odometry uses 11 driving sequences.
- Depth evaluation: The method achieves the best performance among self-supervised methods across training-data settings, with particularly significant gains on Sq Rel.Sq Rel penalizes large short-range errors, where textureless regions are more common.
- Depth evaluation: Qualitative results show improvements over MonoDepth2 in low-texture regions and finer details, including walls, billboards, human silhouettes, and poles.The paper attributes this to feature-metric loss helping optimization escape photometric-loss local minima.
- Odometry evaluation: The method reaches top performance on two odometry metrics and comparable performance on the others, while translation is comparable to traditional SLAM but rotation remains weaker.The authors relate the rotation gap to bundle adjustment over long sequences and limited sensitivity to rotation variation in current reconstruction processes.
- Ablation study: Feature-metric loss gives the largest performance boost over first- and second-order smoothness baselines and supports long-range propagation through a larger convergence radius.It also benefits more from online refinement because discriminative features provide larger penalties for incorrect depth values.
- Ablation study: Combining reconstruction, discriminative, and convergent losses yields the best single-view reconstruction performance and a smooth slope-like feature landscape suited to gradient descent.Discriminative loss alone can create a disordered zigzag landscape, while the full combination produces a basin-like landscape.
5 Conclusion
The paper proposes an end-to-end self-supervised framework that learns feature representations for feature-metric depth and egomotion estimation. It reports state-of-the-art depth results comparable to supervised methods and improved visual odometry.
- Conclusion: Feature-metric loss learns feature representations with two regularizers to promote convergence toward correct depth and pose.The framework is end-to-end trainable in a self-supervised setting.
- Conclusion: The approach achieves state-of-the-art depth estimation that is comparable to supervised learning methods.This conclusion concerns the reported self-supervised framework.
- Conclusion: Visual odometry based on estimated egomotion significantly outperforms previous state-of-the-art methods.