Source-linked AI summary
Self-Supervised Monocular Depth Estimation with Internal Feature Fusion
Hang Zhou, David Greenwood, Sarah Taylor
TL;DR
Monocular depth estimation needs spatial and semantic representations, yet self-supervised geometry is vulnerable to difficult pixels and separate semantic models can undermine its advantages. DIFFNet builds on HRNet with internal multi-resolution feature fusion and decoder attention, outperforming state-of-the-art methods on KITTI, including at higher resolution, while introducing challenging-case evaluation.
Problem
Self-supervised depth estimation needs accurate spatial and semantic representations, but SfM violations and separate semantic networks limit existing approaches.
Method
DIFFNet adapts HRNet for depth estimation using multiple-resolution feature fusion and an attention mechanism in its encoder-decoder network.
Results
DIFFNet outperforms state-of-the-art self-supervised methods on KITTI, with further accuracy gains at higher input resolution.
Takeaways & Limitations
The method shows potential for high-resolution self-supervised depth estimation and motivates evaluating robustness on empirically selected difficult cases.
Takeaways & Limitations
The self-supervised framework assumes a static world and view changes caused only by a moving camera.
Abstract
from arXiv · showhide
Self-supervised learning for depth estimation uses geometry in image sequences for supervision and shows promising results. Like many computer vision tasks, depth network performance is determined by the capability to learn accurate spatial and semantic representations from images. Therefore, it is natural to exploit semantic segmentation networks for depth estimation. In this work, based on a well-developed semantic segmentation network HRNet, we propose a novel depth estimation network DIFFNet, which can make use of semantic information in down and upsampling procedures. By applying feature fusion and an attention mechanism, our proposed method outperforms the state-of-the-art monocular depth estimation methods on the KITTI benchmark. Our method also demonstrates greater potential on higher resolution training data. We propose an additional extended evaluation strategy by establishing a test set of challenging cases, empirically derived from the standard benchmark.
1 Introduction
Self-supervised monocular depth estimation uses image-sequence geometry to avoid costly labelled depth data, but SfM violations and reliance on separate semantic networks remain challenges. DIFFNet addresses these issues by integrating semantic information through internal feature fusion and attention, achieving strong benchmark performance.
- Self-supervised methods exploit scene geometry in sequential images, reducing dependence on costly labelled depth data.
- SfM-based supervision is vulnerable to low-texture regions, occlusions, and moving objects that violate its assumptions.
- Semantic information complements geometry because semantic and depth networks both depend on accurate object boundaries.
- Many prior methods require a separately trained semantic network, weakening self-supervision and introducing domain-gap concerns.
- DIFFNet applies internal feature fusion and decoder attention to built-in semantic information, advancing KITTI results and enabling difficult-case evaluation.
2 Related Work
Self-supervised monocular depth estimation has developed from SfM-based depth-and-pose learning, while related work increasingly incorporates semantic information and redesigned representations. These approaches address geometric ambiguity, invalid SfM pixels, and representation quality through losses, auxiliary models, and feature fusion.
- Monocular depth estimation is ill-posed because multiple 3D depth configurations can project onto the same image pixel.
- SfM-based self-supervision jointly trains depth and pose networks from sequential video frames, forming the basis for many later methods.
- Semantic approaches either add constraints to photometric losses or identify pixels that violate the static-world assumption.
- Other methods use semantic information for representation learning, including consistency objectives and pretrained semantic networks.
- Feature-fusion approaches redesign encoder-decoder skip connections, while multi-task models share encoders across geometry and semantic representations.
3 Self-supervised Monocular Depth Estimation Framework
The framework trains depth and pose models from sequential RGB frames, synthesizing the target view through predicted geometry and camera motion. Photometric reconstruction and edge-aware smoothness provide supervision, supplemented by masking and multi-scale losses.
- The framework jointly trains a depth model and pose model using target and neighboring sequential RGB frames.
- It assumes a static world and camera-induced view changes when synthesizing the target frame from source-frame pixels.
- Known camera intrinsics and predicted depth support reprojection and projection operations that generate a synthesized target view.
- Photometric loss compares the synthesized and target images using structural similarity and L1-based appearance matching.
- Edge-aware smoothness regularizes depth in low-gradient regions, while auto-masking, minimum photometric error, and multi-scale loss refine supervision.
- The objective averages the combined photometric and smoothness losses across pixels, pyramid scales, and image batches.
4 DIFFNet
DIFFNet combines HRNet’s multi-resolution representations with internal feature fusion and attention-based decoding for monocular depth estimation. The design enriches feature maps across stages and processes encoder skip connections while restoring depth features at multiple scales.
- DIFFNet combines multiple-resolution feature fusion and a spatial attention mechanism in an encoder-decoder depth network.
- High-Resolution Depth Encoder: HRNet maintains high-resolution representations through multiple streams and stages that exchange information across resolutions.Each stream contains feature maps at a shared resolution, while stages operate at different resolutions.
- High-Resolution Depth Encoder: Using HRNet as the depth encoder yields significant improvements over approaches using ResNet encoders.The HRNet encoder has four streams and four stages and outputs five feature maps at different scales.
- High-Resolution Depth Encoder: DIFFNet stacks multi-stage feature maps to give low-level features more semantic information without changing their scale.The feature stack is formed through concatenation, and its benefit is examined through ablation.
- Attention-based Depth Decoder: The decoder applies attention to encoder skip connections and uses upsampling, concatenation, and 3×3 convolution to restore feature maps.The decoder architecture is based on U-Net and processes features at multiple scales.
- Attention-based Depth Decoder: Among channel-wise, spatial, and channel-spatial attention, channel-wise attention gives the best performance.The attention module generates channel or spatial attention maps and combines them through element-wise multiplication.
5 Experiments
Experiments evaluate DIFFNet on KITTI through benchmark comparisons, ablations, qualitative visualizations, and a challenging-case test set. DIFFNet outperforms competing methods while producing detailed depth maps with fewer parameters and performs competitively on difficult images.
- 5 Experiments: KITTI training uses 39,810 monocular frame triplets, with 4,424 validation triplets and an assumed universal camera intrinsic matrix.The intrinsic assumption is valid only when the capturing cameras are similar.
- 5 Experiments: DIFFNet outperforms state-of-the-art approaches on KITTI Absolute Relative Error and RMSE, achieving the best results on all metrics with stereo training.At 1024×320 resolution, accuracy further increases while DIFFNet continues to outperform competing methods.
- 5 Experiments: DIFFNet provides greater detail for roadside items than contemporary methods while using fewer trainable parameters.Figure 4 compares DIFFNet with PackNet, HR-depth, and Monodepth2.
- 5 Experiments: The ablation study progressively evaluates ImageNet pre-training, multi-stage fusion, channel-wise attention, and space-wise attention.The largest performance gain comes from encoder pre-training, while channel-wise attention also increases accuracy.
- 5 Experiments: The challenging test set unions the ten highest-error images from four approaches, yielding 23 images with three common hard cases.DIFFNet achieves the lowest Absolute Relative Error on this difficult test set; foliage and difficult lighting are hypothesized to contribute to the challenge.
- 5 Experiments: More semantic information improves predicted depth precision, and DIFFNet produces fewer artefacts than the baseline method.These effects are illustrated in the ablation visualization.
6 Conclusion
The paper proposes DIFFNet, an HRNet-based self-supervised monocular depth estimator using multi-resolution feature fusion and channel attention. It reports stronger performance with fewer trainable parameters, especially at high resolution, and introduces difficult-case evaluation.
- 6 Conclusion: DIFFNet adapts HRNet for self-supervised monocular depth estimation with multi-resolution feature fusion and channel attention.The encoder computes semantically rich feature maps, and ablations show gains from the proposed modifications.
- 6 Conclusion: DIFFNet outperforms other state-of-the-art self-supervised methods with fewer parameters, especially when high-resolution input is available.
- 6 Conclusion: The paper introduces difficult-test-case evaluation by investigating challenging images and inviting future authors to adopt the strategy.