DeFeat-Net: General Monocular Depth via Simultaneous Unsupervised Representation Learning

1. Introduction

• Existing monocular depth approaches tend to fail in adverse weather and nighttime conditions, where photometric consistency assumptions break down in dimly lit environments.Source: §1 · p. 1
• Feature-learning approaches can require pixel-wise ground-truth correspondences, which are difficult to obtain across seasons because alignment is unreliable and manual annotation is costly.Source: §1 · p. 1
• Depth estimation and feature representation complement one another: depth enables ground-truth feature matches across views, while robust features support depth estimation despite lighting and appearance changes.Source: §1 · p. 1
• DeFeat-Net jointly learns monocular depth, dense feature representations, and vehicle ego-motion using only a monocular image stream.The framework is entirely self-supervised and requires no ground-truth correspondences.Source: §1 · pp. 1–2
• Predicted depth and ego-motion produce correspondence maps for photometric losses, while those correspondences also serve as positive examples for metric learning and learned features provide a more robust loss at night.Source: §1 · pp. 1–2
• DeFeat-Net reports robust depth and invariant features across weather and lighting conditions, establishing state-of-the-art performance.Source: §1 · p. 2

2.1. Depth Estimation

• Depth estimation progressed from stereo correspondence and supervised disparity prediction to monocular deep learning and unsupervised methods using photometric warping, with added temporal, semantic, geometric, and cycle-consistency constraints.Recent methods still compute photometric losses in RGB, which makes cross-domain learning challenging.Source: §2.1 · p. 2

2.2. Feature Learning

• Feature learning spans hand-crafted descriptors, learned sparse detection and description, and dense representations trained with correspondence-based objectives.Some dense-feature methods target SLAM, localization, or optical flow rather than jointly learning features with monocular depth and odometry.Source: §2.2 · p. 3
• DeFeat-Net unifies monocular depth and odometry estimation with feature learning: depth-derived pixel correspondences support dense feature training without ground-truth labels, and learned-space match costs improve robustness in cross-domain scenarios.Source: §2.2 · p. 3

3. Methodology

• DeFeat-Net jointly learns monocular depth and dense features for robust estimates in adverse weather, while also learning ego-motion between consecutive frames as a byproduct.Its only ground-truth inputs are the original images and camera intrinsics.Source: §3 · p. 3, §3.3 · p. 4

3.1. Networks

• Complementary networks predict depth, pose, and dense features; the feature network produces a dense n-dimensional map, and the pose network predicts transforms between consecutive frames.The depth decoder also produces disparity maps at four scales.Source: Figure 2 · p. 4, §3.1 · p. 4

3.2. Correspondence Module

• Depth and predicted transforms backproject target-frame pixels into 3D and project them into support frames, producing correspondences for the photometric warp and pixel-wise contrastive loss.These correspondences provide sampling locations for the photometric warp and positive matches for feature learning.Source: §3.2 · p. 4

3.3. Losses

• Pixel-wise contrastive loss trains dense features from positive correspondences and spatially mined negative examples; photometric loss supports early training, while edge-aware smoothness regularizes predicted depth.The smoothness weighting is reduced at strong image-intensity gradients to avoid smoothing over edges.Source: §3.3 · pp. 4–5

3.4. Masking & Filtering

• Minimum reprojection selects correspondences with lower photometric error to filter occlusions, while automasking excludes pixels whose unwarped-frame error is lower than the warped-frame error.The minimum-error assumption is used when multiple consecutive support frames are available.Source: §3.4 · p. 5

4. Results

• RobotCar Seasons covers diverse seasons and weather, but has no depth ground truth; a manually created 12,000-image RobotCar validation set supplies LiDAR depth split evenly between day and night.The KITTI split contains only sunny daytime driving.Source: §4 · pp. 5–6

4.1. Single Domain Evaluation

• DeFeat-Net is competitive with MonodepthV2 and outperforms most other state-of-the-art monocular depth techniques on single-domain KITTI.The depth evaluation is reported in Table 1.Source: §4.1 · p. 6, Table 1 · p. 6
• Unsupervised DeFeat-Net features are competitive with contemporary supervised feature learning and outperform pretrained features in local evaluation.L2-normalizing pretrained features can improve their global performance.Source: §4.1 · p. 6, Table 2 · p. 6

4.2. Multi-Domain Evaluation

• Around 10% reduction in all error measures on RobotCar Night versus the previous state-of-the-art technique; daytime improvements are modest.The proposed approach outperforms that technique across all error measures on the more challenging RobotCar task.Source: §4.2 · p. 6, Table 3 · p. 7
• In well-lit conditions photometric loss already provides good supervision, while at night less discriminative photometric matching makes feature-based invariance and robustness valuable.This is the authors’ proposed explanation for the larger nighttime gains.Source: §4.2 · pp. 6–7
• The A1, A2, and A3 measures remain fairly consistent between day and night for DeFeat-Net, indicating graceful rather than catastrophic errors in uncertain-depth regions.The authors identify under-exposure and over-saturation as sources of uncertain depth.Source: §4.2 · p. 7
• Feature visualizations show scene-structure and road-position distinctions, while single-domain KITTI features can behave strangely around nighttime lights.The authors suggest broader appearance coverage may explain lower contrast in RobotCar Seasons features, and brighter training imagery may bias KITTI features around lights.Source: §4.2 · p. 7, Figure 4 · p. 8

4.3. Ablation

• Removing concurrent feature learning has a small, inconsistent effect on KITTI and RobotCar Day, but the full approach performs markedly better on RobotCar Night.For many error measures, performance doubles; the reduction in outliers brings a three-fold reduction in Sq-Rel error.Source: §4.3 · p. 8, Table 4 · p. 8

• DeFeat-Net unifies robust monocular depth estimation and dense feature representations, functioning across a wide range of appearance domains.Source: §5 · p. 8
• The framework learns feature representations without explicit ground truth, and the authors suggest co-training may apply beyond monocular depth estimation.Source: §5 · p. 8
• Feature consistency cannot currently be enforced across seasons; whether cross-season feature matching is possible remains unclear.Source: §5 · p. 8