PaperlayerView exact revision on arXiv

Source-linked AI summary

Regularizing Nighttime Weirdness: Efficient Self-supervised Monocular Depth Estimation in the Dark

Kun Wang, Zhenyu Zhang, Zhiqiang Yan, Xiang Li, Baobei Xu, Jun Li, Jian Yang

arXiv:2108.03830v2cs.CV

TL;DR

Self-supervised monocular depth estimation has strong daytime results, but nighttime low visibility and changing illumination challenge depth prediction. The paper combines prior-based regularization, mapping-consistent enhancement, and statistics-based masking to address these conditions. It reports significant improvement over current state-of-the-art methods and fewer weird nighttime depth outputs.

  • Problem

    Nighttime low visibility and varying illumination challenge self-supervised depth estimation, which has relatively few methods addressing these scenarios.

    Source: §1 · pp. 1–2

  • Method

    The framework combines unpaired-reference depth regularization, brightness-consistent image enhancement, and dynamic-statistics masking of textureless regions.

    Source: §1 · p. 2

  • Results

    The framework significantly outperforms current SOTA methods and greatly alleviates weird outputs in nighttime depth estimation.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The paper reports that its three improvements effectively address self-supervised nighttime depth estimation and reduce weird depth outputs.

    Source: §5 · p. 8

Abstract

from arXiv · show

Monocular depth estimation aims at predicting depth from a single image or video. Recently, self-supervised methods draw much attention since they are free of depth annotations and achieve impressive performance on several daytime benchmarks. However, they produce weird outputs in more challenging nighttime scenarios because of low visibility and varying illuminations, which bring weak textures and break brightness-consistency assumption, respectively. To address these problems, in this paper we propose a novel framework with several improvements: (1) we introduce Priors-Based Regularization to learn distribution knowledge from unpaired depth maps and prevent model from being incorrectly trained; (2) we leverage Mapping-Consistent Image Enhancement module to enhance image visibility and contrast while maintaining brightness consistency; and (3) we present Statistics-Based Mask strategy to tune the number of removed pixels within textureless regions, using dynamic statistics. Experimental results demonstrate the effectiveness of each component. Meanwhile, our framework achieves remarkable improvements and state-of-the-art results on two nighttime datasets.

1. Introduction

Self-supervised depth estimation performs well in daytime settings, but nighttime low visibility and changing illumination can produce holes and non-smooth depth maps. The paper proposes three improvements for nighttime estimation: prior-based regularization, mapping-consistent image enhancement, and a statistics-based mask.

1. Introduction

  • Nighttime low visibility creates textureless areas associated with large depth-map holes, while varying illumination breaks brightness consistency and is associated with non-smooth predictions.The paper links the resulting incorrect depth predictions to failure in training the depth network.Source: §1 · pp. 1–2
  • Priors-Based Regularization learns depth-distribution knowledge from unpaired references to constrain incorrect predictions and prevent incorrect training.The method also uses 2D coordinates to find depth distributions related to pixel location.Source: §1 · p. 2
  • Mapping-Consistent Image Enhancement extends CLHE to improve visibility while preserving brightness consistency across video frames.The paper notes that ordinary enhancement methods do not address correspondence among frames, which is essential for self-supervised depth estimation.Source: §1 · p. 2
  • Statistics-Based Mask uses dynamic statistics to flexibly tune the number of masked pixels in textureless regions.This addresses Auto-Mask's inability to adjust removed-pixel counts because it depends on photometric loss.Source: §1 · p. 2
  • The paper reports that the three contributions yield state-of-the-art nighttime depth estimation and reduce weird depth outputs.Source: §1 · p. 2

2. Related Work

Prior video-based self-supervised depth methods use reconstruction and geometry cues, but most work discussed focuses on daytime settings. Nighttime methods address domain transfer or thermal imagery, while existing enhancement methods generally do not preserve inter-frame brightness correspondence.

Self-supervised Depth Learning from Videos

  • Video-based self-supervised depth methods use reconstruction, optical flow or segmentation for moving objects, reprojection for occlusion, geometry priors, and feature representations, but the discussed methods focus little on nighttime scenarios.Source: §2 · p. 2

Nighttime Self-supervised Learning Methods

  • Nighttime depth approaches include thermal-image prediction, cross-domain feature learning, and daytime-to-nighttime adaptation, with the paper distinguishing its use of daytime depth distributions as training regularization.The paper says thermal images have less texture detail and limited resolution, and thermal cameras are expensive; it also says Defeat-Net does not tackle low visibility and varying illumination.Source: §2 · p. 2

Domain Adaptation in Depth Estimation

  • Depth-estimation domain adaptation uses source-domain knowledge, including geometry-based approaches and test-video adaptation, while this paper uses prior knowledge to regularize training.Source: §2 · p. 2

Low-light Image Enhancement

  • Low-light enhancement methods improve image brightness and contrast, but the discussed approaches do not account for brightness correspondence across frames needed for self-supervised depth estimation.Source: §2 · p. 3

3. Method

The framework adapts self-supervised view synthesis for nighttime depth estimation with priors-based regularization, mapping-consistent image enhancement, and statistics-based masking. Its training objective combines photometric and smoothness losses with PBR regularization.

3.1. Self-supervised Training

  • Self-supervised training reconstructs a target frame from source views using predicted depth and relative pose, optimizing photometric error and edge-aware depth smoothness.The photometric error combines L1 and SSIM; smoothness is intended to address depth ambiguity.Source: §3.1 · p. 3

3.2.1 Priors-Based Regularization

  • PBR uses an adversarial discriminator to regularize nighttime depth predictions toward unpaired reference depth maps, with pixel coordinates supplied as additional input.The reference maps are produced by a depth network trained self-supervised on daytime data; PBR is intended to flag outlier depths rather than instantiate a specific sample's depth.Source: §3.2.1 · p. 4

3.2.2 Mapping-Consistent Image Enhancement

  • MCIE applies one brightness mapping to target and source frames, improving visibility while preserving their brightness consistency for photometric supervision.Its mapping is computed by clipping high brightness frequencies, redistributing the clipped frequency, and using the cumulative distribution.Source: §3.2.2 · p. 4

3.2.3 Statistics-Based Mask

  • SBM uses an exponentially weighted moving average of target–source differences and a percentile threshold to tune masked pixels, improving masking of textureless regions.The final mask combines SBM with Auto-Mask, which can prevent large errors from being incorporated.Source: §3.2.3 · p. 5

3.2.4 Final Loss

  • The final objective combines enhanced-image photometric loss, edge-aware smoothness loss, and PBR discriminator and generator losses.The stated objective weights the terms with η, ξ, and τ.Source: §3.2.4 · p. 5

4. Experiment

The experiments evaluate the framework on RobotCar-Night and nuScenes-Night, compare it with existing methods, and assess the proposed components through ablations.

4.1. Dataset

  • RobotCar-Night contains more than 19k training sequences and 411 test samples; nuScenes-Night contains more than 10k training sequences and 500 test samples.The datasets use carefully generated depth ground truth and differ in their source datasets and test-split construction.Source: §4.1 · p. 5

4.2. Implementation Detail

  • The depth network uses a U-Net with a ResNet-50 encoder, while pose prediction uses ResNet-18 and PBR uses a Patch-GAN discriminator.Models were trained for 20 epochs on four RTX2080TI GPUs; evaluation capped depth at 40m for RobotCar-Night and 60m for nuScenes-Night.Source: §4.2 · pp. 5–6

4.3. Compare with SOTA Methods

  • 93.0% and 91.0% improvements over the baseline method are reported on RobotCar-Night and nuScenes-Night, respectively.Against PackNet, improvements are 87.1% and 93.8%; against FM, they are 93.1% and 90.9% on the two datasets, respectively.Source: §4.3 · p. 6, Table 1
  • Day-trained models score higher on the first four error metrics but worse on the last three accuracy metrics; PBR-only training does not work well.The authors therefore use photometric loss as the primary constraint and PBR loss as regularization.Source: §4.3 · pp. 6–7
  • Qualitative results contrast the method with SC-SfMLearner, MonoDepth2, and FM; the paper reports that these methods miss details and produce non-smooth maps.The proposed framework is reported to alleviate non-smoothness and produce higher-quality depth outputs.Source: §4.3 · p. 7, Figure 5 · p. 7

4.4. Ablation Study

  • Among individual components, PBR Only performs best, improving Sq Rel by 92.8% on RC and 90.2% on NS.SBM Only improves Sq Rel by 27.7% and 39.0%; MCIE Only improves it by 9.6% and 3.7% on RC and NS, respectively.Source: §4.4 · p. 7
  • Enabling more components improves performance: Full Method improves Sq Rel over PBR Only by 3.5% on RC and 8.1% on NS.Compared with PBR + MCIE, Full Method improves Sq Rel by 1.5% on RC and 5.3% on NS.Source: Table 2 · p. 8, §4.4 · p. 8
  • Removing the coordinates image from Full Method worsens Sq Rel by 11.6% on RC and 15.1% on NS.The authors interpret this as validating an association between image coordinates and depth distribution.Source: §4.4 · p. 8, Table 2 · p. 8

5. Conclusion

The conclusion presents three components for self-supervised nighttime depth estimation and reports improved performance over current SOTA methods.

5. Conclusion

  • Priors-Based Regularization uses prior depth distributions from reference maps to regularize model training.It is intended to prevent incorrect training.Source: §5 · p. 8
  • Mapping-Consistent Image Enhancement brightens and increases image contrast while maintaining brightness consistency in low visibility.Source: §5 · p. 8
  • Statistics-Based Mask uses dynamic statistics to flexibly remove pixels in textureless regions and mitigate depth ambiguity.The paper reports that the combined framework significantly outperforms current SOTA methods and alleviates weird nighttime depth outputs.Source: §5 · p. 8

A. Dataset Construction

The paper constructs two nighttime datasets from Oxford RobotCar and nuScenes, with depth ground truth prepared for evaluation.

A. Dataset Construction

  • RobotCar-Night and nuScenes-Night are built from video clips in Oxford RobotCar and nuScenes, respectively, with ground truth generated using official toolboxes.Source: A. Dataset Construction · p. 11
  • RobotCar ground truth combines multiple LiDAR frames using Structure-from-Motion, so the authors select scenes without moving objects and manually choose high-quality test outputs.This differs from prior work, which randomly samples test examples.Source: A. Dataset Construction · p. 11, Figure 8 · p. 11
  • nuScenes ground truth uses single-frame LiDAR data with more than 3,000 valid points per frame, and random sampling forms its test split.Daytime clips from both source datasets are also used to build RobotCar-Day and nuScenes-Day reference maps.Source: A. Dataset Construction · p. 11

B. Parameter Setting

The section explains parameter choices for MCIE and SBM and compares reference-depth scenarios for training.

B. Parameter Setting

  • Low-light noise motivates a smaller σ to avoid excessive noise enhancement, while darker scenes motivate a larger ϵ to mask more textureless pixels.The settings are (σ, ϵ) = (0.008, 10) for RC-N and (0.004, 20) for NS-N.Source: B. Parameter Setting · p. 11
  • On RC-N, σ and ϵ have little impact overall, and the framework performs best at σ = 0.008 and ϵ = 10.Figure 9 plots RMSE error against the parameter values; zero indicates the corresponding module is disabled.Source: B. Parameter Setting · p. 11, Figure 9 · p. 11
  • The suggested proper ranges are [0.002, 0.01] for σ and for ϵ.The paper recommends comparison tests to choose the best setting.Source: B. Parameter Setting · p. 11

C. Selection of Referenced Scene

  • Using RobotCar-Day rather than nuScenes-Day depth references yields slightly worse but similar performance on nuScenes-Night, while both reference scenarios outperform other SOTA methods.The authors conclude that depth maps from other scenarios can also regularize training.Source: C. Selection of Referenced Scene · p. 11, Table 3 · p. 11

D. Comparison with ADFA in Challenging Cases

The paper compares its method with ADFA in blurred, saturated, and very low-illumination cases. The method presents object shapes better in blurred and saturated regions, while low-light results range from plausible predictions to coarse or incomplete depth.

D. Comparison with ADFA in Challenging Cases

E. More Qualitative Result

The section presents qualitative comparisons on RobotCar-Night and nuScenes-Night against five established methods.

E. More Qualitative Result

F. Evaluation Metrics

The section evaluates depth predictions using seven standard metrics, covering error magnitudes and threshold-based accuracy.

F. Evaluation Metrics

  • Seven standard metrics are used: Abs Rel, Sq Rel, RMSE, RMSE log, δ1, δ2, and δ3.Source: F. Evaluation Metrics · p. 13
  • The δ_i metrics report the percentage of valid depth pixels whose predicted-to-ground-truth ratio is below the threshold 1.25^i.The threshold expression is shown in the metric definition.Source: F. Evaluation Metrics · p. 13
  • The error metrics use predicted depth d, ground-truth depth d∗, and the set D of valid ground-truth depth values.The passage provides formulas for Abs Rel, Sq Rel, RMSE, and RMSE log.Source: F. Evaluation Metrics · p. 13

G. More discussion on experiment results

The discussion contrasts error and threshold-accuracy metrics, then considers mixed-data training. MonoDepth2 (Day) and FM (Day) show metric trade-offs, while mixed-data variants improve on their baselines but remain behind Our.

G. More discussion on experiment results

  • MonoDepth2 (Day) and FM (Day) perform better on the first four error metrics but worse on the last three accuracy metrics than their counterparts.The proposed explanation is illustrated by MonoDepth2 versus MonoDepth2 (Day): the former has detailed outputs with large holes, while the latter is blurry without holes.Source: G. More discussion on experiment results · p. 13, Figure 15 · p. 14
  • The large holes in MonoDepth2’s output indicate very large depth values and contribute to higher average error, while its non-hole predictions are more accurate by the threshold metrics.The comparison concerns Abs Rel, Sq Rel, RMSE, RMSE log, and δ1, δ2, δ3.Source: G. More discussion on experiment results · p. 13
  • Mixed-data MonoDepth2 and FM perform better than their respective baselines but still have a large gap to Our.Table 4 reports quantitative results for mixed daytime and nighttime nuScenes data.Source: G. More discussion on experiment results · p. 14, Table 4 · p. 14

Related papers

Loading 2108.03830v2…