PaperlayerView exact revision on arXiv

Source-linked AI summary

Weather-Conditioned Depth Anything

Zhaoming Xu, Chan-Wei Hu, Kuan-Ru Huang, Zihao Zhu, Renjie Li, Yang Zhou, Zhengzhong Tu

arXiv:2609.04827v1cs.CV

TL;DR

Depth foundation models remain brittle under adverse weather because degradation alters appearance and geometric cues. DA-W disentangles weather style from scene content with a Style Filter and lightweight conditioning of a frozen Depth Anything backbone. Experiments report state-of-the-art performance across general and adverse-weather benchmarks while preserving clean-domain generalization.

  • Problem

    Depth foundation models degrade under fog, snow, rain, and nighttime conditions, while weather and scene geometry become entangled in their representations.

  • Method

    DA-W learns domain-aligned weather embeddings from mixed real and synthetic data and injects them into a frozen Depth Anything backbone through lightweight AdaLN-Zero modulation.

  • Results

    DA-W achieves state-of-the-art performance across general and adverse-weather benchmarks without degrading generalization performance on clean data.

  • Takeaways & Limitations

    A unified weather-conditioned adaptation strategy can improve robustness across adverse-weather benchmarks while retaining clean-domain generalization.

  • Takeaways & Limitations

    Controlled adaptation experiments focus on DA-v2-S, and larger variants, other foundation models, and nighttime shifts beyond weather appearance remain challenging.

Abstract

from arXiv · show

Monocular depth estimation foundation models, such as the Depth Anything series, have achieved remarkable performance across diverse domains. However, they still suffer from critical failures under adverse weather conditions, such as fog, rain, snow, or at night. To address this, we present Weather-Conditioned Depth Anything (DA-W), a framework that explicitly disentangles style from content for weather-robust depth estimation. Specifically, we introduce a Style Filter trained on a curated mix of real and synthetic degradation datasets to extract content-independent, degradation-aware weather embeddings. This style embedding is then injected into the Depth Anything backbone using a parameter-efficient, zero-initialized adapter. Such a lightweight modulation allows a single unified model to robustly adapt to diverse conditions, including fog, rain, snow, and low-light, while avoiding catastrophic forgetting of its core generalization abilities in normal conditions. We train the adapter using a pseudo-label distillation and alignment strategy. Our comprehensive experiments demonstrate that our proposed DA-W achieves state-of-the-art robust depth estimation, improving AbsRel by an average of 3.7% on our curated weather benchmarks, while matching or slightly outperforming performance on standard clean benchmarks. Our project page is available at https://zhaoming-tamu.github.io/WCDA/.

1 Introduction

Existing depth foundation models degrade under adverse weather because weather appearance changes entangle with scene geometry. DA-W addresses this by learning weather embeddings from mixed real and synthetic data and injecting them into a frozen Depth Anything model, achieving strong adverse-weather performance while preserving clean-data generalization.

  • Adverse weather causes dramatic depth-estimation degradation through scattering, occlusion, specular artifacts, and low-light noise that alter image appearance.
  • Depth Anything features entangle weather-specific style with scene geometry, preventing separable weather representations and synthetic-to-real adaptation.
  • DA-W trains a Style Filter with contrastive learning on seven real adverse-weather datasets, four clean datasets, and corresponding synthetic weather views.
  • The learned weather embedding conditions a frozen Depth Anything backbone through lightweight AdaLN-Zero modulation, preserving clean-scene performance while adapting predictions to weather.
  • DA-W achieves state-of-the-art performance across general and adverse-weather benchmarks without degrading clean-data generalization.

2 Related Work

Depth estimation research has improved cross-domain generalization through larger, more diverse data and models, but adverse weather still corrupts geometric cues. Existing remedies either restore images before estimation or incorporate weather robustness directly into depth models.

  • Monocular depth estimation supports 3D perception in robotics, augmented and virtual reality, and autonomous driving, but early supervised and self-supervised methods had domain bias.
  • MiDaS, Depth Anything, ZoeDepth, UniDepth, and Metric3D improve generalization through broader datasets, larger models, or alternative depth-estimation designs.
  • Generative depth methods formulate prediction as conditional sampling, supporting uncertainty-aware inference and tasks such as depth inpainting.
  • Rain, fog, and nighttime noise remain challenging because they corrupt or remove geometric cues through scattering, occlusions, and low illumination.
  • Prior robust-depth approaches restore degraded images before estimation or train depth models with weather-aware objectives, but restoration can alter geometric cues.

3 Proposed Method

DA-W separates weather style from scene content with a Style Filter, then injects the resulting embedding into Depth Anything through lightweight decoder modulation. Its training combines teacher distillation with clean-synthetic alignment and augmentation consistency.

  • Weather Embedding: The Style Filter learns weather-discriminative embeddings that remain invariant to scene content and aligned between real and synthetic domains.Contrastive learning clusters same-weather images and separates different weather categories, while real-synthetic positive pairs share the embedding space.
  • Weather Adaptation: DA-W injects a 64-dimensional weather embedding at five decoder sites using lightweight AdaLN-Zero modulation.An MLP produces per-channel shifts, scales, and gates; zero initialization preserves the base model’s initial behavior.
  • Weather Adaptation: The modulation updates only the decoder head and small modulation heads, providing a parameter-efficient weather adapter for the pretrained backbone.This design targets weather robustness while preserving clean-domain behavior and mitigating catastrophic forgetting.
  • Training Data: The training data comprise clean images, geometry-preserving synthetic weather corruptions, and real weather images without clean counterparts.Synthetic views preserve the underlying geometry needed for paired supervision.
  • Training Objectives: A frozen teacher supplies pseudo-labels, while the student is trained with affine-invariant distillation, clean-synthetic pair alignment, and real-image augmentation consistency.The total objective combines L_dis, L_pair, and L_aug.

4 Implementation

The implementation curates real and synthetic weather data, trains the Style Filter with balanced sampling, and evaluates robustness across adverse-weather, mixed-weather, and clean-domain benchmarks. Qualitative comparisons assess structural detail and depth smoothness under synthetic degradations.

  • Dataset Curation: The adverse-weather collection covers approximately 15K images from seven real-world datasets spanning fog, rain, snow, and low-light conditions.Four clean datasets provide source images for corresponding synthetic weather views.
  • Dataset Curation: Stage I uniformly subsamples 20K clean images and generates paired synthetic weather corruptions to balance real and synthetic training data.The synthetic degradation recipe follows RobustSAM and preserves shared geometry between clean and corrupted views.
  • Qualitative Results: Qualitative comparisons show DA-W retaining clearer structural details and smoother depth transitions under synthetic weather degradations.Other methods exhibit noisy artifacts and depth discontinuities in highlighted regions.
  • Training: Stage II uses all real adverse-weather images and synthetic corruptions, masking invalid or sky regions during depth supervision.Provided validity masks are used when available; otherwise, sky masks are generated with Lang-SAM.
  • Training: The two-stage protocol trains a 64-dimensional Style Filter, then optimizes only the DPT decoder head and AdaLN-Zero layers at five decoder stages.Stage II runs for 20 epochs with AdamW, a 5e-6 learning rate, 0.01 weight decay, and batch size 16.
  • Evaluation Protocol: Evaluation uses AbsRel and δ1 on adverse-weather, mixed-weather, and clean-domain benchmarks.Clean benchmarks include KITTI, NYU-Depth v2, Sintel, ETH3D, and DIODE.

5 Experiment

Experiments show that DA-W improves robustness across real, synthetic, and mixed adverse-weather conditions while preserving clean-scene performance. Ablations attribute this balance to decoder-only adaptation, real-weather data, and weather-conditioned modulation, while nighttime domain shifts remain a limitation.

  • Real adverse weather: DA-W achieves the highest robustness within the DepthAnything family on NuScenes-night and DS-rain, while remaining competitive on DS-cloud and DS-fog.It outperforms DepthAnything v2 and DepthAnything-AC on both AbsRel and δ1 for NuScenes-night and DS-rain.
  • Synthetic corruptions: DA-W performs best on Dark, Snow, and Motion corruptions and remains competitive on Fog in KITTI-C.Compared with DepthAnything v2, it improves both reported metrics across all four settings, with the largest gain on Snow.
  • Clean benchmarks: DA-W preserves clean-scene behavior, with small consistent gains on KITTI and Sintel and parity on ETH3D.The results are consistent with decoder-side AdaLN and zero-initialized heads preserving the backbone’s clean-domain generalization.
  • Mixed weather: Adaptation transfers to coexisting degradations, producing fewer structural artifacts and more stable geometry under simultaneous low illumination and snowfall.The mix-weather evaluation includes synthetic compositions and qualitative Boreas night-plus-snow sequences without restoration preprocessing.
  • Ablations: The complete decoder-only configuration with real-world degradations and weather injection offers the best overall ablation trade-off.Decoder-only tuning improves NuScenes-night δ1 from 0.727 to 0.741 and supports clean-scene generalization, while full fine-tuning can improve real-night averages but not global performance.
  • Ablations: Real-weather data primarily benefits real adverse-weather domains, while AdaLN conditioning particularly improves nighttime robustness but slightly reduces DS-fog performance.A residual domain mismatch affects NuScenes-night, and not all synthetic corruptions benefit equally from explicit weather conditioning.
  • Feature analysis: Feature differences after AdaLN modulation correlate with degradation characteristics and propagate to the final depth predictions.Nighttime modulation is stronger in darker regions, while foggy scenes show stronger changes in texture-rich regions.
  • Limitations: The controlled adaptation experiments focus on DA-v2-S, and real nighttime scenes include non-weather shifts that remain challenging.These shifts include camera response, exposure dynamics, motion blur, and scene-distribution changes; larger backbones require matched adaptation heads and optimization settings.

6 Conclusion

The paper presents Weather-Conditioned Depth Anything for robust monocular depth estimation across adverse weather. Its Style Filter and AdaLN-Zero decoder adapt a frozen backbone using mixed real-and-synthetic data, achieving strong weather robustness without sacrificing clean-domain performance.

  • 6 Conclusion: Weather-Conditioned Depth Anything targets robust monocular depth estimation across diverse adverse-weather conditions.The model is designed for zero-shot depth estimation under weather degradations.
  • 6 Conclusion: The Style Filter learns domain-aligned weather embeddings, which an AdaLN-Zero decoder injects into a frozen Depth Anything backbone.Training uses distillation, pairwise alignment, and augmentation consistency with mixed real-and-synthetic weather data.
  • 6 Conclusion: Experiments on real and synthetic weather benchmarks show consistent gains over strong depth-foundation and restoration-based baselines without sacrificing clean-dataset performance.The paper presents the method as a direction for condition-aware depth foundation models.

A.1 Implementation of Style Filter Network

The Style Filter Network represents weather degradations as styles decoupled from scene content and learns weather-discriminative embeddings from Gram statistics and contrastive learning. It is trained on curated real and synthetic weather data to align their style distributions.

  • Style representation: The Style Filter treats weather degradations as weather-specific image styles decoupled from scene content and domain.Its operational goal is to produce weather-discriminative embeddings rather than entangled appearance-and-geometry features.
  • Network design: The network learns weather-discriminative embeddings using Gram matrix statistics and contrastive learning.It uses a two-scale patch-based transformer encoder with 4×4 patch tokenization.
  • Training data: Training combines seven real-world adverse-weather datasets with four synthetic weather datasets and corresponding synthetic degradations.The curated data are used to learn domain-aligned weather embeddings across real and synthetic distributions.

A.2 Dataset Curation

The dataset curation combines real-world adverse-weather sources spanning fog, rain, snow, and nighttime conditions with synthetic weather data. Selection emphasizes authentic degradations and coverage of challenging weather domains.

  • Real-world datasets: ACDC provides 4,006 adverse-condition images paired with normal-scene views across fog, nighttime, rain, and snow.Only its designated training split is used for Style Filter training.
  • Distribution analysis: The curated sources are visualized with t-SNE to examine real and synthetic dataset distributions.The figure presents the distributions used in the dataset-curation analysis.
  • Real-world datasets: MUSES contributes snow sequences from its training split to improve coverage of an underrepresented degradation.The dataset contains varying weather and illumination conditions with uncertainty-aware severity annotations.
  • Real-world datasets: NightCity contributes more than 4,000 authentic nighttime images spanning diverse urban illumination conditions.Its scenes include non-uniform artificial lighting, strong contrast variation, and low signal-to-noise ratios.
  • Real-world datasets: RID and RIS provide real rainy images from driving videos and surveillance cameras, respectively.RID contains 2,496 driving images, while RIS contains 2,048 images from 154 surveillance cameras.
  • Real-world datasets: RTTS supplies 4,322 real-world hazy images with varying fog densities and authentic atmospheric scattering.Its traffic scenes include object-detection annotations for five categories.
  • Real-world datasets: Snow100K contributes 1,329 realistic snowy photographs while excluding its 100,000 synthetic paired images.The selection prioritizes authentic weather degradations for Style Filter training.

B.1 Qualitative results under different degradation

Qualitative comparisons show that DA-W preserves finer details and sharper geometric structure under real and synthetic adverse-weather degradations. Its weather-aware conditioning is designed to address detail loss and diffuse boundaries observed in prior Depth Anything models.

  • Real-World Weather Degradations: DA-W preserves fine-grained details and robust depth structure across real-world snow, fog, rain, and low-light conditions.The qualitative comparisons cover real-weather degradation results against Depth Anything v1, v2, and DepthAnything-AC.
  • Comparison with Prior Models: Depth Anything v1 loses fine-grained details, while v2 produces discontinuous edges and DepthAnything-AC generates diffuse depth boundaries under adverse weather.These failure patterns are reported for both real-world and synthetic degradation comparisons.
  • Comparison with Prior Models: DepthAnything-AC’s uniform degradation modeling and clean-image distillation encourage conservative, smoothed estimates that weaken boundaries near high-frequency texture or depth discontinuities.The limitation is attributed to spatially homogeneous synthetic corruptions and distillation toward clean-image predictions.
  • DA-W Conditioning: DA-W disentangles weather style from scene content and modulates decoder features to preserve sharp boundaries and fine-grained details under challenging conditions.The method is presented as addressing the over-smoothing artifacts observed in prior approaches.
  • DA-W Conditioning: Weather-aware modulation becomes stronger for foggy and snowy scenes than for clear scenes and is spatially concentrated in severely affected regions.In the snowy example, dense snowflake regions receive stronger modulation than less-affected areas.

C.1 Visualization Results of Decoder Feature Modulation

The study compares prompt-driven and direct image-image CLIP alignment for weather embeddings, but both alternatives fail to align real and synthetic weather domains reliably. The visualizations identify semantic overlap and real-synthetic separation as the central failure patterns.

  • Alternative Alignment Methods: Prompt-driven and direct image-image CLIP alignment were evaluated as alternative approaches for learning weather embeddings.The alternatives use prompt-based image-text alignment or direct alignment of real-synthetic image pairs with the same weather condition.
  • Negative Results: Both CLIP-based variants failed to achieve satisfactory domain alignment despite extensive tuning across layers, prompts, margins, and alignment strategies.The reported sweeps included different CLIP encoder depths, prompt templates, margin settings, and both image-text and image-image objectives.
  • Prompt-Driven Alignment: Prompt-driven alignment causes text-adapted rain embeddings to overlap heavily with clean-weather embeddings, indicating confusion between rainy and clean conditions.Figure 12 visualizes text-adapted rain in blue and clean weather in pink.
  • Direct Image-Image Alignment: Direct image-image alignment separates synthetic rain from clean weather but leaves real rain in a distinct region, failing to bridge the real-synthetic domain gap.The result shows that purely visual alignment does not make synthetic and real rain embeddings coincide.
  • Analysis: CLIP features may be insufficient for weather-domain alignment because they emphasize high-level semantics rather than low-level physics-driven appearance variations.The cited analysis highlights atmospheric scattering and noise patterns as relevant real-versus-synthetic differences.
Loading 2609.04827v1…