Source-linked AI summary
Both Style and Fog Matter: Cumulative Domain Adaptation for Semantic Foggy Scene Understanding
Xianzheng Ma, Zhixiang Wang, Yacheng Zhan, Yinqiang Zheng, Zheng Wang, Dengxin Dai, Chia-Wen Lin
TL;DR
Foggy scene understanding is challenged by visibility degradation and limited annotated data, motivating transfer from clear scenes. CuDA-Net uses an intermediate domain to cumulatively adapt style, fog, and their combined factor; the paper reports state-of-the-art results on three SFSU datasets and generalization to rainy and snowy scenes.
Problem
Fog reduces visibility in semantic foggy scene understanding, and dense-fog images and annotations are scarce, motivating transfer from labeled clear images to unlabeled foggy images.
Source: §1 · p. 1Method
CuDA-Net uses an intermediate domain to separate style and fog gaps, then disentangles those factors and their combination with a cumulative loss.
Source: §1.1 · p. 2, §2.3 · p. 5Results
CuDA-Net outperforms state-of-the-art methods on three SFSU datasets and shows generalization ability in rainy and snowy scenes.
Source: §4 · p. 8, §3.3 · p. 8Takeaways & Limitations
The paper reports that two-step adaptation performs better than direct adaptation in rainy and snowy scenes, indicating potential for other adverse-scene understanding.
Source: §3.3 · p. 8The reported generalization to rainy and snowy scenes is based on primary experiments.
Source: §4 · p. 8
Abstract
from arXiv · showhide
Although considerable progress has been made in semantic scene understanding under clear weather, it is still a tough problem under adverse weather conditions, such as dense fog, due to the uncertainty caused by imperfect observations. Besides, difficulties in collecting and labeling foggy images hinder the progress of this field. Considering the success in semantic scene understanding under clear weather, we think it is reasonable to transfer knowledge learned from clear images to the foggy domain. As such, the problem becomes to bridge the domain gap between clear images and foggy images. Unlike previous methods that mainly focus on closing the domain gap caused by fog -- defogging the foggy images or fogging the clear images, we propose to alleviate the domain gap by considering fog influence and style variation simultaneously. The motivation is based on our finding that the style-related gap and the fog-related gap can be divided and closed respectively, by adding an intermediate domain. Thus, we propose a new pipeline to cumulatively adapt style, fog and the dual-factor (style and fog). Specifically, we devise a unified framework to disentangle the style factor and the fog factor separately, and then the dual-factor from images in different domains. Furthermore, we collaborate the disentanglement of three factors with a novel cumulative loss to thoroughly disentangle these three factors. Our method achieves the state-of-the-art performance on three benchmarks and shows generalization ability in rainy and snowy scenes.
1. Introduction
Foggy scene understanding is important for autonomous driving, but fog degrades visibility and foggy images are scarce and difficult to label. The paper motivates and proposes cumulative adaptation that separates style and fog gaps before addressing their combination.
1. Introduction
- Fog degrades semantic scene understanding, while sparse dense-fog images and annotations make transferring knowledge from labeled clear scenes to unlabeled foggy scenes important.Semantic foggy scene understanding is relevant to autonomous driving.Source: §1 · p. 1
- Existing adaptation methods align domains without accounting for the cause of the gap, while defogging and synthetic fog approaches have reported limitations.Defogging can introduce artifacts; synthetic fog may be unrealistic and widen the clear-to-foggy domain gap.Source: §1 · pp. 1–2
1.1. Motivation
- The authors hypothesize that an intermediate domain separates the mixed gap into style and fog gaps that can be closed separately, with the dual gap accumulating both.Their framework adapts style, fog, and then the combined dual factor.Source: §1 · p. 2, §1.1 · p. 2
- After adaptation with the intermediate clear domain, the style gap falls from 0.089 to 0.067, while the fog gap remains approximately unchanged at 0.037 versus 0.033.The paper uses mean variance values to represent domain performance and their differences to represent domain gaps.Source: §1.1 · p. 2
- The proposed CuDA-Net separates style and fog factors before jointly disentangling the dual factor, using a cumulative loss to model their relationship.The paper reports state-of-the-art performance on three SFSU datasets and generalization ability in rainy and snowy scenes.Source: §1.1 · p. 3
2. Method
CuDA-Net transfers segmentation knowledge from labeled clear source images to unlabeled foggy target images through an intermediate domain. It separates style, fog, and combined gaps using feature disentanglement networks and cumulative training.
2. Method
- An unlabeled intermediate domain m bridges source s and target t, whose mixed style-and-fog gap is decomposed into style and fog gaps.The method assumes s and m are clear, while m and t share a city; it trains separate networks for s→m, m→t, and s→t.Source: §2 · p. 3, §2.2 · pp. 4–5
2.1. Feature Disentanglement Networks
- Feature Disentanglement Networks (FDNs) separate domain-invariant content from domain-specific features, using shared content and private encoders with a shared decoder.Within-domain reconstruction and cross-domain translation supervise disentanglement; a segmentation head predicts heatmaps from content features using source labels.Source: §2.1 · p. 3
2.2. Style and Fog Decomposition
- The three networks address style, fog, and dual gaps in sequence, passing shared segmentation components forward and using Fs→m to pseudo-label unlabeled domain m.The method initializes Fm→t from Fs→m and then initializes Fs→t from Fm→t; the target domain is also unlabeled.Source: §2.2 · pp. 4–5
2.3. Cumulative Domain Adaptation
- Cumulative training penalizes deviations from the assumed relation that style and fog discrepancies sum to the dual discrepancy.After initialization, the method uses the cumulative loss as an auxiliary loss, alternately freezing fog or dual-factor encoders while training other modules; it cycles three times and uses the trained Fs→t content encoder and segmentation head for testing.Source: §2.3 · p. 5
3. Experiments
The experiments evaluate CuDA-Net across foggy-scene datasets, compare it with prior methods, and test its components and transfer to other adverse weather. Results report top performance against state-of-the-art methods, gains from synthetic fog data, and benefits of cumulative adaptation.
3.1. Datasets
- Evaluation uses clear Cityscapes as the source, manually selected Clear Zurich images as an intermediate dataset, and medium-fog Foggy Zurich as the target, with Foggy Zurich-test and Foggy Driving for testing.The request also includes the ACDC rainy and snowy subsets for generalization experiments.Source: §3.1 · pp. 5–6
3.2. Performance Comparison
- CuDA-Net with a DeepLab-v2 backbone achieves top performance, outperforming state-of-the-art methods; the paper also reports state-of-the-art results on ACDC.Table 1 reports mean IoU over all classes on Foggy Zurich and Foggy Driving.Source: Table 1 · p. 6, §3.2 · p. 7
- Comparisons cover backbone, defogging, domain-adaptation, synthesis, and combined methods, with domain-adaptation methods given the same amount of training data.Synthesis-based methods are fine-tuned using synthetic foggy images and corresponding clear-weather labels; defogging methods process real foggy test images before segmentation.Source: §3.2 · p. 6
- Adding 498 dense synthetic foggy images improves CuDA-Net by 0.9% on FZ; CuDA-Net+ beats CMAda3+ by 2.3% on FZ and 3.7% on FD.Both CuDA-Net and CuDA-Net+ also outperform CMAda3+ using less synthetic and real foggy data.Source: Table 2 · p. 6, §3.2 · p. 7
3.3. Discussion
- Ablations show style adaptation raises mIoU from 25.89 to 39.16, sequential fog adaptation adds 3.33, and dual-factor adaptation reaches 43.06.Sequential style-then-fog adaptation is 2.28 higher than direct style-and-fog transfer; cyclical training with T=2 adds 2.72, and L2 distance attains top performance among tested cumulative-loss metrics.Source: Table 3 · p. 6, §3.3 · p. 7
- Performance is not sensitive to λcum, set to 0.25 for its top result; manual selection of intermediate-domain images outperforms CNN-based selection.The selection comparison uses three random seeds.Source: Figure 5 · p. 7, Table 4 · p. 7, §3.3 · p. 8
- On rainy and snowy ACDC scenes, two-step adaptation outperforms direct source-to-target adaptation.The experiments train on the corresponding ACDC subsets and test on their validation sets.Source: §3.3 · p. 8, Table 5 · p. 8
4. Conclusion
CuDA-Net addresses semantic foggy scene understanding by disentangling style, fog, and their combined factors. It outperforms state-of-the-art methods on three widely used datasets, with reported generalization to rainy and snowy scenes.
4. Conclusion
- CuDA-Net outperforms state-of-the-art methods on three widely used semantic foggy scene understanding datasets and shows generalization ability to rainy and snowy scenes.The authors note that their rainy- and snowy-scene experiments are preliminary.Source: §4 · p. 8
- CuDA-Net is a cumulative style-fog-dual disentanglement domain adaptation method for semantic foggy scene understanding.The method assumes style, fog, and dual factors have a cumulative relationship.Source: §4 · p. 8
- The method uses DISE as its baseline, which the authors say could be replaced by stronger disentanglement-based domain adaptation methods.They believe this replacement could improve CuDA-Net's performance.Source: §4 · p. 8
- The authors report only primary rainy- and snowy-scene experiments and say further analysis is needed to verify the cumulative relationship in other adverse settings.Source: §4 · p. 8