PaperlayerView exact revision on arXiv

Source-linked AI summary

FIFO: Learning Fog-invariant Features for Foggy Scene Segmentation

Sohyun Lee, Taeyoung Son, Suha Kwak

arXiv:2204.01587v1cs.CV

TL;DR

Fog reduces segmentation performance, and annotated real-fog scenes are difficult to collect. FIFO uses learned fog factors to align representations across fog conditions; it outperforms prior work on foggy scenes while improving performance on clear scenes too.

  • Problem

    Fog reduces image visibility and segmentation performance, while collecting and labeling fully annotated real-fog scenes is difficult.

    Source: §1 · p. 1

  • Method

    FIFO learns fog factors from neural style representations and alternately trains the filters and segmentation network to align factors across fog conditions.

    Source: §1 · pp. 1–2

  • Results

    FIFO outperforms previous work and improves performance on both foggy and clear-weather domains, while existing methods often degrade clear-scene performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    FIFO enables end-to-end training without extra modules or human intervention for training.

    Source: §6 · p. 8

    Real-fog images are unlabeled, so their segmentation predictions lack ground-truth labels during training.

    Source: §4.2 · p. 5

Abstract

from arXiv · show

Robust visual recognition under adverse weather conditions is of great importance in real-world applications. In this context, we propose a new method for learning semantic segmentation models robust against fog. Its key idea is to consider the fog condition of an image as its style and close the gap between images with different fog conditions in neural style spaces of a segmentation model. In particular, since the neural style of an image is in general affected by other factors as well as fog, we introduce a fog-pass filter module that learns to extract a fog-relevant factor from the style. Optimizing the fog-pass filter and the segmentation model alternately gradually closes the style gap between different fog conditions and allows to learn fog-invariant features in consequence. Our method substantially outperforms previous work on three real foggy image datasets. Moreover, it improves performance on both foggy and clear weather images, while existing methods often degrade performance on clear scenes.

1. Introduction

Fog degrades segmentation, while labeled real-fog data are difficult to collect; FIFO treats fog as image style and learns features insensitive to fog variation. It outperforms prior methods while improving performance on foggy and clear scenes.

1. Introduction

  • Fog seriously reduces visibility and segmentation performance, while large fully annotated real-fog datasets are difficult to collect and label.Source: §1 · p. 1
  • Prior methods use synthetic fog and curriculum adaptation, but require fog-density control and often degrade clear-scene performance.These approaches additionally introduce computation and hyper-parameters.Source: §1 · p. 1
  • FIFO treats fog condition as image style and closes style discrepancies across clear, synthetic-fog, and real-fog domains to learn fog-invariant features.Clear-weather and synthetic-fog images are labeled; real-fog images are unlabeled.Source: §1 · pp. 1–2
  • Ordinary neural styles also reflect capture conditions and image content, so directly matching them can alter content and yield suboptimal solutions.Source: §1 · p. 2
  • Learnable fog-pass filters extract fog factors from neural styles, and alternating optimization aligns factors across domains alongside segmentation training.The filters are auxiliary training modules, not required at testing.Source: §1 · p. 2
  • FIFO outperforms previous work and improves performance on both foggy and clear-weather domains, where existing methods often degrade clear-scene performance.It also avoids controlling fog-density levels of synthetic and real images during training.Source: §1 · p. 2

2. Related Work

Prior fog-segmentation methods combine synthetic fog with curriculum adaptation, while dehazing can be computationally heavy and often does not improve recognition. FIFO relates to style-based robustness and feature-level domain adaptation but uses learned fog factors.

Semantic Foggy Scene Segmentation

  • Prior fog-segmentation methods simulate fog on labeled clear images and use curriculum learning to adapt from light synthetic fog to dense real fog, but may hurt clear-weather performance.They also require extra modules to control real-image fog density.Source: §2 · p. 2

Image Dehazing

  • Dehazing models restore clean images but are often too computationally heavy for recognition pipelines, and many do not improve foggy-scene recognition.Source: §2 · p. 2

Robustness; Unsupervised Domain Adaptation

  • FIFO resembles RobustNet and feature-level domain adaptation, but explicitly extracts fog-relevant factors rather than relying on generic style or domain-discriminator matching.The paper states that matching fog factors is more effective than fooling a fog-domain classifier for fog-invariant learning.Source: §2 · p. 3

Style Transfer

  • Gram matrices represent neural style and support domain adaptation, but can reflect factors beyond fog or image content [39].FIFO therefore uses fog-pass filters to isolate fog-relevant information.Source: §2 · p. 3

3. Configuration of Training Data

FIFO trains on clear-weather, synthetic-fog, and real-fog domains, using labeled images for the first two and unlabeled real-world images for the third. It uses fixed-density synthetic fog and the entire real-fog dataset rather than density-based subsets.

3. Configuration of Training Data

  • The three training domains are clear weather (CW), synthetic fog (SF), and real fog (RF).Source: §3 · p. 3
  • Cityscapes supplies labeled CW images, Foggy Cityscapes-DBF supplies labeled SF images, and Foggy Zurich supplies unlabeled real-world RF images.Source: §3 · p. 3
  • FIFO fixes synthetic-fog attenuation at β = 0.005 and uses the full Foggy Cityscapes-DBF and Foggy Zurich datasets, unlike prior density-varying or subset-based training.Prior methods divide Foggy Zurich by estimated density and vary synthetic fog levels.Source: §3 · p. 3

4. Proposed Method

FIFO alternates fog-pass filter and segmentation-network training to reduce differences between fog domains in feature space. The empirical analysis reports reduced fog-style gaps across all three domain pairs and sharper, less artifact-prone reconstructions.

4. Proposed Method

  • FIFO alternates fog-pass-filter and segmentation-network training, first training filters alone for 5K iterations to avoid cold-start.The filters are frozen during segmentation-network training and are used only for training, adding no inference-time complexity.Source: §4 · p. 3

4.1. Fog-pass Filtering Modules

  • Fog-pass filters map upper-triangular Gram-matrix entries to fog factors, grouping same-domain factors and separating factors from different domains.The Gram matrix captures correlations between feature channels; using its upper triangle avoids redundant entries because the matrix is symmetric.Source: §4.1 · p. 4

4.2. Segmentation Network

  • The segmentation network combines pixelwise segmentation supervision and fog-style matching; only same-origin CW–SF pairs also receive prediction-consistency loss.CW and SF counterparts share the same semantic layout, supporting consistency between their predicted segmentation maps.Source: §4.2 · pp. 4–5

4.3. Empirical Verification

  • FIFO closes fog-style gaps across all three domain pairs, and its learned features yield sharper reconstructions, clearer object boundaries, and fewer artifacts.The gap is measured by average Hausdorff distance between sets of fog factors.Source: Figure 4 · p. 5, §4.3 · pp. 5–6, Figure 5 · p. 6

4.2. Segmentation Network

  • Real-fog images contribute to fog-style matching but receive no segmentation loss because their segmentation labels are absent.For domain pairs involving RF, the objective includes segmentation loss on the labeled image and fog-style matching.Source: §4.2 · p. 5

5. Experiments

FIFO outperforms prior methods on the real-fog datasets and improves performance on clear-weather images. Its experiments examine the evaluation setup, qualitative predictions, comparisons with UDA, and the contributions of training choices and fog-pass filtering.

5.1. Implementation Details

  • FIFO uses RefineNet-lw with a ResNet-101 backbone and fog-pass filters on Conv1 and ResBlock1, without reconstructing images.The filters are implemented as multilayer perceptrons, and the segmentation model is trained with SGD.Source: §5.1 · p. 6

5.2. Datasets for Evaluation

  • Evaluation covers Foggy Zurich test v2, Foggy Driving, Foggy Driving Dense (a subset of Foggy Driving), and Cityscapes lindau 40 for clear scenes.The three foggy datasets contain fully annotated images with various fog densities.Source: §5.2 · p. 6

5.3. Quantitative Analysis

  • FIFO outperforms CMAda3+ [8] on all three real-fog datasets and enhances performance on clear-weather images.The authors suspect CMAda3+'s clear-weather drop may reflect overfitting to foggy scenes through catastrophic forgetting.Source: §5.3 · p. 6, Table 1 · p. 7

5.4. Qualitative Results

  • On real-fog examples, FIFO with fog-pass filtering gives the best segmentation; unfiltered FIFO beats the baseline, especially for cars, roads, and vegetation.Figure 6 compares input images, the baseline, FIFO without fog-pass filtering, FIFO, and ground truth.Source: §5.4 · p. 6, Figure 6 · p. 7

5.5. Comparison with UDA

  • The tested UDA methods underperform FIFO and CMAda3+, which the authors relate to fog style combining fog with other scene factors.Fog also increases within-domain variation because its effect depends on scene 3D configuration.Source: §5.5 · p. 7

5.6. Ablation Study

  • Using more domain pairs generally boosts performance, suggesting that CW–SF, CW–RF, and SF–RF pairs all contribute on real foggy images.CW, SF, and RF denote clear weather, synthetic fog, and real fog.Source: Table 2 · p. 8, §5.6 · p. 8
  • Fog style matching, prediction consistency, and fog-pass filtering all contribute, with style matching having the largest impact and fog factors outperforming Gram matrices.Table 3 compares FIFO variants with and without the losses and filtering modules.Source: Table 3 · p. 8, §5.6 · p. 8
  • β = 0.005 is the optimal tested attenuation coefficient for generating synthetic fog.FIFO uses a single β value rather than varying fog density during curriculum adaptation.Source: Table 4 · p. 8, §5.6 · p. 8

6. Conclusion

FIFO learns fog-invariant features by quantifying fog style and closing the gap between images from different fog conditions. It outperforms previous methods without sacrificing clear-weather performance and supports end-to-end training without an extra module or human intervention.

6. Conclusion

  • FIFO quantifies fog style and closes gaps between fog conditions, outperforming prior methods without sacrificing clear-weather performance or requiring an extra module or human intervention.The conclusion describes this as learning a segmentation network in fog-style space.Source: §6 · p. 8

A. Algorithm of FIFO

  • The training algorithm takes clear, synthetic-fog, and real-fog images as inputs and outputs an optimized segmentation network.The inputs include segmentation labels for clear-weather and real-fog images.Source: A. Algorithm of FIFO · p. 9
  • The stated total objective combines fog-pass filtering losses with fog-style matching, consistency, and segmentation losses.The objective is written as a fog-pass filtering term plus a minimization over the segmentation network.Source: A. Algorithm of FIFO · p. 9

B. Generalization to Other Weather Conditions

FIFO generalizes to rainy and frosty Cityscapes corruptions, remaining more robust than a clear-weather baseline as corruption severity increases. On ACDC, it outperforms the reported foggy-scene segmentation methods across all four conditions.

B. Generalization to Other Weather Conditions

C. Independence Analysis of Fog Factors

The independence analysis compares fog factors and Gram matrices by how independently they group from content factors. Fog factors are reported as more independent of content and more clustered by fog condition than Gram matrices.

C. Independence Analysis of Fog Factors

  • The content-pass module learns content-factor embeddings that group clear-weather–synthetic-fog pairs with the same content and separate other pairs.Its pairwise loss uses cosine distance and a margin.Source: C. Independence Analysis of Fog Factors · p. 10
  • The independence score averages, across images, the overlap between each image’s nearest fog-factor neighbors and nearest content-factor neighbors.The procedure selects k=200 similar factors for each representation.Source: C. Independence Analysis of Fog Factors · p. 10
  • Fog factors are more independent of content factors than Gram matrices, supporting the fog-pass module’s extraction of fog-relevant information apart from image content.The comparison is reported in Figure A2.Source: Figure A2 · p. 10, C. Independence Analysis of Fog Factors · p. 10

D. Effect of Fog Style Matching Loss

Bidirectional fog-style matching outperforms unidirectional matching when the same domain pairs are used, while Fog-to-Clear matching performs better on clear weather. Separately, fog factors are more clustered by fog condition than Gram matrices across the reported measures.

D. Effect of Fog Style Matching Loss

E. Impact of Fog Factors

  • Fog factors are more clustered by fog condition than Gram matrices under normalized mutual information and adjusted mutual information as well as the other reported measures.The comparisons appear in Figure A3 and Table A5.Source: E. Impact of Fog Factors · p. 11

F. Empirical Verification Using Evaluation Dataset

The supplementary evaluation reports consistency with the main-paper verification, reduced distances between deep features from different domains, and analysis of FIFO against UDA methods.

F. Empirical Verification Using Evaluation Dataset

G. Generalization on Deep Features

H. Comparison with UDA

  • FDA's low-frequency style transfer also transfers content, while FIFO outperforms DANN and avoids adversarial-training instability.The authors attribute FIFO's advantage to modeling within-domain fog-style variation while closing gaps between fog domains.Source: H. Comparison with UDA · p. 12, Figure A6 · p. 12

I. Comparison with Variants of CMAda

FIFO outperforms every reported CMAda variant, irrespective of curriculum stage count or fog densification.

I. Comparison with Variants of CMAda

  • FIFO outperforms all CMAda variants regardless of their number of curriculum stages or use of fog densification.In Table A6, a “+” marks models using additional fog densification for real foggy training datasets.Source: §I · p. 13, Table A6 · p. 13

J. Additional Qualitative Results

Additional qualitative examples show higher-quality FIFO segmentation in fog and improved image reconstruction, including on clear-weather images.

J. Additional Qualitative Results

  • FIFO produces higher-quality segmentation than the baseline across fog densities and datasets, particularly in dense-fog regions.Additional reconstruction examples also show clearer image quality in dense fog and clearer features in clear-weather images.Source: J. Additional Qualitative Results · p. 13, Figure A8 · p. 15, Figure A9 · p. 16
  • Additional qualitative examples also cover frost and rain corruptions.Figure A7 presents corrupted inputs, baseline and FIFO outputs, and ground truth.Source: Figure A7 · p. 14

Related papers

Loading 2204.01587v1…