PaperlayerView exact revision on arXiv

Source-linked AI summary

Model Adaptation with Synthetic and Real Data for Semantic Dense Foggy Scene Understanding

Christos Sakaridis, Dengxin Dai, Simon Hecker, Luc Van Gool

arXiv:1808.01265v1cs.CV

TL;DR

Semantic scene understanding remains underdeveloped for dense fog because recognition benchmarks and annotation-intensive training largely emphasize clear weather. The paper proposes CMAda, which progressively adapts segmentation models from light synthetic fog to dense real fog using synthetic and unlabeled real data, alongside semantic fog simulation and Foggy Zurich. The authors report that their simulation is competitive with a prior method and that curriculum adaptation significantly improves dense-fog SFSU performance.

  • Problem

    Semantic scene understanding methods and benchmarks mainly target clear weather, while collecting and manually annotating diverse adverse-weather images is difficult to scale.

  • Method

    CMAda progressively adapts segmentation models from light synthetic fog to dense real fog using synthetic and unlabeled real foggy data, supported by semantic fog simulation.

  • Results

    CMAda significantly improves state-of-the-art semantic segmentation models for dense real-fog SFSU, while the proposed fog simulation performs competitively with the prior simulation.

  • Takeaways & Limitations

    The work demonstrates that partially synthetic data and unlabeled real foggy data can progressively improve semantic segmentation in dense real fog.

  • Takeaways & Limitations

    The curriculum relies on complementary domain assumptions: dense synthetic fog has comparable visibility but artifacts, whereas light real fog captures spatial variation at a different density.

Abstract

from arXiv · show

This work addresses the problem of semantic scene understanding under dense fog. Although considerable progress has been made in semantic scene understanding, it is mainly related to clear-weather scenes. Extending recognition methods to adverse weather conditions such as fog is crucial for outdoor applications. In this paper, we propose a novel method, named Curriculum Model Adaptation (CMAda), which gradually adapts a semantic segmentation model from light synthetic fog to dense real fog in multiple steps, using both synthetic and real foggy data. In addition, we present three other main stand-alone contributions: 1) a novel method to add synthetic fog to real, clear-weather scenes using semantic input; 2) a new fog density estimator; 3) the Foggy Zurich dataset comprising $3808$ real foggy images, with pixel-level semantic annotations for $16$ images with dense fog. Our experiments show that 1) our fog simulation slightly outperforms a state-of-the-art competing simulation with respect to the task of semantic foggy scene understanding (SFSU); 2) CMAda improves the performance of state-of-the-art models for SFSU significantly by leveraging unlabeled real foggy data. The datasets and code are publicly available.

1 Introduction

The paper targets semantic scene understanding under dense fog, where existing recognition benchmarks and annotation-heavy training practices largely focus on clear weather. It combines semantic fog simulation, synthetic and real foggy data, and curriculum adaptation to improve SFSU without extra human annotations.

  • Dense fog severely reduces scene visibility, making semantic understanding important for outdoor applications such as automated driving.
  • Clear-weather benchmarking and the difficulty of collecting and annotating adverse-weather images limit straightforward extension of standard semantic-understanding training.
  • The method combines high-quality synthetic fog with many unlabeled real foggy images to improve SFSU without requiring extra human annotations.
  • The proposed fog simulator adds synthetic fog to clear real scenes using a semantic-aware filter that exploits object-instance structure.
  • CMAda progressively adapts a segmentation model from light synthetic fog to dense real fog in multiple steps using synthetic and real foggy data.
  • The paper contributes an automatic scalable fog-generation pipeline, a curriculum adaptation method, and Foggy Zurich with 3808 real foggy images, including 16 densely foggy images with fine annotations.

2 Related Work

The related work spans dehazing, foggy-scene recognition, and domain adaptation, while this paper addresses the domain gap by generating synthetic fog and applying gradual adaptation.

  • Prior dehazing research aims to increase visibility in foggy scenes, including road-specific and trainable end-to-end approaches.
  • Foggy-scene recognition research includes detection, traffic analysis, and pixel-level semantic segmentation, but mainly uses clear-weather data.
  • Existing fog-related work also studies fog detection, scene classification, and visibility estimation.
  • Domain-adaptation methods have transferred segmentation models across weather conditions and from simulated to real environments using pixel- and feature-level approaches.
  • This work addresses the domain gap through synthetic fog generation and gradual adaptation, while combining it with transfer-learning methods remains a proposed direction.

3 Fog Simulation on Real Scenes Using Semantics

The fog simulation pipeline replaces color-only depth postprocessing with a dual-reference filter using semantic and color structure, then applies the optical fog model to synthesize foggy images. This design aims to preserve semantic and depth edges while reducing erroneous texture transfer.

  • Motivation and pipeline: The pipeline adapts prior depth completion and filtering by using semantic labels alongside color as references for transmittance-map postprocessing.It begins with depth outlier detection, robust SLIC-superpixel plane fitting, and depth-map postprocessing, then replaces the prior guidance strategy with a semantic-aware design.
  • Dual-reference filtering: The dual-reference filter combines a semantic equality term with a CIELAB color term, allowing semantic boundaries and true depth edges to be preserved.Pixels sharing a semantic label contribute through the semantic term, while color similarity helps preserve depth edges that do not coincide with semantic boundaries.
  • Dual-reference filtering: Two separate bilateral grids implement semantic and color filtering independently, reducing computation relative to a single joint grid.The semantic and color domains use 3D and 5D grids, respectively, instead of one 6D grid.
  • Fog synthesis: The optical model synthesizes fog from clear-scene radiance, atmospheric light, and a complete transmittance map whose density is controlled by β.The pipeline completes an initial transmittance map from noisy disparity, filters it with the dual-reference filter, and uses the result for synthesis; larger β denotes denser fog.
  • Evaluation: At β = 0.02, corresponding to approximately 150 m visibility, the method generally preserves depth-discrepant object boundaries better than Foggy Cityscapes.The semantic boundaries reduce incorrect transfer of color textures into the transmittance map, improving the visual quality of the synthetic fog.

4 Semantic Segmentation of Scenes with Dense Fog

The paper addresses synthetic-to-real domain discrepancy in dense fog by adapting segmentation models through progressively harder fog conditions. CMAda combines labeled synthetic fog, weak labels from unlabeled real fog, and fog-density ranking.

  • Domain discrepancy: Synthetic-fog training improves segmentation but remains limited by the discrepancy between synthetic and real fog, especially at higher densities.The method therefore incorporates unlabeled real foggy data rather than relying only on synthetic images.
  • Curriculum Model Adaptation: CMAda uses light and dense synthetic fog together with unlabeled real foggy scenes spanning variable, unknown fog densities.Its curriculum moves from easier light-fog tasks toward dense real-fog scene understanding.
  • Curriculum Model Adaptation: The pipeline estimates fog density, ranks real images, trains on light synthetic fog, and generates noisy labels for lower-density real-fog images.These stages prepare supervisory information before adaptation to denser fog.
  • Fog density estimation: The fog-density estimator regresses directly to simulated β values using multiple Foggy Cityscapes versions, avoiding handcrafted fog features.Training uses β ∈{0, 0.005, 0.01, 0.02}, and predictions correlate well with subjective human density judgments.
  • Curriculum Model Adaptation: The final segmentation model learns from human annotations on dense synthetic fog and generated labels on light real fog.The formulation maps dense-fog inputs to semantic labels using both labeled and weakly labeled data.
  • Training objective: CMAda mixes synthetic and real weakly labeled images with relative real-image weight w = 1/3 during CNN training.The weighting balances contributions from the two data sources in the optimization.

5 The Foggy Zurich Dataset

Foggy Zurich provides a large collection of real foggy driving scenes and a finely annotated dense-fog test subset. Its unlabeled scale supports adaptation, while its labeled subset targets evaluation in particularly challenging dense fog.

  • Dataset collection: 3808 foggy images were collected from four Zurich-area driving video sequences at 1920×1080 resolution.Frames were sampled at one frame per second and manually cleaned to remove duplicates.
  • Dense-fog test set: 16 images with dense fog and diverse scenes form Foggy Zurich-test, with fine pixel-level annotations using Cityscapes evaluation classes.Uncertain or nonmatching pixels receive the void label.
  • Motivation: Foggy Zurich addresses the scarcity of foggy scenes in major road-scene datasets, where fog is rare and difficult to annotate.Existing datasets such as KITTI, Cityscapes, and Mapillary Vistas contain few or no foggy scenes.
  • Dataset significance: Compared with Foggy Driving's 101 annotated images, Foggy Zurich offers many more high-resolution foggy images and fine annotations for dense fog.Its larger unlabeled portion is intended for unsupervised or semisupervised methods, while Foggy Zurich-test supports dense-fog evaluation.
  • Data splitting: Images resembling Foggy Zurich-test scenes were excluded from training sets to support separate training and evaluation.The filtering was performed manually on the unlabeled portion.

6 Experiments

Experiments evaluate fog-density ranking, synthetic-fog adaptation, and CMAda on dense real-fog benchmarks. CMAda’s progressive use of synthetic and unlabeled real fog improves segmentation, while the proposed simulation remains competitive with Stereo-GF.

  • 6.1 Fog Density Estimation with Synthetic Data: 89.3% agreement between the fog-density ranking method and human judgment validates the estimator on 10400 annotations.The evaluation retained 87% of 4000 HITs after validation screening.
  • 6.2 Benefit of Adaptation with Our Synthetic Fog: RefineNet adapted with lighter synthetic fog consistently outperforms the clear-weather baseline and denser synthetic fog across the evaluated dense-fog benchmarks.The comparison uses Foggy Zurich-test and Foggy Driving-dense.
  • 6.2 Benefit of Adaptation with Our Synthetic Fog: Stereo-DBF beats Stereo-GF in most cases by a small margin and is consistently better for generating denser synthetic fog, although Stereo-GF with light fog is slightly better on Foggy Zurich-test.Both simulations are evaluated by fine-tuning RefineNet on Foggy Cityscapes-refined variants.
  • 6.3 Benefit of Curriculum Adaptation with Synthetic and Real Fog: CMAda-7 consistently improves upon CMAda-4 by combining dense synthetic fog with light real fog from Foggy Zurich-light.Foggy Zurich-light contains 1556 images selected using the density estimator.
  • 6.3 Benefit of Curriculum Adaptation with Synthetic and Real Fog: 5.9% and 7.9% mean-IoU improvements over the clear-weather baseline are achieved on Foggy Zurich-test for all and frequent classes, respectively.The best result uses the proposed simulation for dense synthetic fog in CMAda-7.
  • 6.3 Benefit of Curriculum Adaptation with Synthetic and Real Fog: Foggy Zurich-light also provides a generalization benefit on Foggy Driving-dense, which uses different camera sensors.The evaluated CMAda results are reported in Tables 3 and 4.

7 Conclusion

The paper combines semantic-aware synthetic fog, curriculum adaptation, and unlabeled real foggy data to improve semantic segmentation in dense real fog. It introduces Foggy Zurich and reports gains without labeled real foggy images, while the simulation performs competitively with state-of-the-art alternatives.

  • 7 Conclusion: The curriculum framework progressively improves state-of-the-art semantic segmentation using partially synthetic and unlabeled real foggy data.The reported improvements target dense real fog.
  • 7 Conclusion: The fog simulator uses scene semantics through a dual-reference cross-bilateral filter to generate synthetic fog on real clear-weather scenes.The method is applied to Cityscapes.
  • 7 Conclusion: The curriculum method significantly boosts dense-real-fog performance without using labeled real foggy images, and the fog simulation remains competitive with state-of-the-art counterparts.These conclusions follow the paper’s detailed evaluation.
Loading 1808.01265v1…