PaperlayerView exact revision on arXiv

Source-linked AI summary

Curriculum Model Adaptation with Synthetic and Real Data for Semantic Foggy Scene Understanding

Dengxin Dai, Christos Sakaridis, Simon Hecker, Luc Van Gool

arXiv:1901.01415v2cs.CV

TL;DR

Semantic scene understanding remains difficult in fog because existing methods and benchmarks primarily address clear weather. The paper proposes CMAda, which progressively adapts segmentation from light synthetic fog to dense real fog using labeled synthetic and unlabeled real data; experiments report stronger results as adaptation stages increase, while performance remains below clear-weather results.

  • Problem

    Semantic scene understanding methods are mainly developed for clear-weather images, while fog creates increasingly severe visibility and annotation challenges for outdoor applications.

  • Method

    CMAda progressively adapts a segmentation model through fog-density-ordered domains using labeled synthetic fog, unlabeled real fog, and knowledge distillation.

  • Results

    CMAda variants outperform the clear-weather baseline at higher fog densities, with more adaptation stages producing increasingly accurate segmentation results.

  • Takeaways & Limitations

    Synthetic fog and unlabeled real fog can be combined in a curriculum framework to improve semantic segmentation in dense real fog.

  • Takeaways & Limitations

    Despite significant improvement, semantic segmentation on foggy scenes remains substantially worse than reported performance on clear-weather scenes.

Abstract

from arXiv · show

This work addresses the problem of semantic scene understanding under fog. Although marked progress has been made in semantic scene understanding, it is mainly concentrated on clear-weather scenes. Extending semantic segmentation methods to adverse weather conditions such as fog is crucial for outdoor applications. In this paper, we propose a novel method, named Curriculum Model Adaptation (CMAda), which gradually adapts a semantic segmentation model from light synthetic fog to dense real fog in multiple steps, using both labeled synthetic foggy data and unlabeled real foggy data. The method is based on the fact that the results of semantic segmentation in moderately adverse conditions (light fog) can be bootstrapped to solve the same problem in highly adverse conditions (dense fog). CMAda is extensible to other adverse conditions and provides a new paradigm for learning with synthetic data and unlabeled real data. In addition, we present three other main stand-alone contributions: 1) a novel method to add synthetic fog to real, clear-weather scenes using semantic input; 2) a new fog density estimator; 3) a novel fog densification method to densify the fog in real foggy scenes without using depth; and 4) the Foggy Zurich dataset comprising 3808 real foggy images, with pixel-level semantic annotations for 40 images under dense fog. Our experiments show that 1) our fog simulation and fog density estimator outperform their state-of-the-art counterparts with respect to the task of semantic foggy scene understanding (SFSU); 2) CMAda improves the performance of state-of-the-art models for SFSU significantly, benefiting both from our synthetic and real foggy data. The datasets and code are available at the project website.

1 Introduction

The paper targets semantic scene understanding in fog, where existing methods and annotations largely focus on clear weather. It combines synthetic fog, unlabeled real fog, and curriculum adaptation to improve dense-fog segmentation without extra real-fog annotations.

  • Motivation: Fog severely reduces visibility, while semantic scene understanding remains primarily benchmarked on clear-weather images.The challenge intensifies as fog density increases, motivating methods for adverse-weather outdoor applications.
  • Motivation: Manual annotation is difficult to scale for adverse weather, motivating limited-supervision, transfer-learning, and synthetic-data approaches.The paper positions its method between synthetic-data learning and adaptation using unlabeled real images.
  • Contributions: CMAda gradually adapts segmentation from clear weather through light fog to dense fog using labeled synthetic fog and unlabeled real fog.A fog-density estimator orders real images, while knowledge from easier conditions is reused for harder conditions.
  • Contributions: Experiments report that CMAda achieves the best results on two dense-real-fog datasets and one dataset with varying fog density.The method is iterative, supports different numbers of adaptation steps, and is presented as extensible to other adverse conditions.
  • Contributions: The paper additionally contributes semantic fog simulation, fog densification, model selection across weather conditions, and an enlarged Foggy Zurich annotated set.The dense-fog annotated subset is increased from 16 to 40 images.

2 Related Work

Related work spans defogging, semantic foggy-scene understanding, filtering, and domain adaptation. CMAda differs by jointly using synthetic and real fog data and by assigning real images to fog-density domains.

  • Scope: Prior research addresses fog visibility enhancement, road-scene recognition, fog detection, and semantic segmentation, but much recognition work focuses on clear weather.The paper situates foggy semantic understanding alongside defogging and broader road-scene recognition research.
  • Scope: Creating fine pixel-level annotations for dense foggy scenes is very difficult.This difficulty motivates methods that reduce reliance on human annotations for real foggy data.
  • Image filtering: The proposed filter extends cross-bilateral filtering with a discrete semantic-label reference alongside the color reference.The semantic reference helps preserve scene boundaries while the color reference preserves true depth edges.
  • Domain adaptation: CMAda complements adversarial adaptation methods by jointly adapting with synthetic foggy data and real foggy data.Its curriculum requires assigning real foggy images to target domains using fog-density estimation.

3 Fog Simulation on Real Scenes Using Semantics

The paper simulates fog on clear real scenes using a dual-reference cross-bilateral filter guided by color and semantics. The resulting data preserve semantic boundaries and support foggy-scene learning.

  • Dual-reference filtering: The simulation pipeline adapts prior depth completion and filtering steps, while retaining robust depth-plane fitting and adding semantic guidance.The semantic labeling is used as a reference rather than replacing the superpixel-based depth-plane fitting.
  • Dual-reference filtering: The dual-reference filter combines color and semantic references so semantic edges resist blurring while nonsemantic depth edges remain recoverable.Its semantic term weights pixels sharing labels, while its color term preserves true depth discontinuities.
  • Fog model: The fog simulator uses scene radiance, atmospheric light, and a complete transmittance map within the standard optical fog model.Transmittance depends on scene distance, and larger attenuation coefficients produce denser fog.
  • Results: Synthetic foggy images generally preserve edges between adjacent objects with large depth differences better than Foggy Cityscapes images.The comparison uses Cityscapes examples generated at β = 0.02, corresponding to visibility of approximately 150m.
  • Dataset: Applying the method to Cityscapes produces the publicly available Foggy Cityscapes-DBF dataset.The dataset applies semantic-guided fog simulation across the entire Cityscapes collection.

4 Semantic Foggy Scene Understanding

This section introduces supervised learning with synthetic fog and then CMAda, which combines synthetic and real foggy data for semantic segmentation under dense fog.

  • Overview: The section first evaluates standard supervised segmentation with synthetic fog before presenting CMAda with synthetic and real foggy data.This establishes the progression from single-source synthetic training to curriculum adaptation.

4.1 Learning with Synthetic Fog

The method generates synthetic fog on clear-weather scenes while preserving their existing human annotations, enabling segmentation-model training for foggy scenes.

  • Synthetic fog can inherit human annotations from clear-weather scenes, enabling standard segmentation-model training.The approach evaluates the fog simulator by adapting a clear-weather segmentation model to synthesized foggy images.
  • The learned model can operate standalone for semantic foggy scene understanding or initialize CMAda using synthetic and real data.

4.2 Curriculum Model Adaptation (CMAda)

CMAda progressively adapts segmentation models from clear weather through increasingly dense fog using labeled synthetic data and pseudo-labeled real images. It addresses growing domain discrepancies by ordering real images by fog density and extending adaptation to multiple stages.

  • Dense fog is difficult because synthetic-to-real and cross-density domain discrepancies increase as fog becomes denser.
  • A fog-density estimator trained on synthetic images with controlled attenuation coefficients ranks unlabeled real images for curriculum adaptation.The estimator regresses attenuation coefficient β and orders Foggy Zurich images by estimated density.
  • CMAda adapts segmentation models through ordered domains from clear weather to light and dense fog, with optional intermediate stages.The method supports single-stage, two-stage, and general multiple-stage adaptation.
  • At each stage, CMAda combines human-labeled synthetic fog matching the target density with pseudo-labeled real images from a less dense domain.The previous model supplies supervisory labels for real images whose fog density is below the preceding target density.
  • The initialization stage trains on labeled synthetic fog, after which later stages iteratively adapt toward the ultimate dense-fog target.CMAda1, CMAda2, and CMAda3 use progressively longer attenuation-coefficient sequences.

4.3 CMAda+ with Synthetic and Densified Real Fog

CMAda+ densifies real foggy images to reduce the remaining density gap between real training images and the target domain, while avoiding depth-based processing. The method uses estimated input density and controlled target density to create training data.

  • Real training images have lower fog density than the target, leaving a domain gap that fog densification is designed to reduce.
  • Fog densification estimates the input density, selects a higher target attenuation coefficient, and synthesizes a denser image without explicit depth.The output is computed from estimated transmittance values, the input image, and atmospheric light.
  • Figure 4 compares a Foggy Zurich input with its densified output and shows their separate R, G, and B histograms.
  • The method bypasses explicit clear-radiance recovery and dehazing, reducing inference to attenuation-coefficient estimation under globally constant transmittance.
  • Using one target fog density for every image could ignore true image-level variability and introduce additional domain discrepancies, so CMAda+ assigns targets independently.
  • The densified real dataset is trained jointly with the synthetic dataset using the same CMAda formulation.

4.4 Semantic Scene Understanding in Multiple Weather Conditions

The Model Selection extension handles mixtures of clear and foggy test images by choosing between specialized clear-weather and fog models. A weather classifier determines which segmentation model supplies the output.

  • Model Selection addresses changing weather by selecting an appropriate segmentation model for each encountered condition.
  • A two-class AlexNet classifier distinguishes clear weather from fog using Cityscapes and synthetic foggy images as training samples.
  • The system applies the clear-weather model when the classifier predicts clear weather and the fog-specialized model otherwise.The clear-weather and fog models are denoted φ1 and φZ, respectively.
  • The method can be generalized directly to multiple adverse conditions, including rain or snow.

5 The Foggy Zurich Dataset

Foggy Zurich provides a large collection of real foggy road scenes, with a densely fogged annotated test subset for semantic segmentation evaluation.

  • Data collection: The dataset was collected during car rides using a camera mounted inside the windshield to avoid severe lens-condensation blur and contrast degradation.Inside-windshield mounting produced sharper images with minimal windshield artifacts compared with outside mounting.
  • Dense-fog test set: A fog-density estimator ordered the images, after which 40 dense-fog scenes were manually selected to ensure diverse visual content for Foggy Zurich-test.The unlabeled images were filtered to exclude scenes resembling the test images from CMAda training sets.
  • Annotations: Person, rider, car, truck, bus, train, motorcycle, and bicycle instances receive separate annotations, additionally providing bounding boxes for these eight classes.The paper focuses on semantic segmentation despite these additional object-detection annotations.
  • Dataset significance: Foggy Zurich offers more high-resolution foggy images than Foggy Driving, while its large unlabeled portion supports unsupervised or semi-supervised learning.Foggy Driving has 101 annotated images, mostly with light fog and coarser annotations for dense fog.

6 Experiments

CMAda substantially improves semantic segmentation in dense fog by progressively adapting from synthetic and lighter fog to denser real fog. Multiple adaptation stages and fog densification provide further gains, although performance remains below clear-weather benchmarks and the method has unresolved scalability and self-training limitations.

  • Performance on Foggy Scenes: mIoU rises from 34.6% to 46.8% on Foggy Zurich-test and from 35.8% to 43.0% on Foggy Driving-dense versus clear-weather RefineNet.These improvements use no extra human annotations beyond Cityscapes and generalize across different camera sensors.
  • Benefit of Adaptation with Synthetic and Real Fog: Two-stage curriculum adaptation consistently outperforms single-stage adaptation using either only synthetic or only real foggy training data.This holds across fog simulation and fog-density estimation choices, supporting progressive adaptation from easier to harder conditions.
  • Benefit of Adaptation at Finer Scales: Adding a third adaptation stage improves all three test sets, including a 3.3% gain over the best CMAda2 model on Foggy Driving-dense.The authors connect finer-scale adaptation with improved performance but note that many stages increase computational cost and require selecting intermediate fog densities.
  • Qualitative Results and Discussion: Across increasing fog densities, CMAda variants increasingly outperform the clear-weather baseline, with more adaptation stages yielding larger benefits in heavier fog.In the most heavily fogged example, the bus is recognized only after all three adaptation stages.
  • Qualitative Results and Discussion: Despite substantial gains, foggy-scene segmentation remains much worse than reported clear-weather performance because fog introduces more variation and visibility-related difficulty.The authors identify additional real-fog annotations as a promising direction for obtaining significantly better results.

6.2 Performance in Multiple Weather Conditions

The paper evaluates semantic understanding across clear, foggy, and mixed-weather scenes. Model Selection combines clear- and fog-trained experts and achieves the strongest performance, especially on mixed weather.

  • CMAda adaptation to dense fog also benefits lower fog densities on the Foggy Driving dataset.
  • Model Selection with RefineNet and CMAda3+ yields the best performance, especially on the mixed Clear-Foggy-80 dataset.The comparison covers Cityscapes-lindau-40, Foggy Zurich-test, and their union.
  • Adapted models significantly improve recognition on foggy scenes, but adaptation can forget knowledge learned from clear-weather images.Adding clear-weather images can alleviate this issue, although the paper reports a cost in foggy-scene performance.

6.3 Investigating the Utility of Dehazing Preprocessing

The experiments compare dehazing preprocessing options before segmentation on dense-fog benchmarks. MSCNN provides the strongest dehazing baseline, but real dense fog remains difficult for these methods.

  • The comparison evaluates no dehazing, MSCNN, DCP, and Non-local preprocessing for RefineNet with and without fine-tuning.Fine-tuning uses Foggy Cityscapes-DBF with attenuation coefficient β = 0.005, matching the preprocessing used at testing.
  • MSCNN slightly enhances contrast with minor artifacts and outperforms DCP and Non-local dehazing for segmentation preprocessing.Figure 11 qualitatively illustrates the artifacts and contrast changes on Foggy Zurich-test images.

7 Conclusion

The paper combines semantic-aware fog simulation, fog-density estimation, and real foggy data in a curriculum adaptation framework. Experiments report gains on dense real fog without labeled real foggy images, alongside the Foggy Zurich dataset and public resources.

  • Curriculum adaptation combines partially synthetic and unlabeled real foggy data to progressively improve segmentation in dense real fog.
  • The paper introduces semantic-aware fog simulation using a dual-reference cross-bilateral filter and applies it to Cityscapes to create Foggy Cityscapes-DBF.
  • Foggy Zurich provides real foggy scenes with pixel-level semantic annotations for 40 dense-fog scenes, and the authors report that CMAda boosts performance without labeled real foggy images.
Loading 1901.01415v2…