Source-linked AI summary
Semantic Foggy Scene Understanding with Synthetic Data
Christos Sakaridis, Dengxin Dai, Luc Van Gool
TL;DR
The paper addresses the limited study of semantic understanding in foggy scenes and the difficulty of obtaining annotated foggy images. It generates synthetic fog on real clear-weather images, trains models with supervised and semi-supervised strategies, and evaluates them on real foggy scenes, finding improvements from synthetic data and further gains from supervision transfer.
Problem
Semantic foggy scene understanding has received little attention, while collecting and annotating foggy images is difficult.
Method
The paper adds synthetic fog to real clear-weather images using incomplete depth information, then trains CNNs with supervised and semi-supervised learning using transferred clear-weather supervision.
Results
Synthetic data improve semantic foggy scene understanding on real foggy scenes, semi-supervised learning adds a 3.4% improvement, and dehazing provides no clear mean-IoU benefit.
Takeaways & Limitations
Partially synthetic Foggy Cityscapes data can boost state-of-the-art CNN performance for semantic segmentation and object detection on challenging real foggy scenes.
Takeaways & Limitations
Object-detection evaluation excludes coarsely annotated Foggy Driving images, and dehazing experiments use a constant gamma value that may be suboptimal.
Abstract
from arXiv · showhide
This work addresses the problem of semantic foggy scene understanding (SFSU). Although extensive research has been performed on image dehazing and on semantic scene understanding with clear-weather images, little attention has been paid to SFSU. Due to the difficulty of collecting and annotating foggy images, we choose to generate synthetic fog on real images that depict clear-weather outdoor scenes, and then leverage these partially synthetic data for SFSU by employing state-of-the-art convolutional neural networks (CNN). In particular, a complete pipeline to add synthetic fog to real, clear-weather images using incomplete depth information is developed. We apply our fog synthesis on the Cityscapes dataset and generate Foggy Cityscapes with 20550 images. SFSU is tackled in two ways: 1) with typical supervised learning, and 2) with a novel type of semi-supervised learning, which combines 1) with an unsupervised supervision transfer from clear-weather images to their synthetic foggy counterparts. In addition, we carefully study the usefulness of image dehazing for SFSU. For evaluation, we present Foggy Driving, a dataset with 101 real-world images depicting foggy driving scenes, which come with ground truth annotations for semantic segmentation and object detection. Extensive experiments show that 1) supervised learning with our synthetic data significantly improves the performance of state-of-the-art CNN for SFSU on Foggy Driving; 2) our semi-supervised learning strategy further improves performance; and 3) image dehazing marginally advances SFSU with our learning strategy. The datasets, models and code are made publicly available.
1 Introduction
The paper targets semantic understanding of foggy outdoor scenes, a setting where existing vision systems and labeled data are limited. It proposes partially synthetic foggy data, new datasets, and supervised and semi-supervised learning to address this gap.
- Foggy outdoor scene understanding remains comparatively understudied despite fog’s impact on visibility and computer vision systems.
- Collecting and annotating foggy images is difficult, motivating synthetic fog generation on real clear-weather outdoor images.
- The pipeline simulates physically sound fog on clear-weather datasets and applies the resulting images to semantic segmentation and object detection.Fog simulation uses a standard daytime optical model and incomplete depth information; the pipeline is illustrated in Figure 1.
- Foggy Cityscapes contains 550 finely annotated synthetic foggy images and 20000 additional synthetic foggy images without fine annotations.
- The models are trained either supervised on annotated synthetic fog or semi-supervised with additional foggy images receiving transferred supervision from clear-weather counterparts.
- The paper introduces Foggy Driving for evaluation and studies image dehazing’s utility for foggy scene understanding and human perception.Foggy Driving contains 101 annotated foggy driving images.
2 Related Work
Related work spans dehazing, depth completion, semantic understanding, synthetic visual data, and transfer learning. The paper distinguishes its approach by synthesizing fog in real outdoor urban scenes and transferring supervision across paired weather conditions.
- Dehazing research seeks to restore visibility using contrast enhancement, depth, and natural-image statistics as priors.
- Synthetic fog generation generally requires accurate depth, while this work addresses more complex outdoor urban depth configurations from Cityscapes.
- The depth-processing method builds on stereoscopic superpixel inpainting and introduces a theoretically grounded superpixel-matching objective.
- Prior outdoor semantic-understanding work mainly targets clear weather, whereas this paper handles 19 semantic classes and 8 distinct object classes in foggy scenes.
- Synthetic visual-data research uses rendered or modified images with annotations to train models for tasks including optical flow, text detection, pedestrian detection, and segmentation.
- The approach closes a clear-to-foggy domain gap and transfers supervision through paired samples, relating it to transfer learning and model distillation.
3 Fog Simulation on Real Outdoor Scenes
The paper simulates fog on real clear-weather outdoor images using an optical formation model and incomplete depth, refined through a multi-stage depth-completion pipeline. The method estimates transmission from completed depth while addressing unreliable regions through superpixel-based matching and RANSAC plane fitting.
- 3.1 Optical Model of Choice for Fog: Fog synthesis models the observed image from clear-scene radiance, atmospheric light, and distance-dependent transmission.The optical model assumes globally constant atmospheric light, generally valid for daytime images.
- 3.1 Optical Model of Choice for Fog: Larger attenuation coefficient β produces thicker fog, while visibility is defined by the maximum distance with transmission t(x) ≥ 0.05.Under the homogeneous model, meteorological optical range is MOR = 2.996/β.
- 3.2 Depth Denoising and Completion for Outdoor Scenes: The pipeline converts raw disparity into depth, completes and denoises it, computes scene distance, estimates transmission, and applies guided filtering.The completed depth map provides the basis for calculating the final transmission map.
- 3.2 Depth Denoising and Completion for Outdoor Scenes: Superpixels guide depth completion by approximating local 3D regions as planes and separating reliable from unreliable depth areas.Reliable superpixels receive RANSAC-fitted depth planes; unreliable ones are matched to reliable superpixels for plane assignment.
- 3.2 Depth Denoising and Completion for Outdoor Scenes: The proposed matching objective uses CIELAB color distance and centroid distance to improve pairwise assignment of unreliable to reliable superpixels.It addresses cases where prior color or adjacency costs can assign misleadingly low penalties to visually or spatially unsuitable matches.
4 Foggy Datasets
The paper introduces Foggy Cityscapes, a synthetic fog dataset derived from Cityscapes, and Foggy Driving, a real-world fog benchmark with semantic and object annotations. Foggy Cityscapes provides multiple visibility conditions and inherited annotations, while Foggy Driving supports evaluation of foggy-scene understanding.
- Dataset overview: Foggy Cityscapes is generated from Cityscapes using synthetic fog, while Foggy Driving contains 101 real-world foggy driving scenes for benchmarking.Foggy Cityscapes inherits semantic annotations from clear-weather counterparts; Foggy Driving includes annotations for semantic segmentation and object detection.
- Foggy Cityscapes: The large-scale Foggy Cityscapes collection contains 20,000 images from coarsely annotated Cityscapes data, retained without refinement filtering.This design prioritizes scale and variability over the highest visual quality of every synthetic image.
- Foggy Cityscapes: A refined Foggy Cityscapes subset contains 550 high-quality synthetic foggy images selected from finely annotated Cityscapes images.The subset includes 498 training images and 52 validation images, with inherited semantic segmentation and object-detection annotations.
- Foggy Cityscapes: Five Foggy Cityscapes versions simulate different visibility conditions using β values from 0.005 to 0.06, corresponding approximately to 600 m to 50 m MOR.Each version uses a constant attenuation coefficient and therefore a constant simulated meteorological optical range.
- Foggy Driving: Foggy Driving provides dense pixel-level annotations across Cityscapes’ 19 evaluation classes and separately labels instances in eight object classes.It includes 33 finely annotated images and 68 coarsely annotated images; Figures 5 and 6 summarize pixel and object counts by class.
- Foggy Driving: Coarsely annotated Foggy Driving images are excluded from object-detection evaluation but retained for semantic-segmentation evaluation because unlabeled pixels are ignored.Foggy Driving contains more than 500 vehicles and almost 300 humans, and its finely annotated subset has per-image human and vehicle counts roughly comparable to Cityscapes.
5 Supervised Learning with Synthetic Fog
Supervised fine-tuning on synthetic Foggy Cityscapes-refined improves semantic segmentation and object detection on real Foggy Driving scenes. Dehazing preprocessing generally offers no clear advantage, with performance depending on fog density and method.
- Semantic Segmentation: Fine-tuning Dilation10 on Foggy Cityscapes-refined improves Foggy Driving semantic segmentation over the original clear-weather model.The protocol fine-tunes a Cityscapes-trained model using synthetic foggy images and evaluates it on Foggy Driving.
- Semantic Segmentation: 3% mean IoU over all classes and 5% over frequent classes separate FT-0.01 from Dilation10.FT-0.01 uses only 498 Foggy Cityscapes-refined training images, versus 2,975 Cityscapes training images for Dilation10.
- Semantic Segmentation: Fine-tuned foggy models remain more robust to changes in β, while Dilation10 performance drops rapidly as fog thickens.The comparison evaluates the models across five foggy validation versions and the original clear-weather version.
- Dehazing Preprocessing: No dehazing is the best or second-best option across measures and fog densities, while MSCNN only marginally exceeds it in some cases.The authors attribute limited dehazing gains partly to optical-model mismatch and artifacts introduced by inverse dehazing.
- Dehazing Preprocessing: 3 dehazing methods provide no clear mean-IoU benefit over directly processing foggy images, and human evaluations show the same broad pattern.The no-dehazing option generally ranks highest, with DCP sometimes close behind; pairwise agreement is small and no option dominates.
- Object Detection: 2.4% average improvement on the two frequent detection classes is achieved by FT-0.005 over the baseline.Both fine-tuned models significantly improve car detection, while person detection remains on par with the baseline.
6 Semi-supervised Learning with Synthetic Fog
The paper extends supervised semantic segmentation to semi-supervised learning by transferring labels from clear-weather images to paired synthetic foggy images. On Foggy Driving, this strategy improves mean IoU beyond both the original and fog-fine-tuned RefineNet models.
- Semi-supervised learning: Semi-supervised learning uses clear-weather models to transfer supervision to unlabeled paired foggy images.The approach is motivated by easier recognition in clear weather and supplements human annotations with transferred labels.
- Training data: The unlabeled training set contains 20000 Foggy Cityscapes-coarse images, while the labeled set contains 498 finely annotated Foggy Cityscapes-refined images.The unlabeled foggy images receive labels predicted by a model trained using the labeled data.
- Semi-supervised learning: The training objective combines categorical cross-entropy on human labels with transferred labels from unlabeled foggy-image pairs.The relative contribution of unlabeled images is controlled by λ, with implementation based on mixing labeled and transferred-label data.
- Results: 44.3%, 46.3%, and 49.7% mean IoU are obtained by original RefineNet, fog-fine-tuned RefineNet, and the semi-supervised model, respectively, on Foggy Driving.Fog fine-tuning improves the baseline by 2%, while semi-supervised learning adds a further 3.4% improvement over fog fine-tuning.
- Results: Qualitative results show improved segmentation of pedestrians, trams, and sidewalks that the original RefineNet misclassifies.Figure 13 presents results for both coarsely and finely annotated Foggy Driving images.
- Implication: The approach reduces reliance on large-scale foggy-scene annotation by combining existing clear-weather annotations, generated fog, and transferred supervision.The stated benefit concerns training and evaluating semantic understanding on real foggy scenes using partially synthetic data.
7 Conclusion
The paper demonstrates that synthetic fog based on real images supports semantic understanding of real foggy scenes. Foggy Cityscapes improves segmentation and detection on Foggy Driving in supervised and semi-supervised settings, while dehazing is only marginally helpful.
- Conclusion: The paper constructs Foggy Cityscapes from Cityscapes and Foggy Driving as a real-world foggy dataset, both with dense semantic annotations.The annotations cover 19 semantic classes and bounding boxes for objects in 8 classes.
- Conclusion: Foggy Cityscapes boosts state-of-the-art CNN performance on the challenging real foggy scenes of Foggy Driving in supervised and semi-supervised settings.The conclusion attributes the benefit to synthetic data based on real images.
- Conclusion: Image dehazing is marginally helpful for semantic foggy scene understanding because it faces difficulties when applied directly to real outdoor foggy data.The paper proposes future integration of dehazing and semantic understanding in an end-to-end learned pipeline.
- Resources: The datasets, models, and code are publicly available.The conclusion provides a public-access statement for the released resources.