Source-linked AI summary
Dark Model Adaptation: Semantic Image Segmentation from Daytime to Nighttime
Dengxin Dai, Luc Van Gool
TL;DR
Nighttime semantic segmentation is difficult because existing progress and annotation practices center on daytime scenes. The paper progressively adapts daytime models through twilight stages, and reports effective daytime-to-nighttime transfer without extra human annotation.
Problem
Semantic image segmentation has mainly progressed for daytime scenarios, while collecting large human-annotated datasets across nighttime conditions is costly.
Method
The method progressively self-learns from daytime through civil, nautical, and astronomical twilight stages toward nighttime.
Results
All adaptation variants improve the original daytime-trained semantic model, while the complete three-step pipeline outperforms incomplete alternatives.
Takeaways & Limitations
Twilight provides a trajectory between daytime and nighttime that makes unsupervised daytime-to-nighttime knowledge transfer feasible.
Takeaways & Limitations
Nighttime performance remains substantially worse than daytime performance, and adding reasonable nighttime human annotation is expected to improve results.
Abstract
from arXiv · showhide
This work addresses the problem of semantic image segmentation of nighttime scenes. Although considerable progress has been made in semantic image segmentation, it is mainly related to daytime scenarios. This paper proposes a novel method to progressive adapt the semantic models trained on daytime scenes, along with large-scale annotations therein, to nighttime scenes via the bridge of twilight time -- the time between dawn and sunrise, or between sunset and dusk. The goal of the method is to alleviate the cost of human annotation for nighttime images by transferring knowledge from standard daytime conditions. In addition to the method, a new dataset of road scenes is compiled; it consists of 35,000 images ranging from daytime to twilight time and to nighttime. Also, a subset of the nighttime images are densely annotated for method evaluation. Our experiments show that our method is effective for model adaptation from daytime scenes to nighttime scenes, without using extra human annotation.
I. INTRODUCTION
Nighttime semantic recognition remains difficult because visible-light systems are mainly designed for well-illuminated daytime scenes. The paper therefore proposes progressively adapting daytime semantic models through twilight scenes while compiling a dataset for evaluation.
- Visible-light perception algorithms are mainly designed for daytime images under good illumination, leaving nighttime driving scenes challenging.Nighttime recognition is further affected by disrupted structural, textural, and color features, camera noise, and motion blur.
- Collecting and densely annotating large image sets across every weather and illumination condition is difficult and costly.
- The paper progressively adapts semantic models trained on daytime scenes to nighttime scenes using twilight images as intermediate stages.
- Nighttime Driving contains 35,000 unlabeled images and 50 densely annotated images spanning nighttime and twilight driving scenes.The dataset supports learning and evaluating semantic segmentation methods for nighttime driving scenes.
A. Semantic Understanding of Nighttime Scenes
Prior nighttime scene-understanding work includes detection with FIR or visible-light cameras and methods for robustness to illumination changes. This paper positions nighttime semantic understanding as an extension of recent daytime road-scene models.
- Nighttime object detection and recognition has used FIR cameras, visible-light cameras, or combinations of both, especially for human detection.
- Other research addresses traffic-object detection and robustness to illumination changes for road-area detection and semantic labeling.
- Recent semantic scene-understanding methods have made daytime object detection feasible, motivating their extension to nighttime conditions.
B. Model Adaptation
The work relates to transfer learning, including adaptation across weather conditions and from synthetic images to real environments. Its supervision transfer from daytime to nighttime is inspired by model distillation and imitation learning.
- Prior transfer-learning research studied model adaptation across weather conditions and domain adaptation from synthetic images to real environments.
- This paper’s daytime-to-nighttime supervision transfer is inspired by model distillation and imitation learning.
C. Road Scene Understanding
Road scene understanding supports assisted and autonomous driving applications. This work aims to adapt advanced daytime road-scene models to nighttime without manually annotating nighttime images.
- Road scene understanding supports assisted and autonomous driving applications, including detecting roads, traffic lights, cars, and pedestrians.
- The paper aims to extend advanced daytime road-scene models to nighttime without manually annotating nighttime images.
III. APPROACH
The approach progressively adapts a daytime semantic segmentation model to nighttime through three unlabeled twilight domains. It uses model-generated labels to bridge increasingly larger illumination gaps while retaining daytime supervision.
- Principle: The method transfers semantic knowledge through unlabeled twilight images rather than requiring equivalent human annotation across illumination conditions.Its design is inspired by model distillation and addresses the expense of annotating many conditions and their combinations.
- Motivation: Twilight provides intermediate domains between daytime and nighttime, comprising civil, nautical, and astronomical twilight categories.The method relies on continuous illumination changes and assumes each adjacent domain gap is smaller than the direct daytime-to-nighttime gap.
- Data and initialization: Daytime images with human annotations initialize the segmentation model, while the three twilight datasets remain unlabeled.Images are denoted x0 through x3 for daytime and the three twilight categories, with daytime labels y0 supplied by humans.
- Progressive adaptation: The model generates noisy labels for civil twilight, fine-tunes on daytime and civil-twilight data, then repeats this process for nautical and astronomical twilight.Each successive model is duplicated from the preceding model and applied to the next twilight domain before fine-tuning.
- Final model: After three adaptation stages, the final model is fine-tuned on daytime data and all three pseudo-labeled twilight datasets before segmenting nighttime images.The method is termed Gradual Model Adaptation, and equal dataset weights are used empirically during training.
A. Data Collection
Nighttime Driving was collected from car rides across Swiss cities and suburbs, producing 35,000 frames spanning daytime through twilight to nighttime. A manually selected subset of 50 diverse nighttime images was densely annotated for evaluation.
- Dataset construction: 35,000 images were extracted at one frame per second from five roughly two-hour driving videos recorded with a GoPro Hero 5.The recordings began during daytime, passed through twilight, and ended at full nighttime.
- Evaluation set: 50 diverse nighttime images were manually selected to form the Nighttime Driving-test set.Manual selection was intended to provide visual diversity despite the test set’s relatively small size.
- Annotations: The test images received fine pixel-level annotations using the 19 evaluation classes from Cityscapes.Pixels outside those classes or uncertain because of insufficient illumination receive the void label and are ignored during evaluation.
B. Experimental Evaluation
The experiments evaluate progressive adaptation from daytime to nighttime using twilight as a bridge, finding improved segmentation over direct daytime models and stronger performance with three progressive steps. Qualitative and quantitative results support the method, while nighttime performance remains substantially below daytime performance and annotation ambiguity remains a limitation.
- Experimental setup: RefineNet models are adapted progressively through twilight, with each adapted model labeling data used in the next fine-tuning stage.The pipeline fine-tunes on the union of Cityscapes daytime data and previously segmented twilight datasets.
- Quantitative results: All adaptation variants improve the original daytime-trained semantic model on overall intersection over union (IoU).The authors attribute this to smaller domain gaps between twilight and the daytime or nighttime domains than between daytime and nighttime directly.
- Progressive adaptation: The complete three-step pipeline outperforms incomplete alternatives across civil, nautical, and astronomical twilight stages.The reported sequence is daytime to civil twilight, civil to nautical twilight, and nautical to astronomical twilight.
- Progressive adaptation: The three-step progressive pipeline outperforms the one-step alternative because it learns from less noisy generated labels closer to each preceding model’s training domain.The method balances label quality control against computational cost by adapting in smaller domain steps.
- Qualitative results: Qualitative examples generally show better segmentation than the original daytime RefineNet, including fewer road-area pixels misclassified as cars.The comparison is shown on the Nighttime Driving-test set.
- Limitations: Nighttime segmentation still performs substantially worse than daytime segmentation, and uncertain nighttime regions may require a separate special class.The authors identify human annotation uncertainty in many nighttime regions as a limitation and future-work direction.
V. CONCLUSIONS
The paper addresses nighttime semantic image segmentation by progressively adapting daytime-trained models through twilight stages. It introduces the Nighttime Driving dataset and reports effective unsupervised knowledge transfer without human supervision.
- Conclusions: The method progressively adapts daytime-trained semantic models to nighttime scenes through three twilight subgroups in an unsupervised manner.Twilight spans the transition between daytime and nighttime and serves as the adaptation bridge.
- Conclusions: The Nighttime Driving dataset contains 35,000 images spanning daytime, twilight, and nighttime, including 50 densely annotated nighttime images.The densely annotated subset supports method evaluation.
- Conclusions: The experiments show effective knowledge transfer from daytime to nighttime without human supervision.