Source-linked AI summary
DANNet: A One-Stage Domain Adaptation Network for Unsupervised Nighttime Semantic Segmentation
Xinyi Wu, Zhenyao Wu, Hao Guo, Lili Ju, Song Wang
TL;DR
Nighttime semantic segmentation lacks high-quality labels and is difficult under poor illumination, motivating adaptation from labeled daytime data. DANNet performs one-stage adversarial multi-target adaptation using coarsely aligned day-night pairs and static-category pseudo supervision, and it achieves state-of-the-art performance on the reported nighttime benchmarks. The approach also uses re-weighting to improve small-object segmentation.
Problem
Nighttime semantic segmentation is important for autonomous driving but difficult because poor illumination and visual hazards make high-quality pixel-level annotations arduous.
Method
DANNet performs one-stage adversarial adaptation from labeled Cityscapes to unlabeled Dark Zurich day-night pairs, using static-category daytime predictions as nighttime pseudo supervision and probability re-weighting.
Results
DANNet achieves state-of-the-art nighttime semantic segmentation performance on Dark Zurich and Nighttime Driving datasets.
Takeaways & Limitations
Coarsely aligned daytime data can bridge labeled daytime and unlabeled nighttime domains, while re-weighting boosts performance for small objects.
Takeaways & Limitations
Prior transfer-based pipelines can be time-consuming and closely dependent on the preprocessing transfer model, particularly when the domain gap is large.
Abstract
from arXiv · showhide
Semantic segmentation of nighttime images plays an equally important role as that of daytime images in autonomous driving, but the former is much more challenging due to poor illuminations and arduous human annotations. In this paper, we propose a novel domain adaptation network (DANNet) for nighttime semantic segmentation without using labeled nighttime image data. It employs an adversarial training with a labeled daytime dataset and an unlabeled dataset that contains coarsely aligned day-night image pairs. Specifically, for the unlabeled day-night image pairs, we use the pixel-level predictions of static object categories on a daytime image as a pseudo supervision to segment its counterpart nighttime image. We further design a re-weighting strategy to handle the inaccuracy caused by misalignment between day-night image pairs and wrong predictions of daytime images, as well as boost the prediction accuracy of small objects. The proposed DANNet is the first one stage adaptation framework for nighttime semantic segmentation, which does not train additional day-night image transfer models as a separate pre-processing stage. Extensive experiments on Dark Zurich and Nighttime Driving datasets show that our method achieves state-of-the-art performance for nighttime semantic segmentation.
1. Introduction
Nighttime semantic segmentation is important for autonomous driving but difficult because poor illumination and visual hazards make high-quality pixel annotations arduous. DANNet addresses this challenge with one-stage adversarial adaptation using labeled daytime data and unlabeled, coarsely aligned day-night pairs.
- Motivation: Nighttime semantic segmentation supports autonomous driving applications, but poor illumination, indiscernible regions, and visual hazards make pixel-level annotation difficult.Examples of hazards include under/overexposure and motion blur.
- Limitations of prior work: Prior daytime-to-nighttime approaches often train an image-transfer model as an additional preprocessing stage, creating a time-consuming two-stage pipeline.The second stage closely relies on the first, especially when the domain gap is large.
- Proposed method: DANNet is a multi-target adversarial adaptation network with an image relighting network, a semantic segmentation network, and two discriminators.It adapts from labeled Cityscapes to Dark Zurich daytime and nighttime domains.
- Proposed method: DANNet uses Dark Zurich-D predictions for static object categories as pseudo supervision for corresponding Dark Zurich-N images.The strategy exploits coarsely aligned day-night image pairs without labeled nighttime data.
- Proposed method: A probability re-weighting strategy is designed to significantly enhance segmentation of small objects.It addresses category imbalance and inaccuracies associated with pseudo supervision.
- Results: Experiments on Dark Zurich-test and Nighttime Driving report new state-of-the-art nighttime semantic segmentation performance, with ablations verifying component effectiveness.The reported evaluation covers both benchmark datasets and component-level ablations.
2. Related Work
Prior nighttime adaptation methods use adversarial learning, image translation, self-training, or gradual transfer through intermediate domains, often involving additional preprocessing. DANNet instead presents a one-stage framework using multi-target adaptation and day-night correspondence.
- Domain adaptation for semantic segmentation: Domain adaptation methods transfer knowledge from labeled source domains to target domains with similar objects but different data distributions.Semantic segmentation has used domain-adversarial learning and output-space adversarial adaptation.
- Domain adaptation for semantic segmentation: Image translation, style transfer, and self-training have also been used to obtain domain-invariant representations or iteratively refine pseudo labels.These approaches provide alternatives to adversarial learning for source-target adaptation.
- Nighttime semantic segmentation: Nighttime segmentation methods have used twilight domains, stylized synthetic images, and unlabeled real images to exploit cross-time-of-day correspondence.Gradual adaptation approaches commonly require multiple semantic-segmentation stages in preprocessing.
- Nighttime semantic segmentation: Other nighttime approaches use thermal infrared inputs or two-stage adversarial training for rainy night scenes.These methods address nighttime segmentation through complementary sensing or near-scene day-night adaptation.
- DANNet: DANNet differs by performing one-stage end-to-end adaptation with an image relighting network, a segmentation network, and two discriminators.Its architecture simultaneously adapts from Cityscapes to Dark Zurich-D and Dark Zurich-N.
3. Proposed Method
DANNet jointly adapts labeled Cityscapes daytime data to Dark Zurich daytime and nighttime domains using relighting, segmentation, and adversarial discrimination. Its re-weighting and static pseudo-supervision mechanisms address illumination differences, category imbalance, and day–night correspondence.
- Network architecture: DANNet simultaneously adapts Cityscapes to Dark Zurich daytime and nighttime domains with an image relighting network, a segmentation network, and two discriminators.The segmentation network shares weights across relighted images from all three domains, while the discriminators distinguish source from each target domain.
- Network architecture: The relighting network uses four convolutional layers, three residual blocks, two transposed convolutional layers, and residual addition to produce relighted images.Each convolutional layer is followed by batch normalization.
- Probability re-weighting: The re-weighting strategy assigns larger weights to underrepresented categories, uses logarithms to limit overweighting, and applies normalized weights to prediction channels before argmax decoding.This design targets small-object categories such as poles, signs, and lights.
- Objective functions: DANNet’s light loss combines total variation, exposure-control, and structural-similarity losses to align intensity distributions, smooth noise, control exposure, and preserve image structure.The total variation term removes rough textures, while structural similarity preserves the original image structure.
- Objective functions: Source-domain segmentation uses weighted cross-entropy, while static loss transfers daytime predictions as pseudo supervision for static categories in corresponding nighttime images.Static categories include road, sidewalk, wall, fence, pole, light, sign, vegetation, terrain, and sky; focal loss addresses category imbalance.
- Objective functions: Two least-squares adversarial losses encourage predictions from both target domains to approach the source-domain output distribution.The generator combines light, segmentation, static, and adversarial losses into its total objective.
4. Experiments
Experiments evaluate DANNet on Dark Zurich and Nighttime Driving using mIoU, comparisons with existing methods, and ablations of its components. DANNet achieves strong nighttime segmentation performance, while pseudo supervision and re-weighting materially improve results.
- Experimental setup: The experiments use mIoU, Cityscapes for adversarial training, 2,416 coarsely aligned Dark Zurich night-day pairs, and Nighttime Driving only for evaluation.Dark Zurich images are unlabeled, while Nighttime Driving contains 50 pixel-level annotated nighttime images.
- Comparison with state-of-the-art methods: DANNet with PSPNet improves overall mIoU by 2.7% over MGCDA, the strongest existing method, on Dark Zurich-test.DANNet variants using DeepLab-v2, RefineNet, or PSPNet all perform better than or tie existing methods.
- Comparison with state-of-the-art methods: DANNet also outperforms existing methods on several Dark Zurich categories, including road, sidewalk, and sky.The reported visualizations on Dark Zurich-val support the comparison of segmentation outputs across methods.
- Ablation study: Removing pseudo labels from Dark Zurich-D decreases mIoU by 13.78%, showing their key contribution in the ablation study.The study also identifies coarsely aligned Dark Zurich-D, the image relighting network, and its light loss as useful components.
- Ablation study: The specialized static loss outperforms directly applying cross-entropy or focal loss for calculating the static loss.This comparison is part of the component ablation on Dark Zurich-val.
- Ablation study: The re-weighting strategy improves small-object segmentation, with optimal performance at std = 0.16 and a 10% additional increase over state-of-the-art approaches on Dark Zurich-val.Using OCNet’s commonly used weights instead achieves 35.05 mIoU, below DANNet’s performance.
5. Conclusion
DANNet is an end-to-end, unsupervised nighttime segmentation network that adapts from labeled daytime data through unlabeled, coarsely aligned day-night pairs. Its multi-target adaptation and re-weighting strategy yield state-of-the-art performance on Dark Zurich and Night Driving while improving small-object prediction.
- 5. Conclusion: DANNet adapts from labeled Cityscapes daytime images to unlabeled Dark Zurich day-night pairs using an image relighting network and a special light loss.The method bridges Cityscapes, Dark Zurich-D, and Dark Zurich-N through multi-target domain adaptation.
- 5. Conclusion: Coarse alignment of static categories between Dark Zurich-D and Dark Zurich-N supplies pseudo supervision, while re-weighting boosts small-object performance.The conclusion identifies both mechanisms as central components of the proposed adaptation framework.
- 5. Conclusion: Experiments demonstrate component effectiveness and state-of-the-art performance on Dark Zurich and Night Driving test datasets.The reported conclusion covers both the designed components and the two evaluation datasets.