PaperlayerView exact revision on arXiv

Source-linked AI summary

Map-Guided Curriculum Domain Adaptation and Uncertainty-Aware Evaluation for Semantic Nighttime Image Segmentation

Christos Sakaridis, Dengxin Dai, Luc Van Gool

arXiv:2005.14553v2cs.CV

TL;DR

Nighttime semantic segmentation lacks robust adaptation and evaluation when visual content becomes ambiguous. The paper introduces map-guided curriculum adaptation, uncertainty-aware evaluation, and Dark Zurich, substantially improving competing methods on real nighttime sets.

  • Problem

    Semantic segmentation models and annotations are predominantly developed for daytime, while nighttime images contain challenging and sometimes indiscernible content.

  • Method

    MGCDA gradually adapts daytime models using progressively darker data and cross-time-of-day map correspondences, alongside UIoU and the Dark Zurich benchmark.

  • Results

    MGCDA substantially improves upon competing state-of-the-art methods on real nighttime sets under standard IoU evaluation.

  • Takeaways & Limitations

    Uncertainty-aware evaluation shows that invalidating predictions can be useful for ambiguous nighttime content and safety-oriented applications.

  • Takeaways & Limitations

    The method relies on a simple map-matching procedure that may be insufficient for other tasks.

Abstract

from arXiv · show

We address the problem of semantic nighttime image segmentation and improve the state-of-the-art, by adapting daytime models to nighttime without using nighttime annotations. Moreover, we design a new evaluation framework to address the substantial uncertainty of semantics in nighttime images. Our central contributions are: 1) a curriculum framework to gradually adapt semantic segmentation models from day to night through progressively darker times of day, exploiting cross-time-of-day correspondences between daytime images from a reference map and dark images to guide the label inference in the dark domains; 2) a novel uncertainty-aware annotation and evaluation framework and metric for semantic segmentation, including image regions beyond human recognition capability in the evaluation in a principled fashion; 3) the Dark Zurich dataset, comprising 2416 unlabeled nighttime and 2920 unlabeled twilight images with correspondences to their daytime counterparts plus a set of 201 nighttime images with fine pixel-level annotations created with our protocol, which serves as a first benchmark for our novel evaluation. Experiments show that our map-guided curriculum adaptation significantly outperforms state-of-the-art methods on nighttime sets both for standard metrics and our uncertainty-aware metric. Furthermore, our uncertainty-aware evaluation reveals that selective invalidation of predictions can improve results on data with ambiguous content such as our benchmark and profit safety-oriented applications involving invalid inputs.

1 INTRODUCTION

This work introduces Map-Guided Curriculum Domain Adaptation (MGCDA) to adapt daytime segmentation models to nighttime without nighttime annotations, alongside uncertainty-aware evaluation for visually indiscernible regions. It also presents Dark Zurich, a corresponding day–twilight–night dataset and 201-image nighttime benchmark.

  • Method: MGCDA adapts semantic segmentation models from daytime to nighttime without annotations in the nighttime domain.The method uses continuity of illumination, prior knowledge of place, and data to transfer semantic knowledge across times of day.
  • Method: The method exploits shared content between images captured at different times with the same 6D camera pose to guide transfer from daytime to nighttime.Intermediate illumination domains support a smoother transfer of semantic knowledge, while cross-time correspondences provide place-specific guidance.
  • Evaluation: The uncertainty-aware framework distinguishes valid from invalid regions, requiring models to express high uncertainty where nighttime degradation makes semantic content indiscernible even to humans.Privileged information from daytime counterparts enables reliable invalid-region annotation and evaluation, unlike existing benchmarks that completely exclude such regions.
  • Dataset: Dark Zurich contains 8779 corresponding daytime, twilight, and nighttime images and provides a benchmark of 201 annotated nighttime images for uncertainty-aware evaluation.The dataset supplies real data for MGCDA and supports a nighttime segmentation challenge; the dataset and code are publicly available.
  • Contributions: Compared with the conference version, this paper adds geometry-aware correspondence refinement, 50 annotated images for a total of 201, and more extensive experiments.The revised adaptation method is reported to improve over the conference version and includes comparisons with more recent state-of-the-art methods.

2 RELATED WORK

Prior work spans nighttime vision, adaptation to adverse conditions, and semantic-segmentation evaluation, while maps have mainly supported localization and related vision tasks. This work positions daytime imagery and distilled semantics as map-based prior knowledge for nighttime segmentation.

  • Vision at Nighttime: Nighttime vision research includes human, car, and rear-light detection, alongside illumination-robust representations, multimodal fusion, and adverse-illumination datasets,,,,,,,,,,,,.
  • Domain Adaptation: Domain adaptation has increasingly targeted adverse conditions, including clear-weather-to-fog adaptation using labeled synthetic and unlabeled real images across increasing fog densities,,,,,,,.
  • Semantic Segmentation Evaluation: Semantic segmentation is commonly evaluated with IoU, with related metrics including instance-level IoU, mean average precision, panoptic quality, and WildDash’s combination of standard IoU with additional evaluation,,.
  • Map-Guided Vision Applications: Maps have supported robot localization and vision tasks such as road-surface detection, navigation, object detection, tracking, and forecasting,,,,,,,,, [67].
  • Map-Guided Vision Applications: This work instead uses georeferenced daytime images and their distilled semantics as map-based prior knowledge to supplement segmentation under challenging lighting or weather, especially at nighttime.

3 MAP-GUIDED CURRICULUM DOMAIN ADAPTA-

MGCDA adapts semantic segmentation models from daytime to nighttime through an ascending darkness curriculum that uses twilight as an intermediate domain. It refines unlabeled-target pseudo-labels with corresponding daytime map knowledge, geometry-aware alignment, and confidence-based fusion.

  • Curriculum domain adaptation: MGCDA transfers a model through daytime, twilight, and nighttime domains ordered by increasing darkness, distilling knowledge from each domain while introducing synthetic data for the next.The framework uses the current domain’s unlabeled real data and newly stylized data from the next domain to adapt progressively.
  • Guided label refinement: The curriculum addresses error accumulation by refining predictions on darker images with weak supervision from corresponding daytime images rather than directly propagating intermediate-domain predictions.The method assumes objects are generally easier to recognize in lighter conditions and solves domains in ascending darkness.
  • Map-guided supervision: The method relies on geo-referenced daytime images and semantic pseudo-labels as map knowledge, while acknowledging that its simple map-matching procedure may not generalize to other tasks.The authors treat more sophisticated map matching as orthogonal to the learning algorithm.
  • Geometry-aware alignment: Geometry-aware warping aligns daytime soft predictions to dark-image viewpoints using dense correspondences, preserving small objects such as traffic signs and poles better than a cross-bilateral filter.The formulation explicitly incorporates two-view geometry and uses forward warping based on the daytime source depth map.
  • Confidence-based fusion: Adaptive fusion combines aligned daytime and initial dark-image predictions using per-pixel confidence, downweighting daytime predictions when high-confidence disagreement indicates dynamic content.The fusion scheme regulates the daytime contribution to address imperfect alignment and differences caused by dynamic objects.

4 UNCERTAINTY-AWARE EVALUATION

The section introduces an uncertainty-aware annotation and evaluation framework that distinguishes invalid from valid pixels using day–night correspondences, and proposes UIoU to evaluate predictions that may explicitly invalidate uncertain regions.

  • Framework: The framework evaluates invalid regions alongside valid pixels but treats them differently to account for uncertainty in their semantic content.It generalizes negative-test-case evaluation from whole images to intra-image regions that can coexist with valid content.
  • Annotation: Annotations are created by first marking valid pixels using only the nighttime image, then assigning semantic labels with a corresponding daytime image captured under more favorable conditions.The protocol produces an invalid mask J and semantic labeling H over 19 Cityscapes classes, using cross-time-of-day scene overlap as privileged information.
  • Prediction handling: Predictions use a confidence threshold θ to invalidate low-confidence pixels without adding an explicit invalid class, producing an evaluation curve as θ varies.The model’s soft class distribution is converted to a hard label, and pixels below θ are assigned invalid; increasing θ therefore invalidates more pixels.
  • UIoU metric: UIoU generalizes IoU to invalid predictions, rewarding true invalids like correct labels while penalizing invalidating valid pixels like an incorrect prediction.It averages class-wise UIoU over the C classes and reduces to standard IoU when no pixels are predicted invalid, including at θ = 1/C.
  • UIoU metric: Under the framework’s confidence-separation assumption, UIoU is guaranteed to exceed IoU for some θ > 1/C when errors occur in ground-truth invalid regions.The theorem assumes invalid-region predictions have lower confidence than valid-region predictions and that at least one standard IoU error lies in an invalid region.

5 THE DARK ZURICH DATASET

Dark Zurich provides corresponding daytime, twilight, and nighttime image sets collected in Zurich, with held-out laps for validation and testing. Its nighttime annotations show high inter-annotator consistency and are positioned against existing annotated nighttime datasets.

  • Data collection: Dark Zurich was recorded in Zurich with a 1080p GoPro Hero 5 mounted above a car’s front windshield, using multiple drives and laps to establish correspondences.The collection protocol is described in Section 3.
  • Dataset split: Dark Zurich contains 3041 daytime, 2920 twilight, and 2416 nighttime images, with one lap reserved for validation, another for testing, and remaining laps used unlabeled for training.Images were extracted at 1 fps and correspond across the three time-of-day sets.
  • Annotation quality: 93.5% of labeled pixels and 95% of invalid-mask pixels were consistent across 20 images annotated independently by two subjects.The dataset is compared with existing annotated nighttime sets, while Cityscapes and Mapillary Vistas contain few or no nighttime scenes and Nighttime Driving and Raincouver provide coarse annotations.

6 RESULTS

MGCDA and GCMA substantially outperform competing nighttime adaptation methods, achieving a 10% overall mIoU improvement on Dark Zurich-test and strong generalization across additional benchmarks. Ablations and uncertainty-aware evaluation further show the value of curriculum learning, guided refinement, and selective invalidation of uncertain predictions.

  • Comparison with state-of-the-art: MGCDA and GCMA significantly outperform other adaptation methods on Dark Zurich-test, improving overall mIoU by 10% over the next-best method, especially for dark nighttime classes.The advantage is pronounced for sky, vegetation, building, and person, indicating effective handling of large daytime-to-nighttime domain shifts.
  • Comparison with state-of-the-art: The method generalizes well beyond Dark Zurich, with comparisons repeated on Nighttime Driving and BDD100K despite the latter’s unreliable ground-truth annotations.BDD100K evaluation required manually identifying a list to address annotation reliability.
  • Ablation study: Two-stage curriculum training with twilight data improves mIoU by 9.7% over direct nighttime adaptation, while fully fledged MGCDA is needed to exploit real data beyond the synthetic-only CycleGAN baseline.Pseudo-labels are inferred by models already adapted to the real images, enabling more reliable gradients during subsequent training; the method uses map-guided refinement of twilight labels.
  • Test-time map guidance: At test time, map guidance is evaluated for MGCDA and DMAda by comparing original predictions with map-guided alternatives on Dark Zurich-test.This experiment examines deployment scenarios such as autonomous-vehicle segmentation, where map guidance remains relevant after training.
  • Image enhancement: Enhancement-based preprocessing is insufficient to close the nighttime domain gap: ZeroDCE adaptation reaches 38.4% mIoU but remains below synthetic-nighttime adaptation without enhancement.The comparison includes ZeroDCE, MBLLEN, CLAHE, and CycleGAN-based preprocessing alternatives.
  • Uncertainty-aware evaluation: UIoU exceeds standard IoU by approximately 2–3% at its maximum across methods, supporting selective invalidation of low-confidence predictions on ambiguous nighttime images.The curves evaluate MGCDA, GCMA, DMAda, and RefineNet across confidence thresholds θ on Dark Zurich-test.

7 CONCLUSION

The paper introduces MGCDA for gradual daytime-to-nighttime semantic segmentation adaptation, UIoU for evaluating images with indiscernible content, and the Dark Zurich dataset with cross-time-of-day correspondences and 201 annotated nighttime scenes.

  • MGCDA gradually adapts semantic segmentation models from daytime to nighttime using stylized data and unlabeled real data with increasing darkness.
  • UIoU is a novel semantic-segmentation evaluation metric designed for images containing indiscernible content.
  • Dark Zurich contains real scenes captured at multiple times of day with cross-time-of-day correspondences and 201 annotated nighttime scenes.

APPENDIX A PROOF OF THEOREM 1

The proof links standard IoU pixel sets to the corresponding UIoU sets, then derives the target inequality by applying the theorem’s assumptions and analyzing two cases. It concludes both cases using nonnegativity, set inclusions, and the definitions of IoU and UIoU.

  • Proof setup: The proof first establishes an association between standard-IoU pixel sets and UIoU pixel sets for every θ ∈ [1/C, 1].The resulting decomposition includes TP(θ), FN(θ), TI(θ), and FI(θ).
  • Proof setup: Under the first assumption, FI(θ1) is empty because no false-invalid pixel exists for any θ < θ2, including θ1.This is then substituted into the relation obtained for θ = θ1.
  • Key relation: The proof combines the first and second assumptions, then uses disjointness and set inclusions among FN and FP sets to derive the key relation.Specifically, FN(1/C) ∩ FP(1/C) = ∅, while FN(θ1) ⊆ FN(1/C) and FP(θ1) ⊆ FP(1/C).
  • Case analysis: Because both terms in the key relation are nonnegative and at least one is positive, the proof splits into two corresponding cases.Each case supplies the strict inequality needed to complete the argument.
  • First case: In the first case, algebraic bounds using the IoU definition, equation (16), FP-set inclusion, and UIoU definition establish the desired strict inequality.The chain concludes with UIoU(θ1) in equation (21).
  • Second case: The second case follows analogously, applying nonnegativity and the preceding bounds before concluding with UIoU(θ1) in equation (24).The proof explicitly cites the IoU and UIoU definitions together with equations (16), (22), and (23).

APPENDIX B ADDITIONAL QUALITATIVE RESULTS

Additional Dark Zurich-test examples qualitatively compare MGCDA with GCMA, AdaptSegNet, and DMAda, further demonstrating MGCDA’s superiority. The examples also show that many annotated invalid regions receive reliable semantic labels and can be evaluated.

  • APPENDIX B ADDITIONAL QUALITATIVE RESULTS: MGCDA qualitatively outperforms GCMA, AdaptSegNet, and DMAda on additional Dark Zurich-test images.Figure 13 presents nighttime images alongside qualitative predictions from all four approaches.
  • APPENDIX B ADDITIONAL QUALITATIVE RESULTS: A significant portion of ground-truth invalid regions receives reliable semantic labels under the annotation protocol, enabling their inclusion in evaluation.This supports evaluating regions that would otherwise be treated as invalid.
  • APPENDIX B ADDITIONAL QUALITATIVE RESULTS: Figure 13 includes invalid-mask and semantic annotations for the additional nighttime examples.Valid pixels are overlaid in green in the invalid-mask annotations.

APPENDIX C PARAMETER SELECTION FOR PREDICTION FUSION

Appendix C shows that the confidence-adaptive fusion parameter α_h must balance exploiting easier-domain daytime predictions against their imperfect alignment with dark images.

  • APPENDIX C PARAMETER SELECTION FOR PREDICTION FUSION: The fusion parameter α_h controls refinement strength, with increasing values producing progressively refined predictions for dark images.The appendix illustrates this effect in Fig. 14(f)–(h).
  • APPENDIX C PARAMETER SELECTION FOR PREDICTION FUSION: Low α_h values can leave erroneous regions in the initial dark-image prediction unrefined, such as the central region shown in Fig. 14(f).
  • APPENDIX C PARAMETER SELECTION FOR PREDICTION FUSION: Higher reliance on daytime predictions is expected to become more effective as daytime-view depth-map accuracy improves alignment with dark images.The fusion step can then assign greater weight to the daytime prediction during geometrically guided segmentation refinement.
Loading 2005.14553v2…