PaperlayerView exact revision on arXiv

Source-linked AI summary

Night-time Scene Parsing with a Large Real Dataset

Xin Tan, Ke Xu, Ying Cao, Yiheng Zhang, Lizhuang Ma, Rynson W. H. Lau

arXiv:2003.06883v3cs.CV

TL;DR

Scene-parsing research has focused mainly on well-lit daytime images, while labeled night-time data are scarce and co-occurring over- and under-exposures are not explicitly modeled. The paper introduces NightCity and an exposure-aware two-stream framework for night-time parsing. The authors report that NightCity benefits existing methods and their model outperforms existing methods, while noting failures in extremely challenging large under-exposure regions.

  • Problem

    Scene parsing has mainly focused on daytime images, with scarce labeled night-time data and over- and under-exposures not explicitly modeled in existing pipelines.

    Source: §I · p. 1

  • Method

    The paper constructs NightCity, a pixel-annotated real night-time dataset, and proposes a two-stream framework that learns exposure features to augment scene parsing.

    Source: §I · p. 1, §VI · p. 11

  • Results

    The authors report that NightCity benefits existing scene-parsing methods on night-time scenes and that their model outperforms existing methods.

    Source: §VI · p. 12

  • Takeaways & Limitations

    NightCity is proposed for NTSP training and benchmarking, and the authors report that its data can benefit existing scene-parsing methods on night-time images.

    Source: §I · p. 1, §VI · p. 12

    The model may fail in extremely challenging cases, such as large under-exposed regions containing visually difficult objects.

    Source: §VI · p. 12

Abstract

from arXiv · show

Although huge progress has been made on scene analysis in recent years, most existing works assume the input images to be in day-time with good lighting conditions. In this work, we aim to address the night-time scene parsing (NTSP) problem, which has two main challenges: 1) labeled night-time data are scarce, and 2) over- and under-exposures may co-occur in the input night-time images and are not explicitly modeled in existing pipelines. To tackle the scarcity of night-time data, we collect a novel labeled dataset, named {\it NightCity}, of 4,297 real night-time images with ground truth pixel-level semantic annotations. To our knowledge, NightCity is the largest dataset for NTSP. In addition, we also propose an exposure-aware framework to address the NTSP problem through augmenting the segmentation process with explicitly learned exposure features. Extensive experiments show that training on NightCity can significantly improve NTSP performances and that our exposure-aware model outperforms the state-of-the-art methods, yielding top performances on our dataset as well as existing datasets.

I. INTRODUCTION

Night-time scene parsing is under-served by day-focused methods and scarce labeled night data, while co-occurring over- and under-exposure complicates parsing. The paper introduces NightCity and an exposure-guided, two-stream framework, reporting improved NTSP performance and state-of-the-art results.

I. INTRODUCTION

  • Scene parsing supports applications including human parsing, image editing, and autonomous driving, while prior methods mainly focus on day-time scenes.The authors motivate night-time parsing in part because day and night each cover roughly half the year on average.Source: §I · p. 1
  • Night-time images can combine over- and under-exposure, and existing scene-parsing methods do not explicitly model these conditions.The paper describes resulting detection and segmentation difficulties in dark, bright, and mixed-exposure regions.Source: §I · p. 1
  • The paper identifies two NTSP challenges: scarce large-scale labeled night data and exposure conditions not explicitly modeled by existing methods.Existing large scene-parsing datasets mainly contain day-time images, with few or no night-time images.Source: §I · p. 1
  • NightCity contains 4,297 real night-time images with pixel-wise annotations and is presented as the largest labeled NTSP dataset.It covers more diverse and challenging exposure conditions than Cityscapes and can serve as an NTSP benchmark.Source: §I · p. 1
  • EGNet explicitly learns over- and under-exposure features to guide NTSP through segmentation and exposure streams.The exposure stream predicts an exposure map and augments the segmentation stream via attention.Source: §I · pp. 1–2
  • The proposed model outperforms state-of-the-art methods on night-time scenes in extensive evaluations.The authors also report that NightCity can help advance NTSP performance.Source: §I · pp. 1–2

II. RELATED WORK

Prior scene-parsing work largely targets well-illuminated day-time scenes, using methods such as multi-level features, attention, two-stream architectures, and domain adaptation. This paper focuses instead on night-time scenes with poor lighting.

A. Scene Parsing

  • Multi-level methods combine global context with low-level detail, while attention methods model local-global, long-range, or height-dependent relationships.Examples include position and channel attention, criss-cross attention, and height attention.Source: A. Scene Parsing · p. 2
  • Other scene-parsing work uses object-contextual representations, video prediction, adaptive receptive fields, scale adaptation, and domain adaptation.Domain adaptation methods use knowledge from synthetic scenes to enrich scene representations.Source: A. Scene Parsing · p. 2

II. RELATED WORK

  • Prior scene-parsing methods achieved state-of-the-art results on day-time datasets, whereas this work focuses on poorly lit night-time scenes.The reviewed approaches include multi-level features, attention, two-stream networks, and domain adaptation.Source: A. Scene Parsing · p. 2

B. Scene Parsing in Adverse Conditions

NightCity addresses the scarcity of annotated night scenes, while the paper contrasts its exposure conditions with existing datasets and motivates exposure-aware parsing without image pre-processing.

B. Scene Parsing in Adverse Conditions

  • Earlier adverse-condition work included small or narrowly labeled night datasets, whereas this work provides real-night semantic annotations across 21 categories.The related-work discussion cites examples with 95, 151, and 226 night-time images.Source: B. Scene Parsing in Adverse Conditions · p. 2

C. Image Enhancement/Correction

  • Image enhancement before parsing cannot address the problem well, so the proposed approach uses exposure-aware features without pre-processing.The paper notes that remapping may not recover meaningful values in regions with pixel values close to zero or one.Source: C. Image Enhancement/Correction · pp. 2–3

III. THE NIGHTCITY DATASET

  • Three annotators reconcile labels; 9.8% of pixels initially differed, and 47.9% of those differing annotations were finally corrected.A third annotator reviews disagreements and can also correct labels when both initial annotators agree but are judged incorrect.Source: §III · p. 3
  • NightCity images are challenging even for humans because insufficient lighting, street lights, and headlights create under- and over-exposure.The paper presents the dataset as challenging for both training and evaluation.Source: Figure 2 · p. 3, §III · p. 4
  • NightCity has significantly more under-exposed pixels than Cityscapes, while similar over-exposed pixel counts arise in different scene regions.NightCity over-exposure can come from street lights and headlights, whereas Cityscapes over-exposure is mostly in the sky.Source: Figure 4 · p. 4, §III · p. 4
  • NightCity contains 4,297 finely annotated real night-time images, making it larger than the cited adverse-condition datasets.All images have manually assigned fine class labels.Source: §III · p. 4

IV. EXPOSURE-GUIDED NETWORK

The Exposure-Guided Network augments segmentation with learned exposure features using coupled segmentation and exposure streams. Exposure guidance layers connect the streams so segmentation can better distinguish exposed regions.

IV. EXPOSURE-GUIDED NETWORK

  • EGNet explicitly learns exposure features and uses them to augment the segmentation process.The network is designed around coupled exposure and segmentation streams.Source: §IV · pp. 4–5
  • The segmentation stream predicts a semantic label map, while the exposure stream predicts an exposure map and guides segmentation through exposure guidance layers.The architecture depicts the segmentation stream in blue and the exposure stream in yellow.Source: Figure 5 · p. 5, §IV · p. 5
  • Exposure predictions guide the segmentation stream so it can discriminate under- and over-exposed regions more effectively.Source: §IV · p. 5

A. Network Architecture

The architecture splits encoded image features into segmentation and exposure streams, which output a semantic label map and a pixel-wise exposure map. The exposure map is approximated using the input image’s normalized HSV V channel.

A. Network Architecture

  • A backbone encoder transforms the input image into features, and the two streams output a class label map and a pixel-wise exposure map.The exposure map indicates the magnitude of exposure at each pixel.Source: A. Network Architecture · p. 5
  • The normalized HSV V channel of the input image approximates the exposure map.Source: A. Network Architecture · p. 5
  • The network uses ResNet-101 features, with cross-entropy for segmentation and L1 loss for exposure combined in a weighted final loss.The segmentation output has 19 channels, while the exposure output is a soft binary map.Source: A. Network Architecture · p. 5

B. Exposure Guidance Layer

The Exposure Guidance Layer uses exposure-stream features to refine segmentation features through spatial attention. The discussion distinguishes noisy low-illuminated scenes from night-time urban scenes, where noise is typically negligible.

B. Exposure Guidance Layer

  • The Exposure Guidance Layer uses exposure features to augment intermediate segmentation features, guiding NTSP toward more discriminative features in under- and over-exposed regions.It weights segmentation-stream features based on exposure features and also learns exposure features to predict exposure maps.Source: B. Exposure Guidance Layer · p. 5
  • The layer forms a soft spatial attention map from exposure-stream features; its differentiable operations enable end-to-end training and exposure-map gradients to reach the segmentation stream.The exposure-aware features are then passed to the next network stage.Source: B. Exposure Guidance Layer · p. 5

C. Discussion: Is Noise Always Vital In Night-time Scenes?

  • Man-made light sources typically raise illuminance in night-time urban scenes, allowing noise to be ignored at low ISO when the overall photon count is sufficient.The authors attribute this to a relatively high signal-to-noise ratio.Source: C. Discussion: Is Noise Always Vital… · pp. 5–6
  • Above 10 Lux, noise is negligible in the authors’ night-time urban scenes; below 1 Lux, low-light scenes can have significant noise.The authors conclude their NTSP method does not require denoising mechanisms.Source: C. Discussion: Is Noise Always Vital… · p. 6

V. EXPERIMENTS

The experiments compare scene-parsing methods and analyze model components. They evaluate NTSP with mean IoU and an exposure-aware F1 score that measures performance across exposure levels.

V. EXPERIMENTS

  • The experiments compare the proposed method with mainstream day-time parsing methods, assess NightCity, compare ad hoc NTSP methods, and include ablations.The reported evaluation metrics are mean IoU and exposure-aware F1.Source: §V · p. 6

A. Evaluation Metrics

  • Mean IoU evaluates NTSP performance, while exposure-aware F1 (EF1) evaluates segmentation across exposure levels by grouping night-time images according to exposure.EF1 takes both precision and recall into consideration.Source: A. Evaluation Metrics · p. 6
  • EF1 balances precision and recall within each exposure group, and mEF1 is the mean of the groups’ EF1 scores.The authors set β to 1 for the precision-recall balance.Source: A. Evaluation Metrics · p. 6

B. Comparison to Day-time Scene Parsing Methods

Day-time scene-parsing methods trained on Cityscapes perform significantly worse than the proposed model trained on NightCity. Applying image enhancement before day-time parsing provides no obvious performance gains.

B. Comparison to Day-time Scene Parsing Methods

  • NightCity-trained model results significantly exceed Cityscapes-trained day-time models on mIoU and mEF1.Table III compares the methods on the NightCity test set, including Cityscapes training and image-enhancement preprocessing conditions.Source: B. Comparison to Day-time Scene… · p. 7, Table III · p. 7
  • Image enhancement as preprocessing gives no obvious performance gains for the Cityscapes-trained methods.In the DRHT examples, under-exposed regions improve while over-exposed regions worsen, and the enhanced images remain unlike daytime images.Source: B. Comparison to Day-time Scene… · p. 7, Table III · p. 7
  • The authors report that enhanced images’ different appearance from daytime images causes day-time parsing methods to fail.The examples use DRHT.Source: B. Comparison to Day-time Scene… · p. 7

C. Evaluation of the Proposed NightCity Dataset

The authors evaluate NightCity by training methods on different dataset combinations and by comparison with BDD100K-night. NightCity improves existing methods, and the proposed model performs favorably on examples with varied exposure.

C. Evaluation of the Proposed NightCity Dataset

  • The proposed model outperforms other models across all 10 exposure groups, including near-under- and near-over-exposure.Table IV reports EF1 by exposure level, ordered from under-exposure to over-exposure.Source: C. Evaluation of the Proposed… · p. 7, Table IV · p. 8
  • Training existing methods on NightCity significantly improves mIoU and mEF1 over Cityscapes training; the proposed model outperforms the compared methods and achieves the best mEF1.When trained on both datasets, some existing models gain slightly while others lose slightly.Source: C. Evaluation of the Proposed… · p. 7
  • Training on NightCity significantly improves three methods’ performance on the BDD100K-night test set.BDD100K-night contains 314 training images and 31 test images; the comparison uses PSPNet, CCNet, and HANet.Source: C. Evaluation of the Proposed… · p. 8
  • Qualitative examples show the proposed model handling varied exposures, with sharper building boundaries, recognized traffic lights, and detection of a small tree in an under-exposed region.The comparison also describes clearer sidewalks and a more intact pedestrian segmentation.Source: C. Evaluation of the Proposed… · p. 8

D. Comparison to Night-time Scene Parsing Methods

The method is reported to outperform comparison methods on Dark Zurich, and qualitative comparisons show cleaner or more accurate segmentation under challenging lighting. The visual comparisons include results from methods trained on NightCity.

D. Comparison to Night-time Scene Parsing Methods

  • Our method is reported to outperform seven domain-adaptation methods on Dark Zurich's 151-image evaluation set.The compared methods are GCMA, AdaptSegNet, ADVENT, BDL, DMAda, MGCDA, and DANNet.Source: D. Comparison to Night-time Scene… · p. 8
  • On the eight-image comparison, our model produces clearer tree boundaries, cleaner buildings, and identifies a wall missed by the other methods.Other methods sometimes label sky as road, while our model also detects a clear car shape.Source: D. Comparison to Night-time Scene… · pp. 8–9
  • Figure 8 compares our results with state-of-the-art methods, with all methods trained on NightCity.White boxes mark the model's advantages; yellow boxes mark drawbacks of other methods.Source: Figure 8 · p. 9

E. Model Analysis

The analyses test the model on daytime images and examine exposure guidance, exposure-map alternatives, and learned attention. They report second-best daytime performance, benefits from the full exposure-guidance design, and failures in large under-exposed regions.

Model Robustness on Day-time Dataset

Ablation Study

  • Removing exposure guidance significantly lowers performance, removing the exposure stream lowers it further, and concatenation or sum fusion still trails our model.The ablations compare removing the exposure guidance layer or stream and replacing attention with concatenation or summation.Source: E. Model Analysis · pp. 9–10, Table VIII · p. 11

Strategies of Using the Exposure Map

  • Direct exposure-map attention may favor over-exposed regions and give less attention to under-exposed ones, while learned attention attends to both.The learned attention maps emphasize distant cars with headlights and under-exposed regions.Source: E. Model Analysis · p. 11

Ablation Study

  • The authors attribute the method's effectiveness to learning an exposure representation, then fusing semantic and exposure information through the exposure guidance layer.Source: E. Model Analysis · p. 11

Strategies of Using the Exposure Map

  • mIoU falls from 51.8 to 35.6 when the exposure map is used as extra input; as an attention map, it scores 40.2 versus 40.1 for w/o ES.These comparisons are reported in Table IX and across Tables IX and VIII, respectively.Source: Table IX · p. 11, E. Model Analysis · p. 11

Model Robustness on Day-time Dataset

  • Thin poles and trees in large under-exposed regions can go undetected.Source: Figure 11 · p. 11

VI. CONCLUSION

The paper presents NightCity and an exposure-feature-based two-stream framework for night-time scene parsing. It reports that NightCity benefits existing methods and that its model outperforms existing methods, while noting failures in large under-exposed regions.

VI. CONCLUSION

  • The paper contributes a large real-nighttime dataset with fine semantic annotations and a two-stream framework that explicitly learns exposure features.Source: §VI · pp. 11–12
  • Training on NightCity benefits existing scene-parsing methods on night-time scenes, and the proposed model outperforms existing methods.Source: §VI · p. 12
  • The model may fail to detect objects in large under-exposed regions, including thin poles and trees.The paper identifies these as extremely challenging situations.Source: §VI · p. 12
Loading 2003.06883v3…