UG$^{2+}$ Track 2: A Collective Benchmark Effort for Evaluating and Advancing Image Understanding in Poor Visibility Environments

Ye Yuan; Wenhan Yang; Wenqi Ren; Jiaying Liu; Walter J. Scheirer; Zhangyang Wang

Source-linked AI summary

UG$^{2+}$ Track 2: A Collective Benchmark Effort for Evaluating and Advancing Image Understanding in Poor Visibility Environments

Ye Yuan, Wenhan Yang, Wenqi Ren, Jiaying Liu, Walter J. Scheirer, Zhangyang Wang

arXiv:1904.04474v4Apr 9, 2019cs.CV

TL;DR

Poor-visibility enhancement is expected to help high-level recognition, but existing methods are not consistently aligned with detection performance. The paper introduces three real-world benchmarks and evaluates enhancement–detection cascades, finding difficult tasks and substantial room for improvement.

Problem
Existing enhancement and recognition methods are often developed separately, and it remains questionable whether restoration-based preprocessing improves high-level visual understanding.
Method
The paper constructs three real-world poor-visibility benchmarks for haze, low light, and rain, with annotated detection data and varied supervision protocols.
Results
The benchmarks are highly challenging: winners remain below 65 mAP in Challenges 2.1 and 2.2, while no participant surpasses the baseline in Challenge 2.3.
Takeaways & Limitations
The results indicate substantial room for technical innovation in robust detection under realistic poor-visibility conditions.

Abstract

from arXiv · show

The UG$^{2+}$ challenge in IEEE CVPR 2019 aims to evoke a comprehensive discussion and exploration about how low-level vision techniques can benefit the high-level automatic visual recognition in various scenarios. In its second track, we focus on object or face detection in poor visibility enhancements caused by bad weathers (haze, rain) and low light conditions. While existing enhancement methods are empirically expected to help the high-level end task, that is observed to not always be the case in practice. To provide a more thorough examination and fair comparison, we introduce three benchmark sets collected in real-world hazy, rainy, and low-light conditions, respectively, with annotate objects/faces annotated. To our best knowledge, this is the first and currently largest effort of its kind. Baseline results by cascading existing enhancement and detection models are reported, indicating the highly challenging nature of our new data as well as the large room for further technical innovations. We expect a large participation from the broad research community to address these challenges together.

I. INTRODUCTION

The paper frames poor-visibility recognition as a joint human–machine vision challenge and introduces three realistic benchmarks for evaluating object and face detection under haze, low light, and rain. Results indicate that these tasks remain highly challenging, with substantial room for improvement.

Motivation: Existing enhancement and visual-understanding methods are usually developed separately, despite real applications requiring robust sensing in unconstrained degraded environments.The paper identifies mismatched training and testing conditions and task-specific guidance as contributors to fragile real-world systems.
Motivation: Restoration may not improve recognition because enhancement quality is not necessarily aligned with the target high-level task.The paper specifically questions whether restoration-based approaches boost visual understanding when enhancement is treated as post-processing.
Benchmark design: The challenge provides realistic benchmarks covering synthetic and real haze, under-exposure, rain streaks, and raindrops.These datasets are designed to support measurement of degradation properties and development of methods that recover cleaner images.
Benchmark design: The three sub-challenges support full-, semi-, unsupervised, and zero-shot learning through paired, unpaired, annotated, and unavailable training data.Challenges 2.1 and 2.2 include paired and unpaired data, while Challenge 2.3 provides no training data and is therefore zero-shot.
Results: Winners remain below 65 mAP in Challenges 2.1 and 2.2, while no participant surpasses the baseline in Challenge 2.3.These outcomes are reported as evidence that the datasets are highly challenging and leave substantial room for technical improvement.
Scope: The challenge evaluates high-level machine vision and, through paired data in Challenges 2.1 and 2.2, can also support human-vision evaluation.The paired data enables assessment of enhancement for human visual experience alongside recognition-oriented tasks.

II. RELATED WORK

Related work spans image-restoration datasets, recognition in degraded environments, and visibility-enhancement methods. The paper positions UG2+ within this literature by focusing on dehazing, low-light enhancement, and deraining for poor-visibility conditions.

Datasets: Earlier restoration datasets primarily evaluate PSNR, SSIM, or subjective visual quality, with later datasets expanding scene diversity and scale.Examples include Set5, Set14, LIVE1, BSD500, Urban100, DIV2K, and MANGA109.
Recognition datasets: A small number of prior datasets and studies address recognition in degraded environments, including UFDD for face detection under adverse conditions.UFDD contains 6,425 images with 10,897 face annotations, according to the cited passage.
Enhancement methods: The paper focuses its visibility-enhancement review on dehazing, low-light enhancement, and deraining within the UG2+ Track 2 scope.Other enhancement areas mentioned include denoising, inpainting, deblurring, super-resolution, and interpolation.
Dehazing: Dehazing research progresses from handcrafted priors and depth statistics to CNN-based estimation and unified modeling of haze variables.The literature also includes night-time dehazing and methods such as AOD-Net that jointly consider dehazing and object detection.
Deraining: Deraining methods include rain modeling, joint rain detection and removal, density-aware multi-stream networks, and temporal cues for video.The cited work addresses both rain streaks and accumulation, with video methods additionally using motion information.

C. Visual Recognition under Adverse Conditions

Recognition performance can remain stable under mild degradation but may drop sharply beyond a threshold, motivating approaches that account for degraded inputs directly. Prior work shows that separate enhancement can introduce artifacts, while joint restoration and recognition can improve performance.

Recognition under degradation: Mild degradation may barely change recognition, but performance can drop substantially once degradation exceeds a threshold.The passage gives a 32×32-pixel resolution example associated with significant drops in object and scene recognition.
Recognition under degradation: Separate enhancement before recognition may fail because artifacts introduced during the first stage can damage the downstream task.This limitation is described for methods that enhance an input image and then forward it to a classifier.
Joint approaches: Joint optimization has been applied to restoration and recognition, including dehazing with object detection and low-resolution recognition.The cited work reports better performance for two-stage joint optimization than previous one-stage methods in dehazing and detection.
Haze benchmark: The haze benchmark uses 4,322 annotated real-world training or validation images, 4,807 unannotated images, and a held-out test set of 2,987 images.Five traffic-object categories are annotated with tight bounding boxes, and the held-out set uses the same classes.

B. (Semi-)Supervised Face Detection in the Low Light Condition

DARK FACE provides a large low-light face-detection benchmark with annotated and unannotated images, plus optional paired low-light/normal-light data. Its images capture substantial variation in face scale, pose, occlusion, appearance, and illumination.

DARK FACE contains 10,000 underexposed images with annotated faces, 9,000 similarly captured unannotated images, and 789 optional paired low-light/normal-light images.
The dataset includes 43,849 annotated faces in training and evaluation and 37,711 annotated faces in the held-out test set.
DARK FACE captures faces at varied scales and poses under under-exposure, with additional degradations including intensive noise.
Each annotated image contains 1-34 faces, while face resolutions range from 1×2 to 335×296 pixels.

C. Zero-Shot Object Detection with Raindrop Occlusions

The rain sub-challenge evaluates object detection on real rainy driving and surveillance imagery, including raindrop occlusions and other degradations. Participants may use pretrained or external data under reproducibility requirements, and ranking uses mAP on held-out data.

The held-out test set contains 2,495 real rainy images from high-resolution driving videos collected across diverse traffic locations and scenes.
Rainy test images contain raindrops on the camera lens alongside accumulation, blurring, reflectance, and occlusion, creating domain shift.
Bounding boxes cover five traffic-object categories: car, person, bus, bicycle, and motorcycle.
Participants may use pretrained models or external data, but submissions must disclose these resources and ensure they were publicly available for reproducibility.
Ranking uses mean average precision on each held-out test set with a default IoU threshold of 0.5, followed by higher thresholds for ties.

1) Baseline Composition:

The baseline combines pretrained object detectors with off-the-shelf dehazing models in a cascade. On original real hazy images, pretrained detectors miss many objects and achieve low overall detection performance.

Baseline Composition: The baseline tests Mask R-CNN, RetinaNet, YOLO-V3, and Feature Pyramid Network as object detectors.
Baseline Composition: It pairs these detectors with AOD-Net, MSCNN, and DCPDN, using officially released dehazing models.
Baseline Composition: 41.83% mAP is obtained with Mask R-CNN and 42.54% with YOLO-V3 on original hazy images, without preprocessing or dehazing.
Baseline Composition: The pretrained detectors fail to predict a large amount of objects in hazy images; person has the highest category AP and bus the lowest.
Baseline Composition: The held-out test set has a relatively larger image size, while its bounding-box count, size, and relative scale distributions are similar to RTTS.
Baseline Composition: Synthetic haze is generated with transmission assumed correlated with scene depth, whereas real haze may be depth-uncorrelated and more variable within one image.

3) Effect of Dehazing:

Dehazing can improve both image quality and detection in the hazy benchmark, but visual quality does not consistently predict detection performance. Real haze and lighting effects remain challenging for pretrained detectors.

Effect of Dehazing: DCPDN and MSCNN dehazed images achieve detection mAPs 1% higher on average than hazy images.
Effect of Dehazing: MSCNN has much better detection performance than DCPDN despite DCPDN having the best SSIM scores and MSCNN the worst visual quality.
Effect of Dehazing: The comparison evaluates four pretrained face detectors together with seven light or contrast enhancement algorithms for low-light imagery.
Effect of Dehazing: Figure 7 compares ground-truth boxes on hazy images with Mask R-CNN detections on hazy and AOD-Net-, MSCNN-, or DCPDN-dehazed images.
Effect of Dehazing: Real haze can be uncorrelated with scene depth and can vary across objects at different depths, unlike the more homogeneous distribution produced by synthetic haze.

2) Results and Analysis:

On DARK FACE, pretrained face detectors struggle under severe low-light degradation, while enhancement improves precision but does not restore normal-light performance.

Without preprocessing or adaptation, state-of-the-art face detectors trained on WIDER FACE do not achieve desirable detection rates on DARK FACE.The results suggest that prior face datasets do not sufficiently capture facial features under highly under-exposed conditions.
Heavy degradation, small scale, pose variation, and occlusion produce both false negatives and false positives in low-light face detection.The failure analysis associates false negatives with these factors and reports false positives in the examined cases.
Enhancement increases detector precision in most instances compared with unenhanced data, with different methods showing similar improvements.The comparison uses two pretrained face detectors on enhanced versions of the annotated dataset.
Overall detector performance remains substantially below normal-light datasets, leaving considerable room beyond the simple low-light-enhancement and face-detection cascade.

C. Sub-challenge 2.3 Baseline Results

The rainy-image sub-challenge evaluates deraining followed by object detection and finds that existing deraining methods generally hurt detection rather than help it.

Five deraining algorithms—JORDER, DDN, CGAN, DID-MDN, and DeRaindrop—are retrained on the provided training set for comparison.The evaluation combines these methods with four object detectors: FRCNN, YOLO-V3, SSD-512, and RetinaNet.
Almost all existing deraining algorithms reduce object-detection performance relative to directly using rainy images for YOLO-V3, SSD-512, and RetinaNet.FRCNN is the only reported exception.
The reported explanation is that deraining is not trained for detection and may remove discriminative, semantically meaningful information.
YOLO-V3 achieves the best detection performance independently of the deraining algorithm, which the authors attribute to its multi-scale prediction structure for small objects.
More than 260 teams registered, 82 completed final submissions, and six teams were selected as winners across the three sub-challenges.

A. Sub-challenge 2.1: Competition Results and Analysis

In Sub-challenge 2.1, competition methods substantially outperform the baseline, with the strongest teams favoring end-to-end adapted detection over explicit dehazing cascades.

Competition results: Seven teams outperform the best baseline mAP 18.09, while HRI DET and superlab403 reach record-high mAP values of 52.71 and 49.22.
Method analysis: The top two teams use end-to-end trained or adapted detectors on hazy training data without an explicit dehazing module.This differs from the dehazing-detection cascade used by many other teams.
Method analysis: Using unlabeled hazy data appears promising, while multi-scale testing and ensembling contribute to reported performance gains.
Winning method: HRI DET combines Faster R-CNN with a ResNeXt-101 and FPN backbone, mixed-dataset tuning, semi-supervised reconstruction training, and three-scale testing.
Runner-up method: The runner-up uses Cascade R-CNN with a ResNeXt-101 backbone, target-specific anchor ratios, label cleaning, and augmentations including blur and illumination changes.
Other methods: Mt. Star ranks third with mAP 31.24 using jointly tuned dehazing and Faster R-CNN models plus multi-scale testing, while ilab ranks sixth with mAP 19.15.
Cross-challenge comparison: In Sub-challenge 2.2, three teams outperform the best baseline mAP 39.30, and CAS-Newcastle and CAS NEU achieve mAP values of 62.45 and 61.84.The most successful solutions there use enhancement-detection cascades, unlike the dominant trend in Sub-challenge 2.1.

C. Sub-challenge 2.3: Competition Results and Analysis

Sub-challenge 2.3 evaluates detection under realistic rainy conditions without closely matched training data, making it a difficult zero-shot setting. No team surpassed the baseline, while the results expose limitations in existing rain datasets and separated enhancement-detection pipelines.

Competition results: No participating team outperformed the baseline in the zero-shot Sub-challenge 2.3.The challenge provides no training set close to the testing distribution, making the task closer to zero-shot or unsupervised learning.
Data characteristics: Real rain test images contain accumulation, blurring, reflectance, and occlusions beyond rain streaks or raindrops.These degradations arise in driving and surveillance scenarios and increase restoration and detection difficulty.
Data characteristics: Existing paired rain datasets are poorly related to real-world object detection because they typically model only one degradation or use visually inauthentic synthetic rain.Rain800 and raindrop datasets focus on rain streaks or raindrops, while synthetic and captured paired images may retain clear backgrounds and appropriate object placement.
Submitted methods: Deep learning dominated submissions, while some low-light challenge teams used hand-crafted enhancement as preprocessing.The reported flexibility of deep learning methods comes from tuning them to the task and dataset.
Submitted methods: Semi-supervised learning with reconstruction loss improved performance for the top two teams in Sub-challenge 2.1, highlighting unlabeled data as a direction for future work.The method reconstructs the full image from intermediate features.
Submitted methods: Enhancement and detection choices varied by task, and separate preprocessing could reduce detection performance.One winner used one-step detection without implicit dehazing, another cascaded enhancement and detection, and several dehazing or deraining combinations underperformed direct detection.
Overall assessment: The three tasks remain highly challenging: winners stayed below 65 MAP in Sub-challenges 2.1 and 2.2, and no Sub-challenge 2.3 participant exceeded the baseline.These outcomes indicate substantial room for further improvement.

VII. CONCLUSIONS

The challenge presents poor-visibility vision as a difficult, under-explored practical problem and reports broad participation. It concludes that substantial room for improvement remains, with UG2+ intended to continue as an evolving benchmark effort.

VII. CONCLUSIONS: The three Track 2 sub-challenges were widely regarded as difficult, under-explored, and meaningful computer vision problems in practice.The conclusion attributes this assessment to post-challenge feedback from participating teams.
VII. CONCLUSIONS: The large volume of team participation produced promising progress, but substantial room for improvement remains.The conclusion frames participation as a source of progress rather than as evidence that the challenges are solved.
VII. CONCLUSIONS: The authors intend UG2+ to become a recurring event with evolving and updated problems and datasets.The stated goal is to continue broad community engagement with these challenges.