Source-linked AI summary
LLVIP: A Visible-infrared Paired Dataset for Low-light Vision
Xinyu Jia, Chuang Zhu, Minzhen Li, Wenqi Tang, Shengjie Liu, Wenli Zhou
TL;DR
Low-light vision lacks datasets that combine useful visible detail, infrared target information, strict alignment, and pedestrian annotations across relevant tasks. The paper introduces LLVIP with aligned, annotated low-light pairs and evaluates fusion, detection, and translation methods. Across these tasks, the experiments indicate that existing algorithms still need substantial improvement.
Problem
Existing visible-infrared datasets do not simultaneously provide large scale, strict alignment, low-light scenes, and many pedestrians for low-light visual tasks.
Method
LLVIP constructs binocular-camera visible-infrared pairs, registers and crops them for alignment, and labels pedestrians using aligned infrared images.
Results
Experiments on LLVIP indicate that existing image fusion, low-light pedestrian detection, and image-to-image translation methods all need improvement.
Takeaways & Limitations
LLVIP provides aligned, low-light, pedestrian-annotated data for fusion, detection, translation, registration, and related studies.
Takeaways & Limitations
Existing fusion algorithms lose dark visible details and perform worse when source-image intensities are extremely low or high.
Abstract
from arXiv · showhide
It is very challenging for various visual tasks such as image fusion, pedestrian detection and image-to-image translation in low light conditions due to the loss of effective target areas. In this case, infrared and visible images can be used together to provide both rich detail information and effective target areas. In this paper, we present LLVIP, a visible-infrared paired dataset for low-light vision. This dataset contains 30976 images, or 15488 pairs, most of which were taken at very dark scenes, and all of the images are strictly aligned in time and space. Pedestrians in the dataset are labeled. We compare the dataset with other visible-infrared datasets and evaluate the performance of some popular visual algorithms including image fusion, pedestrian detection and image-to-image translation on the dataset. The experimental results demonstrate the complementary effect of fusion on image information, and find the deficiency of existing algorithms of the three visual tasks in very low-light conditions. We believe the LLVIP dataset will contribute to the community of computer vision by promoting image fusion, pedestrian detection and image-to-image translation in very low-light applications. The dataset is being released in https://bupt-ai-cz.github.io/LLVIP. Raw data is also provided for further research such as image registration.
1. Introduction
LLVIP addresses the lack of a large, aligned visible-infrared dataset combining low-light scenes and many pedestrians. It constructs and annotates such pairs, then evaluates fusion, pedestrian detection, and image-to-image translation, finding substantial challenges for existing methods.
- Motivation: Low-light visible images lose effective target areas, while infrared images highlight pedestrians but lack texture; fusion can combine these complementary properties.Visible images provide texture and detail, whereas infrared images are less affected by lighting conditions.
- Motivation: Existing datasets do not simultaneously provide large scale, strict alignment, low-light scenes, and many pedestrians for fusion and low-light detection.The paper identifies this combination as an unmet dataset requirement.
- Dataset construction: LLVIP uses a binocular camera, registration, and cropping to produce visible-infrared pairs with consistent time, space, field of view, and size.The construction process supports image fusion and image-to-image translation.
- Dataset construction: LLVIP labels pedestrians by reverse-mapping annotations from aligned infrared images to low-light visible images.The method addresses the difficulty of accurately identifying and marking pedestrians directly in dark visible images.
- Evaluation: Evaluations cover image fusion, pedestrian detection, and image-to-image translation, and report that LLVIP remains highly challenging for existing methods.The paper reports poor image-to-image translation performance and substantial room for improvement in pedestrian detection.
2. Related Datasets
Existing visible-infrared datasets support particular applications but differ in scale, alignment, lighting, scene type, and pedestrian coverage. These limitations motivate LLVIP's combination of aligned pairs, low-light scenes, and pedestrian-focused data.
- TNO: TNO is widely used for visible-infrared image fusion but contains only 261 pairs, many similar consecutive images, and few pedestrians.These properties limit its suitability for deep-learning fusion and post-fusion object detection.
- INO: INO provides varied scenes and weather conditions but contains few pedestrians and few low-light images.Its scene diversity is not matched by coverage of the target low-light pedestrian setting.
- OTCBVS OSU: OTCBVS OSU Color-Thermal images contain many pedestrians, but all were collected during daytime when pedestrians are already clear in visible images.Under these conditions, the advantages of infrared imagery are less prominent.
- CVC-14: CVC-14 targets automatic pedestrian detection and driving, but its images are not dark enough and its visible-infrared pairs are not strictly aligned in time.The temporal mismatch prevents its use for image fusion.
- FLIR: FLIR supports CNN training for automotive applications, but its visible and infrared images are not registered and therefore cannot be used for image fusion.The registration limitation distinguishes it from datasets designed for aligned fusion.
3. The LLVIP Dataset
LLVIP is a high-quality, strictly synchronized and spatially registered visible-infrared dataset collected under low-light conditions, with pedestrian annotations and applications across several visual tasks.
- Image capture and registration: A binocular camera platform captures visible and infrared images, which are registered and cropped to share the same field of view and image size.Registration uses manually selected corresponding points, projection transformation, and cropping.
- Dataset collection: 15488 visible-infrared image pairs containing pedestrians were collected from 26 different locations after time alignment and manual filtering.Each pair is time-synchronized and described as high quality.
- Annotations: Pedestrians are labeled on infrared images and copied to aligned visible images, addressing the difficulty of accurate annotation in low-light scenes.Pedestrians are more distinguishable in infrared images than in low-light visible images.
- Advantages and applications: Synchronous visible-infrared pairs support image fusion and supervised image-to-image translation, while low-light conditions support fusion and pedestrian detection.Infrared images provide supplementary information for low-light visible images.
- Advantages and applications: The dataset provides high-resolution paired images, with original visible resolution of 1920 × 1080 and infrared resolution of 1280 × 720.The paper characterizes LLVIP as high quality compared with other visible-infrared paired datasets.
- Disadvantages: Most images contain medium-distance, medium-sized pedestrians, so LLVIP is not suitable for long-distance small-target pedestrian detection.This is identified as a dataset disadvantage and scope boundary.
4. Tasks
LLVIP supports image fusion, low-light pedestrian detection, and image-to-image translation by pairing complementary visible and infrared information. The evaluated methods expose both fusion behavior and the difficulty of existing pedestrian-detection algorithms in poor light.
- Image Fusion: Image fusion combines visible-image details with infrared heat-source prominence into a single image.The dataset is intended to evaluate fusion methods, including deep learning approaches.
- Image Fusion: Fusion methods include deep learning models such as DenseFuse and FusionGAN, which use feature encoding or adversarial generation to combine source information.DenseFuse uses dense encoder connections and fusion strategies, while FusionGAN uses infrared intensity and visible gradients or texture.
- Fusion metrics: Fusion performance is evaluated with multiple metrics, including EN, MI, SSIM, Qabf, and VIFF, because no single metric is sufficient.These metrics assess information content, similarity, transferred salient information, and visual information fidelity.
- Low-light Pedestrian Detection: Low-light pedestrian detection remains limited, partly because pedestrians are difficult to label accurately in visible images; aligned infrared labeling addresses this difficulty.The paper evaluates YOLOv3 and YOLOv5 on LLVIP.
- Low-light Pedestrian Detection: YOLOv3 and YOLOv5 experiments demonstrate that existing pedestrian-detection algorithms do not perform well in low-light conditions.The result is reported for the LLVIP dataset.
- Image-to-image Translation: Image-to-image translation converts images between domains and can construct infrared data from more easily obtained visible images.The paper discusses physical-model and deep-learning approaches, noting that thermal-image mappings are difficult to manually model accurately.
5. Experiments
Experiments evaluate image fusion, pedestrian detection, and image-to-image translation on LLVIP. Existing methods show complementary benefits but substantial weaknesses in very low-light conditions.
- Image Fusion: Densefuse l1 and IFCNN perform best overall for nighttime fusion, retaining visible detail while highlighting human bodies.Subjective evaluation found these methods preserved more information from both modalities.
- Image Fusion: Fusion preserves bright visible details such as license plates and traffic lights, but loses dark scene textures and clothing information.Leaves, stones, and people’s clothing are among the missing details.
- Image Fusion: When one source image has very low or very high pixel intensity, fusion quality worsens because algorithms poorly balance the two sources.The authors conclude that existing fusion algorithms still have substantial room for improvement.
- Pedestrian Detection: Infrared images highlight pedestrians and achieve better detection effects than visible images, which contain many missed detections in low light.The experiments use YOLOv5 and YOLOv3 with visible and infrared inputs.
- Image-to-image Translation: pix2pixGAN performs very poorly on LLVIP, producing messy backgrounds, unclear object contours, incorrect details, artifacts, and extremely low SSIM and PSNR.Its performance is reported as much worse on LLVIP than on the KAIST multispectral pedestrian dataset.
6. Conclusion
LLVIP is a strictly aligned visible-infrared dataset containing pedestrians, low-light images, and detection annotations. Experiments indicate that fusion, low-light pedestrian detection, and image-to-image translation still require improvement.
- Conclusion: LLVIP provides aligned visible-infrared pairs, many low-light pedestrian scenes, and annotations for pedestrian detection.The dataset supports fusion, low-light pedestrian detection, image-to-image translation, image registration, and domain adaptation.
- Conclusion: Experiments on LLVIP indicate that visible-infrared fusion, low-light pedestrian detection, and image-to-image translation all need improvement.