Source-linked AI summary
HIT-UAV: A high-altitude infrared thermal dataset for Unmanned Aerial Vehicle-based object detection
Jiashun Suo, Tianyi Wang, Xingzhou Zhang, Haiyang Chen, Wei Zhou, Weisong Shi
TL;DR
UAV object detection lacks broadly applicable datasets that support nighttime operation, privacy protection, flight-condition analysis, and diverse high-altitude scenarios. The paper introduces HIT-UAV, an infrared thermal dataset with flight metadata and standard and oriented annotations, and evaluates established detectors on it. The algorithms exhibit exceptional performance on HIT-UAV compared with visual-light datasets, supporting its use in UAV detection research and potential nighttime search-and-rescue studies.
Problem
Existing UAV datasets have limited nighttime and privacy suitability, insufficient flight metadata, and narrow coverage of scenes, altitudes, or object categories.
Method
The paper constructs HIT-UAV from sampled infrared thermal UAV video frames, records flight and imaging metadata, and provides standard and oriented bounding-box annotations.
Results
The evaluated YOLOv4, YOLOv4-tiny, Faster R-CNN, and SSD algorithms exhibit exceptional performance on HIT-UAV compared with visual-light datasets.
Takeaways & Limitations
HIT-UAV supports research on infrared thermal UAV detection across varied scenes, flight conditions, object categories, and potential nighttime search-and-rescue applications.
Abstract
from arXiv · showhide
We present the HIT-UAV dataset, a high-altitude infrared thermal dataset for object detection applications on Unmanned Aerial Vehicles (UAVs). The dataset comprises 2,898 infrared thermal images extracted from 43,470 frames in hundreds of videos captured by UAVs in various scenarios including schools, parking lots, roads, and playgrounds. Moreover, the HIT-UAV provides essential flight data for each image, such as flight altitude, camera perspective, date, and daylight intensity. For each image, we have manually annotated object instances with bounding boxes of two types (oriented and standard) to tackle the challenge of significant overlap of object instances in aerial images. To the best of our knowledge, the HIT-UAV is the first publicly available high-altitude UAV-based infrared thermal dataset for detecting persons and vehicles. We have trained and evaluated well-established object detection algorithms on the HIT-UAV. Our results demonstrate that the detection algorithms perform exceptionally well on the HIT-UAV compared to visual light datasets since infrared thermal images do not contain significant irrelevant information about objects. We believe that the HIT-UAV will contribute to various UAV-based applications and researches. The dataset is freely available at https://github.com/suojiashun/HIT-UAV-Infrared-Thermal-Dataset.
Background & Summary
HIT-UAV addresses limitations in UAV object-detection datasets by providing diverse, high-altitude infrared thermal imagery with flight metadata and dual bounding-box annotations. Evaluation with established detectors indicates exceptional performance compared with visual-light datasets.
- Dataset motivation: Existing UAV datasets often rely on visual-light imagery, limiting nighttime use and raising privacy concerns.Infrared thermal images can support nighttime imaging while representing people without personal appearance, clothing, or gender information.
- Dataset motivation: Many UAV datasets omit flight altitude and camera perspective, preventing investigation of their influence on detection accuracy.HIT-UAV records altitude, perspective, daylight intensity, and image shooting date for each image.
- Dataset motivation: Existing datasets frequently emphasize synthetic scenes, low altitudes, single scenes, or specific categories, restricting coverage across practical scenarios.High-altitude data can cover broader areas and access regions with tall buildings, while diverse scenes and categories broaden applicability.
- Dataset contribution: HIT-UAV contains 2,898 infrared thermal images extracted from 43,470 frames across schools, parking lots, roads, and playgrounds.The dataset spans altitudes of 60–130 meters and camera perspectives of 30–90 degrees.
- Dataset contribution: HIT-UAV provides standard and oriented bounding boxes for 24,899 objects across five categories, with 2,029 training, 579 test, and 290 validation images.Oriented boxes address substantial overlap between aerial object instances, while standard boxes support efficient dataset use.
- Dataset contribution: The authors describe HIT-UAV as the first publicly available high-altitude UAV-based infrared thermal dataset for detecting persons and vehicles.The dataset was evaluated using YOLOv4, YOLOv4-tiny, Faster R-CNN, and SSD; these algorithms showed exceptional performance compared with visual-light datasets.
Methods
HIT-UAV was constructed from UAV-captured thermal video using documented flight hardware, systematic frame sampling, dual bounding-box annotation, and generated dataset splits.
- Video capture: The capture setup used a DJI Matrice M210 V2 carrying a Zenmuse XT2 longwave infrared camera with 640×512 thermal resolution.The UAV cost approximately 10,000 US dollars and the camera approximately 8000 US dollars.
- Video capture: Videos covered multiple scenes, altitudes from 60 to 130 meters, perspectives from 30 to 90 degrees, and both daytime and nighttime conditions.Flight altitude, camera perspective, date, and daylight intensity were recorded for each video.
- Frame extraction and data cleaning: Sampling one image every 15 frames from 7-FPS video reduced adjacent-frame redundancy and produced 2,898 thermal images from 43,470 original frames.The images have 640×512 resolution, 8-bit depth, and an average compression rate of 21.059%.
- Object annotation: Objects were annotated with standard boxes (xc,yc,w,h) and θ-based oriented boxes (xc,yc,w,h,θ) to reduce overlap in aerial scenes.Oriented boxes cannot exceed image edges, so boundary boxes remain standard; both formats were provided for usability.
- Object annotation: Manual annotation was performed with a modified LabelImg tool, cross-verified by three annotators, and supported conversion from oriented to standard boxes.Difficult and truncated instances were also labeled.
- Dataset generation: A dataset-generation tool created XML and JSON labels and split the data into training, test, and validation sets using a 70%, 20%, and 10% ratio.The tool organized images according to flight data and generated corresponding label files.
Data Records
The dataset records acquisition conditions and supports standard and rotated annotation formats for several object categories, including difficult-to-identify instances.
- Folder structure and recording format: Annotation files are supplied in VOC XML and MS COCO JSON formats, with separate folders for standard and oriented bounding boxes.The folders are normal_json, normal_xml, rotate_json, and rotate_xml.
- Folder structure and recording format: Image filenames encode shooting time, flight altitude, camera perspective, weather condition, and serial number.Day is encoded as 0, night as 1, altitudes range from 60 to 130 meters, and perspectives range from 30 to 90 degrees.
- Object categories: The labeled categories are Person, Car, Bicycle, OtherVehicle, and DontCare.DontCare marks unrecognizable or easily confused objects whose specific type cannot be identified reliably.
- Data distribution: Person is the most prevalent category, while substantial numbers of Car and Bicycle instances support common detection tasks.Flight altitudes were recorded at 10-meter intervals from 60 to 130 meters for high-altitude missions.
- Data distribution: The dataset includes sample images distinguishing nighttime and daytime imagery and showing how object appearance varies across altitude and camera perspective.At 80 meters, objects appear smaller at 30° and 90° than at 50° under the described field-of-view conditions.
Technical Validation
Technical validation evaluates four established detectors on HIT-UAV and examines how infrared imagery, altitude, and camera perspective relate to detection performance.
- Baseline evaluation: Four detectors were trained on 2,029 training, 290 validation, and 579 test images: YOLOv4, YOLOv4-tiny, Faster-RCNN, and SSD.YOLOv4 and YOLOv4-tiny used Darknet, while Faster-RCNN with ResNet-101 and SSD-512 used MMDetection.
- Baseline evaluation: 84.75% mAP was achieved by YOLOv4 on the HIT-UAV10 test set.The comparison includes models trained on HIT-UAV10, YOLOv4 variants trained on COCO, and RRNet’s VisDrone-2019 result.
- Baseline evaluation: Infrared thermal images were reported to filter extraneous information and support strong identification performance.The authors also report strong performance from common detectors with limited image data because object features are easily recognizable.
- Altitude and perspective analysis: YOLOv4 AP remained stable within a certain altitude and angle range, indicating limited impact from these variations for robust algorithms.This observation concerns the YOLOv4 results analyzed across altitude and camera perspective.
- Altitude and perspective analysis: YOLOv4-tiny Person AP tended to decrease as altitude increased, reflecting sensitivity when objects moved outside a suitable size range.The decrease was observed across 60–80 m, 80–90 m, and 100–130 m stages.
- Altitude and perspective analysis: YOLOv4-tiny Person AP first increased and then decreased as camera angle increased, with 90° views making individuals appear as points.The reported pattern highlights the role of visible object surface in detection precision.
Usage Notes
HIT-UAV is openly available with common annotation formats and was collected across varied environments, supporting training and investigation of nighttime UAV applications.
- Access and formats: The dataset can be downloaded from GitHub with labels in VOC and MS COCO formats for object-detection training.These formats are commonly used benchmarks, and users can readily use the dataset.
- Potential applications: Images were collected in schools, parking lots, roads, and playgrounds, allowing models to be applied to these scenarios and other environments through deep-learning generalization.The authors identify nighttime UAV search-and-rescue feasibility as a potential application.
Code availability
The HIT-UAV data-processing tools support annotation conversion, visualization, dataset generation, and YOLO-format label creation.
- labelTransformer.py converts oriented bounding boxes to standard bounding boxes and generates the dataset.
- visualization.py displays images with bounding boxes.
- voc2yolo.py generates YOLO-format label files for training YOLO object detectors.