Source-linked AI summary
Real-Time Drone Detection and Tracking With Visible, Thermal and Acoustic Sensors
Fredrik Svanstrom, Cristofer Englund, Fernando Alonso-Fernandez
TL;DR
The paper addresses the need for robust drone detection amid scarce multi-sensor evidence and public datasets. It builds an automatic system combining visible, thermal infrared, acoustic, and auxiliary sensors, and evaluates sensor performance, distance effects, fusion, and false detections. Thermal detection performs similarly to visible detection, while fusion improves system-level robustness and the work contributes public multi-sensor data.
Problem
The paper addresses limited research on multi-sensor fusion, distance-dependent classification, and public benchmark datasets for distinguishing drones from other flying objects.
Method
The system combines thermal infrared, visible video, audio, fish-eye motion sensing, pan/tilt control, and ADS-B data, with machine-learning detectors coordinated by parallel software workers.
Results
78% of detection opportunities produced a drone classification after sensor fusion, compared with 67% for the visible-camera detector.
Takeaways & Limitations
Thermal infrared is feasible for drone detection, with performance similar to visible video, while sensor fusion reduces false detections at system level.
Abstract
from arXiv · showhide
This paper explores the process of designing an automatic multi-sensor drone detection system. Besides the common video and audio sensors, the system also includes a thermal infrared camera, which is shown to be a feasible solution to the drone detection task. Even with slightly lower resolution, the performance is just as good as a camera in visible range. The detector performance as a function of the sensor-to-target distance is also investigated. In addition, using sensor fusion, the system is made more robust than the individual sensors, helping to reduce false detections. To counteract the lack of public datasets, a novel video dataset containing 650 annotated infrared and visible videos of drones, birds, airplanes and helicopters is also presented (https://github.com/DroneDetectionThesis/Drone-detection-dataset). The database is complemented with an audio dataset of the classes drones, helicopters and background noise.
I. INTRODUCTION
Drone detection matters because misuse can threaten safety, while research still lacks robust multi-sensor evidence and public benchmark datasets. This work addresses these gaps across multiple sensing technologies and target classes.
- Drones can support socially beneficial applications but may threaten safety when misused, including through collisions with aircraft.
- Multi-sensor fusion is identified as important for improving detection accuracy, but prior research in this direction is scarce.
- The study addresses the lack of public reference databases by collecting and annotating data from multiple sensors.
- The target classes include three consumer drones together with birds, airplanes, and helicopters to capture objects that may be mistaken for drones.
- Related work spans visible, thermal infrared, acoustic, radar, radio-frequency, laser, human, and animal-based detection approaches.
III. METHODS AND MATERIALS
The methods section presents the automatic detection system at system level and in hardware and software detail, responding to incomplete reporting in prior studies.
- The system description covers hardware and software after prior studies indicated that key acquisition and evaluation details were often unspecified.
- Earlier work often failed to specify the acquisition device, drone type, detection range, or dataset.
- The paper organizes its method around an automatic drone detection system described first at system level and then in detail.
A. System Architecture
The architecture combines wide-field motion sensing with steerable visible, thermal, and acoustic sensing, plus aircraft tracking, to support portable real-time airspace monitoring.
- The system combines wide- and narrow-field cameras on a moving pan/tilt platform to balance airspace coverage with target resolution.
- The thermal infrared and visible cameras perform primary detection, while audio detects nearby drones or helicopters and ADS-B tracks cooperative aircraft.
- The fish-eye camera covers 180° horizontally and 90° vertically to detect moving objects and guide the narrower cameras toward directions of interest.
- The thermal infrared camera uses a 320×256-pixel detector with a 24° horizontal and 19° vertical field of view.
- The platform uses servo-driven pan and tilt hardware controlled from a laptop, with outdoor power supplied through a battery-compatible DC-DC solution.
C. Software
Deployment software uses parallel sensor workers coordinated by a main script, enabling asynchronous detection, tracking, platform control, and user-interface updates.
- Deployment software consists of a main script and five independently runnable workers operating as parallel Matlab threads.
- Parallel workers process sensors asynchronously to handle frames without inter-sensor delays or waiting time.
- The main script coordinates worker communication, servo commands, platform position updates, and the graphical user interface.
- The fish-eye worker detects moving objects with a GMM foreground detector and tracks them using a multi-object Kalman filter.
- The infrared and visible workers run trained YOLO v2 detectors and return target classes, confidence, and image-center offsets for servo control.
D. Graphical User Interface
The graphical interface integrates displays for ADS-B, platform and camera fields of view, tracking history, altitude, and sensor-worker results. It also exposes control and operational information for the pan/tilt system.
- Interface overview: The GUI combines ADS-B displays with altitude information and a Plan Position Indicator.The ADS-B area presents aircraft-related information through a PPI-type display and altitude display.
- Spatial visualization: Dashed green lines show pan/tilt motion and fish-eye-camera fields of view, while red lines represent the platform direction and primary-camera fields of view.
- Tracking visualization: A solid cyan line marks the tracked object direction and displays its track history.
- Dataset displays: The interface includes output-class and infrared/visible-video distribution tables for the sensor datasets.
- Results and control: The center results panel reports worker outputs, servo angles, time, GPS position, elevation, azimuth, and detected ADS-B targets.
IV. DRONE DETECTION DATASET
The dataset combines recordings from three Swedish airports, three drone types, varied daylight weather, and supplementary aircraft footage. It contains public, annotated infrared, visible, and audio clips organized for sensor and distance-based evaluation.
- Collection setting: Recordings were collected at three Swedish airports using Hubsan H107D+, DJI Phantom 4 Pro, and DJI Flame Wheel F450 drones.The drones range from 0.1 m to 0.4 m motor-to-motor side length.
- Collection setting: The dataset spans clear, partly cloudy, and overcast weather, but flight recordings were made in daylight because the drones had to remain within visual range.The system can nevertheless operate at night using thermal infrared and acoustic sensors.
- Dataset contents: 650 ten-second videos contain 203328 annotated images, with 365 infrared videos at 320×256 pixels and 285 visible videos at 640×512 pixels.The dataset also contains 90 audio clips and reaches 200 m for the greatest drone sensor-to-target distance.
- Dataset contents: Supplementary non-copyrighted airplane and helicopter clips were added because the study could not film all suitable targets during reduced flight operations.
- Distance organization: Video samples are divided into Close, Medium, and Distant bins using Detect, Recognize, and Identify requirements.The bins correspond to target widths of 15 pixels for identification, 15 to 5 pixels for recognition, and below 5 pixels for the Distant category in the IRcam image.
V. RESULTS
The evaluation measures individual-sensor detection using precision, recall, and F1-score across distance bins, with separate evaluation data for video and audio. The infrared detector achieves an average F1-score of 0.7601, while higher thresholds trade recall for precision and clouds can cause false alarms.
- Evaluation design: Evaluation measures precision, recall, and F1-score for individual sensors and after sensor fusion across distance-based video data.The video evaluation uses 120 infrared and 120 visible clips, while the audio evaluation includes five ten-second clips from each output category.
- Thermal infrared results: 0.7601 is the average of the three F1-scores reported for the thermal infrared sensor.
- Thermal infrared results: A higher detection threshold, such as 0.8, increases precision at the cost of lower recall, especially as sensor-to-target distance increases.
- Thermal infrared results: Small clouds and sunlit edges of large clouds are a common source of false alarms for the infrared detector.
B. Visible Camera (Vcam)
The visible-camera detector uses the same methods and settings as the infrared detector, differing primarily in YOLO input size. Its results differ by no more than 3% from infrared detection, supporting comparable performance despite the infrared camera’s lower resolution and grayscale imagery.
- Detector setup: The visible-camera detector follows the infrared detector’s methods and settings but uses 416×416 YOLO input images.The visible-camera training set contains 37519 images, with confidence and IoU thresholds of 0.5.
- Performance comparison: The visible-camera and infrared-detector results differ by no more than 3%.
- Performance comparison: The infrared sensor performs as well as the visible sensor despite its lower resolution and grayscale images.The visible-camera input image size is 1.625 higher than the infrared-camera input size.
- Performance comparison: A prior YOLO v2 result reported an F1-score of 0.728, just below the infrared and visible results in this work.That prior detector used one output class and did not report input image size or sensor-to-target distance.
C. Audio Sensor
The audio sensor uses MFCC features with an LSTM classifier to identify drones, helicopters, and background noise. Its F1-score exceeds a prior binary MFCC-LSTM result.
- Related sensor result: The visible-camera results table reports an average of 0.7849 across three F1-scores.The supplied table caption identifies this as the visible-camera result, not the audio-detector result.
- Results: The audio-detector results are presented in Table V.The supplied passage does not include the individual Table V values.
- Results: The audio detector achieves a higher F1-score than the 0.6984 reported by a prior MFCC-based LSTM-RNN classifier.The prior classification task was binary, distinguishing drones from background.
D. Sensor Fusion
Sensor fusion combines sensor class outputs and confidence scores, smooths results over time, and supports configurable sensor requirements. In evaluation, fused outputs were more stable, covered more detection opportunities than the visible camera, and suppressed listed false detections at system level.
- Fusion method: Sensor fusion combines class outputs and confidence scores from available sensors, then smooths the result over approximately one second.The GUI supports varying included sensors, required detections, and sensor weights.
- Evaluation limitation: The evaluation against airplanes was severely limited by the COVID-19 pandemic.The pandemic drastically reduced the possibility of a thorough airplane evaluation.
- Detection performance: The fused system output was more stable and lasted for more frames than the IRcam and Vcam outputs individually.This frame-by-frame analysis indicates a benefit from sensor fusion.
- Detection performance: 78% of detection opportunities produced a drone classification after fusion, compared with 67% for the visible-camera detector.The fusion setting used sensor weight 1.0 and required at least two sensors.
- False detections: Requiring at least two sensors prevented all false detections in the reviewed table from becoming system-level false detections.The reviewed ten-minute section included false detections caused by insects and clouds.
VI. DISCUSSION AND CONCLUSIONS
The system combines visual, thermal, acoustic, wide-field, and cooperative-aircraft sensing for drone detection and tracking. Results support infrared detection and a public multi-sensor dataset, while highlighting unresolved evaluation and deployment boundaries.
- System design: The system integrates visible video, audio, thermal infrared, fish-eye, and ADS-B sensors on a pan/tilt platform.The fish-eye camera detects moving objects to steer other cameras, while ADS-B tracks cooperative aircraft.
- Results: Infrared detection achieves an F1-score of 0.7601 versus 0.7849 for visible video, while audio classification reaches 0.9323.The study also evaluates detection performance across sensor-to-target distance using bins derived from Johnson-criteria DRI requirements.
- Dataset: A publicly available multi-sensor dataset addresses the lack of reference data and supports comparison between infrared and visible detectors.The dataset was designed with similar conditions and target types across those modalities.
- Limitations: System-level comparison remains difficult because existing research evaluating complete drone detection systems is sparse.Individual sensor evaluation is more straightforward, but whole-system assessment lacks established comparators.
- Limitations: Most false detections arise from insects or clouds, and broader drone coverage would require hexacopters, octocopters, fixed-wing, and single-rotor examples.The authors also identify YOLO v3 and distance estimation as possible extensions before real-world deployment.
- Applications: The sensing and tracking framework could be adapted to road-traffic surveillance by retraining components for pedestrians or vehicle classes.The proposed transfer excludes the ADS-B receiver.