Source-linked AI summary
Empirical Study of Drone Sound Detection in Real-Life Environment with Deep Neural Networks
Sungho Jeon, Jong-Woo Shin, Young-Jun Lee, Woong-Hee Kim, YoungHyoun Kwon, Hae-Yong Yang
TL;DR
The paper studies how to detect commercial hobby drones from sound in real-life environments, where diverse background noise and limited drone recordings complicate detection. It augments drone recordings with environmental sounds and evaluates GMM, CNN, and RNN classifiers. RNN achieves the best reported F-Score, 0.8009, using 240 ms of audio input, while processing-time and data requirements constrain practical deployment.
Problem
Drone sound detection requires evidence from noisy real-life environments, but available drone recordings and prior studies are limited in environmental diversity.
Method
The study augments recorded commercial drone sounds with diverse background sounds and evaluates GMM, CNN, and RNN binary classifiers.
Results
RNN achieves the best reported F-Score of 0.8009 with 240 ms of audio input in the evaluated detection experiments.
Takeaways & Limitations
The results support using augmented training data and RNN-based detection for real-time drone sound detection within the evaluated real-life setting.
Takeaways & Limitations
Reported processing time is only minimal partial execution time, and neural-network models require more input data than GMM for optimal performance.
Abstract
from arXiv · showhide
This work aims to investigate the use of deep neural network to detect commercial hobby drones in real-life environments by analyzing their sound data. The purpose of work is to contribute to a system for detecting drones used for malicious purposes, such as for terrorism. Specifically, we present a method capable of detecting the presence of commercial hobby drones as a binary classification problem based on sound event detection. We recorded the sound produced by a few popular commercial hobby drones, and then augmented this data with diverse environmental sound data to remedy the scarcity of drone sound data in diverse environments. We investigated the effectiveness of state-of-the-art event sound classification methods, i.e., a Gaussian Mixture Model (GMM), Convolutional Neural Network (CNN), and Recurrent Neural Network (RNN), for drone sound detection. Our empirical results, which were obtained with a testing dataset collected on an urban street, confirmed the effectiveness of these models for operating in a real environment. In summary, our RNN models showed the best detection performance with an F-Score of 0.8009 with 240 ms of input audio with a short processing time, indicating their applicability to real-time detection systems.
I. INTRODUCTION
The study addresses drone sound detection in noisy real-life environments, where prior work largely used isolated or calm settings. It investigates GMM, CNN, and RNN methods, using audio augmentation to support practical real-time detection.
- Real-life drone detection is motivated by threats from commercial hobby drones to people and important facilities.
- Prior drone sound studies mainly used isolated or calm environments rather than polyphonic real-life settings.
- The study investigates GMM, CNN, and RNN models for drone sound detection in highly noisy real environments.
- Audio augmentation synthesizes raw drone sound with diverse background sounds to address limited drone classification training data.
- The evaluation uses real-life testing data and considers F-Score and processing time for real-time systems.
A. Data Augmentation
The study augments limited drone recordings with diverse environmental sounds to approximate realistic acoustic conditions. It also examines feature inputs and finds that GMM and neural-network models require different audio-window lengths.
- Data Augmentation: Drone recordings are augmented with diverse real-life environmental sounds because collecting varied public drone data is limited.The augmented clips combine raw and overlapped background sounds with repeated drone sounds.
- Data Augmentation: Drone sound is amplified to exceed background sound by 5% in maximum peak, preserving its characteristics during augmentation.
- Feature Engineering: MFCC captures periodicity from drone rotor frequencies, with observed harmonic regions below 1500 Hz and between 5000 Hz and 7000 Hz.These characteristics were not exhibited by all drone models.
- Feature Engineering: 40 ms with 50% overlap gives the best GMM performance, whereas CNN and RNN perform best with at least 240 ms mel-spectrogram inputs.
C. Classifier1: Gaussian Mixture Model
The GMM detector uses separate positive and negative models and compares their log-likelihoods against a threshold to classify drone presence. The selected configuration uses 13 Gaussians, 20 MFCCs, and 40 mel-bins.
- The GMM detector trains separate models on positive and negative data, then compares their log-likelihoods to a decision threshold.The decision is Labelpredicted = L1 − L2 > θdecision.
- The best GMM configuration uses 13 Gaussian components, 20 MFCCs, and 40 mel-bins.
- Higher parameter values caused overfitting, improving training performance but producing unsatisfactory testing results.
D. Classifier2: Convolutional Neural Network
The CNN approach uses spectral features and localized convolution to capture time-frequency patterns in drone sounds. The study favors a simple architecture because greater complexity can promote overfitting, while RNNs use accumulated temporal context.
- Convolutional Neural Network: CNNs use localized convolution over spectral representations to capture invariant time-frequency patterns in sound data.
- Convolutional Neural Network: The proposed CNN has nine stages because more complex architectures can lead to overfitting rather than improved performance.
- Recurrent Neural Network: RNNs repeatedly process information with memory that represents accumulated context and helps prevent vanishing-gradient effects.
- Detection Visualization: The spectrogram examples pair negative and positive data with detection labels, where black regions indicate predicted drone periods.
III. EXPERIMENT
The experiment compares GMM, CNN, and RNN drone-detection performance, evaluates unseen data and real-time computing cost, and averages reported values across 10 evaluation results.
- The study compares GMM, CNN, and RNN for drone detection, including performance on unseen data and computing cost for real-time systems.Reported performance values are averaged across 10 evaluation results.
A. Data description
The training data combines manually recorded drone sounds with diverse real-life background sounds, while separate real-world and unseen-type test sets measure performance beyond augmentation.
- Data collection: The training set combines Phantom3 and Phantom4 drone recordings with background sounds from ordinary real-life situations and public data.Drone sounds covered hovering and approaching at distances of 30m, 70m, and 150m.
- Data augmentation: 64 seconds of drone recordings and 677 seconds of background audio were expanded through augmentation to 9556 seconds of training audio.
- Evaluation datasets: The datasets were separated rather than evaluated with k-fold cross validation to measure performance strictly.The authors note that augmentation cannot completely reproduce the complexity of real datasets and environmental influence.
- Evaluation datasets: The main test set was collected on an urban street for 151 seconds, with equal positive and negative data across normal and construction-site conditions.
- Evaluation datasets: An additional test set measured detection on unseen drone and background types not included in training.
B. Testing: detection performance
The models were evaluated with multiple detection measures, and RNN achieved the strongest training-set F-Score while CNN architecture remained difficult to optimize in real environments.
- Detection performance: 0.8009 F-Score was achieved by RNN, exceeding CNN at 0.6415 and GMM at 0.5232 on the training datasets.RNN also had precision and recall of 0.7953 and 0.8066, respectively.
- Detection performance: RNN provided the most balanced precision and recall among the evaluated models.Its precision was 0.7953 and recall was 0.8066.
- Detection performance: CNN and GMM showed contrasting precision-recall tendencies, with CNN favoring recall and GMM favoring precision.CNN had precision 0.5346 and recall 0.8019, while GMM had precision 0.9031 and recall 0.3683.
- CNN limitation: The authors could not identify a CNN architecture from previously proposed models that performed reliably for this real-environment audio.Drone sound affected only a small spectrogram area, leaving CNN vulnerable to other environmental-sound regions.
C. Testing: unseen types of data
On unseen drone or background types, RNN retained the best reported performance, while CNN failed to classify the data reliably and GMM's performance declined substantially.
- Unseen-data performance: 0.6984 F-Score was achieved by RNN on unseen types of data, with precision 0.5477 and recall 0.9635.
- Unseen-data performance: CNN failed to classify the unseen data and treated all data as positive.The authors associate this behavior with unseen highly noisy background sound that CNN could not distinguish from drone sound.
- Unseen-data performance: GMM achieved more accurate detection than CNN on unseen data but had a 0.3910 F-Score, which the authors judged unsuitable for practical operation.
- Model tendencies: The unseen-data experiment showed that GMM tended to predict negative data, whereas the deep-neural-network models tended to predict positive data.
- Implications: The experiment confirmed that collecting diverse data for the target environment is essential.Additional training data improved RNN precision but did not significantly improve GMM.
D. Required cost versus Detection performance
Practical deployment depends on both detection performance and computational cost. The study reports that all three models appear suitable for real-time operation, while input duration and platform affect actual latency.
- All three proposed models appear appropriate for application to real-time systems.The reported processing measurements exclude Python-program overhead and vary with the target platform.
- 145 ms is required to create the MFCC vector for a 1-minute audio clip, making feature engineering the most time-consuming stage.Data-loading time varies by platform, while classification time does not adversely affect real-time operation.
- 240 ms > 40 ms: deep neural network models require more input data than GMM for optimal performance.This larger input amount can increase initial detection time, especially on low-performance embedded platforms or when importing the program.
IV. CONCLUSION
The paper presents an audio-based binary classifier for drone detection and evaluates its performance and real-time operating cost. RNN achieves the best reported F-Score, while detection distance remains constrained by the recording setup.
- 0.8009 is the best F-Score, achieved by the RNN model with 240 ms of audio input data.The study also reports that augmentation with diverse background sounds alleviates the shortage of drone training data.
- The experiment confirmed that audio augmentation with diverse background sounds can alleviate the shortage of drone training data.
- 150 m was the maximum evaluated drone distance because practical constraints prevented testing farther distances.With a single microphone, recordings beyond 150 m did not show noticeable spectrogram characteristics because background data covered the weakened drone sound.