Source-linked AI summary
Audio Surveillance: a Systematic Review
Marco Crocco, Marco Cristani, Andrea Trucco, Vittorio Murino
TL;DR
Video-based surveillance often lacks the robustness required in real applications, motivating research on audio sensors used alone or with video. This paper systematically reviews audio surveillance through a four-task taxonomy, examines audio features and representative methods, and discusses their application-specific advantages and limitations. It provides an application-oriented synthesis spanning background subtraction, event classification, object tracking, situation analysis, and multimodal surveillance.
Problem
Video-based automated surveillance often lacks the robustness and reliability required in several real applications, while audio-feature evaluation is hindered by scarce benchmark datasets and fragmented literature.
Method
The paper systematically reviews audio-based surveillance methods using a taxonomy covering background subtraction, event classification, object tracking, and situation analysis, including audio-only and multimodal approaches.
Results
The review compares methods across surveillance tasks and discusses their advantages, disadvantages, and application contexts, including representative approaches for segmentation, normal-event modeling, localization, and beamforming.
Takeaways & Limitations
The survey provides an application-oriented framework for relating audio-surveillance algorithms to task requirements and surveillance scenarios.
Takeaways & Limitations
Audio surveillance remains constrained by variable, reverberant, multi-source backgrounds and lower SNR, while single-microphone localization cannot infer source position without additional assumptions or spatialization.
Abstract
from arXiv · showhide
Despite surveillance systems are becoming increasingly ubiquitous in our living environment, automated surveillance, currently based on video sensory modality and machine intelligence, lacks most of the time the robustness and reliability required in several real applications. To tackle this issue, audio sensory devices have been taken into account, both alone or in combination with video, giving birth, in the last decade, to a considerable amount of research. In this paper audio-based automated surveillance methods are organized into a comprehensive survey: a general taxonomy, inspired by the more widespread video surveillance field, is proposed in order to systematically describe the methods covering background subtraction, event classification, object tracking and situation analysis. For each of these tasks, all the significant works are reviewed, detailing their pros and cons and the context for which they have been proposed. Moreover, a specific section is devoted to audio features, discussing their expressiveness and their employment in the above described tasks. Differently, from other surveys on audio processing and analysis, the present one is specifically targeted to automated surveillance, highlighting the target applications of each described methods and providing the reader tables and schemes useful to retrieve the most suited algorithms for a specific requirement.
1 Introduction
The paper frames audio as a complementary surveillance modality addressing weaknesses of video, then organizes audio surveillance research across four surveillance tasks and multimodal systems.
- Video surveillance can fail under adverse weather, sudden illumination changes, reflections, shadows, and nighttime conditions.
- Audio offers broader sensing advantages, including operation despite obstacles, immunity to illumination and temperature issues, and detection of events with little or no video counterpart.
- Audio monitoring is generally perceived as less invasive than video, although audio surveillance usually excludes automatic speech detection and recognition.
- The review organizes audio surveillance by background subtraction, event classification, object tracking, and situation analysis, paralleling the established video-surveillance workflow.
- Audio background subtraction is more challenging than its video counterpart because audio backgrounds vary over time, combine multiple sources, undergo reverberation, and often have lower SNR.
- Audio features affect all four tasks, but no generally best feature set is established because benchmark datasets are scarce and the literature is fragmented.
- The paper reviews audio-only and multimodal surveillance methods because audio information is complementary to other modalities and may support more accurate and reliable performance.
2 Background subtraction
Audio background subtraction separates recurring background features from foreground events using monomodal energy-based or multimodal feature models. The reviewed methods range from thresholding and adaptive segmentation to learned statistical models designed for complex, variable environments.
- Background subtraction identifies recurring audio features as background and departures from them as foreground events.
- Monomodal analysis: Monomodal approaches use energy thresholding, assuming background features follow a single distribution or recurring pattern.They segment fixed-length windows and discard windows whose energy falls below a predefined threshold.
- Monomodal analysis: Thresholds may be fixed from background statistics, tuned for deterministic variation, or adapted to unpredictable changes over time.Adaptive thresholding can use median-filtered energy and the standard deviation of a long-term windowed sequence.
- Monomodal analysis: Time-frequency thresholding detects foreground when energy exceeds a threshold in at least one scale-frequency bin, while autoregressive segmentation avoids chopping events at fixed boundaries.The autoregressive method detects boundaries from prediction errors before adaptive foreground classification.
- Monomodal analysis: Segmentation comparisons found energy-based segmentation least performing, whereas model-based and metric-based methods use learned classes or inter-segment distances.A hybrid method clusters short segments, trains a statistical model per cluster, and then segments the stream with those models.
- Limitations: Energy thresholding has limited performance when high-energy sounds recur as background, so complex environments require additional acoustic features.The survey notes car engines in a parking area as an example of recurring high-energy background.
- Multimodal analysis: Multimodal background models address complex environments by representing repeated background sounds with multiple distributions rather than one.GMMs model feature densities as mixtures of Gaussian components, while related approaches use HMMs or GMM clustering.
- Multimodal analysis: Semi-supervised modeling combines offline background and foreground models with a separate background-change model to detect slowly varying foreground.GMM clustering showed better performance than standard GMM and HMM when many different normal sounds occurred in complex environments.
3 Audio events classification
Audio event classification learns classifiers from labeled features and applies them to unseen samples, using generative, discriminative, hierarchical, and hybrid strategies. The review finds that environmental audio remains difficult to classify reliably because surveillance conditions differ substantially from established speech and instrument-recognition settings.
- Classification framework: Audio classification extracts features from labeled signals, trains a classifier offline, and applies it to recognize unseen samples.The taxonomy separates generative models, which assign class-specific posterior probabilities, from discriminative models, which learn decision boundaries.
- Generative models: Generative classification commonly uses GMMs and HMMs, with HMM variants modeling temporal signatures of transient or stationary sounds.Left-right HMMs capture transient events such as gunshots or screams, whereas ergodic HMMs model stationary sounds.
- Discriminative and hybrid models: Discriminative classification commonly uses ANNs and SVMs to separate classes directly in feature space.The review also describes hybrid GMM-SVM strategies that combine normalized model scores using error-dependent weights.
- Open challenges: No common audio-event classification framework has emerged, and methods developed for speech or musical instruments generally transfer poorly to environmental surveillance sounds.Differences in duration, spectral content, volume, microphone distance, echoes, and reverberation constrain straightforward extension of non-surveillance findings.
- Applications and evaluation: Reported applications range from alarm and gunshot detection to multiclass environmental sound recognition, but small evaluations limit broad conclusions.One comparison achieved 70% correctly classified samples with Dynamic Time Warping, while the authors cautioned that the training and test sets were small.
- Hierarchical classification: Hierarchical classifiers can improve performance as class counts increase by progressively separating broad sound groups before finer classes.Reported applications include five sound classes and shout detection, where staged exclusion of background and non-target sounds improved precision.
4 Source localization and tracking
Audio source localization requires multiple sensors because a single microphone cannot infer source position in general scenes. The review organizes localization methods by propagation cues and learning, then covers audio-visual localization and tracking approaches with their practical trade-offs.
- Localization foundations: A single microphone generally cannot recover sound-source location, so localization relies on microphone arrays or networks, sometimes with heterogeneous sensors.A single sensor provides a one-dimensional signal without sufficient spatial-scene information.
- Localization taxonomy: Source-localization methods are grouped into time-delay, energy-ratio, and learning-based families, with time-delay methods further divided into beamforming, high-resolution spectral estimation, and TDOA.The taxonomy links each family to a different propagation assumption or learned relation between features and location.
- Time-delay methods: Steered beamforming scans candidate directions using delay compensation and summed signal energy, but has relatively poor spatial resolution and high computational cost.The cost becomes especially significant for large arrays and fine direction grids.
- Time-delay methods: High-resolution spectral methods provide higher spatial resolution than steered beamforming but are sensitive to reverberation, source-count limits, and the need for statistically stationary windows.These constraints limit their suitability for surveillance settings.
- Time-delay methods: TDOA methods estimate pairwise arrival-time differences before geometrically inferring positions, supporting microphone networks despite two-step information loss and possible local minima.They can tolerate microphones occluded by architectural barriers and fit networks without a predefined dense geometry.
- Energy-ratio methods: Energy-ratio methods reduce wireless data-transfer demands and tolerate synchronization mismatch, although their precision is generally inferior to propagation-delay methods.For narrowband sounds, they can consistently improve TDOA performance.
- Learning-based methods: Learning-based localization trains classifiers on microphone features but is less common because reliable location-labeled databases are difficult to acquire in uncontrolled environments.The training phase must be repeated when the operating conditions change.
- Audio-visual localization: Audio-visual localization exploits temporal or statistical correspondence between sound and image regions, but fails when multiple matches are ambiguous or no visual counterpart exists.The reviewed approaches include mutual information, component analysis, sparse coding, and temporal onset matching.
5 Situation analysis
Situation analysis interprets complex semantic situations by integrating temporal and spatial information from multiple heterogeneous sensors. The surveyed approaches include direct scene classification, hierarchical fusion, and hybrid systems, with surveillance applications often distinguishing normal from environment-specific relevant situations.
- Scope: Situation analysis extracts complex semantic concepts by integrating temporal and spatial data from multiple heterogeneous sensors.It represents a higher abstraction level than sound classification or foreground extraction.
- Approaches: Direct approaches define finite environmental states and classify general scene descriptions from audio or audio-video features.This strategy treats situation analysis as a sound classification problem and can use machine learning algorithms.
- Approaches: Hierarchical approaches detect individual sound sources or video objects before fusing them with explicit rules or machine learning.The fusion stage incorporates relationships among the detected elements.
- Approaches: HMM-based systems model basic audio events and group likely simultaneous events using predefined transition probabilities to represent structured scenes.Each basic event may belong to multiple groups, with each group identifying a different structured audio scene.
- Applications: Surveillance applications commonly distinguish normal activity from a specific relevant situation, including aggression, intrusion, or fights, using audio-video fusion and probabilistic models.Examples include aggression estimation from speech and body-articulation cues, and fight detection using audio, visual, thermal, and microphone-array data.
6 Audio Features
The survey organizes audio features into six classes according to signal representation and describes their roles across surveillance tasks. It also records feature use by task while noting limitations such as low-dimensional spectral representations and MFCC sensitivity to narrowband noise.
- Feature taxonomy: Audio features compress high-dimensional signals into compact values that preserve information useful for detection or classification while discarding noise and redundancy.The survey divides them into temporal, spectral, time-frequency, cepstrum-based, energy-based, and biologically or perceptually driven classes.
- Feature taxonomy: Temporal, spectral, and time-frequency features are classified by the intermediate signal representation from which they are extracted.Temporal features use signal samples or autocorrelation, spectral features use the spectrum, and time-frequency features use a joint representation.
- Feature roles: Localization and tracking rely mainly on interaural time and phase differences, whereas spectral and temporal features describe signal structure for classification.ITD measures arrival-time differences, while IPD measures phase differences between microphone signals.
- Limitations: Most spectrum-shape features perform poorly alone because of intrinsic low dimensionality and are therefore combined with higher-size features such as MFCC or DWT.The survey reports this limitation for the spectral feature group rather than for a single feature.
- Feature roles: MFCCs are commonly combined with simpler spectral and temporal features, but they have low robustness to narrowband noise because the transform spreads noise across coefficients.Typical MFCC representations use 3 to 15 selected coefficients and have moderate computational complexity.
7 Conclusions
The paper presents an application-oriented survey of automated surveillance methods that use audio alone or with other sensors. Its central contribution is a global taxonomy spanning surveillance tasks, supported by pros-and-cons analyses and retrieval-oriented tables and diagrams.
- Scope: The survey covers automated surveillance methods based on or including audio devices.It addresses audio as a surveillance modality across the reviewed literature.
- Contribution: Its global taxonomy spans typical surveillance tasks from low-level background subtraction to semantic analysis of a whole scene.The conclusion identifies this cross-task organization as a distinguishing feature for audio surveillance reviews.
- Contribution: The application-oriented review discusses method pros and cons for surveillance needs and provides tables and diagrams to help select methods for specific tasks or operating conditions.The paper also includes general works that may affect future developments of the field.