PaperlayerView exact revision on arXiv

Source-linked AI summary

Computational bioacoustics with deep learning: a review and roadmap

Dan Stowell

arXiv:2112.06725v1cs.SDeess.ASq-bio.QM

TL;DR

Bioacoustic deep learning is expanding, but the field remains immature and faces limited labelled data, domain shift, resource constraints, and representation challenges. This paper reviews the state of the art and proposes a roadmap based on survey and thematic analysis. It identifies adaptations and open topics relevant to using acoustic data for zoological and ecological questions.

  • Problem

    Bioacoustic deep learning lacks sufficient labelled datasets and must address changing data conditions, resource constraints, and representations that may miss biologically relevant acoustic detail.

  • Method

    The paper reviews deep learning for computational bioacoustics through a constrained literature survey, thematic analysis, and synthesis of standard practices and knowledge gaps.

  • Results

    The review synthesises an emerging field in which deep learning is adapted to bioacoustic requirements, with CNNs, embeddings, and compressed representations used across diverse tasks and taxa.

  • Takeaways & Limitations

    The roadmap highlights under-explored topics that can guide future deep-learning research using audio to investigate animal behaviour, populations, and ecosystems.

  • Takeaways & Limitations

    Wider deployment remains constrained by fundamental resource requirements, including rare minerals required for batteries, and batteryless sensing has not yet been combined with deep-learning analytical power.

Abstract

from arXiv · show

Animal vocalisations and natural soundscapes are fascinating objects of study, and contain valuable evidence about animal behaviours, populations and ecosystems. They are studied in bioacoustics and ecoacoustics, with signal processing and analysis an important component. Computational bioacoustics has accelerated in recent decades due to the growth of affordable digital sound recording devices, and to huge progress in informatics such as big data, signal processing and machine learning. Methods are inherited from the wider field of deep learning, including speech and image processing. However, the tasks, demands and data characteristics are often different from those addressed in speech or music analysis. There remain unsolved problems, and tasks for which evidence is surely present in many acoustic signals, but not yet realised. In this paper I perform a review of the state of the art in deep learning for computational bioacoustics, aiming to clarify key concepts and identify and analyse knowledge gaps. Based on this, I offer a subjective but principled roadmap for computational bioacoustics with deep learning: topics that the community should aim to address, in order to make the most of future developments in AI and informatics, and to use audio data in answering zoological and ecological questions.

INTRODUCTION

Computational bioacoustics has expanded through affordable digital recording and advances in informatics, while deep learning brings new opportunities alongside field-specific challenges. This review surveys the emerging area, clarifies concepts, identifies knowledge gaps, and proposes a roadmap.

  • Motivation: Affordable digital recording, storage, and sharing have enabled continuous, large-scale bioacoustic monitoring and created an analyst-time bottleneck.Automation is increasingly important because audio volumes exceed the person-time available to trained analysts.
  • Motivation: Deep learning is increasingly used to solve and automate computational bioacoustic problems previously considered intractable.
  • Field development: Recent bioacoustic deep-learning work adapts methods to the field’s specific analytical requirements, but reference works remain scarce because the area is immature.
  • Review aims: The review surveys the state of the art, clarifies key concepts, and identifies knowledge gaps and under-explored topics for research.
  • Review aims: The paper uses the survey and thematic analysis to develop a principled roadmap drawing on deep learning and broader bioacoustics topics.

SURVEY METHODOLOGY

The review uses constrained keyword searches in Google Scholar and Web of Science to identify deep-learning literature in animal bioacoustics. After screening, merging, and deduplication, the selected corpus contains 159 articles, growing from 5 in 2016 to 63 in 2021.

  • Search strategy: The literature search targeted bioacoustics, ecoacoustics, vocalisations, animal calls, passive acoustic monitoring, and deep-learning terms across animal groups.The query combined domain terms with deep-learning architectures and animal-related keywords.
  • Corpus construction: 987 Google Scholar entries and 55 Web of Science entries were screened, merged, and deduplicated into 159 selected articles.
  • Corpus construction: The selected literature grew from 5 articles in 2016 to 63 in 2021, indicating rapid expansion of the sub-field.
  • Corpus construction: Screening excluded off-topic, duplicate, review, abstract-only, non-English, and unavailable works, while peer-reviewed publications received priority over preprints.

STATE OF THE ART AND RECENT DEVELOPMENTS

Bioacoustic deep learning commonly uses spectrogram inputs, established CNN or CRNN architectures, augmentation, and separated evaluation datasets. Standard practice is useful but remains constrained by representation choices, limited generalisation, and data requirements.

  • Standard recipe: The standard workflow commonly uses established CNN architectures, often with AudioSet pretraining, to process fixed-size spectrogram clips.
  • Standard recipe: Labels may represent species, individuals, call types, sound presence, or multiple events, with modern deep learning scaling to many hundreds of species.
  • Standard recipe: Data augmentation expands small or unbalanced training datasets through methods including noise mixing, time shifting, and mixup.
  • Standard recipe: CRNNs extend CNNs by adding recurrent LSTM or GRU layers after convolutional layers to model the task.
  • Evaluation: Separate training, validation, and testing datasets are recommended, with testing under novel conditions providing a better estimate of generalisability.
  • Representation limits: Mel scales, AudioSet pretraining, magnitude spectrograms, and compact CNN filters may overlook details relevant to fine discrimination, animal perception, or broad-band events.
  • Representation limits: Alternative representations can improve selected difficult tasks, but improvements rarely generalise across many tasks, while raw-waveform networks require larger training datasets.

Taxonomic Coverage

Deep-learning bioacoustics spans many taxa and also operates at ecosystem or soundscape level. Birds and marine mammals receive especially substantial attention, while taxonomic coverage reflects both scientific value and recording practicality.

  • Taxa: Birds are the most commonly studied group, appearing in at least 65 selected papers.
  • Taxa: Cetaceans and other marine mammals form another large subfield, represented by 30 selected papers.
  • Taxa: The literature also includes bats, terrestrial mammals, rodent ultrasonic vocalisations, anurans, insects, and fish.Rodent ultrasonic-vocalisation studies are particularly associated with laboratory mice and rats.
  • Analytical scope: Some studies avoid taxon-specific framing by analysing ecosystems or soundscapes through ecoacoustic approaches.
  • Drivers of coverage: Taxonomic emphasis reflects biodiversity monitoring, conservation, comparative behaviour, vocal complexity, and practical recording considerations.Terrestrial and diurnal species are often easier to record, while standardised datasets and challenges have stimulated bird classification.

Neural Network Architectures

CNNs have replaced manually featured MLPs as the dominant architecture in computational bioacoustics, while recurrent, temporal-convolutional, capsule, attention, and automated architecture-search approaches extend the design space.

  • Neural Network Architectures: CNNs and RNNs supersede MLPs by exploiting time-series or time-frequency structure instead of manually designed summary features.This preserves higher-dimensional acoustic information for nonlinear processing.
  • Neural Network Architectures: 3 of 6 teams in the 2016 BirdCLEF challenge submitted spectrogram-based CNN systems, including the highest-scoring team.An entry reused AlexNet, a CNN originally designed for images.
  • Neural Network Architectures: At least 80 surveyed articles used CNNs, sometimes combined with other modules.Many studies compare architectures and CNN configuration options such as layer count.
  • Neural Network Architectures: Off-the-shelf CNN architectures are increasingly used or evaluated, often through pretrained models available in deep-learning frameworks.Common examples include architectures influential beyond bioacoustics.
  • Neural Network Architectures: CRNNs model temporal dependencies beyond local convolutional receptive fields but can require more computation, with benefits that are not universal.Temporal CNNs offer an alternative through dilated one-dimensional convolutions.
  • Neural Network Architectures: Architecture selection remains difficult because the option space is too large for exhaustive search and offers little a priori guidance.Automated workflow construction has been proposed to select architectures for particular tasks.

Acoustic Features: Spectrograms, Waveforms, and More

Spectrograms remain the prevailing input representation because they retain diverse time-frequency information and align with image-based deep learning, while waveform and adaptive front-ends offer less manual preprocessing at greater data demands.

  • Spectrograms: Magnitude spectrograms are the most common input, representing audio as a two-dimensional time-frequency energy grid.Using the grid directly lets deep-learning systems access diverse information and exploit image-DL advances.
  • Spectrograms: Spectrogram design involves time-frequency resolution, windowing, frequency scale, and normalization choices, with no strong consensus on the best format.Careful parameter selection can provide mild benefits.
  • Alternative Representations: MFCCs are often outperformed by less-preprocessed representations such as mel spectrograms in deep-learning evaluations.MFCC compression may poorly match CNN assumptions about shift invariance along the coefficient axis.
  • Alternative Representations: Wavelets, pitch-tracking traces, and chirplets provide alternative time-frequency representations motivated by target-signal characteristics.Chirplets have been used as a match to whale sound characteristics.
  • Waveforms: Raw-waveform deep learning removes spectrogram preprocessing but often requires larger training datasets than spectrogram-based methods.WaveNet and temporal CNN architectures facilitate waveform input.
  • Adaptive Front-Ends: Trainable filterbank representations occupy an intermediate position between raw waveforms and spectrograms.SincNet achieved competitive birdsong-classification results with short training time.
  • Summary: Spectrograms, often with mel or CQT frequency scales, remain appropriate in many cases, while PCEN is frequently useful and adaptive front-ends may gain prominence.The review anticipates increased use of waveform and adaptive representations within effective off-the-shelf architectures.

Classification, Detection, Clustering

Classification and detection dominate computational bioacoustics, with detection commonly framed as binary occupancy, sound-event transcription, or spectrogram bounding-box localization; CNNs perform strongly across these settings.

  • Task Overview: Classification and detection are by far the most common tasks and fundamental building blocks of many workflows.They are also the tasks most comprehensively addressed by current deep learning.
  • Classification: Classification predicts one or more categorical labels such as species or call type, most commonly for species classification within a taxonomic family.The review distinguishes this meaning from varied uses of the term detection.
  • Detection: Binary detection returns a yes/no decision for an audio clip and provides occupancy information without requiring high-resolution labels.Its simplicity helps explain its widespread use.
  • Detection: Sound event detection transcribes event start and end times, potentially for multiple classes in parallel, by labeling successive time steps.This corresponds to the terminology used in DCASE challenges.
  • Detection: Object-style detection estimates time-frequency bounding boxes for sound events on spectrograms and remains uncommon but may be gaining interest.It adapts the bounding-box concept from image object detection.
  • Detection: CNN-based networks show strong performance across all three detection settings and outperform other machine-learning techniques.Different data formats require different output layers and loss functions.
  • Workflow: A two-step detect-then-classify workflow is common, allowing detection to reject many negative clips when target sounds are sparse.This tunes the detection stage before classification.

Signal Processing using Deep Learning

Deep learning supports signal enhancement, source separation, privacy filtering, compression, and sound synthesis beyond classification and detection, but preprocessing can also remove informative signal content.

  • Enhancement and Separation: Denoising and source separation improve sound quality before analysis, especially in difficult signal-to-noise conditions.Deep-learning approaches commonly map spectrograms to enhanced spectrograms using denoising autoencoders or U-nets.
  • Enhancement and Separation: Preprocessing is not always necessary or desirable because denoising and source separation may remove information from the signal.Deep-learning recognition may work well despite noise.
  • Privacy: Privacy-oriented processing can detect speech in bioacoustic recordings so corresponding clips can be deleted.This strategy has been investigated for bee-hive sound recordings.
  • Compression: MP3-like compression can have surprisingly small effects on analysis, and CNN AudioSet embeddings can serve as compressed fingerprints.A separate approach optimizes a codec to produce a decodable compressed representation of elephant sounds.
  • Synthesis: Animal-sound synthesis has received occasional attention for applications including playback experimental stimuli.Modern deep-learning methods such as WaveNet and GANs have been studied for birdsong synthesis.

Small Data: Data Augmentation, Pre-training, Embeddings

Deep learning bioacoustics must address scarce expert-labelled data, using augmentation, transfer-based methods, embeddings, and related learning strategies to make training feasible. Pretrained-network embeddings are widely useful, while several alternatives can exploit unlabelled or limited data.

  • Rare species or calls and limited expert annotation make large labelled datasets difficult to obtain, especially for fine distinctions and large-scale monitoring.
  • Data augmentation expands training data through modifications such as time shifts, low-amplitude noise, audio mixing, and spectrogram warping.
  • Pretraining transfers representations from a network trained on another related task, with AudioSet and VGG-Sound commonly providing general-purpose audio models.
  • Embeddings represent acoustic input as vector coordinates useful for classification or other tasks, often by removing the final classification head from a pretrained network.
  • Autoencoders can learn embeddings from unlabelled data, although ensuring semantic content is encoded may be difficult.
  • Multi-task, semi-supervised, weakly-supervised, self-supervised, and few-shot learning provide additional ways to train despite limited labelled data.Few-shot learning targets unseen tasks using one or very few examples; the other approaches alter how labelled and unlabelled information is used.

Generalisation and Domain Shift

Generalisation is threatened by domain shift when deployment conditions differ from training data. Evaluation on changed conditions and methods such as adaptation, contextual inputs, transfer learning, and augmentation address this concern.

  • Domain shift occurs when background soundscape, sub-population, event frequency, or microphone type changes the input distribution and affects inference outputs.
  • Testing on locations, signal-to-noise ratios, or seasons that differ from training helps avoid overestimating practical generalisation.
  • Domain adaptation can automatically adjust network parameters, while contextual correlates can be supplied as inputs for automatic adaptation.
  • In a cross-condition bird audio challenge, leading submissions relied on pretrained general-purpose representations and data augmentation rather than explicit domain adaptation.

Open-set and Novelty

Open-set recognition addresses novel species, individuals, or sound types that fixed-label systems cannot represent. Embeddings, hierarchical classification, novelty detection, and contextual information offer routes toward handling unfamiliar inputs, but novelty remains an ongoing concern.

  • Fixed-label recognition cannot identify species or individuals absent from the training label set, a frequent issue in wild recordings and individual identification.
  • Open-set recognition detects sound types beyond known target classes, while novelty detection more broadly seeks novel occurrences in data.
  • Hierarchical classification can assign a novel sound to a higher-level taxon even when its lower-level class is unseen.
  • Semantic embeddings may allow novel classes to cluster usefully, supporting methods such as nearest-neighbour classification.
  • Novelty and open-set issues are likely to remain practical concerns, although general-purpose embeddings offer part of a solution.
  • Short audio or spectrogram segments can omit contextual information from prior soundscape activity or other external correlates.

On-device Deep Learning

On-device deep learning can enable live monitoring while reducing power, bandwidth, and storage demands, but constrained hardware limits model speed and capability. Deployment is also moving toward integration with broader monitoring workflows and usable interfaces.

  • On-device processing enables live readouts and rapid responses, with potential savings in power and data-transmission costs.
  • Raspberry Pi, Jetson Nano, and Google Coral have been evaluated for running acoustic detection or classification on-device.
  • AudioMoth-class devices reduce power consumption, ecological footprint, and form factor but cannot run many deep-learning algorithms.
  • A depthwise-separable CNN on AudioMoth outperformed an HMM detector but was not efficient enough for real-time operation.
  • Small-footprint implementations reduce power, bandwidth, and storage demands, while batteryless sensing is proposed to address battery-resource constraints.
  • As systems mature, research increasingly addresses integration into biodiversity-monitoring workflows and graphical interfaces for configuring and exploring outputs.

A ROADMAP FOR BIOACOUSTIC DEEP LEARNING

The roadmap emphasizes open, collaborative, and human-integrated systems, while identifying data, representation, architecture, workflow, and inference challenges for future bioacoustic deep learning.

  • Principles: AI should augment rather than replace expertise, with future systems integrating deep-learning agents, experts, and crowdsourcing.Active learning allows agents to learn from human criticism of their decisions.
  • Principles: Open sharing of datasets, architectures, pretrained weights, source code, audio, and standardized metadata can move analysis beyond single-dataset limitations.The paper identifies incomplete data sharing as a continuing issue in bioacoustics.
  • Representations: Spectrograms remain dominant inputs, while future systems are likely to use off-the-shelf or bioacoustics-specific embeddings, especially where datasets such as AudioSet cannot cover needs such as ultrasound.The choice between waveform and spectrogram may become irrelevant to users of learned features.
  • Architectures: CNNs currently dominate, but temporal convolutions, transformers, perceivers, and memory-based attention architectures may reduce their singular dominance.TCNs are described as simple and relatively efficient, while attention networks offer flexible reference to previous time steps.
  • Data and generalisation: Small-data problems will persist for project-specific and high-resolution tasks, motivating integrated evaluation of pretraining, embeddings, multitask learning, and augmentation.Transfer learning may be inappropriate when it risks bias, while data challenges identify several low-risk routes to improved generalisation.
  • Future directions: Important roadmap directions include efficient active learning, general-purpose individual recognition, richer workflow interfaces, and calibration of outputs for ecological inference.The review also places geographic-scale population distribution outside its scope and highlights taxonomic and dataset imbalance, including under-studied insects and rare species.

CONCLUSIONS

Deep learning has substantially improved automatic computational bioacoustic systems, but progress depends on addressing bioacoustics-specific data characteristics and scientific questions rather than simply importing techniques from neighbouring fields.

  • Deep learning has enabled a leap in the performance of automatic computational bioacoustic systems.
  • Future progress will draw on developments adapted from image recognition, speech, and general audio.
  • Bioacoustic deep learning requires topics of study that arise from the field’s specific data characteristics and questions.
Loading 2112.06725v1…