Global birdsong embeddings enable superior transfer learning for bioacoustic classification

Burooj Ghani; Tom Denton; Stefan Kahl; Holger Klinck

Source-linked AI summary

Global birdsong embeddings enable superior transfer learning for bioacoustic classification

Burooj Ghani, Tom Denton, Stefan Kahl, Holger Klinck

arXiv:2307.06292v2Jul 12, 2023eess.AScs.SD

TL;DR

Limited labeled data restricts classification of rare species, non-bird taxa, and fine-grained call types. The study evaluates fixed embeddings from large-scale acoustic models for transfer across diverse bioacoustic tasks and finds that global bird embeddings consistently outperform general-audio embeddings, including in few-shot settings.

Problem
Rare species, non-bird taxa, and call types often lack enough labeled data to train robust classifiers from scratch.
Method
The study evaluates fixed feature embeddings from large-scale bird and general-audio classifiers with simple downstream probes across diverse bioacoustic datasets.
Results
Global bird embeddings consistently produced higher-quality classification than embeddings from general-audio models across the evaluated bioacoustic tasks.
Takeaways & Limitations
Global bird acoustic embeddings can support few-shot transfer learning for novel bioacoustic classes with limited training data.
Takeaways & Limitations
Example-level train/test splits can overestimate generalization when multiple examples come from the same recording, affecting Bats, RFCX, and some Watkins data.

Abstract

from arXiv · show

Automated bioacoustic analysis aids understanding and protection of both marine and terrestrial animals and their habitats across extensive spatiotemporal scales, and typically involves analyzing vast collections of acoustic data. With the advent of deep learning models, classification of important signals from these datasets has markedly improved. These models power critical data analyses for research and decision-making in biodiversity monitoring, animal behaviour studies, and natural resource management. However, deep learning models are often data-hungry and require a significant amount of labeled training data to perform well. While sufficient training data is available for certain taxonomic groups (e.g., common bird species), many classes (such as rare and endangered species, many non-bird taxa, and call-type) lack enough data to train a robust model from scratch. This study investigates the utility of feature embeddings extracted from audio classification models to identify bioacoustic classes other than the ones these models were originally trained on. We evaluate models on diverse datasets, including different bird calls and dialect types, bat calls, marine mammals calls, and amphibians calls. The embeddings extracted from the models trained on bird vocalization data consistently allowed higher quality classification than the embeddings trained on general audio datasets. The results of this study indicate that high-quality feature embeddings from large-scale acoustic bird classifiers can be harnessed for few-shot transfer learning, enabling the learning of new classes from a limited quantity of training data. Our findings reveal the potential for efficient analyses of novel bioacoustic tasks, even in scenarios where available training data is limited to a few samples.

1 Introduction

The study addresses limited labeled data for rare species, non-bird taxa, and fine-grained call types by testing pre-trained acoustic embeddings for transfer across bioacoustic tasks. It compares global bird models with general-audio models and evaluates their broader classification utility.

Motivation: Rare and endangered species, non-bird taxa, and call types often lack sufficient labeled data for robust classification models.Fine-grained vocalization classes can matter for understanding behaviors such as foraging or breeding.
Motivation: Feature embeddings are intermediate-layer vectors that support few-shot transfer learning to novel tasks and domains.This approach reuses learned representations rather than training every classifier from scratch.
Motivation: Embeddings can capture subtle acoustic differences such as song dialect variation and transfer knowledge from data-rich species to data-poor target species.The proposed use is especially relevant when target classes have insufficient training data.
Study aim: The study compares embeddings from AudioSet-trained general-audio models with embeddings from large-scale global bird-vocalization models across bird, bat, marine-mammal, and amphibian tasks.The evaluation covers both fine-grained bird classification and related non-bird acoustic events.
Practical motivation: Fixed pre-trained embeddings can make large-scale passive-acoustic workflows more efficient by allowing datasets to be embedded once and reused for downstream analyses.The paper motivates this workflow for deployments that may accumulate tens to hundreds of terabytes in one field season.
Study aim: Prior transfer-learning studies often focused on single use cases or older general-audio embeddings, leaving breadth of generalization insufficiently assessed.This study provides a broader comparison of global bird embeddings against general-audio models.

2 Methods

The method extracts fixed embeddings from pre-trained acoustic models and trains simple linear classifiers for diverse downstream bioacoustic tasks. Experiments vary data availability, model inputs, network design, embedding size, and custom-trained baselines.

Core pipeline: The pipeline converts spectrograms into embeddings with pre-trained models, then applies logistic regression or a linear classifier head for downstream classification.Keeping embeddings fixed lets the linear probe assess task-specific information already encoded by each model.
Core pipeline: Each model’s embeddings are computed for the full dataset after adapting audio to the model’s native sample rate and window size.Short inputs receive centered zero-padding, while alternative audio handling is examined experimentally.
Evaluation: Performance is evaluated using macro-averaged ROC-AUC and Top-1 Accuracy averaged across five training runs.Macro-averaged ROC-AUC averages class-specific ROC-AUC values across classes.
Few-shot experiments: Few-shot robustness is tested by varying k as powers of two from 4 to either 32 or 256, depending on dataset size.Results are reported across tested values and specifically for k = 32 in Table 3.
Additional experiments: Additional experiments test t-SNE projections, AudioMAE pre-trained versus fine-tuned embeddings, native-rate audio handling, and a two-layer classification network.These experiments probe embedding geometry and whether alternative AudioMAE configurations improve classification.
Additional experiments: Embedding-size ablations compare BirdNET versions of 320 and 1024 dimensions with Perch versions ranging from 160 to 2560 dimensions.General-audio models are excluded because their training infrastructure was unavailable.
Baselines: A custom-trained baseline uses MFCC features and a two-layer neural network with 2048 hidden features on each target dataset.This baseline mirrors one considered in the BEANS benchmark.
Baselines: The study also compares transfer performance with Sierra Birds, a regional EfficientNet B0 model trained on 89 Sierra Nevada bird species.Its training data primarily comes from XC recordings and includes additional non-bird noise data.

3 Models and Datasets

The study compares bird-trained and general-audio embedding models, then evaluates them across diverse bioacoustic datasets spanning birds, bats, marine mammals, and frogs. The datasets and splits vary in recording characteristics, and some limitations may affect estimated generalization or model quality.

Embedding models: Three bird-trained models—BirdNET, Perch, and Sierras—are compared with four AudioSet-trained general-audio models.The AudioSet models are AudioMAE, PSLA, YAMNet, and VGGish.
Bird-trained models: Perch is trained exclusively on bird sounds, whereas BirdNET combines global bird recordings with labeled soundscape data and some non-bird sounds.Both models use EfficientNet architectures, but their training data and target outputs differ.
Bird-trained models: Sierra Birds is a regional EfficientNet B0 model trained on 89 bird species from California’s Sierra Nevada mountains.Its training data primarily comes from Xeno-canto and includes additional non-bird noise augmentation.
General-audio models: AudioMAE, PSLA, YAMNet, and VGGish provide general-audio comparisons using transformer or convolutional architectures trained on AudioSet.AudioMAE uses encoder embeddings from masked-spectrogram reconstruction, while PSLA summarizes final embeddings after replacing its attention layer.
Evaluation datasets: The evaluation covers bird calls and dialects, four bat species, marine mammals, and Puerto Rican birds and frogs, using datasets with differing recording conditions.The datasets include Godwit Calls, Yellowhammer Dialects, Bats, Watkins Marine Mammal Sounds, and RFCX recordings.
Dataset limitations: Train/test splits may overestimate generalization for Bats, RFCX, and some Watkins species because examples from the same source recording can be separated across splits.Additional unlabeled vocalizations in some Watkins and RFCX frog recordings may instead underestimate model quality.

4 Results

Across few-shot bioacoustic tasks, global bird embeddings consistently outperformed general-audio and regional alternatives, with performance remaining above chance from very limited training data. Embedding geometry and dimension influenced downstream separability and classification quality.

Linear Probes on Feature Embeddings: Perch and BirdNET 2.3 outperformed AudioSet models across all tasks, with Perch achieving the most consistent Top-1 accuracy and AUC.Perch reached AUCs of 0.99 for Godwit Calls and 0.97 for Bat Species.
Few-shot learning: Global bird models consistently outperformed general event-detection models across training-data sizes, with ROC-AUC significantly above 0.5 using only 4 examples per class.Figure 2 reports ROC-AUC across experiments ranging from 4 to 32 or 256 examples per class.
Visualizing Embedding Spaces: Perch embeddings formed clean Godwit clusters with large linear-separability margins, whereas YAMNet lacked clean margins and AudioMAE produced noisier margins.For Yellowhammer, classes remained intermixed in YAMNet, while Perch and BirdNET showed only partial clustering associated with slower improvement.
AudioMAE comparisons: Global bird embeddings uniformly outperformed AudioMAE despite substantial AudioMAE tuning, while PSLA remained below both AudioMAE and global bird classifiers.AudioMAE performed best with a 2-layer perceptron and no resampling; PSLA benefited from a two-layer probe but still underperformed.
Embedding size ablation: Embedding dimension affected performance: reducing Perch to 320 dimensions degraded quality across tasks, while expanding it to 2560 improved some downstream tasks.Increasing BirdNET to 1024 dimensions produced performance similar to Perch on most downstream tasks.
Baselines and regional models: The MFCC baseline barely exceeded chance, and global bird models significantly outperformed the regional Sierra Birds model, which underperformed general audio embeddings on some datasets.These comparisons indicate that broad bird-vocalization training produced stronger transfer than the regional alternative in the evaluated tasks.

5 Discussion

Global bird models produced more generalizable bioacoustic embeddings than broad audio models, including for novel taxa and low-data tasks. Discussion also identifies data diversity, embedding size, and subtle class distinctions as important factors shaping performance.

Model generalization: Global bird models such as BirdNET and Perch consistently outperformed general event-detection models on previously unencountered bioacoustic patterns.The comparison included AudioMAE, PSLA, YAMNet, and VGGish.
Model generalization: Bird vocalization diversity may promote generalized representations through broad temporal, spectral, species, and acoustic variation.The authors connect this diversity to possible transfer across taxa, while presenting the mechanism as a hypothesis for future research.
Training-data diversity: Global bird models trained across many classes outperformed the regional Sierra Birds model, supporting class diversity as a contributor to generalization.The regional model was also weaker than general audio embeddings on some datasets.
Architecture versus data: AudioMAE was outperformed by EfficientNet-B1, and global bird embeddings still performed better without architectural or preprocessing adjustments.The authors conclude that training-data effects dominated architecture effects in these comparisons.
Low-data performance: As few as 4 training samples still yielded promising recognition results, although linear probes showed calibration difficulties and substantial marine-mammal confusions.The Watkins dataset included 18.6% confusion between bearded seals and bowhead whales, whose calls often co-occur in recordings.
Task difficulty: Low-data Yellowhammer dialect classification showed high variance because the classes differ by a subtle note-order cue, encouraging over-generalization from few examples.Other timbral and pitch-sweep variations do not distinguish the dialects.
Embedding design: Increasing embedding size improved bioacoustic separability but increased model size and reduced inference speed.The study notes that newer BirdNET versions adopted larger embeddings following these observations.

6 Conclusions

The study evaluates whether large-scale audio embeddings can generalize to previously unencountered bioacoustic groups and fine-grained calls. Bird-trained embeddings consistently supported better classification and few-shot transfer across the tested tasks.

Scope and approach: The study tests feature-embedding generalizability across marine mammals, bats, frogs, bird calls, and dialects.It focuses on applying large-scale audio classification models to taxonomic groups and call types absent from their original training tasks.
Main conclusion: Bird-trained embeddings consistently facilitated higher-quality classification than embeddings trained on broader audio datasets.The conclusion identifies these embeddings as effective representations for high-dimensional bioacoustic categories.
Main conclusion: Global bird acoustic classifiers can support few-shot transfer learning for novel bioacoustic tasks.This conclusion follows from their performance across previously unencountered taxonomic groups and fine-grained vocalization problems.