Source-linked AI summary
Do bioacoustic foundation models hear through the noise ? SNR and generalization in underwater detection
Lucie Jean-Labadye, Nicolas Farrugia, Axel Marmoret, Dorian Cazau, Olivier Adam
TL;DR
The paper addresses limited evidence about whether bioacoustic foundation models generalize under operational recording conditions. It evaluates four models as frozen encoders on continuous underwater data using controlled SNR datasets and linear probes. Evaluation SNR accounts for more performance variation than model choice, and performance remains low, motivating evaluation under realistic noise and deployment conditions.
Problem
Benchmarks do not consistently reflect practitioners’ data conditions, leaving limited evidence about foundation-model performance under operational conditions.
Source: §1 · p. 2Method
The study evaluates four bioacoustic foundation models as frozen encoders with linear probes on continuous PAM data across controlled SNR strata.
Source: §1 · p. 2Results
Evaluation SNR accounts for more variation in detection performance than model choice, and AP is highest when training and evaluation share a noise regime, though absolute values remain low.
Source: §4 · p. 4, §3.3 · p. 4Takeaways & Limitations
Bioacoustic foundation models should be evaluated on data conditions practitioners actually encounter, including realistic noise and deployment conditions.
Source: §4 · p. 4Low-SNR calls are harder to annotate, so incomplete ground truth may contribute to the observed performance gap.
Source: §3.3 · p. 4
Abstract
from arXiv · showhide
Bioacoustic foundation models are claimed to generalize across taxa and recording conditions, but are trained mainly on terrestrial archives of focal, high-quality recordings and evaluated on benchmarks emphasizing architecture over data conditions. We evaluate four such models as frozen encoders on an underwater detection task, using linear probing on continuously recorded data from the Kerguelen Plateau. Four datasets of identical size were built to differ only in the signal-to-noise ratio (SNR) of their positive segments, and probes were trained and evaluated across all combinations. Evaluation SNR accounts for more variation in average precision (AP) than model choice, and performance peaks when training and evaluation share a noise regime --- the configuration least representative of a continuous deployment. Training on mixed SNR redistributes performance toward low-SNR conditions without improving it overall, and AP remains low throughout. In light of these results, we call for evaluation standards grounded in realistic noise and deployment conditions rather than curated, high-quality recordings.
1. INTRODUCTION & PRIOR WORK
Passive acoustic monitoring makes long-term cetacean observation feasible, but its data volumes make expert annotation difficult to scale and reproduce. The paper examines whether bioacoustic foundation models generalize to continuous underwater detection when signal-to-noise ratio (SNR) is controlled.
1. INTRODUCTION & PRIOR WORK
- Continuous passive acoustic monitoring supports long-term cetacean study, but its data volume makes manual analysis intractable and expert annotation time-consuming, expertise-dependent, and difficult to reproduce.PAM is particularly suited to ecosystems where visual surveys are difficult and species are vocally active.Source: §1 · p. 1
- Bioacoustic foundation models promise task transfer with little retraining, but existing models are often developed for specific settings rather than transfer across locations or recording conditions.Deep learning has improved PAM analysis, yet these models are often neither designed nor evaluated for transfer beyond their development setting.Source: §1 · p. 1
- Available benchmarks and underwater datasets provide limited evidence about operational conditions, so reported performance may not reflect the conditions practitioners face.Underwater resources may be dated, focal excerpts, or partially annotated, while benchmarks often give comparatively little attention to partition differences and data quality under deployment conditions.Source: §1 · pp. 1–2
- SNR affects detection difficulty and is used to model detection probability for population-density estimates, making it consequential for interpreting acoustic trends.Unaccounted model biases could make trends in estimated call densities difficult to attribute reliably to biological drivers.Source: §1 · p. 2
- Prior SNR work studied terrestrial recordings and task-specific models, leaving foundation-model generalization under continuous underwater noise untested to the authors’ knowledge.The study proposes frozen-encoder linear probes on continuous PAM data and controlled SNR strata to test that gap.Source: §1 · p. 2
2. MATERIAL & METHODS
The methods use continuously recorded Kerguelen Plateau data, stratify positive segments by SNR, and compare four frozen foundation-model encoders with linear probes across matched dataset conditions.
2.1. Data collection
- Recordings came from a continuously operating moored hydrophone off the Kerguelen Plateau in 2014 and 2015, with seven annotated call types pooled into weakly labeled presence/absence segments.The recorder sampled at 12 kHz with 16-bit resolution; annotations were time–frequency bounding boxes.Source: §2.1 · p. 2
2.2. Data preprocessing
- Positive segments were divided into three equal-size SNR levels, with boundaries at −8.66 dB and −4.01 dB; each one-hour recording yielded overlapping 5 s segments.A segment was positive when it contained at least 50% of an overlapping event’s duration; the resulting labels were 96% negative.Source: §2.2 · p. 2
2.3. Pre-trained models
- The four encoders were esp-effnet, esp-eat, Perch2.0, and multi-species-whales, trained on differing bioacoustic and general-purpose data mixtures.The models include an underwater-data-trained model, multi-species-whales, which was mostly trained on NOAA acoustic data.Source: §2.3 · pp. 2–3
2.4. Experimental set-up
- Four equal-sized datasets differed only in positive-segment SNR composition, and each model’s embeddings were evaluated with linear probes across all training–test SNR combinations.Training and validation sets were balanced by negative subsampling, while test sets retained the original class distribution; the SNR distribution and example spectrograms are shown in Fig. 1.Source: §2.2 · p. 2, §2.4 · p. 3, Figure 1 · p. 3
- Average precision summarized each probe’s precision–recall curve, and repeated-measures analyses tested model, training-SNR, and test-SNR effects across folds.The mixed-SNR control was compared with the mean of the three pure SNR conditions using planned contrasts.Source: §2.4 · p. 3
3. RESULTS & DISCUSSION
Across models, evaluation SNR matters more to AP than model choice, and performance is low on continuous marine recordings. AP is highest when training and evaluation SNR match, while mixed-SNR training shifts performance toward lower-SNR conditions.
3.2. Effects of the SNR
- AP rises with test SNR regardless of training stratum; low-SNR-trained probes improve from 0.115 AP on low-SNR data to 0.180 AP on high-SNR data.The reported test-SNR effect is F(2, 8) = 3.9 × 10^4, partial η2 = 0.999.Source: §3.2 · p. 3
- High-SNR training transfers poorly to low-SNR evaluation: its 0.094 AP is below medium-trained (0.107) and low-trained (0.115) probes.Performance is highest when training and evaluation strata match, an oracle condition that assumes deployment noise is known in advance.Source: §3.2 · pp. 3–4
3.1. Model comparison
- esp-effnet exceeds msw, perch, and esp-eat by +0.017, +0.061, and +0.062 AP, respectively, while test-set spread (0.139 AP) exceeds model spread (0.062 AP).Fig. 2 reports pairwise AP differences averaged across SNR conditions.Source: §3.1 · p. 3, Figure 2 · p. 4
3.2. Effects of the SNR
- Mixed-SNR training gains +0.0221 AP on average across models and evaluation strata, improving esp-effnet on low- and medium-SNR data while reducing its high-SNR AP.For esp-effnet, the control-trained probe scores 0.105 vs. 0.094 on low-SNR, 0.137 vs. 0.129 on medium-SNR, and 0.382 vs. 0.421 on high-SNR data, compared with the high-trained probe.Source: §3.2 · p. 4
3.3. Discussion
- AP remains low on continuous marine recordings, and evaluation conditions matter more than model choice; high-SNR training and evaluation reaches 0.421 AP versus 0.094 AP on low-SNR evaluation.The high-SNR benchmark-like condition is least representative of deployment, where distant low-SNR calls dominate.Source: Figure 3 · p. 4, §3.3 · p. 4
- Low-SNR calls may be harder to annotate, so some apparent performance gaps may reflect incomplete ground truth rather than model capability.A preliminary inspection found high-confidence detections with spectrogram patterns consistent with unannotated calls; the authors suggest using models to direct annotation effort.Source: §3.3 · p. 4
4. CONCLUSION
Evaluation noise conditions account for more variation in detection performance than foundation-model choice, and performance peaks when training and evaluation share a noise regime—a condition rarely met in long-term deployments. The authors argue for evaluation under practitioners’ conditions and propose testing other sources of variability and using detections to guide annotation.
4. CONCLUSION
- Evaluation SNR accounts for more variation in detection performance than foundation-model choice, and performance is highest when training and evaluation share a noise regime rarely met in long-term deployment.Long-term deployment noise conditions are neither known in advance nor stable.Source: §4 · p. 4
- Future work should test geographic area, time period, and recording hardware, while detections on unannotated segments suggest confidence-guided annotation of faint calls.The proposed active-learning pipeline directs annotation effort toward the faintest calls.Source: §4 · p. 4
- Evaluation should reflect conditions practitioners encounter because acoustic monitoring feeds conservation policy and cetaceans face sustained pressure.The authors characterize foundation-model bias as an immediate practical requirement.Source: §4 · p. 4