PaperlayerView exact revision on arXiv

Source-linked AI summary

BEANS-Next and ROOTS: Broadening Audio-Language Capabilities for Bioacoustics

Christos Plachouras, David Robinson, Marius Miron, Gagan Narula, Paul Laisné, Anthony L. T. Fine, Benno Weck, Ellen Gilsenan-McMahon, Diane Kim, Laura Hay Mack, Maddie Cusimano, Sara Keen, Lukas Rauch, Benjamin Hoffman, Emmanuel Chemla, Emmanouil Benetos, Johan Pauwels, Milad Alizadeh, Matthieu Geist, Olivier Pietquin

arXiv:2610.10663v1cs.SDcs.LG

TL;DR

Bioacoustic evaluation and training have focused on a narrow subset of the tasks used in ethology. This paper introduces the four-group BEANS-Next benchmark and ROOTS training dataset, and reports that ROOTS training substantially improves performance across BEANS-Next tiers. The authors conclude that targeted data construction can begin to close gaps in broader bioacoustic capabilities.

  • Problem

    Existing benchmarks center on species and narrow call-type classification, while many abilities used in bioacoustic workflows remain untested and training supervision for them is limited.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces BEANS-Next across four task groups and builds ROOTS from curated archives, metadata, text synthesis, and synthetic audio, then trains tier-specific models on its supervision.

    Source: §1 · p. 2

  • Results

    ROOTS training substantially improves performance across all BEANS-Next tiers, while existing models perform poorly on broader tasks beyond familiar ones.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that shared-taxonomy evaluation and targeted data construction can begin to close capability gaps, with substantial headroom remaining.

    Source: §6 · p. 9

    Public-archive taxonomic and geographic biases may limit reliability for underrepresented taxa and ecosystems; model errors could also affect downstream decisions, so expert validation is recommended.

    Source: §6 · p. 9

Abstract

from arXiv · show

Bioacoustics and ethology encompass a wide range of audio understanding tasks, many of which stand to benefit from recent advances in large audio-language models. However, progress in the field has so far been assessed on a narrow set of tasks, primarily centered on label-centric biological category recognition, such as species and call-type classification. In this work, we introduce BEANS-Next, a benchmark grounded in a taxonomy of bioacoustics tasks spanning acoustic perception, biological category recognition, scene understanding, and in-context learning. Using BEANS-Next, we show that existing models exhibit limited performance beyond the task families emphasized by existing evaluations, constraining their usefulness for broader bioacoustic applications. To support progress on this broader task space, we also introduce ROOTS, a large-scale training resource built from expanded curated real-world data and previously underused behavioral and acoustic metadata, supplemented by audio-derived information and scalable synthetic generation where labeling is insufficient. We demonstrate that training on this dataset yields substantial progress across all task groups of BEANS-Next, moving audio-language models closer to their potential as general-purpose assistants for bioacoustics and ethology. To accelerate progress in the field, we open-source our benchmark, dataset, and data pipelines.

1 Introduction

The introduction argues that bioacoustic evaluation and training remain narrower than ethology’s task needs, then presents BEANS-Next and ROOTS as aligned evaluation and training resources. It reports that existing models are limited on broader tasks and ROOTS training improves performance across BEANS-Next tiers.

1 Introduction

  • Existing benchmarks emphasize species or narrow call-type classification, leaving abilities such as acoustic measurement, counting, temporal reasoning, comparison, and adaptation untested.The introduction also identifies limited training supervision for these abilities, in part because archives are often weakly labeled and semi-structured.Source: §1 · pp. 1–2
  • BEANS-Next shows bioacoustic models perform competitively on familiar tasks but poorly on broader ones, while general-purpose audio-language models remain poor across most tasks.Source: §1 · p. 2
  • BEANS-Next broadens evaluation across acoustic perception, biological category recognition, scene understanding, and in-context learning.Source: §1 · pp. 2–3
  • Tier-specific ROOTS training substantially improves performance across all BEANS-Next tiers, with curated metadata, text synthesis, and synthetic audio supporting distinct ability groups.The paper links metadata to biological category recognition, text synthesis to richer description and acoustic perception, and synthetic audio to scene understanding and in-context capabilities.Source: §1 · p. 2
  • ROOTS is introduced as a large-scale bioacoustic audio-language training dataset built from curated archives, underused and audio-derived metadata, LLM-based text synthesis, and synthetic audio pipelines.Source: §1 · p. 3
  • The authors release BEANS-Next, an evaluation library, ROOTS, and its synthetic-audio subset to support reproducible research.Source: §1 · p. 3

2 Related Work

Bioacoustic machine learning and language models have largely been evaluated on recognition tasks, while broader abilities needed in ethology remain underexplored. BEANS-Next and ROOTS address this gap with a broader benchmark and an aligned training corpus.

General audio-language models

  • General audio-language competence does not directly establish the fine-grained acoustic perception, biological knowledge, or study-specific adaptability needed for bioacoustic research.General LALMs and benchmarks are largely centered on human-centric audio distributions (Gong et al., 2024; Tang et al., 2024; Chu et al., 2024; Ghosh, Goel, Jayakumar, et al., 2026; Ghosh et al., 2024; Sakshi et al., 2025).Source: §2 · p. 3

Bioacoustic representations and benchmarks

  • Earlier bioacoustic machine-learning resources primarily support fixed-output classification, detection, retrieval, and representation-learning tasks.BEANS, BIRB, BirdSet, and encoder-focused representation models provide strong task-specific resources (Hagiwara et al., 2023; Hamer et al., 2023; Rauch, Schwinger, et al., 2025).Source: §2 · p. 3

Language models for bioacoustics

  • Bioacoustic language-model evaluation remains concentrated on biological category recognition, leaving acoustic measurement, event counting, temporal reasoning, call comparison, and study-specific adaptation underexplored.BioLingual and NatureLM-audio demonstrate language-conditioned bioacoustic modeling, but evaluations focus on labels (D. Robinson, A. Robinson, and Akrapongpisak, 2024; D. Robinson, Miron, et al., 2025).Source: §2 · p. 3

Gap addressed by this work

  • BEANS-Next and ROOTS address the broader ethology task gap with a benchmark and aligned training corpus organized across four tiers.Large archives offer taxonomic breadth but often have weak, heterogeneous, or semi-structured annotations (Vellinga and Planqué, 2015; Chasmai et al., 2024; Museum für Naturkunde Berlin, 2013).Source: §2 · p. 4

3 Data construction

The paper organizes bioacoustic abilities into four tiers, then constructs instruction-following examples and resources for evaluation and training. BEANS-Next uses held-out real recordings and annotations, while ROOTS combines curated data, metadata, audio-derived information, and synthetic examples.

3.1 A taxonomy of bioacoustic audio-language abilities

  • Four tiers organize bioacoustic tasks around ethological workflows and model capabilities, from low-level acoustic perception to category recognition, scene understanding, and in-context learning.The taxonomy includes acoustic properties, biological categories, structured multi-event recordings, and inference-time adaptation to text or audio examples.Source: §3.1 · p. 4

3.2 Text creation methods

  • Templated prompts convert structured annotations into task variants, while LLM-based question-answering and descriptions capture nuance in free-text and semi-structured notes.Dedicated prompts specify evidence, task, output format, constraints, and skip conditions; spectrogram descriptions are restricted to acoustic properties, and generated outputs receive automated checks and manual sample review.Source: §3.2 · p. 5

3.3 BEANS-Next benchmark construction

  • BEANS-Next is a held-out evaluation suite built from real recordings and human- or dataset-provided annotations, with its recordings and derivatives disjoint from ROOTS.For tasks centered on species identity, the benchmark also follows a held-out species protocol where applicable.Source: §3.3 · p. 5

3.4 ROOTS training data construction

  • ROOTS combines curated archives, annotated datasets, metadata, and synthetic audio to provide instruction-following supervision across the four tiers.Synthetic strongly labeled recordings address scarce temporal labels, label noise, and missing labels; strong detection supports scene tasks, while detect-diarize adds proxy speaker IDs and preserves call order and timing.Source: §3.4 · pp. 5–6

4 Experimental Setup

The experiments compare bioacoustic and general-purpose generative audio-language models using a shared evaluation pipeline. ROOTS training fine-tunes separate tier-specific models from a common base to isolate supervision effects.

4.1 Models

  • The evaluation compares NatureLM-audio v1.0 with Audio Flamingo Next and Qwen3-Omni, using a shared inference, parsing, and scoring pipeline.Generative models are selected because many benchmark tasks require open-ended or multi-audio answers; proprietary API models are excluded due to failure rates and cost.Source: §4.1 · p. 6

4.2 Training

  • Separate ROOTS fine-tunes for each tier start from NatureLM-audio v1.1, holding architecture and initialization fixed to isolate training-data contributions.The audio encoder and Q-former are unfrozen, and the LLM is fine-tuned with LoRA; a joint multi-tier model is left to future work.Source: §4.2 · p. 6

5 Results

ROOTS tier-specific training improves performance across several bioacoustic abilities, with especially strong results on in-context learning. Improvements vary by task: counting, open-ended duration identification, and some acoustic-description settings remain challenging.

5 Results

  • On established benchmarks, NatureLM-audio models outperform general-purpose LALMs on species-level classification, while v1.1 improves over v1.0 on some tasks and regresses on others.The reported regressions include ESC-50, HumBugDB, life stage, and individual counting; improvements include CBI, call type, and BirdSet.Source: §5 · p. 7

5.1 Tier 1: Acoustic Perception

  • Tier 1 training improves acoustic descriptions, SNR estimation, and mean f0 prediction, including unseen-species f0 estimation.Crow vocalization description is the exception, where general-purpose LALMs remain better; the passage suggests residual sensitivity to domain and annotation format.Source: §5.1 · p. 7

5.2 Tier 2: Biological Category Recognition

  • Tier 2 training improves presence, call-type, and behavior prediction, while captioning remains weak in absolute terms.The reported gains suggest mined metadata provides supervision beyond species labels alone.Source: §5.2 · p. 7

5.3 Tier 3: Scene Understanding

  • Tier 3 training improves scene parsing into events and relations, especially binding species identities to event-level attributes, but counting remains difficult and duration-based identification does not improve.Synthetic scenes control species identity, timing, overlap, duration, and frequency range, providing direct supervision for structured reasoning; species-count error is slightly worse than base models.Source: §5.3 · pp. 7–8

5.4 Tier 4: In-Context Learning

  • Tier 4 training yields the best results on all five in-context tasks, while gains are smallest on low-SNR Gibbons recordings.Audio Flamingo Next and the NatureLM-audio v1.1 base remain near chance on matching tasks; Qwen3-Omni performs above chance on call-type and unseen-species matching.Source: §5.4 · p. 8

6 Conclusion

This section outlines how a shared capability taxonomy and targeted data construction can address limited coverage of bioacoustic workflows, while noting remaining headroom, inherited biases, deployment risks, and the need for expert validation. It also describes prompt templates for summarizing vocalization counts, timing, and frequency; describing acoustic properties; and detecting animal and non-animal sounds.

6 Conclusion

  • A shared capability taxonomy exposed gaps in existing audio-language models, and curated metadata, LLM-based text synthesis, and synthetic audio pipelines can begin to close them.The authors frame this as a step toward supporting broader bioacoustic workflows, while noting substantial headroom remains.Source: §6 · p. 9
  • Taxonomic and geographic biases may limit reliability for underrepresented taxa and ecosystems, while hallucinated species, behaviors, or counts could affect scientific and conservation decisions.The authors excluded endangered species from their Xeno-canto sources to partially mitigate misuse risks involving species detection and geolocation metadata, and recommend expert validation.Source: §6 · p. 9
  • Gemma 4 26B MoE generated data descriptions and question-answer pairs and judged candidate examples, with an author manually reviewing samples before and after filtering.The judging criteria included instruction following, hallucination, audibility, and difficulty.Source: AI use statement · p. 9
  • AI coding assistants supported parts of the training, evaluation, data-processing, and synthesis code, and routine cluster orchestration; an author reviewed and tested all code before reported experiments.Source: AI use statement · p. 9
  • BEANS-Next, ROOTS, synthetic audio subsets, and the evaluation library are publicly available, with appendices documenting data construction, training, and evaluation.ROOTS audio is not redistributed; the authors provide guidance for retrieving and preprocessing the publicly available audio used.Source: Reproducibility statement · p. 9

B.1 Templated Data Construction

  • The templates ask for per-species call counts, first-vocalization order, overall and species-specific frequency ranges, and combined count-and-frequency summaries.Source: §B.1 · p. 17
  • A templated multilabel task asks which non-animal AudioSet sounds are present.Source: §B.1 · p. 17
  • Other prompts request mean fundamental frequency or species-conditioned descriptions of F0 range and vocalization duration.Source: §B.1 · p. 17
  • Binary-presence prompts cover birds, whales or dolphins, mammals, insects, amphibians, and animal vocalizations generally.Source: §B.1 · pp. 17–18

B.2 LLM-based Data Construction

The section presents LLM-based data construction and task prompts, then describes synthetic scenes assembled from extracted vocalizations and real environmental backgrounds.

B.2.1 Overview

  • Prompts cover when vocalizations occur, which species vocalizes in a specified order, and which species has the highest, lowest, or longest call.The examples also include co-occurrence, vocal dominance, and structural captioning.Source: §B.2.1 · p. 19
  • The overview includes open-ended and multiple-choice questions about vocalization order, pitch, duration, and frequency or dominance.Some prompts provide candidate species; others ask for an open-ended answer.Source: §B.2.1 · p. 19

B.3 Synthetic Audio Creation

  • Synthetic scenes mix extracted vocalizations called pseudovox into real environmental backgrounds, with configurable events, SNRs, and inter-event gaps.Four datasets cover call-type identification or detection, same/different call-type judgments, and few-shot detection; the three call-type tasks define call type as species intersected with acoustic cluster.Source: §B.3 · p. 19
  • Synthetic-dataset pseudovox clips come from Xeno-canto’s train_unseen split and are excluded from evaluation.Scenes are rendered at 16 kHz; the few-shot query scene is 3–10 seconds and may include a background clip.Source: §B.3 · pp. 19–20

C.1 Training Hyperparameters

The section specifies staged training and per-tier fine-tuning, describes noise and mixup augmentations, and reports compute use and corpus-statistics summaries.

C.1 Training Hyperparameters

  • The stage-2 model, trained for 1.5 million steps on 10-second clips, formed the base checkpoint for fine-tuning.Stage 1 used open-ended species classification data for 350,000 steps; both stages used a global batch size of 128 and peak learning rate of 1e−4.Source: §C.1 · p. 20

C.2 Fine-tuning Hyperparameters

  • Per-tier fine-tuning ran for two epochs from the stage-2 checkpoint, using LoRA on projection layers and clips up to 30 seconds.Fine-tuning used rank 32, α = 32, dropout 0.1, and a global batch size of 128.Source: §C.2 · p. 20

C.3 Augmentations

  • Stage-2 training adds noise augmentation, while detection tasks also use noise masking and task-aware mixup.Noise is added with probability 0.4; noise masking is used with probability 0.15 for detection, and mixup with probability 0.3 for detection from stage 2 onward.Source: §C.3 · p. 20

C.4 Computational resources

  • Fine-tuning used eight H100 80GB GPUs per run and totaled approximately 64 H100 days.Tier 1 and Tier 4 each took about one day; Tier 2 and Tier 3 each took about three days.Source: §C.4 · p. 20

C.5 Corpus statistics

  • The corpus-statistics section points to source-level audio and pair counts and describes LLM text construction that leverages nuanced behavioral and acoustic descriptions.Figure 4 presents bigram diversity for acoustic and behavioral description tasks.Source: §C.5 · p. 20

D.1 Data Curation

The curation section describes evaluation tasks spanning acoustic perception, biological detection, scene understanding, and in-context learning, with task-specific datasets and splits.

D.1 Data Curation

  • Expert acoustic-description matching uses four candidate descriptions from the same species, with one correct answer per recording.For acoustic-description tasks, mined text retains directly acoustic cues while excluding species names, call-type labels, and behavioral context.Source: §D.1 · pp. 20, 22
  • Mean F0 targets are computed over valid contour points and rounded to the nearest 10 Hz, with a separate spotted-hyena held-out split.SNR evaluation uses mined Xeno-canto field-note annotations and includes open-ended regression and four-bin multiple choice.Source: §D.1 · p. 22
  • Tier 2 includes balanced taxon-presence tasks, call-type presence or multi-label tasks, and LLM-synthesized semantic captioning and behavior questions.Semantic tasks allow behavioral meaning and semantic labels when supported by metadata, unlike Tier 1 acoustic captions.Source: §D.1 · p. 22
  • Tier 3 scene-understanding questions use annotated recordings and cover counts, timing, co-occurrence, vocalization order, pitch, duration, dominance, and structural captions.Ambiguous or insufficiently supported annotation rows are excluded from the scene-understanding evaluation.Source: §D.1 · p. 22
  • Tier 4 evaluates in-context adaptation through gibbon and DCASE few-shot detection, call-type matching, and unseen-species matching.The gibbon evaluation includes full naturally imbalanced and balanced splits; unseen-species examples use held-out taxa and, when available, hard related-species distractors.Source: §D.1 · pp. 22–23

D.1.1 Evaluation tasks and abbreviations

The appendix maps BEANS-Next task labels, describes its held-out taxa and long-tail species distribution, and profiles the audio-language models used in the experiments.

D.1.1 Evaluation tasks and abbreviations

  • Table 9 uses higher-level task groups followed by abbreviated result-table column names; examples include Tier 3 species identification by characteristic (order and highest-pitched species) and Tier 4 multi-audio few-shot.Source: Table 9 · p. 23, §D.1.1 · p. 24

D.2 List of BEANS-Next held-out taxa

  • BEANS-Next broadens benchmark coverage beyond species recognition and isolated acoustic perception, while its 2,698-species distribution includes 1,183 species (44%) represented once.A few focal species contribute hundreds of samples, with a rapid drop-off after the top few.Source: Table 10 · p. 25, Figure 6 · p. 25, Figure 7 · p. 26

E.1 Models

  • The appendix profiles NatureLM-audio, Audio Flamingo Next, and Qwen3-Omni as audio-language models used in the experiments.NatureLM-audio combines a BEATs encoder and Llama 3.1 8b backbone; v1.1 is the common base checkpoint for tier-specific ROOTS experiments. AF-Next targets speech, environmental sounds, and music, while Qwen3-Omni is used in its instruction-tuned variant for zero-shot audio understanding and captioning.Source: §E.1 · pp. 27–28

E.2 Evaluation library

The evaluation runner separates model serving from local post-processing and metric computation. The appendix specifies task-dependent parsing and scoring, then reports the computational resources used.

E.2 Evaluation library

  • The runner sends each example's audio and prompt to a model endpoint, then post-processes predictions and computes deterministic metrics or routes non-deterministic tasks to an LLM judge.It writes per-example raw, processed, and scored predictions, plus aggregate per-task metrics.Source: §E.2 · p. 28

E.3 Post-processing: parsing and label matching

  • Post-processing parses outputs into segments before a cleaner normalizes them and aligns labels; fuzzy matching is reserved for fixed-vocabulary tasks, not free-form responses.Free-form tasks receive only whitespace and end-of-sequence cleanup to avoid corrupting natural-language outputs.Source: §E.3 · p. 28

E.4 Task types and metrics

  • Each task type determines its post-processing configuration and metric family.The appendix defines separate metric procedures for classification, detection, and captioning.Source: §E.4 · p. 29

E.4.1 Classification

  • Classification uses exact-match accuracy and equally averaged per-class precision, recall, and F1 after label alignment.These metrics are computed from post-processed predictions.Source: §E.4.1 · p. 29

E.4.2 Detection (multi-label presence)

  • Detection uses macro average precision from per-label precision–recall curves, plus macro precision, recall, and F1 over predicted and target label sets.Source: §E.4.2 · p. 29

E.4.3 Captioning

  • Captioning is scored at corpus level with mean CIDEr normalized to [0, 1], because single-sample CIDEr is degenerate.Source: §E.4.3 · p. 29

E.5 Computational resources

  • Main evaluations used about 4 A100 days and 1 H100 day; ablations used about 1 A100 day and 0.5 H100 day.The ablations covered only BEANS-Next, around an eighth of the samples.Source: §E.5 · p. 29

E.6 Text-only and Gaussian-noise controls

The controls compare real audio with noise and no-audio conditions to probe whether models rely on audio, while also tracking answer validity. Real audio helps on several tasks, but output validity and compatibility complicate interpretation.

E.6 Text-only and Gaussian-noise controls

  • Text-only and noise controls use distinct missing-audio conditions, and scoring tracks answer validity alongside task performance.The informed condition adds an unavailable-audio instruction; the silent condition removes audio without that instruction, while noise replaces real audio at −20 dBFS RMS.Source: §E.6 · pp. 29–30
  • Informed LLMs produced valid answers on 99.5–99.6% of Tier 1 MCQs, versus 1.3–5.8% silently; valid answers could still be uninformative.Silent Qwen3 and Llama answered “Yes” on all 2,000 bird-presence examples, scoring 50% accuracy.Source: §E.6 · p. 30
  • Qwen3-Omni bird and amphibian presence accuracy fell from 90.0% and 86.5% with real audio to 50.0% for both with noise.Audio Flamingo Next showed similar reductions, from 88.3% and 80.4% to 50.0% and 49.5%.Source: §E.6 · p. 30

F Examples

  • The examples table pairs task family and prompt with reference output, including scene descriptions and audio-exemplar matching tasks.Source: Table 15 · p. 35, F Examples · p. 36
Loading 2610.10663v1…