Source-linked AI summary
AnimalCLAP: Taxonomy-Aware Language-Audio Pretraining for Species Recognition and Trait Inference
Risa Shinoda, Kaede Shiohara, Nakamasa Inoue, Hiroaki Santo, Fumio Okura
TL;DR
Species unseen during training are difficult to recognize, especially when rare species lack sufficient training data. AnimalCLAP addresses this gap with a taxonomy-aware audio-text framework and an annotated vocalization dataset, improving unseen-species generalization and ecological-trait classification over CLAP.
Problem
Species unseen during training remain difficult to recognize, while animal-specific taxonomic and ecological knowledge is not fully explored in audio-text learning.
Method
AnimalCLAP trains audio-text representations with taxonomic prompts on vocalizations from 6,823 species annotated with 22 ecological traits.
Results
AnimalCLAP outperforms CLAP across ecological-trait classification tasks and improves generalization to unseen species.
Takeaways & Limitations
Animal vocalizations can support species recognition and inference of diverse ecological and biological traits, including for unseen species.
Abstract
from arXiv · showhide
Animal vocalizations provide crucial insights for wildlife assessment, particularly in complex environments such as forests, aiding species identification and ecological monitoring. Recent advances in deep learning have enabled automatic species classification from their vocalizations. However, classifying species unseen during training remains challenging. To address this limitation, we introduce AnimalCLAP, a taxonomy-aware language-audio framework comprising a new dataset and model that incorporate hierarchical biological information. Specifically, our vocalization dataset consists of 4,225 hours of recordings covering 6,823 species, annotated with 22 ecological traits. The AnimalCLAP model is trained on this dataset to align audio and textual representations using taxonomic structures, improving the recognition of unseen species. We demonstrate that our proposed model effectively infers ecological and biological attributes of species directly from their vocalizations, achieving superior performance compared to CLAP. Our dataset, code, and models will be publicly available at https://dahlian00.github.io/AnimalCLAP_Page/.
1. INTRODUCTION
AnimalCLAP addresses the challenge of recognizing species unseen during training by combining taxonomy-aware audio-text learning with ecological trait inference. It introduces a large annotated vocalization dataset and reports improved generalization over CLAP.
- Automated acoustic recognition supports biodiversity monitoring, especially where dense forests limit visual species identification.
- Species unseen during training remain difficult to recognize because rare species provide insufficient training data.
- AnimalCLAP introduces a taxonomy-aware language-audio framework that incorporates hierarchical biological information into audio-text representations.
- The dataset covers 6,823 species with 22 ecological traits, including recordings from rare species for audio-text learning and biodiversity monitoring.
- AnimalCLAP generalizes to unseen species and maintains robust trait classification performance for unseen species.
2. AnimalCLAP DATASET
The AnimalCLAP dataset combines extensive animal vocalizations with taxonomic and ecological-trait annotations, using carefully designed splits to evaluate generalization to rare, unseen species.
- The dataset contains 4,225 hours of vocalizations covering 6,823 species, with 22 ecological trait labels.
- The test set contains 300 rare species selected to be disjoint from training and validation species for generalizability evaluation.
- Recordings were collected from iNaturalist and Xeno-canto, spanning observations through the first half of 2025.
- Trait annotation defines 22 ecological traits using categorical and multi-label values, with labels manually verified after extraction.
- Unseen test species retain genus and family connections to training species, supporting evaluation of cross-species generalization.
3. AnimalCLAP MODEL
AnimalCLAP aligns audio and textual representations using prompts that include species names and taxonomic sequences. Its training and fine-tuning procedures support species recognition and ecological-trait prediction from vocalizations.
- AnimalCLAP learns a joint audio-text embedding space in which taxonomic structure enhances generalization to unseen species.
- Training randomly selects among five prompts combining common names, scientific names, and taxonomic sequences.
- The model uses CLIP contrastive loss to increase similarity for correct audio-text pairs and decrease similarity for incorrect pairs.
- Audio recordings are resampled to 48 kHz and cropped into 10-second clips before encoding with HTS-AT and a RoBERTa-based Transformer.
- Trait fine-tuning uses an audio encoder, two MLP layers, and a linear classifier to predict the 22 annotated ecological traits.
4. EXPERIMENTS
Experiments evaluate AnimalCLAP’s taxonomy-aware training for species recognition, hierarchical consistency, and ecological-trait prediction. The model achieves strong generalization across prompts, benefits from ordered taxonomy, and outperforms CLAP across ecological-trait tasks.
- Species classification: AnimalCLAP consistently achieves the highest species-classification performance across test metrics and maintains robust performance across prompt types.Single-type models excel on their respective query types but generalize less effectively to other settings.
- Species classification: 26.1% top-1 accuracy with scientific-name prompts exceeds 21.4% with common-name prompts.Scientific names provide less ambiguous and more structured signals than culturally variable common names.
- Biological hierarchy: Ordered taxonomic prompts significantly outperform randomized orders in top-1 accuracy across all test prompts.The broad-to-narrow class-to-species sequence supports learning biological hierarchies.
- Biological hierarchy: Ordered prompts produce more taxonomically coherent errors, with higher matches from class through genus than randomized prompts.The error analysis measures cases that miss species-level predictions but correctly identify higher taxonomic ranks.
- Ecological traits: AnimalCLAP consistently outperforms CLAP across ecological-trait classification tasks.The improvement is particularly pronounced for Activity pattern, Locomotion, and Migration.
- Ecological traits: Acoustic information provides a powerful tool for classifying species’ behavioral and ecological strategies.Gains are more modest for broad environmental traits such as forest Habitat and tropical or subtropical Climate, though these traits remain learnable from acoustic data.
5. CONCLUSION
AnimalCLAP integrates taxonomy into audio-text embeddings and improves generalization to unseen species. Its dataset also provides a benchmark for trait prediction in unseen species.
- AnimalCLAP integrates taxonomy structure into audio-text embeddings and improves generalization to species unseen during training.
- The AnimalCLAP dataset can serve as a benchmark for trait prediction of unseen species.