PaperlayerView exact revision on arXiv

Source-linked AI summary

BioVITA: Biological Dataset, Model, and Benchmark for Visual-Textual-Acoustic Alignment

Risa Shinoda, Kaede Shiohara, Nakamasa Inoue, Kuniaki Saito, Hiroaki Santo, Fumio Okura

arXiv:2603.23883v1cs.CV

TL;DR

Integrating audio with visual and taxonomic-text representations remains an open challenge for comprehensive animal-species understanding. BioVITA addresses this gap with a unified tri-modal dataset, two-stage alignment pipeline, and cross-modal retrieval benchmark, achieving superior performance across diverse retrieval scenarios and ecological trait prediction.

  • Problem

    Visual-textual-acoustic alignment remains underexplored, while existing datasets often cover only modality pairs and differ in taxonomic hierarchies and scale.

  • Method

    BioVITA combines a large-scale tri-modal dataset with 1.3 million audio clips, 2.3 million images, and 34 ecological trait labels, a two-stage training pipeline, and six-direction retrieval evaluation.

  • Results

    BioVITA demonstrates superior performance across diverse retrieval scenarios and ecological trait predictions while capturing nuanced behavioral and ecological signals.

  • Takeaways & Limitations

    The framework provides a unified representation space for multimodal biodiversity understanding beyond taxonomy.

  • Takeaways & Limitations

    Potential test-time data leakage from benchmark-overlapping training datasets prevents direct benchmarking against some prior models.

Abstract

from arXiv · show

Understanding animal species from multimodal data poses an emerging challenge at the intersection of computer vision and ecology. While recent biological models, such as BioCLIP, have demonstrated strong alignment between images and textual taxonomic information for species identification, the integration of the audio modality remains an open problem. We propose BioVITA, a novel visual-textual-acoustic alignment framework for biological applications. BioVITA involves (i) a training dataset, (ii) a representation model, and (iii) a retrieval benchmark. First, we construct a large-scale training dataset comprising 1.3 million audio clips and 2.3 million images, covering 14,133 species annotated with 34 ecological trait labels. Second, building upon BioCLIP2, we introduce a two-stage training framework to effectively align audio representations with visual and textual representations. Third, we develop a cross-modal retrieval benchmark that covers all possible directional retrieval across the three modalities (i.e., image-to-audio, audio-to-text, text-to-image, and their reverse directions), with three taxonomic levels: Family, Genus, and Species. Extensive experiments demonstrate that our model learns a unified representation space that captures species-level semantics beyond taxonomy, advancing multimodal biodiversity understanding. The project page is available at: https://dahlian00.github.io/BioVITA_Page/

1. Introduction

BioVITA addresses the open challenge of integrating visual, textual, and acoustic representations for species understanding. It combines a large tri-modal dataset, a unified model, and a cross-modal retrieval benchmark.

  • Motivation: Visual-textual-acoustic alignment remains an open challenge despite advances in biological image-text and animal audio-text modeling.Existing approaches establish image-taxonomy and audio-text alignment, but do not integrate all three modalities.
  • Motivation: Current multimodal datasets mainly contain image-text or audio-text pairs, creating a need for a unified ecological dataset across modalities.Existing datasets also differ in taxonomic hierarchies and scale.
  • BioVITA: BioVITA comprises BioVITATrain, BioVITAModel, and BioVITABench for training, representation learning, and species-level retrieval evaluation.The framework is presented as a three-part visual-textual-acoustic alignment system.
  • BioVITA: 1.3 million audio clips and 2.3 million images span 14k species and include textual taxonomic annotations and 34 ecological traits.The dataset supports the framework's unified multimodal training setting.
  • BioVITA: The model is evaluated across six cross-modal retrieval directions after learning unified representations.The directions include image-to-audio, audio-to-text, text-to-image, and their reverse directions.

2. Related Works

Prior work advances visual and acoustic species recognition but largely treats modalities separately. BioVITA builds on this landscape with broader multimodal integration while addressing dataset and evaluation constraints.

  • Species Recognition: BioCLIP and BioCLIP2 advance image-text representation learning for biodiversity and cross-domain species understanding.These models align biological images with textual taxonomic information.
  • Species Recognition: CLAP and related systems link animal vocalizations with text for species classification, detection, retrieval, and biodiversity indexing.The related work describes acoustic models including NatureLM-Audio, BirdNET, and Perch.
  • Multi-modal Recognition: Visual-acoustic research remains limited, with SSW60 integrating video, audio, and images but covering only 60 bird species.The passage contrasts this limited scope with larger unified multimodal representation efforts.
  • Evaluation Caveat: Some compared models may overlap with the benchmark data, so the authors do not directly benchmark against them because of possible test-time leakage.This caveat concerns evaluation against models trained on datasets overlapping the benchmarks.

3. Training Dataset for BioVITA

BioVITATrain is a large-scale multimodal dataset unifying audio, images, textual labels, and ecological traits across a broad animal taxonomy. Its construction combines curated bioacoustic data, fine-grained annotation, held-out benchmarking data, and aligned visual samples.

  • Dataset Overview: BioVITATrain contains 1.3 million audio clips and 2.3 million images covering 14k species with 34 fine-grained traits.The data use publicly available sources under a consistent, license-compatible protocol.
  • Dataset Construction: The dataset construction pipeline comprises audio curation, fine-grained annotation, and visual data consolidation.Audio is collected from iNaturalist, Xeno-Canto, and the Animal Sound Archive, totaling 1.3 million Creative Commons clips.
  • Fine-Grained Annotation: 34 ecological traits include categories such as diet type, activity pattern, and habitat, with LLM-assisted annotation followed by manual verification of changes.Trait labels are extracted, completed, and reviewed before changed values are manually checked.
  • Benchmark Construction: 325 low-sample species and an additional random 10% of remaining data are held out to construct BioVITABench.The held-out data are reserved from training for performance evaluation.
  • Dataset Statistics: The dataset spans 5 classes, 84 orders, 538 families, 3,612 genera, and 14,133 species, with Aves showing the greatest diversity.Audio durations average 24.6 seconds, sampling rates are predominantly 44.1 kHz, and most images range from 119×119 to 2048×2048 pixels.

4. BioVITA Model

BioVITAModel aligns audio with BioCLIP2 image and text representations through a two-stage training framework. It first trains audio-text alignment, then activates audio-image and image-text objectives for unified VITA alignment.

  • 4.1. Architectures: BioVITAModel uses audio, image, and text encoders, building on BioCLIP2 image-text encoders and a CLAP-based audio encoder.The audio encoder uses HTS-AT to extract representations from mel-spectrogram inputs.
  • 4.2. Two-Stage Training: Stage 1 trains the audio encoder with only the audio-text contrastive loss using paired audio clips and species-label text prompts.Audio-text similarities are computed between normalized audio and text embeddings, with randomly selected prompt templates.
  • 4.2. Two-Stage Training: Stage 2 activates audio-image and image-text contrastive losses after audio-text convergence to achieve visual-textual-acoustic alignment.The three losses are combined as a weighted sum, while audio and text encoders become trainable.
  • 4.3. BioVITA Benchmark: BioVITA retrieval tasks identify relevant samples from databases of 100 samples, providing the model’s evaluation setup.Figure 7 illustrates queries retrieving relevant samples marked by green rectangles from databases containing 100 candidates.
  • 4.2. Two-Stage Training: Stage 2 continues for 10 epochs with a halved learning rate and gradually increases λ from 0 to 0.1 over the first 2 epochs.The schedule is intended to prevent an undesirable increase in the Stage 1 audio-text contrastive loss.

5. BioVITA Benchmark

BioVITA Bench evaluates cross-modal species retrieval across all six modality directions, three taxonomic levels, and seen or unseen species groups. Its design yields 36 retrieval scenarios for analyzing multimodal performance, ecological granularity, and generalization.

  • 5. BioVITA Benchmark: BioVITA Bench is a benchmark for cross-modal species-level retrieval across image, text, and audio data.It evaluates retrieval rather than only single-modality species recognition.
  • 5.1. Benchmark Design: The benchmark defines six retrieval directions: I2A, A2I, I2T, T2I, A2T, and T2A.These exhaustive directions also support comparisons with bi-modal models on modality-specific subsets.
  • 5.1. Benchmark Design: Retrieval is evaluated at Species, Genus, and Family taxonomic levels, with Family retrieval described as more challenging because characteristics vary more widely within higher-level taxa.Higher-level tasks draw queries and candidates from different species within the same genus or family.
  • 5.1. Benchmark Design: Seen and unseen species groups test generalization by excluding unseen species from the training dataset.The unseen subset is intended to reflect ecological settings involving previously unobserved taxa.
  • 5.1. Benchmark Design: 36 retrieval scenarios result from combining six modality directions, three ecological levels, and two generalization groups.Each scenario consists of independent query-database retrieval tasks.
  • 5.1. Benchmark Design: Each retrieval database contains 100 samples, including one relevant target and 99 distractors.Queries use one modality, while each database contains samples from one of the two remaining modalities.

6. Experiments

BioVITA is evaluated across cross-modal retrieval, higher taxonomic levels, unseen species, ecological traits, and training-component ablations. It achieves strong unified retrieval performance, benefits from two-stage alignment and taxonomy-aware prompting, and captures ecological signals particularly well in audio.

  • Species-Level Cross-Modal Retrieval: 71.7% average Top-1 and 89.2% Top-5 accuracy demonstrate BioVITA’s effectiveness across six cross-modal retrieval directions.The model significantly outperforms ImageBind, while Stage 1 already provides substantial gains and Stage 2 further improves retrieval.
  • Prompt Settings: Scientific-name prompts achieve higher accuracy than common-name prompts, indicating clearer taxonomic information for the model.This prompting strategy helps the model exploit hierarchical taxonomic structure learned during training.
  • High-Level Retrieval: BioVITA Stage 2 outperforms ImageBind and BioCLIP 2 across genus- and family-level retrieval directions, with especially pronounced gains for A2I and I2A.Higher-level retrieval evaluates shared genus or family membership across different species, requiring broader ecological and taxonomic similarity.
  • High-Level Retrieval: Retrieval performance decreases at genus and family levels because related species can differ substantially in visual appearance and acoustic properties.Family-level classes therefore do not necessarily form tightly clustered embedding regions.
  • Unseen-Species Generalization: 51.9% average Top-1 and 73.0% Top-5 accuracy on 325 unseen species demonstrate robust generalization to novel taxa.The unseen-species evaluation uses taxa completely withheld during training.
  • Ecological Trait Prediction: BioVITA improves ecological trait learning particularly in audio, especially for behavioral traits such as trohabitat and migration.The authors associate these gains with acoustic representations capturing temporal and behavioral characteristics of vocalizations.
  • Ablation Study: Removing Stage 1 degrades performance, training from scratch is suboptimal, and both findings support the contribution of staged alignment and BioCLIP 2 initialization.The ablation study attributes meaningful contributions to each training component.

7. Conclusion

BioVITA unifies biological audio, image, and text representations through a large-scale tri-modal dataset and two-stage training pipeline. Experiments show strong retrieval and ecological trait prediction performance, including nuanced behavioral and ecological signals.

  • Conclusion: BioVITA combines a 14k-species tri-modal dataset with a two-stage pipeline to unify audio, image, and text representations.The framework supports retrieval across diverse scenarios and ecological trait prediction.

Supplementary Material

The supplementary material provides implementation details, image-dataset collection procedures, additional analyses, visualizations, training-data-size studies, and further dataset details.

  • Supplementary Material: The supplement covers baseline and prompt implementations, image-data collection, additional comparisons, visualizations, training-data-size analyses, and dataset details.It also includes dataset visualizations and examples.

A. Implementation Details

The supplementary implementation details describe baseline models, taxonomy-aware prompts, BioVITA’s encoder architecture, data splits, benchmark construction, leakage prevention, and comparisons with BioLingual.

  • Model Details: CLIP, CLAP, ImageBind, and BioCLIP 2 provide unimodal-pair or tri-modal baselines for evaluating cross-modal alignment.CLIP and BioCLIP 2 support image-text, CLAP supports audio-text, and ImageBind integrates audio, image, and text.
  • Prompt Design: Taxonomy-aware audio-text prompts randomly select common-name, scientific-name, taxonomic, or combined templates during training.The templates augment species names with scientific and taxonomic information.
  • Model Architecture: BioVITA uses BioCLIP2 image and text towers, a CLAP audio encoder, and a linear audio-to-vision/text projection layer.The image tower is frozen, most of the text encoder is frozen, and the full model contains 581.5M parameters with 154.5M trainable.
  • Data Splits: The audio data split holds out 325 species entirely from training, producing approximately 1.3M training samples and 44K test samples.Evaluation limits audio clips to around ten per species for efficiency.
  • Benchmark Construction: The benchmark uses 100-option multiple-choice questions across species, genus, and family levels, covering 9,725 species.It includes roughly 30K species-level, 10K genus-level, and 1.7K family-level questions per task type.
  • Leakage Prevention: Image training and test data are curated to prevent overlap, including exclusion of matching iNaturalist observation IDs.The training image collection covers 12,916 species, or 91.4% of the audio training split.
  • Model Comparison: BioVITA outperforms BioLingual for species-level classification and clearly outperforms it with family- and genus-level prompts.The authors attribute this result to taxonomy-aware prompting inherited from BioCLIP.

C.2. Evaluation on Other Datasets

BioVITA is evaluated against TaxaBind and on external benchmarks, while embedding visualizations and dataset-size experiments assess representation quality and robustness. Results show consistently stronger retrieval, preserved audio structure, and substantial benefits from larger training sets.

  • External retrieval evaluation: BioVITA consistently outperforms TaxaBind on species retrieval measured by top-5 accuracy.The comparison attributes the improvement to the larger dataset and VITA training strategy.
  • External retrieval evaluation: Zero-shot species-level retrieval is evaluated with top-1 accuracy across CUB-200, BioCLIP-Rare, and iSoundNat.Results are averaged over both retrieval directions, and the authors report model generality.
  • Representation analysis: Stage 2 preserves the inherent structure of audio clusters after tri-modal learning.t-SNE visualizations cover the top six categories at species, family, and order taxonomy levels.
  • Training-scale analysis: Downsampling the training set to one-fourth or one-half is used to study how dataset size affects performance.At least one sample per species is retained when downsampling would otherwise remove a species.
  • Training-scale analysis: Larger training audio datasets substantially improve performance and help BioVITA learn robust audio representations.The authors report that model performance benefits greatly from extensive training data.
  • Dataset examples: Annotation examples include images, audio recordings, taxonomic information, and trait annotations for two animal species.The examples feature Tokay Gecko and Schlegel’s Green Tree Frog.

D.2. Dataset Distribution

The dataset distribution section presents training–test genus distributions, dataset examples, and evaluation-table contexts. These materials document how data are organized across splits and experiments.

  • Genus distribution: Figures 12–15 show genus distributions with training data in blue and test data in orange.The figures provide split-wise distribution views across the dataset.
  • Evaluation tables: Table 10 reports classification results for BioLingual and BioVITA using a new 2024 audio test set.The new test set addresses possible overlap between BioLingual’s training data and the original test data.
  • Evaluation tables: Table 11 compares BioVITA with TaxaBind, while Table 12 evaluates the model on other image and audio benchmarks.Table 13 varies the training dataset size and distinguishes scientific-name and common-name inference prompts.
  • Representation visualization: Figure 10 visualizes t-SNE embeddings and reports that Stage 2 learns all three modalities without collapsing the audio feature embedding space.The visualization is used to inspect the learned multimodal representation.
  • Dataset examples: Figure 11 presents dataset examples for which each species includes images, audio recordings, taxonomic information, and trait annotations.The examples are Tokay Gecko and Schlegel’s Green Tree Frog.
Loading 2603.23883v1…