Source-linked AI summary
Automatic Image-Level Morphological Trait Annotation for Organismal Images
Vardaan Pahuja, Samuel Stevens, Alyson East, Sydne Record, Yu Su
TL;DR
Morphological trait extraction is slow and lacks large image–trait datasets, limiting scalable ecological analysis. The paper uses sparse autoencoders and multimodal prompting to localize parts and generate descriptions, producing BIOSCAN-TRAITS with 80K annotations across 19K insect images and improving zero-shot classification after fine-tuning.
Problem
Manual morphological annotation is slow, expert-driven, and limited by the lack of high-quality datasets linking biological images to trait-level annotations.
Method
The pipeline applies sparse autoencoders to foundation-model features, localizes salient morphological regions, and uses vision-language prompting to generate interpretable trait descriptions.
Results
80K morphological trait descriptions across 19K insect images were produced, and fine-tuning on BIOSCAN-TRAITS improved zero-shot species classification on an in-the-wild benchmark.
Takeaways & Limitations
The modular pipeline offers a scalable way to add biologically meaningful supervision to foundation models and support large-scale morphological analyses.
Takeaways & Limitations
The method assumes backbone features encode morphology-relevant signals, while latent entanglement, hallucination, and limited ground-truth annotations constrain trait correctness.
Abstract
from arXiv · showhide
Morphological traits are physical characteristics of biological organisms that provide vital clues on how organisms interact with their environment. Yet extracting these traits remains a slow, expert-driven process, limiting their use in large-scale ecological studies. A major bottleneck is the absence of high-quality datasets linking biological images to trait-level annotations. In this work, we demonstrate that sparse autoencoders trained on foundation-model features yield monosemantic, spatially grounded neurons that consistently activate on meaningful morphological parts. Leveraging this property, we introduce a trait annotation pipeline that localizes salient regions and uses vision-language prompting to generate interpretable trait descriptions. Using this approach, we construct Bioscan-Traits, a dataset of 80K trait annotations spanning 19K insect images from BIOSCAN-5M. Human evaluation confirms the biological plausibility of the generated morphological descriptions. We assess design sensitivity through a comprehensive ablation study, systematically varying key design choices and measuring their impact on the quality of the resulting trait descriptions. By annotating traits with a modular pipeline rather than prohibitively expensive manual efforts, we offer a scalable way to inject biologically meaningful supervision into foundation models, enable large-scale morphological analyses, and bridge the gap between ecological relevance and machine-learning practicality.
1 INTRODUCTION
Morphological traits connect organismal form to ecological function, but manual extraction is slow, heterogeneous, and difficult to scale. The paper proposes a species-contrastive SAE-and-MLLM pipeline that localizes anatomical regions and generates trait descriptions for large image collections.
- Morphological traits provide a mechanistic link between organismal form and environmental interactions, predicting ecological niches and functions with up to 85% accuracy.
- Manual trait measurement remains labor-intensive, with protocols varying across taxa and observations often trapped in notebooks or captions.
- The pipeline uses sparse autoencoders to identify interpretable, spatially grounded visual units corresponding to candidate morphological parts.
- Species-contrastive ranking prioritizes units that activate strongly for a focal species but weakly for closely related species, after which high-score masks are boxed and language-model prompted.
- Using the best-performing configuration, the authors label 19K images with 80K morphological trait descriptions, averaging 4.2 traits per image.
- The resulting modular pipeline is presented as a scalable alternative to expensive manual labeling and as a way to provide biologically meaningful supervision for foundation models.
2 RELATED WORK
Prior work spans interpretable representation learning, fine-grained visual recognition, and automated morphological analysis. These areas motivate localized, language-mediated trait extraction that can handle subtle cues and distribution shifts.
- Sparse Autoencoders: Sparse autoencoders learn disentangled, human-interpretable latent factors in high-dimensional representations, with variants targeting stronger feature disentanglement.
- Fine-grained Visual Recognition: Fine-grained visual recognition distinguishes categories with small inter-class variation and large intra-class variation, often relying on subtle localized cues.
- Fine-grained Visual Recognition: Fine-grained recognition methods localize discriminative regions using weak supervision or self-supervised consistency signals without dense part annotations.
- Fine-grained Visual Recognition: Real-world fine-grained recognition must address distribution shifts, while language-based methods increasingly represent part-level attributes and semantics.
- Morphological Trait Extraction: Traditional morphological analysis relies on labor-intensive manual measurements and qualitative descriptions, motivating scalable representation-learning approaches.
3 METHODOLOGY
The methodology transforms dense vision features into sparse latent factors, selects salient taxonomically focused activations, and prompts multimodal models with localized regions to produce trait descriptions.
- Sparse Autoencoder Background: Sparse autoencoders map dense vision-transformer features into high-dimensional sparse codes and reconstruct the original representation, exposing structured latent factors.
- Sparse Autoencoder Background: The encoder maps z ∈ R^d to latent space using W_e ∈ R^{n×d}, while the decoder reconstructs z̃ using W_d ∈ R^{d×n}, with corresponding bias terms.
- Sparse Autoencoder Background: The training objective minimizes reconstruction error while using a sparsity regularizer, whose coefficient α controls the sparsity–reconstruction trade-off.
- Feature Extraction: The pipeline uses DINOv2-base to extract dense visual representations from specimen images before sparse autoencoder processing.
- Trait Selection: High-activation latents above t_activation are retained, aggregated by species and genus, and filtered to select salient species-focused traits.
- Trait Description: Selected traits are supplied to multimodal language models to generate morphological descriptions, while Figure 2 contrasts this localization approach with Grad-CAM.
4 EXPERIMENTS
The experiments evaluate how SAE-guided localization, multi-image prompting, SAE settings, filtering, and MLLM scale affect morphological trait descriptions and downstream utility. Results support spatially grounded, consensus-based trait generation and interpretable SAE features.
- Comparison with MLLM-only baseline: 3.91 average human rating is achieved with SAE-localized patches in the multi-image setting, up from 3.15 for the MLLM-only baseline.The comparison uses Qwen2.5-VL-72B with a normalized frequency threshold of tfreq = 3e−3 and 1,000 input images.
- Multiple vs. Single Image per Latent: 3.91 average human rating with three images exceeds 3.84 with one image, although multi-image prompts require more tokens.Multiple images encourage attention to shared morphological features and suppress image-specific traits.
- SAE Quality: Lower sparsity performs better for both normalized frequency thresholds, trading stronger latent coverage against reconstruction error and sparsity.The authors hypothesize that broader latent coverage provides richer part proposals and reduces missed discriminative regions.
- SAE Filtering: Increasing tfreq retains fewer latent features while improving precision, creating a coverage–specificity trade-off in trait selection.Lower thresholds include broader trait coverage with more noise, whereas higher thresholds emphasize dominant, taxonomically stable traits.
- MLLM Quality: The Qwen2.5-VL-72B model produces higher human evaluation scores and better spatial grounding than Qwen2.5-VL-7B, avoiding false positive traits.MLLM inference conditioned on three SAE-selected patches takes 4.62 s per annotation, compared with 7.26 ms per image for the preceding computation.
- SAE Interpretability: Neurons 4852 and 13860 consistently activate on insect wings and antennae, respectively, indicating semantically coherent and biologically plausible latent features.The labels denote the highest annotated taxonomic level.
- Fine-tuning with Trait Supervision: Fine-tuning BioCLIP on the trait dataset yields a significant zero-shot species-classification gain on the volunteer-labeled, in-the-wild Insects benchmark.This result provides initial evidence that trait-level supervision supports better generalization.
5 CONCLUSION
The paper presents a sparse-autoencoder and multimodal-language-model pipeline that produces large-scale, interpretable morphological trait descriptions from biological images. BIOSCAN-TRAITS contains over 80K descriptions across 19K insect images, and trait-level supervision improves downstream fine-grained species classification.
- The proposed pipeline distills morphological traits into high-fidelity natural-language descriptions using sparse autoencoders and multimodal language models.
- BIOSCAN-TRAITS contains over 80K trait descriptions spanning 19K insect images and provides structured, interpretable trait-level supervision at scale.
- Trait-level supervision improves generalization in downstream tasks such as fine-grained species classification.
ETHICS STATEMENT
A scalable trait annotation pipeline could support biologically grounded foundation models and biodiversity conservation, but errors may propagate into downstream applications. The authors therefore emphasize collaboration with domain experts to ensure reliability and accuracy.
- Trait annotation tools may support biologically grounded foundation models for biodiversity conservation.
- Hallucination and domain-shift errors may propagate into species classification and conservation decision-making.
- Deployment should occur in close collaboration with domain experts to ensure reliability and accuracy.
REPRODUCIBILITY STATEMENT
The authors provide code, dataset access, and implementation details to support reproducibility. These materials include hyperparameters, the trait-generation algorithm, and prompt templates.
- Code and the BIOSCAN-Traits dataset are publicly accessible through GitHub and Hugging Face.
- Hyperparameter settings, the trait-generation pipeline, and prompt templates are provided in the supplementary materials.
- All experiments were performed on NVIDIA H100 GPUs.
APPENDICES
The supplementary material provides additional details omitted from the main text.
- The supplementary material contains additional details omitted from the main text.
A LIMITATIONS
The approach depends on morphology-relevant backbone features, may conflate co-occurring traits, and remains vulnerable to hallucination and evaluation constraints. The authors address some SAE concerns through filtering, consistency checks, and empirical validation rather than treating SAE features as true traits.
- The method assumes backbone features encode morphology-relevant signals; generic visual biases could underrepresent important biological traits.
- Some SAE latents may combine co-occurring traits, complicating fine-grained and compositional trait disentanglement.
- Smaller MLLMs are susceptible to hallucination when prompted with noisy or background-dominated patches.
- Large-scale trait correctness evaluation remains difficult because ground-truth morphological annotations are unavailable.
- The pipeline filters spurious latents using species-contrastive ranking, frequency thresholds, multi-image consistency, expert ratings, and downstream transfer evaluation.
- The authors treat SAE features as a decomposition to validate and filter rather than as the true underlying traits.
B COMPREHENSIVE RESULTS
The ablations show that spatially localized SAE patches, multiple images, and stronger multimodal models improve morphological trait description quality and grounding, with trade-offs in query cost and model accessibility.
- MLLM Quality Ablations: Multiple images encourage MLLMs to focus on traits common across images, improving precision at the cost of more tokens per query.The setup used Qwen2.5-VL-72B with a normalized frequency threshold of 3e−3 and 1,000 input images.
- MLLM Quality Ablations: GPT-5-mini achieves the highest average rating among the evaluated verbalizers, while the Qwen2.5-VL variants remain usable with in-house hosting for the 72B model.The open-source model shifts cost from per-call API pricing to amortized compute and can support on-premises data governance.
- MLLM Quality Ablations: The larger Qwen2.5-VL-72B model consistently receives higher human evaluation scores than Qwen2.5-VL-7B.In one example, the 72B model recognizes a highlighted region as background while the 7B model hallucinates a body-part description.
- Prompting Setup: The pipeline uses separate prompts for MLLM + SAE and MLLM-only baselines in both multi-image and single-image settings.The corresponding prompt designs are shown in Figures C.2–C.5.
E FEATURE DETECTOR ABLATIONS
Feature-detector ablations compare DINOv2 and CLIP representations, document dataset and evaluation procedures, and outline annotation costs and ecological uses of the resulting trait data.
- Dataset Statistics: Each image is associated with an average of 4.2 trait samples in the dataset statistics.This statistic summarizes the annotation density reported for the generated dataset.
- Feature Backbone: DINOv2-base substantially outperforms CLIP ViT-B/16 on species classification using a kNN classifier.The comparison uses a 1,000-species BIOSCAN-5M benchmark with 20 training and 30 test images per species.
- Evaluation: Trait descriptions were rated solely by the paper’s authors under an evaluation rubric shown in Table F.7.The authors state that the IRB classified the research as exempt from approval.
- Dataset Construction: BIOSCAN-5M supplies the training and dataset-generation images, and the examples include visible wing, antenna, and leg traits.The source dataset is licensed under Creative Commons Attribution 3.0 Unported.
- Ecological Uses: Automated trait labels can expand trait databases and support analyses relating morphology to environmental or behavioral variables.The paper presents the pipeline as a scalable alternative to manual expert annotation for biodiversity and ecological research.
J ADDITIONAL NEURON ACTIVATION ANALYSIS
Additional analyses show that SAE neurons activate on specific insect regions and that combining these localized regions with larger models and multiple images improves trait description precision.
- Neuron Activation Analysis: SAE neuron 4040 consistently activates on the thorax, while neuron 16584 responds to the leg-body junction.These activations provide spatially grounded morphological regions for trait extraction.
- SAE Versus MLLM-Only: The MLLM-only baseline is compared with MLLM + SAE using salient regions selected by high-activation SAE neurons.The comparison is illustrated for Scytodes intricata, Erigone psychrophila, and Morulina thulensis.
- Model Scale: Qwen2.5-VL-72B accurately identifies highlighted insect body parts and avoids hallucinations observed with the 7B counterpart.The comparison includes Agyneta straminicola and Erigone psychrophila.
- Multi-Image Prompting: Using multiple images with SAE-extracted bounding boxes improves precision over single-image prompting.Additional examples compare single-image and three-image settings for Deltocephalus fuscinervosus and Erigone arctophylacis.