PaperlayerView exact revision on arXiv

Source-linked AI summary

BioCLIP: A Vision Foundation Model for the Tree of Life

Samuel Stevens, Jiaman Wu, Matthew J Thompson, Elizabeth G Campolongo, Chan Hee Song, David Edward Carlyn, Li Dong, Wasila M Dahdul, Charles Stewart, Tanya Berger-Wolf, Wei-Lun Chao, Yu Su

arXiv:2311.18803v3cs.CVcs.CLcs.LG

TL;DR

Biology computer-vision workflows are often bespoke and labor-intensive, motivating a general vision foundation model for organismal biology. The paper builds TREEOFLIFE-10M and trains BioCLIP with taxonomic structure, achieving stronger fine-grained classification performance and hierarchical representations. The authors identify limited trait-level representation as a direction for future scaling and richer textual descriptions.

  • Problem

    Biology computer-vision applications require task-specific labels and models, while existing datasets and pre-training methods lack sufficient scale, diversity, fine-grained labels, and taxonomic structure.

  • Method

    The paper curates TREEOFLIFE-10M and trains BioCLIP by matching images with taxonomic names through CLIP-style contrastive learning and mixed text-type training.

  • Results

    BioCLIP consistently outperforms CLIP and OpenCLIP across diverse fine-grained biology tasks, with average absolute improvements of 17% in zero-shot and 16% in few-shot settings.

  • Takeaways & Limitations

    BioCLIP learns image representations that conform to the taxonomic hierarchy and generalize across fine-grained biology classification tasks.

  • Takeaways & Limitations

    BioCLIP is fundamentally trained for classification and lacks richer species-appearance descriptions for extracting fine-grained trait-level representations.

Abstract

from arXiv · show

Images of the natural world, collected by a variety of cameras, from drones to individual phones, are increasingly abundant sources of biological information. There is an explosion of computational methods and tools, particularly computer vision, for extracting biologically relevant information from images for science and conservation. Yet most of these are bespoke approaches designed for a specific task and are not easily adaptable or extendable to new questions, contexts, and datasets. A vision model for general organismal biology questions on images is of timely need. To approach this, we curate and release TreeOfLife-10M, the largest and most diverse ML-ready dataset of biology images. We then develop BioCLIP, a foundation model for the tree of life, leveraging the unique properties of biology captured by TreeOfLife-10M, namely the abundance and variety of images of plants, animals, and fungi, together with the availability of rich structured biological knowledge. We rigorously benchmark our approach on diverse fine-grained biology classification tasks and find that BioCLIP consistently and substantially outperforms existing baselines (by 16% to 17% absolute). Intrinsic evaluation reveals that BioCLIP has learned a hierarchical representation conforming to the tree of life, shedding light on its strong generalizability. https://imageomics.github.io/bioclip has models, data and code.

1. Introduction

Computer vision is increasingly useful for biology, but existing workflows remain task-specific and labor-intensive. The paper addresses this gap with TREEOFLIFE-10M and BioCLIP, using biological taxonomy to support generalization across fine-grained taxa.

  • Biological images support tasks including species classification, individual identification, trait detection, and biodiversity monitoring.
  • Task-specific computer-vision workflows require biologists to label data and train suitable models for each taxa and task.
  • Existing biology datasets and pre-training strategies lack sufficient scale, diversity, fine-grained labels, or use of taxonomic structure.
  • TREEOFLIFE-10M contains over 10 million biology images covering 454 thousand taxa, compared with iNat21’s 2.7 million images covering 10 thousand taxa.
  • BioCLIP matches images with flattened taxonomic names through CLIP-style contrastive learning, supporting zero-shot and few-shot generalization to unseen taxa.
  • 17% zero-shot and 16% few-shot average absolute improvements over CLIP and OpenCLIP accompany BioCLIP’s hierarchical tree-of-life representations.

2. TREEOFLIFE-10M

TREEOFLIFE-10M combines large, diverse biology-image sources with canonicalized taxonomic metadata. Its scale and coverage address limitations in existing datasets, while label granularity remains uneven for some sources.

  • TREEOFLIFE-10M combines iNat21, 6.6 million Encyclopedia of Life images, and BIOSCAN-1M insect images to expand taxonomic and visual diversity.
  • The dataset aggregates images and canonicalizes labels across sources whose taxonomic hierarchies are noisy and inconsistent.
  • 98.6% of BIOSCAN-1M images are labeled only to the family level, while 22.5% include genus labels and 7.5% include species labels.
  • Over 10 million images span more than 450 thousand unique taxonomic names across the tree of life.
  • The dataset is publicly released with metadata, source links, and scripts, subject to primary-source licenses.

3. Modeling

BioCLIP adapts CLIP’s multimodal contrastive learning to biology’s hierarchical taxonomic label space. It represents organisms through taxonomic names and supports flexible text labels at inference time.

  • BioCLIP continually pre-trains CLIP on TREEOFLIFE-10M using multimodal contrastive learning.
  • Biology’s rich hierarchical label space provides signal for representing related taxa and generalizing to unseen species.A species absent from training may still benefit from learned representations of its genus or family.
  • CLIP-style learning encodes taxonomy by conditioning later taxonomic-rank representations on higher ranks rather than treating labels as unrelated class indices.
  • Text Types: A taxonomic name concatenates labels from kingdom through the distal-most taxonomic rank into one text string.
  • Text Types: BIOCLIP’s mixed text-type strategy randomly pairs each image with an available taxonomic, scientific, or common name during training.This retains taxonomic-name generalization benefits while supporting alternative names at inference.

4. Experiments

The experiments evaluate BIOCLIP across diverse biological classification settings, including unseen taxa, varied text types, alternative pre-training objectives, and low-data transfer. BIOCLIP outperforms the baselines and learns fine-grained taxonomic structure in its image representations.

  • Training and Evaluation Details: BIOCLIP is evaluated on 10 classification tasks spanning animals, plants, fungi, and protists across photographs, microscope images, drawings, and museum specimens.The tasks include eight Meta-Album datasets, Birds 525, and the RARE SPECIES task.
  • Can BIOCLIP Generalize to Unseen Taxa?: Using taxonomic names improves rare-species zero-shot accuracy over scientific names, with reported gains from 22.3 to 26.6 and from 28.0 to 30.4.The RARE SPECIES evaluation tests species excluded from training.
  • Can BIOCLIP Generalize to Unseen Taxa?: BIOCLIP substantially outperforms baseline CLIP models and an iNat21-trained CLIP model in zero-shot classification, especially on unseen taxa.The authors attribute its broad zero-shot performance to the diversity of classes in TREEOFLIFE-10M.
  • How Do Text Types Affect Generalization?: Mixed text-type pre-training retains taxonomic-name generalization benefits while supporting consistently strong performance across text types at inference.Using 1M TREEOFLIFE-10M examples also outperforms using 2.7M iNat21 examples, highlighting the role of data diversity.
  • Is the CLIP Objective Necessary?: The CLIP objective massively outperforms simple classification and hierarchical classification baselines in one-shot and five-shot evaluation.The hierarchical classification model outperforms simple classification and is comparable to the CLIP baseline.
  • Can BIOCLIP Classify More Than Species?: BIOCLIP’s mean one-shot accuracy is 9.1% higher than its zero-shot accuracy and it outperforms baselines on plant-disease classification.These results include both zero-shot and few-shot settings on PlantVillage and PlantDoc.
  • Does BIOCLIP Learn the Hierarchy?: BIOCLIP representations form richer and more separable fine-grained taxonomic clusters than CLIP, particularly at lower taxonomic ranks.The visualization uses held-out iNat21 validation images and taxonomic labels.

5. Related Work

Related work establishes multimodal contrastive learning, hierarchical computer vision, and fine-grained biological classification as foundations for BioCLIP. Prior findings also emphasize dataset diversity and image-caption alignment as key factors in vision representation quality.

  • Multimodal foundation model training data: CLIP and related multimodal models use contrastive objectives on web-scale image-text datasets, while later work emphasizes data diversity and image-caption semantic alignment.The cited scaling examples range from 100M-plus images to 6.6B training examples.
  • Hierarchy in computer vision: Hierarchy has been studied in computer vision through ImageNet’s WordNet taxonomy, model confusion patterns, hierarchical architectures, and alternative objectives.This work connects hierarchical structure to classification behavior and error severity.
  • Computer vision for biology: Biological images have long served as benchmarks for fine-grained classification and have supported contrastive, weakly supervised, and semi-supervised computer vision methods.Examples include bird-species classification and other biology-focused learning frameworks.

6. Conclusion

The paper introduces TREEOFLIFE-10M and BIOCLIP, then shows that BIOCLIP supports fine-grained biology classification and learns representations aligned with taxonomic hierarchy. The authors note that its classification-focused training leaves trait-level representation learning as future work.

  • BIOCLIP is a strong fine-grained biology classifier in both zero- and few-shot settings.
  • Using the entire taxonomic name improves generalization compared with other caption types, including on unseen species.The authors also visualize representations and find that BIOCLIP-embedded images better match the taxonomic hierarchy.
  • BIOCLIP efficiently learns visual representations over 450K taxa but is fundamentally trained to do classification.
  • Future work will scale the data and collect richer species-appearance descriptions for fine-grained trait-level representations.

Appendices

The appendices provide additional materials covering reproducibility, ethics, data aggregation, training, evaluation variability, predictions, text-type results, and generalized zero-shot learning.

  • The appendices contain reproducibility, ethics, training-data aggregation, training-detail, and hyperparameter materials.
  • Additional evaluation materials include few-shot standard deviations, zero-shot predictions, text-type results, and generalized zero-shot learning.

A. Reproducibility Statement

The authors release datasets, code, figure-generation tools, and pretrained weights to support reproducibility, while discussing ethical concerns and the intended role of BIOCLIP as an aid rather than a replacement for biologists.

  • Released resources include TREEOFLIFE-10M, RARE SPECIES, preprocessing, training, evaluation, and figure-generation code, plus pretrained model weights.Users with sufficient compute can reproduce the full pipeline, while pretrained weights support evaluation reproducibility with limited compute.
  • The paper provides permanent DOIs for TREEOFLIFE-10M, RARE SPECIES, BIOCLIP, and the code.
  • BIOCLIP does not include specific geographic information or animal conservation status during training, addressing stated concerns about aiding poachers.
  • The stated goal is for BIOCLIP to aid biologists rather than replace them, with users retaining understanding and context when applying it downstream.

C. Training Data Aggregation

TREEOFLIFE-10M aggregates images and labels from multiple biological data sources and constructs taxonomic and common-name mappings using prioritized metadata sources. Full hierarchy labeling is incomplete for some images, and some higher-level mislabels were discovered after training.

  • Images and labels are aggregated from iNat21, BIOSCAN-1M, and the Encyclopedia of Life.
  • Scientific-name mappings connect taxa to taxonomic hierarchies and common names using sources ordered by descending authority.Sources include BIOSCAN-1M metadata, EOL aggregate data, ITIS, iNaturalist, and the Global Names Resolver.
  • 84% of TREEOFLIFE-10M images received full taxonomic labeling, while 10% were labeled only to the family rank.For those family-level images, genus-species information is unavailable.
  • Some hemihomonyms were discovered to be mislabeled at higher taxonomic levels after training.

D. Hyperparameters & Training Details

The training details include a hierarchical classification objective alongside the BIOCLIP training setup. A vision transformer produces features, and separate taxonomic-rank heads are trained by summing cross-entropy losses.

  • Hyperparameters: Training hyperparameters are documented in Tables D1 and D2, including model-specific settings and validation-loss-selected epochs.The validation loss is evaluated with the CLIP objective on TreeOfLife-10M, including for TreeOfLife-1M models.
  • Implementation: The listed implementation uses a vision transformer, rank-specific linear layers, image batches, and hierarchical labels as inputs.The pseudocode defines the forward pass and computes logits for every rank before aggregating losses.
  • Hierarchical classification: The hierarchical classification model uses one linear head for each taxonomic rank and assigns each image seven hierarchical labels.The ViT produces dense image features, which each rank-specific head converts into logits.
  • Hierarchical classification: The training objective sums cross-entropy losses across the taxonomic-rank predictions.This objective trains image features to support classification at multiple taxonomic ranks.

E. Standard Deviation of Main Results

The reported evaluation results include run-to-run variation for few-shot experiments but deterministic zero-shot results. Example predictions illustrate BIOCLIP’s performance across diverse biology tasks and image conditions.

  • Variation across runs: Zero-shot results have no variation, whereas few-shot accuracies vary across five runs because training and test examples are randomly selected.The reported variation is described as small enough for the conclusions to hold.
  • Reported statistics: The evaluation tables report accuracy with standard deviation over five runs for animals, rare species, plants, and fungi.These tables summarize the variability of the corresponding test-set results.
  • Example predictions: BIOCLIP performs well across ten evaluation tasks, including out-of-distribution images and datasets mixing scientific and common names.Figures F1 and F2 show examples where BIOCLIP is correct and where CLIP is incorrect but BIOCLIP is correct.
  • Generalized zero-shot learning: In generalized zero-shot learning, BIOCLIP achieves 26.0% top-1 accuracy, compared with 23.0% for CLIP and 18.2% for OpenCLIP.The evaluation uses 800 labels combining seen and unseen species.
Loading 2311.18803v3…