PaperlayerView exact revision on arXiv

Source-linked AI summary

BioCLIP 2: Emergent Properties from Scaling Hierarchical Contrastive Learning

Jianyang Gu, Samuel Stevens, Elizabeth G Campolongo, Matthew J Thompson, Net Zhang, Jiaman Wu, Andrei Kopanev, Zheda Mai, Alexander E. White, James Balhoff, Wasila Dahdul, Daniel Rubenstein, Hilmar Lapp, Tanya Berger-Wolf, Wei-Lun Chao, Yu Su

arXiv:2505.23883v2cs.CVcs.CLcs.LG

TL;DR

This paper asks what capabilities emerge when hierarchical contrastive training for biological vision is scaled. It curates TREEOFLIFE-200M and trains BIOCLIP 2, finding generalization beyond species recognition alongside ecologically aligned and variation-preserving embeddings. These properties grow more pronounced with scale, though the dataset remains imbalanced in taxonomic coverage and image type.

  • Problem

    It is unclear what properties emerge when hierarchical contrastive training for biological vision is scaled beyond species classification.

    Source: §1 · p. 2

  • Method

    The authors curate TREEOFLIFE-200M and train BIOCLIP 2 with hierarchical taxonomic labels and large-scale contrastive vision-language learning.

    Source: §1 · p. 2

  • Results

    BIOCLIP 2 outperforms vision-language and vision-only baselines by an average margin of 10.3% across biological visual tasks, while larger-scale training improves ecological alignment and separation of intra-species variants.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Species-level supervision can yield embeddings that align species with ecological relationships and preserve intra-species variation in orthogonal subspaces.

    Source: §1 · p. 2

    TREEOFLIFE-200M is imbalanced in both taxonomic coverage and image type, with a long-tailed distribution across taxa.

    Source: A Limitations · p. 15

Abstract

from arXiv · show

Foundation models trained at scale exhibit remarkable emergent behaviors, learning new capabilities beyond their initial training objectives. We find such emergent behaviors in biological vision models via large-scale contrastive vision-language training. To achieve this, we first curate TreeOfLife-200M, comprising 214 million images of living organisms, the largest and most diverse biological organism image dataset to date. We then train BioCLIP 2 on TreeOfLife-200M to distinguish different species. Despite the narrow training objective, BioCLIP 2 yields extraordinary accuracy when applied to various biological visual tasks such as habitat classification and trait prediction. We identify emergent properties in the learned embedding space of BioCLIP 2. At the inter-species level, the embedding distribution of different species aligns closely with functional and ecological meanings (e.g., beak sizes and habitats). At the intra-species level, instead of being diminished, the intra-species variations (e.g., life stages and sexes) are preserved and better separated in subspaces orthogonal to inter-species distinctions. We provide formal proof and analyses to explain why hierarchical supervision and contrastive objectives encourage these emergent properties. Crucially, our results reveal that these properties become increasingly significant with larger-scale training data, leading to a biologically meaningful embedding space.

1 Introduction

The paper asks what emerges when hierarchical contrastive training is scaled, and answers by curating TREEOFLIFE-200M and training BIOCLIP 2. Beyond stronger species classification, the model generalizes to biological tasks and develops ecologically aligned, variation-preserving embeddings.

1 Introduction

  • BIOCLIP 2 improves species classification accuracy by 18.0% over BIOCLIP and outperforms vision-language and vision-only baselines by an average margin of 10.3% across diverse biological visual tasks.Evaluated tasks include habitat classification, trait prediction, new-species identification, and agricultural disease detection.Source: §1 · p. 2
  • At the inter-species level, BIOCLIP 2 embeddings align with ecological relationships, arranging Darwin’s finches by beak size without explicitly labeled traits.The paper attributes this alignment to hierarchical taxonomic labels that encode functional and ecological information.Source: §1 · p. 2
  • At the intra-species level, life-stage and sex variations remain distinct in subspaces orthogonal to species differences, despite not being encoded in taxonomic labels.The paper theoretically proves that, when species prototypes are nearly orthogonal, contrastive training prioritizes this orthogonality over raw magnitude; the separation supports attribute recognition.Source: §1 · p. 2
  • Larger-scale training improves both inter-species ecological alignment and intra-species variant separation, yielding a more interpretable and biologically meaningful embedding space.The paper presents these scale-amplified properties as emergent behaviors from domain-specific scaling with structured supervision.Source: §1 · p. 2

2 Related Work

Prior work studies emergent capabilities in foundation models and biological vision, including hierarchical representations and disentangled subspaces. This paper situates its investigation at the intersection of those themes and computer vision challenges in ecology and evolutionary biology.

2 Related Work

  • Emergent properties in foundation models are capabilities implicitly acquired through training and generalized beyond the initial training objective.Related examples include in-context learning in language models and semantic segmentation in vision models.Source: §2 · pp. 2–3
  • Earlier CLIP studies report visual-semantic hierarchies and disentangled subspaces, while this work investigates emergence when hierarchical labels supervise species distinctions.The cited prior work does not explicitly present hierarchical semantic structures as training supervision in the same way.Source: §2 · p. 3
  • Ecology and evolutionary biology pose vision challenges involving long-tailed distributions, fine-grained classes, and varied image distributions, addressed through tasks such as trait prediction and plant disease detection.Biological vision models have also scaled organism datasets, specialized in camera traps, or incorporated additional modalities.Source: §2 · p. 3

3 TREEOFLIFE-200M

TREEOFLIFE-200M combines broad biological image sources to expand dataset scale and taxonomic coverage. The paper describes its provider composition and filtering, then reports coverage of threatened species and additional representation of underrepresented clades.

3 TREEOFLIFE-200M

  • TREEOFLIFE-200M contains nearly 214 million images representing 952,257 taxa, combining GBIF, EOL, BIOSCAN-5M, and FathomNet with prior sources to build a large, diverse public dataset.The paper contrasts its breadth with TREEOFLIFE-10M and BioTrove, which has 162 million images but does not match TREEOFLIFE-10M’s biodiversity.Source: §3 · p. 3

3.1 Images

  • The dataset draws on GBIF and EOL aggregations alongside expert-annotated BIOSCAN-5M insect images and FathomNet images of ocean animals.Together, these sources cover varied sources, clades, and habitats.Source: §3.1 · p. 4

3.2 Data Curation & Filtering

  • Curation begins with 222,065,140 images and 1,359,405 taxonomic hierarchies, then targets label alignment, image quality, duplication, and data leakage.TaxonoPy resolves labels to a seven-rank Linnaean hierarchy and common name; image screens remove non-organismal content, and hashes are used to remove duplicates and test-image leakage.Source: §3.2 · p. 4

3.3 Taxa Coverage

  • TREEOFLIFE-200M contains nearly 868,000 taxa labeled to species level and represents 77.1% of threatened species on the IUCN Red List assessment.That coverage includes 36,370 threatened species.Source: §3.3 · p. 4
  • TREEOFLIFE-200M adds 1,102 Cephalopoda taxa largely absent from TREEOFLIFE-10M, and its 55,085 Fungi taxa are close to four times TREEOFLIFE-10M’s 14,793.The paper reports a 42.9% absolute improvement over BIOCLIP on the zero-shot Fungi benchmark; comparisons use taxa with non-null species labels because not all images have full taxonomic labels.Source: §3.3 · pp. 4–5

4 BIOCLIP 2 and Species Classification

BIOCLIP 2 builds on hierarchical multimodal contrastive training and expands both the biological dataset and model capacity. It achieves substantial species-classification gains over prior models.

4 BIOCLIP 2 and Species Classification

  • BIOCLIP uses hierarchical labels—including taxonomic labels, scientific names, and common names—in multimodal contrastive training.Unlike one-hot labels, taxonomic labels encode biological information at multiple levels.Source: §4 · p. 5
  • BIOCLIP 2 scales model capacity with a larger ViT-L/14 pretrained on LAION-2B and interleaves a portion of LAION-2B CLIP training data with species contrastive learning; ablations find experience replay improves biological understanding and performance across diverse tasks.The auxiliary replay mechanism is introduced to maintain general-domain understanding for broader applications.Source: §4 · p. 5
  • Training used 214M organism images with hierarchical labels and 26M randomly sampled LAION-2B image-text pairs for 30 epochs.The run used 32 NVIDIA H100 GPUs for 10 days.Source: §4 · p. 5

4.1 Species Classification Performance

  • BIOCLIP 2 surpasses the second-best model by 16.1% on average and improves 30.1% over the original CLIP model.It also shows over 20% zero-shot improvement on Camera Trap, Fungi, and Rare Species.Source: §4.1 · p. 6

5 Emergent Properties from Scaling Hierarchical Contrastive Learning

BIOCLIP 2 transfers beyond species recognition, and larger-scale training strengthens that transfer. Its embedding space also aligns species with ecological relationships while preserving within-species variation in increasingly orthogonal, separable directions.

5.1 Beyond Species Classification

  • BIOCLIP 2 improves average performance by 14.8% over original CLIP and has an 8.9% performance gap over DINOv3 on biological visual tasks beyond species classification.The evaluated tasks include habitat classification, trait prediction, new-species identification, and agricultural disease detection.Source: §5.1 · p. 6

5.2 Scaling Trends

  • Performance on five non-species tasks consistently improves as hierarchical training data scales from 1M to 214M samples.In AwA2, the 1M model underperforms the baseline, while performance improves as data scale increases.Source: §5.2 · p. 7

5.3 Emergent Properties and Qualitative Analysis

  • Larger training sets progressively separate freshwater from non-freshwater fish, while within-species life-stage variations remain distinct and lie in subspaces roughly orthogonal to species differences.In the visualizations, species clusters tighten as scale increases, without collapsing the intra-species variations.Source: §5.3 · p. 7

5.4 Formal Analysis

  • As training scales up, intra-species variations become increasingly orthogonal to species differences and more separable.The explained-variance ratio tracks the variation captured by the species span; its trend supports increasing orthogonality, while Fisher Discriminant Ratio trends support increasing separation.Source: §5.4 · pp. 8–9

6 Ablation Study and Analysis

Ablations indicate that hierarchical contrastive supervision combines the benefits of taxonomic labels and contrastive learning, while larger model capacity and experience replay also improve performance.

Ablation Study

  • Using scientific names without taxonomic labels lowers FishNet performance by 1.3%, while hierarchical contrastive supervision yields the best overall benchmark performance.The study uses TREEOFLIFE-10M with ViT-L/14 as its testbed.Source: §6 · p. 9
  • Replacing contrastive training with cross-entropy on one-hot labels leads to inferior performance on all benchmarks and is optimization-heavy for a 952K-class classifier.Contrastive training still separates species and intra-species variations when trained solely with scientific names.Source: §6 · p. 9
  • Increasing model capacity improves performance across all benchmarks, while experience replay improves species classification and some other visual tasks.These changes were ablated using TREEOFLIFE-10M as a testbed.Source: §6 · p. 9

7 Conclusion

The paper concludes that scaling domain-specific data with structured supervision yields biological embeddings that capture ecological relationships and preserve within-species variation. It also reports generalization beyond the initial training objectives.

7 Conclusion

  • BIOCLIP 2 is trained with hierarchical taxonomic labels on TREEOFLIFE-200M, the paper’s largest and most diverse biological organism dataset to date.The authors report state-of-the-art accuracy on species classification.Source: §7 · p. 9
  • The learned embeddings align species with ecological relationships while preserving and separating appearance variations within species.The authors conclude that domain-specific scaling and structured supervision lead to effective generalization beyond the initial training objectives.Source: §7 · p. 9

Appendix

  • The appendix covers the theorem proof, training implementation, baselines, additional empirical observations, benchmarks, and dataset processing.It also discusses limitations, broader impacts, neural collapse, and author contributions.Source: Appendix · p. 15

A Limitations

The limitations are theoretical and data-related: scale-linked separation of within-species variation is observed but not formally proved, and TreeOfLife-200M is imbalanced across taxa and image types. The broader-impact discussion notes potential misuse while emphasizing that the data omit geotags.

A Limitations

  • Greater separation of intra-species variation with scale is observed empirically, but only its preservation in orthogonal subspaces is formally proved.The authors identify formal analysis of scale-dependent separation as future work.Source: A Limitations · p. 15
  • TreeOfLife-200M has long-tailed taxonomic coverage, even though plants and animals are balanced at the kingdom level.The authors note that described animal species outnumber described plant species.Source: A Limitations · p. 15
  • Image sources create uneven representation: citizen-science images dominate, while camera-trap images are few and generally capture animals large enough to trigger sensors.GBIF contributes 151M citizen-science images and 617.8K camera-trap images; citizen-science images skew toward charismatic species and plants.Source: A Limitations · p. 15
  • Common groups have millions of images, but microscopic organisms such as bacteria and viruses have a dozen or fewer.Dataset-wide scale therefore does not mean even coverage across organisms.Source: A Limitations · p. 15

B Broader Impacts

  • The dataset may support threatened-species recognition, while omitting geotags that could expose precise locations and increase poaching vulnerability.The paper identifies disclosure of precise location information, rather than improved classification, as the primary risk to endangered species.Source: B Broader Impacts · pp. 15–16

C Proof of Theorem 5.1

The proof analyzes contrastive loss around a species prototype plus an intra-species residual. Under its stated separation conditions, first-order effects vanish and orthogonal residuals do not interfere with contrastive optimization.

C Proof of Theorem 5.1

  • The proof starts with contrastive loss for a visual embedding z of class s paired with its corresponding text embedding c_s.The loss compares the matched pair against other class text embeddings.Source: C Proof of Theorem 5.1 · p. 16
  • Assuming the representation is near species prototype μ_s, residual δ represents intra-species variation and the loss is expanded around it.The derivation uses a second-order Taylor expansion and collects first-order terms.Source: C Proof of Theorem 5.1 · pp. 16–17
  • When w_s≈1 and species embeddings are better separated, the first-order terms vanish as the weighted prototype mean m approaches μ_s.Source: C Proof of Theorem 5.1 · p. 17
  • Because the Hessian lies in the span of species prototypes, residuals orthogonal to species differences do not interfere with overall contrastive optimization.Source: C Proof of Theorem 5.1 · p. 17

D Training Implementation Details

The appendix specifies BIOCLIP 2 training and ablation settings, describes the evaluation baselines, and introduces further empirical observations. The supplied passages identify the relevant tables and comparison scope but do not include their hyperparameter values or additional numerical results.

D Training Implementation Details

E Baseline Details

  • Evaluations compare BIOCLIP 2 with general and biology-specific baselines, with vision-only models restricted to few-shot and non-species classification tasks.The listed baselines include CLIP, SigLIP, Supervised-IN21K, DINOv3, BioTrove-CLIP, and BIOCLIP.Source: E Baseline Details · pp. 17–18

F More Empirical Observations

F.1 CLIP Training Data Replay

As training scale increases, BioCLIP 2’s embeddings show stronger biological structure: species align with phylogeny and traits, while within-species differences remain separable. Replay settings affect benchmark performance differently, and the camera-trap evaluation uses balanced test sets across five datasets.

F.1 CLIP Training Data Replay

  • Replay combines hierarchical taxonomic contrastive supervision with 10%–20% as many LAION-2B samples as biological images, using a separate visual projector for replay data.The largest run uses 26M replay samples alongside 214M biological images; both data types share the visual backbone and text encoder.Source: §F.1 · p. 18
  • Replay results vary by benchmark: no replay yields a 5.2% performance improvement, shared-projector replay improves Herb. 19 and PlantDoc by more than 2% but lowers Rare Species accuracy by 1.3%, and separate-projector replay improves INQUIRE-Rerank by 2.1%.The authors attribute inconsistent shared-projector results to the distribution gap between biological taxonomic labels and replay text labels; separate-projector replay improves performance across multiple benchmarks, while replay has limited influence on AwA2.Source: §F.1 · pp. 18–19

F.2 Qualitative Analyses

  • Larger-scale training separates Darwin’s finch groups in ways that align with phylogeny and beak-related function, preserves sex variation orthogonal to species differences, and makes species and disease classes easier to distinguish in PlantDoc.In PlantDoc, healthy apple embeddings also lie close to healthy blueberry embeddings.Source: §F.2 · p. 19

F.3 The decay of FDR numerator/denominator along the data scale.

  • From 1M to 214M training samples, the FDR denominator decays more than its numerator, supporting the observed increase in FDR for life-stage and sex classes.Figure 10 plots the numerator and denominator decay curves for life-stage and sex data.Source: §F.3 · p. 20, Figure 10 · p. 21

F.4 Camera Trap Results

  • The IDLE-OO Camera Traps benchmark draws on five LILA-BC datasets, excludes noisy image-level labels, and samples species-level classes to create balanced test sets.The Island Conservation set uses common names; the other datasets are evaluated on scientific names.Source: §F.4 · pp. 20–21

G Discussion on the Relationship with Neural Collapse

BioCLIP 2 preserves within-species variation rather than exhibiting neural collapse. The discussion relates this outcome to taxonomic text prototypes and a very large label space, while the accompanying sections describe evaluation benchmarks and data-processing procedures.

G Discussion on the Relationship with Neural Collapse

  • BioCLIP 2 preserves intra-species variation in subspaces orthogonal to inter-species differences, so it does not exhibit neural collapse as defined here.The authors identify two relevant conditions: taxonomic text embeddings encode hierarchy and do not become a simplex ETF, while 952K classes exceed the 768-dimensional embedding space.Source: G Discussion on the Relationship with… · p. 21

H Biological Visual Evaluation Details

  • Evaluation uses frozen visual embeddings with standard classifiers to assess representation quality without the influence of complicated optimization loops.FishNet tests habitat and ecological-role prediction; NeWT averages accuracy across 164 natural-world binary tasks; AwA2 tests attribute transfer to five unseen classes; Herb. 19 uses semi-supervised clustering; PlantDoc uses one-shot support examples.Source: H Biological Visual Evaluation Details · pp. 22–23
  • Life-stage and sex benchmarks separately test variant differentiation and cross-variant species classification, using juvenile-to-adult and female-to-male train/test directions, respectively.Both benchmark types use support vector classifiers.Source: H Biological Visual Evaluation Details · p. 23

I.1 Taxonomic Standardization

  • Taxonomic standardization iteratively matches query responses and input hierarchies to predefined profiles; when no match succeeds, the original label is retained.After standardization, approximately 92.26% of the 214M images retain species-level labels.Source: §I.1 · pp. 23–24, Table 9 · p. 24

I.2 Image-quality Screens

  • Downloaded images were checked for a shortest side of at least 224 pixels and resized so the longest side did not exceed 720 pixels.The image-quality-screen code and further processing details are provided in the dataset repository.Source: §I.2 · p. 25

I.2.1 Museum Specimen Processing

  • Museum images can include metadata, duplicate views, multiple specimens, or less informative images under one occurrence; processing groups them by collection area and image type, then classifies them using curated support-set cluster centers.The support examples are embedded with CLIP, not BioCLIP, because museum specimen labels were not filtered from BioCLIP’s training data.Source: §I.2.1 · p. 25

I.2.2 Camera Trap Images

  • To reduce camera-trap label noise from long sequences containing multiple taxa, the dataset retains only occurrences with 15 images or fewer and filters empty frames with MegaDetector.Some occurrences contain up to 10,000 images with a single taxonomic label applied across them.Source: §I.2.2 · p. 25

I.2.3 Citizen Science Images

  • Citizen-science occurrences with mixed-species labels are discarded, same-species occurrences are sampled down to five images, and likely camera-trap images follow camera-trap processing; identifiable people are also filtered out.The processing identifies occurrence types using standardized image embeddings and mean pairwise cosine similarity.Source: §I.2.3 · p. 26

I.3 Duplicate and Leakage Control

The data pipeline uses content-based deduplication to reduce benchmark leakage. The dataset also covers many conservation-prioritized species, and the author contributions span analysis, data curation, tooling, and model design.

I.3 Duplicate and Leakage Control

  • MD5 hashes remove exact duplicates, while PDQ perceptual hashes identify near-duplicates that could otherwise remain in training despite small image changes.PDQ filtering is limited to GBIF citizen-science images and EOL images sourced from Flickr, where it is considered reliable.Source: §I.3 · pp. 26–27
  • TREEOFLIFE-200M contains images of 77.1% of threatened species assessed by IUCN, including 79.7% of Vulnerable, 79.4% of Endangered, and 68.4% of Critically Endangered species.Coverage is 48.4% for data-deficient species; the authors note that only 8% of described species have been evaluated by IUCN.Source: §I.3 · pp. 27–28

J Author Contribution Statement

  • Contributions include dataset cleaning and curation, taxonomic standardization, benchmark and deduplication development, experiment analysis, and guidance on training design.The listed roles include face-detection filtering, camera-trap test-set curation, and development of the PDQ-hash deduplication pipeline.Source: J Author Contribution Statement · p. 28
Loading 2505.23883v2…