Source-linked AI summary
BioBench: A Blueprint to Move Beyond ImageNet for Scientific ML Benchmarks
Samuel Stevens
TL;DR
ImageNet-1K accuracy is widely used to judge visual representations, but it does not reliably predict performance on scientific imagery. BioBench brings nine ecology tasks into a shared benchmark, and ImageNet accuracy explains only 34% of BioBench performance variance across 46 checkpoints.
Problem
Scientific imagery differs from web photographs in content, scale, and acquisition method, while ecological tasks often involve fine-grained, long-tailed distinctions.
Source: §1 · pp. 1–2Method
BioBench consolidates nine application-driven ecology tasks across four kingdoms and six image distributions, using frozen embeddings and lightweight probes for evaluation.
Source: §1 · p. 2Results
Across 46 checkpoints, ImageNet-1K top-1 accuracy explains only 34% of BioBench variance.
Source: §3 · p. 4Takeaways & Limitations
BioBench offers a practical guide for ecological workflows and a template for building grounded benchmarks in other sciences.
Source: §5 · p. 5Because BioBench evaluates frozen features, it may underestimate gains from task-specific fine-tuning.
Source: §5 · p. 5
Abstract
from arXiv · showhide
ImageNet-1K linear-probe transfer accuracy remains the default proxy for visual representation quality, yet it no longer predicts performance on scientific imagery. Across 46 modern vision model checkpoints, ImageNet top-1 accuracy explains only 34% of variance on ecology tasks and mis-ranks 30% of models above 75% accuracy. We present BioBench, an open ecology vision benchmark that captures what ImageNet misses. BioBench unifies 9 publicly released, application-driven tasks, 4 taxonomic kingdoms, and 6 acquisition modalities (drone RGB, web video, micrographs, in-situ and specimen photos, camera-trap frames), totaling 3.1M images. A single Python API downloads data, fits lightweight classifiers to frozen backbones, and reports class-balanced macro-F1 (plus domain metrics for FishNet and FungiCLEF); ViT-L models evaluate in 6 hours on an A6000 GPU. BioBench provides new signal for computer vision in ecology and a template recipe for building reliable AI-for-science benchmarks in any domain. Code and predictions are available at https://github.com/samuelstevens/biobench and results at https://samuelstevens.me/biobench.
1 Introduction
Scientific images differ from the web photographs underlying general vision benchmarks in content, scale, and acquisition method. BioBench consolidates ecology tasks into a shared benchmark after ImageNet-1K accuracy proved an unreliable predictor of ecological performance.
1 Introduction
- Scientific imagery differs from web photographs in content, scale, and acquisition method, including radiographs, micrographs, camera-trap images, and specimen imagery.Ecological tasks also involve fine-grained, long-tailed distinctions that ImageNet’s 1,000 classes represent poorly.Source: §1 · pp. 1–2
- Across three ecology tasks, ImageNet ranking correlation falls below 0.25 once models exceed 75% ImageNet top-1 accuracy.The tasks cover long-tail species identification, drone-video behavior recognition, and specimen-trait inference.Source: §1 · p. 1
- BioBench aims to test whether application-driven benchmarks restore predictive power and to provide a template for other scientific domains.The proposal responds to fragmented datasets and changing checkpoints, hyperparameters, and evaluation scripts that obscured cross-task comparisons.Source: §1 · p. 2
- BioBench consolidates nine application-driven ecology tasks spanning four kingdoms and six image distributions into one benchmark.The corpus contains 3.1 million images and includes drone footage, web video, micrographs, in-situ and specimen photos, and camera-trap frames.Source: §1 · p. 2
- ImageNet-1K explains only 34% of BioBench variance across 46 checkpoints, and mis-ranks models above 75% accuracy 30% of the time.Overall rank concordance is ρ = 0.55; above 75% ImageNet accuracy, it drops to ρ = 0.42.Source: §1 · pp. 2–3
2 Benchmark Suite & Protocol
BioBench is designed to represent ecological diversity, long-tail class balance, and both general capability and operational utility. It combines public tasks with a uniform frozen-embedding evaluation protocol.
2 Benchmark Suite & Protocol
- BioBench addresses ecological diversity, natural class imbalance, varied task types, and both proxy-driven and mission-driven evaluation.Its design also calls for confidence intervals, significance testing, and rank-stability analysis to distinguish performance differences from benchmark lottery effects.Source: §2 · p. 3
- ImageNet lacks ecological diversity, while iNat2021 offers taxonomic breadth but limited task variety and no mission-driven evaluation.The authors report that ImageNet’s predictive power for ecological performance drops once models exceed 75% accuracy.Source: §2 · p. 3
- The suite consolidates nine public tasks across four kingdoms and six image regimes, totaling 3.1 million images.Tasks span species identification, individual re-identification, behavior classification, and functional-trait prediction.Source: §2 · pp. 3–4
- Table 1 distinguishes ecological tasks from general vision benchmarks by dimensions including mission status, image context, and classification target.Mission status marks whether a task serves a specific ecological application; context distinguishes organisms in natural settings from preserved specimens.Source: Table 1 · p. 4
- Models provide frozen image embeddings, then linear or logistic probes are fitted per task; macro-F1 is the default metric, with task-specific metrics for FishNet and FungiCLEF.The protocol also bootstraps confidence intervals and aims to isolate representation quality from task-specific engineering.Source: §2 · p. 4
3 Benchmark Results
Across evaluated checkpoints, ImageNet-1K accuracy has limited predictive alignment with BioBench performance. Only three named model families set new BioBench state-of-the-art scores over time.
3 Benchmark Results
- Across 46 checkpoints, ImageNet-1K top-1 accuracy explains only R2 = 0.34 of BioBench variance, with overall rank concordance ρ = 0.55.At the frontier, the ImageNet-preferred model is worse on BioBench roughly 30% of the time.Source: §3 · p. 4
- Only CLIP, SigLIP, and SigLIP 2 set new BioBench state-of-the-art scores among the released generalist models considered over time.The paper reports BioBench scores over time in Figure 3.Source: §3 · p. 4
4 Related Work
Prior benchmarks and isolated datasets do not capture the range of challenges in ecological monitoring. BioBench pairs consistent evaluation protocols with a broad ecology-focused suite.
4 Related Work
- General vision and transfer benchmarks have little ecological content and miss fine-grained taxonomy, environmental variability, and long-tailed species distributions.The paper also notes that general-domain benchmarks can be fragile under distribution shift and long-tail structure.Source: §4 · p. 4
- Existing ecology datasets are fragmented across task types: iNaturalist focuses on species classification, Pl@ntNet on plants, and iWildCam on camera-trap imagery.The cited comparison notes that these efforts do not jointly cover temporal behavior and ecological trait prediction.Source: §4 · p. 5
- Conservation practitioners lack systematic guidance on which vision architectures transfer best across interconnected ecological-monitoring tasks.The paper presents BioBench as addressing this need with a unified benchmark.Source: §4 · p. 5
- The majority of new models fail to improve on BioBench scores over time.Figure 3 tracks BioBench scores over time.Source: Figure 3 · p. 5
- A single embedding API, class-balanced metrics, and bootstrap uncertainty apply consistent evaluation protocols across tasks.The cited methodological work supports the importance of consistent protocols.Source: §4 · p. 5
5 Limitations & Future Work
BioBench identifies scope, probing, and metric boundaries, while offering its ecology benchmark as a reproducible recipe and a template for other sciences.
5 Limitations & Future Work
- BioBench focuses on ecology; medicine and manufacturing may require different tasks, such as detection, segmentation, or calibration.Source: §5 · p. 5
- Frozen-feature probing isolates representation quality but underestimates gains from task-specific fine-tuning.Source: §5 · p. 5
- Macro-F1 rewards tail performance, though some applications prefer operating-point metrics such as precision@recall.Source: §5 · p. 5
- BioBench is presented as a minimal, reproducible evaluation recipe and a template for benchmarks in other sciences.Source: §5 · p. 5