Source-linked AI summary
BatchBench: Benchmarking AI Robustness to Batch Effects and Domain Shift in Biomedical Images - A JUMP Cell Painting Leaderboard
Sonja Topf, Ana Sanchez-Fernandez, Günter Klambauer
TL;DR
Batch effects shift biomedical-image distributions across batches and laboratories, while the effectiveness of domain adaptation on unseen biomedical conditions remains poorly characterized. BatchBench is a leaderboard for chemical-perturbation identification on JUMP-CP images from unseen sources, using a common interface that supports adaptation and non-adaptation methods. Test-time adaptation raises ResNet18 average overall accuracy from 0.175 without adaptation to 0.244 with AdaBN, with most gain coming from POSCON (0.465 to 0.646).
Problem
Batch effects shift biomedical-image distributions across batches and laboratories, while the effectiveness of domain adaptation on unseen biomedical conditions remains poorly characterized.
Source: §1 · pp. 1–2Method
Results
Test-time adaptation raises ResNet18 average overall accuracy from 0.175 without adaptation to 0.244 with AdaBN, with most gain coming from POSCON (0.465 to 0.646).
Source: §6 · p. 5Takeaways & Limitations
Identifying compounds with weak phenotypes and few training images on unseen sources remains the leaderboard’s main open problem.
Source: §1 · p. 2, §6 · p. 5Current entries remain close to chance for compounds with few training images, and none compensates for their weak phenotypes and limited examples.
Source: §6 · p. 5
Abstract
from arXiv · showhide
Despite recent advances, current AI systems remain highly susceptible to domain shifts, particularly in biomedical imaging, where images are usually acquired under substantially different experimental conditions. Although recent domain adaptation methods have made substantial progress, significant performance gaps under domain shift persist. To measure progress in this regard, we introduce BatchBench, a leaderboard that evaluates and compares the robustness of AI systems on biomedical images from unseen experimental batches and data sources. BatchBench is built on JUMP-CP, one of the largest publicly available biomedical imaging datasets. Generated using the Cell Painting assay, JUMP-CP exhibits pronounced batch effects and domain shifts, making it a challenging testbed for evaluating robustness to distributional changes.
1 Introduction
BatchBench benchmarks chemical-perturbation identification on Cell Painting images from unseen sources, addressing batch effects that can undermine models trained on other distributions. Its JUMP-CP-based leaderboard supports multiple adaptation approaches and reports baseline gaps, especially for compounds with few training images.
Vulnerability of AI systems under domain shift and batch effects
- Batch effects shift biological-image distributions across experimental batches, plates, and sites, and can limit AI use in applications such as drug discovery.A model that performs well on its training distribution can fail on images from a new batch or laboratory.Source: §1 · pp. 1–2
JUMP-CP and CellPainting
- JUMP-CP is a large public biomedical imaging dataset with source, batch, and plate annotations and pronounced shifts across those levels.It was obtained from 13 data-generating centers using different microscopes and laboratory workflows.Source: §1 · p. 2
The BatchBench Leaderboard
- BatchBench is an open leaderboard for comparing chemical-perturbation identification on Cell Painting images acquired from sources unseen during training.It provides standardized source-level evaluation on JUMP-CP with fixed splits and a private test set.Source: §1 · p. 2
- The benchmark accommodates methods that adapt to unlabeled test images and methods that do not through a common adapt() / predict() interface.It evaluates multi-source adaptation under combined covariate and label shift.Source: §1 · p. 2
- BatchBench tests whether pretrained image encoders and specialized architectures generalize to unseen imaging sites, a requirement not captured by in-distribution evaluation.JUMP-CP has substantial batch effects across multiple levels and data from 13 sources.Source: §1 · p. 2
- Six baseline entries show that test-time adaptation reduces the domain gap but does not close it, while current entries remain near chance for compounds with few training images.The leaderboard also includes standard supervised training, domain-invariant training, and meta-learning baselines.Source: §1 · p. 2
2 The BatchBench leaderboard
The leaderboard evaluates compound identification across unseen sources using multi-source adaptation, with source, batch, and plate metadata available to submissions. JUMP-CP-2k provides the dataset and fixed splits, including a private test set, while its evaluation covers compounds with different data availability and class distributions.
Problem setting
- Submissions perform multi-source adaptation to unseen target domains, with domain generalization, test-time adaptation, and in-context adaptation as special cases.Domains may be defined at the source, experimental-batch, or plate level.Source: §2 · p. 2
Task
- The task is to identify which of 2,000 compounds treated the cells in each single five-channel Cell Painting image.The channels are DNA, RNA, ER, AGP, and Mito.Source: §2 · p. 2
Problem setting and splits
- The test images come only from held-out sources, so each also comes from an unseen experimental batch and plate.Sources 4, 6, 9, and 13 form the private test set; source 2 is validation, and eight other public sources are for training.Source: §2 · p. 3
Compound selection and sampling
- JUMP-CP-2k was formed by selecting the 2,000 most frequent compounds and sampling one image per retained well.The source dataset contains more than 100,000 compound and genetic perturbations.Source: §2 · p. 3
Evaluation subsets
- The benchmark distinguishes POSCON, TARGET, and REST, which represent abundant strong-phenotype compounds, a curated mechanism-of-action set, and compounds with few images, respectively.REST compounds have about 12 images each in the public split.Source: §2 · p. 3
Label shift
- The public and test splits differ in class composition: POSCON falls from 51% to 36%, TARGET rises from 25% to 55%, and REST falls from 24% to 9%.This evaluates adaptation under combined covariate and label shift.Source: §2 · p. 3
3 Submitting to the leaderboard
Submissions provide training code and a Dockerfile for evaluation in a container, using a framework-independent interface. The server supplies unlabeled private test images and metadata, and ranks entries by overall accuracy while reporting subset accuracy separately.
Submission format
- A submission is a public Git repository with a commit hash, training code, and a Dockerfile used by the evaluation server to build a container.The container provides a Submission class implementing adapt() and predict().Source: §3 · p. 4
Evaluation
- The server supplies private test images and source, batch, and plate metadata without labels; adapt() may update the model using these unlabeled inputs before predict() returns one prediction per image.Methods that do not adapt implement adapt() as a no-op.Source: §3 · p. 4
Metric and ranking
- Entries are ranked by overall test accuracy, with separate POSCON, TARGET, and REST accuracies reported so progress on rare compounds remains visible.Because POSCON and TARGET together exceed 90% of test images, overall accuracy is dominated by those subsets.Source: §3 · p. 4
Rules
- Training, pre-training, and model selection may not use data from the held-out sources, including through pretrained models, but test-time adaptation on unlabeled test images is permitted.Source: §3 · p. 4
4 Development datasets
Two reduced JUMP-CP-2k variants support faster prototyping before leaderboard submission. Both retain the source-level split, while rankings use the full-resolution private test set.
4 Development datasets
- The two smaller variants are released to lower the entry barrier and enable fast prototyping before leaderboard submission.Source: §4 · p. 4
- JUMP-CP-MNIST includes eight positive-control compounds, downsampled images, and three combined channels for unmodified RGB architectures and pretrained models.Its five imaging channels are combined into three channels, and images are 200 × 200 pixels.Source: §4 · p. 4
- JUMP-CP-2k-200 contains all 2,000 compounds at 200 × 200 pixels with all five imaging channels.Source: §4 · p. 4
- Both variants use JUMP-CP-2k’s source-level split, but leaderboard rankings are computed on the full-resolution private test set.Source: §4 · p. 4
5 Baseline entries
The six baseline entries cover standard supervised training, domain-invariant training, test-time adaptation, and meta-learning, using a ResNet18 backbone with five input channels. Hyperparameters were selected using source 2 accuracy.
5 Baseline entries
- All six baseline entries use a ResNet18 backbone whose first convolution accepts five input channels.Source: §5 · p. 5
- The no-adaptation ResNet18 classifier uses empirical risk minimization on training sources and is applied unchanged to test sources.Source: §5 · p. 5
- DANN and deepCORAL encourage domain-independent representations through adversarial training or second-order feature-statistic alignment, respectively.Both use source identifiers as domain labels.Source: §5 · p. 5
- AdaBN replaces BatchNorm statistics with test-image statistics, while TENT also updates BatchNorm affine parameters by minimizing prediction entropy.Both adapt the trained ResNet18 at inference using test images grouped by plate.Source: §5 · p. 5
- ARM-BN trains episodically with BatchNorm statistics from an unlabeled context batch and computes statistics from unlabeled test images at inference.The passage also lists CS-ARM-BN.Source: §5 · p. 5
- Hyperparameters were tuned by Bayesian optimization, and each method’s model was selected by highest accuracy on source 2.Source: §5 · p. 5
6 Results
On the private test set, test-time adaptation improves overall accuracy over the unchanged ResNet18 baseline, while the domain gap remains for compounds with few images. The leaderboard identifies weak phenotypes and few training images on unseen sources as its main open problem.
6 Results
- 0.244 overall accuracy with AdaBN exceeds 0.175 for non-adapted ResNet18; POSCON accuracy rises from 0.465 to 0.646.TENT improves POSCON further but not TARGET or REST; DANN reaches 0.170.Source: §6 · p. 5, Table 2 · p. 5
- POSCON performance is substantially higher than TARGET and REST, and none of the current entries compensates for the latter groups’ weaker phenotypes and fewer training images.The passage identifies compounds with weak phenotypes and few training images on unseen sources as the leaderboard’s main open problem.Source: §6 · p. 5