PaperlayerView exact revision on arXiv

Source-linked AI summary

Same Encoder, Different Winner: A Paired-View Framework for Cell Painting Encoder Evaluation

Tim Treis, Nikita Moshkov, Johan Fredin Haslum, Shantanu Singh, Fabian J. Theis

arXiv:2609.12761v1cs.CVcs.LGq-bio.QM

TL;DR

Whether Cell Painting evaluation protocols agree—and what each measures—has not been established. CP-BG-Bench compares matched views of the same cells across datasets and protocols, finding systematic ranking disagreements along three interpretable axes.

  • Problem

    It is not established whether the small set of commonly used Cell Painting evaluation metrics agree or what each individually measures.

    Source: §1 · p. 2

  • Method

    CP-BG-Bench holds each cell centre fixed across raw, segmented, and density-augmented views, then evaluates encoders on three datasets using four community-standard protocols.

    Source: §1 · p. 3

  • Results

    The four protocols rank the same encoders systematically differently, with disagreements along cell versus background, morphology versus context, and within-study versus across-batch axes.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Single-metric encoder rankings should be treated as tentative; the C-to-S replicate-mAP gap is a useful diagnostic of signal that may not transfer across acquisition batches.

    Source: §4 · p. 8

    The three datasets differ simultaneously in perturbation modality, cell line, replication depth, and batch structure, preventing attribution of C-to-S gap differences to any single factor.

    Source: §4 · p. 9

Abstract

from arXiv · show

Vision encoders for Cell Painting are typically ranked by a single evaluation, commonly replicate mean average precision (mAP). We introduce CP-BG-Bench, a paired-view evaluation framework that holds the central cell fixed across four matched views (raw crop C, segmented S, and density-augmented variants CD and SD), ablating or augmenting surrounding pixels as a controlled intervention. Instantiating the framework on three datasets (JUMP-CP, RxRx1, RxRx3-core) and three encoders (DINOv3 ViT-B/16, OpenPhenom, SubCell) under four community-standard protocols (replicate mAP, scIB batch integration, CellProfiler feature prediction, cross-batch perturbation recall), we find that the four protocols rank the same encoders systematically differently, with disagreements decomposing along three axes: cell versus background, morphology versus context, and within-study versus across-batch. The largest effect: on RxRx3-core, SubCell with segmented inputs retains 94% of crop replicate mAP but only 32% of crop R@10, so the within-study signal preserved under segmentation is largely non-transferable; density augmentation recovers 84% of the within-study C-to-S gap but only 8% of the cross-batch gap. Segmented views predict CellProfiler features as well as or better than crops on two of three datasets, inverting the replicate-mAP ranking, and the C-to-S gap varies by an order of magnitude across datasets while remaining similar across encoders, indicating that background-driven gain is set by experimental design rather than by the encoder. Single-metric ranking of Cell Painting encoders is therefore sensitive to the protocol used, and protocol disagreements are interpretable as projections onto the three axes the paired-view design exposes. We will release the paired-view datasets, reconstruction pipelines, 36 trained checkpoints, aggregated embeddings, and the full evaluation suite.

1 Introduction

CP-BG-Bench holds each central cell fixed while varying background and density, enabling controlled comparisons across evaluation protocols. The protocols rank encoders differently, with disagreements spanning cell versus background, morphology versus context, and within-study versus across-batch.

1 Introduction

  • Cell Painting uses multiplexed fluorescent dyes to profile cellular changes, and vision encoders are developing against this evaluation backdrop.Earlier workflows segmented cells and extracted hand-crafted features with tools such as CellProfiler.Source: §1 · pp. 1–2
  • The community commonly evaluates encoders with replicate mAP or related metrics, but whether these metrics agree or what each measures remains unsettled.Cell Painting encoders now routinely match or exceed classical hand-crafted features on standard benchmarks.Source: §1 · p. 2
  • Existing Cell Painting datasets entangle cell, neighbour, density, and background signals, while prior observations leave contextual cues difficult to distinguish from within-batch artifacts.Cell count alone can predict bioactivity above chance, and context-containing crops have outperformed tightly cropped cells on retrieval benchmarks.Source: §1 · p. 2
  • CP-BG-Bench provides four identity-matched views that toggle background context and add an explicit local-density signal while holding the central cell fixed.The framework is deployed on JUMP-CP, RxRx1, and RxRx3-core with DINOv3 ViT-B/16, OpenPhenom, and SubCell, using four community-standard evaluation protocols.Source: §1 · p. 3
  • Four protocols rank encoders differently across three axes; RxRx3-core SubCell S retains 94% of C replicate mAP (0.716 vs. 0.760) but 32% of C R@10 (0.100 vs. 0.313).Density augmentation recovers 84% of the within-study C-to-S gap but only 8% of the cross-batch gap.Source: §1 · p. 3
  • The paper’s methodological contribution includes paired-view datasets and reconstruction pipelines, 36 trained checkpoints, aggregated embeddings, and the full evaluation suite.The source images for RxRx1 and RxRx3-core are not redistributed to comply with their originating licences.Source: §1 · p. 3

2 Results

Across the three datasets, the results compare background-sensitive crop views with segmented views and test whether their signals persist across batches or predict classical morphological features. The observed gaps vary with dataset and evaluation protocol, separating within-study performance from cross-batch transfer and morphology prediction.

2.1 Background exploitability varies across datasets

  • Crop-to-segmented replicate-mAP gaps vary across datasets, exceed 0.4 for stronger encoders on RxRx1, and are nearly absent on JUMP-CP.RxRx1 crops reach 0.76–0.84 for SubCell and DINO, while segmented views reach 0.08–0.33; JUMP-CP views remain at 0.21–0.29.Source: §2.1 · pp. 3–4, Figure 2 · p. 4
  • Within each dataset, the C-to-S gap is similar across encoders, indicating that background-driven gain is associated with experimental design rather than encoder choice.RxRx1 and RxRx3-core use tightly controlled single-facility acquisition, whereas randomized layouts across 10 JUMP-CP imaging sources decorrelate background from perturbation identity.Source: §2.1 · p. 4
  • Harmony raises scIB Total scores by +0.070 to +0.103, mostly through batch correction, while biological-conservation deltas remain near zero.The near-zero biological-conservation deltas indicate that post-Harmony retrieval results reflect batch correction rather than artificial inflation of biological similarity.Source: §2.1 · pp. 4–5, Table 1 · p. 4

2.2 Background signal does not transfer across batches

  • On RxRx3-core, SubCell S retains 94% of C replicate mAP (0.716 vs. 0.760) but only 32% of C R@10 (0.100 vs. 0.313); density augmentation recovers 84% of the within-study gap but 8% of the cross-batch gap.For the latter comparison, SD R@10 is 0.117, S is 0.100, and C is 0.313.Source: §2.2 · p. 5
  • Background signal correlates with perturbation identity within a study but does not transfer when queries and galleries come from different batches.On RxRx1, DINO R@10 is 0.742 for C and 0.322 for S; on JUMP-CP, SubCell C and S reach 0.170 and 0.140.Source: §2.2 · p. 5

2.3 Morphology versus context: the background signal is not classical morphology

  • On JUMP-CP and RxRx3-core, segmented views predict cp_measure features better than crops despite comparable or lower replicate mAP.On RxRx3-core, OpenPhenom S reaches median r 0.432 versus 0.311 for C; on JUMP-CP, segmented views outperform crops by 0.02–0.06 in median r.Source: §2.3 · pp. 5–6, Figure 3 · p. 6
  • On RxRx1, the effect of segmentation and density augmentation on feature prediction varies by encoder: SD exceeds crop performance for SubCell and DINO but not OpenPhenom.DINO SD reaches 0.521 versus 0.487 for C, while OpenPhenom SD is 0.500 versus 0.546 for CD.Source: §2.3 · p. 6, Table 3 · p. 6
  • Intensity and texture features are more predictable than shape features across encoders, views, and datasets.Median r is 0.28–0.68 for intensity and texture features and 0.02–0.34 for shape features.Source: §2.3 · p. 6

3 Methods

The methods instantiate CP-BG-Bench across three Cell Painting datasets, using matched image views, adapted vision encoders, and a shared embedding and evaluation pipeline. The evaluation includes classical-feature baselines and four protocols spanning replicate retrieval, batch integration, feature prediction, and cross-batch retrieval.

3.1 Datasets

  • Three Cell Painting datasets span different perturbation modalities and experimental designs, with one batch held out per dataset for early stopping and cross-batch evaluation.The held-out batches are JUMP-CP source_4, RxRx1 HEPG2-07, and RxRx3-core plate_4; remaining batches support training and within-study evaluation.Source: §3.1 · p. 7, Table 4 · p. 7

3.2 Paired-view construction

  • Each cell crop yields segmented, crop-plus-density, and segmented-plus-density views while preserving cell identity.Segmentation zeros pixels outside cell and nucleus masks; four corner patches encode local cell density in the density variants.Source: §3.2 · p. 7

3.3 Encoders and fine-tuning

  • DINO ViT-B/16, OpenPhenom, and SubCell are LoRA-adapted with frozen backbones and projected into a shared 128-dimensional embedding space.The encoders use different pretraining sources and channel-input arrangements.Source: §3.3 · p. 7
  • Training uses a symmetric multi-positive contrastive loss to align image embeddings with perturbation embeddings, with minibatches stratified across imaging batches.Each minibatch contains 512 samples from 64 perturbation classes; positives span imaging batches.Source: §3.3 · p. 7

3.4 Aggregation and batch correction

  • Single-cell embeddings are L2-normalized, mean-pooled per well, spherised, reduced to 50 PCA components, and optionally Harmony-corrected.Pre- and post-Harmony embeddings are retained for downstream evaluations.Source: §3.4 · p. 7

3.5 Baselines

  • cp_measure features provide a classical baseline for replicate mAP and serve as targets for the feature-prediction probe.The features undergo the same aggregation and batch-correction pipeline as learned embeddings.Source: §3.5 · p. 8

3.6 Evaluation protocols

  • Four protocols assess replicate mAP, scIB batch integration, held-out-batch cp_measure prediction, and cross-batch perturbation recall.The feature probe reports median Pearson r across features; perturbation recall reports macro-averaged R@k by cosine similarity.Source: §3.6 · p. 8

4 Discussion

The discussion treats encoder rankings as protocol-dependent and recommends matching evaluation metrics to the intended use. It also bounds the conclusions by the datasets and training setup evaluated, and identifies open questions about transfer and feature predictability.

4 Discussion

  • 33–51% of crop replicate mAP is recovered by segmented views on RxRx1, depending on encoder; the remainder reflects signal that does not survive an acquisition-batch change.The discussion recommends treating single-metric rankings as tentative and preferring CP-prediction or cross-batch recall when generalisation matters.Source: §4 · p. 8
  • Cross-batch recall is most relevant for unseen plates or sites, while replicate mAP suits mechanism-of-action discovery within a single screen.The four protocols capture distinct aspects of encoder representations; scIB tracks closely with replicate mAP and is a less independent check.Source: §4 · p. 8
  • Table 5 summarizes what each evaluation protocol measures, its assumptions, and the failure mode exposed by the paired-view design.The failure modes are listed in the table’s rightmost column.Source: Table 5 · p. 9, §4 · p. 9
  • The evaluation covers three datasets, three encoder families, one fine-tuning recipe, one batch-correction method, and one segmentation pipeline; dataset differences cannot be attributed to a single factor.Each dataset also has a single fine-tuning run, so multi-seed replication would strengthen confidence in smaller effects, particularly RxRx1 CP-prediction inversions.Source: §4 · p. 9
  • Open questions include whether objectives that penalize within-study-only signal can reduce transfer gaps and what causes the observed feature-predictability hierarchy.The possible sources include ViT architecture, fluorescence-image pretraining distributions, and the contrastive objective.Source: §4 · p. 9

A.1 View Construction Details

The view-construction pipeline segments cells, extracts and filters crops, and builds segmented and density-augmented variants. It also controls crop boundaries, channel masking, and sampling across perturbation-source combinations.

A.1 View Construction Details

  • Cellpose-SAM segments nuclei and cells; overlap matches nucleus-cell pairs, unmatched objects are discarded, and 150×150-pixel crops are extracted around cell centroids.Cells within 75 pixels of the field-of-view border are excluded; fluorescent and mask channels are resized to 224×224 with bilinear and nearest-neighbour interpolation, respectively.Source: §A.1 · p. 13
  • Quality filtering excludes cells outside the 0.025–0.975 quantile range for nuclear area, cytoplasmic area, or nucleus-to-cytoplasm ratio; sampling targets 100 or 200 cells per perturbation-source combination.The target is 100 for JUMP-CP and RxRx1, and 200 for RxRx3-core; per-well cell counts vary widely.Source: §A.1 · p. 13
  • Density-augmented views CD and SD add four 22×22-pixel square patches to every channel, with intensity encoding the originating field of view’s cell density.Patches are inset 7 pixels from each corner; density is min-max scaled to [0, 255] across the cell pool.Source: §A.1 · p. 13
  • In segmented views S and SD, DNA-channel pixels outside the nucleus mask are zeroed, while other fluorescent-channel pixels outside the cell mask are zeroed.Source: §A.1 · p. 13

A.2 Encoder Architecture and Training Details

The appendix describes the encoder adaptations, shared embedding head, contrastive objective, and training setup. Perturbation embeddings and image augmentations differ by task and are used alongside batch-stratified sampling.

A.2 Encoder Architecture and Training Details

  • All encoders use LoRA with r=8 and α=16 while the remaining trunk is frozen, then feed a shared two-layer GeGLU MLP with a skip connection to a 128-dimensional embedding.Source: §A.2 · p. 13
  • DINOv3 ViT-B/16 uses a trainable 1×1 channel adapter, OpenPhenom accepts five channels directly, and SubCell maps Cell Painting channels into its four-channel input.DINOv3 yields 768-dimensional CLS tokens; OpenPhenom produces 384-dimensional embeddings; SubCell uses two-head attention pooling to produce 1536-dimensional embeddings.Source: §A.2 · p. 13
  • The symmetric multi-positive contrastive loss aligns image and perturbation embeddings and assigns softmax mass to label-equal partners.Compound perturbations use ECFP4 fingerprints, genetic perturbations use ESM2 embeddings, and a trainable projection head maps perturbation features to the shared 128-dimensional space.Source: §A.2 · pp. 13–14
  • Training runs for up to 25 epochs with early stopping and AdamW, using cosine learning-rate decay, 10% warmup, and bf16 mixed precision.Early stopping monitors macro-averaged image-to-perturbation R@1 on a held-out validation set; learning rates differ for projection heads, LoRA parameters, and temperature.Source: §A.2 · p. 14
  • Each batch contains 512 samples from 64 perturbation classes, stratified across imaging batches; controls are included in every batch.Augmentations include horizontal and vertical flips, 0–90° rotations, and light brightness/contrast jitter of ±10%.Source: §A.2 · p. 14

A.3 Aggregation and Batch Correction Details

Single-cell embeddings are normalized and mean-pooled to wells before whitening and dimensionality reduction. Harmony then corrects batch effects using dataset-specific batch keys, with both pre- and post-correction embeddings retained.

A.3 Aggregation and Batch Correction Details

  • Single-cell embeddings are L2-normalized after the encoder and mean-pooled per well.Source: §A.3 · p. 14
  • Well-level embeddings are ZCA-cor whitened using control wells only, then reduced to 50 PCA components.The whitening is fit on controls and applied to all wells.Source: §A.3 · p. 14
  • Harmony runs with default parameters on the 50-dimensional PCA embeddings, using imaging source for RxRx1 and JUMP-CP and plate for RxRx3-core.Both pre- and post-Harmony embeddings are retained for downstream evaluations, without post-Harmony renormalization.Source: §A.3 · p. 14

A.4 Baseline and Evaluation Protocol Details

This section specifies the classical feature baseline and the scoring and preprocessing details for replicate matching, batch integration, and CellProfiler feature prediction.

A.4 Baseline and Evaluation Protocol Details

  • cp_measure produces well-level baseline features after per-cell extraction, filtering, control-well ZCA whitening, PCA to 50 components, and Harmony correction.The resulting 1,358–1,443 features per dataset enter PCA.Source: §A.4 · p. 14
  • Replicate mAP averages treated-well AP values, excludes same-plate treated-treated pairs, and assesses significance with 10,000 permutations and Benjamini-Hochberg correction.The permutation null uses α=0.05.Source: §A.4 · p. 14
  • scIB batch integration averages five batch-correction metrics and five biological-conservation metrics, using imaging source or plate as the batch key and perturbation identity as the label key.The batch key is imaging source for JUMP-CP and RxRx1, and plate for RxRx3-core.Source: §A.4 · p. 14
  • CellProfiler feature prediction uses ridge regression with α=1.0, training-only embedding standardization, and held-out-batch evaluation on non-control wells.Features with training-set variance below 10^-8 are excluded.Source: §A.4 · p. 14

A.5 Extended Background Exploitability Analysis

The appendix compares crop and segmented views across datasets and encoders to characterize how much replicate matching depends on surrounding image context.

A.5 Extended Background Exploitability Analysis

  • The crop-versus-segmented replicate mAP gap quantifies background exploitability in this analysis.Crop views retain background context, whereas segmented views isolate the cell.Source: Figure A1 · p. 15
  • The background contribution to replicate mAP varies substantially across the three datasets, making it dataset-specific.The figure panels cover RxRx1, RxRx3-core, and JUMP-CP.Source: Figure A1 · p. 15
  • The comparison covers four views—C, CD, S, and SD—across SubCell, DINO ViT-B/16, and OpenPhenom, with cp_measure as a segmented-cell-only reference baseline.The table reports replicate mAP before and after Harmony across RxRx1, RxRx3-core, and JUMP-CP.Source: Table A1 · p. 15

A.6 Extended Batch Integration Analysis

Harmony improves batch correction while largely preserving biological conservation across configurations, although biological conservation and Harmony effects vary by dataset and encoder.

A.6 Extended Batch Integration Analysis

  • Every encoder-view configuration improves in batch correction after Harmony without a downward shift in biological conservation.Figure A2 shows the matched pre- and post-Harmony configurations across all three datasets.Source: §A.6 · p. 16, Figure A2 · p. 16
  • On RxRx1, crop views retain higher post-Harmony biological conservation than segmented views; SubCell CD scores 0.726 versus 0.516 for SubCell S.The crop-versus-segmented gap is also reported before Harmony.Source: §A.6 · p. 16
  • On JUMP-CP, Harmony lowers biological conservation by an average ΔBio of 0.007, with OpenPhenom SD worst at −0.018.The passage relates this trend to strong batch effects and weak perturbation signal, which make batch and biological variance harder to disentangle.Source: §A.6 · p. 16
  • OpenPhenom segmented views on RxRx1 have the weakest replicate matching, at mAP 0.122 and 0.077, and are Harmony-effectiveness outliers.The authors attribute this to little biological signal in the embedding, leaving less meaningful structure for Harmony to correct.Source: §A.6 · p. 16

A.7 Extended CellProfiler Feature Prediction

The extended analyses report how Harmony affects CellProfiler feature prediction and cross-batch perturbation recall, with outcomes differing across datasets and feature groups.

A.7 Extended CellProfiler Feature Prediction

  • On RxRx3-core, Harmony changes CellProfiler prediction by less than 0.002 across configurations; on JUMP-CP, its lift ranges from −0.006 to +0.033, with a median of +0.007.The largest JUMP-CP gains are +0.021 for SubCell C and +0.033 for DINO S.Source: §A.7 · p. 19
  • On RxRx1, Harmony lifts CellProfiler prediction by +0.034 to +0.054 for DINO, +0.011 to +0.020 for SubCell, and +0.006 to +0.013 for OpenPhenom.The table reports median Pearson r for all encoder-view configurations before and after Harmony.Source: §A.7 · p. 19, Table A4 · p. 19
  • Intensity and texture features are best predicted (median r 0.28–0.68), while AreaShape and Zernike are poorly recovered (0.02–0.34); segmented views outperform crops within most feature groups.The authors note that a ridge probe cannot determine whether shape information is absent or nonlinearly encoded.Source: §A.7 · p. 19, Figure A3 · p. 20

A.8 Extended Perturbation Recall

  • The cross-batch recall extension compares pre- and post-Harmony results, with Table A5 listing R@1 and R@10.The table caption specifies that bold marks the best R@10 per dataset within each Harmony stage.Source: §A.8 · p. 21, Table A5 · p. 21
  • On RxRx3-core, Harmony changes cross-batch recall by no more than 0.003 across configurations; on JUMP-CP, R@10 rises +0.01 to +0.035 for SubCell and DINO, with negligible lift for OpenPhenom.The RxRx3-core pattern mirrors the reported lack of change in CellProfiler prediction.Source: §A.8 · p. 21
  • On RxRx1, crop-view R@10 gains range from +0.03 to +0.13, while segmented-view gains are generally +0.02 to +0.06.OpenPhenom CD has the largest reported gains: +0.094 R@1 and +0.126 R@10; the passage contrasts this with the smaller Harmony effect on CellProfiler prediction.Source: §A.8 · p. 21
Loading 2609.12761v1…