PaperlayerView exact revision on arXiv

Source-linked AI summary

Biology-in-the-loop: Amortized Adaptive Hit Discovery in CRISPR Screens

Carl Edwards, Edward De Brouwer, Xiner Li, Namkyeong Lee, Ehsan Hajiramezanali, Anne Biton, Sara Mostafavi, Gabriele Scalia

arXiv:2609.11877v1q-bio.QMcs.AIcs.CLq-bio.GN

TL;DR

Adaptive CRISPR hit discovery must prioritize experiments under constrained budgets, but existing benchmarks are too small and narrow to support transferable acquisition policies. The paper introduces AssayBench-Loop, AssayLoop, and AssayLLM to learn from historical screens while using biological priors, achieving strong held-out performance with limited assays.

  • Problem

    Adaptive CRISPR discovery requires sequential experiment selection under constrained budgets, while existing benchmarks lack sufficient scale and phenotype diversity for rigorous evaluation and transferable policy learning.

  • Method

    AssayLoop combines AssayFormer, a history-conditioned policy trained across historical screens, with LLM-derived biological priors through an adaptive handoff, while AssayLLM applies task-specific post-training directly to an LLM.

  • Results

    5.67-fold enrichment over random selection and 27.7% hit recovery after assaying approximately 5% of the candidate library outperform existing adaptive methods, standalone LLMs, and AssayFormer alone.

  • Takeaways & Limitations

    Historical experimental repositories can support transferable adaptive acquisition policies that combine broad biological priors with feedback from accumulating assay observations.

  • Takeaways & Limitations

    The benchmark is retrospective, focuses on genome-wide binary hit discovery under fixed budgets, and remains to be validated prospectively in a live experimental loop.

Abstract

from arXiv · show

Many biological discovery problems require experiments to be selected sequentially under constrained budgets. CRISPR screening is a prominent example, as exhaustive perturbation testing is often infeasible and candidate perturbations must instead be prioritized over multiple experimental rounds. Despite the importance of this problem, existing benchmarks for adaptive hit discovery remain limited in scale and diversity. Here, we introduce AssayBench-Loop, a large-scale benchmark for adaptive hit discovery comprising 1,389 CRISPR screens across five phenotype categories. Beyond enabling systematic evaluation, its scale makes it possible to learn acquisition strategies across historical experiments. Building on this resource, we introduce AssayLoop, a sequential experimental design framework combining AssayFormer, a transformer-based amortized acquisition policy trained across historical screens to adapt from experimental feedback, with LLM-derived biological priors through an adaptive handoff. In this view, completed experiments become training data for learning how accumulated evidence should guide what to test next, while LLMs provide prior biological knowledge to seed the search. We further introduce AssayLLM, showing that the same principle can be extended directly to an LLM through task-specific post-training. On temporally held-out screens, AssayLoop achieves a 5.67-fold enrichment over random selection and recovers 27.7% of hits after assaying approximately 5% of the candidate library, outperforming existing adaptive-design methods and standalone LLMs, and AssayFormer alone. Performance improves with increasing historical training data and transfers to phenotype categories excluded from training. These results demonstrate the value of learning acquisition policies across historical experiments and combining them with broad biological priors for efficient adaptive hit discovery.

1 Introduction

Adaptive CRISPR hit discovery is constrained by vast candidate spaces, limited benchmarks, and the need to combine biological priors with feedback-driven experimentation. The paper introduces a large benchmark and methods that learn transferable acquisition strategies across historical screens.

  • Problem setting: Sequential design selects perturbations over multiple rounds, observes outcomes, and uses accumulated feedback to guide later experiments under a finite budget.This lab-in-the-loop process is motivated by the infeasibility of exhaustive perturbation testing.
  • Benchmark gap: 1,389 CRISPR screens across five phenotype categories address the lack of large, diverse benchmarks for adaptive hit discovery.The benchmark supports both systematic evaluation and learning from historical experiments rather than fitting each screen independently.
  • Method: AssayLoop combines AssayFormer, trained across historical screens, with LLM-derived biological priors through an adaptive handoff.The LLM guides early acquisitions, then AssayFormer adapts prioritization as assay-specific observations accumulate.
  • Method: AssayLLM extends historical-screen training directly to an LLM through task-specific post-training on acquisition trajectories and reinforcement learning.The 27B-parameter model provides a proof of principle for learning adaptive acquisition behavior within an LLM.
  • Results: 5.67-fold enrichment over random selection and 27.7% hit recovery after assaying approximately 5% of the library outperform existing adaptive methods, standalone LLMs, and AssayFormer alone.The result is reported on temporally held-out screens.
  • Implication: Historical repositories can support transferable decision-making that combines broad biological knowledge with feedback-conditioned policies for adaptive experimentation.The paper frames this as a route toward lab-in-the-loop systems that accumulate experience across campaigns.

2 Results

The study formalizes adaptive hit discovery as sequential selection from a shared gene pool and constructs a temporally split benchmark for learning transferable policies. It evaluates amortized modeling with metrics designed for heterogeneous screens and incomplete retrospective ground truth.

  • Problem formulation: Each policy selects batches of b genes over T rounds, observes binary hit labels, and maximizes discovered hits within the fixed budget T × b.The formal setup uses a natural-language screen description, screen-specific libraries, and a shared candidate gene pool.
  • Benchmark: 1,389 screens are split into 1,349 training, 20 validation, and 20 test screens across five phenotype categories.Validation and test screens are temporally held out and restricted to genome-wide assays with sufficient hit signal and non-trivial baseline recoverability.
  • Evaluation: The benchmark uses hit enrichment factor as its primary metric, with nAUC, fraction of hits found, and pathway diversity providing additional evaluation axes.The metrics account for heterogeneous baseline difficulty, incomplete ground truth, hallucinated genes, and fixed budgets of 10 rounds with 100 requested genes.
  • AssayFormer: AssayFormer learns one history-conditioned acquisition policy across historical screens instead of fitting a separate surrogate for each new experiment.Its transformer processes the screen description and observed gene–outcome history to score candidate genes.
  • AssayFormer: AssayFormer training initializes gene embeddings with cross-screen hit structure, predicts unobserved hit labels, and samples varying observed histories to mimic sequential states.The training procedure uses Bayesian Probabilistic Matrix Factorization followed by supervised learning.
  • Adaptive handoff: LLM-derived biological priors complement AssayFormer because the learned policy initially relies mainly on cross-screen structure and limited assay-specific description information.This motivates the adaptive handoff strategy.

A. Active hit discovery paradigms

ASSAYLOOP combines LLM biological priors for early selection with ASSAYFORMER, a history-conditioned policy learned across historical screens. On held-out screens, this handoff outperforms standalone policies and improves with more training data while transferring beyond observed phenotype categories.

  • A. Active hit discovery paradigms: ASSAYLOOP uses an LLM for the first three rounds, then hands accumulated experimental feedback to ASSAYFORMER for history-conditioned acquisition.This design pairs early biological prior knowledge with later adaptation to assay-specific outcomes.
  • A. Active hit discovery paradigms: EF ≈5.67, nAUC 21.7%, and approximately 27.7% of hits are recovered at 5% effective library coverage, exceeding standalone LLMs and ASSAYFORMER alone.Both Gemini-3.1-Pro and GPT-5.6 Sol warm starts achieve this held-out-screen performance.
  • A. Active hit discovery paradigms: Standalone LLMs use experimental feedback, but their modest label-removal losses indicate that acquisition remains strongly driven by pre-observation information.ASSAYLOOP is designed to overcome this limitation by combining LLM priors with a feedback-conditioned learned policy.
  • A. Active hit discovery paradigms: Historical-screen methods outperform methods that adapt within each screen without historical data, highlighting the value of transferring acquisition knowledge across experiments.BPMF, Screen-kNN, and MAML also benefit from training screens despite not being designed specifically for adaptive hit discovery.
  • A. Active hit discovery paradigms: ASSAYFORMER performance increases with training-set size, with EF@10 more than doubling and no clear saturation at the largest evaluated dataset.Increasing model capacity across nearly 40-fold yields little additional benefit.
  • A. Active hit discovery paradigms: Under leave-one-phenotype-out training, ASSAYFORMER reaches EF 4.59 versus 4.83 with full training, indicating robustness when one phenotype category is excluded.The authors interpret this evaluation primarily as robustness to phenotype exclusion because withheld data amounts vary.

A. Reactome pathway diversity

ASSAYLOOP maintains broad Reactome pathway diversity while improving hit discovery, and ASSAYFORMER encodes asymmetric, context-sensitive gene influences. These learned relationships are presented as interpretable policy updates rather than causal or calibrated interaction estimates.

  • A. Reactome pathway diversity: ASSAYLOOP combines high dataset-level pathway diversity with broad batch coverage, whereas Screen-kNN concentrates most strongly and Gemini-3.1-Pro is narrower within individual batches.The figure reports EP-D 70.4 and EP-B 18.4 for ASSAYLOOP, compared with Screen-kNN EP-D 54.8 and Gemini EP-B 14.5.
  • A. Reactome pathway diversity: ASSAYLOOP reaches EP-B 18.4, EP-S 46.6, and EP-D 70.4 while increasing EF from 4.83 to 5.67 relative to ASSAYFORMER alone.Its diversity remains comparable to or higher than ASSAYFORMER across batch, screen, and dataset scopes.
  • A. Reactome pathway diversity: Influence measures the change in a target gene’s acquisition score when a probe gene is added as a hit, relative to matched random-background histories.The analysis averages over 10 independently sampled contexts containing 50 random genes.
  • A. Reactome pathway diversity: ASSAYFORMER’s influence updates are asymmetric: 44% of reciprocal gene pairs have opposite-sign influences, including MDM2→PFDN4 +0.33 and PFDN4→MDM2 −0.46.These directional updates cannot be explained by purely symmetric gene similarity.
  • A. Reactome pathway diversity: ASSAYLLM transfers teacher-policy behavior through SFT and GRPO, raising standalone EF from 2.65 to 3.69 and achieving EF 5.23 in handoff-aligned training.The handoff-trained variant controls the initial rounds before ASSAYFORMER completes the campaign.

3 Discussion

The paper establishes a large benchmark for transferable adaptive hit-discovery policies, showing that combining LLM biological priors with feedback-trained acquisition improves performance while revealing scope limits for retrospective evaluation.

  • Benchmark and overall performance: 27.7% of hits were recovered after assaying 5% of the candidate library, with AssayLoop achieving the strongest performance among evaluated methods.The benchmark spans 1,389 screens across five phenotype categories.
  • Benchmark and overall performance: LLM warm starts followed by AssayFormer adaptation outperform both components in isolation, supporting complementarity between biological priors and feedback-conditioned policies.The handoff uses LLM prioritization in early rounds before transitioning to AssayFormer.
  • Role of LLMs: Frontier LLMs incorporate experimental feedback to some degree, but explicitly trained policies adapt more effectively, while domain-specific post-training narrows this gap.AssayLLM post-training yields a 39% improvement over the base model.
  • Scaling and transfer: AssayFormer performance improves consistently with more training screens, with no clear plateau across the evaluated range, and generalizes to held-out phenotype categories.These results suggest that larger and more diverse screen collections may further improve transferable acquisition policies.
  • Scope and limitations: The study is limited by its retrospective, genome-wide, hit-signal-focused benchmark and fixed-budget binary-hit formulation, which remains unvalidated prospectively in live experiments.The analysis also does not explicitly model experimental noise across replicates or heterogeneous experimental costs.
  • Future directions: Natural-language task representations could extend the benchmark to richer readouts and perturbation modalities, while adaptive campaigns could jointly advance discovery and model training.AssayLLM provides a mechanism for incorporating new screening data as it becomes available.

4 Methods

The paper formulates adaptive hit discovery as sequential Bayesian experimental design and implements acquisition policies that use screen descriptions and accumulated feedback. ASSAYFORMER learns history-conditioned prioritization across screens, while ASSAYLLM provides a post-trained LLM alternative.

  • Problem formulation: Adaptive hit discovery selects gene batches sequentially and maximizes discovered hits within a fixed acquisition budget.Each screen provides a candidate library and binary hit labels, while the policy updates its history after each batch.
  • Evaluation metrics: The evaluation uses hit enrichment plus trajectory, recall, library-validity, essentiality, and biological-diversity metrics.The enrichment factor penalizes hallucinated genes and unfilled slots while not penalizing valid genes absent from a screen library.
  • ASSAYFORMER architecture: ASSAYFORMER learns a shared acquisition policy across historical screens, scoring untested genes from the screen description and observed hit history.The policy uses a transformer over a description token and history gene tokens, then selects the top-scoring batch.
  • Training procedure: ASSAYFORMER training combines gene-embedding initialization, supervised fine-tuning, and reinforcement-learning fine-tuning.The reinforcement-learning stage optimizes full multi-round trajectories rather than only greedy stepwise decisions.
  • ASSAYLLM: ASSAYLLM is post-trained to return ranked gene batches from screen descriptions and observed histories across sequential acquisition rounds.It supports both standalone control of all rounds and warm-start control before handoff to ASSAYFORMER.
  • Policy analysis: Influence analysis measures how adding an observed hit changes another gene’s acquisition score across matched synthetic histories.The analysis uses 50-gene random backgrounds and averages over 10 independently sampled contexts.

A Dataset Composition

ASSAYBENCH-LOOP uses temporally split CRISPR screens selected for genome-wide coverage, sufficient signal, LLM recoverability, phenotype balance, and within-phenotype diversity. Its eligibility criterion defines a benchmark for screens with detectable description-based predictive signal rather than an unbiased population of all genome-wide screens.

  • Screen selection: ASSAYBENCH-LOOP retains 1,349 training screens and selects 20 validation and 20 test screens from a temporally split AssayBench resource.The benchmark uses genome-wide screens with sufficient hit signal and a minimum LLM performance threshold.
  • Eligibility criteria: The selected screens require libraries of 18,000–22,000 genes, at least 50 hits, hit rates at most 15%, and AnDCG@100 at least 0.05 for the best LLM.Phenotype allocation is proportional to the eligible pool, with a soft cap of six screens per phenotype.
  • Screen selection: Within-phenotype diversity is selected by greedy max-min TF-IDF similarity over screen descriptions.Descriptions include phenotype, condition, cell line, cell type, and library methodology.
  • Scope boundary: Because eligibility requires a minimum LLM-signal criterion, ASSAYBENCH-LOOP evaluates screens with detectable description-based predictive signal rather than an unbiased sample of genome-wide screens.This selection criterion is an explicit scope boundary for interpreting benchmark results.
  • Gene universes: The benchmark distinguishes a 23,724-gene training vocabulary, a 21,147-gene fixed acquisition universe, and screen-specific measured libraries.These gene sets serve different roles in representation learning, policy evaluation, and observed-label availability.
  • Embedding ablation: Embedding initialization materially affects ASSAYFORMER performance, with supervised-fine-tuning EF ranging from 1.06 for GenePT to 3.83 for BPMF.Reinforcement learning improves all tested initializations, while downstream performance does not track recovery of canonical biological relationships.

B.2 Training strategies

The ablations test how ASSAYFORMER’s training objective and screen conditioning affect adaptive acquisition, using BPMF embeddings and reported comparisons in Supplementary Table 3.

  • Training objectives: Terminal enrichment-factor optimization performs worse than the context-delta reward because it provides a weaker signal for history-dependent adaptation.The context-delta reward explicitly rewards improvement over a context-free policy as observations accumulate.
  • Screen conditioning: Removing the screen description has a relatively minor effect, suggesting acquisition relies mainly on cross-screen gene-hit patterns learned from historical data.
  • Latent-factor model: The model represents observed screen–gene outcomes with a latent-factor formulation whose unobserved pairs are omitted from the fitting sums, yielding matrix completion rather than zero imputation.
  • Acquisition space: ASSAYFORMER evaluates the full gene universe, including genes without hit labels in the current screen, rather than ranking only genes in each screen’s library.

D.3 Training settings

The training settings cover supervised and reinforcement-learning optimization for ASSAYFORMER, alongside Screen-kNN and prompt-based active-learning procedures for comparison and evaluation.

  • Screen-kNN: Screen-kNN combines a global gene hit-rate prior with screen-specific evidence weighted by similarity to historical screens.The mixture shifts from the prior toward screen evidence as more genes are observed, and each batch contains the highest-scoring untested genes.
  • Screen objective: The RSL3 screen defines hits as activations that increase viability after ferroptosis-drug exposure, with significance assessed at FDR < 0.05.Genes with lower FDR are ranked higher under the prompt’s stated criteria.

E.2 Qualitative analysis of ASSAYLOOP on two biologically distinct screens

The qualitative analysis compares ASSAYLOOP, ASSAYFORMER, and Gemini-3.1-Pro across an NF-κB/TNF signaling screen and an AAV transgene-silencing screen.

  • Screen comparison: Two biologically distinct screens are examined: NF-κB/TNF signaling and AAV transgene silencing.
  • Model comparison: The figure organizes acquisition-round gene compositions by ASSAYFORMER, Gemini-3.1-Pro, and the Gemini-to-ASSAYLOOP handoff.ASSAYFORMER appears in the top row, the handoff in the middle row, and Gemini-3.1-Pro in the bottom row.

F.1 Embedding drift during training

Embedding drift analyses show that BPMF preserves its geometry during training, while other initializations reorganize more substantially; the preserved structure aligns with screen phenotypes rather than canonical pathways or complexes.

  • Drift measurements: BPMF embeddings are almost perfectly preserved during training, with pairwise cosine similarity Pearson r = 0.999 and median neighborhood RBO above 0.95.These results indicate that ASSAYFORMER learns its policy on top of the BPMF structure rather than substantially reorganizing it.
  • Initialization comparison: Random, SVD, MF, and MF-Sphere initializations undergo substantially greater neighborhood reorganization during supervised fine-tuning than BPMF.GenePT shows relatively little drift despite poor downstream performance, separating stability from task usefulness.
  • Task alignment: BPMF effectiveness is associated with a geometry aligned to adaptive hit discovery rather than with stability alone or recovery of canonical biological relationships.The analysis reports a negative association between canonical relationship recovery and downstream ASSAYFORMER performance.
  • Embedding structure: BPMF embeddings organize primarily by shared screen phenotype, while genes from the same complexes or pathways are generally distributed across the embedding space.HDBSCAN identifies phenotype-related clusters, whereas CORUM and Reactome annotations show minimal spatial clustering except partial organization in the common-essential region.

G Biological Diversity Analysis

Supplementary Figure 5 compares the biological pathway composition of genes acquired by individual LLMs across test screens, alongside diversity-related summaries.

  • G Biological Diversity Analysis: Supplementary Figure 5 compares LLM-acquired genes by their Reactome pathway composition across the 20 test screens.The figure also reports unique genes, annotated picks, and a random-sampling baseline.

H Metrics used in ASSAYBENCH-LOOP

ASSAYBENCH-LOOP evaluates adaptive hit discovery with enrichment, recovery, trajectory, validity, essentiality, biological breadth, and batch-diversity metrics.

  • H Metrics used in ASSAYBENCH-LOOP: Hit enrichment factor measures acquired hits relative to the number expected under random selection.Its normalization penalizes hallucinated genes but not valid genes outside the screen library.
  • H Metrics used in ASSAYBENCH-LOOP: Normalized AUC captures hit-recovery ordering across the acquisition trajectory, while fraction of hits measures endpoint recall.nAUC uses cumulative hits versus effectively queried library fraction; fraction of hits is recovered hits divided by total screen hits.
  • H Metrics used in ASSAYBENCH-LOOP: Shortfall measures acquisitions outside the screen library, including valid out-of-library genes and hallucinated genes.Only in-library genes can contribute to recovered-hit counts.
  • H Metrics used in ASSAYBENCH-LOOP: Percentage of essential genes diagnoses whether recovered hits are concentrated among 1,827 common essential genes.Essential genes can cause general cell death without reflecting screen-specific biology.
  • H Metrics used in ASSAYBENCH-LOOP: Effective Pathways estimates biological breadth as the effective number of equally represented Reactome level-2 pathway groups.Larger values indicate broader pathway coverage, with rarefied sample sizes used for comparability.
  • H Metrics used in ASSAYBENCH-LOOP: Vendi Ratio and Pathway Overlap quantify whether each batch selects functionally diverse genes or concentrates in a narrow biological neighborhood.Vendi Ratio reflects effective gene distinctness, whereas Pathway Overlap compares annotation similarity with random selection.

I ASSAYFORMER Reveals Directional Gene-Gene Relationships

ASSAYFORMER uses experimental feedback to infer context-dependent, directional gene relationships, while its performance depends more on training data than model size and exceeds standalone LLM adaptation.

  • I ASSAYFORMER Reveals Directional Gene-Gene Relationships: ASSAYFORMER learns stronger influences among functionally related genes than among canonical cancer drivers, indicating structured context dependence.Supplementary Figure 7 analyzes 31 representative genes, including 12 cancer drivers and 19 functional-module genes.
  • I ASSAYFORMER Reveals Directional Gene-Gene Relationships: 44% of reciprocal gene pairs have opposite-sign influences, with MDM2 increasing PFDN4’s score by +0.33 while PFDN4 decreases MDM2’s by −0.46.The influence matrix and example demonstrate directional rather than symmetric gene relationships.
  • I ASSAYFORMER Reveals Directional Gene-Gene Relationships: Removing known database gene pairs leaves high-magnitude influences consistent with co-essentiality or synthetic-lethal dependencies.The analysis excludes pairs represented in STRING, CORUM, SIGNOR, MSigDB, and Reactome before examining remaining relationships.
  • I ASSAYFORMER Reveals Directional Gene-Gene Relationships: Increasing parameter count from 0.28M to 10.97M produces no meaningful improvement, making training-screen availability the current binding constraint.The default model contains 4.44M parameters, and the comparison spans a nearly 40-fold capacity range.
  • I ASSAYFORMER Reveals Directional Gene-Gene Relationships: Removing hit labels reduces performance across tested LLMs, but standalone LLMs remain consistently below ASSAYFORMER.This evaluates adaptation from experimental outcomes while preserving previously acquired genes.
Loading 2609.11877v1…