PaperlayerView exact revision on arXiv

Source-linked AI summary

Can AI Scientist Agents Learn from Lab-in-the-Loop Feedback? Evidence from Iterative Perturbation Discovery

Gilles Wainrib, Barbara Bodinier, Haitem Dakhli, Josep Monserrat, Almudena Espin Perez, Sabrina Carpentier, Roberta Codato, John Klein

arXiv:2603.26177v1cs.LG

TL;DR

Whether LLM agents truly learn from experimental feedback remains contested. This paper tests the question through 800 iterative perturbation-discovery experiments on Cell Painting data, comparing feedback-enabled agents with prior-knowledge and randomized-feedback controls. Feedback improves discoveries, but the effect depends on model capability and the structure of the feedback signal.

  • Problem

    Prior studies question whether LLM agents genuinely learn from experimental feedback or instead rely on static biological knowledge.

    Source: §1 · p. 1

  • Method

    The study benchmarks LLM agents across 800 iterative perturbation-discovery experiments on JUMP Cell Painting data, including prior-knowledge, feedback-enabled, and randomized-feedback conditions.

    Source: §1 · p. 1, §3.1 · p. 2

  • Results

    +185% improvement over random selection was achieved by the best feedback-enabled agent, versus +85% for prior knowledge alone, with p = 0.003.

    Source: §1 · p. 1

  • Takeaways & Limitations

    Effective in-context learning from experimental feedback emerges when model capability is sufficient and the feedback retains its gene–outcome structure.

    Source: §1 · p. 1, §6 · p. 6, §7 · p. 7

    The findings are limited to one JUMP Cell Painting dataset, one assay type, two proprietary models, and retrospective data; real noisy or delayed feedback may alter ICL dynamics.

    Source: §4.4 · p. 3, §6 · pp. 6–7

Abstract

from arXiv · show

Recent work has questioned whether large language models (LLMs) can perform genuine in-context learning (ICL) for scientific experimental design, with prior studies suggesting that LLM-based agents exhibit no sensitivity to experimental feedback. We shed new light on this question by carrying out 800 independently replicated experiments on iterative perturbation discovery in Cell Painting high-content screening. We compare an LLM agent that iteratively updates its hypotheses using experimental feedback to a zero-shot baseline that relies solely on pretraining knowledge retrieval. Access to feedback yields a $+53.4\%$ increase in discoveries per feature on average ($p = 0.003$). To test whether this improvement arises from genuine feedback-driven learning rather than prompt-induced recall of pretraining knowledge, we introduce a random feedback control in which hit/miss labels are permuted. Under this control, the performance gain disappears, indicating that the observed improvement depends on the structure of the feedback signal ($+13.0$ hits, $p = 0.003$). We further examine how model capability affects feedback utilization. Upgrading from Claude Sonnet 4.5 to 4.6 reduces gene hallucination rates from ${\sim}33\%$--$45\%$ to ${\sim}3$--$9\%$, converting a non-significant ICL effect ($+0.8$, $p = 0.32$) into a large and highly significant improvement ($+11.0$, $p=0.003$) for the best ICL strategy. These results suggest that effective in-context learning from experimental feedback emerges only once models reach a sufficient capability threshold.

1 Introduction

This study tests whether LLM agents genuinely learn from experimental feedback rather than relying only on static biological knowledge. Across 800 perturbation-discovery experiments, feedback improves discovery performance, with benefits varying by model capability.

  • The study addresses whether experimental improvements reflect genuine feedback learning or retrieval of static prior knowledge.Earlier work reported comparable performance when feedback labels were randomly permuted.Source: §1 · p. 1
  • +185% improvement over random selection was achieved by the best feedback-enabled agent, versus +85% for prior knowledge alone across features.The comparison was significant at p = 0.003.Source: §1 · p. 1
  • Feedback produced significant within-feature gains on all 10 target features, ranging from +3.7 to +27.4 additional hits.The reported range spans F90 to F80, respectively.Source: §1 · p. 1
  • The direction of the ICL effect is consistently positive, although its magnitude varies across features and depends on model capability.Upgrading from Sonnet 4.5 to 4.6 reduced gene hallucination from ∼33%–45% to ∼3–9%.Source: §1 · p. 1

2 Related Work

Prior work applies LLMs to sequential optimization across biological, chemical, materials, and drug-design settings. Other studies question whether such systems genuinely learn from feedback, while Cell Painting remains an underexplored setting for sequential perturbation selection.

  • LLM-guided closed-loop optimization has been applied to genetic perturbation, chemical synthesis, materials discovery, and drug design.Related approaches use feedback as in-context exemplars or as an acquisition function replacing hand-crafted kernels.Source: §2 · p. 1
  • Gupta et al. (2025) found comparable performance with randomly permuted outcomes, suggesting BioDiscoveryAgent gains may reflect prior knowledge rather than in-context adaptation.Other work reports exchangeability violations with long contexts and limited out-of-distribution ICL generalization.Source: §2 · p. 1
  • Cell Painting is a high-content imaging assay, and the JUMP resource contains profiles for over 116,000 chemical and 22,000 genetic perturbations.Prior Cell Painting work aligned language and image representations but did not address sequential perturbation selection.Source: §2 · p. 2

3 Methods

The study evaluates iterative perturbation discovery on JUMP Cell Painting data using multiple LLM agents, feedback conditions, baselines, and feature-level statistical tests. Agent variants differ in whether they use prior knowledge, experimental outcomes, or structured phenotypic hypotheses.

3.1 Data and Task

  • 3.1 Data and Task: The task uses ∼8,000 gene knockouts across 4,672 CellProfiler features, with 10 iterations of 100 perturbations and cumulative unique discoveries as the metric.A hit is defined by p_y(x) < 0.05 for target feature y.Source: §3.1 · p. 2
  • 3.1 Data and Task: Ten target features spanning baseline hit rates of ∼1–2% are evaluated with 10 independent replicates, totaling 800 experiments across eight conditions.Replicates provide repeated measurements for each feature and condition.Source: §3.1 · p. 2

3.2 Agent Architectures

  • 3.2 Agent Architectures: The zero-shot agent selects genes using only the target description and available-gene list, providing a prior-knowledge baseline without outcome feedback.The compared LLM backbones are Claude Sonnet 4.6 and 4.5.Source: §3.2 · p. 2

In-Context Learning from Experimental Feedback

  • In-Context Learning from Experimental Feedback: ICL-EF exposes the full history of tested genes, hit/miss outcomes, and successful-gene prefixes to guide subsequent selections.The agent can identify patterns such as related genes succeeding after one family member is a hit.Source: In-Context Learning from Experimental… · p. 2
  • In-Context Learning from Experimental Feedback: ICBR-EF adds phenotypic signatures and an LLM-managed hypothesis register tracking confidence and status across iterations.The register supports explicit creation, weakening, and abandonment of biological hypotheses.Source: In-Context Learning from Experimental… · p. 2

3.3 Baselines

  • 3.3 Baselines: The study includes uniform random selection, GP-UCB using STRING protein-interaction similarity, and random feedback with permuted hit/miss labels.Random feedback preserves per-iteration hit rates while breaking gene–outcome correlations; GP-UCB is described as deliberately conservative.Source: §3.3 · p. 2

3.4 Statistical significance of pairwise methods comparison across features

  • 3.4 Statistical significance of pairwise methods comparison across features: Replicates are averaged within feature, then paired feature-level comparisons use exact two-sided sign-flip permutation tests with Benjamini–Hochberg FDR control.Hierarchical bootstrap intervals are additionally reported for combined feature and replicate variability.Source: §3.4 · pp. 2–3

4 Results

Across 800 experiments, feedback-enabled agents outperform zero-shot and random baselines, with gains depending on feature structure and model capability. The analyses link these gains to feedback-driven pathway exploitation, phenotypic inference, and faithful use of experimental outcomes.

4.2 ICL-EF vs ICBR-EF: qualitative comparison

  • ICL-EF progressively widens its lead over zero-shot as observations accumulate, reaching 10 additional genes by iteration 10.ICL-EF and ICBR-EF remain comparable through iteration 6, after which ICBR-EF shows only a marginal contrast.Source: §4.2 · p. 3, Figure 2 · p. 4

4.3 Feature-Level Heterogeneity

  • ICBR-EF improves over random selection by up to 6.0× on F70, with large gains also on F80 and F10.Relative improvement reaches 5.1× on F80 and 3.2× on F10; F90 still benefits 4.1× despite being difficult.Source: §4.3 · p. 3, Figure 3 · p. 5
  • Feature structure shapes the effect: gains are largest where gene-family correlations are learnable, while feedback also raises variability through productive and misplaced-obstinacy behaviors.ICL-EF has std=8.2 and ICBR-EF std=9.5 versus std=4.3 for zero-shot.Source: §4.3 · p. 3

4.4 Model Capability as a Critical Factor

  • Sonnet 4.6 reduces gene hallucination from 45.3% to 9.1%, improving instruction-following and increasing ICL-EF discoveries from 21.8 to 29.3.The authors caution that the model upgrade changes several capabilities, so hallucination reduction alone cannot explain the improvement.Source: §4.4 · pp. 3–4
  • With Sonnet 4.6, ICL-EF discovers 29.3 hits versus 20.4 zero-shot (+8.9, p = 0.003), while ICBR-EF reaches 31.4, improving over ICL-EF by +2.1.Source: §4.1 · p. 3, Table 1 · p. 4

4.5 Behavioral Analysis: The Evolution of Search Strategies

  • ICL-EF exploits confirmed hits within biological complexes, using complex, family, or subunit terminology 7.9 times per iteration versus 2.3 for zero-shot.This exploitation lets ICL-EF mine related members of complexes such as SWI/SNF after identifying one hit.Source: §4.5 · p. 4
  • ICBR-EF uses phenotypic state inference to connect genes across disconnected pathways, referencing phenotypic markers 22.6 times per iteration versus 0.7 for ICL-EF.On F80, it linked perturbations across DNA repair, cyclin-CDK, and mitotic kinase pathways to mitotic arrest.Source: §4.5 · p. 5

4.6 Reconciling with Prior Work

  • +13.0 hits: ICBR-EF outperforms random feedback, while permuted feedback falls below zero-shot, showing that performance depends on factual feedback content.ICBR-EF discovers 31.4 hits versus 18.4 for Random FB and 20.4 for zero-shot; random feedback is harmful on 6/10 features.Source: §4.6 · pp. 5–6

5 Future Work

The paper identifies context-window, session-persistence, and training limitations of ICL, while proposing reinforcement learning as a future direction enabled by the JUMP environment.

  • ICL is limited because observations must fit a fixed context window, learning resets each session, and outcomes are not used for explicit training.Source: §5 · p. 6
  • The JUMP dataset provides a simulated environment for reinforcement-learning training from verifiable rewards.Source: §5 · p. 6
  • Reinforcement learning could optimize exploration–exploitation patterns observed in gene-family correlations and misplaced obstinacy.Source: §5 · p. 6
  • An RL-trained agent could learn strategies transferable across feature types, which the authors leave for future investigation.Source: §5 · p. 6

6 Discussion

The discussion presents practical benefits and costs of feedback-enabled agents while emphasizing model dependence, possible data contamination, and unresolved comparisons with hybrid methods.

Practical implications

  • Model capability matters: upgrading from Sonnet 4.5 to 4.6 reduced hallucination from ∼33%–45% to ∼3–9% and increased ICL-EF discoveries by 34%.Source: §6 · p. 6

Cost considerations

  • Each 10-iteration campaign costs approximately $1.50 in LLM API calls, or roughly $0.05 per discovery for ICL-EF.Source: §6 · p. 6

Data contamination

  • Public-dataset contamination may inflate zero-shot prior-only performance, although random-feedback results indicate that the ICL effect depends on feedback content.Source: §6 · p. 6

Relationship to LLMNN

  • The study does not directly compare pure ICL with Gupta et al.’s LLMNN hybrid, leaving their relative performance unresolved.Source: §6 · pp. 6–7

Summarized limitations

  • Results may not generalize beyond two Claude models, the JUMP assay, retrospective noise-free data, or the evaluated feedback design.Source: §6 · pp. 6–7

7 Conclusion

Across 800 experiments, the study finds that capable frontier LLM agents can learn from experimental feedback for iterative perturbation discovery, but the result remains bounded by its evaluation setting.

  • 31.4 mean discoveries represents a +185% improvement over random selection, with the ICL effect significant at p = 0.003.Source: §7 · p. 7
  • +13.0 discoveries over permuted feedback at p = 0.003 indicates that the learning effect depends on structured feedback rather than random labels.Source: §7 · p. 7
  • Model capability is a prerequisite: ICL is non-significant with Sonnet 4.5 but highly significant with Sonnet 4.6.Source: §7 · p. 7
  • The findings are specific to JUMP Cell Painting, one assay type, two proprietary models, and retrospective data, so generalization remains unestablished.Source: §7 · p. 7

A Zero-shot Agent: Prompting Strategy and Execution Trace

The zero-shot agent repeatedly selects genes using biological prior knowledge without experimental feedback, producing modest discoveries that decline as high-confidence categories are exhausted.

A Zero-shot Agent: Prompting Strategy and Execution Trace

A.1 System Prompt

  • Its fixed prompt emphasizes gene families, nuclear-phenotype categories, and known phenotype associations.Source: §A.1 · p. 8

A.2 Per-Iteration User Prompt

  • Each iteration supplies an iteration counter, complete tested and untested gene lists, self-check instructions, and a strict JSON output format.Source: §A.2 · p. 8

A.3 Execution Trace

  • In iteration 1, prior-based reasoning spans chromatin, nuclear transport, cell cycle, DNA damage, and transcription, yielding 5 hits among 100 genes.Source: §A.3 · p. 9
  • Iteration 2 broadens the same prior-based categories and yields 4 hits among 100 genes.Source: §A.3 · p. 9
  • Across iterations 3–10, the strategy does not adapt; hit rates remain low and later gains reflect depleted-prior coverage rather than learning.Source: §A.3 · p. 9
  • After 1,000 tests, the zero-shot agent discovers 28 of 152 significant genes, achieving 18.4% recall.Source: §A.3 · p. 9

B ICL-EF Agent: Prompting Strategy and Execution Trace

The ICL-EF agent combines biological priors with iterative hit/miss feedback, prefix summaries, and gene-family exploitation to select successive perturbation batches. Its execution trace shows adaptive exploitation of productive families followed by pivots when those families are exhausted.

B.2 Per-Iteration User Prompt

  • At each iteration, the agent receives the full tested-gene history, hit/miss outcomes, p-values, prefix frequencies, and tested-gene lists before proposing the next batch.The prompt explicitly balances exploitation of observed patterns with exploration of untested categories and checks that proposed genes are untested.Source: B ICL-EF Agent: Prompting Strategy and… · p. 9, §B.2 · p. 10

B.1 System Prompt

  • The fixed strategy prompt prioritizes related gene families, pathways, protein complexes, and recurring name prefixes as biological routes for exploitation.Successful-gene prefixes are counted and the five most frequent prefix-count pairs are exposed to the model.Source: §B.1 · p. 9, B ICL-EF Agent: Prompting Strategy and… · p. 9

B.3 Execution Trace

  • Initial prior-guided selections span nine mechanistic categories, producing hits in chromatin remodeling, histone regulation, transcription, DNA repair, and nuclear signaling.The first batch includes CHD4, KMT2B, EP300, TAF1, POLR2A, PCNA, TRRAP, and MYC.Source: §B.3 · pp. 10–11
  • The family-exploitation table reports 26 family-assigned hits and 9 singleton hits, with early CHD, TAF, POLR, and MED families exhausted by iteration 7.The exhaustion pattern forces later pivots toward ZSCAN, KLK, and PPP anchors.Source: Table 2 · p. 11, §B.3 · p. 11
  • From iteration 2 onward, the agent expands around successful TAF, POLR, CHD, KMT, MYC, and PCNA anchors to test related subunits, partners, and pathway members.Iteration 2 yields TAF5, SUPT6H, and POLR3B after systematic expansion from the initial hits.Source: §B.3 · p. 11
  • Across later iterations, the agent pivots from transcription machinery and replication to ubiquitin-proteasome, zinc-finger, protease, and phosphatase families as new hits emerge.A CUL3/RBX1/UBE2M and PSMC2 probe yields six hits in one batch, while later ZSCAN, KLK, and PPP expansions produce additional discoveries.Source: §B.3 · pp. 11–12
  • After 1,000 tests, ICL-EF discovers 35 of 152 significant genes, achieving 23.0% recall through family- and pathway-based exploitation.The trace uses 100-gene batches across 10 iterations and records 8 initial hits before later expansions.Source: §B.3 · pp. 10–11

C ICBR-EF Agent: Prompting Strategy and Execution Trace

ICBR-EF extends binary feedback with phenotypic fingerprints and an evolving hypothesis register, allowing the agent to revise mechanistic beliefs as results accumulate. Its trace shows broad initialization, pruning of failed hypotheses, and cascading expansion across phenotypically related pathways.

C ICBR-EF Agent: Prompting Strategy and Execution Trace

  • ICBR-EF supplies target outcomes, top co-affected features, shared phenotypic signatures, and a JSON hypothesis register to guide mechanism-based selection.The register tracks confidence and status, while phenotypic fingerprints support clustering beyond binary hit/miss labels.Source: C ICBR-EF Agent: Prompting Strategy… · p. 12

C.1 System Prompt

  • The system prompt directs the agent to cluster genes by side-effect profiles, infer pathways from shared phenotypes, and update confidence through hypothesis testing.This explicitly shifts selection from gene-name patterns alone toward phenotype-based mechanism inference.Source: §C.1 · p. 12

C.2 Hypothesis Register Schema

  • The hypothesis register stores free-text mechanistic claims with confidence, status, and evidence-based reasoning, and the LLM replaces the array as evidence accumulates.The register begins with a neutral global-exploration hypothesis before later entries are added, weakened, or abandoned.Source: §C.2 · p. 13

C.3 Per-Iteration User Prompt

  • The per-iteration prompt combines phenotypic observations, the current register, and explicit exploitation-exploration instructions while requiring gene-list verification.Once at least three hits are available, recurring side effects are summarized as a shared phenotypic signature.Source: §C.3 · pp. 13–14

C.4 Execution Trace

  • Initial testing yields seven hits, after which the agent abandons several failed branches and refines its search toward chromatin, ubiquitin, transport, Mediator, and transcription hypotheses.Iteration 2 produces one UHRF1 hit; iteration 3 then yields TRIM28, MED27, SUPT6H, TAF1, and TAF5.Source: §C.4 · p. 14
  • Later iterations pivot from transcription machinery into RNA processing, mRNA stability, translation, ribosome biogenesis, proteasome, DNA replication, and DNA repair as surprise hits expand the hypothesis space.Iteration 10 returns zero hits after lower-priority candidates are swept.Source: §C.4 · pp. 14–15
  • The ICBR-EF family table records 20 family-assigned hits and 17 singleton hits, reflecting shallow family probing followed by rapid phenotypic-signature-based pivots.These pivots can cross families with limited direct functional or mechanistic continuity.Source: §C.4 · p. 14, Table 3 · p. 15
  • After 1,000 tests, ICBR-EF discovers 37 of 152 significant genes, corresponding to 24.3% recall across its adaptive trajectory.The trajectory expands from seven broad initial priors to more than 12 mechanistically distinct hypotheses by iteration 9.Source: §C.4 · pp. 14–15

D Additional statistical significance assessment of performance gaps

Additional analyses assess performance gaps with feature-level permutation tests and hierarchical bootstrap intervals. They support consistent gains over random selection and generally favor Sonnet 4.6 ICL methods over zero-shot, while providing weaker evidence for the ICBR-EF versus ICL-EF difference.

D Additional statistical significance assessment of performance gaps

  • Per-feature gaps are tested with paired two-sided sign-flip permutation tests, with the target feature treated as the inference unit.Replicates are averaged within feature before paired method comparisons, and reported p-values use Benjamini-Hochberg correction unless otherwise stated.Source: D Additional statistical significance… · p. 15
  • All investigated approaches, including Random FB, outperform random selection, while Sonnet 4.6 ICL approaches consistently outperform the zero-shot agent.The Random FB result is described as consistent with filtering noisy feedback while retaining prior knowledge.Source: D Additional statistical significance… · p. 15
  • The ICBR-EF versus ICL-EF advantage is not confirmed consistently at the per-feature level.The comparison remains borderline in the hierarchical bootstrap analysis, with the reported interval upper bound equal to zero.Source: D Additional statistical significance… · p. 15
  • Hierarchical bootstrap intervals resample features and within-feature replicates to quantify uncertainty around average feature-wise performance gaps.A significant comparison is defined by a 99% confidence interval that excludes zero.Source: D Additional statistical significance… · p. 15, Table 5 · p. 16

Related papers

Loading 2603.26177v1…