Source-linked AI summary
Automated Hypothesis Validation with Agentic Sequential Falsifications
Kexin Huang, Ying Jin, Ryan Li, Michael Y. Li, Emmanuel Candès, Jure Leskovec
TL;DR
Abstract hypotheses are difficult to validate directly, and LLM-generated hypotheses add scale and reliability challenges. POPPER uses agents to design and run sequential falsification tests with Type-I error control; across benchmarks it showed strong power and expert-comparable validation performance with faster completion.
Problem
Free-form hypotheses are difficult to translate into direct tests, while the volume and variable plausibility of LLM-generated hypotheses make reliable, scalable validation important.
Source: §1 · p. 1Method
POPPER uses LLM agents to design and execute experiments on measurable implications, combining sequential evidence with Type-I error control.
Source: §1 · pp. 1–2, §2.2 · p. 2Results
Across three benchmarks, POPPER had the highest power among methods controlling Type-I error; against human experts, performance closely matched and tasks ran 9.7 times faster.
Source: §4.1 · p. 7Takeaways & Limitations
The results support POPPER as a scalable, statistically rigorous approach to automated validation of free-form hypotheses.
Source: §6 · p. 9Failure analysis identified misinterpreted p-values (35.9%) and ineffective falsification experiment design (28.1%) as the leading failure modes.
Source: §4.2 · p. 8
Abstract
from arXiv · showhide
Hypotheses are central to information acquisition, decision-making, and discovery. However, many real-world hypotheses are abstract, high-level statements that are difficult to validate directly. This challenge is further intensified by the rise of hypothesis generation from Large Language Models (LLMs), which are prone to hallucination and produce hypotheses in volumes that make manual validation impractical. Here we propose Popper, an agentic framework for rigorous automated validation of free-form hypotheses. Guided by Karl Popper's principle of falsification, Popper validates a hypothesis using LLM agents that design and execute falsification experiments targeting its measurable implications. A novel sequential testing framework ensures strict Type-I error control while actively gathering evidence from diverse observations, whether drawn from existing data or newly conducted procedures. We demonstrate Popper on six domains including biology, economics, and sociology. Popper delivers robust error control, high power, and scalability. Furthermore, compared to human scientists, Popper achieved comparable performance in validating complex biological hypotheses while reducing time by 10 folds, providing a scalable, rigorous solution for hypothesis validation.
1. Introduction
POPPER addresses the challenge of validating abstract, free-form hypotheses at scale by having LLM agents design and execute falsification experiments on measurable implications. Its sequential testing framework controls Type-I error while accumulating evidence, and evaluations span six domains and compare its performance with experts.
1. Introduction
- Abstract natural-language hypotheses are difficult to evaluate directly and must be translated into measurable implications for rigorous experiments.The paper frames this as the challenge of rigorously validating free-form hypotheses at scale.Source: §1 · p. 1
- POPPER uses LLM agents to test measurable implications through sequential falsification experiments, including data analysis, simulations, real-world experiments, and interventions.The approach is inspired by Karl Popper’s principle of falsification.Source: §1 · pp. 1–2
- A design agent specifies falsifiable sub-hypotheses and experiments, while an execution agent carries them out and returns p-values for sequential evidence aggregation.The framework adapts its decisions as evidence accumulates while controlling Type-I error.Source: §1 · p. 2
- Across six domains, POPPER controlled Type-I error while achieving power improvements, and matched expert validation performance while reducing validation time by an order of magnitude.The expert study involved nine PhD-level biostatisticians and computational biologists.Source: §1 · p. 2
2. POPPER: a general framework for automated hypothesis validation
POPPER turns free-form hypotheses into sequences of experiments that test measurable implications, then aggregates evidence under conditions that support Type-I error control. Its framework combines agent-led experiment design and execution with sequential statistical validation.
2.1. Background and Problem Formulation
- A hypothesis describes relationships among variables in a context; its null specifies uninteresting data-generating distributions, and validation tests that null against the alternative.The task is represented as a function mapping a hypothesis to an unvalidated or validated status.Source: §2.1 · p. 2
2.2. Overview of POPPER
- POPPER repeatedly proposes and executes experiments on measurable implications, converting test p-values into e-values and continuing until evidence crosses a threshold or the test budget is reached.Experiments may analyze existing data, collect measurements, or run simulations.Source: §2.2 · pp. 2–3
2.3. Validity of Type-I Error Control in POPPER
- The validity conditions require sequentially valid evidence and a stopping time based on observed data; data selection must not use the selected dataset before testing.For actively collected data, the dataset is collected after the experiment is designed.Source: §2.3 · p. 4
- Under Assumptions 1–3, terminal aggregated evidence is a valid e-value, and rejecting when it reaches 1/α controls Type-I error at α.The result holds over randomness in the agents and collected data.Source: §2.3 · p. 4
2.4. Agentic hypothesis validation framework
- POPPER can operate wherever valid p-values can be produced, including laboratory experiments, real-time data collection, and computational simulations.The demonstrated implementation uses a static database.Source: §2.4 · p. 4
- The design agent proposes an implication, null and alternative, and feasible experiment; the execution agent runs it and records a failed attempt if the protocol or data prevents completion.Proposed experiments must be capable of yielding a valid p-value under the null sub-hypothesis.Source: §2.4 · p. 4
- A general p-to-e calibrator supports aggregation and adaptive continuation when likelihood-ratio constructions requiring modeling assumptions do not suit the agents’ flexibility.If cumulative evidence reaches 1/α, the null is rejected; otherwise testing continues until the budget is reached.Source: §2.4 · p. 5
3. Instantiation of POPPER
POPPER’s demonstrated deployment tests hypotheses using a static corpus of large, hypothesis-free datasets, while its general framework can also accommodate laboratory procedures, simulations, or data analyses. Metadata-only experiment design, relevance screening, and code-based execution structure the validation process.
3. Instantiation of POPPER
- POPPER’s deployment uses a static corpus of massive, hypothesis-free datasets, chosen for ease of implementation and reproducibility rather than real-time data acquisition.The broader framework is described as capable of testing sub-hypotheses through laboratory procedures, simulations, or data analyses.Source: §3 · p. 5
- The evaluation covers TargetVal biology and DiscoveryBench’s six domains, with 86 deduplicated non-null hypotheses grounded in peer-reviewed research.TargetVal aggregates 22 tables and about 85 million records; POPPER receives dataset schemas and the main hypothesis, then queries and analyzes raw data.Source: §3 · p. 5
- At each iteration, the Design Agent uses the main hypothesis, prior proposals and p-values, and database metadata to propose a new falsification experiment.It sees table schemas but not raw data or summary statistics, and self-refines proposals for novelty, implementability, and logical relevance.Source: §3 · p. 5
- Table 3 reports Type-I error and power across methods; power is invalid for methods failing the control criterion, and metrics summarize five independent runs.Control is assessed by whether the predefined threshold falls within one standard deviation of a method’s result.Source: Table 3 · p. 6
- A relevance checker discards proposed null sub-hypotheses below a predefined relevance threshold, mitigating the risk that an irrelevant falsification incorrectly supports the main hypothesis.The checker estimates how strongly each proposed null sub-hypothesis is implied by the main hypothesis.Source: §3 · pp. 5–6
- After relevance screening, the Execution Agent queries and analyzes raw data in Python, selecting statistical tests suited to the data distribution.Its ReAct cycle alternates code execution, output inspection, and reasoning, typically including dataset retrieval, preprocessing, statistical testing or modeling, and summarization.Source: §3 · p. 6
4. Experiments
Across the evaluated benchmarks, POPPER combines Type-I error control with higher power than methods that also control error, and its power rises as the test budget grows. In the expert study, its power and Type-I error were similar to human experts, while task completion was faster.
4. Experiments
- The evaluation measured Type-I error on permuted negative examples and power on positive examples, using nominal α = 0.1 and up to three or five tests by benchmark.The maximum was three tests on DiscoveryBench and five on TargetVal because its hypotheses were more complex.Source: §4 · p. 6
4.1. Results
- On DiscoveryBench, POPPER delivers 66.5% greater power than ReAct; on TargetVal-IL2, it outperforms Self-Refine by a factor of 3.17 among methods controlling Type-I error.POPPER also remains below the nominal Type-I error level across all datasets; removing the relevance checker significantly raises Type-I error.Source: §4.1 · pp. 6–7
- 9.7 times faster task completion, 3.6 times more lines of code, and 2.5 times more statistical tests accompanied power and Type-I error rates similar to human experts.The study recruited nine computational biologists and bioinformaticians for TargetVal-IL2; Figure 2 reports the expert comparison.Source: §4.1 · p. 7, Figure 2 · p. 7
- Claude Haiku 3.5 had high Type-I error, whereas Llama, GPT-4o, Sonnet, and o1 maintained reasonable error control; o1 performed best on DiscoveryBench, while power leaders varied by benchmark.GPT-4o excelled in DiscoveryBench power, whereas Sonnet led on TargetVal-IL2.Source: §4.1 · p. 7, Table 4 · p. 7
4.2. Analysis and Discussion
- In TargetVal, POPPER tested biological implications using up to 14 execution steps, selected statistical tests based on modeling assumptions, and often included negative controls.Examples include protein-interaction networks, expression correlations, eQTL, loss-of-function, and genetic perturbation tests; non-parametric tests were most frequent.Source: §4.2 · p. 7, Figure 3 · p. 8
- As the maximum number of tests increased, power rose while Type-I error remained well-controlled; Type-I error control also held across varied nominal levels α.Figure 4 presents empirical Type-I error across α levels and power and Type-I error across test budgets.Source: §4.2 · p. 8, Figure 4 · p. 8
- The ReleCheck agent’s ratings correlated with human judgments (Spearman’s ρ = 0.55, p = 5 × 10−6), but it overestimated strongly implied proposals: 85% versus humans’ 77%.Three authors rated 90 randomly selected proposals, with Kendall’s W = 0.91 after calibration; the authors call for further calibration and domain-specific expertise.Source: §4.2 · p. 8
- Misinterpreted p-values (35.9%) and ineffective experiment design (28.1%) were the leading identified failure reasons; hallucination accounted for 0.8%.Other identified reasons were tests breaking the implication (17.2%) and incorrect implementation (8.6%).Source: §4.2 · p. 8
5. Related Work
Prior work has generated, refined, or grounded LLM hypotheses, but few studies systematically test free-form hypotheses with rigorous statistical controls. POPPER combines sequential falsification with statistical error control to validate both LLM- and human-generated hypotheses.
5. Related Work
- LLM research has explored domain-specific hypothesis generation and comparisons between AI-generated and expert proposals (Wang et al., 2024a; Baek et al., 2024).Other work refines hypotheses or grounds them in datasets (Honovich et al., 2023; Wang et al., 2024c; Majumder et al., 2024).Source: §5 · p. 8
- Studies of LLM-driven experimental protocols and hypothesis-plus-code generation often lack strong error control (Tian et al., 2024; Gu et al., 2024; Li et al., 2024b; Lu et al., 2024; Ifargan et al., 2024; Majumder et al., 2024).Few works systematically test free-form hypotheses under rigorous statistical controls.Source: §5 · pp. 8–9
- Unlike Li et al. (2024a), POPPER targets free-form natural-language hypotheses and offers rigorous error control through sequential falsification.It conducts statistical validation of hypotheses generated by both LLMs and humans.Source: §5 · p. 9
6. Conclusion
The conclusion presents POPPER as a scalable framework for statistically rigorous validation of free-form hypotheses, while identifying test relevance and robust LLM implementations as challenges.
6. Conclusion
- POPPER combines sequential testing with automated experiment design and execution to validate free-form hypotheses at scale with statistical rigor.The paper describes this as an early exploration.Source: §6 · p. 9
- Refining test relevance and ensuring robust LLM implementations remain challenges for POPPER.Source: §6 · p. 9
- Future work could extend POPPER to control other error metrics, such as false discovery rate, broadening its utility in scientific discovery and beyond.Source: §6 · p. 9
Impact Statement
The impact statement links POPPER’s statistical validation and automation to risks from unsupported hypotheses and to access to scientific analysis across resource settings.
Impact Statement
- The statement identifies biology, economics, and social sciences as domains where hypothesis generation and validation matter for advancing knowledge.Source: Impact Statement · p. 9
- POPPER’s Type-I error control and statistical validation are presented as ways to mitigate risks from hallucinated or unsupported hypotheses.The statement says this helps direct research resources toward meaningful and plausible hypotheses and reduces potential wasted effort and false conclusions.Source: Impact Statement · p. 9
- By automating and accelerating validation, POPPER is described as potentially widening access to advanced analyses for smaller teams and resource-limited institutions.Source: Impact Statement · p. 9
A.1. Detailed algorithm for POPPER
Algorithm 1 initializes agents, thresholds, and experiment records, then iterates through proposed tests and returns a summary of validation outcomes.
A.1. Detailed algorithm for POPPER
- The algorithm takes a main hypothesis and dataset, initializes design, relevance-checking, execution, summarization, and coding components, and sets implication, alpha, test-count, and retry thresholds.Source: §A.1 · p. 13
- For each test, the design agent proposes an experiment using prior successful and failed tests; the relevance checker can reject a proposal below threshold.The algorithm also initializes success and failure sets, observations, and a completion flag.Source: §A.1 · p. 13
- The algorithm exits its inner loop when a retry sequence does not succeed, stops the outer loop when done, and returns a summary with accumulated evidence and successful and failed tests.Source: §A.1 · p. 13
A.2. Proof of Theorem 4
The proof shows that, under Assumptions 1 and 2, the sequential aggregate evidence is a super-martingale; optional stopping and Markov’s inequality then yield Type-I error control.
A.2. Proof of Theorem 4
- Under H0 and Assumptions 1 and 2, each e-value has conditional expectation at most 1 given prior data.The proof conditions on the LLM agents’ training process.Source: §A.2 · p. 13
- The aggregate evidence starts at E0 = 1 and is a super-martingale adapted to the filtration of observed data.The proof applies optional stopping to the terminal evidence at any stopping time adapted to that filtration.Source: §A.2 · p. 13
- The terminal evidence yields P(ŷ = 1) ≤ α under H0 by optional stopping and Markov’s inequality.The proof bounds the expected stopped evidence by 1, then applies the threshold 1/α.Source: §A.2 · p. 13
B. Full related works
The related work positions POPPER among philosophy-of-science approaches, LLM hypothesis generation and testing, and broader research automation. It distinguishes POPPER by its focus on statistically grounded validation of free-form hypotheses.
Philosophy of science
- POPPER’s philosophical framing links falsification and observable implications with auxiliary assumptions, adaptive testing, and statistical safeguards.The discussion also notes tensions with Kuhn and Lakatos and the framework’s abstraction of sociotechnical dimensions.Source: B. Full related works · p. 14
LLM for hypothesis generation
- Prior LLM hypothesis-generation work develops ideas or hypotheses, while POPPER takes human- or LLM-generated hypotheses through data-driven statistical evaluation.The cited work spans creative research ideas, commonsense hypotheses, and hypotheses refined from examples or datasets.Source: B. Full related works · p. 14
LLM for hypothesis testing and experiments
- POPPER targets rigorous validation of free-form natural-language hypotheses, unlike work focused on experiment-code generation or predefined models.The passage contrasts its sequential falsification framework and Type-I error control with automated discovery work said to lack rigorous statistical grounding.Source: B. Full related works · p. 14
LLM for automating research
- Other LLM research-automation applications include review generation, literature curation, outcome prediction, and future-work recommendations; POPPER focuses on hypothesis testing.Source: B. Full related works · p. 14
C. Limitations
Type-I error control does not ensure that discoveries are true, particularly when many hypotheses are tested. Family-wise error-rate and false discovery rate extensions are described, but their evaluation is left to future work.
C. Limitations
- Type-I error control does not guarantee true discoveries, especially when validating many hypotheses; if all are null, every rejection is false.The authors urge caution in treating POPPER-validated hypotheses as true discoveries to act upon.Source: C. Limitations · p. 15
- Family-wise error-rate control via Bonferroni and false discovery rate control via eBH are proposed extensions, but their evaluation is outside this work’s scope.The authors leave further development of these capabilities for future investigation.Source: C. Limitations · p. 15
D. Error analysis
The error analysis categorizes 128 failed experiment logs across three benchmarks using O1 labels checked by humans. Misinterpreting p-value context was the most frequently reported failure mode.
D. Error analysis
- 28.1% of failures involved ineffective falsification tests, while 17.2% involved tests that broke the implication.Source: D. Error analysis · p. 15
- The analysis categorized 128 failed logs from TargetVal-IFNG, TargetVal-IL2, and DiscoveryBench; O1 labels matched human judgment in 93.3% of 30 checked logs.The ten failure categories are defined in Table 5.Source: D. Error analysis · p. 15, Table 5 · p. 16
- 35.9% of failures involved misinterpreting p-value context, the most frequently reported failure mode.Figure 5 presents the failure-mode distribution; O1 labels were manually checked for quality assurance.Source: Figure 5 · p. 15, D. Error analysis · p. 15
E. Tests and trajectory analysis
The analysis categorizes the statistical and biological tests used in falsification experiments and summarizes execution-agent trajectories. It describes the agent’s workflow and the sampling used to analyze both.
Test categorization
- The analysis sampled 1,500 falsification test designs and execution logs and used GPT-4o to group the statistical tests performed.For biological tests, it sampled 462 proposals and used GPT-4o to group them into standardized tests; this analysis was limited to biology because the other five domains had too few unique hypotheses.Source: E. Tests and trajectory analysis · p. 16
Agent trajectories
- Trajectory analysis used 11 high-level actions, developed after manual inspection of 20 experiments, to code 80 randomly sampled execution-agent trajectories.GPT-4o converted each sampled trajectory into an action list using the definitions in Table 6.Source: E. Tests and trajectory analysis · p. 16, Table 6 · p. 17
- The agent’s workflow begins with dataset inspection, then cycles through test implementation, execution, and error resolution before summarizing findings.It may check assumptions and sample sizes and refine its approach before reaching a conclusion.Source: E. Tests and trajectory analysis · pp. 16–17
- The action definitions include fitting statistical or machine-learning models, summarizing a hypothesis verdict, and actions outside the other categories.Table 6 names and defines the experiment execution agent’s actions.Source: Table 6 · p. 17
F. Human study details
The human study describes participant backgrounds and the benchmark tasks and conditions used for comparison. A separate annotation study reports how annotators calibrated and rated proposed falsification tests.
F. Human study details
- Nine of 11 recruited computational biologists and bioinformaticians adhered; six held or were pursuing PhDs, and six reported expert genetic or genomic data-analysis experience.Two participants were beginners and one intermediate in data analysis; five reported expert familiarity with statistical hypothesis testing, and six had never performed wet-lab experiments.Source: F. Human study details · p. 17
- The human evaluation sampled 18 TargetVal-IL2 tasks, split evenly between Type-I error and power evaluation, and tested each hypothesis at α = 0.1.Participants received two randomly assigned tasks and could use the internet or LLMs for general coding and statistical questions, but not to query the specific hypothesis.Source: F. Human study details · p. 17
G. Human Annotation Details
- Annotator agreement rose from Kendall’s W of 0.62 before calibration to 0.91 afterward; the relevance checker’s rankings correlated with human annotations at Kendall’s Tau 0.43 and Spearman’s 0.55.The relevance checker ranked 84% of proposals strongly relevant, compared with 77% for human annotators.Source: G. Human Annotation Details · p. 18
H. Qualitative Analysis
The qualitative examples show a successful sequence of falsification experiments and a false-positive trajectory. Together, they illustrate both sequential evidence gathering and a challenge in interpreting test results in context.
Successful trajectory
- Four diverse experiments on a TargetVal-IL2 hypothesis produced cumulative e-values that crossed the α = 0.1 threshold, leading POPPER to reject the main null hypothesis.The tests included expression correlation, regulatory-network analysis, variant–immune-phenotype association, and an eQTL test.Source: H. Qualitative Analysis · p. 18
- After an initial dataset-inspection error, POPPER reached the correct conclusion through further trials, while the agents refined experiment proposals and checked test validity.The execution agent also inspected data sources and model assumptions; the sequential framework provided tolerance for test execution failures.Source: H. Qualitative Analysis · pp. 18–19
Failure case
- A false positive arose when a neutrophil RAB39A eQTL p-value was treated as evidence about IL-2 regulation, exposing difficulty reasoning about effect-size and p-value context.The proposed experiment was valid, but its implementation violated the implication assumption.Source: H. Qualitative Analysis · p. 19
I. Prompting Details
This section presents prompting workflows and examples for sequential falsification of proposed IL-2 regulatory hypotheses, showing how agents propose, assess, refine, and interpret tests across GRAP2, LCP2, and RAB39A. The examples trace varied test outcomes, with conclusions constrained by the evidence observed.
Prompting Details
- The coding agent proposes a data-answerable falsification test whose sub-hypothesis is null whenever the main hypothesis is null, while avoiding redundant tests and targeting strong implications.The prompt requires a theoretically grounded p-value and relevant use of available data.Source: §I · p. 23
- In successive rounds, a critic checks validity, redundancy, failed-test overlap, and data feasibility, then the agent refines or replaces the test until it is acceptable.The process begins with an initial proposal and permits returning a final test definition when it is good enough.Source: §I · p. 23
- A relevance checker scores each sub-hypothesis test from 0.1 to 1.0 using evidence strength, mechanistic connection, and predictive value, with a brief justification.The rubric ranges from irrelevant tests with no predictive value to highly relevant tests offering direct evidence or clear mechanistic insight.Source: §I · p. 24
- The summarizer reports the main hypothesis, sequential-test results, reasoning, and a True/False conclusion, adhering strictly to observations even when they fail to reject the null.The prompt explicitly requires a False conclusion when experiments fail to reject the null hypothesis.Source: §I · p. 24
- The e-value agent estimates P(data|h_1) and P(data|h_0), then critiques and refines both estimates for theoretical rigor and accuracy.It returns final values once estimates meet a rigorous standard, such as a rubric score of 0.9 or 1.0.Source: §I · p. 25
I. Prompting Details
- The initial falsification test correlates GRAP2 and IL-2 TPM across GTEx tissues, but correlation alone cannot establish causation or provide a complete mechanism.The proposal treats significant nonzero correlation as consistent with regulatory coupling and lack of correlation as evidence against the hypothesis.Source: §I · pp. 27–28
- 3.100e-02 was the permutation-based correlation p-value for GRAP2 and IL-2 expression, while sequential testing concluded there was insufficient evidence and should continue.The sequential statistic was 2.8398091712353244.Source: §I · p. 28
- A proposed high-versus-low IL-2 tissue contrast compared GRAP2 expression with a two-sample test, but the proposal was flagged as insufficiently relevant and described as only moderate associative evidence.The contrast was framed as a tissue-specific check, not direct mechanistic evidence.Source: §I · p. 30
- The subsequent Essential Expression Overlap Test proposed a 2x2 presence/absence table for GRAP2 and IL-2 across tissues and Fisher’s exact test.Its premise is that no tissue should express IL-2 without GRAP2 if GRAP2 expression is necessary for IL-2 expression.Source: §I · p. 31
- The test classifies tissue-level GRAP2 and IL-2 expression as present or absent, then uses Fisher’s exact test to assess whether IL-2 is absent whenever GRAP2 is absent.Finding IL-2 expression in tissues without GRAP2 would count against the proposed necessary-expression pattern.Source: §I · pp. 31–32
- Because the test uses tissue-level expression patterns, its results could influence confidence in GRAP2 necessity but would not provide definitive mechanistic proof.The passage characterizes the approach as correlational rather than mechanistic.Source: §I · p. 32
- The Essential Expression Overlap Test returned p-value 1.832e-01, and the sequential test found insufficient evidence to decide and continue.The sequential statistics were 3.317389204898911, with p-values [0.031, 0.1832].Source: §I · p. 34
- A proposed IL-2 Pathway Co-expression Test would correlate GRAP2 across tissues with IL-2 receptor and signaling genes, including IL2RA, IL2RB, JAK3, and STAT5.It adds a pathway-focused angle rather than repeating tests of GRAP2 correlation with IL-2 itself or GRAP2/IL-2 presence and absence.Source: §I · pp. 34–35
- The earlier test correlated GRAP2 with IL-2 signaling genes across tissues, but this was judged only indirectly informative because correlation does not establish a functional regulatory link.The critique noted that knockdown or overexpression would provide a more direct functional test.Source: §I · p. 36
- The proposed test models IL2 expression from GRAP2 while controlling for housekeeping-gene expression; a non-significant association tends to falsify the hypothesis, while a significant one supports it.The null specifies zero partial correlation after controlling for housekeeping expression; the alternative specifies a significant association.Source: §I · p. 38
- The housekeeping-normalized partial-correlation proposal passed the relevance check with a score of 1.0, because it tests GRAP2–IL-2 association beyond general transcription.The evaluation characterized a significant partial correlation as evidence for or against a specific regulatory link.Source: §I · p. 38
- Controlling for GAPDH, partial correlation was 0.3764 with permutation-based p-value 2.80e-02; after three tests, sequential testing returned insufficient evidence—CONTINUE.The partial-correlation test regressed GAPDH from GRAP2 and IL2 expression across tissues and used 500 permutations.Source: §I · pp. 40–41
- The random-gene test compares GRAP2–IL2 correlation across tissues with correlations between IL2 and sampled genes; indistinguishable correlation falsifies, while stronger correlation supports but does not prove regulation.The final test definition proposes sampling 100–1000 genes and estimating a p-value from the fraction of random-gene correlations exceeding GRAP2–IL2’s correlation.Source: §I · p. 42
- The random-gene test reported GRAP2–IL2 correlation 0.35043 and p-value 5.60e-02; after four p-values, sequential statistic 20.944182166241685 yielded PASS.The four p-values were [0.031, 0.1832, 0.028, 0.056].Source: §I · p. 44
- The earlier GRAP2 analysis passed overall, with an E-value of 20.944182166241685, and the sequential tests failed to falsify the hypothesis.The conclusion was that the hypothesis that GRAP2 regulates IL-2 production was not falsified.Source: §I · p. 44
- Across the four GRAP2 tests, statistics were 0.031, 0.1832, 0.028, and 0.056; the account described partial correlation as significant and the random-gene control as borderline.The essential-expression-overlap test did not appear to reject its null sub-hypothesis.Source: §I · p. 44
- The revised LCP2 proposal tested positive expression correlation with IL-2 across tissues, but its stated limitation was that correlation alone cannot establish regulation.The proposal was framed as a logically valid, implementable test using available expression data, although those required genes were ultimately absent from the provided dataset.Source: §I · p. 46
- The LCP2 expression-correlation test could not be performed because the genes were absent from the provided GTEx data; its default p-value was 1.00e+00 and sequential testing returned CONTINUE.The report states that the current dataset lacked expression levels for both genes across tissues.Source: §I · p. 47
- The LCP2 interaction proposal was refined to examine high-confidence physical interactions with established IL-2 transcriptional regulators or upstream signaling proteins, but it did not directly measure IL-2 production.The proposal received a relevance score of 0.8; its limitation was that interaction evidence alone would not complete validation of a regulatory relationship.Source: §I · pp. 48–49
- Among 3 LCP2 physical interactions, 0 involved IL-2 pathway genes; the binomial test gave p = 1.00e+00, indicating no significant enrichment or depletion.The analysis failed to reject the null hypothesis.Source: §I · p. 51
- The proposed GeneBASS follow-up tested whether LCP2 loss-of-function and missense variants associate with autoimmune and T-cell phenotypes, but it remained associative rather than a direct mechanistic test.The proposal specified p > 0.05 under the null and p < 0.05 under the alternative, and received a relevance score of 0.8.Source: §I · pp. 52–53
- The LCP2 pLoF and missense datasets each contained 1423 entries, a matching count the agent flagged as unusual.Source: §I · p. 53
- The strongest reported immune-related pLoF association was with lymphocyte percentage (Pvalue 0.000076; Pvalue_Burden 0.000067; Pvalue_SKAT 0.000937).The reported BETA_Burden was 0.014855 and SE_Burden was 0.003725.Source: §I · p. 54
- For LCP2 variants and immune phenotypes, the minimum p-value of 6.66e-05 became 9.42e-02 after Bonferroni correction, so the null could not be rejected at 0.05.The reported lymphocyte-percentage association had positive BETA_Burden of 0.014855 and t-statistic 3.988.Source: §I · p. 55
- The sequential test remained at “insufficient evidence - CONTINUE” after p-values [1.0, 1.0, 0.0942] and a summarized statistic of 0.40727200778192163.Source: §I · p. 55
- The revised falsification test compared LCP2 expression in immune tissues with non-immune tissues using GTEx data, after an eQTL–IL2 regulatory-region proposal was criticized for causal ambiguity and unavailable annotations.The refined test was judged feasible with the available GTEx data, but its expression pattern would provide circumstantial support rather than directly establish regulation of IL-2 production.Source: §I · pp. 56–57
- The GTEx dataset contained 54 LCP2 tissue entries, and the test treated Whole Blood, Spleen, and EBV-transformed lymphocytes as immune-related tissues.Source: §I · pp. 57–59
- Mann–Whitney U test statistic was 152.0 with p-value 8.06e-05; median expression was 63.6943 in immune tissues versus 4.2643 in non-immune tissues.The immune group comprised Whole Blood, Spleen, and EBV-transformed lymphocytes.Source: §I · p. 59
- The agent rejected a RAB39A tissue-expression proposal because it might remain positive under the main null, lacked granular tissue data, and overlapped an earlier test.It replaced the proposal with a UK Biobank eQTL test, judged to satisfy the nullity, data-availability, and nonredundancy requirements; the test provides correlational rather than direct causal or molecular-mechanism evidence.Source: §I · pp. 62–63
- RAB39A’s sequential test passed after 3 tests with combined e-value approximately 1.01 x 10^6, alongside significant neutrophil eQTLs with a strongest p-value of 2.81e-13.Multiple independent variants near RAB39A showed eQTL effects in neutrophils.Source: §I · p. 65
I. Prompting Details
- The sequential procedure passed with combined e-value >10ˆ6, driven by Test 3 QTL evidence (p=2.81e-13), and rejected the null hypothesis.The procedure concluded that there was sufficient evidence to pass.Source: §I · p. 65
- The first two tests showed modest or no associations, while Test 3 provided extremely strong QTL evidence for associations between RAB39A variants and IL-2 traits.The QTL evidence was described as compelling statistical support.Source: §I · p. 65
- The conclusion that RAB39A regulates IL-2 production was based strictly on the sequential testing procedure’s formal rejection of the null hypothesis.The procedure rejected the null in favor of the alternative hypothesis.Source: §I · p. 65
Related papers
This paper cites
Cited by
- SciGame: Let Agents Play the Scientific Discovery Games in the Real and Counterfactual World2026
- Same Claim, Different Interpretations: How LLM Agents Resolve Ambiguity in Scientific Claims2026
- Open-ended Scientific Discovery with Possibilistic Reasoning2026
- HypoEvolve: Genetic Algorithms Enable Multi-Agent LLMs to Discover Scientific Hypotheses2026
- EvoSCM: Scientific Belief Revision Through Causal Model Evolution and Experimentation2026
- AgentFold: Closed-Loop Agentic Search for Protein Folding Model Design2026