PaperlayerView exact revision on arXiv

Source-linked AI summary

Autonomous Scientific Discovery via Iterative Meta-Reflection

Bingchen Zhao, Sara Beery, Oisin Mac Aodha

arXiv:2607.01131v1cs.CVcs.AI

TL;DR

Existing discovery systems are limited by predefined questions, constrained hypothesis spaces, and weak synthesis of accumulated evidence. DiscoPER performs open-ended, code-based multimodal exploration with statistical validation and periodic meta-reflection, improving the recall and diversity of supported discoveries across ecological and causal benchmarks.

  • Problem

    Existing LLM-based discovery systems lack fully open-ended inquiry from raw data, arbitrary executable hypotheses, and autonomous selection of what to investigate next.

  • Method

    DiscoPER uses a propose–evaluate–reflect loop in which LLM agents generate executable statistical tests, retain validated claims, and periodically identify gaps and confounds to redirect exploration.

  • Results

    DiscoPER improves both the recall and diversity of supported discoveries over systems constrained by fixed edge spaces, predefined questions, or short-term memory.

  • Takeaways & Limitations

    The results support combining open-ended exploration, empirical validation, and reasoning over accumulated evidence in autonomous scientific discovery.

  • Takeaways & Limitations

    DiscoPER cannot discover phenomena unsupported by its input data, and citizen-science biases make human verification of proposed discoveries essential.

Abstract

from arXiv · show

Autonomous scientific discovery systems offer the potential to accelerate research by automating the process of hypothesis generation and validation. However, current systems operate within constrained search spaces or require predefined research questions, limiting their capacity for true open-ended inquiry. Furthermore, while they generate hypotheses iteratively, they largely lack the ability to explicitly synthesize their own accumulated findings to uncover complex, interconnected phenomena. We introduce DiscoPER, an autonomous large language model-powered framework that conducts open-ended research by dynamically generating and executing code to explore datasets without pre-specified research objectives. To ensure rigorous scientific validity, every proposed discovery must pass statistical testing. To overcome the limitations of isolated search, our framework introduces a second-order reasoning mechanism that periodically analyzes its own accumulated discoveries. By treating prior discoveries as empirical data, DiscoPER identifies structural patterns, confounds, and epistemic gaps, actively redirecting hypothesis exploration toward uncharted regions of the search space. The search space is further expanded by incorporating tool use, enabling the system to explore hypotheses beyond structured metadata by seamlessly processing and extracting useful information from multimodal sources like images. Evaluated on iNatDisco, a new multimodal ecological knowledge benchmark with pattern-level ground truth obtained from peer-reviewed literature, DiscoPER recovers 8 of 9 known patterns with a 72.7% hypothesis support rate, outperforming both classical causal discovery and LLM-guided baselines. Ablations show that DiscoPER scales with more data, and confirms the benefits of second-order meta-reflection.

1 Introduction

Existing LLM-based discovery systems generate hypotheses independently and remain constrained by predefined questions or limited hypothesis forms. DiscoPER frames open-ended discovery as a PROPOSE–EVALUATE–REFLECT loop in which agents generate executable statistical analyses and iteratively search based on accumulated findings.

  • Limitations of Existing Systems: Most existing LLM-based systems generate each hypothesis independently, without surveying prior findings or reasoning about what remains untested.Their loop receives a dataset or research question, generates a hypothesis, and then resets.
  • Limitations of Existing Systems: Classical structure-learning methods cannot express higher-order patterns, while full-pipeline AI scientist systems require initial research objectives or seed hypotheses.The cited higher-order examples include interactions, mediation chains, and confound structures.
  • Open-Ended Discovery: Fully open-ended discovery requires starting from raw unstructured data, expressing arbitrary testable hypotheses as executable code, and autonomously choosing subsequent investigations.The system must decide what to investigate next based on what it has already discovered.
  • DiscoPER: DiscoPER satisfies these requirements by having LLM agents propose hypotheses without a priori targets and realize each hypothesis as executable code invoking statistical tools.Claims are grounded in formal tests on real data rather than LLM confidence estimates.
  • DiscoPER: DiscoPER formalizes open-ended discovery as a generalized PROPOSE–EVALUATE–REFLECT loop and instantiates it as a multimodal, code-based system validated on held-out splits.Existing systems are described as restricted instances that lack flexible hypothesis spaces, unrestricted generation, or reflective iterative search.

2 Related work

Prior discovery systems either search predefined spaces, depend on external research direction, or lack structured reflection over accumulated findings. DiscoPER addresses these limitations through data-first exploration, held-out statistical validation, and reflective accumulation to guide future discovery.

  • Causal discovery and structure learning: Traditional causal-discovery methods search predefined variable-edge spaces, while recent LLM-based methods aim to improve search efficiency or inject domain knowledge.The passage identifies constraint-based, score-based, continuous-optimization, and LLM-augmented approaches as related work.
  • Autonomous scientific discovery systems: Existing autonomous discovery systems differ in how they source hypotheses and whether they validate them against data.The AI Scientist operates on machine-learning tasks and uses simulated review, whereas the AI CoScientist generates literature- and domain-originated biomedical hypotheses.
  • Autonomous scientific discovery systems: LLM-driven hypothesis-generation systems use methods including Bayesian surprise, code-executed transformation discovery, and falsification experiments, but some remain largely linear.These approaches focus specifically on generating and testing hypotheses through evidence collection or code execution.
  • Autonomous scientific discovery systems: All surveyed systems either require external research direction, lack structured reflection over accumulated findings, or both.The paper positions its approach as data-first, grounded in held-out statistical validation, and based on reflective accumulation.
  • Benchmarks for scientific discovery: Recent scientific-discovery benchmarks evaluate dynamic, code-driven agents, but typically use predefined research questions or highly constrained search environments.The passage cites BioDSA, HeurekaBench, BLADE, and DiscoveryBench as examples of structured evaluation environments.

3 Method

DiscoPER frames autonomous scientific discovery as an iterative loop that generates executable hypotheses, statistically evaluates them, and reflects over accumulated findings to guide subsequent exploration. Its open-ended formulation combines an unrestricted code-based hypothesis space with no required prior information and explicit meta-level reflection.

  • Setup: A hypothesis is an executable program over dataset X that returns supported or rejected together with statistical evidence, within a hypothesis space H of expressible programs.The setup allows optional accompanying data, such as images, and prior information P ranging from none to specific research questions.
  • Discovery loop: Each iteration proposes K hypotheses from current claims and guidance, evaluates them, and uses reflection to update guidance for the next proposal round.Systems without reflective reasoning set G_t = ∅, whereas DiscoPER uses updated guidance to steer subsequent hypothesis generation.
  • PROPOSE: PROPOSE generates structured natural-language hypotheses with variables and Python statistical-test code from tabular summaries, optional metadata, claims, and guidance.The method supports multimodal hypothesis generation when additional metadata such as images is available.
  • EVALUATE: EVALUATE executes each hypothesis on training and held-out validation splits using statistical tests, accepting claims only when they satisfy the stated validation criteria.Tests include correlation analysis, group comparison, predictive modeling, clustering with enrichment testing, and stratified subgroup re-analysis.
  • Hypothesis space and novelty: DiscoPER’s code-based hypothesis space can encompass all Turing-computable statistical tests, unlike classical edge-based discovery and code-based systems requiring externally supplied questions or target patterns.This formulation uses open code, P = ∅, and explicit reflection, supporting open-ended discovery.
  • REFLECT: REFLECT periodically analyzes accepted and rejected claims collectively to identify explored and missing areas and redirect the search, unlike concurrent systems that do not revise direction from accumulated findings.The reflection agent produces structured guidance that prompts PROPOSE to explore different aspects of the dataset.

4 Experiments

DiscoPER is evaluated on open-ended multimodal ecological discovery using literature-based pattern recall and held-out hypothesis support. It outperforms constrained baselines, scales with data, and uses REFLECT to broaden and ground its search in observed data.

  • Evaluation metrics: DiscoPER evaluates discovery with pattern recall and hypothesis support rate, the fraction of proposed hypotheses passing held-out statistical validation.The benchmark uses peer-reviewed literature annotations for ecological patterns.
  • Benchmark results: Classical causal methods recover at most 1 of 9 iNatDisco-800 patterns, while guided LLM methods recover 3/9.Causal methods cannot express interaction effects, mediation chains, and multi-variable ecological patterns dominating the ground truth.
  • Scaling behavior: More data makes subtle cross-kingdom patterns more discoverable and increases the number of supported patterns recovered.On iNatDisco-50K, scaling dataset size improves recall and supported insights.
  • Counterfactual evaluation: REFLECT increases hypotheses grounded in observed input data rather than only model priors, including on the counterfactual iNatDisco-800-CF dataset.Supported discoveries remain data-grounded even when the counterfactual data reverse relationships known to the model.
  • Reflection analysis: REFLECT reduces simple pairwise hypotheses from 92% to 69% and introduces interaction, correlation, seasonal, and visual hypotheses.Interaction hypotheses shift from 0% →2%, correlation tests from 0% →2%, and “X peaks in Y” patterns from 6% →14%.

5 Conclusion · Appendix · A Additional results

DiscoPER addresses autonomous scientific discovery as a process requiring open-ended exploration, empirical validation, and reasoning over accumulated evidence. The framework formulates and tests hypotheses from multimodal data while reflecting on its claims to identify gaps, confounds, and compound insights.

  • 5 Conclusion: DiscoPER is a code-driven framework for autonomous scientific discovery.It formulates testable, open-ended hypotheses from multimodal data.
  • 5 Conclusion: Autonomous scientific discovery requires open-ended exploration beyond proposing plausible hypotheses.The conclusion identifies empirical validation and reasoning over accumulated evidence as additional requirements.
  • 5 Conclusion: DiscoPER validates proposed discoveries on held-out data splits.This supports empirical validation of its testable hypotheses.
  • 5 Conclusion: Periodic self-reflection enables DiscoPER to identify gaps and confounds in its accumulated claims.The framework reasons over prior findings rather than treating hypotheses as isolated proposals.
  • 5 Conclusion: DiscoPER also derives compound insights from its accumulated claims.This extends the framework’s reasoning beyond individual discoveries.
  • Appendix: The supplied conclusion passage presents multimodal data as part of DiscoPER’s discovery process.No additional appendix or additional-results findings are provided in the supplied passages.
  • 5 Conclusion: The supplied passages include acknowledgements of support from the Royal Society Research Grant and Schmidt Sciences AI2050 Early Career Fellowship.They also acknowledge an NSF CAREER Grant, the NSF and NSERC AI Biodiversity Change Global Center, and associated award numbers.

A.1 Additional ablations · A.2 Classical causal discovery benchmarks

Ablations show that DiscoPER’s discovery performance depends substantially on the backbone LLM while remaining controllable through optional user context. On standard causal-discovery benchmarks, DiscoPER achieves the highest reported SACHS edge-recovery F1 and remains competitive on ASIA.

  • A.1 Additional ablations: Opus 4.6 obtains a 76.5% support rate but recovers only 4/9 patterns, indicating more conservative search behavior.The passage contrasts its higher validation fraction with exploration of fewer benchmarks patterns.
  • A.1 Additional ablations: DiscoPER can be steered by optional user context without modifying its underlying discovery loop.The evaluated contexts include a small set of prior facts or a single broad research interest.
  • A.1 Additional ablations: The model ablation evaluates recall across four LLMs, while the controllability ablation measures topic adherence to a user-specified interest.Table A1 reports these left- and right-panel analyses on iNatDisco-800.
  • A.2 Classical causal discovery benchmarks: DiscoPER is evaluated against PC, GES, NOTEARS, DAG-GNN, and GOLEM on SACHS and ASIA using edge-recovery F1 against validated ground-truth DAGs.SACHS contains 11 proteins and 17 edges; ASIA contains 8 variables and 8 edges, with DiscoPER tested with and without REFLECT.
  • A.2 Classical causal discovery benchmarks: 0.83 is DiscoPER’s highest reported edge-recovery F1 on SACHS, where it outperforms the compared classical structure-learning methods.The passage states that DiscoPER is competitive on ASIA, but the supplied excerpt does not provide its numeric F1.

A.3 Synthetic visual benchmark · B Additional dataset details

The synthetic visual benchmark isolates vision from memorized ecological knowledge by using novel programmatically generated images with controlled metadata and image-grounded patterns. On this benchmark, DiscoPER recovered 3 of 8 ground-truth patterns at 100 iterations, with a 54.2% support rate.

  • A.3 Synthetic visual benchmark: Table A2 reports edge-recovery F1 results for classical causal-discovery benchmarks, where higher scores are better.The classical-method results come from the respective publications.
  • A.3 Synthetic visual benchmark: The benchmark isolates visual reasoning from memorized ecological knowledge using novel colored shapes rather than real species.Its patterns cannot be proposed from prior ecological knowledge alone.
  • A.3 Synthetic visual benchmark: The dataset contains 5,000 programmatically generated images combining visual variables with tabular metadata.Visual variables include color, shape, size, texture, background, and count; metadata include category, region, season, temperature, and elevation.
  • A.3 Synthetic visual benchmark: Eight ground-truth patterns span metadata-only, vision-only, and mixed evidence settings.Some patterns are discoverable only through image analysis, such as red shapes being more common in autumn.
  • A.3 Synthetic visual benchmark: 100 iterations yielded 3 of 8 recovered ground-truth patterns with a 54.2% support rate.Both metadata-only patterns were reliably discovered.
  • A.3 Synthetic visual benchmark: DiscoPER recovered one vision-only color-season pattern by leveraging a vision-language model during hypothesis generation.This result demonstrates image-based feature extraction within the hypothesis-testing pipeline.
  • A.3 Synthetic visual benchmark: Figure A1 presents representative benchmark images whose shapes vary in color, size, and texture across sky, grass, water, and sand backgrounds.The figure illustrates the controlled visual variation used in the benchmark.

B.1 iNatDisco benchmark construction

The iNatDisco benchmarks use iNaturalist research-grade observations and peer-reviewed ecological literature to evaluate recovery of curated ecological patterns. They comprise datasets of different scales and species coverage, with recall based on validated claims recovered from the benchmark observations.

  • Benchmark datasets: iNatDisco-800 contains 800 observations across 8 species with 9 ground-truth patterns, while iNatDisco-50K contains 50,000 observations across 9,776 species with 12 patterns.Both benchmarks use the same image set as INQUIRE, and their ground-truth patterns were curated from peer-reviewed ecological literature.
  • Evaluation protocol: Recall measures whether DiscoPER proposes validated claims that recover curated patterns using the benchmark’s observation data and available statistical tools.Some literature-derived patterns require species-level specificity or other conditions that iNaturalist data cannot always satisfy.

B.2 iNatDisco-800-CF: counterfactual dataset construction · C Additional implementation details

iNatDisco-800-CF is constructed by reversing five established ecological relationships in iNatDisco-800, with each counterfactual statistically verified. Its held-out validation rejects hypotheses suggested by prior knowledge when the modified data supports the opposite pattern.

  • B.2 iNatDisco-800-CF: counterfactual dataset construction: iNatDisco-800-CF modifies iNatDisco-800 to reverse five well-established ecological relationships.For each relationship, the underlying data is altered so the original pattern disappears and the reversed pattern holds.
  • B.2 iNatDisco-800-CF: counterfactual dataset construction: Each counterfactual is verified to be statistically present in the modified dataset.The evaluation uses modified data in which the intended counterfactual relationship is statistically supported.
  • B.2 iNatDisco-800-CF: counterfactual dataset construction: CF1 removes a significant seasonal shift: bird latitude and month have r = 0.02, p = 0.66.This result illustrates that the modified dataset no longer contains the original seasonal relationship.
  • B.2 iNatDisco-800-CF: counterfactual dataset construction: CF2 produces a spring concentration, with 77% of fungal observations falling in spring months.The fungal seasonal pattern is altered so spring becomes the dominant observation period.
  • B.2 iNatDisco-800-CF: counterfactual dataset construction: CF4 narrows mammal latitude variation to 2.5◦, compared with 14.8◦ for insects.The counterfactual therefore changes the relative latitude dispersion between mammals and insects.
  • B.2 iNatDisco-800-CF: counterfactual dataset construction: Held-out statistical validation prevents the system from accepting patterns absent from the modified data.LLM-generated prior-based hypotheses such as “fungi peak in autumn” are rejected when the data shows the opposite.

C.1 Evaluation protocol: LLM-as-a-judge for recall … C.5.2 Experiment planning (EVALUATE)

The paper evaluates open-ended discoveries through semantic, statistically validated matching and compares DiscoPER with controlled LLM and causal-discovery baselines. Its prompts and experiment pipeline enforce novel hypothesis generation, reflective guidance, tool selection, and claim–code alignment.

  • C.1 Evaluation protocol: LLM-as-a-judge for recall: C.1 Evaluation protocol: LLM-as-a-judge for recall: Semantic judging matches supported claims to ground-truth mechanisms despite different wording, species exemplars, or generality levels.Only supported claims are evaluated, ensuring recall requires both a proposed pattern and code validated on held-out data.
  • C.1 Evaluation protocol: LLM-as-a-judge for recall: C.1 Evaluation protocol: LLM-as-a-judge for recall: 8/9 is the reporting format for recall, with a pattern discovered when its highest supported-claim score is ≥1.Scores represent exact, partial, or missed semantic matches, and recall uses the number of discovered patterns over peer-reviewed patterns.
  • C.1 Evaluation protocol: LLM-as-a-judge for recall: C.1 Evaluation protocol: LLM-as-a-judge for recall: 95% agreement with manual assessment was observed for the judge’s semantic score-2 matches across primary experiments.Claude Sonnet 4.5 served as the structured-JSON judge across experiments and baselines.
  • C.2 LLM-based baseline construction: C.2 LLM-based baseline construction: 3/9 patterns were recovered by both HeurekaBench-like and ExperiGen-like baselines on the standard or iNatDisco-800 settings.The baselines respectively provide full or partial guidance, but fixed questions and generic framing miss compound cross-taxon patterns.
  • C.2 LLM-based baseline construction: C.2 LLM-based baseline construction: Baseline comparisons hold the backbone LLM and held-out validation protocol constant, while HeurekaBench-like receives questions corresponding directly to ground-truth patterns.This gives the guided baseline an advantage over open-ended discovery, which must find patterns from scratch.
  • C.3 Additional details: C.3 Additional details: Classical causal methods output directed edges that are converted into natural-language causal claims and scored by the same LLM judge as other methods.The compared algorithms include PC, GES, NOTEARS, DAG-GNN, and GOLEM on SACHS and ASIA.
  • C.4 Statistical tools: C.4 Statistical tools: Claims enter the claim store only when effect size exceeds 0.2 and p ≤0.05 on both training and held-out validation splits.The EVALUATE step selects among seven statistical primitives, including tabular tests and vision-language-model image-feature tests.

D Compute resources

DiscoPER’s compute is dominated by LLM API calls, with each iteration using approximately three calls for hypothesis generation, experiment planning, and periodic reflection. A 50-iteration run takes 30–60 minutes, while 100 iterations require 1–2 hours depending on model latency.

  • LLM API calls account for the majority of DiscoPER’s computing resources.
  • Each iteration uses approximately three LLM API calls for PROPOSE, experiment planning, and REFLECT every K iterations.
  • A 50-iteration run takes 30–60 minutes, whereas a 100-iteration run takes 1–2 hours depending on model latency.

E Limitations · F Broader impact

DiscoPER’s discoveries depend on the available data and should therefore be interpreted as evidence-grounded candidates requiring expert verification. Its potential broader impact is accelerating analysis of large observational datasets while complementing domain expertise.

  • E Limitations: DiscoPER cannot discover phenomena absent from its available input data.Citizen science data may contain biases that do not accurately reflect underlying ecological phenomena.
  • E Limitations: Human verification remains essential for proposed discoveries because input-data biases can distort ecological conclusions.
  • E Limitations: The iNatDisco claim set is incomplete, so valid patterns absent from its annotations may still be proposed by the model.
  • E Limitations: Statistical-significance testing selects DiscoPER claims that are grounded in evidence from the data.
  • F Broader impact: DiscoPER could accelerate discovery on large observational datasets where human analysis capacity is limited.It is positioned as complementing domain expertise rather than replacing it.
  • F Broader impact: Held-out validation and counterfactual evaluation provide safeguards against hallucinated findings.Despite these safeguards, system outputs should be treated as candidates for expert review, not established facts.
Loading 2607.01131v1…