PaperlayerView exact revision on arXiv

Source-linked AI summary

SciPredict: Can LLMs Predict the Outcomes of Scientific Experiments in Natural Sciences?

Udari Madhushani Sehwag, Elaine Lau, Haniyeh Ehsani Oskouie, Shayan Shabihi, Erich Liang, Andrea Toledo, Guillermo Mangialardi, Sergio Fonrouge, Ed-Yeremai Hernandez Cardona, Paula Vergara, Utkarsh Tyagi, Chen Bo Calvin Zhang, Pavi Bhatter, Nicholas Johnson, Furong Huang, Ernesto Gabriel Hernandez Montoya, Bing Liu

arXiv:2604.10718v1cs.AI

TL;DR

Scientific progress could benefit from predicting experimental outcomes before costly validation, but evidence about LLMs’ ability to do so remains limited. SciPredict evaluates this capability using expert-curated empirical tasks across natural sciences and finds low accuracy and poor reliability awareness. The results indicate that useful scientific assistance requires both stronger predictions and better calibration of their reliability.

  • Problem

    LLMs’ ability to predict experimental outcomes reliably before costly physical validation remains underexplored, despite its relevance to scientific research decisions.

  • Method

    SciPredict benchmarks LLMs on 405 expert-curated prediction tasks from recent empirical studies across 33 specialized physics, chemistry, and biology subfields.

  • Results

    Frontier LLMs achieve 14%-26% accuracy versus approximately 20% for human experts, while models remain poorly calibrated and humans’ accuracy rises from approximately 5% to approximately 80% with feasibility.

  • Takeaways & Limitations

    Reliable experimental guidance requires not only higher prediction accuracy but also awareness of which predictions are trustworthy.

  • Takeaways & Limitations

    The benchmark covers only physics, biology, and chemistry, and its 405 questions may not capture the full diversity of experimental paradigms within each subdomain.

Abstract

from arXiv · show

Accelerating scientific discovery requires the identification of which experiments would yield the best outcomes before committing resources to costly physical validation. While existing benchmarks evaluate LLMs on scientific knowledge and reasoning, their ability to predict experimental outcomes - a task where AI could significantly exceed human capabilities - remains largely underexplored. We introduce SciPredict, a benchmark comprising 405 tasks derived from recent empirical studies in 33 specialized sub-fields of physics, biology, and chemistry. SciPredict addresses two critical questions: (a) can LLMs predict the outcome of scientific experiments with sufficient accuracy? and (b) can such predictions be reliably used in the scientific research process? Evaluations reveal fundamental limitations on both fronts. Model accuracies are 14-26% and human expert performance is $\approx$20%. Although some frontier models exceed human performance model accuracy is still far below what would enable reliable experimental guidance. Even within the limited performance, models fail to distinguish reliable predictions from unreliable ones, achieving only $\approx$20% accuracy regardless of their confidence or whether they judge outcomes as predictable without physical experimentation. Human experts, in contrast, demonstrate strong calibration: their accuracy increases from $\approx$5% to $\approx$80% as they deem outcomes more predictable without conducting the experiment. SciPredict establishes a rigorous framework demonstrating that superhuman performance in experimental science requires not just better predictions, but better awareness of prediction reliability. For reproducibility all our data and code are provided at https://github.com/scaleapi/scipredict

1. Introduction

SciPredict addresses the limited evaluation of LLMs for predicting empirical outcomes by introducing a benchmark and assessing prediction accuracy, reliability, and the usefulness of contextual knowledge. Results show substantial accuracy and calibration gaps, despite benefits from expert-curated background knowledge.

  • Existing benchmarks largely emphasize literature work, simulation or reproduction, and hypothesis generation rather than empirical experiment-outcome prediction.
  • SciPredict contains 405 expert-curated tasks from empirical studies spanning 33 specialized fields across physics, chemistry, and biology.
  • 14%-26% model accuracy and ≈20% human accuracy remain insufficient for reliable experimental planning.
  • Model accuracy does not meaningfully improve with higher feasibility scores, whereas human accuracy rises from ≈5% to ≈80% as rated feasibility increases.
  • Expert-curated background knowledge improves accuracy by ≈3% on average, while self-generated knowledge often degrades performance.
  • The evaluation compares 15 state-of-the-art LLMs and human experts on accuracy, reliability ratings, and four forms of background knowledge.

2. Related Works

Related benchmarks evaluate LLMs across scientific and research-oriented tasks, but empirical outcome prediction in the natural sciences remains a distinct and insufficiently covered setting. SciPredict differs by focusing on quantitative predictions of experimental results and their feasibility.

  • Scientific benchmarks cover expert-level science tasks, computational fluid dynamics, healthcare support, biology evaluations, and other domain-specific capabilities.
  • AI/ML research benchmarks primarily assess literature work, idea generation, code execution, experiment reproduction, and end-to-end research cycles.
  • Non-ML scientific benchmarks include computational physics problem solving, equation inference, and simulated equation optimization.
  • Outcome-prediction work has mainly examined behavioral and social sciences, whereas SciPredict targets quantitative empirical results in the natural sciences.
  • Hypothesis-generation benchmarks ask models to produce or rank novel hypotheses using background facts, knowledge graphs, or literature.
  • Existing feasibility studies assess capability judgments, refusal behavior, uncertainty, or feasible-task labels, but not empirical outcome prediction requiring the underlying experiment.

3. SciPredict Curation

SciPredict constructs a broad benchmark from recent empirical studies using expert extraction, multiple prediction formats, structured task fields, and multi-stage quality review. Its design balances predictive difficulty with objectively verifiable outcomes across diverse natural-science domains.

  • Design Principles: SciPredict contains 405 tasks from studies published after March 2025, presenting experimental systems, conditions, measurements, interventions, and outcomes to predict.
  • Expert Annotation: Experts curate experimental setups, measurements, prediction questions, ground truths, and background knowledge while preserving challenge and objective verifiability.
  • Design Principles: The benchmark covers physics, biology, and chemistry, with 33 specialized subdomains and tasks ranging from single-step reasoning to complex multi-hop inference.
  • Quality Control: The curation pipeline combines post-March 2025 data collection, expert annotation, structured fields, and iterative quality assurance.
  • Design Principles: Tasks use multiple-choice, free-form, and numerical formats to assess discrete, explanatory, and quantitative prediction.
  • Quality Control: A separate expert group supplies human baselines, while two expert-review rounds check task clarity, background relevance, ground-truth clarity, difficulty, distractors, rubrics, and numerical ranges.

4. Evaluation Setup and Metrics

SciPredict evaluates model accuracy and self-assessed reliability across three scientific prediction formats, multiple knowledge-context conditions, and human and LLM baselines. The setup also tests whether reliability judgments and evaluation procedures are calibrated and robust.

  • Accuracy Metrics: Accuracy is defined separately for multiple-choice, free-form, and numerical questions to support comparison across formats.MCQ uses exact correctness, free-form uses expert-rubric judging, and numerical predictions are correct when they fall within an acceptable interval.
  • Reliability Metrics: Models provide predictions alongside confidence, difficulty, and feasibility assessments that test whether self-judgments track empirical accuracy.Confidence and feasibility should increase with accuracy, whereas perceived difficulty should decrease with accuracy under calibration.
  • Context Conditions: The benchmark varies context across no background, expert-curated, self-generated, combined, and filtered background knowledge conditions.These conditions test parametric knowledge, explicit expert information, autonomous context generation, source combination, and whether restating unknown information helps prediction.
  • Evaluation Participants: The evaluation compares 15 zero-shot state-of-the-art LLMs with human experts under matched no-background and background conditions.Human experts provide the same confidence, difficulty, and feasibility assessments, enabling direct calibration comparisons.
  • Robustness Checks: Free-form scoring and decoding choices were tested for robustness, with no statistically significant accuracy differences across judge models or tested decoding settings.Free-form evaluations were replicated with GPT-5.2, and temperatures from 0.0 to 1.0 plus several top-p settings were evaluated.

5. Main Results

SciPredict finds that frontier models remain unreliable predictors of experimental outcomes and cannot meaningfully assess their own prediction reliability. Curated background knowledge improves accuracy, but self-generated context and model self-reports do not provide dependable guidance.

  • Predictive performance: 14%-26% model accuracy remains insufficient for reliable experimental planning, despite some models marginally exceeding the approximately 20% human baseline.The benchmark evaluates whether predictions are accurate and reliable enough to guide resource allocation in scientific experimentation.
  • Calibration: Human accuracy rises from approximately 5% on infeasible questions to approximately 80% on feasible questions, whereas models show no systematic relationship between self-reported feasibility or difficulty and accuracy.Human judgments therefore provide a clear ordering of predictability that model self-reports fail to reproduce.
  • Curated background knowledge: Approximately 3% average accuracy improvement from curated background knowledge indicates that supplying relevant context benefits correctness without materially changing confidence.The gain varies across models, while confidence scores remain roughly unchanged between NBK and BK conditions.
  • Curated background knowledge: Most models perform better with full curated background than filtered background, showing that explicitly restating already known facts can improve prediction accuracy.Restated information can foreground relevant priors, reduce ambiguity, align terminology and assumptions, and scaffold application to the prediction task.
  • Self-generated background knowledge: Self-generated background knowledge often lowers accuracy relative to no background, while combining it with expert-curated context rarely yields consistent improvement.Synthetic context can introduce irrelevant, misleading, distracting, or harmful information.
  • Error analysis: Factual and extraction errors average 80.1% and logical and reasoning flaws average 87.4% of incorrect responses, while basic comprehension errors average 10.0%.Frequent fine-grained errors include factual contradiction and information fabrication, rather than primarily surface-level comprehension problems.

6. Discussion and Conclusion

SciPredict reveals that current LLMs remain unreliable guides for experimental decisions because their modest prediction accuracy is paired with severe miscalibration. The paper concludes that useful scientific assistance requires systems that assess prediction reliability, while noting important scope and coverage limitations.

  • 14-26% accuracy remains insufficient for guiding resource-intensive experimental decisions, despite frontier models approaching or exceeding human baselines around 20%.
  • Models maintain approximately 20% performance regardless of self-reported confidence or feasibility, unlike humans whose accuracy rises from approximately 5% to 80%.
  • Expert-curated background knowledge provides approximately 3% gains, but models cannot autonomously identify or generate consistently helpful context.
  • The benchmark covers three natural-science domains, uses a March 2025 temporal cutoff, contains 405 questions, and relies on expert-curated background knowledge that may introduce bias.

A.1 Additional details about task contributors / human baseline participants

The benchmark contributors and human-baseline participants were selected for relevant scientific expertise, with the cohort showing strong credentials and broad geographic diversity.

  • Experts recruited for benchmark construction and human baselines had strong credentials across their respective scientific fields.The paper provides additional visualizations of degree, expertise, and country-of-origin diversity.
  • Human-baseline experts were matched to task domains and subdomains to support relevant performance comparisons.The expertise-to-task mapping is summarized in Table 1.

A.3 Detailed expert task curation

SciPredict curates post-March 31, 2025 empirical studies into expert-extracted prediction tasks, with separate experts providing human baselines and multiple review stages validating task quality.

  • Selected studies were empirical experiments published after March 31, 2025, excluding theoretical analyses and computational simulations to reduce data leakage.Sources included recognized venues such as bioRxiv, chemRxiv, arXiv, PubMed Central, Nature, and Science.
  • Experts extracted domain labels, experimental setups, measurements, prediction questions, ground-truth answers, and supporting background knowledge from each paper.Answers were formatted as multiple-choice, numerical, or free-form tasks.
  • Human-baseline participants answered benchmark questions, explained their reasoning, and reported confidence under a protocol mirroring LLM evaluations.They formed a separate expert cohort from the benchmark-construction annotators.
  • 74.4% of human-baseline participants held doctoral degrees, while their primary expertise covered biology, chemistry, and physics.The cohort also included participants from multiple countries, including the United States, Argentina, the United Kingdom, Mexico, and Colombia.
  • A rigorous assignment process aligned each human subject’s expertise with corresponding task subdomains to reduce domain mismatch.The resulting expertise-to-task mapping appears in Table 1.
  • Multi-stage review filtered questions with timing, simulation, ambiguity, information sufficiency, or source-ground-truth problems before domain-expert review.Reviewers also checked distractor plausibility, free-form rubric coverage, and numerical evaluation criteria.

A.6 Data Diversity Details

The benchmark spans diverse scientific domains, experimental complexities, knowledge requirements, and controlled distributions of domains and question formats.

  • The benchmark covers 33 specialized subdomains across physics, biology, and chemistry.Physics contributes 9 subdomains, biology 14, and chemistry 10.
  • Questions range from controlled laboratory systems to complex biological systems and from single-step reasoning to multi-hop inference.Some tasks require integrating thermodynamics, kinetics, and material properties.
  • 25% of questions come from physics, 50% from biology, and 25% from chemistry.The balanced domain distribution is intended to reduce overfitting to particular experimental contexts.
  • Question formats comprise 40% multiple-choice, 32% free-form, and 28% numerical-value tasks.These controlled format proportions help test general experimental-outcome prediction rather than narrow pattern matching.

A.7 Human baseline expert - Task subdomain mapping

The section maps human annotator expertise and task distributions across biology, physics, and chemistry subfields.

  • Table 1 groups human annotator expertise by task domain and subdomain.
  • Chemistry: The task mapping includes environmental chemistry, polymer chemistry, chemical engineering, and materials and inorganic chemistry.
  • Biology: The listed biology subdomains include antimicrobial resistance, biochemistry, cell biology, genetics, microbiology, neuroscience, plant biology, and related fields.
  • Table 2 reports the number of tasks assigned to biology, physics, and chemistry subfields.

B.1 Task examples

The examples span quantitative, multiple-choice, and free-form experimental predictions across physics, chemistry, biology, and neuroscience. They pair detailed setups and measurements with reported experimental outcomes.

  • Physics: A continuous cold-atom beam combines a 2D MOT with off-axis moving optical molasses for cooling and velocity control.
  • Physics: Increasing MOT power raises atomic flux but affects temperature only weakly.
  • Physics: Warm dense copper experiments use X-ray absorption spectroscopy and VISAR measurements to infer plasma temperature and shock behavior.
  • Physics: The TFLN modulator shows 7.2–8.8 dB of DC bias drift after 16 minutes at 4.3 dBm optical power and 737 nm.
  • Biology: VTA dopamine-terminal activation in pBLA accelerates fear extinction and improves retrieval, whereas aBLA activation worsens later extinction and retrieval.
  • Biology: The GelMA versus GelMA-plus-100 μg GsMTx4 tendon treatment comparison yields a 4–6-point BS difference at eight weeks.
  • Chemistry: In-situ XANES shows rapid PdO reduction to metallic Pd followed by palladium hydride formation during formate exposure.

B.2 Example human responses

The human-response examples combine a measured net-proton cumulant ratio with theoretical background, experimental context, and confidence or feasibility judgments.

  • The graphically estimated ground-truth range for C4/C2 is 0.25–0.40.
  • C4/C2 is identified as κσ^2, with C2 equal to variance σ^2 and κ representing kurtosis.
  • The Poisson baseline for experimentally measured cumulant ratios is unity.
  • The measured net-proton cumulant ratio C4/C2 is 0.4 at 19.6 GeV for the 0–5% centrality class.
  • The response labels the question easy and the answer somewhat confident, while judging it very feasible without running the experiment.

C. Additional Results

Additional results describe evaluation dimensions, experimental controls, and a taxonomy for analyzing prediction errors.

  • Additional results: Performance tables organize LLM evaluations by domain, question format, confidence, difficulty, feasibility, and required background knowledge.The supplied table captions identify these comparison dimensions but do not report the underlying values.
  • Evaluation controls: Web search was disabled to measure predictive reasoning rather than information retrieval.The benchmark uses post-March-2025 papers, so web access could expose the source publications and turn prediction into lookup.
  • Error analysis: The error taxonomy distinguishes five top-level categories, including comprehension, factual extraction, logical reasoning, scientific rigor, and formatting errors.The categories separate task misunderstanding from failures involving explicit information, inference, scientific communication, or mechanics.
Loading 2604.10718v1…