PaperlayerView exact revision on arXiv

Source-linked AI summary

SciGame: Let Agents Play the Scientific Discovery Games in the Real and Counterfactual World

Yusheng Zhao, Bowei He, Xue Liu, Mingxiao Li

arXiv:2610.37919v1cs.AIcs.CL

TL;DR

Scientific discovery benchmarks often cover only parts of law inference and may not test unseen-input prediction, separate evidence from prior knowledge, or include post-cutoff relations. SciGame pairs Original and dimensionally consistent Counterfactual environments, varies task and observation protocols, and finds that priors and adaptive experimentation affect models differently.

  • Problem

    Existing benchmarks cover parts of discovery but often vary tasks, protocols, and metrics together, with limited unseen-input evaluation, post-cutoff relations, or separation of law inference from programming and replication.

    Source: §1 · p. 1

  • Method

    SciGame turns source relations into scientific cards, creates paired Original and dimensionally consistent Counterfactual environments, and tests Identification and Discovery under Prior-only, Fixed, and Adaptive protocols.

    Source: §1 · p. 2

  • Results

    Familiar priors help on Original relations but hurt on Counterfactual ones, while adaptive experimentation helps stronger models but can harm weaker ones.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Good explanations can coexist with incorrect predictors, so explanation quality does not by itself establish predictive correctness in these experiments.

    Source: §1 · p. 2, §3.4 · p. 5

    The study covers 44 task families, mostly in physics, with three repetitions per condition; deterministic simulators omit measurement noise, and the collections use different mutation schemes.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language model agents are increasingly used for scientific discovery. However, existing benchmarks mix law inference with programming, score data fitting instead of prediction on unseen inputs, and do not separate empirical evidence from pretrained priors. We introduce SciGame, a controlled benchmark that converts scientific literature into structured scientific cards. Each card yields a pair of Original and dimensionally consistent Counterfactual environments with the same task interface. SciGame contains a Historical collection of canonical laws and a live Temporal collection curated from papers published after model knowledge cutoffs. It evaluates agents in candidate-guided Identification and open-ended Discovery settings. Each setting runs under Prior-only, Fixed, and Adaptive protocols. Experiments on five frontier models yield three findings. First, familiar priors hurt performance once the governing relation is altered. Second, adaptive experimentation helps the strongest models but harms weaker ones under limited budgets. Third, good explanations can hide quantitative failures. Recovering the exact symbolic form remains the main bottleneck, even when agents interpret evidence correctly. Current agents can investigate systems actively, but inducing exact laws in unfamiliar settings remains an open challenge.

1 Introduction

Scientific law discovery requires agents to generalize to unseen inputs, distinguish evidence from prior knowledge, and explain applicability, but existing benchmarks cover these demands unevenly. SciGame pairs Original and Counterfactual environments and evaluates agents across task settings and protocols, finding that familiar priors and adaptive experimentation affect models differently.

1 Introduction

  • Existing benchmarks often change task, observation protocol, and metric together, while few test unseen-input generalization, post-cutoff relations, and law discovery without programming or replication.A plausible explanation may also score well when the predictor is wrong.Source: §1 · p. 1
  • SciGame converts source relations into scientific cards and pairs each Original environment with a dimensionally consistent Counterfactual mutation under the same interface.The pairing tests whether agents follow evidence when it conflicts with familiar laws.Source: §1 · p. 2
  • Identification supplies candidate relations, whereas Discovery requires agents to construct a relation without candidates; both require executable expressions, explanations, and applicability conditions.Each setting runs under Prior-only, Fixed, and Adaptive protocols.Source: §1 · p. 2
  • SciGame includes a rerunnable Temporal collection built from recent papers, alongside stable Historical relations, to reduce memorization across model cutoffs.It reports hidden-input prediction, hierarchical episode success, and rubric-based explanation scores.Source: §1 · p. 2
  • Familiar priors help on Original relations but hurt on Counterfactual ones, while adaptive experimentation helps stronger models but can harm weaker ones.Wrong functional relations are the main failure mode, and good explanations can coexist with incorrect predictors.Source: §1 · p. 2

2 Relevant work

Prior work studies pieces of scientific discovery, including interactive hypothesis testing, equation recovery, active design, altered laws, and paper-based research. SciGame’s comparison highlights a remaining gap in consistent task comparisons and relation inference apart from broader research skills.

Scientific discovery benchmarks

  • Existing discovery benchmarks span hypothesis revision, equation discovery, active experimental design, and altered scientific laws, but none compares the same task with and without candidate expressions.The resulting candidate effect on prediction under supporting or contradicting observations remains unclear.Source: §2 · pp. 2–3

Paper-based scientific research

  • Paper-based benchmarks assess broader research skills such as hypothesis discovery, programming, or replication, while SciLaws-Bench uses different protocols across its Real and Parallel settings.Its predictive accuracy is therefore not directly comparable across those settings.Source: §2 · p. 3
  • The benchmark comparison covers paper grounding, counterfactual variants, and temporal source selection, and reports that SciGame supports all compared features.The caption defines checkmarks as supported and dashes as unsupported.Source: §2 · p. 3

3 Methodology

SciGame structures scientific relations as cards, pairs Original and Counterfactual environments, and evaluates agents’ predictors, explanations, and investigation traces. Its methodology specifies source curation, task and protocol variants, and complementary evaluation metrics.

3.1 Overview

  • Each card records a relation, parameters, variables, units, input domain, and applicability conditions, then instantiates paired environments with shared constraints and a controlled structural mutation.Episodes track the Player’s experiments, observations or feedback, hypothesis updates, and final predictor and explanation.Source: §3.1 · p. 3

3.2 Data curation

  • Historical relations provide interpretable references, while Temporal relations come from papers selected relative to the evaluated Player’s reported knowledge cutoff.The Temporal pipeline uses an LLM to extract relations, a separate LLM reviewer to check cards, and can be rerun with updated date windows.Source: §3.2 · pp. 3–4
  • Counterfactual environments preserve the inputs, ranges, observables, and constraints while mutating the relation through changes to coefficients or exponents, nonlinear transformations, or cross-input interactions.The mutations preserve dimensional consistency, and counterpart ground-truth equations fail prediction evaluation on the other environment’s observations.Source: §3.2 · p. 4

3.3 Experimental framework and protocol

  • Identification gives the Player a formula combining Original and Counterfactual expressions through hidden s, whereas Discovery provides no candidate expressions or parametric models.Each setting requires executable observable expressions and a scientific explanation.Source: §3.3 · p. 4
  • Prior-only uses no observations, Fixed supplies observations from a preset design, and Adaptive lets the Player propose experiments sequentially and submit a final completion.Adaptive responses include hypotheses and confidence scores, along with either an experiment proposal or a predictor, rationale, and applicability conditions.Source: §3.3 · p. 4

3.4 Evaluation metrics

  • The evaluation combines hidden-input prediction accuracy, hierarchical episode success, and a rubric-based explanation score assessing effects, comparisons or scaling, and evidential scope.Strict episode success additionally requires an exact match to the ground-truth expression; explanations are scored by an LLM judge.Source: §3.4 · pp. 5–6

4 Experiments

Experiments compare models across scientific task collections and protocols, finding that performance depends on whether relations are original or counterfactual and whether models can experiment adaptively. With limited budgets, fixed observations can outperform adaptive search in open-ended counterfactual discovery; failure patterns also vary by model.

4.1 Experimental setup

  • The setup uses 20 Historical task families and 24 Temporal families, with three independent episodes per task, model, and protocol and hidden-input prediction scoring.GPT-6 Astra evaluates explanations separately; Temporal papers span 30 April to 15 September 2026.Source: §4 · p. 6, §4.1 · p. 6, Table 2 · p. 7

4.2 Main benchmark results

  • 58.6% on Historical and 21.3% on Temporal Original Discovery contrast with 0% strict ESR on Historical Counterfactual Discovery and 8.9% on Temporal counterfactuals.These averages span nine LLMs under Fixed; under Adaptive, the top model scores 16.7% on Temporal and 1.7% on Historical counterfactuals.Source: §4.2 · p. 6, Table 2 · p. 7
  • GPT-6 Astra’s Temporal Counterfactual Discovery ESR rises from 9.7% to 16.7% with Adaptive, while Qwen3.7 Plus stays at 6.9% and GPT-5.6 Terra falls from 8.3% to 1.4%.On Temporal Original Discovery, GPT-5.6 Terra drops from 20.8% to 6.9% and Qwen3.7 Plus from 13.9% to 6.9%.Source: §4.2 · p. 6, Table 2 · p. 7
  • Temporal Counterfactual Discovery’s average ESR is about half Temporal Original Discovery’s (8.9% versus 21.3%); all models fall within a narrow range of 5.6% to 12.5%.Only Adaptive lifts the top model above this range.Source: §4.2 · p. 7, Table 2 · p. 7
  • An SES above 0.700 can coexist with strict ESR below 12.0%; GPT-5.6 Terra has SES 0.815, ESR 1.4%, and PS 0.060 in Adaptive Temporal Counterfactual Discovery.DeepSeek-V4 Flash has SES 0.723 and ESR 11.1% in the same setting; the authors say neither metric alone describes an episode.Source: §4.2 · p. 7, Table 2 · p. 7

Weak ESR on the Temporal collection under dif-

  • 47.2%–65.3% Fixed versus 27.8%–34.7% Adaptive: at B = 8, Fixed performs better across three models in Temporal Counterfactual Discovery.Adaptive Discovery success stays near zero for the first two queries and rises mainly after step four, whereas Adaptive Identification saturates after one query; GPT-6 Astra exceeds 95% at step one for both relation types through B = 8.Source: Weak ESR on the Temporal collection… · p. 7, Figure 5 · p. 8
  • With B ≤ 8, preset observations work better than sequential search for counterfactual laws, while Adaptive Discovery’s advantage depends on a larger interaction budget.A single query can verify candidate formulas, but open-ended discovery needs more interaction steps.Source: Weak ESR on the Temporal collection… · p. 7, Figure 5 · p. 8
  • Incorrect relations account for about half of GPT-5.5 and GPT-5.6 Sol failures, while GPT-6 Astra most often fails to recover the exact equation symbolically (48%).The authors report that powerful models tend to fit relations with polynomials rather than approach hypotheses through physical intuition.Source: Weak ESR on the Temporal collection… · p. 8, Figure 6 · p. 8

5 Conclusion

SCIGAME benchmarks scientific law discovery by pairing Original relations with dimensionally consistent Counterfactual mutations under the same interface. It compares protocols across Identification and Discovery and adds a knowledge-cutoff-aware Temporal pipeline.

5 Conclusion

  • SCIGAME separates law inference from coding and pairs Original relations with dimensionally consistent Counterfactual mutations to test whether agents follow evidence over familiar priors.Source: §5 · p. 8
  • The benchmark compares Prior-only, Fixed, and Adaptive protocols in both Identification and Discovery settings.Source: §5 · p. 9
  • Its Temporal pipeline adds newly published relations to reduce contamination from prior knowledge.Source: §5 · p. 9

Limitations

The evaluation covers a small, mostly GPT-family model set in one harness, and its simulator and curation choices constrain how broadly results can be interpreted.

Limitations

  • The experiments use a small set of mostly GPT-family models in one agent harness, limiting model and harness coverage.Source: Limitations · p. 9
  • With three repetitions per condition, small differences between result cells are uncertain.Source: Limitations · p. 9
  • The environments are deterministic and noise-free, while Temporal relations simplify reported results.Source: Limitations · p. 9
  • Counterfactuals use limited synthetic mutations, and the Historical and Temporal collections use different mutation schemes.Source: Limitations · p. 9
  • Provider-reported cutoff dates reduce but cannot rule out contamination, while LLM-based curation and explanation scoring use an evaluated model as judge.Source: Limitations · p. 9

A Details of benchmark implementation

This appendix specifies the Adaptive interaction protocol and the evaluation pipeline.

A Details of benchmark implementation

A.1 Adaptive interaction protocol

Under Adaptive interaction, the Player proposes experiments to an Oracle, updates hypotheses from feedback, and either continues experimenting or submits a predictor under a stopping rule. The protocol specifies hypothesis reports, proposal validation, and deterministic simulated observations.

A.1 Adaptive interaction protocol

  • The interaction loop has the Player submit hypotheses and an experimental proposal, then revise hypotheses after Oracle feedback or terminate with a final predictor.Source: §A.1 · p. 12, Figure 8 · p. 12
  • During exploration, reports include at least two hypotheses with confidence scores summing to one and finite numerical forecasts for each requested readout under each hypothesis.Source: §A.1 · pp. 12–13
  • The final submission designates a selected hypothesis with an executable predictor, scientific explanation, and applicability conditions.Source: §A.1 · pp. 12–13
  • The Oracle validates proposals before execution, and Adaptive experimentation continues only while every hypothesis confidence is below threshold τ; a valid final choice must meet or exceed τ.Source: §A.1 · pp. 12–13
  • Protocol compliance is scored separately from prediction accuracy: violating the stopping rule can still receive a prediction score but automatically fails the episode success rate.Source: §A.1 · p. 13
  • Accepted proposals return deterministic observations, quantized to each observable's declared resolution when specified; otherwise the exact response is returned.Source: §A.1 · p. 13
  • The evaluated Players are five frontier GPT models run through Codex at high reasoning effort, using the same action-selection and forecasting prompt across protocols.Source: §A.1 · p. 13
  • The supplied case defines the world, which may differ from familiar laws, and explanations should not claim mechanisms beyond the case and evidence.Source: §A.1 · p. 13

A.2 Evaluation pipeline details

The evaluation pipeline represents predictions as constrained arithmetic expressions and assesses both predictive validity and scientific understanding. Episode fields, model metadata, and explicit scoring rules structure the evaluation.

A.2 Evaluation pipeline details

  • Completed predictors are represented as pure arithmetic expression trees rather than arbitrary executable scripts.Leaves encode constants or task inputs, while internal nodes specify supported mathematical operations.Source: §A.2 · p. 13
  • Supported operations use real-domain semantics, and the operator table records each operation’s arity.Undefined real operations, including division by zero and non-positive logarithm arguments, invalidate a predictor.Source: §A.2 · p. 13, Table 5 · p. 15
  • Predictors are capped at 1,024 total nodes and depth 64 per tree; unsupported or undefined operations and non-finite outputs invalidate them.Admissible predictors are scored on accuracy at unseen inputs, without clipping, imputation, or post-hoc repair.Source: §A.2 · pp. 13–14
  • Table 3 records episode scientific fields and action structure, while Table 4 lists Player models and dates.Source: Table 3 · p. 14, Table 4 · p. 14
  • An LLM Evaluator judges the Player’s endpoint text against the assigned environment’s reference relation, using the full interaction history.The prompt excludes episode pass/fail and numerical errors and asks the evaluator to assess expressed scientific understanding against available evidence.Source: §A.2 · p. 14
  • Rubric scores distinguish correct scoped understanding (1), partial understanding (0.5), unresolved or materially incorrect content (0), and no assessable endpoint (null).Tentative hypotheses are not explanation errors merely for being wrong; expressed uncertainty is respected, and every non-null score requires a supporting quotation.Source: §A.2 · pp. 14–15

B Addtional experimental results

Additional results show that exact symbolic recovery remains difficult, especially in counterfactual discovery. Adaptive episodes that fail can consume substantial budgets, while successful episodes often resolve with few observations.

B Addtional experimental results

  • One targeted query suffices for successful Adaptive Identification episodes, while successful Counterfactual Discovery episodes cluster at 9–64 queries and can exceed 256.Original Discovery successes peak at steps three to six and most resolve within sixteen queries.Source: B Addtional experimental results · p. 15
  • $5,843 is the total inference cost reported for the main benchmark, including $4,429 for Adaptive, $782 for Fixed, and $631 for Prior-only.Adaptive is the only protocol in which the Player chooses how many observations to take.Source: B Addtional experimental results · p. 15
  • 96% of Adaptive cost comes from unsuccessful episodes, which run 7–20 observations on average; successful episodes stop after 1.3–2.5.One GPT-6 Astra episode reaches 485 observations, and the authors identify inferred relations, rather than collected evidence, as the limiting factor.Source: B Addtional experimental results · p. 15
  • 0.0% strict ESR for most models and 1.7% strict ESR versus 95.0% weak ESR for GPT-6 Astra in Historical Counterfactual Discovery; Temporal Counterfactual Discovery peaks at 16.7% strict ESR.Across conditions, numerical data fitting succeeds more often than recovery of the exact symbolic expression.Source: B Addtional experimental results · p. 15

C Collection sources

The benchmark draws on historical task families and a Temporal collection, with source-grounded extraction and review intended to preserve the support and scope of reported relations.

C Collection sources

  • The extraction prompt requires source-anchored support for every factual statement and numeric value and bars inferring missing facts from outside knowledge.Source: C Collection sources · p. 16
  • Extraction records experimental controls, readouts, fixed conditions, aggregation, sample size, uncertainty, applicability, and limitations, while preserving the source’s stated relation type.A control is a manipulated numeric quantity; a readout is a measured or computed observable, not a latent variable or fitted parameter.Source: C Collection sources · p. 16
  • Review checks source support and formula fidelity, including parameter, control, readout, and domain claims, against the frozen source packet.The review warns against strengthening associations or proportionalities into unsupported equalities or expanding measured covariates beyond reported ranges.Source: C Collection sources · pp. 16–17
  • The 20 Historical task families implement canonical relations in modern simulations rather than reconstructing historical experiments.Cards specify parameters, inputs, observables, units, resolutions, and tolerances; counterfactual mutations can activate correction terms absent from the Original relation.Source: C Collection sources · p. 18
  • The Temporal collection contains 24 families and draws on recent peer-reviewed papers alongside classical scientific history.Collection batch and publication date are distinct quantities.Source: C Collection sources · pp. 15, 18

D LLM Disclosure

This part documents the authors’ limited use of generative AI and specifies benchmark tables and disclosure cards for bounded historical and temporal tasks across physical systems, including their interfaces, assumptions, counterfactual interventions, and shared controls.

LLM disclosure

  • Generative AI tools were used only to summarize existing literature and refine grammar and readability; the authors reviewed and verified all content and take responsibility for the submission.Source: D LLM Disclosure · p. 18

Historical collection

  • The Historical collection comprises 20 deterministic idealizations grounded in cited sources, with specified input domains, observables, validity scopes, and separating counterfactual interventions.Its intervention examples include changing an inverse-square relation to inverse distance and adding an input interaction.Source: Table 8 · p. 19, D LLM Disclosure · p. 20
  • The historical RC-discharge task specifies V(t) = V0 exp[−a(t/RC)^q] with a = q = 1, under ideal lumped first-order RC assumptions and a normalized-time intervention.The task samples normalized times and separately varies capacitance while crossing initial voltage and resistance.Source: D LLM Disclosure · p. 20

Temporal collection

  • The Temporal collection contains 24 families derived from 23 primary papers, with deterministic source-reported branches, benchmark-authored counterfactual responses, and stated legal query domains.The implementations follow selected relations under stated approximations and do not reproduce all experiments or raw data in the papers.Source: Table 9 · p. 21
  • The temporal thin-film reflectance task retains multiple reflections and the analytic derivative, while its counterfactual replaces thickness throughout the differentiated response and probes interior interference extrema.The specified model is a first-order real-index perturbation for normal-incidence 800-nm light in an air/SnS/MgO stack; imaginary-index change is omitted.Source: D LLM Disclosure · p. 22

Scientific-system cards

  • The cards span mantle phase boundaries, CoO magnetoresistance, MnF2 transitions, quasiperiodic localization, suspension rheology, and splashing-drop dynamics.Their inputs and readouts are bounded or normalized, with the suspension card restricted to a constructed packing-related slice and the drop card to an early two-jets regime.Source: D LLM Disclosure · pp. 22–23

Comparison protocols

  • Comparison protocols emphasize interior inputs, coupled readouts, and shared endpoints or controls rather than relying on endpoints alone.Examples include crossing distinct localization couplings, logarithmically separated suspension values, coupled drop speed and diameter, and joint coexistence branches.Source: D LLM Disclosure · pp. 23–25

Quantum-system cards

  • Quantum cards specify trapped-ion survival, parity-prepared oscillator squeezing, microcanonical coexistence, and loss-mediated transfer using explicit physical constraints and multi-output readouts.The trapped-ion card fixes pulse calibration; the oscillator card retains full pure loss and coherences; the coexistence target excludes finite-size dynamics.Source: D LLM Disclosure · pp. 24–25

Counterfactual transformations

  • Counterfactuals warp selected inputs while retaining other interface variables, including position for the ion card, displacement amplitude for squeezing, and transverse field for coexistence.Other cards warp time in complete transfer outputs or Weber number in linked drop-response outputs.Source: D LLM Disclosure · pp. 23–25

Non-Markovian radiation

  • The radiation card returns three unconditional Dicke populations and signed energy-loss rate, with a separate derivative factor applied to the composed intensity.It retains excitation sectors and loss recycling; endpoint populations agree across branches, while endpoint intensities need not.Source: D LLM Disclosure · p. 25

Related papers

Loading 2610.37919v1…