PaperlayerView exact revision on arXiv

Source-linked AI summary

InfiniteScienceGym: An Unbounded, Procedurally-Generated Benchmark for Scientific Analysis

Oliver Bentham, Vivek Srikumar

arXiv:2604.13201v2cs.CLcs.AI

TL;DR

Evaluating empirical-data reasoning and abstention is difficult because published-study benchmarks carry methodological biases and storage costs. InfiniteScienceGym procedurally generates deterministic repositories and verifiable answerable or unanswerable QA tasks, then evaluates proprietary and open-weight models. No evaluated model exceeds 50% overall accuracy, unanswerable questions remain difficult, and stronger models use tools more effectively rather than simply consuming more tokens.

  • Problem

    Published-study and human-annotated benchmarks inherit publication bias, known-knowledge bias, label noise, and substantial storage requirements, limiting controlled evaluation of empirical-data reasoning and abstention.

  • Method

    InfiniteScienceGym combines a seed-deterministic repository simulator, privileged QA generator, and paraphrase module to produce scalable, verifiable answerable and unanswerable tasks.

  • Results

    No evaluated model exceeds 50% accuracy overall, while unanswerable-question recognition remains weak and accuracy increases with tool interaction rather than token usage.

  • Takeaways & Limitations

    InfiniteScienceGym complements real-data benchmarks by stress-testing evidence-grounded reasoning, abstention, and data analysis where ground truth is known exactly.

  • Takeaways & Limitations

    The benchmark focuses on empirical tabular data, excludes other modalities and non-empirical reasoning, and uses a benchmark-specific operational notion of unanswerability.

Abstract

from arXiv · show

Large language models are emerging as scientific assistants, but evaluating their ability to reason from empirical data remains challenging. Benchmarks derived from published studies and human annotations inherit publication bias, known-knowledge bias, label noise, and substantial storage requirements. We present InfiniteScienceGym, a procedurally generated benchmark of scientific repositories paired with a verifiable question-answering task. From a seed, the simulator deterministically generates a self-contained repository with realistic directory structure, files, and tabular data, and a privileged QA generator produces both answerable and unanswerable questions with exact ground truth. This makes it possible to evaluate evidence-grounded reasoning, abstention, and tool-mediated analysis in a controlled setting without distributing a large static corpus. InfiniteScienceGym complements real scientific benchmarks by targeting blind spots and failure modes that are hard to evaluate using published datasets alone. Evaluating both proprietary and open-weight models, we find that none achieve more than 50% accuracy overall, that recognizing unanswerable questions remains a major weakness, and that stronger models tend to use tools more effectively rather than simply consuming more tokens.

1 Introduction

InfiniteScienceGym addresses gaps in evaluating whether scientific assistants reason from available data and recognize when evidence is insufficient. It introduces a procedurally generated benchmark with verifiable QA, including unanswerable questions, and reports weak overall performance alongside more effective tool use by stronger agents.

  • InfiniteScienceGym targets whether models reason from repository data, recognize insufficient evidence, and use tools to support answers.
  • Existing benchmarks inherit publication bias, known-knowledge bias, annotation noise, and the storage burden of large static datasets.
  • The benchmark combines a seed-based repository simulator, a privileged QA generator, and a paraphrase module producing naturalistic questions with exact ground truth.
  • 49.2% accuracy was achieved by the best evaluated model, while identifying unanswerable questions remained a major weakness.
  • Accuracy increased with tool-call interaction, whereas token usage alone misleadingly measured effort or thoroughness.

2 The Case for Procedurally Generated Scientific Evaluation

Existing scientific benchmarks cover different parts of data-driven reasoning, abstention, structured artifacts, and agentic workflows. InfiniteScienceGym is positioned at their intersection, prioritizing reproducibility and exact ground-truth verification while complementing real-data evaluations.

  • Scientific reasoning benchmarks include discovery workflows, research-derived tasks, Bayesian-surprise discovery, and scientific equation discovery.
  • Abstention research studies uncertainty, knowledge gaps, synthetic novelty, and simulation-based evaluation of whether models acknowledge insufficient knowledge.
  • Structured-artifact benchmarks evaluate coding, reproducibility, repository tasks, database-grounded text-to-SQL, and semi-structured table reasoning.
  • These benchmark families trade off workflow fidelity, verifiable ground truth, scalability, bias robustness, and support for file navigation and tool use.
  • InfiniteScienceGym occupies this intersection by prioritizing seed-based reproducibility and exact ground-truth verifiability at effective scale.

3 InfiniteScienceGym: Reaping Repositories from Sown Seeds

InfiniteScienceGym deterministically generates realistic scientific repositories and paired QA tasks from seeds, progressing from scientific context to project, paths, files, and data. Privileged generation supports answerable and unanswerable questions, while paraphrasing adds naturalistic surface variation.

  • The simulator deterministically generates unbounded file systems and QA pairs from random seeds, guaranteeing reproducibility across runs.
  • Repository construction proceeds top-down from scientific context through project specification, directory structure, and tabular file structure.
  • Directories use project-conditioned placeholder variables and delimiters to create plausible paths whose names encode research variables.
  • Tabular files organize identifiers, dates or times, independent variables, and dependent variables, with independent variables sampled from named distributions.
  • Dependent-variable functions can encode linear or nonlinear relationships, noise processes, and partial observability.
  • Human evaluation found statistically significant plausibility across repository organization, naming, schemas, values, and visible simulation artifacts.
  • A privileged QA generator applies filters and computes exact answers for answerable questions while constructing unanswerable cases from insufficient data or ambiguity.
  • A paraphrase module converts controlled templates into naturalistic research queries while retaining verifiability.

4 Experimental Setup

The evaluation tests models on templated and paraphrased answerable and unanswerable questions, measuring accuracy, abstention, resource use, error patterns, and robustness to surface-form variation.

  • Two experiments compare model performance, unanswerable-question detection, token and tool usage, annotated error patterns, and agreement across surface forms.
  • 500 questions were sampled from 15,988 questions associated with the first 500 repositories.
  • The evaluation includes GPT-5.4, Claude Opus 4.6, GPT-OSS 20B, Gemma 3 27B it, and Qwen3 4B Instruct.

5 Experimental Results and Analysis

Models struggle with repository-grounded QA, especially questions requiring schema-aware filtering, computation, and abstention. Higher accuracy is associated with more effective tool use rather than greater token use, while paraphrased questions show moderate agreement with templated variants.

  • Proprietary models outperform open-weight models by at least 14 percentage points on templated questions, with differences significant at p ≤0.001.
  • Filtering questions are especially difficult because they require mapping conditions onto schemas, selecting files and rows, and computing or rejecting an answer.
  • All models struggle with filtering, while stronger models make fewer navigation and schema-grounding errors before reaching later stages.
  • Claude Opus 4.6 and GPT-5.4 exceed 80% precision and recall for unanswerable-question detection, but neither exceeds 83%.
  • Higher accuracy is associated with more tool calls, not more tokens; GPT-5.4 achieves the highest accuracy while averaging about 24,000 tokens per question.
  • Agreement between templated and paraphrased responses ranges from 0.68 to 0.71, while GPT-OSS 20B paraphrases are 96% semantically equivalent, 98% unambiguous, and 100% fluent.

6 InfiniteScienceGym as a Unit Test and a Robustness Probe

InfiniteScienceGym evaluates repository-grounded scientific assistance as a sequence of unit-test capabilities and as a robustness probe. Its controlled generation supports exact-ground-truth stress tests, especially for answerability and abstention, while remaining complementary to real-data benchmarks.

  • Current models struggle to reliably inspect repositories, identify evidence, compute quantities, and recognize insufficient data.
  • Proprietary models outperform open-weight models, but no evaluated model performs particularly well because errors at any reasoning stage can prevent success.
  • No model handles unanswerable questions reliably in all cases, and open-weight models miss many such cases.
  • Higher accuracy is associated with more tool use but not higher token usage, indicating that stronger models gather evidence selectively and operate programmatically.
  • The benchmark functions as unit tests for traversal, filtering, aggregation, and abstention, while also probing robustness to repository and wording variation.
  • InfiniteScienceGym prioritizes control over realism and complements real-artifact benchmarks for stress-testing answerability, abstention, and data analysis with exact ground truth.

7 Conclusion

InfiniteScienceGym provides a controlled, storage-free benchmark for evaluating abstention and evidence-grounded reasoning. Experiments show persistent model weaknesses, while future extensions target broader modalities and messier repositories.

  • The benchmark generates realistic repositories, verifiable answerable and unanswerable questions, and naturalistic paraphrases on demand.Its unanswerable questions are guaranteed by construction, enabling direct abstention measurement.
  • No evaluated model exceeds 50% accuracy, and unanswerable questions remain a major source of failure.Higher-performing models tend to use tools more effectively rather than simply consuming more tokens.
  • The benchmark complements real-data evaluations by directly targeting abstention and evidence-grounded reasoning.
  • InfiniteScienceGym excludes images, video, audio, and non-empirical reasoning, while procedural generation may introduce exploitable regularities.Its operational notion of unanswerability is narrower than uncertainty in real scientific practice.
  • Future directions include adding data modalities and messier repository structures, constructing repositories that contradict prior knowledge, and studying hallucination through access patterns.

Reproducibility Statement

InfiniteScienceGym uses seed-driven generation to make benchmark artifacts exactly reproducible, but model-evaluation results are only partially reproducible. Exact regeneration also depends on pinned software versions and specified hardware.

  • Given the same seed, the simulator deterministically regenerates identical repositories, file contents, and question-answer pairs.This supports exact regeneration of benchmark instances and ground-truth labels from released code and seeds.
  • Model-evaluation results are only partially reproducible because sampled outputs and proprietary backend behavior are not fully controlled.The paper claims exact reproducibility for benchmark artifacts but not replicability for evaluation results.
  • Exact byte-for-byte regeneration depends on library versions and hardware, which the accompanying code documents explicitly.
  • The scientific taxonomy samples fields, domains, and subdomains to introduce diversity in simulated empirical-science data rather than accurately represent scientific areas.

B.1 Repository #118 Description

Repository #118 models an omics-driven study of genetically modified Saccharomyces cerevisiae across fermentation conditions. Its structured files support multi-level analysis of genetic, environmental, phenotypic, and molecular measurements.

  • The project seeks genetic-modification and fermentation-condition combinations that maximize ethanol yield while maintaining strain stability and metabolic robustness.
  • The study tests 12 modified S. cerevisiae strains under 6 fermentation conditions, with triplicate measurements of phenotypic and omics outputs.
  • Independent factors include strain, temperature, pH, glucose concentration, oxygen level, growth phase, measurement time, and genotype type.
  • Measured outcomes include ethanol yield, glucose consumption, biomass, viability, metabolites, gene expression, protein abundance, metabolic flux, and stress markers.
  • Files are organized by strain, condition, and time point, with each run represented as a tabular file containing replicate-level metadata and measurements.
  • The project integrates genomics, transcriptomics, proteomics, metabolomics, and high-throughput phenotyping for data-driven industrial ethanol strain optimization.
  • Repository paths are sampled from Cartesian combinations of placeholder values, with a right-skewed distribution favoring file systems containing fewer than 500 files.
  • Generated dependent-variable functions receive non-dependent variables and an error term, exemplified by the glucose-consumption-rate function.

F Validating Repository Plausibility

The benchmark validates generated repositories through domain-expert judgments and organizes evaluation questions into progressively more difficult metadata, traversal, and statistical tasks. Examples span repository and file metadata, conditional univariate statistics, and bivariate hypothesis testing.

  • Repository plausibility: 19 domain experts evaluated generated repositories across six scientific fields, providing a plausibility validation study.Experts held or pursued Ph.D.s and researched within their evaluated domains.
  • Repository plausibility: All six validation items rejected the neutral-or-worse median-rating null hypothesis at α = 0.05.The sign-test results provide initial evidence that generated repositories are plausible enough for evaluation.
  • Question categories: Evaluation questions progress from repository and file metadata to directory traversal, univariate statistics, and bivariate statistics.The categories move from locating high-level information to computing statistics and testing hypotheses.
  • Question categories: Metadata and traversal questions ask about repository contents, file properties, paths, counts, and conditional directory structure.Examples include checking for a README, identifying file extensions, and counting files matching path or variable conditions.
  • Question categories: Statistical questions require conditional aggregation or testing, including variances, common values, Pearson correlations, and chi-square null-hypothesis decisions.The examples filter files or rows by specified variable values before calculating univariate or bivariate results.
  • Question categories: Paraphrased questions preserve the statistical task while replacing templated wording with more natural formulations.Examples restate relationships involving packet-size misclassification, phase coherence, and experimental conditions.

I Questions considered in Evaluation

The evaluation samples 500 questions across answerable and unanswerable cases, categories, file extensions, and seeded stochastic examples. Question counts are structured so later, stochastic types receive repeated repository-specific instances.

  • Evaluation sample: Table 7 reports the number of sampled questions by category and question type for the 500-question evaluation.The table organizes the evaluation sample across the benchmark’s question taxonomy.
  • Evaluation sample: 500 questions comprise 361 answerable cases (72.2%) and 139 unanswerable cases (27.8%).Each question is evaluated in one templated form and three paraphrase variants.
  • Evaluation sample: The sample covers five progressively difficult categories, with stochastic question types receiving five seeded examples per repository.The final seven question types therefore have about five times as many samples as the first four types.
  • Evaluation sample: The sampled repositories use six file extensions: .csv, .json, .jsonl, .log, .txt, and .xlsx.The evaluation counts are 76, 94, 83, 77, 83, and 87 respectively.

J Evaluation details

Models receive repository and Python tools and are prompted to return structured answers, while deterministic grading handles categorical, integer, and continuous responses. Continuous answers receive tolerance through reduced-significant-digit matching.

  • Model and tool setup: Proprietary models use MCP-compatible tool interfaces, while open-weight models use a ReAct-like framework through smolagents.Both setups support tool-mediated interaction with the repository.
  • Model and tool setup: The benchmark exposes directory and file-reading functions plus a Python interpreter with numpy, pandas, and scikit-learn.The tools allow models to inspect repository contents and programmatically analyze data.
  • Response format: Models return an answer JSON object, and unanswerable questions require the response “not possible.”Invalid JSON responses, occurring in fewer than 3% of cases, are graded as the entire final response.
  • Deterministic grading: Categorical answers are checked for the correct exclusive choice, while integer answers require an exact integer match after limited numeric extraction.The grading procedure normalizes categorical responses to lowercase and can extract numbers from responses containing units.
  • Deterministic grading: Continuous answers are graded using the requested significant figures, with one fewer significant digit accepted for exact matching.For example, a requested three-significant-digit answer of 1.234 accepts 1.235 because both match at two significant digits.

K Example LLM responses

The response examples contrast context-window reasoning with code-based analysis on a file-row-count task. The code-based trajectory obtains the correct result, while the context-window approach gives an incorrect answer after locating the file.

  • Contrasting strategies: The code-based GPT-5.4 trajectory returns 163, the correct answer, whereas Gemma 3 27B it answers incorrectly.The example attributes the difference to solving with code rather than relying on the context window.
  • Contrasting strategies: The Gemma trajectory navigates to the target directory and opens the file before producing the wrong result.The omitted intermediate steps are described as similar to the shown directory-navigation actions.
  • Example task: A repository question asks for the number of data rows excluding the header in a specified JSON file.The task uses repository id 23 and a nested path under cond=skw_paillier_ns=0.05.
  • Paraphrase audit: The paraphrase audit finds only GPT-OSS 20B paraphrases both semantically equivalent and unambiguous among the sampled models.Gemma 3 27B it and Qwen3 4B Instruct paraphrases are treated as robustness checks whose downstream results require caution.
Loading 2604.13201v2…