PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenProblemBench: Benchmarking AI on Open Problems in the Foundational Theoretical Sciences

Zhiyi Li, Sihan Hu, Tianning Xiao, Xiansheng Cai, Xiaojun Tan, Youjin Deng, Kun Chen

arXiv:2610.11118v1cs.AI

TL;DR

Conventional scientific benchmarks usually have known targets, leaving open whether AI can advance unresolved research questions. OpenProblemBench evaluates 82 literature-grounded mathematics and theoretical-physics problems with independent model judges; GPT-6-Astra achieved the highest mean judged solve rate, 14.02%.

  • Problem

    Most scientific benchmarks use published results or known targets, leaving whether AI can advance unresolved research questions as an open evaluation question.

    Source: §1 · pp. 1–2

  • Method

    OpenProblemBench curates 82 literature-grounded mathematics and theoretical-physics problems and has four evaluator models independently assess submissions without reference answers.

    Source: §1 · p. 2, §3.4 · p. 4

  • Results

    Across 82 problems, GPT-6-Astra had the highest mean judged solve rate at 14.02%, while every other evaluated configuration remained below 8%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The benchmark assesses AI progress on unresolved research questions and connects outcomes to representation changes, general arguments, and proofs of decisive steps.

    Source: §5 · p. 9

    The results are model judgments, and independent expert review is needed to establish scientific correctness and novelty.

    Source: §6 · p. 9

Abstract

from arXiv · show

The next frontier for artificial general intelligence is tackling unresolved scientific problems, calling for benchmarks that assess progress beyond established knowledge. We introduce OpenProblemBench, a benchmark of 82 unresolved problems drawn from the mathematics and theoretical physics literature. Each problem supplies the research context, assumptions, and prior progress needed to investigate the question. We select problems whose proposed solutions admit comparatively clear checks of their decisive mathematical or computational claims. Four evaluator models independently assess the correctness, completeness, and degree of progress of each submission without reference solutions. Across seven evaluated configurations, GPT-6-Astra achieves the highest mean judged solve rate of 14.0%, compared with 5.5-6.7% for the evaluated full-size open models and 2.4-3.7% for Flash models. Case comparisons connect stronger outcomes to changes in problem representation, general arguments that extend beyond finite evidence, and proofs of the steps needed to complete a solution. By grounding evaluation in questions arising from the research literature, OpenProblemBench provides a setting for investigating the capabilities and limitations of AI as a contributor to foundational theoretical science.

1 Introduction

Conventional benchmarks generally use known targets, leaving open whether AI can advance unresolved research questions. OpenProblemBench evaluates progress on literature-grounded problems whose proposed solutions admit comparatively structured checks.

1 Introduction

  • Unlike tasks with published results or known targets, unresolved problems require systems to find productive directions and establish what evidence closes the argument.Plausible derivations may still leave quantifiers, parameter regimes, or exceptional cases untreated.Source: §1 · pp. 1–2
  • Open problems can be difficult to solve yet comparatively structured to verify through checking assumptions, necessity and sufficiency, computational certificates, or decisive proof steps.This combination makes selected open problems suitable for evaluating research-oriented systems.Source: §1 · p. 2
  • OpenProblemBench contains 82 mathematics and theoretical-physics problems with research context, assumptions, known progress, and resolution conditions.Four evaluator models independently distinguish solved outcomes from trivial, nontrivial, and breakthrough partial progress, without reference answers.Source: §1 · p. 2
  • 14.02% was GPT-6-Astra’s mean judged solve rate, the highest across seven configurations; all others remained below 8%.Other systems often received substantive-progress judgments, and selected cases link outcomes to problem representation and proof of decisive steps.Source: §1 · p. 2
  • The benchmark compares how systems select productive directions, develop arguments, and establish progress beyond the current expert literature.It measures complete solutions and useful advances where no reference answer is available.Source: §1 · p. 2

2 Related Work

Prior benchmarks evaluate demanding scientific reasoning, research workflows, and unresolved-question tasks, but OpenProblemBench focuses on literature-grounded problems with no known final answer. It uses independent model judges and comparatively clear checks of proposed solutions.

2.1 Scientific reasoning with known targets

  • Benchmarks for scientific reasoning and research workflows evaluate tasks with known targets, whereas OpenProblemBench asks about unresolved questions.These works assess reasoning, evidence use, research processes, reproduction, or generation, but retain a published result or empirical objective.Source: §2.1 · pp. 2–3

2.2 Research reproduction and generation

  • Research-scale benchmarks test reproduction, generation, or empirical advances against published results or empirical objectives, unlike unresolved questions in the source literature.Examples include PaperBench, PRL-Bench, Terminal-Bench-Science, and MLRC-Bench.Source: §2.2 · p. 3

2.3 Evaluation on unresolved questions

  • OpenProblemBench extends unresolved-problem evaluation with literature-grounded mathematics and physics questions, independently reviewed by multiple agent judges without requiring proof-assistant formalization.Its problems include assumptions and unresolved conclusions from prior work, and selected solutions admit comparatively clear checks.Source: §2.3 · p. 3

3 Benchmark Construction and Evaluation

The benchmark collects literature-grounded questions, runs solvers under documented configurations, and uses independent model judges to assess solutions and partial progress. Its reports define outcome categories and aggregate them across solver–evaluator pairs.

3.1 Literature-grounded problem collection

  • Problems are sourced from literature-derived candidates, checked against source and subsequent work, and written as self-contained statements with context, significance, progress, and references.Selection favors proposed solutions that can be scrutinized through decisive proof steps, assumptions, or consequential calculations.Source: §3.1 · p. 3

3.2 Composition and scope

  • The 82 problems span mathematics and theoretical physics, with origins ranging from recent work to questions open for centuries.The collection contains 46 mathematics and 36 physics problems across multiple subfields.Source: §3.2 · p. 4

3.3 Solver configurations and execution

  • Seven solver configurations submit one solution per problem; runs allow local computation, disable network retrieval in the primary setting, and permit up to 12 hours of work.The comparison is of model–execution configurations, and an additional Qwen condition compares OpenCode with Claude Code.Source: §3.3 · p. 4

3.4 Agent-as-judge assessment

  • Four evaluator models independently check obligations, assumptions, quantifiers, decisive steps, and computations, assigning solved, partial, or unsolved outcomes.Partial progress is graded as trivial, nontrivial, or breakthrough; reports also document established contributions, remaining gaps, and overclaiming.Source: §3.4 · pp. 4–5

3.5 Reported outcomes

  • Solve rate is the number of problems judged solved divided by 82, and overall outcome percentages average equally across the four evaluators.The study reports partial-progress levels, nonsolved outcomes, and incomplete assessments; all 28 primary solver–evaluator cells contain 82 reviews.Source: §3.5 · p. 5

4 Results and Analysis

Astra leads the overall judged solve-rate comparison, while the outcome distributions also reveal differences in substantive partial progress. The sensitivity tests and case comparisons show how execution settings and research approaches relate to completion, and the disciplinary comparison is bounded by the benchmark’s task composition.

4.1 Overall performance across evaluators

  • GLM-Flash receives fewer solved judgments on average than GLM (3.66% versus 5.49%), yet their combined solved, breakthrough, and nontrivial shares are close (63.72% versus 62.80%).Evaluator-specific solved subsets overlap only partially, making per-problem comparisons more informative than a single aggregate ordering.Source: §4.1 · p. 5
  • 14.02% is Astra’s highest mean judged solve rate; it also has the largest combined share of solved, breakthrough, or nontrivial outcomes, at 84.45%.Figure 1 averages each outcome category equally across evaluators.Source: §4.1 · p. 5, Figure 1 · p. 6

4.2 Partial progress and reporting reliability

  • Among completed reviews not judged solved, substantive-progress rates vary by evaluator; overclaim diagnostics include conditional results presented as unconditional and finite checks extrapolated to infinite families.A total of 2,095 partial judgments comprise 768 trivial, 1,252 nontrivial, and 75 breakthrough outcomes.Source: §4.2 · p. 6

4.3 Harness and web-access sensitivity

  • For Qwen, changing harness reduced solved counts from 7 to 5 and substantive partials from 68 to 59; web access left solved counts at 5 but raised substantive partials from 59 to 68.With web access, breakthrough outcomes rose from 4 to 7; three problems gained a solved judgment and three lost one.Source: §4.3 · pp. 6–7, Table 1 · p. 7

4.4 How models approach the same open problems

  • On knot and percolation problems, Astra reformulates the target to reach a counterexample or finite-torus identity, while other attempts stop at computations, bounds, or intermediate quantities.The comparisons contrast a permitted refutation route with work that does not establish the requested conclusion.Source: §4.4 · pp. 7–8
  • On entropy and algebra-classification problems, exact finite computations or restricted regimes do not settle the general question; Astra’s graph-size-independent bound solves the entropy problem.The algebra classification remains incomplete without excluding all further isomorphisms.Source: §4.4 · p. 8
  • Fusion cases distinguish complete correspondence from promising construction: Astra closes proof gaps, while Flash models solve selected tasks but leave auxiliary assumptions or original-task coverage unresolved.For non-orientable surface counting, implementation agreement was insufficient where implementations shared an incorrect dual convention.Source: §4.4 · pp. 8–9

4.5 Disciplinary differences and evaluator agreement

  • Ten problems were judged solved by all evaluators, eight by Astra; disciplinary comparisons remain bounded by curation toward checkable mathematics and physics construction extensions.The authors caution that this task composition does not establish general disciplinary proficiency.Source: §4.5 · p. 9

5 Conclusion

OpenProblemBench evaluates AI on unresolved research questions in mathematics and theoretical physics. Across 82 problems, Astra leads under every evaluator, while case comparisons connect outcomes to problem representation, general arguments, and completion of decisive proof steps.

5 Conclusion

  • Across 82 problems, Astra has the highest solve rate under every evaluator, while several other models make substantive partial progress and Flash models solve a subset of tasks.The case comparisons examine changing problem representation, extending finite results to general arguments, and proving steps needed for a full conclusion.Source: §5 · p. 9

6 Limitations and Future Work

The benchmark’s scope and evaluation design impose limits on how broadly its results can be interpreted. The authors identify uneven field coverage, possible training-data exposure, and variability in historical runs as issues for future work.

6 Limitations and Future Work

  • The results are model judgments, so independent expert review is needed to establish scientific correctness and novelty.Evaluators may share errors, and agreement alone does not estimate false-positive or false-negative rates.Source: §6 · p. 9
  • The 82 questions are uneven across subfields, covering broad mathematics and a mathematically intensive subset of theoretical physics.Source: §6 · p. 9
  • Literature searches may miss solutions, and relevant statements may occur in training data.The authors recommend specifying a literature cutoff and extending future releases to additional fields.Source: §6 · p. 10
  • Historical runs vary in resources and execution software, while one run per ablation condition cannot measure run-to-run variance.Fine comparisons require repeated runs under common budgets.Source: §6 · p. 10

A Comparison of evaluation designs

Table 2 presents representative evaluation designs and distinguishes tasks with published targets from unresolved research questions.

A Comparison of evaluation designs

  • Table 2 distinguishes evaluation designs for tasks with published targets from designs for unresolved research questions.Source: Table 2 · p. 13

B Detailed Outcome and Diagnostic Counts

The appendix reports detailed outcome and diagnostic counts, with captions specifying the scope and interpretation of each table.

B Detailed Outcome and Diagnostic Counts

  • Eight of 2,296 primary reviews are marked Incomplete; all other primary reviews have completed outcome assessments.Five are GPT-Sol reviews of DeepSeek-Flash and three are Qwen reviews of Qwen.Source: B Detailed Outcome and Diagnostic Counts · p. 13
  • Table 3 reports submissions judged solved out of 82, with percentages in parentheses and incomplete assessments separately reported.Each cell contains 82 review records.Source: Table 3 · p. 13
  • Table 4 gives nontrivial, breakthrough, and all-partial counts for completed assessments, counting review events rather than unique discoveries.Source: Table 4 · p. 14
  • Table 5 reports overclaim counts as detected, not detected, or unknown; unknown usually means coverage was partial.A dash means there are no reviews for the entire cell.Source: Table 5 · p. 14
  • Table 6 reports determined overclaim-verdict coverage as a percentage of 82 submissions, with the complement representing the unknown share.Source: Table 6 · p. 14

C Cross-judge Solved Outcomes

The cross-judge outcomes show that Astra’s solved judgments are concentrated in a small subset of its submissions.

C Cross-judge Solved Outcomes

D Paired Harness Comparison

The section compares paired harness judgments and uses selected problem cases to show how exact identities and uniform proofs distinguish completed solutions from bounded or conditional progress.

D Paired Harness Comparison

  • 75 OpenCode and 64 Claude Code submissions reached solved or at least nontrivial progress; among four discordant solve outcomes, (g, l) = (1, 3) gave an exact conditional McNemar probability of 0.625.The comparison covers archived configurations; repeated runs under common resource budgets would be needed to isolate harness effects from run-to-run variation.Source: D Paired Harness Comparison · p. 15

E.1 Selection and comparison procedure

  • The ten-question qualitative sample compares observed approaches, while its four-example table holds the evaluator fixed to Qwen and adds GPT-Sol.The sample supports comparisons of approaches, not estimates of their frequency across the benchmark.Source: §E.1 · p. 16, Table 7 · p. 16

E.2 ORB-PHYS-57: Exact critical manifold of the inhomogeneous bow-tie lattice

  • For the bow-tie lattice, Astra replaces a problematic local-probability interpretation with a finite-torus identity and a limiting argument; Astra’s and Kimi’s solutions are judged solved by Qwen.Qwen’s bounds do not collapse to the claimed surface, while GLM’s argument lacks full-domain completion.Source: §E.2 · p. 16

E.3 ORB-PHYS-62: Pauling’s residual entropy estimate at growing degree

  • O((log d/d)^(1/3)) is Astra’s n-independent bound after spectral truncation, and all four evaluators judge its residual-entropy answer solved.Other submissions leave a size restriction, a conditional local-limit estimate, or a uniform bound unresolved.Source: §E.3 · p. 17

E.4 ORB-PHYS-69: Sklyanin algebra classification in exceptional fields

The Sklyanin classification case turns on proving completeness across exceptional fields and strata, while the generalized lambda-function case supplies a shared success for several models; the coverage appendix lists the retained problems and their subfield assignments.

E.4 ORB-PHYS-69: Sklyanin algebra classification in exceptional fields

  • The Sklyanin task requires necessary and sufficient conditions over arbitrary fields, including exceptional strata and descent; Astra and GPT-5.6-Sol are judged solved, while Qwen leaves proof obligations unresolved.Astra uses a contraction-determinant invariant in characteristic three and a central-quotient Hilbert-series argument; Qwen’s remaining gaps include a special-line hypothesis and characteristic-two proof.Source: §E.4 · p. 17

E.5 ORB-PHYS-82: Integrality of generalized lambda-function values

  • All seven evaluated models are judged solved on the generalized lambda-function task, which extends Ishii (2011) to excluded parameter regimes.The proof routes differ: GLM-Flash combines cyclotomic valuation cases with symbolic elimination, while Qwen and DeepSeek-Flash use other valuation-based arguments.Source: §E.5 · p. 18

F Problem Coverage

  • The coverage appendix provides primary-subfield assignments for the 82 retained problems and lists the retained problem statements.The titles are inherited from the archive.Source: Table 8 · p. 18, Table 9 · p. 18
Loading 2610.11118v1…