PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Selected Breakthroughs: Evaluating Language Models on Randomly Sampled Mathematical Conjectures

Helen Jenne, Davis Brown, Jesse He, Henry Kvinge

arXiv:2610.40382v1cs.AImath.HO

TL;DR

Evidence on LLM mathematics has focused on major breakthroughs, leaving open how often models resolve randomly sampled conjectures and what their solutions contribute. This paper builds OpenConjecture and studies three-pass GPT attempts on 200 conjectures; models proposed resolutions for 125, while the authors emphasize evaluating more than correctness.

  • Problem

    The paper asks how often LLMs prove versus disprove conjectures, how many they resolve with simple prompting, and how novel their solutions are on a random sample.

    Source: §1 · pp. 1–2

  • Method

    The authors introduce OpenConjecture and prompt frontier GPT models in three sequential passes on 200 sampled conjectures, analyzing candidate solutions beyond correctness.

    Source: §1 · p. 2

  • Results

    Models proposed resolutions for 125 of 200 conjectures, including 70 counterexamples and 55 proofs; 84% of proposed resolutions held up to LLM review.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Evaluation of mathematical outputs should consider correctness, fidelity to the intended statement, relation to existing literature, and mathematical contribution.

    Source: §7 · p. 9

    Proposed resolutions were not expert-verified, and each was reviewed by a new instance of the same model that produced it.

    Source: §6 · p. 9

Abstract

from arXiv · show

The last year has seen a shift in AI-assisted mathematics, with frontier language models producing proofs of and counterexamples to long-standing open conjectures. However, studying only the current set of major breakthroughs provides an incomplete picture of LLM capabilities and the computational resources needed to achieve reported results. To address this, we introduce a new dataset, OpenConjecture, which collects conjectures from mathematics papers published on the arXiv since February 2026. We use these conjectures as a source of open mathematical problems, and apply frontier GPT models to a random sample of 200 of these conjectures over three passes using a reproducible protocol without problem-specific mathematical guidance. Rather than attempt expert verification of these candidate solutions, we analyze the types of problems for which LLMs produce solutions and the characteristics of these solutions. The models we test produce candidate solutions for roughly 60% of the conjectures, with counterexamples about 1.3 times more common than proofs. We estimate solution novelty, assess how solutions cite and relate to existing literature, and analyze whether the proposed resolution relies on either an error in or underspecification of the conjecture. We find that models sometimes propose resolutions that are outside the authors' intent (e.g., proposing the trivial group as a counterexample) and that a number of proofs follow closely from existing results, but that some resolutions exhibit at least moderate novelty. Our findings underscore the need for evaluations that go beyond correctness to assess fidelity to the intended statement, attribution, and the nature of the mathematical contribution.

1 INTRODUCTION

The paper asks how frontier LLMs resolve randomly sampled conjectures and what their solutions contribute beyond correctness. It introduces OpenConjecture, analyzes three-pass GPT attempts on 200 conjectures, and finds that proposed resolutions vary substantially in mathematical contribution.

1 INTRODUCTION

  • The study asks whether LLMs more often disprove or prove conjectures, how often one naïve prompt suffices, and how much solutions draw on existing literature.It also asks how often repeated prompting or domain expertise is needed and how much novelty solutions introduce.Source: §1 · pp. 1–2
  • OpenConjecture is a dataset of conjectures from recent arXiv mathematics preprints, used as a source of conjectures drawn ‘from the wild’ to study LLM mathematical ability beyond correctness.The authors study 200 conjectures, one drawn from each of 200 randomly sampled papers, prompting GPT-5.6 Sol, GPT-5.6 Sol Pro, and GPT-6 Astra Pro across three passes, each receiving the previous pass’s unresolved attempts as context.Source: §1 · p. 2
  • Among first- and second-pass counterexamples that passed LLM review, roughly 60% were classified as off-the-shelf and the remainder as standard constructions.Third-pass counterexamples typically involved nontrivial modifications or more involved arguments; some proposed counterexamples exploited missing hypotheses rather than offering an interesting construction.Source: §1 · p. 2
  • 70 conjectures received proposed counterexamples and 55 received proposed proofs; 84% of proposed resolutions held up to LLM review.The paper also analyzes solution length, counterexample novelty and author-intent alignment, and relationships between proposed proofs and existing literature.Source: §1 · p. 2
  • The study does not attempt expert verification or formalization of the conjecture attempts.It is not presented as a benchmark for comparing models.Source: §1 · p. 2

2 RELATED WORK

Related work includes benchmarks of research-level problems and datasets sourced from recent arXiv papers. OpenConjecture differs by collecting conjectures posed by human mathematicians rather than lemmas or synthetic questions.

2 RELATED WORK

  • The paper situates its methodology within a broader and growing field of AI for mathematics.It points readers to a survey (Ju & Dong, 2026) and a curated list of papers.Source: §2 · p. 2
  • Unlike LemmaBench and arXivMath, OpenConjecture collects conjectures made by human mathematicians, rather than lemmas or questions generated from abstracts.LemmaBench rewrites lemmas from recent arXiv preprints, while arXivMath generates questions from abstracts and applies additional filtering and preprocessing.Source: §2 · p. 2
  • First Proof’s second batch tested and expert-graded four systems on 10 problems; at least seven problems received a passing grade.The passage also reports solutions differing from known ones on three problems and significant attribution failures in several solutions.Source: §2 · p. 3
  • Other research-level benchmarks include Benchmarks in Leipzig, IMProofBench, Soohak, Riemann-Bench, and FrontierMath.Benchmarks in Leipzig contains 100 research-level questions with unique, well-defined answers contributed by 49 mathematicians.Source: §2 · p. 3

3 OPENCONJECTURE

OpenConjecture is built from recent arXiv mathematics preprints by extracting conjectures and filtering for author-posed, open problems. The June 21 snapshot and further filtering yield a dataset whose remaining conjectures are concentrated in several mathematical subfields.

3 OPENCONJECTURE

  • The dataset ingests recent arXiv math preprints, extracts conjecture blocks, and uses GPT-5 Mini to label author-posed conjectures and estimate mathematical interestingness and near-term solution viability.Its first snapshot was published March 17, 2026, and the dataset is updated weekly.Source: §3 · p. 3
  • The June 21 snapshot contained 3,030 conjectures from 1,675 papers labeled open by GPT-5 Mini; filtering removed 769 conjectures from papers outside the intended math-primary-classification scope.The removed papers were cross-listings or updates rather than papers from the intended window.Source: §3 · p. 3
  • A second GPT-5.4 open/not-open filtering round reclassified 337 of 2,261 downselected conjectures as not real, leaving 1,924 labeled open.The filtering round was motivated by arXivMath’s finding that 30% of 20 reviewed questions were potentially answerable using cited prior work.Source: §3 · p. 3
  • Combinatorics, number theory, and algebraic geometry account for the majority of the remaining 1,924 conjectures.Figure 1 shows the distribution by math primary subject classification.Source: §3 · p. 4

4 FINDING CANDIDATE SOLUTIONS TO OPEN CONJECTURES

The study samples one conjecture from each of 200 uniformly sampled eligible papers, then applies up to three sequential GPT solver passes. Later passes receive unresolved attempts as context, with a stronger model used at each stage.

Sampling

  • The 200 papers were sampled uniformly, followed by one uniformly sampled conjecture per paper to avoid overrepresenting conjecture-heavy works.This aimed to reflect the global population of recent arXiv math preprints containing conjectures without overweighting subfields that publish few conjectures.Source: §4 · p. 4

Solver

  • Up to three sequential passes used GPT-5.6 Sol, GPT-5.6 Sol Pro, and GPT-6 Astra Pro; only unresolved conjectures advanced.The model strength increased across passes to approximate a cost-constrained workflow.Source: §4 · p. 4
  • The first pass gave GPT-5.6 Sol the conjecture, full paper, and a simple instruction to attempt a proof or refutation, with proof, counterexample, or unresolved as verdicts.The model was instructed to cite external results, and web search was permitted in this pass.Source: §4 · p. 4
  • For unresolved cases, later passes received necessary conjecture context and preceding model responses; the third pass tested whether a stronger model could resolve them.Web search was removed from the second and third passes after first-pass responses appeared focused on literature review rather than the problem.Source: §4 · p. 4

5 RESULTS AND ANALYSIS

Across three passes, models proposed 70 counterexamples and 55 proofs for 200 conjectures, with LLM review used to screen responses. Analyses find varied counterexample novelty and fidelity, while proof responses sometimes draw on prior work or earlier attempts.

5 RESULTS AND ANALYSIS

  • 70 counterexamples and 55 proofs were proposed across three passes; counterexamples outnumbered proofs in the first two passes, but proofs outnumbered them in the third.The pass-by-pass counts were 49, 15, and 6 counterexamples, and 33, 11, and 11 proofs; Figure 2 also shows final status and pass of resolution.Source: §5 · p. 4, Figure 2 · p. 5
  • All third-pass resolutions and 80–85% of first- and second-pass resolutions passed LLM review, but expert verification was outside the study’s scope.Researchers manually checked for obvious issues; subsequent analyses focus on responses that passed the LLM check.Source: §5 · p. 5
  • First-pass responses averaged 3.3 citations and a median of 2 distinct citations; later passes had a median of 2, with about 60% citing named theorems or results.Citation requests brought a trade-off: first-pass responses seemed more focused on literature review than working on the problem, with a median of 131,783 input tokens versus 6,309 in the second pass.Source: §5 · pp. 5–6

5.1.1 METHODS

  • The review assessed counterexample object and argument novelty as well as whether authors plausibly intended to exclude the case.It distinguished off-the-shelf examples, standard constructions, nontrivial modifications, and genuinely novel constructions.Source: §5.1.1 · p. 7

5.1.2 RESULTS

  • About 60% of verified first- and second-pass counterexamples were classified as off-the-shelf, while 5 of 6 third-pass examples were nontrivial modifications.Third-pass counterexample responses were longer and more often assessed as nonroutine by LLM reviewers.Source: §5.1.2 · p. 7
  • A significant portion of first- and second-pass counterexamples were judged plausibly due to imprecision in the conjecture statement.Reviewers identified possible convention mismatches, misstatements or order-of-operations errors, and missing projectivity or properness hypotheses.Source: §5.1.2 · p. 7

5.2 PROOF ANALYSIS

  • Proofs settled in later preprints, straightforward consequences of existing results, apparently original proofs, and unclear cases formed the literature-relation categories; 3 of 9 later-settled proofs included an independent argument.The classifications drew on LLM verification responses and were made in a non-specialist review.Source: §5.2 · p. 8
  • Third-pass proofs built on earlier-pass work in 7 of 11 cases, while unresolved responses had medians of 6, 5, and 8 proved statements across the three passes.The authors caution that apparent progress in unresolved responses does not establish that the work is useful or correct.Source: §5.2 · pp. 8–9

6 LIMITATIONS

The evaluation reports candidate resolutions and LLM-review outcomes, not expert-verified solutions. It covers three GPT models without controlled model-family comparisons or testing each model’s best achievable performance.

Correctness of proposed resolutions

  • Proposed resolutions were reviewed by a new instance of the same model that generated them, so solver and reviewer may share mathematical errors.The proposed proof and counterexample counts and review-pass rates should not be read as rates of newly solved open problems.Source: §6 · p. 9

Scope of the evaluation

  • The study evaluated three GPT models, not a controlled comparison across model families or each model’s best achievable performance.It did not test multiple attempts or recommended harnesses; run-to-run variability and effects of alternative prompts and tools remain for future work.Source: §6 · p. 9

7 DISCUSSION

The study applies frontier GPT models to conjectures from recent arXiv preprints rather than benchmark-selected problems, and proposed resolutions for 125 of 200 conjectures. Its analysis emphasizes that mathematical contributions involve correctness, fidelity to intent, relation to literature, and contribution type.

7 DISCUSSION

  • Models proposed resolutions for 125 of the 200 conjectures, including literature-based arguments, standard constructions, potentially intent-mismatched counterexamples, and potentially original arguments.The paper argues that assessing mathematical contributions involves correctness, fidelity to the intended statement, relation to existing literature, and the nature of the contribution.Source: §7 · p. 9
  • The authors used generative AI for methodology feedback, prompt writing, figure and table creation, coding, and readability feedback, while taking responsibility for the final content..Source: AI USE STATEMENT · p. 9

A OPENCONJECTURE DATASET DESCRIPTIVE STATISTICS

The June 21 snapshot contains 3,030 conjectures labeled real by GPT-5 Mini, drawn from 1,675 papers. The appendix describes the labeling criteria and reports that the per-paper conjecture counts are heavily right-skewed.

A OPENCONJECTURE DATASET DESCRIPTIVE STATISTICS

  • GPT-5 Mini labeled a conjecture real when its extracted statement remained unresolved as written and was posed as an active open target.Statements already proved or refuted, solved special cases, background or inactive statements, and claims immediate from known results were labeled not real; insufficient context was uncertain.Source: A OPENCONJECTURE DATASET DESCRIPTIVE… · p. 11
  • For labeling, GPT-5 Mini received the conjecture block plus 20 lines before and after it.The listed labels distinguish real open conjectures, not-real conjectures, and uncertain cases.Source: A OPENCONJECTURE DATASET DESCRIPTIVE… · p. 11
  • 3,030 of 3,738 conjectures in the June 21 snapshot were labeled real, 689 not real, and 19 uncertain; the real conjectures came from 1,675 papers.Most papers with conjectures contributed only one or two, and the per-paper count distribution was heavily right-skewed.Source: A OPENCONJECTURE DATASET DESCRIPTIVE… · p. 11, Table 1 · p. 11
  • 769 conjectures came from 432 papers without a math.* primary category; 95 were cross-listings and 337 had an empty primary category.Most of the 337 papers with empty primary categories appeared to be updates, with publication dates typically before the first snapshot.Source: A OPENCONJECTURE DATASET DESCRIPTIVE… · p. 11

B ADDITIONAL RESULTS

The appendix presents distributions of solver token use and first-pass web searches, alongside response-citation, length, and definition-count breakdowns. The supplied captions identify what these visuals measure but do not provide their plotted values.

B ADDITIONAL RESULTS

  • Figure 6 shows input- and output-token distributions by solver pass; only the first pass allowed web search, which counts toward input tokens.Input-token sample sizes were 197, 118, and 92 for the first, second, and third passes, respectively; three of 200 CIDs lacked first-pass token data.Source: B ADDITIONAL RESULTS · p. 11, Figure 6 · p. 12
  • Figure 7 presents the first-pass distribution of web-search calls per response; its caption does not report median call counts, query counts, or a citation-request trade-off.Source: Figure 7 · p. 13
  • Figures 8–10 classify response citations and show citation counts per response, excluding citations to the source paper.The available captions identify first-, second-, and third-pass breakdowns but do not give plotted values.Source: B ADDITIONAL RESULTS · p. 11, Figure 8 · p. 13, Figure 9 · p. 13
  • Figure 11 measures prose word count and total claims with proof, while Figure 12 groups prose word count by math.* classification.The passage introducing these figures labels them in the reverse order, so the captions identify the measures used here.Source: B ADDITIONAL RESULTS · p. 11, Figure 11 · p. 14, Figure 12 · p. 15
  • Table 2 counts solver-introduced definitions by pass and response type, excluding definitions from the source paper, external literature, or standard mathematical background.Response types are proof, counterexample, and unresolved.Source: B ADDITIONAL RESULTS · p. 11, Table 2 · p. 15

C CONJECTURE SOLUTION PIPELINE: ADDITIONAL DETAILS

The pipeline compared giving the model the full paper with supplying the conjecture and extracted relevant context. Context extraction often remained incomplete, particularly for assumptions stated elsewhere or left implicit.

C CONJECTURE SOLUTION PIPELINE: ADDITIONAL DETAILS

  • The first pass tested full-paper input against a conjecture plus relevant context; later passes used the conjecture and context with prior attempts.The later-pass setup was intended to have models build on earlier attempts.Source: C CONJECTURE SOLUTION PIPELINE:… · p. 15
  • Even after prompt iteration and tests with stronger models, extracted context packs were often judged insufficient by an LLM review.The context was extracted from the paper to create a self-contained conjecture statement.Source: C CONJECTURE SOLUTION PIPELINE:… · p. 15
  • Implied assumptions can be difficult to extract because they may appear in the introduction or be omitted while remaining understood within the subfield.Source: C CONJECTURE SOLUTION PIPELINE:… · p. 16

C.1 SOLVER PROMPTS

The solver prompts direct models to attempt proofs or counterexamples while checking literature, interpretation, and prior attempts; separate prompts specify context extraction and solution review. The protocols also set explicit standards for novelty assessment and for when a result should remain unresolved.

C.1 SOLVER PROMPTS

  • The solver must give exactly one verdict: PROOF, COUNTEREXAMPLE, or UNRESOLVED, with unresolved work stating established results and the remaining gap.A claimed proof or counterexample with an unresolved substantive gap is worse than an honest unresolved verdict.Source: §C.1 · p. 16
  • The second pass treats earlier work as untrusted notes, asks for sustained attempts by different routes, and accepts UNRESOLVED when a gap remains.The prompt requires checking claims before relying on them and downgrading unsupported arguments to UNRESOLVED.Source: §C.1 · pp. 16–17
  • The first-pass prompt asks for a rigorous proof or explicit counterexample, precise citations for external results, and explicit handling of settled or ambiguous statements.It also specifies mathematical theorem, lemma, and proof formatting.Source: §C.1 · pp. 16–17
  • The third-pass prompt adds web search and frames prior attempts as exploratory first-pass work followed by a closed-book second pass.Source: §C.1 · pp. 16–17

C.2 SYSTEM PROMPT FOR CONTEXT EXTRACTION

  • The context-extraction prompt identifies definitions, notation, applicable assumptions, and directly referenced results without solving or evaluating the conjecture.It orders selected spans by mathematical dependency and classifies dependencies as included, statement-bound, standard, external, or unresolved.Source: §C.2 · pp. 17–18
  • Context extraction distinguishes indispensable external results from missing material internal to the paper and dependencies whose location is unknown.Source: §C.2 · p. 18

D.1 COUNTEREXAMPLES

  • Counterexample assessment requires a verdict on whether the witness refutes the statement, while novelty assessment ends in a separate novelty verdict.Source: §D.1 · pp. 19–20
  • Counterexample review separates correctness from novelty and classifies the construction, argument, prior appearance, and possible mismatch with author intent.The novelty scale runs from off-the-shelf examples through standard constructions and nontrivial modifications to genuinely novel constructions.Source: §D.1 · pp. 19–20

D.2 PROOFS

The proof-review prompt instructs an independent audit of a proposed proof against the exact conjecture, including its reasoning and dependencies. It standardizes verdicts and reporting so that defects, routine completions, and later-settlement claims are distinguished.

D.2 PROOFS

  • The proof auditor independently checks implications, equivalences, external theorem hypotheses, hidden assumptions, circular reasoning, and missing cases against the original definitions.Minor routine details may be filled in only when explicitly identified; substantive gaps must not be silently repaired.Source: §D.2 · p. 20
  • Web search is reserved for specialized external results whose statements, hypotheses, definitions, or versions are not fully supplied.Source: §D.2 · p. 21
  • If a proof invokes a later settlement, the auditor checks whether it relies on that result or proves the claim independently; locating another proof does not validate the submitted proof.Source: §D.2 · pp. 21–22
  • The audit distinguishes verified proofs, invalid proofs, disproved statements, and inconclusive cases, with invalidity not implying that the conjecture is false.Source: §D.2 · p. 21
  • The required report records the exact statement, strategy, decisive checks and analysis, routine completions, dependencies, and whether later work settled the conjecture.The later-work field does not determine the proof-verification verdict.Source: §D.2 · p. 21
Loading 2610.40382v1…