PaperlayerView exact revision on arXiv

Source-linked AI summary

ResearchMath-14K: Scaling Research-Level Mathematics via Agents

Guijin Son, Seungyeop Yi, Minju Gwak, Hyunwoo Ko, Wongi Jang, Youngjae Yu

arXiv:2605.28003v1cs.CL

TL;DR

Research-level mathematics lacks broad, open training data, and it remains unclear whether language models can engage with unsolved problems. The paper builds datasets from academic open questions and filtered model trajectories, finding that fine-tuning on them improves Qwen3 models by 9.2 percentage points on average.

  • Problem

    Research-level mathematics lacks broad, open training data, limiting evidence about language models’ ability to engage with unsolved problems.

  • Method

    A multi-agent pipeline extracts and refines open problems from mathematical literature, then filters model-generated reasoning trajectories into a training-ready subset.

  • Results

    9.2 percentage points: fine-tuning three Qwen3 models on filtered trajectories improves them over base models, while newer models produce more fake references.

  • Takeaways & Limitations

    Filtered, imperfect open-problem attempts can provide useful supervision for research-level mathematical reasoning without ground-truth solutions.

  • Takeaways & Limitations

    An unfiltered fine-tune degenerated on nearly every evaluation, scoring close to zero.

Abstract

from arXiv · show

The frontier of mathematics is defined by problems whose solutions are not yet known, yet it remains unclear whether language models can meaningfully engage with such problems without human intervention. A major obstacle is the lack of large-scale research-level math datasets. To this end, we introduce ResearchMath-14k, a set of $14{,}056$ problems curated from academic sources via a multi-agent pipeline, making it the largest collection of research-level mathematical problems to date. We further generate ResearchMath-Reasoning, $220$K teacher trajectories from two open models, where we observe recurring avoidance behaviors such as non-attempts and fabricated references. Interestingly, across eight open-weight models, newer generations produce $5.6\times$ more references and $5.0\times$ more fake references per trace. After agentic filtering of ResearchMath-Reasoning, fine-tuning Qwen3 models from 4B to 30B parameters improves over base models by $9.2$ points on average. This shows that filtered open-problem attempts can provide useful supervision even without fully correct reasoning traces. We make ResearchMath-14k publicly available for future works on research-level mathematical reasoning.

1 Introduction

ResearchMath-14K addresses the shortage of open research-level mathematics data by using agents to extract and normalize questions from mathematical literature. Its accompanying reasoning corpus reveals recurring model failures, yet filtered trajectories improve Qwen3 models despite lacking ground-truth solutions.

  • Publicly available math training data mostly targets olympiad-level or easier problems, while research-frontier datasets are scarce and often held out to prevent contamination.
  • 1,233 open-problem lists and research papers are processed by agents that recover definitions, assumptions, and standalone formulations.The sources come from zbMATH, arXiv, and academic repositories.
  • 30% of 100 manually reviewed trajectories are visibly problematic, including non-attempts, narrower substitutions, and fabricated arXiv or PDF URLs.The same failure types recur in trace-level analysis across eight open-weight models.
  • 9.2 percentage points is the average improvement from fine-tuning 4B, 8B, and 30B-A3B Qwen3 models on a filtered 5,000-trace subset.The result indicates that filtered research-level attempts can provide useful supervision without ground-truth solutions.
  • 14,056 research-level mathematical questions comprise RESEARCHMATH-14K, accompanied by 220K teacher reasoning trajectories in RESEARCHMATH-REASONING.The datasets are openly released under the MIT license for future research-level mathematical reasoning work.

2 RESEARCHMATH-14K

ResearchMath-14K is built through an agentic extraction-and-refinement pipeline that creates a large, self-contained corpus of research-level mathematical problems from diverse sources. The dataset is broad and difficult, and its paired reasoning release exposes substantial teacher-model non-attempt behavior while providing large-scale supervision for open problems.

  • Pipeline: 20,835 problems are produced from 1,233 source documents by an Extractor that finds problem statements and a Refiner that makes them self-contained.The sources include arXiv open-problem papers, open-problem web pages, and problem-session sheets or curated lists.
  • Pipeline: 27.0 percentage points: refined statements are self-contained at a higher rate than original extractions, rising from 67.2% to 94.2%.Refined statements average 1,192 characters, compared with 290 at the Extractor stage, a 4.1× expansion.
  • Composition: 8,971 problems (63.82%) belong to four large areas—Analysis/PDEs/Dynamics, Mathematical Physics, Discrete Mathematics/Combinatorics, and Geometry/Topology.The taxonomy also assigns each problem to one of 28 macrosubjects and a research-level hierarchy containing 11,611 unique tags.
  • Difficulty: Roughly 400 Elo points: RESEARCHMATH-14K ranks above the compared existing math datasets on Knowledge, Novelty, and Procedural difficulty judgments.Ratings are computed from randomized cross-dataset pairwise comparisons judged by GPT-5-mini.
  • ResearchMath-Reasoning: 25/100 reviewed teacher trajectories contain no attempt, while 21/100 list related references and output “open” as the answer.Unfiltered fine-tuning led to repetitive outputs, frequent non-attempts, and degeneration, scoring close to zero on nearly every evaluation.
  • ResearchMath-Reasoning: 220K responses pair with 14K prompts, approximately 16 responses per prompt, forming ResearchMath-Reasoning from two teacher models.The release is described as the largest publicly available collection of model attempts on research-level mathematics.

3 Experiment Setup

The experiments compare older and newer model generations across research-level and control benchmarks, using complementary rule-based and agent-judged metrics to analyze reasoning behavior and citation factuality. The benchmark suite yields 446 short-form-answer prompts after sampling and filtering.

  • Models: The study evaluates DeepSeek, Kimi, and Qwen model families, grouping them into four matched older→newer pairs.The evaluated set includes substantially larger systems and both older and newer generations within each family.
  • Benchmarks: Four control benchmarks isolate the effects of research-level difficulty and AI-refined problem surface form in RESEARCHMATH-14K.SOOHAK and Leipzig Tier-4 are research-level controls for the AI-refining artifact.
  • Benchmarks: 446 prompts remain after sampling 90 items from each other benchmark, restricting SOOHAK to graduate-or-beyond challenge items, and filtering all benchmarks to short-form-answer questions.SOOHAK contributes 86 items after filtering.
  • Evaluation: Two complementary methods measure trace behavior and citation factuality, covering both axes of reasoning-trace analysis.The methods are designed to assess how models reason and whether their citations are factual.
  • Evaluation: Rule-based counting matches curated phrase lists for citation-like nouns, abandonment, and related recurring patterns, reporting the fraction of traces with at least one match.The rules are transparent and inexpensive, but matching alone cannot determine whether a phrase represents a contextual failure.
  • Evaluation: GPT-5.5 judges lemma decomposition in the first 30% of traces and verifies whether referenced sources exist on the web.Both judge outputs measure reasoning-trace properties rather than solution correctness.

4 Analyzing Reasoning Behavior on RESEARCHMATH-14K

Across eight models, newer generations cite substantially more often and fabricate more references on RESEARCHMATH-14K, while lemma decomposition is nearly absent. The results indicate that models often reproduce the style of research mathematics without engaging its underlying reasoning.

  • Citation behavior: 30–80 percentage-point increases in citation-like row-hit rates occur on the harder benchmarks, but the effect weakens on easier tasks.The increase appears on RESEARCHMATH-14K, Leipzig Tier-4, and SOOHAK, is modest on HLE, and is near zero on AIME.
  • Citation behavior: 87.4% of traces contain at least one reference-like object, while 54.0% contain at least one fake reference.The Agent-Judge examined 720 traces, with 19,864 extracted mentions and 17.6% labeled fake at the reference level.
  • Citation behavior: 5.6× more references and 5.0× more fake references per trace are produced by newer model generations across matched comparisons.Per-trace references increase sharply for DeepSeek, Kimi, and Qwen3 pairs, while fake-reference counts also rise.
  • Citation behavior: Models released in 2026 cite more and produce more fake citations than 2025 models across DeepSeek, Kimi, and Qwen families.The passage characterizes this pattern as factuality on research-level prompts moving backward and notes it is unlikely to be specific to one training set.
  • Research-style behavior: 94.0% of traces match the assume counter, whereas only 17.4% match the abandon counter.Models rarely give up outright, but the passage cautions that citations and compression are not inherently failures when correctly grounded.
  • Research-style behavior: 11/720 RESEARCHMATH-14K traces are positive for lemma decomposition, compared with only 18/2,128 across three benchmarks.The authors argue that research-level problems require decomposition into checkable subproblems rather than single-pass solutions.

5 Learning from RESEARCHMATH-14K

Filtering out non-attempts, unsupported claims, and fake citations yields open-problem trajectories that improve fine-tuned models despite lacking verified-correct solutions. The filtered data outperforms olympiad reasoning data in nearly every setting, especially on research-level evaluations.

  • Data filtering: 5,000 filtered traces are produced by verifying reference-like spans with web search and removing any trace containing a judged-fake reference.The filtering uses the Agent-Judge pipeline and forms RESEARCHMATH-REASONING-FILTERED.
  • Interpretation: Open-problem trajectories teach research-level reasoning more than olympiad data, even without solving the problems.Their advantage over DASD is clearest on research-level evaluations, indicating that generic mathematical reasoning exposure alone does not explain the gains.
  • Fine-tuning results: 8/9 cells favor RESEARCHMATH-REASONING-FILTERED over DASD, with a +2.6-point average advantage across HLE and SOOHAK.The largest gaps are +4.4 at HLE for the 30B model and +3.8 at SOOHAK for the 4B model; DASD wins AIME for 30B by 11.1 points.
  • Interpretation: Removing non-attempts, unsupported claims, and fake citations allows wrong-but-reasonable attempts to improve student models without verified-correct supervision.The authors leave testing this signal at larger scale for future work.

6 Related Works

Prior mathematical reasoning resources for LLMs have focused mainly on problems with known answers and generally fall below the research frontier. Research-level mathematical data remains difficult to scale because existing resources are often solved, synthetic, textbook- or olympiad-derived, formalized, expert-authored, private, gated, or small.

  • Research-Level Mathematics with LLMs: Existing LLM mathematical-reasoning resources mainly target problems with known answers rather than research-frontier mathematics.The cited examples span resources associated with known-answer mathematical reasoning.
  • Research-Level Mathematics with LLMs: These resources are commonly solved, verifiable, synthetic, textbook-derived, olympiad-derived, or tied to formal proof environments.The passage lists these categories as characteristic of existing resources.
  • Research-Level Mathematics with LLMs: Research-level mathematical data is expensive and nontrivial to scale, with existing resources often expert-authored, private or gated, small, or continuously maintained.The passage identifies these constraints as barriers to scaling research-level mathematical data.

7 Conclusion and Future Work · Appendix Contents

The work uses ResearchMath-14K to analyze open reasoning models on research-level mathematical problems, finding more citation-heavy responses and fake references in newer generations. Despite these imperfections, filtered trajectories improve fine-tuned models by an average 9.2 percentage points over their base versions.

  • 7 Conclusion and Future Work: ResearchMath-14K supports analysis of open reasoning models on research-level mathematical problems whose complete solutions are often unavailable.The study focuses on model behavior when definitive solutions may not exist.
  • 7 Conclusion and Future Work: Newer model generations produce more citation-heavy responses than earlier generations.The trace-level analysis identifies this as a concerning shift in model behavior.
  • 7 Conclusion and Future Work: Newer model generations also produce more fake references than earlier generations.The increase in fabricated citations accompanies the growth in citation-heavy responses.
  • 7 Conclusion and Future Work: Imperfect open-problem attempts still contain useful supervision for training.The conclusion treats flawed reasoning traces as potentially valuable rather than wholly unusable.
  • 7 Conclusion and Future Work: 9.2 percentage points is the average improvement from fine-tuning on filtered trajectories over base versions.The reported gain applies to models fine-tuned using the filtered trajectories.
  • 7 Conclusion and Future Work: The findings motivate using filtered, imperfect research-level math attempts as supervision for future model development.This follows the study’s observation that such attempts retain useful training signal despite citation problems.

A Example Source Comparisons

The section illustrates source-level contrasts in ResearchMath-14K through two side-by-side examples. These compare broad grand challenges with narrower contemporary open problems across arithmetic geometry and algebraic geometry.

  • Example Source Comparisons: Two side-by-side examples illustrate source-level contrasts in ResearchMath-14K.The examples are presented in Tables 3 and 4.
  • Example Source Comparisons: The first comparison pairs a classical arithmetic-geometry grand challenge with a narrower contemporary open-problem source.Both sources remain within number theory and arithmetic geometry.
  • Example Source Comparisons: The second comparison pairs a broad algebraic-geometry grand challenge with a narrower contemporary problem about hyperkähler Chow rings.This example contrasts broad algebraic geometry with a focused hyperkähler/Chow-ring problem.

B Self-Containment Audit

The self-containment audit measures whether refinement makes extracted questions understandable without their source documents. Codex evaluates both original and refined questions on 500 randomly sampled released records.

  • B Self-Containment Audit: 500 randomly sampled released records were audited by Codex for self-containment before and after refinement.A question counts as self-contained only when a mathematically trained reader can understand it without source-local notation, missing definitions, or external references.

C Keyword and Judge Metric Details … D.1 Pairwise Similarity Distribution

The appendix defines descriptive keyword counters and judge-based annotations for analyzing mathematical reasoning traces, then explains aggregation across models and benchmarks. It also reports pairwise embedding-similarity distributions for original statements and self-contained rewrites, with maxima below the 0.90 duplicate threshold.

  • C Keyword and Judge Metric Details: Lowercasing precedes keyword matching, and each group count sums exact substring occurrences; these descriptive features are not standalone hallucination classifiers.The specified groups include abandon, cite, and assume, covering stuckness, source references, and assertive or remembered-result language.
  • C.1 Keyword Groups: The abandon counter captures claims of being stuck, time-limited, unable to complete a solution, or resorting to guesses.Its phrases include “i’m stuck,” “cannot solve,” “given the time,” and “hazard a guess.”
  • C.1 Keyword Groups: The cite counter detects references to source objects, external databases, and citation-like artifacts such as papers, arXiv, DOI strings, and online sources.The list also includes books, journals, Wikipedia, MathWorld, OEIS, StackExchange, and related resources.
  • C.1 Keyword Groups: The assume counter combines assertive shortcuts with remembered-result language that substitutes confident assertion for derivation.Examples include “clearly,” “obviously,” “known result,” “by symmetry,” and “I recall.”
  • C.2 LLM-Judge and Agent-Judge Annotations: Agent-Judge verification searches mentioned references and records whether the cited source appears to exist, separating provenance signals from hallucinated bibliographic support.The annotation covers papers, books, articles, arXiv identifiers, DOI-like strings, named sources, and database references.
  • C.2 LLM-Judge and Agent-Judge Annotations: LLM-Judge annotations measure constructive proof organization by requiring reusable intermediate lemmas, claims, subgoals, or cases that structure subsequent reasoning.GPT-5.5 also marks explicit counterexample, edge-case, small-instance, or adversarial testing of conjectures and candidate solutions.
  • C.3 Aggregation: Row-hit rates aggregate counters and judged positives by model, family, and benchmark, while benchmark trends average newer-minus-older deltas across DeepSeek, Kimi, and Qwen pairs.The row-hit rate treats each trace as a binary hit for a counter or, for judged annotations, as positive when marked by the judge.
  • D.1 Pairwise Similarity Distribution: 98,778,540 problem pairs have pairwise embedding-similarity distributions whose maximum values remain below the 0.90 duplicate threshold for both original statements and self-contained rewrites.Figure 6 presents original-statement similarities on the left and rewrite similarities on the right.

D.2 GPT-5.5 Judgments Near Decision Boundary … F License and Release

Near the 0.9 similarity threshold, GPT-5.5 judged example problem pairs distinct, supporting a conservative threshold. The paper also reports behavior-counter rates and releases both ResearchMath artifacts under the MIT License from publicly available, non-paywalled academic sources.

  • D.2 GPT-5.5 Judgments Near Decision Boundary: Near the 0.9 similarity threshold, GPT-5.5 judged several problem pairs to be distinct.Tables 7, 8, and 9 provide these examples and support the conservative threshold choice.
  • D.2 GPT-5.5 Judgments Near Decision Boundary: GPT-5.5 distinguished two problems sharing the same quadratic family and Bruno/Brjuno condition because their boundary questions differ.One asks whether a Siegel disk boundary is not a Jordan curve; the other asks whether it is a C^∞ embedded circle.
  • E. Reasoning Behavior Details: Table 10 reports absolute behavior-counter row-hit rates across eight paper models.The assume, cite, and abandon counters use rule-based keyword groups, while lemma-decomposition rows are judged separately by the Agent-Judge.
  • E.1 Behavior-Counter Rates: Behavior-counter rates measure the fraction of benchmark traces triggering each counter across the eight evaluated models.The counters cover assume, cite, and abandon behaviors, with lemma-decomposition rows available for three research-level benchmarks.
  • F License and Release: The RESEARCHMATH family is released under the MIT License.The release contains two artifacts: RESEARCHMATH-14K and RESEARCHMATH-REASONING.
  • F License and Release: RESEARCHMATH-14K contains 14,056 research-level mathematical problems, while RESEARCHMATH-REASONING contains 220K reasoning trajectories.Both artifacts are described in Section 2.
  • F License and Release: Both released artifacts derive from publicly available academic sources, including arXiv preprints, open-problem web pages, and workshop or conference problem sheets.The Extractor agent discards documents hidden behind a paywall before extraction, so paywalled or restricted sources are not represented.

G Training Details

Fine-tuning uses LoRA on three Qwen3 base models with 5,000 sampled traces from either filtered RESEARCHMATH-14K or DASD-Thinking, averaging results across three seeds. The setup specifies LoRA parameters, batch sizes, and model-dependent sequence lengths.

  • G Training Details: LoRA fine-tuning uses three Qwen3 base models and 5,000 randomly sampled traces from either filtered RESEARCHMATH-14K or DASD-Thinking, averaged across three seeds.The models are Qwen3-4B-base, Qwen3-8B-base, and Qwen3-30B-A3B-base.
  • G Training Details: LoRA uses rank r = 64, alpha α = 128, dropout 0.05, no bias, and targets attention and MLP projections in every transformer block.Targeted projections are q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj.
  • G Training Details: Global batch size is 16 for the 30B run and 32 for the smaller models, with per-device batch size 1.
  • G Training Details: Maximum sequence length is 24,512 tokens for 4B/8B runs and 32,768 tokens for the 30B run.

H Prompts

This section documents the prompts used across dataset generation, difficulty comparison, response generation, and factuality evaluation. The factuality workflow also specifies source resolution, provenance, and restrictions on acceptable extraction sources.

  • H.1 Dataset Generation Agents: Tables 11 and 12 provide the Extractor and Refiner prompts used by the dataset-generation agents in Section 2.1.These prompts correspond to the Extractor and Refiner agents, respectively.
  • H.2 Difficulty Comparison: Table 13 gives the prompt used for difficulty comparison in Section 2.3.
  • H.3 Response Generation: Table 14 presents the prompt used to generate model responses in Section 2.4.
  • H.4 Factuality Metrics: Tables 15–17 specify factuality-metric prompts, with Table 17 adapting Table 16 for detected blocks shorter than 200 characters and added surrounding reasoning context.The short-block variant is used when detected blocks fall below the 200-character threshold.
  • H.4 Factuality Metrics: The factuality workflow processes exactly one scholarly source per run, resolves it to readable full text, and returns strict JSON with source, accepted, needs_review, and trace keys.Inputs may be local PDF paths or supported scholarly URLs, and the final JSON is auto-saved each run.
  • H.4 Factuality Metrics: Metadata APIs may identify the exact work, discover candidate full-text URLs, verify DOIs, and check access status, but they cannot serve as the technical extraction source.The workflow instead requires readable source content containing the problem statements themselves.
  • H.4 Factuality Metrics: Source provenance records preserve the original locator, resolved locator, non-empty title, and title origin for every final JSON object.Title selection prioritizes the visible title from the chosen primary source, then trusted resolver metadata, then an AI-extracted title.
  • H.4 Factuality Metrics: When no readable exact-work copy is recoverable, the workflow stops equivalent resolver retries and moves the item to needs_review.Candidate links are followed only when the resulting page or file still matches the same DOI or exact title.
Loading 2605.28003v1…