PaperlayerView exact revision on arXiv

Source-linked AI summary

ACADREASON: Exploring the Limits of Reasoning Models with Academic Research Problems

Xin Gui, King Zhu, JinCheng Ren, Qianben Chen, Zekun Moore Wang, Yizhi LI, Xinpeng Liu, Xiaowan Li, Wenli Ren, Linyu Miao, Tianrui Qin, Ziqi Shu, He Zhu, Xiangru Tang, Dingfeng Shi, Jiaheng Liu, Yuchen Eleanor Jiang, Minghao Liu, Ge Zhang, Wangchunshu Zhou

arXiv:2510.11652v1cs.CL

TL;DR

Existing evaluations do not provide a rigorous combination of broad academic coverage and deep reasoning challenges. ACADREASON evaluates LLMs and agents on research questions drawn from academic papers, using expert annotation and quality controls. General models underperform reasoning models and agents, and GPT-5 scores 16.0 points.

  • Problem

    Existing evaluations often focus on math, code, or general tasks, while multi-domain academic benchmarks lack sufficient reasoning depth.

    Source: §1 · p. 1

  • Method

    ACADREASON builds a 50-question benchmark from academic literature, with research questions, golden answers, hints, and checklists for evaluating LLMs and agents.

    Source: §1 · p. 2

  • Results

    General models underperform reasoning models and agents; GPT-5 achieves 16.0 points.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The results indicate that current models still have considerable room for improvement on academic research problems.

    Source: §6 · pp. 8–9

Abstract

from arXiv · show

In recent years, the research focus of large language models (LLMs) and agents has shifted increasingly from demonstrating novel capabilities to complex reasoning and tackling challenging tasks. However, existing evaluations focus mainly on math/code contests or general tasks, while existing multi-domain academic benchmarks lack sufficient reasoning depth, leaving the field without a rigorous benchmark for high-level reasoning. To fill this gap, we introduce the Acadreason benchmark, designed to evaluate the ability of LLMs and agents to acquire and reason over academic knowledge. It consists of 50 expert-annotated academic problems across five high-reasoning domains, including computer science, economics, law, mathematics, and philosophy. All questions are sourced from top-tier publications in recent years and undergo rigorous annotation and quality control to ensure they are both challenging and answerable. We conduct systematic evaluations of over 10 mainstream LLMs and agents. The results show that most LLMs scored below 20 points, with even the cutting-edge GPT-5 achieving only 16 points. While agents achieved higher scores, none exceeded 40 points. This demonstrates the current capability gap between LLMs and agents in super-intelligent academic research tasks and highlights the challenges of Acadreason.

1 Introduction

Existing benchmarks do not consistently combine broad academic coverage with the professional depth needed to evaluate advanced reasoning. ACADREASON uses research questions from academic literature to assess LLMs and agents, with results showing substantial room for improvement.

1 Introduction

  • Existing benchmarks often lack either coverage of fields such as science and humanities or the professional rigor and reasoning depth of cutting-edge academic tasks.The introduction describes prior benchmarks as often targeting simpler arithmetic, algebra, grade-school knowledge, or commonsense tasks.Source: §1 · p. 1
  • ACADREASON evaluates academic-level reasoning by extracting research questions and golden answers from authoritative, timely academic literature.The construction process develops hints and checklists alongside the questions; the benchmark contains 50 questions.Source: §1 · pp. 1–2
  • GPT-5 achieves 16 points in pass rate and 40.6 points in checklist score, while reasoning models and agents generally perform more strongly than general models.The reported results also find that methodology hints yield the largest gains among the hint types.Source: §1 · p. 2
  • The benchmark combines detailed evaluation metrics and knowledge hints to assess LLMs’ and agents’ capabilities and inform future improvement directions.Source: §1 · p. 2

2 Related Work

Prior research benchmarks assess research-related reasoning, agentic abilities, or information synthesis, but the related work identifies gaps in breadth and depth. ACADREASON is presented as combining both dimensions in academic research questions.

2 Related Work

  • Large reasoning models have strong reasoning capabilities but limited internal knowledge, while agent frameworks add tools that support acquiring external knowledge.The related work describes OAgents’ GAIA and BrowseComp results and MiroFlow’s leaderboard performance.Source: §2 · p. 3
  • Existing benchmarks include paper reproduction, research-oriented tasks, real-world tool use, and web search, but some focus narrowly on math and coding while others test only superficial information integration.Source: §2 · p. 3

3 ACADREASON Benchmark

ACADREASON evaluates research reasoning through tasks drawn from theoretical academic papers, with annotations, validation, and scoring designed to make questions challenging and assessable.

3 ACADREASON Benchmark

  • ACADREASON uses human annotation, hints, checklists, and corresponding metrics to build a benchmark focused on cutting-edge reasoning capabilities.Source: §3 · p. 3

3.1 Task Specification

  • Candidates solve research questions from theoretical articles without the original text, using internal knowledge or search tools; questions include a specific problem and minimal background.Hints cover background, definitions, and methodology; dynamic checklists capture reasoning milestones, and golden answers provide complete solution trajectories.Source: §3.1 · pp. 3–4

3.2 Data annotation

  • Annotators select challenging theoretical papers, formulate comprehensive questions and reasoning-rich golden answers, then derive hints and verifiable, independent checklist items.Hints draw on background, definitions, and methodology from the source paper.Source: §3.2 · p. 4

3.3 Validation Process

  • The benchmark retains 50 theoretical papers published in 2023–2025 across five fields, and questions pass review by three domain experts for clear boundaries, complete information, and domain logic.Source: §3.3 · p. 4

3.4 Evaluation method and metrics

  • GPT-5-mini judges candidate answers against golden answers and checklist items; pass rate measures full golden-answer matches, while checklist score measures satisfied checklist criteria.Source: §3.4 · p. 5

4 Experiments

The experiments compare mainstream language models and agent frameworks on ACADREASON, then examine performance across disciplines and the effects of hints. OAgents leads the reported model and framework results, while hint benefits vary by type and discipline.

4.1 Experiments Settings

  • The experiments assess general and reasoning models, agent frameworks, hinted models, and representative failure cases.Models answer benchmark questions directly, while agents retain their basic tool configurations; the hint ablation tests GPT-5, GPT-4.1, o3, and others.Source: §4.1 · pp. 5–6

4.2 Main Experiment Result

  • Computer Science and Economics have lower checklist-score distributions than Law and Philosophy, which the authors identify as more challenging disciplines.The reported comparison uses checklist scores across academic disciplines.Source: §4.2 · p. 6, Figure 2 · p. 6
  • GPT-5 scores 16 and 40.5 on pass rate and checklist, respectively, versus GPT-4.1’s 0 and 21.0.DeepSeek-V3.1 also scores 2.0/24.8 versus DeepSeek-V3’s 2.0/15.9.Source: §4.2 · p. 7
  • OAgents scores 34.0 pass rate and 65.1 checklist, while GPT-5 scores 16 pass rate and 40.6 checklist.OAgents achieves the best overall results among evaluated models and frameworks.Source: §4.2 · pp. 6–7, Table 1 · p. 6

4.3 Ablation Study

  • GPT-5 rises from 16.0/40.5 without hints to 40.0/67.8 with all hints, surpassing OAgents.Table 2 reports pass rate/checklist score pairs across hint settings.Source: §4.3 · p. 7, Table 2 · p. 7
  • Methodology hints yield the largest gains for most models, while background hints provide the smallest relative gain.Figure 3a presents performance gain per model.Source: §4.3 · p. 7, Figure 3 · p. 8
  • STEM subjects improve less than humanities subjects; Law and Philosophy benefit more from methodology and background hints, while Economics emphasizes definitions.Figure 3b reports average improvement across disciplines.Source: §4.3 · p. 8, Figure 3 · p. 8

5 Case study

The legal case study compares OAgents and GPT-5 against a four-action checklist. OAgents addresses all four actions, while GPT-5 addresses two and misses deeper synthesis.

5 Case study

  • OAgents satisfies all four legal-analysis checklist actions, while GPT-5 satisfies only two.The checklist covers identifying the legal fallacy, refuting a false safety proposition, analyzing root cause, and identifying judicial impact.Source: §5 · p. 8
  • GPT-5’s misses reflect shallow association and incomplete synthesis of political and economic context, whereas OAgents identifies the coordinated lobbying strategy.GPT-5 associates “counterfeit” with consumer harm and identifies procedural leverage, but does not provide the required deeper analysis.Source: §5 · p. 8

6 Conclusion

ACADREASON evaluates research capabilities across 50 items in five domains; GPT-5 scores 16 points, while an advanced agent scores 34. The authors release annotated data and preliminary benchmarks to support evaluation and enhancement of LLM reasoning.

6 Conclusion

  • ACADREASON contains 50 evaluation items across five domains, and GPT-5 scores 16 points versus 34 for an advanced agent.The authors present these results as evidence of the benchmark’s difficulty and current models’ room for improvement.Source: §6 · p. 8
  • The authors release the annotated data and preliminary benchmarking results to help the research community evaluate and enhance LLM reasoning.They frame the work as diversifying reasoning benchmarks and using academic research artifacts in LLM research.Source: §6 · p. 9

A Data Statistics

The final benchmark contains 50 academic questions, with five samples in each category; Table 3 lists 20 representative papers.

A Data Statistics

B Prompt for Infer and Evaluation

The appendix documents separate inference and evaluation prompts, instructions for question-and-answer annotation, and criteria for screening and validating benchmark items.

B Prompt for Infer and Evaluation

C LLM Usage

  • LLMs were used only to polish writing, not for experimental design, methods, or scientific interpretation.Source: C LLM Usage · p. 16

RESEARCH QUESTIONS ANNOTATION GUIDELINE

HINTS AND CHECKLIST ANNOTATION GUIDELINE

VALIDATION GUIDELINE

  • Validation reviewers refine items containing a research question, hints, checklist, and golden answer.Source: VALIDATION GUIDELINE · p. 17
  • Screening checks source authority and timeliness, theoretical content, reasoning difficulty, and whether each question is well-defined and answerable.The validation guideline also includes a quality-validation figure.Source: VALIDATION GUIDELINE · p. 17, Figure 10 · p. 17

E More Experiment Result

The section provides model-performance tables under three hint conditions and a domain-level pass-rate figure.

E More Experiment Result

  • Table 4 reports pass rate and checklist score for models evaluated with background hints.Source: Table 4 · p. 18
  • Table 5 reports pass rate and checklist score for models evaluated with definition hints.Source: Table 5 · p. 18
  • Table 6 reports pass rate and checklist score for models evaluated with methodology hints.Source: Table 6 · p. 19
  • Figure 11 presents general model performance across domains using pass rate.Source: Figure 11 · p. 19

F Specific Case of Acadreason benchmark

The examples include questions about moral understanding, sorting permutations, legal statelessness, retrieval structures, and investor participation.

PHILOSOPHY

  • The philosophy sample asks why shared moral understanding is important.Source: PHILOSOPHY · p. 20, Figure 12 · p. 20
  • The math sample asks how to characterize and enumerate permutations sortable by a pop stack with a bypass.It also asks about forbidden patterns, Motzkin-path bijections, preimages, and extensions to parallel pop stacks.Source: PHILOSOPHY · p. 21, Figure 13 · p. 21
  • The law sample presents the “Schrödinger’s citizen” dilemma of whether someone can hold citizenship while being stateless.Source: PHILOSOPHY · p. 22, Figure 14 · p. 22

COMPUTER SCIENCE

ECONOMICS

  • The economics sample asks about threshold participation equilibrium and the limiting Poisson distributions of participants conditional on project type.Source: ECONOMICS · p. 24, Figure 16 · p. 24

Related papers

Loading 2510.11652v1…