Source-linked AI summary
ELAIPBench: A Benchmark for Expert-Level Artificial Intelligence Paper Understanding
Xinbang Dai, Huikang Hu, Yongrui Chen, Jiaqi Li, Rihui Jin, Yuyang Zhang, Xiaoguang Li, Lifeng Shang, Guilin Qi
TL;DR
ELAIPBench aims to provide reliable multiple-choice evaluation with challenging questions and answerable items supported by evidence. ELAIPBench pairs 403 multiple-choice questions from AI papers with answers and evidence, using incentive-driven annotation and three difficulty levels. 48.14% human-expert accuracy within 20 minutes exceeded the best-performing LLMs’ 39.95% accuracy.
Problem
ELAIPBench aims to provide reliable multiple-choice evaluation with challenging questions and answerable items supported by evidence.
Source: §1 · p. 2Method
ELAIPBench pairs 403 multiple-choice questions from AI papers with answers and evidence, using incentive-driven annotation and three difficulty levels.
Source: §1 · p. 2Results
48.14% human-expert accuracy within 20 minutes exceeded the best-performing LLMs’ 39.95% accuracy.
Source: §1 · p. 2Takeaways & Limitations
The results indicate a substantial gap in current models’ fine-grained, evidence-grounded reasoning over complex scientific texts.
Source: §5 · p. 9The benchmark is limited to AI, text-only content, and English, with expansion constrained by the practical challenges and cost of annotation.
Source: §6 · p. 9
Abstract
from arXiv · showhide
While large language models (LLMs) excel at many domain-specific tasks, their ability to deeply comprehend and reason about full-length academic papers remains underexplored. Existing benchmarks often fall short of capturing such depth, either due to surface-level question design or unreliable evaluation metrics. To address this gap, we introduce ELAIPBench, a benchmark curated by domain experts to evaluate LLMs' comprehension of artificial intelligence (AI) research papers. Developed through an incentive-driven, adversarial annotation process, ELAIPBench features 403 multiple-choice questions from 137 papers. It spans three difficulty levels and emphasizes non-trivial reasoning rather than shallow retrieval. Our experiments show that the best-performing LLM achieves an accuracy of only 39.95%, far below human performance. Moreover, we observe that frontier LLMs equipped with a thinking mode or a retrieval-augmented generation (RAG) system fail to improve final results-even harming accuracy due to overthinking or noisy retrieval. These findings underscore the significant gap between current LLM capabilities and genuine comprehension of academic papers.
1 Introduction
ELAIPBench targets deep comprehension and reasoning over full-length AI papers, addressing benchmarks that rely on retrieval, shallow questions, or unreliable evaluation. It uses evidence-supported multiple-choice questions and an incentive-driven annotation process; reported results show a substantial performance gap between LLMs and human experts.
Motivation
- Existing academic-paper benchmarks often test explicit retrieval or use open-ended evaluation metrics that may not reliably assess fine-grained technical knowledge.Crowdsourced annotation can also produce superficial questions despite some validation mechanisms.Source: §1 · p. 2
Benchmark and contribution
- ELAIPBench contains 403 expert-created questions across three difficulty levels, designed to require deep understanding beyond simple paper retrieval.It covers machine learning, computer vision, and natural language processing, and provides each question with its source paper, answer, and supporting evidence.Source: §1 · p. 2
Approach
- The benchmark uses a competitive, performance-linked annotation mechanism to generate high-quality questions.The process assigns question writers, evidence verifiers, and answer verifiers distinct roles.Source: §1 · p. 2
Evaluation
- 48.14% human-expert accuracy within 20 minutes exceeds the best-performing LLMs’ 39.95% accuracy.The authors report these results as evidence of the benchmark’s difficulty.Source: §1 · p. 2
- LLMs augmented with reasoning or retrieval capabilities generally underperform non-augmented baselines on ELAIPBench.The authors suggest that naive augmentation methods may lead to paper misinterpretations.Source: §1 · p. 2
2 Related Works
Prior scientific-comprehension benchmarks either omit full-paper context or ask questions answerable through direct matching. Existing annotation approaches also face quality-control or depth limitations, motivating ELAIPBench’s game-theoretic approach.
2.1 Scientific Comprehension Benchmarks
- Benchmarks using short or no contextual passages often neglect academic-paper comprehension, while full-paper benchmarks may rely on superficial questions answerable by text matching.The cited examples span domain-specific and full-paper benchmark groups.Source: §2.1 · p. 3
2.2 Annotation Methods
- Crowdsourced annotation depends heavily on reviewer expertise, while LLM-generated datasets can lack the diversity and depth needed for complex academic questions.The authors state that, to their knowledge, they are the first to apply game-theoretic principles to annotation for academic QA benchmarks.Source: §2.2 · p. 3
3 Construction of ELAIPBENCH
ELAIPBench evaluates whether a model can select all correct answers to questions about AI papers. Its construction combines expert annotators, question-generation guidelines, evidence checks, and timed answer verification to produce challenging, answerable questions.
3 Construction of ELAIPBENCH
- The construction process comprises task definition, a game-theoretic annotation mechanism, and benchmark statistics.The section introduces these stages before presenting the statistics.Source: §3 · p. 3
3.1 Task Overview
- The task asks models to identify all correct choices for a multiple-choice question about an AI-domain academic paper; missing or incorrect choices receive zero points.The source paper is a conference or journal publication.Source: §3.1 · p. 3
3.2 Game-Theoretic Annotation Mechanism
- Twenty AI-specialized annotators with prior publication experience are assigned writer, evidence-verifier, or answer-verifier roles.The team includes a professor, a postdoctoral researcher, Ph.D. students, and master’s students.Source: §3.1 · p. 3, §3.2 · p. 3
3.2.1 QA Pair Generation
- Writers select long papers, create single- or multiple-answer questions, and provide verbatim evidence; guidelines require reasoning beyond keyword matching and constrain multiple-answer questions to two or three correct options.Papers under 4,096 tokens are rejected, and questions undergo review by three LLMs; any question answered correctly by one is revised as insufficiently challenging.Source: §3.2.1 · pp. 4–5
3.2.2 Evidence Verification
- Evidence verifiers must answer using only the supplied excerpts, and failed checks trigger revision before the question can proceed.Successful verification rewards both the writer and verifier, while failure removes their bonus allocation.Source: §3.2.2 · p. 5
3.2.3 Answer Verification
- Answer verifiers use only the paper and have 20 minutes; questions are classified as easy, moderate, or hard according to correctness and time.Writer and verifier bonuses vary by outcome, and the authors describe cross-validation as making careful annotation a Nash equilibrium.Source: §3.2.3 · p. 5
3.3 Benchmark Statistics
- Table 1 presents statistics describing ELAIPBench’s data distribution.The text directs readers to Appendix A.4 for more statistics and examples.Source: §3.3 · p. 5, Table 1 · p. 5
4 Experiments
Across the evaluated models, ELAIPBench exposed persistent difficulty with academic-paper understanding. Reasoning and retrieval augmentation often failed to improve performance, with error analyses identifying harmful verification and longer reasoning chains as important patterns.
4.5 Difficulty vs. Reasoning Length
- LRMs generated longer outputs than CoT, but reasoning lengths were similar across question complexity categories, indicating little adaptation to difficulty.Source: §4 · p. 5, §4.5 · p. 7, Figure 5 · p. 8
4.2 Results for LLMs
- 39.95% was the highest accuracy on ELAIPBENCH; CoT degraded all models except Qwen3-8B, and GPT-5-thinking-all and DeepSeek-R1 trailed their base counterparts.The authors call this performance pattern reasoning paralysis.Source: Table 2 · p. 6, §4.2 · p. 6
4.1 Baseline LLMs
- Seven base models and seven LRMs were evaluated, with reported results averaged across three independent API calls.Source: §4.1 · p. 6
4.2 Results for LLMs
- LLMs underperformed human experts on both task formats; GPT-5-thinking-all scored 46.59% versus GPT-5’s 31.82% on SA-MCQ, while accuracy fell on MA-MCQ.The authors hypothesize that selecting multiple correct answers makes MA-MCQ more challenging.Source: §4.2 · p. 6
4.3 Reasoning Error Analysis
- Harmful verification accounted for over half of errors across integrated reasoning settings, often overturning initially correct answers; the analysis examined 30 incorrect answers.The authors identify harmful verification as the primary reason reasoning paradigms underperformed base models, with greater reasoning paralysis tendency on MA-MCQ.Source: Figure 3 · p. 7, §4.3 · p. 7
4.2 Results for LLMs
- No LLM exceeded human experts on easy or moderate questions; a few models’ hard-question advantage was only 0.96%, and underperformance held across AI disciplines.Source: §4.2 · p. 7
4.4 Accuracy vs. Reasoning Length
- Accuracy declined as completion length increased across LRMs, while self-verification made up approximately 50% of reasoning traces and added largely ineffective steps.Source: Figure 4 · p. 7, §4.4 · p. 7
4.6 RAG Performance
- Intra-paper DPR and BM25 retrieval degraded accuracy, while web retrieval gains remained marginal amid noisy results, most of which were other relevant papers.The evaluation retrieved top-five passages from 512-token paper segments or top-five web pages; Figure 6’s “None” condition corresponds to Table 2 Total accuracy.Source: Figure 6 · p. 8, §4.6 · p. 8
5 Conclusion
ELAIPBench evaluates LLM understanding of AI research papers; the best-performing model reaches 39.95%, revealing a substantial gap in fine-grained, evidence-grounded reasoning over complex scientific texts.
5 Conclusion
- 39.95%: the best-performing state-of-the-art LLM achieves only moderate performance on ELAIPBench.The evaluation is of LLM understanding of AI research papers.Source: §5 · p. 9
- ELAIPBench is an expert-level benchmark for evaluating LLM understanding of AI research papers.Source: §5 · p. 9
- The results indicate a substantial gap in current models’ fine-grained, evidence-grounded reasoning over complex scientific texts.Source: §5 · p. 9
6 Limitations
ELAIPBench is limited to English-language AI papers in text form, and expert annotation imposes substantial time and financial costs.
6 Limitations
- The benchmark covers only the AI domain and text modality, constraining its scope.Source: §6 · p. 9
- ELAIPBench is English-only, limiting evaluation of models’ crosslingual capabilities.Source: §6 · p. 9
- Collecting the current 403 questions required six months and over 50,000 CNY (~7,000 USD), reflecting the practical challenges and high cost of expert annotation.Source: §6 · p. 9
- The authors aim to expand coverage to other scientific disciplines and to multimodal evaluation.Source: §6 · p. 9
A.1 Academic Paper Understanding Dataset Comparison
ELAIPBench is distinguished from related resources by combining full-paper context, human annotation, objective MCQ evaluation, and a focus on deep comprehension rather than retrieval or code generation.
A.1 Academic Paper Understanding Dataset Comparison
- ELAIPBench is among the small subset of benchmarks built on complete scientific papers, alongside PaperBench and AcadReason.The authors argue that full-paper understanding requires sustained comprehension of complex methods, nuanced arguments, and long-range dependencies.Source: §A.1 · p. 11
- AcadReason has 50 questions, while PaperBench has 20 instances and primarily targets code generation; the authors describe these sizes or aims as limiting their fit for comprehension evaluation.SciQAG has more than 180,000 automatically generated instances, which the authors contrast with the authenticity and depth needed for expert-level reasoning.Source: §A.1 · p. 11
- Many benchmarks use open QA, which the authors say makes objective and reproducible evaluation difficult.Source: §A.1 · p. 11
- The authors argue that human annotation supports high-quality questions grounded in scientific reasoning, and that their game-theoretic protocol incentivizes careful reading and verification.They contrast this with automatic generation, which they say can fall short in subtlety and cognitive rigor.Source: §A.1 · p. 11
- ELAIPBench combines full-text papers, large-scale human annotation, objective MCQ evaluation, and a focus on deep comprehension rather than citation retrieval or code generation.Source: §A.1 · p. 11
A.2 Annotation Guidelines
The annotation guidelines seek challenging, answerable questions through constrained MCQ design, evidence checks, timed human answering, and incentives tied to question quality and difficulty.
A.2 Annotation Guidelines
- Question design calls for English, objective wording, avoiding commonly known papers, and preferably selecting documents between 8,192 and 128,000 words.Source: §A.2 · pp. 11–12
- Questions must require reasoning, summarization, or integrating information across a paper rather than keyword matching or direct retrieval.Guidelines also require terms and abbreviations to be defined in the source paper and distractors to use misleading phrasing or partial truths without creating ambiguity.Source: §A.2 · p. 12
- Single-answer questions have exactly one correct option, while multiple-answer questions have two or three.Source: §A.2 · p. 12
- Question writers receive 30 CNY per well-written, verified question, with reduced rewards for failed evidence checks, overly easy questions, or guideline violations.Questions that fail evidence verification must be revised and resubmitted.Source: §A.2 · p. 12
- Evidence verifiers read the paper segment, question, and proposed evidence, then attempt to answer using only the paper; an incorrect answer signals that evidence may be insufficient.Questions can be returned to their authors for revision.Source: §A.2 · pp. 12–13
- Answer verifiers’ correctness and response time help estimate question difficulty and determine rewards for both the verifier and question author.These estimates also inform quality control and benchmarking statistics.Source: §A.2 · p. 13
A.3.1 Annotator Statistics
The annotation team had advanced academic credentials and publication experience. Most annotators were aged 22–30, and 80% identified as male.
A.3.1 Annotator Statistics
- All annotators held or were pursuing at least a master’s degree and had authored at least one academic paper.The team included eight master’s students, two people with completed master’s degrees, eight PhD students, one postdoctoral researcher, and one professor.Source: §A.3.1 · p. 14
- 55% of annotators had published more than five papers.Source: §A.3.1 · p. 14
- Most annotators were aged 22–30 (41% aged 22–26; 36% aged 26–30), and 80% identified as male.The remaining age groups were under 22 (14%) and over 30 (9%); 20% identified as female.Source: §A.3.1 · p. 14
A.3.2 Annotation Platform
The appendix describes the annotation workflow and benchmark statistics, then specifies evaluation and retrieval settings and analyzes reasoning errors and efficiency. Longer reasoning chains did not necessarily improve accuracy and could be detrimental.
A.3.2 Annotation Platform
- The platform routes submitted question-evidence pairs through evidence verification and answer verification, with writers revising pairs when evidence is insufficient or incorrect.Answer-verifier rewards use accuracy and response time; the outcome also determines whether the writer receives a performance bonus.Source: §A.3.2 · p. 14, Figure 12 · p. 17, Figure 13 · p. 17
A.4 Benchmark Statistics
- 76% of papers contain at least one Hard question, and the benchmark’s four answer options are relatively balanced, ranging from 23.25% to 26.45%.Questions range from 29 to 492 tokens, and source papers from 5,479 to 31,333 tokens.Source: §A.4 · p. 19
A.5 Evaluation Metric and LLM Settings
- MA-MCQ scoring requires all correct options with no errors, while SA-MCQ scoring requires an exact match.The base-model prompts distinguish one correct option from two or three; CoT adds “Let’s think step by step,” and evaluated models have context windows of at least 32k tokens.Source: §A.5 · p. 19
A.6 Error and Efficiency Analysis
- The analysis illustrates analytical error, harmful verification, and overconfidence: models can commit to a wrong answer early, reject a correct answer, or rely on internal priors instead of paper context.Source: §A.6 · p. 23, Figure 18 · p. 23, Figure 19 · p. 23
- Longer reasoning chains did not necessarily improve accuracy and could be detrimental; QwQ-32B had the longest chains but lowest performance, while DeepSeek-R1 was the most accurate of three LRMs with shorter chains.DeepSeek-R1’s performance remained significantly below Qwen3-32B+CoT.Source: Figure 21 · p. 23, §A.6 · pp. 23–24
A.7.1 Hyper-parameter Setting
- Dense passage retrieval uses BGE-m3 embeddings and cosine similarity to return the top five passages; BM25 uses k1 = 1.5 and b = 0.75.Web retrieval queries combine the question and paper title, return at most ten results, and select five after relevance filtering.Source: §A.7.1 · p. 24
A.7.2 RAG Configuration
- Retrieved passages and web documents are normalized, trimmed to sentence boundaries, ordered by retrieval score, and kept within a combined 32,768-token context budget.If the limit is exceeded, retrieved items are truncated round-robin.Source: §A.7.2 · p. 24
A.7.3 RAG Prompts
- RAG prompts instruct models to answer using both the paper and retrieved knowledge snippets, with separate formats for single-answer and multiple-answer questions.Source: §A.7.3 · p. 24