AAAR-1.0: Assessing AI's Potential to Assist Research

1. Introduction

• Researchers face intensive work in reading papers, generating ideas, conducting rigorous experiments, and handling growing peer-review demands.Source: §1 · p. 1
• Prior LLM research-assistance studies often address subjective, expertise-heavy problems that are laborious and difficult to evaluate reproducibly.Source: §1 · p. 1
• AAAR-1.0 assesses four expert-level tasks drawn from researchers’ daily activities, with input-output illustrations of their distinct challenges.The tasks are equation correctness inference, experiment design, paper-weakness identification, and review critique.Source: §1 · pp. 1–2, Figure 1 · p. 2
• EQINFER scores remain near chance: the random-guess baseline is 40% F1, and top models reach around 46%.The authors note this difficulty despite the task relying primarily on local-context reasoning.Source: §1 · p. 2
• LLM-designed EXPDESIGN experiments are more innovative and diverse than human-designed ones, but many are trivial, infeasible, or stray from research objectives.Source: §1 · p. 2
• LLM-identified paper weaknesses often lack depth and specificity, while models also struggle to identify deficient human reviews.Source: §1 · p. 2

2. Related Work

• Prior work studies LLM-generated research ideas and end-to-end research agents, while systematic evaluation of individual research-step outputs remains lacking.Si et al. (2024) studied idea generation; Lu et al. (2024) proposed an agent for a multi-step research workflow.Source: §2 · p. 3
• Existing AI-research benchmarks mainly assess implementation and execution, such as MLAgentBench’s evaluation of generated code and trained-model test performance.The cited benchmark is Huang et al. (2024).Source: §2 · p. 3
• AAAR-1.0 focuses on expertise-intensive research activities and develops curated, task-specific metrics for more efficient and accurate evaluation.Source: §2 · p. 3

3. AAAR-1.0

• The benchmark’s data-construction overview covers EQINFER, EXPDESIGN, and PAPERWEAKNESS, with workflows illustrated for three tasks.Source: §3 · p. 3, Figure 2 · p. 4

3.1. EQUATIONINFERENCE

• EQINFER frames equation validation as binary inference, pairing each human-written equation with three GPT-4-generated negatives and filtering context-unaligned or actually correct negatives.Expert checks retain 1,049 positive equation sets after filtering; each retained positive has three negative counterparts.Source: §3.1 · pp. 3–4

3.2. EXPERIMENTDESIGN

• EXPDESIGN uses pre-experiment paper context and figures as input, with expert-annotated experiment plans and corresponding explanations as output.Experts identify and summarize key experiments, then review and discuss annotations until their opinions align; the final dataset contains 100 instances.Source: §3.2 · pp. 4–5

3.3. PAPERWEAKNESS

• For PAPERWEAKNESS, PDF parsing supplies paper text and extracted figure and table images because most under-review submissions lack available source files.A manual check of 200 cases found GPT-4 missed no more than 1% of reviewer-written weaknesses.Source: §3.3 · p. 5

3.4. REVIEWCRITIQUE

• REVIEWCRITIQUE reuses expert-labeled data from 100 papers, comprising 380 reviews divided into 11,376 sentence-level segments labeled deficient or not.The dataset includes detailed human explanations for the labels.Source: §3.4 · p. 6

4. Evaluation Criteria

• EQINFER uses F1, while free-form EXPDESIGN and WEAKNESS outputs are evaluated with task-specific metrics alongside ROUGE.Source: §4 · p. 6
• EXPDESIGN entailment precision and recall compare predicted items with the full ground-truth list in both directions, using GPT-4o as evaluator.For explanation generation, one-to-one predicted and ground-truth experiments are compared semantically with SentenceBERT (Reimers, 2019).Source: §4 · p. 6
• WEAKNESS evaluation preserves separate reviewers’ lists rather than flattening perspectives, and uses a semantic-based metric to evaluate predicted weaknesses.The ITF-IDF review-diversity metric combines weakness informativeness and specificity, reflecting overall weakness diversity.Source: §4 · p. 6
• REVIEWCRITIQUE uses F1 for deficiency classification and ROUGE plus BERTScore (Zhang et al., 2020) to assess explanation alignment with expert annotations.Source: §4 · pp. 6–7

5.1. EQUATIONINFERENCE

• 47.98% was the best EQINFER result, while nearly all open-source models failed to beat the 40% All-Positive baseline.All models showed high recall but low precision, which the authors identify as a risk when using LLMs to check equation validity in reviews.Source: §5.1 · p. 7

5.2. EXPERIMENTDESIGN

• EXPDESIGN closed-source models generally outperformed open-source models, but all scored only 20%–30%, with low recall for ground-truth experiments and low precision for novel ones.For explanations, closed-source models scored higher on S-Match, while open-source models often copied or paraphrased experiment wording, inflating ROUGE overlap.Source: §5.2 · p. 7
• Whole-list prompting generally improved closed-source models’ experiment-explanation performance by preserving relationships among experiments.The authors’ manual checks found that models could refer to other experiments and better grasp the motivation for a given experiment.Source: §5.2 · p. 8
• More paper context improved experiment-design performance up to 8k tokens, after which the benefit stopped; figure input did not improve results and could harm them.The authors suggest figures may add tokens while providing little information beyond the text.Source: §5.2 · p. 9

5.3. PAPERWEAKNESS

• Closed-source models generally outperformed open-source models on PAPERWEAKNESS, yet their weaknesses remained less diverse, vaguer, and less informed by frontier research than human reviews.AI-SCI performed worse than its GPT-4o backbone, especially on ITF-IDF.Source: §5.3 · p. 9
• For PAPERWEAKNESS, split-combine generally outperformed full-paper input, and long-context models could perform worse than when given only the first 3,000 words.The authors observed that full-paper inputs led models to neglect important sections, whereas splitting encouraged piece-by-piece weakness brainstorming.Source: §5.3 · pp. 9–10

5.4. REVIEWCRITIQUE

• In REVIEWCRITIQUE, closed-source models generally had higher F1 than open-source models, but F1 remained low and recall exceeded precision across models and prompting strategies.This pattern indicates that models often incorrectly flag review segments as deficient.Source: §5.4 · p. 10
• LLM explanations of deficient review segments scored relatively low overall, although Claude Opus had the highest scores across metrics and open-source models generally scored lower.The scores indicate that models could identify some deficient segments but struggled to articulate their reasoning in line with human explanations.Source: §5.4 · pp. 10–11

• AAAR-1.0 targets comprehensive evaluation of current LLMs’ capacity to assist AI research.Source: §6 · p. 11
• The benchmark combines distinct expertise-intensive tasks with curated evaluation metrics.Source: §6 · p. 11
• Senior AI researchers and strict data examinations were used to collect high-quality data.Source: §6 · p. 11

• LLMs trained on papers from open-source sources such as arXiv and OpenReview may have seen benchmark source data, potentially affecting comparison fairness and conclusions.Source: Limitations · p. 11
• High annotation costs from senior researchers leave some tasks, including EXPDESIGN, relatively small, which may introduce variability across repeated runs.Source: Limitations · p. 11
• The authors plan larger-scale AAAR-2.0 datasets covering more diverse research tasks.Source: Limitations · p. 11

• The study examines whether LLMs can assist human researchers in AI research, not replace them.Source: Impact Statement · p. 11
• The authors state that primary responsibility for scientific research should remain with humans to prevent societal risks.Source: Impact Statement · p. 11
• The work analyzes LLM strengths and weaknesses to support judicious use of these tools and mitigate risks while maximizing benefits.Source: Impact Statement · p. 11

• The supplementary material elaborates on these topics.Source: Appendices · p. 15

Appendix A

• Appendix A presents data statistics and diversity.Source: Appendices · p. 15

Appendix B

• Appendix B provides implementation details.Source: Appendices · p. 15

Appendix C

• Appendix C presents more experiment results and details.Source: Appendices · p. 15

Appendix D

• Appendix D provides data cases and an illustration of the annotation platform.Source: Appendices · p. 15

Appendix E

• Appendix E contains prompt templates.Source: Appendices · p. 15

A. Data Statistics and Diversity

• The appendix presents detailed statistics for all three benchmark datasets, tokenizing words with NLTK and simplifying equations with pylatexenc before length calculations.Source: A. Data Statistics and Diversity · p. 15
• WEAKNESS papers span 13 tracks, and most have review scores from 5 to 8, corresponding to weak rejection or acceptance.The score and track distributions are illustrated in Figure 3.Source: A. Data Statistics and Diversity · p. 15
• Table 9 reports EQINFER statistics and distinguishes paper context before and after a missed equation from positive and negative equation examples.Positive equations come from source-paper authors; negative equations are GPT4-synthetic wrong equations.Source: Table 9 · p. 15

B.1. Metric Details

• Similarity-based scores encode each segment with SentenceBERT and calculate cosine similarity.The stated single-A100 memory use is about 1GB.Source: §B.1 · p. 15

B.2. LLMs Running Details

• The experiments include open- and closed-source LLMs, with repeated runs used to address performance instability.Each model is run three times and the median result is selected; inference uses VLLM, PyTorch 2.4.0, CUDA 12.1, and eight A100 GPUs.Source: §B.2 · p. 16

C.1. Input Context Scaling Investigation

• Context-scaling results for EQINFER, EXPDESIGN, and WEAKNESS are shown in Figure 4, Figure 5, and Table 12, respectively; Table 12 compares WEAKNESS input-processing methods using GPT-4o and GPT-4-Turbo.Both models accept a maximum of 128k input tokens, and AI-SCI results are included for reference.Source: §C.1 · p. 17, Table 12 · p. 17

C.2. Human Evaluation on LLM-Generated Novel Experiments

• Three senior PhD students evaluate each LLM-generated novel experiment, with a third annotator deciding when the first two disagree.Figure 6 gives the evaluation guideline, and Table 13 shows example evaluations.Source: §C.2 · p. 18

C.3. Human Evaluation on LLM-Generated Explanation

• Five annotators judge whether anonymized model-generated explanations are acceptable for their corresponding experiments.Each annotator receives four or five papers and evaluates outputs from five models; results are reported in Table 5.Source: §C.3 · p. 18

C.4. Multi-Modal Input Ablation

• The paper reports multimodal ablation studies for EXPDESIGN and WEAKNESS, presenting the studies in Table 14 and Table 15, respectively.Source: §C.4 · p. 18

D. Data cases and Annotation Platform Illustration

• Sample cases cover EQINFER, EXPDESIGN, and WEAKNESS, and the annotation platform transfers PDF comments into a Google Doc for verification and discussion.Source: D. Data cases and Annotation Platform… · p. 18, Figure 7 · p. 21, Figure 8 · p. 21

E. Prompt Templates

• Prompt templates for data collection and model prediction are shown for EQINFER, EXPDESIGN, and WEAKNESS.The appendix also includes human-evaluation examples and multimodal ablation tables for EXPDESIGN and WEAKNESS.Source: E. Prompt Templates · p. 18, Table 13 · p. 22, Table 14 · p. 23

D. Data cases and Annotation Platform Illustration

• The appendix includes figures for WEAKNESS data diversity, context-length scaling, and novel-experiment evaluation guidance.The captions identify score and track distributions, input-context trends, and an evaluation guideline, respectively.Source: Figure 3 · p. 19, Figure 4 · p. 20, Figure 5 · p. 20