Source-linked AI summary
AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot
Joydeep Biswas, Sheila Schoepp, Gautham Vasan, Anthony Opipari, Arthur Zhang, Zichao Hu, Sebastian Joseph, Matthew Lease, Junyi Jessy Li, Peter Stone, Kiri L. Wagstaff, Matthew E. Taylor, Odest Chadwicke Jenkins
TL;DR
Rapidly growing submission volumes have made it difficult to maintain peer-review quality, consistency, and timeliness, leaving open whether AI-generated reviews can work at major-conference scale. The paper reports a full-scale AAAI-26 deployment using a multi-stage, multi-tool AI review system and evaluates it through surveys and a scientific-review benchmark. The system was feasible at conference scale, participants preferred its reviews on several criteria, and it outperformed a base LLM at detecting scientific errors, while respondents also identified technical and prioritization limitations.
Problem
Rapid submission growth is straining peer review, and prior evidence had not established whether AI-generated reviews could be technically sound at real-world major-conference scale.
Method
The study deployed one clearly labeled AI review for every AAAI-26 main-track paper entering full review and evaluated the system through a multi-stage workflow, surveys, and a synthetic-perturbation benchmark.
Results
Participants preferred AI reviews to human reviews on several criteria, and the system significantly outperformed a base LLM at detecting scientific errors across five review dimensions.
Takeaways & Limitations
The findings indicate that the system’s technical capabilities were sufficient to usefully assist scientific peer review in ways the community found helpful.
Takeaways & Limitations
Respondents reported equation and table-reading errors, difficulty prioritizing issue significance, and reviews longer than preferred, alongside concerns about misleading decision-makers and effects on reviewing skill.
Abstract
from arXiv · showhide
Scientific peer review faces mounting strain as submission volumes surge, making it increasingly difficult to sustain review quality, consistency, and timeliness. Recent advances in AI have led the community to consider its use in peer review, yet a key unresolved question is whether AI can generate technically sound reviews at real-world conference scale. Here we report the first large-scale field deployment of AI-assisted peer review: every main-track submission at AAAI-26 received one clearly identified AI review from a state-of-the-art system. The system combined frontier models, tool use, and safeguards in a multi-stage process to generate reviews for all 22,977 full-review papers in less than a day. A large-scale survey of AAAI-26 authors and program committee members showed that participants not only found AI reviews useful, but actually preferred them to human reviews on key dimensions such as technical accuracy and research suggestions. We also introduce a novel benchmark and find that our system substantially outperforms a simple LLM-generated review baseline at detecting a variety of scientific weaknesses. Together, these results show that state-of-the-art AI methods can already make meaningful contributions to scientific peer review at conference scale, opening a path toward the next generation of synergistic human-AI teaming for evaluating research.
1 Introduction
Rapid submission growth is straining the quality, consistency, and timeliness of human peer review, while prior studies had not established whether AI-generated reviews could work at major-conference scale. AAAI-26 deployed labeled AI reviews across all 22,977 full-review main-track papers and found them operationally feasible, useful, and preferred on several criteria, while identifying important limitations.
- Motivation: More than 30,000 initial submissions to AAAI-26 for 2026 nearly doubled the approximately 15,000 received for 2025.Submission growth has also occurred at other major venues.
- Motivation: 28,000+ program committee members, senior program committee members, and area chairs were required for AAAI-26, nearly three times the AAAI-25 committee size.The resulting scale increased reviewer workload and compressed timelines.
- Research gap: Before AAAI-26, no major conference had conducted a conferencewide live study of AI-generated reviews on real submissions.Earlier live studies provided limited assistance rather than official AI-generated reviews.
- Deployment: 22,977 full-review main-track papers received one clearly labeled AI review alongside at least two human reviews, without replacing human reviewers.The AI review entered Phase 1, and decision-makers could consider it during promotion and final deliberations.
- Findings: 5,834 survey responses indicated that participants found AI reviews helpful and preferred them to human reviews on technical accuracy, review focus, and research suggestions.Respondents also believed AI reviews would be useful in future peer review processes.
- Limitations: Reported weaknesses included errors reading some equations and tables, difficulty prioritizing issue significance, and reviews longer than readers preferred.The paper identifies tighter output-length controls as a straightforward mitigation for excessive length.
2 The AAAI-26 AI Review System
The AAAI-26 AI Review System uses a multi-stage, multi-tool pipeline designed to assess scientific accuracy across story, presentation, evaluations, correctness, and significance. It combines paper preprocessing, staged analysis, self-critique, revision, quality checks, and auditing to produce structured textual reviews.
- System design: The system targets scientific accuracy across story, presentation, evaluations, correctness, and significance.These criteria reflect the system’s design goal of assessing multiple forms of scientific validity.
- Preprocessing: Each PDF paper is preprocessed into a consistent image resolution and markdown representation, while both PDF and markdown versions are supplied to later stages.The figure specifies 250 DPI image resampling and conversion with olmOCR.
- Core review stages: Five core review stages separately assess the story, presentation, evaluations, correctness, and significance before their findings are compiled into an initial review.Later stages receive the prompts and results from all previous stages.
- Safeguards: A self-critique stage checks for unsupported claims, missing details, and inconsistencies, after which a final stage revises and compiles the review.The workflow also includes quality checking and citation-hallucination checks.
- Oversight: Logs, checkpoints, and review reports are saved for auditing and human oversight, with manual inspections performed during staged deployment.An initial 30% batch was inspected before processing the remaining papers.
- Tool use: The evaluations and correctness stages provide a Python code interpreter so the LLM can test for errors through code execution.This tool use is part of the staged review workflow.
3 Review Survey and Findings
The survey assessed AI and human reviews across participants and review-quality criteria, finding broad usefulness and several AI advantages alongside recurring weaknesses. Written feedback emphasized actionable, thorough analysis but also concerns about judgment, verbosity, and factual errors.
- Survey design: 5,834 responses compared AI and human reviews across nine review-quality criteria using participant questionnaires.The survey covered authors, PC, SPC, and AC respondents and used a five-point Likert scale.
- Quantitative findings: AI reviews were preferred on six of nine criteria, with all nine AI-human differences statistically significant at α = 0.01.The largest advantages concerned technical-error identification, new points, presentation and research-design suggestions, and thoroughness.
- Quantitative findings: 53.9% judged AI reviews useful and 61.5% expected them to help future peer review, while 55.6% saw capabilities beyond their expectations.Respondents also viewed AI reviews as complementary: they could catch concerns humans might miss while overlooking concerns humans might catch.
- Qualitative findings: Written feedback most often praised actionable revision guidance, breadth, thoroughness, and technical-error detection.These themes describe reviews that turn criticisms into revision suggestions and flag mistakes overlooked by human reviewers.
- Qualitative findings: Common weaknesses included poor big-picture judgment, nitpicking, excessive verbosity, factual errors, and equation or table misreadings.Respondents also raised concerns that AI reviews could mislead decision-makers, encourage optimization for AI preferences, or erode reviewing skills.
4 The SPECS Review Benchmark
The SPECS benchmark evaluates free-form AI reviews across five scientific criteria using controlled perturbations to published papers. The full AAAI-26 system significantly outperformed a single-prompt baseline across every criterion, while criterion-targeted stages generally improved detection further.
- Benchmark design: SPECS evaluates whether free-form reviews detect errors in Story, Presentation, Evaluations, Correctness, and Significance.Unlike benchmarks focused on one criterion or structured outputs, SPECS assesses multiple criteria in full review text.
- Benchmark design: The benchmark creates synthetic perturbations in accepted papers and judges whether reviews explicitly identify each injected error with supporting evidence.The evaluation compares the complete system, a single-prompt baseline, and criterion-targeted intermediate stages.
- Results: For every SPECS criterion, the final AAAI-26 system significantly improved recall over the baseline at α = 0.01, with an average gain of +0.21.Table 2 reports criterion-level recall for Final, Baseline, and Targeted conditions.
- Results: The five targeted stages averaged +0.19 over baseline, with every targeted stage except Correctness showing a statistically significant improvement.The Significance stage detected some errors that were not explicitly preserved in the final review.
- Results: Each intermediate stage was most effective at detecting errors in its intended criterion, although cross-criterion detections also occurred.The stage-by-criterion matrix uses rows for perturbation types and columns for core stages; cells are independent detection fractions.
5 Conclusions
The pilot found AI-generated reviews operationally feasible at conference scale and helpful to authors and reviewers, while identifying limitations in technical reading, issue prioritization, and review length. The evidence supports exploring complementary human-AI integration rather than treating AI reviews as interchangeable with human review.
- Conclusions: The pilot demonstrated that AI-generated reviews were operationally feasible at conference scale and helpful to authors and reviewers.The conclusion combines quantitative and qualitative analyses of the AAAI-26 deployment.
- Conclusions: Reported limitations included technical errors in reading equations and tables, difficulty prioritizing issue significance, and excessive review length.The conclusion presents these as areas for improvement rather than as a complete failure of the system.
- Conclusions: The analyses indicate complementary strengths between AI systems and human reviewers and motivate further study of how to integrate them.The supported direction is to leverage both systems’ strengths in evaluating and advancing scientific research.
A.1 Prompt design
The AAAI-26 review pipeline uses layered, specialized prompts to decompose review criteria, synthesize findings, critique the draft, and produce a revised final review. The exact prompts are withheld to reduce risks from prompt-targeted optimization or prompt injection.
- Prompt disclosure: The exact prompt wording is withheld to reduce prompt-targeted optimization and prompt injection risks in future deployments.The paper reports the prompt structure and design without exposing the prompts themselves.
- Prompt architecture: The system uses layered prompts rather than one monolithic instruction to improve coverage across review criteria and minimize errors.A persistent base instruction defines the reviewing objective, factual analysis, and joint use of PDF and OCR-derived markdown.
- Prompt architecture: Specialized components separately address story, presentation, evaluations, correctness, significance, initial review, self-critique, and final review.The components are invoked in a fixed order and play distinct roles in the pipeline.
- Review synthesis: The final stages compile findings, generate a self-critique, and revise the review to address that critique.This produces the final review after the core criterion-focused stages are complete.
A.2 Quantitative Analysis Details
The analysis compares response distributions for AI and human reviews across authors and reviewers, using collection means and statistical tests.
- Responses were collected separately for human and AI reviews from authors and reviewers.The analysis defines separate response collections for each respondent group and review type.
- Mean response differences were computed separately for authors and reviewers.The mean M(R) is defined over each response collection, with |R| denoting its size.
- Overall AI and human review means aggregate responses from both authors and reviewers.The overall means are defined from the corresponding AI and human response collections across both respondent groups.
- Mann-Whitney U tests assessed whether AI and human response distributions differed significantly.The null hypothesis was that the samples came from the same distribution, tested separately for overall, author, and reviewer comparisons.
A.3 Survey Questionnaire
The survey used four questionnaire variants to compare AI and human reviews across authors and reviewers, with standardized ratings and open-ended comments.
- Four questionnaire variants crossed respondent type with whether the questionnaire accompanied an AI or human review.Respondents were authors or reviewers, with reviewers including PC, SPC, and AC roles.
- All closed-form items used the same five-point Likert scale.
- Each questionnaire included an open-ended field for free-form comments.
A.4 Preprocessing PDFs
Paper PDFs were resampled to 250 DPI to control multimodal context usage, while testing showed that PDF inputs alone could misread equations and tables.
- 250 DPI was used as the consistent resolution for all paper PDFs.The resampling was intended to keep multimodal tokenization within the LLM context window when high-resolution images were included.
- PDF inputs alone were insufficient for accurately reading equations and tables.Initial testing linked review errors to misinterpretation of mathematical notation and table structures.
A.5 LLM Details for AAAI-26 AI Review System
The system used gpt-5 with high reasoning effort across review stages, with multimodal PDF processing and code-interpreter support for selected evaluations.
- gpt-5 with high reasoning effort was used for all AAAI-26 AI Review System stages.The model was described as the most capable OpenAI model available at deployment and had a September 30, 2024 knowledge cutoff.
- Table 3 documents the prompt components, their usage locations, and their high-level instructions.
- PDF processing used both extracted text and rendered page images as multimodal context.
- The evaluations and correctness stages used the built-in code interpreter tool.The tool used an auto container type.
A.6 Quality-Checking and Human Oversight of AI Reviews
The pilot used layered automated checks, manual inspection, and human oversight to assess review quality, policy concerns, citation reliability, and benchmark validity. Survey feedback highlighted thorough technical scrutiny alongside limits in contextual prioritization and nuanced judgment.
- Automated and manual quality checks: A critic LLM screened reviews for identity disclosure, offensive content, bias, missing structure, ethical concerns, policy violations, apparent authorship, effort, and overall quality.Flagged reviews were then manually inspected for potential issues.
- Automated and manual quality checks: Manual inspection identified anonymity violations, potential software-license violations, and possible identity leakage in released datasets.Some papers contained acknowledgments or links to public non-anonymous websites.
- Citation checking: Among 1,356 citations sampled from 100 AI reviews, GPTZero marked 1,346 valid, 8 unsure, and 2 fake before manual inspection.The text reports that the uncertain and fake labels were manually inspected.
- Survey feedback analysis: Survey responses were coded into positive and negative themes, then applied across written feedback with excerpt-level attribution for each identified category.The coding process used LLM-assisted taxonomy creation and classification of individual feedback items.
- Survey feedback analysis: Respondents praised technical thoroughness, actionable suggestions, consistency checks, and references, while noting weaker contextual prioritization and nuanced assessment of novelty.The feedback described AI reviews as sometimes exhaustive checklists rather than judgments about which issues most affect overall contribution.
- Benchmark human oversight: Human reviewers judged 22 benchmark perturbations valid scientific errors, while additional review found 39 of 40 sampled LLM judgments unanimously correct.Validity agreement varied by perturbation type: story, correctness, and evaluation achieved unanimous agreement on 5/6, 6/7, and 5/8 cases, respectively.