Streamlining the review process: AI-generated annotations in research manuscripts

1 Introduction

• Rising submissions increase reviewer workload: reviewers spend over 15 million hours annually, and reviewing is often done without sufficient care.Academics average 14 manuscript reviews per year, with each review taking about five hours.Source: §1 · p. 1
• Standalone AI-generated reports may lack high-level analysis, independent judgment, and the questioning needed for rigorous scholarly discourse.Yet AI systems can surpass human reviewers in aspect coverage and informativeness.Source: §1 · pp. 1–2
• LLMs can support reviewers by identifying relevant manuscript excerpts, but the paper positions annotation as a complement to human judgment, not a replacement.The work introduces AnnotateGPT and evaluates it through a TAM questionnaire (n=9), while also aiming to generalize results to other stakeholders.Source: §1 · p. 2
• LLM-authored highlighting is proposed to guide reviewer attention and help prevent valuable excerpts from being overlooked.The authors build on the comprehension benefits of highlighting and LLMs' ability to identify relevant information.Source: §1 · p. 2
• The introduction frames two questions: integrating LLM-authored annotations into review and adapting traditional annotation tools for LLM-assisted review.Source: §1 · p. 2

2 Related work

• AI interventions can be organized by target stakeholder—editor, reviewer, or author—and by whether AI automates or augments human work.Source: §2 · p. 2
• Editor-focused examples predict acceptance or review outcomes from manuscripts and reviews, with authors noting potential bias and ethical concerns.Source: §2 · pp. 2–3
• Reviewer-focused systems include automated scoring and report generation, including ReviewRobot [20], ReviewAdvisor, and standalone ChatGPT reviewing.ReviewRobot reported 71.4%–100% accuracy in NLP and ML tests.Source: §2 · p. 3
• Prior reviewer-focused systems automate scores, comments, or review reports, while this work distinguishes itself by using LLMs to support rather than replace human judgment.The paper contrasts its approach with [20].Source: §2 · p. 3
• The authors advocate annotation as a middle ground: it helps reviewers locate important excerpts while preserving their control over interpretation and value.They frame augmentation as an ethical alternative to unsupervised automation, consistent with a balanced AI-human approach.Source: §2 · p. 3

3 RQ1: How could the task of LLM-authored annotations be integrated into the review process?

• Contextualized feedback makes review criteria explicit and may distinguish positive from negative sentiment to support balanced feedback.Different research communities assess different aspects, and manuscript review is more critical than directive student feedback.Source: §3 · p. 4
• Specific feedback identifies the manuscript excerpts where comments apply, including gaps in arguments or methods.Reviewers commonly annotate while reading and quote the relevant excerpts in their reviews.Source: §3 · p. 4
• LLM annotations are expected to improve feedback timeliness by helping reviewers focus on highlighted manuscripts compared with unannotated ones.Timely feedback is described as reinforcing correct actions or signaling a need for change.Source: §3 · pp. 4–5
• Color-coded excerpts tied to review criteria and automatic excerpt integration are proposed to support contextualized and specific feedback.The section connects these features to the expected focus benefits of reading annotated manuscripts.Source: §3 · p. 5
• A review report can organize criterion-level compilations and annotation-derived comments by review criterion or sentiment.Annotations may include highlighted excerpts, comments, and sentiment; criterion reviews can retain compilations and alternative viewpoints.Source: §3 · pp. 5–6
• Reviews develop bottom-up: reviewers build criterion-based annotations and evaluations, then compile them into a structured report with optional LLM assistance.Reviewers can request relevant excerpts, add comments or sentiments, and use the LLM to compile annotations from either source.Source: §3 · p. 6

4 RQ2: How could traditional annotation tools be tuned for LLM-assisted academic review?

• AnnotateGPT’s contribution is an annotation-based interaction model for LLM–human collaboration rather than a novel prompting approach.It presents a proof-of-concept browser extension that overlays a PDF manuscript and lets reviewers participate in annotation creation, updating, rendering, and assembly.Source: §4 · pp. 6–7

4.1 Annotation

• For each review criterion, GPT-4 can generate three evidence excerpts and assess their sentiment, with prompts built using Reverse Prompt Engineering.Reviewers can request further evidence, fact-check annotations, raise social concerns, ask clarifying questions, or add their own excerpts.Source: §4.1 · pp. 7–9

4.2 Criterion Review

• Criterion reviews begin empty and can accumulate annotations, while GPT can compile an evaluation summary or provide alternative viewpoints.Review criteria can be represented by color-coded highlights, with criterion descriptions and optional actionable recommendations incorporated into prompts.Source: §4.2 · p. 9

4.3 Full Review

• AnnotateGPT compiles criterion-review summaries from annotations into an editable final report organized by criterion or sentiment.The report draws on excerpts and comments assembled in annotations.Source: §4.3 · p. 9

5 Evaluation

• The evaluation used the Technology Acceptance Model to assess reviewers’ perceived ease of use and usefulness in criteria-based manuscript reviewing.The stated evaluation goal was to assess the platform from human reviewers’ point of view.Source: §5 · p. 9
• Nine experienced manuscript reviewers tried AnnotateGPT on one of their own papers, whose familiar content was intended to help them understand its output.Source: §5 · p. 11
• Participants generally valued AnnotateGPT for focus and criterion consistency more than productivity, and reported seamless PDF-viewer integration that may reduce onboarding friction.The reported findings concern perceived usefulness and ease of use.Source: §5 · p. 11

5.1 Threats to validity

• Cronbach’s α was 0.8 for usefulness and 0.83 for ease of use, but usefulness ratings may reflect GPT’s excerpt accuracy, while reviewer expertise and topic may confound results.Source: §5.1 · p. 11
• Review criteria can differ across disciplines, and further evaluation is needed to determine how those differences affect annotation beyond the information-systems context.The paper distinguishes variation in review criteria from variation in annotation practice.Source: §5.1 · p. 12

6.1 Generalization of the Solution Instance

• False negatives may cause reviewers to overlook relevant paragraphs, making AnnotateGPT’s performance dependent on current LLM precision.The paper contrasts this with false positives, which may cause unnecessary attention fatigue.Source: §6.1 · p. 13
• Reviewer feedback on paragraph relevance could let AnnotateGPT refine recommendations and adapt them to individual preferences.Source: §6.1 · p. 13
• Explanations for selected paragraphs are proposed as a way to improve interpretability and help reviewers understand and trust recommendations.Source: §6.1 · p. 13
• Reliance on highlights may cause reviewers to miss unannotated context, so AnnotateGPT allows annotations to be toggled off.Source: §6.1 · p. 13

6.2 Generalization of the Problem Instance

• The discussion extends potential use beyond reviewers to conference organizers, who could customize preset criterion prompts and distribute the platform as a browser plugin.The paper suggests this could support more consistent reviews and conference quality.Source: §6.2 · p. 13
• With confidentiality addressed, conference organizers could use reviews from past editions to fine-tune general-purpose LLMs for annotation-based reviewing.Source: §6.2 · p. 14
• Authors could use LLMs to assess their own drafts against venue criteria, prompting reflection on strengths and weaknesses rather than generating manuscript content.Source: §6.2 · p. 14

• AnnotateGPT is a web-based PDF visualizer combining criterion-driven prompts with language models such as GPT-4 to help reviewers identify important manuscript sections.Source: §7 · p. 14
• The authors call for larger quantitative evaluations with more subjects and qualitative evaluations involving conference endowment.Source: §7 · p. 14
• Future work will assess open-source LLMs for reducing review expenses and explore fine-tuning to improve review effectiveness.Source: §7 · p. 14
• LLM fine-tuning uses labeled examples to update model weights and improve performance on specific tasks.Source: §7 · p. 14