PaperMentor: A Human-Centered Multi-Agent Writing Tutor for AI Research Papers on Overleaf

1 Introduction

• Junior AI researchers often learn scientific writing through trial and error, although venues assess clarity, narrative, organization, and conventions alongside technical merit.Weak presentation can obscure strong ideas and affect a manuscript's final acceptance (Widom, 2006; Peyton Jones, 2014; Jin, 2024).Source: §1 · p. 1
• Grammar assistants focus on sentence edits and AI reviewing tools judge paper quality, leaving drafting-stage, text-anchored guidance on narrative, organization, and technical presentation unprovided.The cited examples are Grammarly and Writefull for grammar assistance, and Liang et al. (2024), Liu and Shah (2023), and Zhou et al. (2024) for AI-powered reviewing.Source: §1 · p. 1
• PaperMentor combines over 40 expert skill files with 12 specialized agents and delivers suggestions as Overleaf-native inline comments while authors retain revision control.The agents review aspects including methods, results, style, formatting, and terminology; feedback is consolidated and mapped back to LaTeX source files.Source: §1 · p. 1, Figure 1 · p. 2
• PaperMentor improved validity by 6.5 percentage points and actionability by 4.1 percentage points over direct prompting with the same LLM without the skill library.The study involved 14 AI researchers annotating comments on 80 papers from ICLR 2026 submissions and internal student drafts.Source: §1 · p. 1

LLM-Based Automated Paper Review

• Unlike review-level LLM systems that judge soundness, novelty, or acceptance, PaperMentor gives authors concrete, text-anchored writing and structure comments during drafting.Prior LLM review work has emphasized surface-level summaries and shown limited correlation with human scoring (Liang et al., 2024; Liu and Shah, 2023; Thakkar et al., 2025; Zhuang et al., 2025; Gao et al., 2025).Source: §2 · p. 2

Human-AI Collaborative Writing

• Unlike grammar-focused tools such as Writefull (Writefull, 2026) and Grammarly (Grammarly, 2026), PaperMentor uses Overleaf comments for structural and organizational guidance rather than rewriting.Following feedback-based assistance that preserves authorial agency (Lee et al., 2024; Dhillon et al., 2024; Han et al., 2024), specialized agents use an expert skill library for domain-specific, venue-aware guidance while authors make revisions.Source: §2 · p. 2

3 Task Definition

• Each review comment specifies the source file, the character span to highlight, the feedback text, and a severity label.The source file identifies the project file, while the span marks the beginning and end of the highlighted passage.Source: §3 · pp. 2–3
• PaperMentor’s input for this task is a LaTeX project, and its output is a collection of review comments.Source: §3 · p. 2
• The severity label indicates the issue’s importance and uses categories including CRITICAL and WARNING, plus a third label shown as “SUGGES-” in the passage.Source: §3 · p. 3

4.1 The Skill Library

• The skill library combines faculty feedback, public writing guidance, 32 published examples, and 350 conference reviews, standardized with Claude Opus 4.5 and human-refined.It contains over 40 skill files and more than 16,000 words; experts organize skills into six categories: setup, venues, paper types, sections, figures and tables, and writing style.Source: §4.1 · p. 3

4.2 Input Processing

• The system consolidates nested LaTeX files, extracts the abstract and section headers, identifies paper type, and assigns draft content to review domains.An LLM selects among analysis, dataset, method, engineering, interdisciplinary, and position paper types, leaving unsupported types unspecified; it maps lowest-level section headers to one or more domains and also defines global domains.Source: §4.2 · p. 3

4.3 Multi-Agent Review

• Twelve concurrent agents cover seven section domains, three global domains, and two agents instantiated for the paper type and target venue.Agents receive relevant LaTeX source and applicable skill files, paper-type and venue guidance, and any role-model paper; long inputs can be split into lower-level subtasks.Source: §4.3 · p. 4

4.4 Comment Aggregation

• The system removes near-duplicate comments with substantially overlapping spans and lexically similar text, keeping the higher-severity comment and preferring section-specific agents when severity ties.It uses agent-produced character spans to map comments to their source files and render them in Overleaf.Source: §4.4 · p. 4

• PaperMentor uses Overleaf Community Edition and its ShareJS protocol to add AI comments to the native review panel like human reviewer comments.This preserves the familiar Overleaf environment and requires no change to researchers’ existing workflow.Source: §5 · p. 4

Frontend

• The frontend adds an Overleaf sidebar panel for selecting a model, specifying an optional venue, uploading optional role-model papers, and starting a full review.The panel is implemented as a React component in TypeScript.Source: §5 · p. 4, Figure 2 · p. 5
• After review, users see a collapsible summary with detected paper type and file-by-file comments, while highlighted spans are anchored to corresponding LaTeX source locations.Comments also appear in Overleaf’s native review panel.Source: §5 · p. 5, Figure 2 · p. 5

Backend

• The backend retrieves project documents, merges them into a TEX file, runs the three-phase review pipeline, and returns results organized by source file.It is implemented as Express.js route handlers and a review orchestration engine within the existing Overleaf web service; the frontend sends the project ID and selected model to /ai-tutor-review.Source: §5 · p. 5

6.1 Experimental Setup

• The study compares GPT-5.2 with and without the skill library, using 80 papers sampled from student submissions and ICLR 2026 submissions.The baseline uses the same LLM and other prompt components as PaperMentor; the papers include compilable LaTeX sources and span a broad range of quality.Source: §6 · p. 5, §6.1 · p. 5

6.2 User Study Design

• Fourteen AI researchers evaluated comments on four papers each, rating validity, actionability, and conciseness with binary judgments.Each paper had 30 PaperMentor and 30 baseline comments, presented without layout distinctions to avoid bias; annotators received evaluation guidelines.Source: §6.2 · pp. 5–6, §6.1 · p. 5

6.3 Results

• PaperMentor significantly outperforms direct prompting in validity and actionability, while baseline comments score higher in conciseness.The authors attribute this trade-off to longer comments produced when the skill library’s structured writing guidelines are used.Source: §6.3 · p. 6, Table 2 · p. 6
• Approximately 40% of comments address Methods and Results, while PaperMentor gives relatively more attention to the Abstract and Methods than to appendices.Annotation scores remain consistent across major sections, suggesting comment quality generalizes across different parts of papers.Source: §6.3 · p. 6
• Annotators generally viewed the feedback positively, citing professor-like tone, understandability, usefulness, and balanced critique.They particularly noted effectiveness for clarity, depth of analysis, and grammar, alongside moderate improvements in thesis clarity and supporting evidence.Source: §6.3 · p. 6

7 Skill Library Extensibility

• Researchers can update the living skill library through text edits, adapting venue expectations, paper types, and disciplinary norms independently as standards change.This makes the system easier to adapt than a fixed prompt or monolithic reviewer.Source: §7 · p. 7
• The authors envision senior AI researchers contributing advice across subfields to build shared infrastructure for collecting and operationalizing paper-writing knowledge.Contributions could enhance existing skills or become additional modules, potentially transforming PaperMentor from a single assistant into shared infrastructure.Source: §7 · p. 7

8 Conclusion

• PaperMentor delivers expert-guided, actionable feedback directly within the Overleaf drafting workflow.Its specialized review agents draw on a curated skill library distilled from senior researchers’ guidance.Source: §8 · p. 7
• The skill-library-grounded agents significantly improve comment validity and actionability over a direct prompting baseline.Source: §8 · p. 7
• Research-paper AI support should move beyond generic rewriting toward mentor-like feedback that helps authors revise while preserving their authorship and judgment.Source: §8 · p. 7

Limitations

• Because PaperMentor primarily analyzes LaTeX source, it may miss rendered-PDF, visual-figure, and numerical-verification issues, while its 80-paper and 14-annotator evaluation does not capture broad research diversity.Its feedback also depends on skill-library coverage and LLM reliability, so it should be treated as drafting assistance rather than authoritative review.Source: Limitations and Future Work · p. 7

Future Work

• The ablation against the same LLM without expert writing skills does not directly compare PaperMentor with experienced researchers’ feedback; benchmarking expert-authored Overleaf comments could provide a stronger reference.Source: Limitations and Future Work · p. 7
• Section-specific agents can flag terms, definitions, or experimental details as missing when they appear elsewhere; full-paper context could help but would raise computational cost and API usage.The paper proposes lightweight document-wide grounding through shared summaries, global definitions, or structured representations accessible to all agents.Source: Limitations and Future Work · p. 7
• The authors are improving PaperMentor and invite community contributions to extend its skill library and refine the system as AI research writing practices and standards evolve.Source: Limitations and Future Work · p. 7

Ethical Considerations

• External evaluation papers were publicly available arXiv preprints used under their licenses for non-commercial research; the user study followed ETH Zurich guidelines and was exempt from formal ethics review.Annotation data were stored securely and used exclusively for the paper’s evaluation.Source: Ethical Considerations · p. 7
• PaperMentor is intended to support junior researchers without experienced mentors and reduce inequalities in scientific writing guidance, but over-reliance could homogenize styles or suppress diverse rhetorical voices.The system generates suggestions rather than rewrites, deliberately preserving authorial agency.Source: Ethical Considerations · p. 8
• Its skill library reflects predominantly English-language, Western AI-venue norms that may not generalize equitably across cultural or disciplinary backgrounds; community contributions are encouraged to mitigate bias.Source: Ethical Considerations · p. 8

A Agent Configuration and Skill Assignment

• PaperMentor assigns more comments to core sections such as the Abstract and Methods than to less critical sections such as the Appendix.Source: A Agent Configuration and Skill… · p. 10
• Each run instantiates twelve review agents: ten with fixed scope and two configured dynamically.Section agents review assigned sections with the abstract and introduction for context; global agents review merged source, and the figures agent reviews extracted figure and table environments.Source: A Agent Configuration and Skill… · p. 10
• The paper-type agent uses conventions for the type identified in Phase 1, while the venue agent uses requirements for the user-selected venue.Source: A Agent Configuration and Skill… · p. 10

B Distribution of Comments and Annotation Scores Across Review Domains

• Figure 4 shows generated-comment distributions across review domains alongside mean human annotation scores on three evaluation metrics, with standard-deviation error bars.The supplied passage does not provide domain-specific values.Source: Figure 4 · p. 10, B Distribution of Comments and… · p. 10

C Section Level Comment Distribution vs. Text Length Distribution

• Table 4 compares each main section’s percentage of total comments with its percentage of total text length.The supplied passages do not provide the table’s values.Source: C Section Level Comment Distribution… · p. 10, Table 4 · p. 11