Source-linked AI summary
PaperMentor: A Human-Centered Multi-Agent Writing Tutor for AI Research Papers on Overleaf
Jiarui Liu, Terry Jingchen Zhang, Ryan Faulkner, X. Angelo Huang, Vilém Zouhar, Dominik Glandorf, Isabel Dahlgren, Van Q. Truong, Rishit Dagli, Yuen Chen, Felix Leeb, Punya Syon Pandey, Yves Bicker, Suvajit Majumder, Wenyuan Jiang, Zeju Qiu, Sankalan Pal Chowdhury, Bernhard Schölkopf, Mona Diab, Zhijing Jin
TL;DR
Early-career researchers often lack structured, actionable mentorship for scientific writing, while existing AI tools focus on grammar or simulated review. PaperMentor combines a curated expert skill library with specialized agents to deliver Overleaf-native comments while leaving revisions to authors. In a 14-researcher study, it improved validity and actionability over direct prompting without the skill library, with scope limited by LaTeX-centered analysis and evaluation diversity.
Problem
Existing AI writing and reviewing tools do not provide drafting-stage, text-anchored guidance on narrative, organization, and technical presentation for authors lacking experienced mentorship.
Method
PaperMentor combines over 40 curated expert skill files with 12 specialized agents that generate expert-guided inline comments in Overleaf while authors retain revision control.
Results
PaperMentor significantly outperforms direct prompting without the skill library in comment validity and actionability, improving validity by 6.5 percentage points and actionability by 4.1 percentage points.
Takeaways & Limitations
The results support structured, mentor-like commenting that helps authors revise their own research papers without replacing authorship or judgment.
Takeaways & Limitations
The system primarily analyzes LaTeX source, and the evaluation does not capture the full diversity of writing styles, venues, disciplines, or researcher backgrounds.
Abstract
from arXiv · showhide
Expert writing feedback from experienced researchers is critical for early-career scholars to improve their manuscripts, yet high-quality feedback often remains scarce because reviewing research papers is labor-intensive. Emerging AI-powered writing assistants largely focus on grammar fixes or simulating peer review with final scores, yet they fall short of providing concrete, actionable suggestions that help students improve their papers during drafting. We present PaperMentor, a human-centered writing assistant system that delivers actionable suggestions as Overleaf-native inline comments while leaving the actual writing entirely to human authors. PaperMentor integrates an expert skill library carefully curated from established researchers' writing advice with 12 specialized agents covering different aspects of paper writing, such as formatting compliance, phrasing accuracy, and terminology consistency. In a user study (n=14), 90.6% of the generated comments were rated actionable and 67.5% were rated valid, significantly outperforming a GPT-5.2 baseline uswithout the skill library. We release PaperMentor as open source for public use. Our code is publicly available under the AGPL-3.0 license at https://github.com/jiarui-liu/overleaf
1 Introduction
PaperMentor addresses the lack of structured, drafting-stage mentorship by providing expert-guided inline feedback in Overleaf while preserving authors’ control over revisions. Its skill library and specialized agents improve feedback quality over direct prompting, though the system is presented as open source.
- Junior AI researchers often learn scientific writing through trial and error, while presentation quality can affect how their work is evaluated.
- Existing grammar assistants and AI reviewing tools do not provide text-anchored drafting feedback on narrative, organization, and technical presentation.
- PaperMentor combines over 40 expert skill files with 12 specialized agents covering methods, results, style, formatting, and terminology.
- PaperMentor delivers suggestions as native Overleaf inline comments, allowing authors to revise within their workflow while retaining control over the writing.
- In a study of 14 AI researchers, PaperMentor improved validity by 6.5 percentage points and actionability by 4.1 percentage points over direct prompting without the skill library.
2 Related Work
Prior AI paper-review systems often emphasize surface-level summaries and correlate weakly with human judgments, while writing tools mainly target language quality. PaperMentor instead focuses on structural feedback and commenting-based collaboration for scientific writing.
- LLM-based automated review often emphasizes surface-level summaries over deeper methodological weaknesses and shows limited correlation with human scoring.
- Existing writing workspaces and language tools address sentence-level quality but do not focus on structural and organizational feedback important for junior researchers.
- Feedback-based assistance through commenting rather than rewriting better preserves authorship and control in human-AI writing collaboration.
3 Task Definition
PaperMentor represents review output as anchored comments on a LaTeX project, pairing each comment with its source file, highlighted character span, feedback text, and severity.
- Each PaperMentor review comment identifies a source file, highlighted character span, feedback text, and severity label.
- The source file and character span locate the issue in the project, while the comment text gives feedback and the severity label marks its importance.
4 System Design
PaperMentor builds a modular review pipeline around curated expert guidance, paper- and venue-specific context, and specialized agents that analyze LaTeX content before consolidating comments back into Overleaf.
- 4.1 The Skill Library: The skill library curates expert guidance from faculty feedback and public writing guides into categories covering setup, venues, paper types, sections, figures and tables, and style.
- 4.1 The Skill Library: The library contains over 40 skill files totaling more than 16,000 words, derived from writing guidance, 32 published examples, and 350 conference reviews.
- 4.2 Input Processing: Input processing consolidates nested LaTeX files, extracts structural headers, identifies paper type, and uses venue and role-model context to guide review.
- 4.2 Input Processing: The system assigns sections to review domains such as abstract, introduction, methods, results, conclusion, appendix, style, formatting, and captions.
- 4.3 Multi-Agent Review: Example comments illustrate feedback anchored to text spans with concrete suggestions across agents and severity levels.
- 4.3 Multi-Agent Review: PaperMentor runs twelve agents concurrently, including seven section agents, three global agents, and two agents instantiated from paper type and venue.
- 4.3 Multi-Agent Review: Section agents receive focused text with abstract and introduction context, whereas global agents receive the full merged source.
- 4.3 Multi-Agent Review: Overlapping comments are deduplicated, higher-severity feedback is retained, and character spans map the final comments back to original LaTeX files.
5 System Demonstration
PaperMentor extends Overleaf with a sidebar panel that runs reviews and presents file-anchored comments in the native review interface.
- The Overleaf integration preserves native review behavior by injecting comments with highlighted spans anchored to LaTeX source locations.
- Users select a backbone model, optionally specify a target venue and role model paper, then start a full review from the sidebar.
- A progress indicator remains visible during review, after which results appear in a collapsible summary and a file-by-file comment list with severity indicators.
- The backend retrieves project documents, merges them into a TEX file, runs the three-phase pipeline, and returns results organized by source file.
6 User Study for Evaluation
The user study compares PaperMentor with a same-model baseline lacking the skill library, using human ratings of comment quality across papers and researchers.
- 6.2 User Study Design: The baseline uses the same GPT-5.2 model and identical prompt components but omits the skill library, isolating the library as the comparison factor.
- 6.2 User Study Design: Comments are rated for validity, actionability, and conciseness using binary judgments, with Table 2 reporting mean ratings and 95% confidence intervals.
- 6.2 User Study Design: 14 AI researchers evaluate comments on 80 papers, including internal student submissions and randomly sampled ICLR 2026 submissions.
- 6.3 Results: PaperMentor significantly outperforms direct prompting in validity and actionability, while baseline comments are more concise on average.
- 6.3 Results: The skill library creates a trade-off: comments become longer while validity and actionability improve under structured writing guidelines.
- 6.3 Results: Approximately 40% of comments address Methods and Results, while section-normalized attention favors the Abstract and Methods over appendices.
- 6.3 Results: Annotators generally found the feedback understandable, useful, professor-like in tone, and balanced in critique, especially for clarity, analytical depth, and grammar.
7 Skill Library Extensibility
PaperMentor’s skill library is designed as an evolving, community-extensible resource rather than a fixed prompt or monolithic reviewer.
- Researchers can add new skills or refine existing ones through simple text-based edits, allowing the library to evolve over time.
- Independent updates to venue expectations, paper types, and disciplinary writing norms make the system more adaptable than a fixed prompt.
- The proposed community-driven model encodes writing advice from senior researchers across HCI, NLP, and computer vision into reusable skill modules.
- The authors envision PaperMentor becoming shared infrastructure for collecting, maintaining, and operationalizing practical paper-writing knowledge.
8 Conclusion
PaperMentor combines expert-guided, specialized review agents with Overleaf-native comments to support actionable drafting feedback while preserving authorship and judgment.
- PaperMentor delivers expert-guided, actionable feedback directly within the Overleaf drafting workflow.
Limitations and Future Work
The paper identifies technical, evaluative, and architectural limitations, and proposes broader expert comparisons and document-wide grounding as future work.
- PaperMentor may miss issues requiring rendered PDFs, visual figure assessment, or numerical verification.
- The evaluation covers 80 papers and 14 annotators but does not represent the full diversity of writing contexts.
- The system depends on skill-library coverage and underlying LLM reliability, so its output is drafting assistance rather than authoritative review.
- Evaluation: The study compares against an LLM without expert skills, not against comments authored by experienced researchers.
- Global document awareness: Section-specific agents can produce validity errors because limited context omits information introduced elsewhere in the document.
Ethical Considerations
PaperMentor aims to broaden access to writing support while acknowledging risks of stylistic homogenization and limited cross-cultural or cross-disciplinary generalization.
- PaperMentor targets junior researchers lacking experienced mentors and seeks to reduce inequalities in scientific writing guidance.
- The system generates suggestions rather than rewrites, preserving authorial agency while over-reliance could homogenize styles or suppress diverse voices.
- The skill library reflects predominantly English-language, Western AI-venue norms and may not generalize equitably across backgrounds.
B Distribution of Comments and Annotation Scores Across Review Domains
The paper presents distributions of PaperMentor’s comments across review domains and sections alongside human annotation scores and text-length comparisons.
- Review domains: Figure 4 compares comment distributions across review domains with mean human annotation scores on validity, actionability, and conciseness.Error bars indicate standard deviation.
- Section-level distribution: PaperMentor generates comments more frequently in core sections such as the Abstract and Methods, prioritizing important document content.
- Section-level distribution: Table 4 compares each main section’s percentage of total text length with its percentage of total comments.