Source-linked AI summary
Metag: A dataset to build agentic meta-reviewing capabilities
Anirudh Sundar, Min Chen, Divya Tadimeti, Gemma Zhang, Alice Li, Nigel Boachie Kumankumah, Pavan Uttej Ravva, Sadid Hasan, Somya Chatterjee, Pruthvi Prakash Navada, Xiao Wang, Yue Kang, Sulaiman Vesal, Larry Heck
TL;DR
Meta-reviewers need traceable links between reviewer feedback and the manuscript changes authors make during rebuttal. Metag constructs a human-reviewed dataset by aligning dialogue-derived action items with structured PDF differences between pre-review and camera-ready papers. The dataset contains 349 samples, and GPT-5.6-Sol achieves an F1-score close to 0.40 among the evaluated models.
Problem
Meta-reviewing requires connecting reviewer concerns and author responses to concrete manuscript changes, which existing review aggregation and document-difference tools do not generally provide together.
Method
Metag computes structured differences between pre-review and camera-ready PDFs and links selected edits to action items extracted from reviewer-author dialogues through human annotation.
Results
349 high-quality human-reviewed samples were collected, and GPT-5.6-Sol achieved an F1-score close to 0.40 among the evaluated models.
Takeaways & Limitations
Metag supports agentic meta-reviewing systems that verify author statements and ground judgments in manuscript evidence.
Takeaways & Limitations
Metag is derived exclusively from ICLR 2024 and relies on linking camera-ready submissions with preprints, while intermediate revisions are not always publicly available.
Abstract
from arXiv · showhide
AI tools increasingly support tasks across the scientific research cycle, from experiment design and manuscript preparation to peer review. At the same time, the continuing growth in conference submissions has increased the burden on meta-reviewers, who must synthesize reviewer feedback, author rebuttals, and manuscript revisions. To address this concern, this paper introduces Metag, a dataset to accelerate the development of meta-reviewing agents, specifically to identify changes made to scientific articles during the review-rebuttal process. Each instance contains a reviewer concern, the author's proposed resolution, and the manuscript diffs implementing the stated change. Metag is collected by obtaining manuscript versions from before the review deadline and after acceptance, computing differences between the two documents, and asking human annotators to align these differences with action items from OpenReview discussions. The resulting dataset consists of 349 high-quality action items tied to paper differences and will enable building methods to empower meta reviewers to quickly identify whether authors have addressed reviewer statements and where in the paper those changes have been made, resulting in additional transparency and traceability throughout peer review. The dataset is publicly available at https://github.com/microsoft/Metag-dataset.
1 Introduction
AI tools assist authors and reviewers, but meta-reviewers still need traceable links between review concerns, author commitments, and concrete manuscript revisions. Metag addresses this gap by pairing reviewer-author action items with the relevant diffs between original and revised papers.
- Motivation: AI tools support authoring and reviewing, while meta-reviewing remains relatively under-explored.The paper frames meta-reviewing as a distinct stage requiring additional assistance.
- Motivation: Meta-reviewers must synthesize reviews, rebuttals, and discussions whose revision evidence is distributed across interconnected texts.Author responses may describe revisions without clearly locating them in the final manuscript.
- Research gap: Existing document-comparison tools identify PDF differences but generally do not explain which reviewer action item motivated each change.The paper identifies this missing link as an underexplored extension of document comparison.
- Contribution: Metag releases a dataset and collection methodology linking reviewer-author action items to corresponding changes between original and revised scientific papers.The dataset is intended to support agentic human-in-the-loop systems for navigating revisions and verifying author statements.
2 Related Work
Prior datasets separately model scientific reviews or document revisions, while Metag focuses on linking reviewer dialogue to observable manuscript changes. Its formulation combines structured PDF differences with action-item annotations.
- Scientific reviews without document edits: Review-focused datasets capture questions, discussions, summaries, or reviewer suggestions but do not model the resulting manuscript edits.The cited resources emphasize review content, intent, or organization.
- Document edits without scientific reviews: Revision-focused datasets extract document changes or classify edit intentions without including the reviewer feedback that prompted scientific revisions.These resources study iterative revision independently of peer review.
- Scientific reviews and document edits: ARIES most closely relates to Metag by aligning reviewer comments with scientific-manuscript revisions.ARIES aligns actionable comments with textual spans, whereas Metag selects structured PDF diffs implementing dialogue-derived action items.
- Metag’s distinction: Metag preserves insertions, deletions, replacements, moved text, page locations, and surrounding context when linking review dialogue to revisions.This representation targets traceability between reviewer feedback and observable manuscript changes.
3 Method
Metag constructs paired reviewer-action-item and manuscript-diff data through document acquisition, PDF differencing, action-item extraction, and human filtering and linking. The resulting pipeline supports lexical and language-model-based diff classification.
- Pipeline: Metag collection proceeds through six stages: scraping, PDF acquisition, difference computation, action-item extraction, annotation and filtering, and dataset assembly.The stages are designed to connect review-rebuttal dialogue with concrete manuscript revisions.
- Paper and PDF acquisition: OpenReview submissions and dialogues are linked to pre-review arXiv PDFs through Semantic Scholar before comparing them with camera-ready papers.The pipeline retains the arXiv version before submission and excludes papers without an obtainable version history.
- PDF Difference Computation: PDF differencing parses both documents and produces text-block edits tagged as insert, delete, or replace with text spans, context, pages, and word-level bounding boxes.The structured diff follows Git’s histogram algorithm after parsing with PyMuPDF.
- Action Item Extraction: Action items are extracted from reviewer-author dialogues as reviewer-concern and author-response pairs, then filtered by annotators before diff linking.The extraction prioritizes explicit commitments to modify the manuscript and references to specific sections, tables, equations, or figures.
- Human Annotation and Filtering: Annotators use a side-by-side PDF viewer to select color-coded diff regions corresponding to each retained action item.Selections record the pane, page, change type, diff text, and surrounding context; the interface supports navigation, search, synchronized viewing, and batch reuse of computed diffs.
- Agreement and classification: Two annotators retain only intersecting linked diffs, with exact-set agreement of 37% and mean Jaccard similarity of 50.1%.The dataset also benchmarks BM25, TF-IDF, embedding-based, and language-model approaches for diff classification.
4 Results
The experiments compare lexical retrieval and language-model approaches for linking review action items to manuscript diffs. GPT-5.6-Sol performs best on test data, while lexical methods and Gemma variants struggle with the task.
- Lexical retrieval baselines: BM25 reaches micro-F1 .108 on validation and .098 on test, while TF–IDF reaches .092 and .096, respectively.Their low recall suggests direct lexical overlap often misses revisions that express an action item's intent differently.
- LLM-based diff classification: Hosted-model evaluation uses three independent runs, reporting means and sample standard deviations after rerunning failed API or parsing requests.Candidate diffs are processed in windows; window size 80 gives DeepSeek-V4-Pro the highest mean validation micro-F1 while reducing repeated instructions and API calls.
- Main findings: GPT-5.6-Sol achieves the strongest test performance, with micro-F1 .360 ± .005 and macro-F1 .398 ± .018.Its precision is .270 ± .004 and recall is .540 ± .008.
- Main findings: DeepSeek-V4-Pro records the highest validation micro-F1 at .410 ± .121, but its test micro-F1 falls to .261 ± .021.The large validation standard deviation indicates substantial run-to-run variability.
- Main findings: Kimi-K2.5 has high recall on validation and test, but lower precision limits its test micro-F1 to .254 ± .009.
- LLM-based diff classification: Gemma-3-27B-IT zero-shot achieves recall of .634 on validation and .601 on test but predicts over 200 diffs per action item, yielding test micro-F1 .037.LoRA fine-tuning improves its test micro-F1 to .083, while the embedding-based MLP reaches .069.
5 Conclusions and Future Work
Metag is a human-reviewed dataset linking OpenReview action items to manuscript changes, benchmarked with lexical retrieval and language models. GPT-5.6-Sol is the strongest evaluated model, with an F1-score close to 0.40.
- Conclusions: Metag contains 349 high-quality human-reviewed samples linking reviewer action items to specific scientific-manuscript changes.The dataset is collected by scraping OpenReview reviews, identifying action items, and linking them to revisions.
- Conclusions: Metag is benchmarked with lexical retrieval baselines and open- and closed-source language models for the revision-linking task.
- Conclusions: GPT-5.6-Sol is the most capable evaluated model, achieving an F1-score close to 0.40.
- Future work: The dataset is limited to ICLR 2024, and extending it to more venues is left for future work.The authors also identify broader benchmarking with open-source models as a future direction.
6 Limitations
Metag’s coverage and revision alignment are constrained by its reliance on ICLR 2024 and on identifying a conference-submitted manuscript version.
- Scope: Metag is derived exclusively from ICLR 2024, so additional venues, years, disciplines, and publication formats are needed to assess generalizability.
- Data availability: The method relies on accurately identifying the manuscript version submitted to the conference, but intermediate review and rebuttal versions are not always publicly available.Timestamped intermediate revisions could support finer-grained analysis of manuscript evolution.
A Prompt for Action Item Extraction
The extraction prompt instructs a model to identify actionable reviewer requests and author statements indicating that manuscript changes were made, with outputs represented in a structured schema.
- Prompt inputs: The prompt supplies a review, an author-reviewer dialogue, and instructions to identify action items for authors to fix in the paper.
- Action-item criteria: It targets reviewer-noted typos, grammatical errors, and author statements that they will fix content in the paper.
- Action-item criteria: The prompt retains cases where dialogue indicates a completed manuscript change, such as updating, revising, or editing the paper.Responses should be extracted directly from the dialogue and paraphrase the reviewer comment and author response.
- Action-item criteria: Specific references such as sections or equations are prioritized because they help locate where changes were made.
- Examples: The schema example includes reviewer comments and author responses describing requested fixes and proposed resolutions.
- Output representation: Diff-classification instances provide review and dialogue context alongside candidate diffs, with labels marking whether each diff is relevant to an action item.The relevant_diff_indices field lists the indices of all positively labeled diffs.
B Dataset Schema
Each dataset entry represents one action-item/paper pair, with labels aligned to the paper’s complete diff list. Relevant diff indices are also stored separately for convenience.
- Each entry corresponds to one action item and paper.
- The labels array aligns positionally with all_diffs.labels[i] indicates whether all_diffs[i] is relevant to the action item.
- The relevant_diff_indices field lists the indices of all relevant diffs.This provides a convenient index-based representation of the true labels.
C Windowing
The method selects a diff window size by sweeping candidate values and balancing evaluation cost against the granularity available to the model. Validation performance was optimal at a window size of 80.
- 80 diffs per prompt produced optimal validation performance.The sweep evaluated window sizes of 10, 40, 80, 160, and 320.
- Larger windows reduce the number of prompts and evaluation cost.
- Larger windows also provide the model with more granularity.
D Hyperparamter Settings
The experiments tune retrieval, prompting, generative, and embedding-based settings for diff classification. BM25 and TF-IDF use validation-selected k = 6, while the prompting setup uses batched candidate diffs and specified decoding controls.
- Retrieval settings: k = 6 was the best validation-set value for BM25 and TF-IDF retrieval.Values from 1 to 100 were swept, and Figure 5 marks the selected cutoff.
- Model settings: Gemma-3-27B-IT was evaluated with zero-shot prompting, LoRA fine-tuning, and an embedding-based classifier.
- Prompting settings: Candidate diffs were grouped into batches of 50 after trivial formatting changes were removed.Decoding used temperature 0.1, nucleus-sampling probability p = 0.95, and a maximum of 1,024 generated tokens.
- Fine-tuning settings: LoRA training used three epochs, learning rate 2×10−5, rank r = 8, α = 16, and dropout 0.05.The effective batch size was 4 and input sequences were truncated to 4,096 tokens.
- Embedding settings: The embedding classifier used 4-bit NF4 quantization and 5,376-dimensional mean-pooled representations.A three-layer MLP received the action-item embedding, diff embedding, and their element-wise product.
- Evaluation settings: Closed-source LLM experiments used Azure OpenAI and cost USD 534.57 in total.Default settings were used for reasoning effort, random seed, and output length.
E Annotator Details
The annotation process used two annotators per sample, with agreement assessed across doubly annotated action items. The reported results show substantial annotation subjectivity and cases where review commitments had no corresponding manuscript diff.
- Annotation process: Each sample was annotated by two annotators, with eleven annotators supporting the labeling effort.The annotation effort took between 2 and 3 hours depending on the annotator.
- Agreement: 37% of 575 doubly annotated action items received identical diff sets.
- Agreement: Only one annotator selected a diff for 112 of 575 items.Both selected at least one diff for 404 items, while neither selected one for 59 items.
- Annotation outcomes: 10.3% of action items had no relevant diff, indicating that authors did not always commit promised changes.
NeurIPS Paper Checklist
The checklist evaluates whether the paper clearly states its claims and limitations, supports reproducibility, reports experimental details and statistical significance, and documents new assets and potential societal impacts.
- Claims and limitations: The paper states that its abstract and introduction accurately reflect its contributions and scope, and that limitations are discussed in Section 6.
- Theoretical foundations: The paper reports no theoretical proofs, so theoretical proof requirements are marked not applicable.
- Reproducibility and access: Reproducibility is addressed through planned release of code and models after acceptance, with dataset examples provided in the paper.
- Experimental details: The checklist asks for sufficient training and test details, including data splits, hyperparameters, and optimizer information, with supporting details cited in the appendices.
- Statistical significance: Statistical reporting includes standard deviation in Table 3 and error bars in Figure 4.
- Assets and societal impact: The dataset is identified as a new asset, with examples provided in Figure 1 and Listing 1, while responsible-release guidance addresses potential misuse and safeguards.