PaperlayerView exact revision on arXiv

Source-linked AI summary

Re2: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussions

Daoze Zhang, Zhijian Bao, Sihang Du, Zhiyi Zhao, Kuangling Zhang, Dezheng Bao, Yang Yang

arXiv:2505.07920v2cs.CLcs.AIcs.LG

TL;DR

Peer-review datasets have limited venue diversity, may pair reviews with revised rather than initial submissions, and often lack well-structured rebuttal discussions. Re2 assembles consistency-ensured review data and formats rebuttal interactions as multi-turn conversations; it provides a broad dataset for static review tasks and interactive rebuttal support.

  • Problem

    Existing peer-review datasets have limited source diversity, may use revised papers instead of the initial submissions reviewers assessed, and often lack consistent multi-turn rebuttal data.

    Source: §1 · p. 2

  • Method

    Re2 collects initial submissions and full-stage review records from OpenReview, then structures accessible rebuttal and discussion exchanges as multi-turn conversations.

    Source: §1 · pp. 2–3

  • Results

    Re2 includes 19,926 initial submissions and 70,668 reviews across 24 conferences and 21 workshops; its rebuttal subset pairs 14,830 submissions with 53,818 rebuttal discussions.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The dataset supports traditional review tasks and interactive rebuttal discussions, with the stated aim of helping authors self-evaluate and easing review burden.

    Source: §1 · p. 3, §5 · p. 9

    The experiments cover textual and tabular paper components, excluding visual elements such as figures.

    Source: §5 · p. 10

Abstract

from arXiv · show

Peer review is a critical component of scientific progress in the fields like AI, but the rapid increase in submission volume has strained the reviewing system, which inevitably leads to reviewer shortages and declines review quality. Besides the growing research popularity, another key factor in this overload is the repeated resubmission of substandard manuscripts, largely due to the lack of effective tools for authors to self-evaluate their work before submission. Large Language Models (LLMs) show great promise in assisting both authors and reviewers, and their performance is fundamentally limited by the quality of the peer review data. However, existing peer review datasets face three major limitations: (1) limited data diversity, (2) inconsistent and low-quality data due to the use of revised rather than initial submissions, and (3) insufficient support for tasks involving rebuttal and reviewer-author interactions. To address these challenges, we introduce the largest consistency-ensured peer review and rebuttal dataset named Re^2, which comprises 19,926 initial submissions, 70,668 review comments, and 53,818 rebuttals from 24 conferences and 21 workshops on OpenReview. Moreover, the rebuttal and discussion stage is framed as a multi-turn conversation paradigm to support both traditional static review tasks and dynamic interactive LLM assistants, providing more practical guidance for authors to refine their manuscripts and helping alleviate the growing review burden. Our data and code are available in https://anonymous.4open.science/r/ReviewBench_anon/.

1 Introduction

The paper identifies gaps in peer-review datasets in breadth, submission-version consistency, and support for rebuttal discussions. Re2 addresses these gaps with a consistency-ensured dataset spanning initial submissions, reviews, and structured multi-turn rebuttal discussions.

1 Introduction

  • Prior datasets cover few conferences and do not comprehensively capture publicly available OpenReview reviews, limiting source diversity and scale.Most surveyed datasets draw on ICLR, while some include venues such as NeurIPS and ACL.Source: §1 · p. 2
  • Existing datasets often rely on revised rather than initial submissions, risking inconsistency between the paper models see and the reviews they are meant to address.The paper argues that review tasks should use the initial submission because reviewers’ comments address that version.Source: §1 · p. 2
  • Many existing datasets omit rebuttals, while those that include them often handle discussions coarsely and lack consistency for multi-turn tasks.The paper frames rebuttal–discussion as a conversation in which models can respond to authors’ rebuttals, beyond static review generation.Source: §1 · p. 2
  • Re2 combines 19,926 initial submissions and 70,668 reviews with 14,830 submissions paired with 53,818 rebuttal discussions, spanning 24 conferences and 21 workshops.The dataset includes initial submissions and review stages, while its rebuttal subset structures author–reviewer interactions as multi-turn conversations.Source: §1 · pp. 2–3
  • Re2 supports both traditional review tasks and interactive models for rebuttal and discussion, with the stated aim of helping authors self-evaluate and easing review burden.Its reported contributions include statistical analysis and experiments across four review-related tasks.Source: §1 · p. 3

2 The Re2 Dataset

Re2 collects peer-review records across venues and years, verifies that papers are initial submissions, and standardizes full-stage review data. Its rebuttal subset converts author–reviewer exchanges into structured multi-turn dialogues, and the dataset statistics describe its coverage and score distributions.

2.1 Data Collection and Processing

  • The Re2-Review collection spans 24 conferences and 21 workshops from 2017 onward, using deadline-based revision histories to select each paper’s latest version before submission.The authors first crawled publicly accessible records across 68 conferences from 2013 to 2025, then applied the initial-submission filter.Source: §2.1 · p. 3
  • Manual audits and conference-year-specific extraction logic address the differing review formats across the 45 venues.The paper content is also converted from PDF to LaTeX and Markdown for accessibility and downstream use.Source: §2.1 · p. 3
  • The rebuttal subset retains papers with publicly accessible rebuttal stages and organizes author rebuttals and subsequent discussions into structured multi-turn dialogues.Consecutive posts from the same role are merged into a turn; reminder posts are excluded, while global responses are inserted as reference context rather than direct replies.Source: §2.1 · pp. 4–6

2.2 Dataset Statistics

  • Most papers range from 6,000 to 16,000 tokens and most reviews from 300 to 1,100 tokens; ICLR and NeurIPS account for the largest paper and review shares.Those two conferences also account for the highest proportion of rebuttals in the conference-by-year distribution.Source: §2.2 · p. 6
  • Acceptance proportions rank highest for self-supervised learning and second-highest for generalization, while deep learning and reinforcement learning have the most papers.These are distributions across the ten most frequent keywords.Source: §2.2 · p. 6
  • Normalized median review scores across the top ten conferences fall between 5 and 7, and ICLR’s score distribution is the most uniform.The paper notes that kernel-density smoothing can extend violin tails beyond the score-range boundary.Source: §2.2 · pp. 6–7

3 Experiment

Experiments evaluate prediction, review generation, and rebuttal-discussion tasks, with results indicating that fine-tuning on Re2 data improves language-model performance across these settings. Fine-tuned LLaMA-3.1-8B leads on review-generation overlap metrics and on four of five rebuttal-discussion similarity metrics.

3.1 Experimental Setup

  • The experiments sample 1,000 Re2-Review papers and 500 Re2-Rebuttal papers for testing, fine-tune open-source LLMs with LoRA, and report zero-shot and fine-tuned results.The evaluated tasks are acceptance and score prediction, review generation, and rebuttal-discussion conversation.Source: §3.1 · p. 7
  • Acceptance prediction is evaluated with accuracy, precision, recall, and F1, while score prediction uses MAE and MSE.Review generation uses BLEU, ROUGE, BERTScore, and EmbedCos; rebuttal-discussion also uses an LLM judge across five dimensions.Source: §3.1 · p. 7

3.2 Experimental Results

  • After fine-tuning, LLaMA-3.1-8B and Qwen2.5-7B show lower MAE and MSE for score prediction, while their untuned versions accept every paper.SEA-E and DeepReviewer-14B perform strongly on accuracy, recall, and F1; LLaMA-OpenReviewer-8B and CycleReviewer-8B show higher precision but lower F1.Source: §3.2 · p. 7, Table 2 · p. 8
  • Fine-tuned LLaMA-3.1-8B ranks first among all models on BLEU and ROUGE-L for review generation.Fine-tuned LLaMA-3.1-8B and Qwen2.5-7B also improve EmbedCos by 49.9% and 20.5%, respectively, compared with the third place.Source: §3.2 · p. 8, Table 3 · p. 8
  • Fine-tuned LLaMA-3.1-8B achieves the best performance on four of five rebuttal-discussion similarity metrics and the best overall LLM-as-judge performance.Both fine-tuned models generally outperform their zero-shot versions across the similarity and judge evaluations.Source: §3.2 · p. 8, Table 4 · p. 8, Table 5 · p. 9

4 Related Work

Prior work spans static review datasets and datasets that incorporate rebuttal content, but the paper identifies unresolved consistency, diversity, scale, and interaction-coverage gaps. Re2 addresses these with initial-submission data and a multi-turn conversation framing for rebuttals and discussions.

Static Review Datasets

  • Static review datasets support analysis, acceptance or score prediction, and review generation, but often cannot ensure that manuscript versions match those reviewed or include rebuttals.The cited work includes datasets and methods spanning venues, fairness analysis, aspect-prompted reviews, and review-revision questions.Source: §4 · p. 9

Rebuttal-included Datasets

  • Rebuttal-included datasets add argument annotations or model reviewer-author and inter-reviewer discussions, yet remain limited by version reliability, diversity, scale, or simulated-data reliability.The passage describes Kennard et al. [2021], Jin et al. [2024], and Wu et al. [2022].Source: §4 · p. 9

Re2 Advances

  • Re2 ensures its 19,926 papers are initial submissions and frames rebuttal and discussion data as multi-turn reviewer-author conversations.The paper presents Re2 as a large real-world dataset with broad conference coverage and comprehensive review stages.Source: §4 · p. 9

5 Conclusion

Re2 is presented as a consistency-ensured real-world peer-review dataset for training and evaluating review and rebuttal tasks, including interactive dialogue. The conclusion connects this resource to reviewer burden and author self-evaluation, while noting that experiments exclude visual paper content.

Conclusion

  • Re2 is presented as a consistency-ensured real-world peer-review dataset for training and evaluating review and rebuttal tasks.The conclusion describes it as the largest such dataset.Source: §5 · p. 9
  • The dataset supports both traditional static tasks and interactive dialogue-based models for rebuttal and discussion.The conclusion states these capabilities can support author self-evaluation and manuscript improvement before submission and reduce reviewer burden.Source: §5 · pp. 9–10

Limitations

  • Experiments are limited to textual and tabular paper components and exclude figures, although figures are included in the released dataset.The authors identify integration of vision-language models as future work.Source: §5 · p. 10

A Details of the Violin Plot

Violin plots show normalized review-score distributions by conference, combining kernel-density contours with boxplot summaries.

A Details of the Violin Plot

B Details of the Peer Review Methods in the Experiment

The experiment evaluates four peer-review-focused systems, spanning standardized-review fine-tuning, expert-review tuning, evidence-based review, and iterative feedback.

B Details of the Peer Review Methods in the Experiment

C Details of the Evaluation Metrics

The section explains four text-generation evaluation metrics: BLEU and ROUGE-L measure lexical overlap, while BERTScore and EmbedCos measure semantic similarity. It also describes the metrics’ main limitations and the comparison presented in Table 6.

C Details of the Evaluation Metrics

  • BLEU combines n-gram precisions using a geometric mean and applies a brevity penalty to discourage overly short outputs.The penalty compares generated-text length with reference-text length.Source: C Details of the Evaluation Metrics · pp. 14–15
  • BLEU is insensitive to synonyms and grammatical variation, and is better suited to longer texts or whole documents than single sentences.The authors say this makes it suitable for their review-generation tasks.Source: C Details of the Evaluation Metrics · p. 15
  • ROUGE-L reports an F1 score based on the longest common subsequence, and selects the highest score when multiple reference texts are available.The longest common subsequence preserves word order but need not be contiguous; ROUGE’s lexical matching does not effectively measure semantic similarity.Source: C Details of the Evaluation Metrics · p. 15
  • BERTScore compares contextual word embeddings using pairwise cosine similarities, matching each word to its most similar counterpart to calculate precision and recall.It can recognize synonyms to some extent, but remains word-level, omits sentence structure and grammatical order, and requires substantial computation.Source: C Details of the Evaluation Metrics · pp. 15–16
  • EmbedCos measures cosine similarity between sentence-level embeddings produced with sentence-transformers/all-mpnet-base-v2.The authors chose MPNet for efficient generation of high-quality sentence embeddings.Source: C Details of the Evaluation Metrics · p. 16
  • Table 6 compares BLEU, ROUGE-L, BERTScore, and EmbedCos.The supplied caption identifies the metrics but gives no comparison values.Source: Table 6 · p. 16, C Details of the Evaluation Metrics · p. 16

D Details of the LLM as Judge

The authors use LLaMA-3.1-8B-Inst to judge generated rebuttal-discussion dialogue statements across five dimensions. The judge is instructed to score conservatively on a 1–10 scale.

D Details of the LLM as Judge

  • The LLaMA-3.1-8B-Inst judge scores generated dialogue statements for accuracy, clarity, constructiveness, completeness, and overall quality.Accuracy and clarity assess technical output; constructiveness addresses practical relevance, completeness concerns coverage, and quality is the global indicator.Source: D Details of the LLM as Judge · pp. 16–17
  • The rubric defines accuracy as factual and logical correctness, clarity as fluent and understandable expression, constructiveness as practical value, and completeness as coverage of key issues.Overall quality integrates the other dimensions.Source: D Details of the LLM as Judge · p. 16
  • The judge assigns each dimension a 1–10 score and is instructed to grade conservatively, reserving high scores for exceptionally strong responses.The prompt also asks for a reasonable score distribution and objective grading.Source: D Details of the LLM as Judge · p. 17
Loading 2505.07920v2…