PaperlayerView exact revision on arXiv

Source-linked AI summary

Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactions

Cheng Tan, Dongxin Lyu, Siyuan Li, Zhangyang Gao, Jingxuan Wei, Siqi Ma, Zicheng Liu, Stan Z. Li

arXiv:2406.05688v1cs.CLcs.AIcs.LG

TL;DR

Existing LLM peer-review applications primarily generate static reviews, missing the iterative dialogue of real review. This paper introduces ReviewMT and role-specific metrics for a multi-turn, long-context process involving reviewers, authors, and decision makers. The resulting dataset captures interactions across ICLR and Nature Communications, while the paper identifies limited venue coverage as a scope boundary.

  • Problem

    Existing LLM peer-review research primarily generates static reviews and does not capture the dynamic, iterative nature of real-world peer review.

    Source: §1 · p. 2

  • Method

    The paper reformulates peer review as a long-context, multi-turn dialogue among reviewers, authors, and decision makers, and constructs an annotated dataset with role-specific evaluation metrics.

    Source: §1 · p. 2

  • Results

    ReviewMT collects annotated peer-review dialogues from ICLR and Nature Communications, covering interactions among authors, reviewers, and decision makers.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The role-based, iterative framework aligns with real-world peer review and provides a foundation for future research on LLMs in this process.

    Source: §1 · p. 2, §6 · p. 9

    The dataset is limited primarily to ICLR and Nature Communications and may not represent the diversity of academic publishing.

    Source: §6 · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) have demonstrated wide-ranging applications across various fields and have shown significant potential in the academic peer-review process. However, existing applications are primarily limited to static review generation based on submitted papers, which fail to capture the dynamic and iterative nature of real-world peer reviews. In this paper, we reformulate the peer-review process as a multi-turn, long-context dialogue, incorporating distinct roles for authors, reviewers, and decision makers. We construct a comprehensive dataset containing over 26,841 papers with 92,017 reviews collected from multiple sources, including the top-tier conference and prestigious journal. This dataset is meticulously designed to facilitate the applications of LLMs for multi-turn dialogues, effectively simulating the complete peer-review process. Furthermore, we propose a series of metrics to evaluate the performance of LLMs for each role under this reformulated peer-review setting, ensuring fair and comprehensive evaluations. We believe this work provides a promising perspective on enhancing the LLM-driven peer-review process by incorporating dynamic, role-based interactions. It aligns closely with the iterative and interactive nature of real-world academic peer review, offering a robust foundation for future research and development in this area. We open-source the dataset at https://github.com/chengtan9907/ReviewMT.

1 Introduction

Existing LLM peer-review work mainly generates static reviews, whereas this paper frames peer review as a long-context, multi-turn dialogue among reviewers, authors, and decision makers. It introduces ReviewMT and role-specific evaluation metrics for that setting.

1 Introduction

  • Static review generation misses the dynamic, iterative peer-review process, which the paper models as a dialogue among reviewers, authors, and decision makers.The framework combines long-context grounding in the paper with multiple rounds of interaction.Source: §1 · p. 2
  • ReviewMT collects multi-turn peer-review dialogues from ICLR and Nature Communications, including accepted and rejected ICLR papers across a range of domains.Entries are annotated to capture interactions among authors, reviewers, and decision makers.Source: §1 · p. 2
  • The proposed role-specific metrics assess response validity and text quality, as well as final-review scores and decision-maker decisions.The authors aim for a fair and comprehensive assessment of LLM performance in the reformulated setting.Source: §1 · p. 2

2 Related Work

Prior instruction-tuning datasets largely focus on single-turn interactions, while LLM peer-review research has mainly studied review generation rather than the complete iterative process.

2.1 Instruction Tuning Dataset

  • Most instruction-tuning datasets focus on single-turn interactions rather than the multi-turn, long-context dialogues characteristic of peer review.The passage describes instruction tuning as training LLMs to follow instructions and perform designated tasks more reliably.Source: §2.1 · p. 3

2.2 LLM in Review

  • Studies report LLM-generated reviews comparable to human reviews and helpful GPT-4 feedback, but existing work focuses on generating reviews rather than simulating the complete multi-round process.MARG uses multiple LLM instances to divide paper sections and provide feedback, including for papers exceeding a model’s context size.Source: §2.2 · p. 3

3 Preliminaries

The paper contrasts static review generation with a multi-turn process in which reviewers, authors, and decision makers have distinct roles. The dialogue proceeds from initial reviews through rebuttals and updated reviews to a decision.

3 Preliminaries

  • Existing LLM peer-review work formulates review generation as producing a static review for a submitted paper, while this framework models the complete process as a multi-turn dialogue.The proposed process includes reviewer and author exchanges and a decision based on the dialogue.Source: §3 · pp. 3–4
  • Reviewers first assess the paper and raise questions, then produce a final review after considering the author’s rebuttal and clarifications.There may be multiple reviewers for each paper.Source: §3 · p. 3
  • Authors respond to reviewer comments by addressing them, clarifying misunderstandings, and describing changes or improvements to the paper.The rebuttal also defends the paper’s validity and significance while engaging with constructive criticism.Source: §3 · p. 3
  • In the fourth turn, the decision maker synthesizes the paper and dialogue, including the initial reviews, to make an informed decision.The framework describes the decision as following the exchanges among reviewers and authors.Source: §3 · pp. 3–4

4 ReviewMT Dataset

ReviewMT combines peer-review records from ICLR and Nature Communications in a structured, multi-turn dataset. The section describes its construction, dataset statistics, and metrics for evaluating LLM responses.

4.1 Data Source

  • ReviewMT draws on ICLR and Nature Communications, split into ReviewMT-ICLR and ReviewMT-NC to reflect their distinct review processes.The sources offer review data across machine-learning and multidisciplinary research.Source: §4.1 · p. 4

4.2 Data Processing

  • Collection converts papers to Markdown text and structures each record as a four-turn reviewer–author–reviewer–decision-maker exchange.Records are stored in JSON; when initial ICLR ratings are unavailable, final ratings are used, and Nature Communications Peer Review Files provide review information.Source: §4.2 · p. 4, Figure 2 · p. 5

4.3 Dataset Statistics

  • 5,760 ICLR papers in 2024 versus 485 in 2017; papers average approximately 13,000–26,000 tokens and typically receive three to four reviews.Average length dips in 2023 and 2024 because rebuttal-phase replies are absent; 38.1% of papers were accepted and 61.9% rejected.Source: Figure 3 · p. 5, §4.3 · pp. 5–6
  • Figure 4 separates ICLR keywords, including machine-learning topics, from Nature Communications keywords spanning scientific fields such as cell, human, gene, and immune.The combined keyword analysis illustrates the dataset's range of research topics.Source: Figure 4 · p. 6, §4.3 · p. 6
  • 26,841 papers, 92,017 reviews, and 567,207,583 tokens make up ReviewMT, with token counts calculated using the LLaMA-3 tokenizer.Table 1 presents detailed dataset statistics.Source: §4.3 · p. 6, Table 1 · p. 7

4.4 Evaluation Metrics

  • Evaluation uses paper, review, and decision hit rates for response validity, MAE for review scores, and F1-score for accept-or-reject decisions.A response receives a zero hit rate if it fails to address the paper, include a score, or give a clear decision, respectively.Source: §4.4 · p. 7
  • Text replies are assessed against references using BLEU-2 and BLEU-4, ROUGE-1/2/L, and METEOR.The text-quality measures cover initial and final reviews, author responses, and decision-maker meta reviews.Source: §4.4 · p. 7

5 Experiments

The experiments evaluate open-source LLMs on ReviewMT-ICLR and ReviewMT-NC, comparing zero-shot and supervised fine-tuned performance. Fine-tuning improves performance on ICLR, while gains on the one-turn NC evaluation are more modest.

5 Experiments

  • For ICLR, papers from 2017–2023 train the models and 100 papers from 2024 form the test set; NC uses 100 test papers and the remainder for training.Both zero-shot and supervised fine-tuned performance are reported using several open-source LLMs.Source: §5 · p. 7
  • Supervised fine-tuning improves ICLR performance across metrics, while zero-shot models show lower review and decision hit rates despite high paper hit rates.Yuan is an exception: it has a high review hit rate but an extremely low decision hit rate, which the authors attribute possibly to strict constraints on making decisions.Source: §5 · p. 8, Table 2 · p. 8, Figure 5 · p. 8
  • On NC, supervised fine-tuning generally improves metrics, but gains are more modest than on ICLR; the evaluation is one-turn because its open reviews are less complete.The authors suggest ICLR's multi-turn dialogue may offer richer learning context.Source: §5 · p. 8, Table 3 · p. 9, Figure 6 · p. 9

6 Conclusion and Limitation

ReviewMT is constructed and evaluated as a dataset for LLM peer review, modeling interactions among reviewers, authors, and decision makers. Its scope and visual-data coverage are limited, and biased reviews or decisions may be learned and replicated.

6 Conclusion and Limitation

  • ReviewMT models peer review as a multi-turn dialogue among reviewers, authors, and decision makers, with annotations spanning reviews, rebuttals, final reviews, and decisions.It draws on ICLR and Nature Communications papers and is intended as a resource for fine-tuning and evaluating LLMs.Source: §6 · p. 9
  • The dataset includes no figures in its main text, limiting its ability to handle visual data that are integral to some academic papers.Source: §6 · p. 9
  • ReviewMT currently draws primarily on ICLR and Nature Communications, which may not represent the full diversity of academic publishing.The authors call for expanding sources across disciplines.Source: §6 · p. 9
  • Biased training reviews or decisions could lead LLMs to replicate those biases and unfairly evaluate some groups or topics.Source: §6 · p. 9

Related papers

Loading 2406.05688v1…