MOPRD: A multidisciplinary open peer review dataset

1 Introduction

• Existing datasets often omit parts of the peer review process, including initial submissions needed to compare manuscripts before and after review.A complete dataset should include initial submissions, comments, meta-reviews, revisions, rebuttals, and editorial decisions.Source: §1 · pp. 2–3
• Many available datasets focus on computer science, limiting their disciplinary diversity and reuse for tasks in other fields.The introduction notes that models trained on single-discipline data can be difficult to reuse for tasks in other disciplines.Source: §1 · p. 3
• MOPRD brings together metadata, initial and revised manuscripts, reviews, meta-reviews, rebuttals, and editorial decisions across disciplines.The authors describe this coverage as making the dataset more complete and diverse.Source: §1 · p. 3
• The paper also proposes modular guided review-comment generation and explores meta-review generation, editorial-decision prediction, rebuttal generation, and scientometric analysis.Source: §1 · p. 3

2 Related work

• Peer-review datasets are grouped by their main applications: prediction, analysis, and generation.Table 1 presents their domains and applications.Source: §2 · pp. 3–4
• Prediction datasets include PeerRead (Kang et al., 2018), which pairs papers with decisions and often reviews, and collections supporting score or rebuttal studies.The ACL-18 Numerical Peer Review Dataset (Gao et al., 2019) includes review comments, rebuttals, and scores before and after rebuttal; the Interspeech 2019 collection includes rejected submissions but was not publicly available at the time described.Source: §2 · p. 4
• Analysis datasets cover reviewer argumentation, citation impact, comparative discussions, review requirements, and layered properties of review comments.Examples include AMPERE, CiteTracked, COMPARE, ReAct, and a multilayered dataset of 17k sentences from 1,199 comments.Source: §2 · pp. 4–5
• Generation datasets include ASAP-Review (Yuan et al., 2022) for review comments and MReD (Shen et al., 2022) for labeled meta-reviews.ASAP-Review contains 28,119 review comments; MReD contains 45,929 sentences from 7,089 labeled meta-reviews.Source: §2 · p. 5
• The authors position MOPRD as, to their knowledge, the largest multidisciplinary peer-review dataset with complete review histories.They argue its completeness supports different applications and more universal, generalized studies.Source: §2 · p. 5

3.1 Data source

• The project begins with a filtered source selection rather than an exhaustive, continuously updated collection, which the authors describe as too demanding to maintain.They identify future exhaustive collection, potentially through crowdsourcing, as a long-term goal.Source: §3.1 · pp. 5–6
• PeerJ was selected after considering scope, influence, data integrity, format, and licensing; its seven journals span multiple research disciplines.Source: §3.1 · p. 6
• PeerJ histories can include initial submissions, revisions, comments, meta-reviews, rebuttals, and editorial decisions, enabling reconstruction of the process through acceptance.Original manuscripts are available, unlike at many venues that provide only published papers; PeerJ also supplies reviews in plain text, structured into four segments, and permits reuse under a Creative Commons Attribution License.Source: §3.1 · pp. 6–7

3.2 Construction

• A special crawler collects paper metadata, review comments, meta-reviews, and editorial decisions, then downloads submissions, revisions, and rebuttal letters.MOPRD is provided in a Native version with collected files and a Processed version with parsed HTML information in JSON and converted manuscripts and rebuttals.Source: §3.2 · pp. 7–8

3.3 Properties

• MOPRD contains 6,578 papers, and Fig. 2 presents the disciplines represented in the collection.Table 2 lists statistical features of the dataset, but the supplied passage does not give its individual values.Source: §3.3 · p. 8, Table 2 · p. 8

4 Review comment generation

• Automated scholarly paper review is described as a way to improve efficiency and save resources for academic and publishing communities.Source: §4 · p. 8
• Review-comment generation is identified as one of the key steps in achieving automated scholarly paper review.Source: §4 · p. 8
• The authors describe MOPRD as a large-scale review-comment dataset suited to review-comment generation.Source: §4 · p. 8

4.1 Method

• Long manuscripts strain model input limits, while extraction loses information; the method segments manuscripts by review structure and uses existing plus auto-labeled review segments.The authors identify input length and labeled data as the two main challenges.Source: §4.1 · pp. 9–10

4.1.1 Preprocessing

• GROBID converts PDFs to structured text, then SciBERT classifies sections into summary, methods-and-results, and other subsets, achieving 0.986 accuracy and 0.986 weighted F1.The subsets comprise general-summary sections, methods and results, and the full manuscript.Source: §4.1.1 · pp. 10–11

4.1.2 Preparation for generation

• LED is fine-tuned into five module-specific models, pairing manuscript subsets with review segments; Questions and Proposals are auto-labeled, while figure/table and PeerJ sentences are filtered.LED supports inputs up to 16,384 tokens.Source: §4.1.2 · pp. 11–13

4.1.3 Generation

• Module-specific prefixes guide five models to generate distinct review sections, addressing the missing contextual coherence and schematic structure of pretrained-model reviews.Prefixes are randomly selected from predefined sets of phrases or sentences with similar meanings.Source: §4.1.3 · p. 13

4.1.4 Postprocessing

• Postprocessing concatenates the five model outputs into one review comment and converts it to a reader-friendly format.Source: §4.1.4 · p. 14

• The generation process is illustrated in Fig. 3.Source: §4.2 · p. 14, Figure 3 · p. 14
• Modular processing is designed to handle long manuscripts and generate longer, more comprehensive review comments.The method uses LED and manuscript segmentation to process long inputs, with five generation models allowing output up to five times the length of a traditional model.Source: §4.2 · pp. 14–15
• Guidance gives each module direction, while modular generation establishes an overall structure intended to improve coherence and readability.Source: §4.2 · p. 15
• The method, developed around PeerJ’s four review segments, is reusable for papers with similar structures and can support special structures when related labeled data are available.Source: §4.2 · p. 15
• The method has so far been evaluated on MOPRD as a baseline, and the authors say enough data are needed to address multidisciplinary bias.Source: §4.2 · p. 15

4.3 Experiments

• The experiments use initial submissions and their corresponding comments, yielding 10,998 manuscript-comment records split randomly into training, validation, and test sets at 8:1:1.Manuscripts under 2,000 words and comments under 100 words are filtered out; decoding uses joint Top-P and Top-K sampling with p=0.92 and k=50, plus a no-repeat 3-gram penalty.Source: §4.3 · pp. 15–16

4.3.1 Sanity test

• In the sanity test, the generated comment addresses each target module’s focal points and the paper’s content, without conventional template-like rigidity.The authors also report that some output has grammatical errors or inexplicable phrasing, which they identify as a focus for future improvement.Source: §4.3.1 · p. 16, Figure 4 · p. 17

4.3.2 Comparison of generation methods over automatic metrics

• The modular guided method outperforms the comparison methods on all reported automatic metrics except ROUGE-L, where extract-then-generate scores highest.The authors attribute the ROUGE-L result to beam search, which they say can also cause repetition; they report that modular and guided designs work best together.Source: Table 3 · p. 18, §4.3.2 · p. 18

4.3.3 Comparison of generation methods over human evaluation

• The modular guided method surpasses the compared method on every human-evaluation metric, with its highest score, 3.45, for structure & logicality.Its specialty & accuracy rating is relatively lower; both methods are described as generating useful and inspiring comments.Source: Table 4 · p. 19, §4.3.3 · p. 19

4.3.4 Single-discipline training data VS multi-discipline training data

• The multi-discipline model performs better than the single-discipline model on all metrics in the computer-science test.The authors note that scores in this experiment are lower than in prior experiments, citing the unseen test discipline and halved training-data size as reasons.Source: Table 5 · p. 20, §4.3.4 · p. 20

5.1 Meta-review generation

• MOPRD’s initial submissions and revisions support meta-review generation across different review rounds, potentially incorporating manuscript content.The task is framed as multi-document summarization, with review comments as input and a meta-review as output.Source: §5.1 · pp. 20–21

5.2 Editorial decision prediction

• MOPRD’s multidisciplinary data permit comparison of review-comment sentiment across disciplines, which previous analyses could not make.Editorial decision prediction uses review comments to predict acceptance, revision, or rejection.Source: §5.2 · p. 21

5.3 Author rebuttal generation

• Author rebuttal generation can use review comments, meta-reviews, and the manuscript as inputs to produce point-by-point responses.Segmentation and classification help link review points to relevant manuscript content, guiding the generated response.Source: §5.3 · p. 22

5.4 Scientometric analysis

• MOPRD contains 20% more data across a wider variety of disciplines than a prior PeerJ scientometric study, potentially supporting more profound analysis.The cited comparison is with Matsui et al. (2021), which analyzed approximately 5,000 papers from three disciplines.Source: §5.4 · p. 22

6 Conclusion

• The modular guided review comment generation method outperforms compared methods on ROUGE and BARTScore and across all three human-evaluation aspects.Every human-evaluation aspect score is above the mean, with structure & logicality scoring highest.Source: §6 · p. 23
• MOPRD contains 6,578 papers with metadata and six types of material spanning submission through publication.These include multiple manuscript versions, review comments, meta-reviews, author rebuttal letters, and editorial decisions.Source: §6 · p. 23
• For review comment generation in unknown disciplines, models trained on multi-discipline data work better than models trained on single-discipline data.Source: §6 · p. 23
• The paper also explores MOPRD applications in meta-review generation, editorial decision prediction, author rebuttal generation, and scientometric analysis.Source: §6 · p. 23

Data availability

• The method for obtaining the dataset is provided within the paper.Source: Data availability · p. 23