PaperlayerView exact revision on arXiv

Source-linked AI summary

Summarizing Multiple Documents with Conversational Structure for Meta-Review Generation

Miao Li, Eduard Hovy, Jey Han Lau

arXiv:2305.01498v4cs.CLcs.AI

TL;DR

Existing multi-document summarization datasets lack explicit inter-document relationships, limiting study of relationship comprehension for information integration. The paper introduces PeerSum and RAMMER for meta-review generation, and human assessment finds most meta-review content grounded in its sources.

  • Problem

    Existing multi-document summarization datasets lack explicit inter-document relationships, making it hard to study relationship comprehension for information integration and aggregation.

    Source: §1 · p. 1

  • Method

    The paper introduces PeerSum for meta-review generation and RAMMER, which uses relationship-aware sparse attention and a multi-task objective to leverage metadata.

    Source: §1 · pp. 1–2, §4 · p. 5, §6 · p. 9

  • Results

    Human assessment found that most meta-review content is grounded in source documents despite being highly abstractive.

    Source: Limitations · p. 9

  • Takeaways & Limitations

    PeerSum serves as a probe of how machines reason, aggregate, and summarize potentially conflicting opinions.

    Source: §1 · p. 2

    Human validation uses a simple assessment, larger open-source model comparisons are absent, and the authors characterize the results as preliminary.

    Source: Limitations · p. 9

Abstract

from arXiv · show

We present PeerSum, a novel dataset for generating meta-reviews of scientific papers. The meta-reviews can be interpreted as abstractive summaries of reviews, multi-turn discussions and the paper abstract. These source documents have rich inter-document relationships with an explicit hierarchical conversational structure, cross-references and (occasionally) conflicting information. To introduce the structural inductive bias into pre-trained language models, we introduce Rammer ( Relationship-aware Multi-task Meta-review Generator), a model that uses sparse attention based on the conversational structure and a multi-task training objective that predicts metadata features (e.g., review ratings). Our experimental results show that Rammer outperforms other strong baseline models in terms of a suite of automatic evaluation metrics. Further analyses, however, reveal that RAMMER and other models struggle to handle conflicts in source documents of PeerSum, suggesting meta-review generation is a challenging task and a promising avenue for further research.

1 Introduction

PeerSum frames meta-review generation as multi-document summarization over reviews, discussions, and paper abstracts, with explicit conversational relationships and sometimes conflicting opinions. RAMMER incorporates that structure and metadata, but models still struggle with conflicts.

1 Introduction

  • PeerSum supplies a dataset for studying meta-review generation from reviews, multi-turn author-reviewer discussions, and paper abstracts.The authors suggest generated drafts could reduce meta-reviewers’ workload in a time-consuming process.Source: §1 · p. 1
  • Its source documents have hierarchical conversational threads, occasional conflicts with explicit indicators, and metadata including review ratings, confidence, and paper acceptance outcome.These features make PeerSum a probe for how models reason over and summarize potentially conflicting opinions.Source: §1 · pp. 1–2
  • RAMMER uses relationship-aware sparse attention over conversational relations and a multi-task objective to predict document types, review ratings and confidence, and paper acceptance.The approach targets limited research on abstractive MDS methods that recognize relationships among source documents.Source: §1 · p. 2
  • RAMMER performs strongly against baseline models across automatic metrics and human evaluation, but models still struggle to recognize and resolve conflicts in source documents.The experiments also include an automatic metric based on predicting paper acceptance outcomes.Source: §1 · p. 2

2 Related Work

Prior MDS datasets and methods generally leave inter-document relationships or discourse structure implicit. RAMMER applies attention manipulation to capture broader relationships across documents.

2.1 MDS Datasets

  • The related-work comparison identifies WCEP, Multi-News, Multi-XScience, and WikiSum as abstractive MDS datasets spanning news, scientific, and Wikipedia domains.The discussion notes that Multi-XScience summaries are not always reflective of their cited papers.Source: §2.1 · p. 2
  • Earlier MDS datasets lack explicit inter-document structure and conflict information, while peer-review datasets either use official reviews alone or lack PeerSum’s hierarchical conversational structure.The related-work discussion connects these gaps to underexplored research on inter-document relationship comprehension.Source: §2.1 · p. 2

2.2 Structural Inductive Bias for Summarization

  • Many MDS systems flatten documents, while graph methods add parameters and can be difficult to construct; attention manipulation has mostly addressed sentence or single-document structure.RAMMER’s stated novelty is using attention manipulation for broader inter-document relationships.Source: §2.2 · pp. 2–3

3 The PEERSUM Dataset

PEERSUM pairs peer-review documents and meta-reviews across ICLR and NeurIPS while preserving their hierarchical conversation structure and metadata. Its summaries are highly abstractive yet largely grounded in their sources, including in samples with conflicting reviews.

3.1 Dataset Construction

  • PEERSUM contains 14,993 samples from ICLR 2018–2022 and NeurIPS 2021–2022, split into 11,995 training, 1,499 validation, and 1,499 test samples.Samples pair a meta-review with source documents, including reviews, discussions, and the paper abstract; the authors note private discussions may also influence meta-reviews.Source: §3.1 · p. 3
  • Source documents form tree-like conversational structures, with an average tree height of 3.63 and width of 5.31.The dataset also includes paper acceptance outcome and, for each official review, a rating and confidence.Source: §3.1 · p. 3, §3.2 · p. 3

3.2 Conflicts in Source Documents

  • 13.6% of PEERSUM samples are CF, defined as having at least one pair of official reviews with a rating difference ≥4.Meta-reviews for these samples need to handle conflicting information or viewpoints.Source: §3.2 · p. 3

3.3 Abstractiveness and Faithfulness of Summaries

  • PEERSUM summaries are highly abstractive, particularly for bigrams and trigrams, although Multi-XScience is more abstractive.The authors note that Multi-XScience summaries are not always reflective of their source content.Source: §3.3 · p. 4, Table 5 · p. 4
  • Almost 80% of words in Non-CF meta-reviews and 72% in CF meta-reviews are grounded in source documents.The manual analysis covered 60 samples annotated by 10 volunteers.Source: §3.3 · p. 4, Table 6 · p. 5

4 The RAMMER Model

RAMMER extends an encoder-decoder pretrained language model for meta-review generation by encoding relationships among source documents and predicting metadata alongside text.

4 The RAMMER Model

  • RAMMER takes concatenated source documents with delimiters as input to an encoder-decoder pretrained language model that generates a meta-review.Source: §4 · p. 5

4.1 Relationship-Aware Sparse Attention

  • RSAttn restricts and weights attention using seven document relationships, including ancestor, descendant, sibling, self, and same-thread links.Its block-multiplication implementation computes attention only between related documents, reducing memory use for long inputs without substantially increasing computation complexity.Source: §4.1 · pp. 5–6

4.2 Multi-Task Learning

  • RAMMER jointly trains meta-review generation with auxiliary predictions of review ratings, review confidence, paper outcome, and source-document type.The objective balances generation and metadata-task losses with weights α.Source: §4.2 · p. 6

4.2.1 Encoder Auxiliary Tasks

  • Encoder delimiter embeddings support regression of official-review ratings and confidence and classification of each source document's type.Ratings and confidence are normalized to the [0,1] range for regression.Source: §4.2.1 · pp. 6–7

4.2.2 Decoder Auxiliary Tasks

  • The decoder's sole auxiliary objective predicts whether the paper was accepted or rejected from its output embeddings.Source: §4.2.2 · p. 7

5 Experiments

Experiments compare RAMMER with strong summarization baselines using automatic metrics and human judgments. RAMMER generally performs better, but models struggle to recognize and resolve conflicts.

5.1 Experimental Setup

  • The experiments compare large BART, PEGASUS, LED, PegasusX, and PRIMERA models, dividing each sample's input-length budget among its source documents.RAMMER's validation-tuned objective weights give greater weight to review confidence and paper acceptance outcome.Source: §5.1 · p. 7

5.2 Automatic Evaluation on Generated Meta-Reviews

  • Evaluation uses ROUGE, BERTScore, UniEval consistency and relevance, and ACC, which predicts paper outcome from generated meta-reviews as a reference-free consistency measure.Source: §5.2 · p. 7
  • Among RAMMER variants initialized with BART, PRIMERA, or LED, the LED variant performs best on validation and is selected as the backbone.Source: §5.2 · p. 8
  • Across test metrics, RAMMER consistently beats baselines except for near-tied CF ROUGE-L; CF scores trail Non-CF, especially on ACC and BERTScore.Source: §5.2 · p. 8, Table 8 · p. 8
  • Ablations leave the relative impact of RSAttn and multi-task learning unresolved across metrics, but combining them produces the best performance.ROUGE-L, BERTScore, and UniEval generally favor RSAttn, whereas ACC appears to favor the other component.Source: §5.2 · p. 8, Table 8 · p. 8

5.3 Human Evaluation on Conflict Recognition and Resolution

  • On conflict samples, RAMMER recognizes conflicts better than baselines but still misses them in most cases, while all models perform poorly at conflict resolution.Source: §5.3 · p. 9, Table 9 · p. 9

6 Conclusion

The paper introduces PeerSum, a dataset for meta-review generation, and RAMMER, which models inter-document relationships and metadata. Although RAMMER performs favorably against baselines, conflict recognition and resolution remain difficult.

6 Conclusion

  • PeerSum provides highly abstractive meta-reviews grounded in source documents, with metadata, explicit inter-document structure, and conflicting information to handle.Source: §6 · p. 9
  • RAMMER combines sparse attention for inter-document relationships with a multi-task objective for metadata, and performs favorably against most strong baselines.The attention approach could also be adapted to tasks whose inputs contain inter-document relationships.Source: §6 · p. 9
  • RAMMER often fails to recognize and resolve conflicting viewpoints in source documents, leaving conflict handling a challenging research problem.Source: §6 · p. 9

Limitations

The paper identifies limits in its human assessment, model comparisons, and treatment of source-document structure and conflicts.

Limitations

  • The human assessment is simple and could use more objective subtasks, such as linking meta-review statements to source sentences.The authors note that most meta-review content is grounded in source documents, while acknowledging that writing also involves judging opinions.Source: Limitations · p. 9
  • The experiments omit comparisons with larger models, and the authors describe their results as preliminary.They suggest exploring how RAMMER’s RSAttn could be adapted for large autoregressive models.Source: Limitations · p. 9
  • The study considers explicit conversational structure only; it helps recognize conflicts to some degree but does not resolve them.The authors leave implicit structures, such as argument and discourse links, for future exploration.Source: Limitations · p. 9

A Appendix: Different types of source documents in PEERSUM

The appendix presents a randomly selected, anonymized PeerSum example to illustrate how meta-reviews draw on peer-review materials and discussions.

A Appendix: Different types of source documents in PEERSUM

B Appendix: Hyper-parameters for fine-tuning pre-trained text summarization models

The appendix points readers to Table 10 for the experiment models’ hyper-parameters.

B Appendix: Hyper-parameters for fine-tuning pre-trained text summarization models

C Appendix: Instructions for annotation of PEERSUM quality

The appendix instructs annotators to identify meta-review assertions drawn solely from PeerSum source documents and illustrates how source-grounded content differs from outside knowledge or judgment.

C Appendix: Instructions for annotation of PEERSUM quality

  • The annotation task asks whether meta-review assertions are drawn exclusively from reviews, discussions, and the paper abstract.The instructions direct annotators to highlight assertions based solely on those source documents.Source: C Appendix: Instructions for… · p. 13
  • Assertions grounded in source documents are highlighted, even when they contain light meta-reviewer judgment.The instructions contrast these with assertions relying heavily on the reviewer’s own knowledge, full-paper reading, or other external knowledge.Source: C Appendix: Instructions for… · p. 13
  • Annotators are asked to read the source documents and the corresponding meta-review before marking source-grounded statements.The instructions describe the source materials as the abstract, reviews, and author-reviewer discussions and responses.Source: C Appendix: Instructions for… · p. 13
  • The examples exclude meta-reviewer suggestions about impact and judgments about whether a question was addressed as not source-grounded.One example treats a repository suggestion and an assessment of the paper’s unanswered question as outside the source documents.Source: C Appendix: Instructions for… · p. 14
  • The area-chair example includes judgments based on field experience and full-paper reading, alongside claims about reviewer positions.The example’s annotations distinguish those outside-source judgments from assertions such as reviewers being inclined to reject.Source: C Appendix: Instructions for… · pp. 14–16

D Appendix: Generated meta-reviews for PEERSUM by different models

The appendix compares generated meta-reviews from PRIMERA, LED, PegasusX, and RAMMER on random PEERSUM test examples. The examples show that RAMMER’s benchmark advantage does not eliminate output-quality problems.

D Appendix: Generated meta-reviews for PEERSUM by different models

  • RAMMER outperforms strong baselines on evaluation metrics, yet its sampled meta-reviews still need improvement, reinforcing that PEERSUM is challenging.The table caption identifies the samples as random and says ROUGE scores compare generated outputs with ground truth.Source: D Appendix: Generated meta-reviews for… · p. 16, Table 11 · p. 24
  • For the neural-network analysis paper, RAMMER describes entropy as negatively correlated with training accuracy and modularity as correlated, while scalability and practicality remain concerns.Other sampled reviews question comparison with existing methods, experimental validation, and generalization beyond small or simple settings.Source: D Appendix: Generated meta-reviews for… · pp. 17–18
  • For the batch-size-invariance paper, sampled reviews describe decoupling proximal and behavior policies, alongside concerns about theoretical justification and evaluation.The examples report that reviewers did not reach consensus and called for stronger analysis of ablations.Source: D Appendix: Generated meta-reviews for… · pp. 19–20
  • For the facial-privacy paper, sampled reviews describe poisoning defenses as vulnerable to future recognition systems, but reviewers differ on the paper’s contribution and acceptance.The RAMMER example says the paper concludes that legislation, rather than technical solutions, is needed to protect privacy.Source: D Appendix: Generated meta-reviews for… · pp. 20–22
  • For the time-series augmentation paper, PRIMERA reports improvements across datasets, while other sampled reviews question the results’ adequacy or recommend revision.The reviews differ in their assessments of the approach and experimental evidence.Source: D Appendix: Generated meta-reviews for… · pp. 22–23
  • For the all-layer-margin paper, reviews describe generalization bounds and a margin-maximizing algorithm, while retaining concerns about notation, residual networks, and remaining reviewer comments.The PRIMERA example also says the work extends its bounds to robust classifiers.Source: D Appendix: Generated meta-reviews for… · pp. 23–24

Related papers

Loading 2305.01498v4…