PaperlayerView exact revision on arXiv

Source-linked AI summary

A Sentiment Consolidation Framework for Meta-Review Generation

Miao Li, Jey Han Lau, Eduard Hovy

arXiv:2402.18005v2cs.CLcs.AI

TL;DR

LLMs can generate plausible multi-document summaries, but it remains uncertain whether they consolidate opinionated information accurately. This paper proposes a three-layer sentiment-consolidation framework for meta-review generation and finds that framework-based prompting produces better meta-reviews.

  • Problem

    Whether LLMs genuinely consolidate opinionated information across documents, rather than generating plausible summaries with incorrect overall sentiment, remains uncertain.

    Source: §1 · p. 1

  • Method

    The paper proposes a three-layer sentiment-consolidation framework for meta-review generation and prompting methods that integrate its process.

    Source: §1 · p. 2, §5.1 · p. 6

  • Results

    Prompting LLMs to follow the framework's processes results in better meta-reviews.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Experiments empirically validate the framework as a description of the meta-review writing process.

    Source: §6 · p. 9

    The paper focuses on sentiment prediction; future work must consider argumentation in source reviews and justification in meta-reviews.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Modern natural language generation systems with Large Language Models (LLMs) exhibit the capability to generate a plausible summary of multiple documents; however, it is uncertain if they truly possess the capability of information consolidation to generate summaries, especially on documents with opinionated information. We focus on meta-review generation, a form of sentiment summarisation for the scientific domain. To make scientific sentiment summarization more grounded, we hypothesize that human meta-reviewers follow a three-layer framework of sentiment consolidation to write meta-reviews. Based on the framework, we propose novel prompting methods for LLMs to generate meta-reviews and evaluation metrics to assess the quality of generated meta-reviews. Our framework is validated empirically as we find that prompting LLMs based on the framework -- compared with prompting them with simple instructions -- generates better meta-reviews.

1 Introduction

The introduction frames meta-review generation as scientific sentiment summarization, where correct consolidation requires more than extracting majority sentiment. It proposes a three-layer framework, annotation resources, and sentiment-focused evaluation metrics, and reports that framework-based prompts improve generated meta-reviews.

1 Introduction

  • Scientific sentiment summarization uses peer-review meta-reviews as summaries, addressing a domain underexplored by product-review datasets.Those datasets often use extractive summaries and majority-sentiment consolidation, unlike the complex judgements involved in meta-reviewing.Source: §1 · p. 1
  • Meta-reviewers must account for multi-turn discussions, argument logic, and conflicts or consensus rather than rely on majority voting.Their comments support manuscript acceptance decisions.Source: §1 · p. 2
  • The proposed framework hypothesizes that human meta-reviewers follow three layers of sentiment consolidation when writing meta-reviews.The paper collects human annotations of meta-reviews and source documents based on the framework.Source: §1 · p. 2
  • Framework-based prompting generates better meta-reviews than simple instructions, validating the proposed sentiment-consolidation process.The paper also proposes reference-free and reference-based metrics for assessing sentiment in generated meta-reviews.Source: §1 · p. 2

2 Related Work

Related work covers information consolidation in abstractive summarization and automated sentiment summarization. The paper situates meta-review generation as a scientific sentiment-summarization task and focuses on facet-level sentiment fusion.

2.1 Large-Scale Information Consolidation

  • Prior summarization work emphasizes generation quality, while whether models truly consolidate information across source documents remains uncertain.One cited approach uses heterogeneous graphs and graph compression to improve generated summaries.Source: §2.1 · p. 2

2.2 Automated Sentiment Summarization

  • Meta-review generation is commonly modeled end-to-end, but prior models do not explain how human meta-reviewers write meta-reviews.This work instead uses a three-layer sentiment-consolidation framework focused on review facets and sentiment fusion.Source: §2.2 · p. 2

3 Sentiment Consolidation Over Multiple Opinionated Documents

This section frames meta-review generation as scientific sentiment summarization and models it as staged consolidation of reviewer judgements. Its analyses find non-majority decisions and suggest judgement extraction may help recover facet-level meta-review sentiment.

3.1 Hierarchical Sentiment Consolidation

  • The three-layer framework moves from reviews and multi-turn discussions, through extracting, facet-organizing, and consolidating judgements, to writing meta-reviews.A judgement combines sentiment on a review facet with components including content expression, sentiment expression, sentiment level, and convincingness level.Source: §3.1 · p. 3, §3.2 · p. 3

3.2 Judgement Identification and Extraction

  • Human annotators show generally moderate-to-high agreement, whereas GPT-4 agreement is moderate for meta-reviews and official reviews but low for discussions.GPT-4 also has poor agreement on convincingness, for which human agreement is not strong.Source: §3.2 · p. 4, Figure 2 · p. 4, Figure 3 · p. 4

3.3 Sentiment Fusion for Consolidation

  • Soundness and Advancement are the most important review facets in meta-reviews, while Compliance is rarely an issue.The reported facet frequencies appear in Table 3.Source: §3.3 · p. 4, Table 3 · p. 4
  • 23.7% of PeerSum samples have an acceptance decision inconsistent with the prediction from majority voting on review ratings.A sample is counted as consistent when reviews rated ≥5 outnumber those rated <5 and the final decision is Accept.Source: §3.3 · p. 4, Table 4 · p. 5
  • Using annotated judgements rather than full source texts, GPT-4 predicts meta-review sentiment better on 4 of 6 facets.The comparison uses 100 randomly sampled human-annotated instances.Source: §3.3 · p. 5, Table 5 · p. 5

4 Sentiment-Aware Evaluation on Information Consolidation

The section introduces two sentiment-focused metrics for meta-reviews: one compares generated and human-written reviews, while the other checks generated sentiments against predictions from source documents. Together they assess sentiment consistency across review facets and source materials.

4.1 Measuring Sentiment Similarity to Human-Written Meta-Review

  • FacetEval measures sentiment consistency between generated and human-written meta-reviews across review facets, rather than relying mainly on surface-form matching.It represents facet sentiments and compares the resulting vectors with cosine similarity.Source: §4.1 · p. 5
  • Official reviews show higher sentiment consistency for Compliance than for Clarity and Novelty; conflict reviews are associated with conflicts in Advancement, Novelty, Clarity, and Overall.The reported comparison of facet consistency is in Table 6.Source: §4.1 · p. 6, Table 6 · p. 6

4.2 Measuring Sentiment Fusion for Individual Facets

  • FusionEval compares generated-review sentiments with GPT-4 zero-shot predictions from the source reviews and discussions, using precision because the number of judgements to synthesize is unknown.It evaluates sentiment levels for extracted expressions against predictions based on source-document judgements for the same facet.Source: §4.2 · p. 6

5 Enhancing LLMs with Explicit Information Consolidation

The section tests prompts that incorporate a five-step sentiment-consolidation process against simpler and LLM-generated-step baselines. Across most LLMs, the proposed prompts perform better, with different approaches favored by text-overlap and sentiment metrics.

5.1 Prompting LLMs with Sentiment Consolidation Logic

  • The proposed process extracts judgements, assigns facets, sentiments, and convincingness, clusters by facet, creates facet-level mini-summaries, and generates a final meta-review.Prompt-Ours gives all five steps in one prompt; Pipeline-Ours uses separate sequential prompts.Source: §5.1 · p. 6

5.3 Reference-Based and Reference-Free Automatic Evaluation

  • Most LLMs perform better with Prompt-Ours or Pipeline-Ours than with Prompt-Naive or Prompt-LLM; Prompt-Ours favors ROUGE, while Pipeline-Ours is marginally better on FusionEval and FacetEval.GPT-4 generally performs best on the sentiment metrics.Source: Table 7 · p. 7, §5.3 · p. 7

5.2 Baselines

  • Experiments compare the proposed prompts with Prompt-Naive's simple instruction and Prompt-LLM's LLM-generated steps included in the final prompt.The experiments cover GPT-4, GPT-3.5, LLaMA2-70B, and LLaMA2-7B on 500 PeerSum samples.Source: §5.1 · p. 6, §5 · p. 6, §5.2 · p. 7

5.3 Reference-Based and Reference-Free Automatic Evaluation

  • Automatic evaluation uses ROUGE, FacetEval, and FusionEval; the paper excludes other metrics because they had only been validated for news or general-domain summarization.The Table 7 comparison reports the different prompting methods across LLMs.Source: §5.3 · p. 7, Table 7 · p. 7

5.4 Reference-Free Human Evaluation

  • For Prompt-Naive versus Prompt-Ours using LLaMA2-70B, annotators selected Prompt-Ours-generated meta-reviews more often in a comparison based on 30 randomly selected samples.In a separate comparison, annotators much more preferred Prompt-Ours (GPT-4) to human-written meta-reviews.Source: §5.4 · p. 8

5.5 Case Study on Generated Meta-Reviews

  • Generated meta-reviews are plausible but often longer than human-written ones, with details that can be unnecessary or redundant.In the example, Prompt-Ours includes both strengths and weaknesses for Clarity, unlike Prompt-Naive; this accords with its better sentiment results.Source: §5.5 · p. 9

6 Conclusions and Future Work

The paper studies sentiment-focused information consolidation for scientific summarization through meta-review generation. It concludes that framework-based prompting produces better meta-reviews in experiments, while adapting the framework to other domains remains future work.

6 Conclusions and Future Work

  • The paper introduces a three-layer sentiment-consolidation framework for generating meta-reviews that account for sentiments on review facets in reviews and discussions.It also proposes automatic metrics for evaluating overall sentiment in generated meta-reviews.Source: §6 · p. 9
  • Experiments show that prompting LLMs through the framework's processes results in better meta-reviews, empirically validating it as a description of meta-review writing.Source: §6 · p. 9
  • Adapting the framework to domains such as politics and advertising is identified as future work.Source: §6 · p. 9

Limitations

The authors identify limitations in data coverage, language, model adaptation, and the scope of meta-review generation. They also note that GPT-4’s sentiment prediction and failure cases require further understanding.

Limitations

  • Experiments use peer-review data from only some AI conferences, which may bias the models.The authors hope to include data from more diverse areas.Source: Limitations · p. 9
  • The experiments cover English texts only, not other languages.Source: Limitations · p. 9
  • The framework is applied to prompting-based models, not fine-tuned models.The authors plan to investigate its use with fine-tuned models.Source: Limitations · p. 9
  • The authors say more work is needed to understand how GPT-4 predicts meta-review sentiments from source judgments and what causes its errors.Source: Limitations · p. 9
  • Future work should consider argumentation in source reviews and justification in meta-reviews, beyond sentiment prediction.Source: Limitations · p. 9

Ethics Statement

The authors report that generated meta-reviews can be satisfactory to a degree, but warn against relying on them without human verification because hallucinations occur. They do not advocate replacing human meta-reviewers, while noting that models may make the process more efficient and effective.

Ethics Statement

  • Generated meta-reviews can be satisfactory to a degree, but hallucinations mean they should not be used without manual verification and review.Source: Ethics Statement · p. 9
  • The authors do not advocate replacing human meta-reviewers with LLMs.Source: Ethics Statement · p. 9
  • The authors state that LLMs have the capacity to make the meta-reviewing process more efficient and effective.Source: Ethics Statement · p. 9

A Review Criteria in Different Reviewer Guidelines

The appendix includes review guidelines from different academic presses and screenshots of the human-annotation instructions.

A Review Criteria in Different Reviewer Guidelines

  • Table 10 presents review guidelines from different academic presses.Source: Table 10 · p. 11

B Annotation Instructions for Human Annotation

C Inter-Annotator Agreement Among Human Annotators and GPT-4

The paper measures agreement differently for highlighted text spans and categorical annotation labels. It reports human annotator agreement and GPT-4 agreement with human annotators in separate tables.

C Inter-Annotator Agreement Among Human Annotators and GPT-4

  • For Content Expression and Sentiment Expression, agreement uses character-level Krippendorff’s α and Cohen’s κ on highlighted spans.The agreement indicates whether annotators identify sentiments from similar text spans.Source: C Inter-Annotator Agreement Among… · p. 11
  • For Review Facet, Sentiment Level, and Convincingness Level, annotators first match sentiment spans using a combined ROUGE-1, ROUGE-2, and ROUGE-L threshold above 2.0, then compare predicted values.Source: C Inter-Annotator Agreement Among… · p. 11
  • Tables 11–13 report agreement between human annotators, while Tables 14–16 report GPT-4’s average agreement with the two human annotators.Source: C Inter-Annotator Agreement Among… · p. 11

D Prompt to Get Content and Sentiment Expressions with GPT-4

The appendix prompts GPT-4 to identify manuscript-quality judgments, extract their content and sentiment expressions, and predict judgment components. It also contrasts prediction from extracted judgments with prediction from full source documents.

D Prompt to Get Content and Sentiment Expressions with GPT-4

  • The appendix includes human annotator agreement tables for meta-reviews, official reviews, and discussions, plus GPT-4 agreement with two annotators for each source type.The supplied captions identify the tables’ scopes but do not provide agreement values.Source: Table 11 · p. 11, Table 12 · p. 12, Table 13 · p. 12
  • The prompt first identifies sentences judging manuscript quality across six review facets, then extracts each sentence’s content expression and sentiment expression.The requested output pairs the judgment sentence with the content it concerns and the phrase expressing sentiment.Source: D Prompt to Get Content and Sentiment… · p. 13
  • Examples show the format separating a judgment sentence from its content and sentiment expressions.Examples include judgments about writing quality, experimental support, and novelty.Source: D Prompt to Get Content and Sentiment… · p. 13

E Prompt to Get Judgement Component Predictions with GPT-4

  • The facet prompt defines six categories: novelty, soundness, clarity, advancement, compliance, and overall manuscript quality.The prompt supplies descriptions of what each facet covers.Source: E Prompt to Get Judgement Component… · p. 14
  • The component-prediction prompt asks for the judgment’s facet, original sentiment expresser, justification strength, and sentiment polarity and strength.The listed expressers are Self and Others; convincingness ranges from Not at all to Highly Convincing, with Not applicable for explicitly quoted sentiment.Source: E Prompt to Get Judgement Component… · pp. 14–15

F.1 Prediction with Judgements of Source documents

  • For prediction from source judgments, the prompt uses extracted judgments in the same review facet as the target meta-review judgment.It asks for an implicit meta-review and a predicted sentiment level based on those judgments.Source: §F.1 · p. 15

F.2 Prediction with Full Texts of Source documents

  • For full-text prediction, the source texts are concatenated, and the prompt predicts a sentiment level for a supplied content expression using related document information.The output pairs the content expression with its predicted sentiment level.Source: §F.2 · p. 16

G Prompts for Meta-Review Generation with Integration of Information Consolidation Logic

The appendix contrasts descriptive consolidation instructions with simpler prompts and a prompt-generated-steps approach. It also shows the input and annotation context used for the Prompt-LLM approach.

G.1 Prompt with Descriptive Consolidation Logic

  • The descriptive prompt asks the model to follow underlying steps to generate better meta-reviews from reviews and discussions.The prompt introduces the task before referring to those steps.Source: §G.1 · p. 16

G.2 Prompts Used in the Pipeline Generation

  • The pipeline prompts extract content and sentiment expressions, predict other judgment components, summarize judgments within each facet, then combine facet summaries into a meta-review.The first two prompts match those in Appendices D and E.Source: §G.2 · p. 17

G.3 Prompts from Prompt-Naive

  • Prompt-Naive asks directly for a meta-review based on the reviews and discussions.Source: §G.3 · p. 17

G.4 Prompts from Prompt-LLM

  • Prompt-LLM first generates steps with GPT-4, then asks for a meta-review following those steps.The generated steps are inserted into the second prompt.Source: §G.4 · p. 18
  • The annotation context treats a meta-review as the final summary and judgment based on reviewers’ comments and author-reviewer discussions.Annotators are instructed to ignore comments added after the meta-review is released.Source: §G.4 · pp. 19–20
  • The paper organizes peer-review judgments by criteria facets and annotates each judgment’s sentiment and, sometimes, its justification.Tables 1 and 2 are identified as presenting the facet typology and fine-grained annotation aspects.Source: §G.4 · p. 19
  • The appendix includes a sample review passage with a criticism of a method’s novelty and supporting questions about its components.Source: §G.4 · p. 20

Related papers

Loading 2402.18005v2…