LLMs as Meta-Reviewers' Assistants: A Case Study

1 Introduction

• Meta-reviewing gathers expert opinions, synthesizes shared strengths, weaknesses, and suggestions, then adds the meta-reviewer’s judgment and final recommendation.The paper frames its task as assisting the human meta-reviewer with summaries, rather than generating meta-reviews.Source: §1 · p. 1
• Meta-reviewing is arduous and time-consuming, and rising submission volume adds to the workload and may increase susceptibility to inconsistency or missed details.The authors study whether LLM-generated multi-perspective summaries can assist meta-reviewers in working more consistently and efficiently.Source: §1 · p. 1
• The case study examines 40 ICLR papers using three LLMs and human evaluation of their generated summaries against five criteria.The evaluation included 480 distinct LLM-generated summaries and a GPT-4-based assessment whose correlation with human judgments was also examined.Source: §1 · pp. 1–2
• GPT-3.5 and PaLM2 were rated higher than LLaMA2 by humans; PaLM2 generally had better recall, while GPT-3.5 generally had better precision.Humans rated GPT-3.5 poorly at the LLM level, and GPT-4 evaluations correlated poorly with human judgments of predecessor-generated summaries.Source: §1 · p. 2

2 Related Work

• Earlier meta-review generation work used rule-based and neural methods, encoder-decoder methods, and more recently LLM-based approaches.The cited work includes Bhatia et al. (2020) and Pradhan et al. (2021) for rule-based and neural techniques, and Kumar et al. (2021) and Li et al. (2023) for encoder-decoder methods.Source: §2 · p. 2
• LLM performance on multi-constraint, multi-perspective tasks such as meta-review composition remains understudied, as does systematic comparison using a standard prompting taxonomy.The authors identify these as two gaps motivating their case study.Source: §2 · p. 2
• The study applies TELeR, a prompting taxonomy, to compare three LLMs on meta-review composition and report prompt categories used.The authors state that using the taxonomy enabled more meaningful comparisons and helped derive more accurate conclusions about relative performance.Source: §2 · p. 3

3.1 Data-set Details

• The dataset contains 40 papers selected from ICLR submissions from 2020–2023, with detailed reviews and meta-reviews; 10 were rejected papers.Selection favored detailed meta-reviews, at least three reviews addressing different aspects, and, especially, recent papers from 2021–2023.Source: §3.1 · p. 3

3.2 Large Language Models

• The study compares GPT-3.5, PaLM2, and LLaMA2 in zero-shot settings without further fine-tuning.GPT-3.5 and LLaMA2 were accessed through APIs; PaLM2 was used manually through a web browser because it had no API.Source: §3.2 · p. 3

3.3 Prompt Design using TELeR Taxonomy

• TELeR prompt levels progress from a general instruction to multi-part instructions, then bulleted subtasks, with level 4 also asking models to explain their outputs.At level 2 and above, subtasks cover core contributions, common strengths and weaknesses, improvement suggestions, and optionally missing references.Source: §3.3 · p. 3

3.4 Evaluation Criteria and Process

• Ten NLP researchers evaluated the generated summaries against five criteria, using five-point agreement ratings and anonymized model identities.Each annotator assessed summaries for four papers using publicly available reviews; criteria included core contributions, common strengths and weaknesses, improvement tips, and literature review.Source: §3.4 · pp. 3–4
• The evaluation also assessed each LLM’s overall performance on instruction adherence, summary usefulness, and similarity to expert-written meta-reviews.This macro-evaluation used five-point agreement ratings and included an essay question about model performance.Source: §3.4 · p. 4

4.1 Results from Micro-Evaluation

• Level 3 and 4 prompts were more effective overall than Levels 1 and 2; common-strength recall peaked at Level 3, while literature-review ratings showed little gain beyond Level 2.For core contributions, GPT-3.5 rated best at Levels 2–4, while PaLM2 rated best at Level 1.Source: §4.1 · p. 5
• Human ratings favored LLMs for core contributions, common strengths, and literature review, but were mixed for common weaknesses and suggestions.The overall core-contributions ratings also showed humans voting in favor of LLMs more often than otherwise.Source: §4.1 · pp. 5–6
• Humans rated GPT-3.5 and PaLM2 above LLaMA2 across all five micro-evaluation aspects, with PaLM2 generally stronger in recall and GPT-3.5 in precision.Exceptions were common strengths for recall, and common weaknesses and literature review for precision.Source: Table 1 · p. 7, §4.1 · p. 7

4.2 Results from Macro-Evaluation

• PaLM2 was the best performer and GPT-3.5 the worst across all three macro-evaluation aspects.Annotators suggested that fine-grained micro-evaluation judgments may be more reliable; why GPT-3.5 rated highly in micro-evaluation but poorly in macro-evaluation remains open.Source: Table 2 · p. 7, §4.2 · p. 7

4.3 Automatic Evaluation with GPT-4o

• GPT-4o was not a reliable judge of MPS quality: it correlated poorly with human ratings, often scored GPT-3.5 summaries higher, and showed no significant correlation for GPT-3.5 macro-evaluations.For macro-evaluation, GPT-4o correlated better with human judgments for LLaMA2, particularly on adherence and usefulness.Source: §4.3 · pp. 7–9

5 Conclusion

• GPT-3.5 and PaLM2 were rated higher than LLaMA2 across all five aspects, with comparable average performance but stronger recall for PaLM2 and stronger precision for GPT-3.5.Although micro-evaluation found GPT-3.5 and PaLM2 comparable, macro-evaluation favored PaLM2.Source: §5 · p. 9
• GPT-4 evaluations correlated poorly with human evaluations, making GPT-4 unreliable for assessing this facet-aware MPS task.Source: §5 · p. 9
• LLM reliability varied across evaluation criteria, models, and prompt types in the study.Source: §5 · p. 9

6 Limitations

• The study tested only three LLMs and 40 manuscripts, limiting the basis for general conclusions about LLM performance on meta-review composition.Source: §6 · p. 9
• Annotation took an estimated four hours per paper, totaling 160 hours across ten annotators, apart from other study work.Source: §6 · p. 9
• Prompt-sensitivity findings received limited main-paper discussion; performance improved substantially from level 1 to level 2, but not always from levels 2 to 3 or 4.Source: §6 · p. 9
• Annotators read abstracts, reviews, comments, and meta-reviews but not the original papers, and therefore assumed reviewers’ comments were correct and justified.Source: §6 · p. 9
• The authors note that biases and inconsistencies may also affect their NLP-researcher annotators’ evaluations of meta-reviews.Source: §6 · p. 9
• GPT-4 evaluations correlated poorly with human judgments, and time and budget constraints prevented another human evaluation of newer models.Source: §6 · p. 9

7 Ethics Statement

• The stated goal is to help meta-reviewers understand reviewers’ consensus, not to have an LLM write the meta-review independently.Source: §7 · p. 10
• Meta-reviewers must verify summary claims against the paper before manually writing the meta-review.Source: §7 · p. 10
• Meta-reviewing can also require resolving reviewer conflicts and adding expert judgment, beyond summarizing reviews.Source: §7 · p. 10

Alexander Goldberg, Ivan Stelmakh, Kyunghyun Cho,

• The references include work on meta-review generation using neural architectures and conversational multi-document summarization.Source: Alexander Goldberg, Ivan Stelmakh,… · p. 11

Tribikram Pradhan, Chaitanya Bhatia, Prashant Kumar,

• Other listed topics include a meta-review dataset, evaluation metrics, and prompt sensitivity and taxonomy.Source: Tribikram Pradhan, Chaitanya Bhatia,… · p. 11
• The bibliography lists research on language models, including LaMDA, LLaMA, and LLaMA 2.Source: Tribikram Pradhan, Chaitanya Bhatia,… · pp. 11–12

A.1 Background on TELeR Taxonomy

• Turn classifies prompts by the number of shots, distinguishing single-turn from multi-turn prompting.The passage describes turn as the number of turns or shots used to accomplish a complex task.Source: §A.1 · p. 12
• Expression distinguishes interaction styles such as questioning and instructing.Source: §A.1 · p. 12
• Level of detail ranges from no directive at Level 0 to granular Level 4 prompts specifying subtasks, examples, or evaluation criteria.Source: §A.1 · p. 12
• Role captures how specifying an LLM’s system role can change its response even when prompt content stays unchanged.Source: §A.1 · p. 12

A.2 Detailed Plots for Five Criteria

• The plots compare three LLMs’ human-perceived precision and recall distributions across four criteria.The criteria are Common Strengths, Common Weaknesses, Common Suggestions, and Literature Review.Source: §A.2 · p. 12
• The distributions are derived from qualitative rating counts across Strongly Agree, Agree, Neutral, Disagree, and Strongly Disagree.Source: §A.2 · p. 12
• A normalization term converts the rating-derived values into valid distributions with individual values from 0 to 1.Source: §A.2 · p. 12

A.3 Prompt Design Details

• Table 4 presents prompt designs used during the preliminary study.Source: §A.3 · p. 12
• Table 4 also presents prompt designs used during the final case study.Source: §A.3 · p. 12
• Tables 5 and 6 present prompt designs for automatic evaluation.Source: §A.3 · p. 12

A.4 Responsible NLP Checklist

• The annotation task involved eight graduate students and two undergraduates, paid at university-prescribed rates or $15 per hour, respectively.Source: §A.4 · p. 12
• Detailed instructions for micro- and macro-evaluation are presented in Figures 13, 14, and 15.Source: §A.4 · p. 12
• The study reports no collected annotator behavior data and says survey data contained no demographic or identifying information.Human-subject data were reviewed for confidentiality and privacy before manuscript preparation.Source: §A.4 · p. 12
• The authors characterize the work as having no potential risk and being a curiosity-driven academic study.Source: §A.4 · p. 12
• The authors plan to use a Creative Commons Attribution license that permits reuse and adaptation with credit to the NSF-funded project.Source: §A.4 · p. 13

• Three reviews were used for the Preliminary Study, with their full texts following.Source: §A.5 · p. 14

A.5.1 Review 1

• Review 1 found the approach novel and theoretically grounded and leaned toward acceptance, but questioned its Olympic-dataset samples, unclear debiasing choice, and qualitative-only denoising and inpainting evaluation.The reviewer specifically questioned whether samples beat MADE and noted that intersections between rings appeared less well modeled.Source: §A.5.1 · pp. 16–18

A.5.2 Review 2

• Review 2 praised Gaussian smoothing and posterior modeling for making the data distribution easier to capture, while requesting analysis of optimal noise, alternatives to Gaussian noise, and further theoretical clarification.The reviewer later raised their score and recommended acceptance, while still noting gaps in analysis of sigma, smoothing distributions, and the Lipschitz constant.Source: §A.5.2 · pp. 20, 22

A.5.3 Review 3

• Review 3 saw improved sample quality from a two-stage method with one denoising step, but requested deeper theoretical and related-work discussion, flow experiments, and stronger sample-quality evaluation.The reviewer noted that the best-performing method doubles sampling time for already slow autoregressive models and questioned the claimed capacity limitation.Source: §A.5.3 · pp. 23–24

CONCLUSION

• The conclusion favors acceptance, and the reviewer reports raising the score to 7 after considering the response and updated manuscript.The reviewer says the updated version was clearly improved and agrees with the other reviewers about the paper’s quality.Source: §A.5.2 · p. 22, CONCLUSION · p. 25

A.6 Preliminary Study with GPT-3.5’s and TELeR Taxonomy Based Prompting

• GPT-3.5 captured several shared strengths and weaknesses, but its coverage scored 4/10 and suggestions scored 4/10; a second response recommended acceptance with minor revisions.The evaluation also rated common strengths and weaknesses 6/10 each, and coverage 4/10.Source: §A.6 · p. 26
• High-level prompts differed: the system-role response omitted missing references and answered only four of five questions, whereas the user-role response mentioned missing references incorrectly but answered all questions.System-role strengths and coverage scored 6/10 and 7/10; user-role strengths and coverage each scored 8/10.Source: §A.6 · pp. 27–28
• High-level question prompts yielded brief or insufficiently comprehensive summaries, with missing references omitted and coverage judged poor.For both variants, strengths and weaknesses were rated 3/10, suggestions 6/10, and coverage 5/10.Source: §A.6 · pp. 28–29
• The system-role question prompt scored 10/10 for adherence, 8/10 for strengths, 9/10 for weaknesses, 8/10 for suggestions, and 9/10 for coverage.It covered all three reviews and answered all questions, but its missing-reference answer was inaccurate and it omitted the manifold hypothesis.Source: §A.6 · p. 30
• The system-role question-paragraph response scored 10/10 for adherence and coverage, but 6/10 for strengths and 7/10 for weaknesses and suggestions.The evaluation said it considered all reviews, while its strengths content was inadequate and its weaknesses response was stronger.Source: §A.6 · p. 31
• The user-role question-paragraph response scored 10/10 for adherence and coverage, 8/10 for strengths, and 9/10 for weaknesses and suggestions.The evaluation found that it considered all three reviews and that its content was adequate.Source: §A.6 · p. 32
• The system-role bulleted-instruction response followed instructions but was weak on strengths and weaknesses, duplicated those categories in its suggestions, and gave an incorrect missing-reference answer.Its evaluation also noted that the strengths and weaknesses lacked content and that weaknesses and suggestions were almost the same.Source: §A.6 · p. 33

A.5.1 Review 1

• Across L3 assessments, strengths were often judged correct but inadequate, while weaknesses and suggestions varied in quality and sometimes overlapped.One analysis found suggestions correct but insufficient; another judged weaknesses better than strengths, and another said the weaknesses included points from suggestions.Source: §A.6 · pp. 34, 35, 39
• L3 prompt scores varied: adherence was 10/10 throughout, coverage ranged from 8/10 to 10/10, and suggestions ranged from 5/10 to 8/10.The three score sets were (10/10, 5/10, 5/10, 7/10, 8/10), (10/10, 6/10, 7/10, 5/10, 8/10), and (10/10, 6/10, 7/10, 8/10, 10/10), ordered as adherence, strengths, weaknesses, suggestions, and coverage.Source: §A.6 · pp. 34–35
• 10/10 scores for strengths, weaknesses, suggestions, and coverage contrasted with 8/10 adherence for the L4 system prompt requesting supporting evidence.The analysis said the response did not explicitly provide support or explanations and included process details in its summary of contributions.Source: §A.6 · p. 38
• For GPT-3.5’s response, section scores were 8/10 for instruction adherence and common strengths, 9/10 for common weaknesses and improvement suggestions, and 10/10 for coverage.The analysis noted that explanations were not explicitly provided in any of the answers.Source: §A.6 · p. 40
• The review characterized the two-stage denoising approach as improving sample quality over direct distribution fitting, citing strong theoretical grounding, empirical evaluation, clear presentation, and a well-motivated, straightforward method.It framed the method as addressing mismatched conditionals in autoregressive models and the manifold hypothesis.Source: §A.6 · p. 41
• The review called for analysis of optimal randomized-smoothing noise, fuller discussion of Theorem 2’s Laplacian term, normalizing-flow experiments, clearer terminology, and expanded related work, including Song et al. (2020).It noted that “Denoising Diffusion Probabilistic Models” by Song et al. (2020) could be relevant to denoising methods in generative modeling.Source: §A.6 · p. 41
• GPT-3.5 was judged to have followed almost all instructions and covered all reviews, scoring 8/10 for adherence and weaknesses and 10/10 for strengths, suggestions, and coverage.The analysis said the weaknesses were less good than those elicited by system-content questions and overlapped with improvement suggestions.Source: §A.6 · p. 41