Reviewer2: Optimizing Review Generation Through Prompt Generation

• Open-ended review generation leaves the focus unspecified, while standard fine-tuning can produce generic reviews with reduced specificity that fail to cover the range of human-identified aspects.Source: §1 · pp. 1–2
• Reviewer2 uses one fine-tuned model to generate aspect prompts from a paper and a second to generate a review conditioned on the paper and prompt.Source: §1 · p. 2
• PGE generates and self-evaluates aspect prompts from reviews, supporting a dataset of 27k papers and 99k reviews from six venues.Source: §1 · p. 2
• Across multiple venues, Reviewer2 trained with PGE-generated prompts substantially outperforms existing methods in review quality, specificity, and coverage.Source: §1 · p. 2
• The work also introduces gLoRA for 32k-context, low-memory fine-tuning and metrics for specificity and coverability.Source: §1 · p. 2

• Instruction-tuning approaches are described as improving task performance and adaptability, but depend heavily on human-written instruction data constrained in quantity and diversity (Wei et al., 2022; Sanh et al., 2022; Ouyang et al., 2022).Source: §2 · p. 2
• Self-alignment work includes unsupervised data generation and post-hoc output refinement, such as principle-based responses, synthetic prompts, and iterative feedback (Li et al., 2023; Sun et al., 2023; Madaan et al., 2023).Source: §2 · p. 2
• Automated peer-review systems address reviewer matching, plagiarism detection, desk-rejection prediction, and review generation from papers (Stelmakh et al., 2019; Kobren et al., 2019; Cohan et al., 2020; Yuan et al., 2021; Lin et al., 2023).Source: §2 · pp. 2–3

• Reviewer2’s explicit aspect control is intended to improve review coverage and specificity.Source: §3 · p. 3
• At inference, one model generates an aspect prompt for a new paper, and a second generates a review conditioned on that paper and prompt.Source: §3 · p. 3
• Aspect prompts diversify the issues addressed to better reflect variability across human reviews; the paper explains that this reduces noise in mapping prompts to reviews and enables more specific outputs.Source: §3 · p. 3
• For long-context fine-tuning and inference, the implementation adapts LoRA+ and S2-Attn; S2-Attn groups input tokens to address self-attention’s quadratic complexity.Source: §3 · p. 3

4.1 PGE: Prompt Generation with Evaluation

• PGE generates a prompt for each review, evaluates it on a 5-point scale, stores score-5 prompts, and regenerates lower-scoring prompts.The iterative process addresses the absence of ground-truth prompts and is intended to ensure quality without human supervision.Source: §4.1 · p. 4
• PGE seeds its prompt pool with 100 zero-shot prompts generated by Llama-2-70B-Chat and manually refined for relevance and open-endedness.The initial examples are used as in-context examples during generation.Source: §4.1 · p. 4
• Generation uses randomly sampled in-context examples from the growing prompt pool, with as many examples as fit the context-length constraint.As accepted prompts accumulate in S, they expand the pool available for sampling.Source: §4.1 · p. 4
• Llama-2-70B-Chat scores review-prompt pairs using five in-context examples per score, and prompts are regenerated up to five times if they score below 5.93.60% of reviews reach a score of 5 in three generations or fewer.Source: §4.1 · p. 5

4.2 Dataset Details

• The corpus combines PeerRead and NLPeer data with crawled ICLR and NeurIPS reviews, retaining review metadata and applying open-license and consent conditions.The crawls cover ICLR 2017–2023 and NeurIPS 2016–2022; personal and reviewer metadata are excluded.Source: §4.2 · pp. 5–6
• The unified resource pairs each paper’s full text and metadata with its reviews and generated prompts, while prompt generation uses only review text.Paper PDFs are parsed into structured JSON, and review content and scores are divided according to venue-specific formats.Source: §4.2 · p. 5
• More than 27k papers and 99k reviews span multiple domains; the dataset has the most papers and reviews and is the only one with aspect prompts.Average paper lengths range from 4k to 7k, review lengths from 400 to 500, and prompt lengths from 45 to 60.Source: §4.2 · pp. 5–6
• Table 3 compares dataset scale and prompt availability, while Table 2 reports dataset statistics.The paper reports that its dataset has the largest number of papers and reviews and is the only one containing aspect prompts.Source: §4.2 · pp. 5–6, Table 3 · p. 5, Table 2 · p. 5

5 Experiments

• The experiments compare REVIEWER2 with prompt-free and extractive-summary variants, using maximum similarity across multiple references and GPT-4 judgments of sampled reviews.The evaluation covers six venues; other venues are held out to test adaptability to different review formats.Source: §5 · p. 6, §5.1 · p. 7

5.1 Quality Analysis

• REVIEWER2 outperforms all methods across metrics and datasets, and achieves higher GPT-4-judged winrates than baselines on four review qualities.The reported qualities are faithfulness, coverage, coherence, and specificity.Source: Figure 4 · p. 6, §5.1 · p. 7

5.2 Specificity Analysis

• Specificity rises with training steps for prompt-based methods but falls for prompt-free methods; the gain from prompts is larger with full context than extracted context.Specificity is measured by the BertScore drop when a generated review is paired with references for a different paper.Source: §5.2 · pp. 7–8, Figure 5 · p. 7

5.3 Control Analysis

• REVIEWER2 and SINGLES have similar average reference similarity, but REVIEWER2 has higher maximum similarity across all six venues.The authors interpret this as REVIEWER2 more consistently generating reviews close to one particular reference.Source: §5.3 · p. 8

5.4 Coverage Analysis

• REVIEWER2 has significantly better coverability than REVIEWER2-E, indicating more tailored responses across prompts for a paper.The authors also identify use of full context as important to this result.Source: §5.4 · p. 8, Table 6 · p. 8

• The paper reports that its two-stage framework generates high-quality, specific reviews controllable by aspect prompts, and releases a review dataset annotated with aspect prompts.Source: §6 · p. 8

7.1 Disjoint Processes for Generation

• Prompt generation and review tuning operate independently, reducing prompt effectiveness; the authors suggest integrating the processes to align prompts during fine-tuning.Source: §7.1 · p. 9

7.2 Input Inconsistency

• PGE uses human-written reviews while REVIEWER2 also uses papers; the 4,096-token context limit of Llama-2-70B-Chat constrains PGE inputs, while GPT-4's longer context is costly.The paper reports an average paper context length of 6,229 tokens and notes that the performance improvement may not justify the cost.Source: §7.2 · p. 9

7.3 Limited Domain Knowledge

• REVIEWER2 relies on pretrained knowledge being adequate, which may lead to inaccurate reviews for papers requiring substantial in-domain expertise.The authors identify second-stage pretraining or domain adaptation as possible future work.Source: §7.3 · p. 9

8 Ethics

• Generated reviews are exploratory aids for authors and human reviewers, not replacements for reviewers or definitive authentic reviews of papers.The released datasets may also support NLP research on review generation, instruction following, and self-alignment.Source: §8 · p. 9

A Geometric Intuition of Illustration

• Ten reviews per model are embedded with SFR-Embedding-Mistral4 and visualized using t-SNE; REVIEWER2 generations are more dispersed and closer to human reviews than SINGLES generations.SINGLES generations have smaller coverage and lie around the middle of the space spanned by human reviews; Figure 6 shows the embedding visualization and two-standard-deviation confidence interval.Source: A Geometric Intuition of Illustration · p. 13, Figure 6 · p. 13

Prompt for Generation

• The generation prompt asks an LLM to infer open-ended questions a reviewer is addressing, using sampled review-prompt pairs as examples before generating questions for a new review.The examples are inserted as sampled reviews and prompts, followed by the review for which questions are to be generated.Source: Prompt for Generation · p. 14

Prompt for Evaluation

• The evaluation prompt scores how well generated questions match a candidate answer on a five-point scale, from substantial deviation to comprehensive coverage without irrelevant information.The prompt provides example question-answer-assessment-score records before requesting an assessment of a new pair.Source: Prompt for Evaluation · p. 15

C Example Review-Prompt Pair

• The example review discusses a semantic-correspondence paper’s contribution and concerns, and its paired prompt distills those topics into five questions.The questions cover contribution, neural representation, experimental weaknesses, representation concerns, and matching-cost limitations.Source: C Example Review-Prompt Pair · p. 16

D In-Context Example for Evaluation

• The answer summarizes an alternating minimization algorithm for dictionary learning and notes that the paper provides theoretical guarantees.Each step estimates coefficients with thresholding and then updates the dictionary with a gradient step.Source: D In-Context Example for Evaluation · p. 17
• The answer identifies guarantees for overcomplete dictionaries and a factor-of-log-d improvement in the sparsity-level requirement, then raises six questions about assumptions, theorems, and the algorithm.Its questions also address initialization, permutation invariance, algorithm notation, terminology, and simulation evidence for the convergence rate.Source: D In-Context Example for Evaluation · p. 17
• The answer receives a score of 3 because it fails to address the reproducibility question.The example prompt asks about the paper’s contribution, theoretical analysis, assumptions, theorems and algorithm, and reproducibility.Source: D In-Context Example for Evaluation · p. 17

E.1 Paper Contents

• Paper-content records include the paper title, publication year, abstract, authors, emails, sections, and references.Sections carry headings and text; references include titles, authors, venues, citation expressions, and publication years.Source: §E.1 · p. 18
• Reference records include where each reference is mentioned and its numerical ID and context within the paper.The listed reference fields include title, authors, venue, citation expressions, and year.Source: §E.1 · p. 18

E.2 Metadata Contents

• Metadata records include a paper ID, conference, decision, PDF URL, review URL, title, and author names.The listed decision values are accept or reject.Source: §E.2 · p. 18

F Experimental Details

• REVIEWER2’s prompt-generation model is Llama-2-7B-Chat, taking the full paper to produce aspect prompts; its review-generation model takes the full paper and aspect prompt to produce the review.REVIEWER2 and SINGLES have 32,768-token context lengths, while other models have 4,096. Except for SINGLES-E0, models are fine-tuned for 2 epochs with 8 A6000 GPUs, batch size 64, gradient accumulation 8, 100 warm-up steps, and AdamW at learning rate 1e−5.Source: F Experimental Details · p. 18

G Winrate Details

• Winrate computations use the gpt-4-0613 checkpoint to compare reviews on faithfulness, coverage, coherence, and specificity.The prompt asks GPT-4 to explain its comparison and then select review A or B.Source: G Winrate Details · p. 19, Prompt for Winrate · p. 19

H Generated Aspect Prompts and Review for this Paper

• The example generated review summarizes the two-stage framework and PGE pipeline, and reports that the paper evaluates review quality, specificity, and aspect coverage.It also describes ablations and the paper’s conclusion about using both full-paper context and aspect prompts.Source: H Generated Aspect Prompts and Review… · p. 20
• The example review praises the two-stage architecture’s aspect modeling and reported detail and coverage, while questioning baseline strength, architectural motivation, and prompt-evaluation detail.The review states that generated reviews better cover aspects identified by human reviewers.Source: H Generated Aspect Prompts and Review… · p. 20