PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review

Siming Yuan, Xueyi Zhang, Wangze Ni, Tianfang Xiao, Shimin Di, Jia Zhu, Zhuoren Jiang, Rong Tan, Lei Chen, Kui Ren

arXiv:2609.05947v1cs.AI

TL;DR

Existing AI-assisted peer-review evaluations provide limited evidence about whether decisions are supported by reliable review evidence. This benchmark aligns review records into process variables and evaluates their quality and relationships. Across datasets and models, Gold-process variables generally have higher decision value, while final decisions are not consistently supported by preceding evidence.

  • Problem

    Existing evaluations focus mainly on generated-review quality or final-decision accuracy, providing limited evidence about whether decisions are supported by reliable review evidence.

    Source: Introduction · p. 2

  • Method

    The benchmark aligns paper content, summary, critique, suggestion, and decision across three datasets, then evaluates stage quality, process-variable decision value, chain consistency, and interventional sensitivity.

    Source: Introduction · p. 2

  • Results

    Across three datasets and six models, Gold-process variables generally have higher decision value; complete Gold-only improves over Direct by 0.336–0.434.

    Source: Process-variable Decision Value: Gold… · p. 5, Conclusion · p. 7

  • Takeaways & Limitations

    The benchmark provides an auditable diagnostic for AI systems intended to assist rather than replace human reviewers.

    Source: Conclusion · p. 7

    Chain-consistency results mainly reflect limits of converting stage quality into binary error labels, and sensitivity results should not be interpreted as causal evidence.

    Source: Chain-consistency Evaluation · p. 6, Interventional Sensitivity Analysis · p. 7

Abstract

from arXiv · show

Peer review is central to quality control in science. However, existing evaluations of AI-assisted peer review mainly focus on the overall quality of generated reviews or the accuracy of final decisions. They therefore provide limited evidence about whether model decisions are supported by sufficient and reliable review evidence. We introduce a process-centric diagnostic benchmark for AI-assisted peer review. It uses (x,$z_s$,$z_c$,$z_r$,y) to represent the paper content, summary, critique, suggestion, and decision. We convert heterogeneous review records from PeerRead, NLPeer ARR-22, and OpenReview-ICLR into process-aligned data. Our benchmark uses direct decision prediction from the paper content (Direct) as its baseline. It compares the decision value of Gold-process variables and Predicted-process variables, and conducts stage-level evaluation, chain-consistency evaluation, and interventional sensitivity analysis. Experiments across three datasets and six models show that Gold-process variables generally have higher decision value. For the main analysis model, the Gold--Predicted gap remains stable across datasets and random seeds. This gap is also reproduced in most model--dataset combinations. Although model-generated intermediate review texts show relatively high local consistency across adjacent stages, the final decisions are not consistently supported by the preceding review evidence. Our benchmark targets AI systems designed to assist rather than replace human reviewers. It provides a transparent and auditable diagnostic tool for evaluating the reliability of their review processes.

Introduction

The introduction argues that evaluating AI-assisted peer review requires looking beyond final decisions and overall review quality to the evidence formed across review stages. It introduces a process-aligned benchmark to examine those stages and finds that human-aligned process variables generally have greater decision value than model-predicted variables.

Introduction

  • As conference submission volumes grow, peer-review workload rises, while human reviewers must retain final decision authority and accountability.AI-assisted peer review has raised concerns about fairness and research integrity.Source: Introduction · p. 1
  • Existing evaluations rarely assess process-variable decision value, chain consistency, and intermediate evidence sensitivity within one framework.Review stages are open-ended texts, so a reasonable-looking stage alone does not establish consistency or show that the final decision uses its evidence.Source: Introduction · pp. 1–2
  • The benchmark represents each review as paper content, summary, critique, suggestion, and decision, aligning records from PeerRead, NLPeer ARR-22, and OpenReview-ICLR.Heterogeneous datasets differ in field structure, text granularity, and label sources.Source: Introduction · p. 2
  • The evaluation compares paper-only, Gold-process, and Predicted-process inputs, alongside stage quality, chain consistency, interventional sensitivity, and conditional error analysis.Together, these analyses diagnose how models generate and use review evidence.Source: Introduction · p. 2
  • Gold-process variables generally have higher decision value, while final decisions are not consistently supported by preceding review evidence.The Gold–Predicted gap remains stable across datasets and random seeds for Qwen2.5-14B-Instruct and appears in most model–dataset combinations.Source: Introduction · p. 2

Related Work

Prior AI-assisted review work has mainly aimed to improve generated reviews or their final outputs, while process-oriented evaluation has broadened the dimensions used to assess review quality. The paper distinguishes its approach by aligning review stages with decisions and evaluating their relationships as process variables.

AI-Assisted Peer Review

  • AI-assisted review systems have sought stronger review outputs through methods such as retrieval augmentation and multi-agent collaboration, while this work examines whether evidence is consistent and reflected in decisions.Earlier work explored automatic review-comment generation (Yuan, Liu, and Neubig 2022), and PeerRead enabled analysis of review texts and acceptance decisions (Kang et al. 2018).Source: Related Work · p. 2

Process-Oriented Evaluation

  • Evaluation has expanded beyond score or decision accuracy: Beyond Rating, ReviewEval, MMReview, PRISM, and CoCoReviewBench assess generated reviews for factuality, completeness, constructiveness, and alignment with human reviews.Source: Related Work · p. 2
  • Existing evaluations rarely align summary, critique, suggestion, and decision as unified, intervenable process variables; this work evaluates stage quality, decision value, consistency, sensitivity, and conditional error associations.PeerRead contains papers, reviews, and acceptance labels (Kang et al. 2018), while NLPeer standardizes multiple peer-review data sources (Dycke, Kuznetsov, and Gurevych 2023).Source: Related Work · pp. 2–3

Process-Centric Benchmark

The benchmark aligns peer-review records into shared process variables and evaluates how models generate and use review evidence. It compares human-aligned and model-generated evidence, then diagnoses individual stages, links between stages, and decision sensitivity.

Process-Centric Benchmark

  • The benchmark aligns PeerRead, NLPeer ARR-22, and OpenReview-ICLR records into a shared representation for diagnosing AI-assisted peer-review processes.The representation separates paper content, summary, critique, suggestion, and decision.Source: Process-Centric Benchmark · p. 3

Process-Aligned Data Construction

  • Field-aware alignment constructs Gold-process variables from reviewer summaries when available, abstracts otherwise, and weakness- and suggestion-related review fields.Rule-based extraction supplies critique and suggestion fields when structured fields are missing.Source: Process-Aligned Data Construction · p. 3
  • NLPeer ARR-22 decisions use an average review-score threshold of 3.5, while the aligned datasets contain 562 PeerRead, 364 NLPeer, and 12,002 ICLR samples.OpenReview-ICLR test evaluation draws 500 label-stratified samples per seed.Source: Process-Aligned Data Construction · p. 3

Evaluation Tasks and Stages

  • The framework compares direct paper-only decisions with decisions using seven subsets of Gold-process or Predicted-process variables.Complete-process settings are Gold-only and Pred-only; Paper + Gold and Paper + Pred add paper content to the complete process variables.Source: Evaluation Tasks and Stages · p. 3
  • Interventional sensitivity changes one generated process variable at a time, while conditional error analysis compares decision accuracy across stage-quality groups.Interventions add acceptance-supporting, rejection-supporting, or neutral text, or remove the target variable; conditional accuracy gaps indicate association, not causation.Source: Evaluation Tasks and Stages · p. 4

Evaluation Metrics

  • Stage metrics include unit-matched critique weakness coverage, suggestion alignment and judged quality, and Macro-F1 as the primary decision metric.Critique units are greedily matched one-to-one at a TF–IDF cosine similarity threshold of 0.25; suggestions need not match a reference directly.Source: Evaluation Metrics · p. 4

Evaluation Tasks and Stages

Experiment

The experiments compare decision value across review-process inputs, assess stage quality and chain consistency, test decisions’ sensitivity to process interventions, and examine generalization across models. Across these analyses, Gold-process variables generally outperform Predicted-process variables, while generated review evidence does not consistently support final decisions.

Experimental setup

  • Qwen2.5-14B-Instruct runs the complete six-stage protocol across all datasets and three seeds; five other models run Stages 1–3 on seed-42 PeerRead and NLPeer ARR-22 splits.The six-model selection spans general-purpose open models at different scales, a proprietary API model, and a peer-review-oriented model; shared prompts, deterministic decoding, and a 32K input budget are used.Source: Experimental setup · p. 5

Process-variable Decision Value: Gold versus Predicted Process

  • Gold-process variables outperform Predicted-process variables in all 21 subset–dataset comparisons; Gold-only improves over Direct by 0.336–0.434, while Pred-only improves by 0.194–0.275.Pred-only remains 0.142–0.159 below Gold-only, and the best variable combination differs by dataset: Critique + Suggestion leads on PeerRead, while the complete process leads on NLPeer ARR-22 and OpenReview-ICLR.Source: Process-variable Decision Value: Gold… · p. 5, Table 1 · p. 6

Stage-level Evaluation

  • Summary BERTScore-F1 is 0.866–0.890 across datasets, but predicted critiques have Weakness F1 and Coverage close to zero; suggestion scores are 4.336–4.493 while Critique–Suggestion Alignment is 0.484–0.495.Direct Macro-F1 is 0.235–0.347 and shows an accept bias; low strict alignment means identified issues rarely match specific reference weaknesses under this protocol, not that critiques are entirely invalid.Source: Stage-level Evaluation · p. 5, Table 2 · p. 6

Chain-consistency Evaluation

  • S–C scores are close to 4.0 and C–S scores close to 5.0, but P–D scores are 1.31–1.41, with 97.0%–97.9% of samples rated 1 or 2.Conditional error analysis shows no consistent error-propagation pattern; severe group imbalance affects critique and suggestion estimates, so these results are not evidence of causal propagation.Source: Chain-consistency Evaluation · p. 6, Table 3 · p. 6

Interventional Sensitivity Analysis

  • Process-only interventions produce expected acceptance-probability directions across datasets, and critique has the largest effect: average |∆P| is 0.468–0.491 and DCR is 0.485–0.523.With paper content included, most intervention effects are substantially smaller, though negative critique yields DCR of 0.987–0.997 and decreases acceptance probability by 0.927–0.963; these results indicate sensitivity, not causality.Source: Interventional Sensitivity Analysis · pp. 6–7, Figure 3 · p. 7

Model and Dataset Generalization

  • Gold-process variables outperform Predicted-process variables in all ten combinations of five general-purpose models and two datasets, with gaps of 0.071–0.265 on PeerRead and 0.137–0.360 on NLPeer ARR-22.A small gap need not indicate stronger process generation: DeepReviewer-7B’s similar PeerRead results are both approximately 0.35, while predicted-process gains over Direct vary across models and datasets.Source: Model and Dataset Generalization · p. 7, Table 4 · p. 7

Conclusion

The benchmark evaluates AI-assisted peer review as a process, comparing the decision value of human-aligned and model-generated review variables. It finds that locally consistent review texts do not ensure final decisions are supported by preceding evidence.

Conclusion

  • Gold-process variables generally have higher decision value than Predicted-process variables across datasets and models.The benchmark also evaluates stage-level quality, chain consistency, and interventional sensitivity.Source: Conclusion · p. 7
  • Model-generated review texts show high local consistency across adjacent stages, but final decisions are not consistently supported by preceding review evidence.Source: Conclusion · p. 7
  • The benchmark is designed to assist rather than replace human reviewers, providing a transparent, auditable tool for identifying deficiencies in review-evidence generation and use.Future work will incorporate expert validation and examine bias and uncertainty under human oversight.Source: Conclusion · p. 7
Loading 2609.05947v1…