A Missing Piece for Trustworthy AI Reviewers: From Benchmarking Rhetorical Robustness to SciCore Review

1 Introduction

• Existing evaluations have examined human alignment extensively, while rhetorical robustness has remained comparatively neglected.The introduction motivates this gap by noting that meaning-preserving revisions can alter automated evaluations.Source: §1 · pp. 1–2
• Rhetorical robustness requires both stable judgments across content-preserving rewrites and discrimination among papers; human alignment is a distinct property.The introduction notes that agreement with human judgments on original manuscripts does not establish stability across rhetorical variants.Source: §1 · p. 2
• The evaluated content-focused prompting protocol does not consistently improve robustness across backbones.This finding motivates the introduction of SciCore as a dual-branch framework.Source: §1 · p. 2
• RobustReview is a controlled full-manuscript benchmark spanning rewrite strategies, reviewer families, and review protocols.The benchmark evaluates whether content-focused review is nontrivial and exposes configuration-dependent sensitivity and false robustness.Source: §1 · p. 2
• SciCore averages a full-manuscript judgment with a content-normalized judgment from an extracted science core, improving the joint robustness profile while maintaining competitive human alignment in the primary evaluation.The introduction presents the content-normalized branch as a complement to manuscript-level assessment.Source: §1 · p. 2

2.1 Rhetorical Robustness

• Rhetorical robustness jointly requires within-paper stability across rhetorical variants and between-paper discrimination; stability alone can mask score collapse.Human alignment is measured separately, and the joint metrics relate within-paper consistency to cross-paper variation or separation.Source: §2.1 · p. 3

2.2 Benchmark Construction

• RobustReview contains 60 ICLR 2026 submissions and 1,200 rhetorical variants, generated under 10 conditions by two independent LLM systems.The submissions are sampled evenly across six mean human-review score intervals; rewrites operate on complete LaTeX projects under content-preservation and structural controls.Source: §2.2 · p. 4

2.3 Reviewer Configurations

• The evaluation covers 30 reviewer configurations, including general-purpose LLMs prompted under Standard, Strict, and Persistent protocols, specialized models, and agentic systems.Persistent prompting directly tests whether content-only instructions suffice to separate scientific judgment from rhetorical presentation.Source: §2.3 · p. 4

2.4 Evaluation Metrics

• Seven metrics separately assess rewrite stability, joint stability-discrimination, and agreement with human scores and rankings.MAD and Drift SD measure within-paper stability; ICC, SPR, and discriminability are joint metrics, while Human MAE and Spearman correlation assess human alignment.Source: §2.4 · p. 4

3 Benchmark Findings: Limits of Current AI Reviewers

• RobustReview reports seven metrics because no single column captures rewrite movement, joint stability-discrimination, and human alignment.Source: §3 · p. 5

3.1 Content-Only Review Is Not Reliably Achieved by Prompting Alone

• Only GPT-5.5 improves all five robustness metrics under Persistent; Claude Sonnet 5 lowers MAD and Drift SD but also lowers ICC, SPR, and discriminability, while other backbones are mixed.Source: §3.1 · p. 6

3.2 False Robustness: Within-Paper Stability without Discrimination

• 0.205 MAD for Gemini-3.5-Flash-Lite under Standard pairs with 0.199 ICC and 0.511 discriminability, illustrating false robustness; GPT-5.5 Persistent leads joint metrics.DeepReviewer and OpenJudge likewise show relatively low drift but weak ICC and discriminability.Source: §3 · p. 5, §3.2 · p. 6

3.3 Human Alignment and Rhetorical Robustness Are Distinct

• GPT-5.5 Strict has the strongest existing human-alignment profile, while Persistent has weaker alignment but higher ICC, SPR, and discriminability.Human agreement on original manuscripts does not establish stability across rhetorical variants.Source: §3.3 · p. 6

4 SciCore: Dual-Branch Review for Rhetorical Robustness

• SciCore averages a full-manuscript judgment with a science-core judgment, preserving the conventional review alongside a content-normalized view.Source: §4 · p. 6

4.1 Design Principle: A Stable Scientific Branch without Discarding the Manuscript

• The science-core branch is intended to reduce rhetorical sensitivity, but its extracted representation must be tested for stability and paper separation rather than assumed to achieve them.Averaging may reduce sensitivity when the science-core branch varies less across matched rewrites, while joint metrics guard against cross-paper score collapse.Source: §4.1 · p. 6

4.2 Science-Core Extraction: Isolating the Reported Scientific Record

• The extractor records scientific claims, methods, assumptions, evidence, results, contributions, reproducibility information, and limitations in objective, attributed language.It is instructed not to score persuasiveness or significance, and to retain results and numerical evidence while flagging information it cannot reliably recover.Source: §4.2 · p. 7

4.3 Science-Core Branch: Evaluating the Extracted Record

• The science-core reviewer judges only the extracted record, connecting claims with methods, assumptions, and evidence while using stated limitations to assess evidence scope.It retains ICLR-style criteria and scoring, and treats explicit omissions as missing support while handling extraction uncertainty separately.Source: §4.3 · p. 7

4.4 Dual-Branch Fusion: Integrating Manuscript and Science-Core Judgments

• The manuscript and science-core branches produce overall scores that SciCore combines by an unweighted arithmetic mean, avoiding a tuned fusion parameter.Source: §4.4 · pp. 7–8

5.1 Main Results: Rhetorical Robustness on RobustReview

• SciCore leads the primary comparison on ICC (0.775), SPR (0.652), and discriminability (0.726), with the lowest Human MAE (1.072) and second-highest human Spearman correlation (0.488).It does not have the lowest MAD or Drift SD, and its human Spearman correlation remains below GPT-5.5 under Strict.Source: §5.1 · p. 8

5.2 Dissecting the Gains: Architecture and Review-Policy Ablations

• Direct science-core review outperforms reconstruction in this pipeline: ReconstructReview worsens MAD from 0.598 to 0.651 and Drift SD from 0.956 to 1.101 versus Manuscript-Standard.Core-Standard then improves all seven metrics over ReconstructReview; the authors note that the theoretical rationale does not predict this pipeline ordering.Source: §5.2 · p. 8
• With the extracted core held fixed, Core-Persistent worsens all five robustness metrics relative to Core-Standard, while Core-Adapted leads the core-only configurations on four specified robustness metrics.Core-Adapted has the lowest MAD and Drift SD and the highest ICC and SPR among core-only configurations.Source: §5.2 · p. 8
• Fusion reduces MAD from 0.766 to 0.476 and increases ICC from 0.632 to 0.775 over Manuscript-Strict, improving all five robustness metrics; it also improves human alignment, ICC, and discriminability over Core-Adapted.Against the three-review control, SciCore has better point estimates on five metrics, while the control has lower Drift SD and higher ICC; the table compares GPT-5.5 configurations and reports direct and joint metrics.Source: §5.2 · p. 8, Table 2 · p. 9

5.3 Representation Analysis: Science-Core Stability and Paper Separation

• For GPT-5.5, matched science cores have similarity 0.978, versus 0.616–0.619 for cross-paper cores, indicating rewrite stability alongside paper-specific differences.Table 3 compares matched and different-paper cores across extractors and rewrite producers.Source: §5.3 · p. 9, Table 3 · p. 9

5.4 Criterion-Level Behavior of the Science-Core Branch

• Science-core review generally improves soundness stability, but contribution and human-alignment gains are less consistent; constant confidence scores can make low drift misleading.Core-Adapted raises confidence SPR to 0.394, whereas Core-Standard and Core-Strict assign constant confidence scores.Source: §5.4 · p. 9, Table 4 · p. 10

5.5 Cross-Backbone Generalization of the Science-Core Branch

• Core-Adapted improves ICC for all three backbones, but other robustness and human-alignment outcomes vary, and these tests evaluate the branch rather than final fusion.GPT-5.5 improves on all five robustness metrics, GPT-5-mini improves MAD, Drift SD, and ICC but weakens SPR and discriminability, and GLM-5.2 improves all seven reported metrics. Extraction and review change together, so the experiments do not isolate the limiting stage.Source: §5.5 · p. 10, Table 5 · p. 10

5.6 Science-Core Weight Sensitivity

• Larger science-core weights tend to improve robustness while weakening human alignment; under Manuscript-Strict, α = 0.5 lies near the transition and maintains strong ICC, SPR, and discriminability.Equal weighting is the prespecified, Pareto non-dominated default across the evaluated protocol-weight combinations on seven point estimates; Figure 2 shows normalized performance across weights and protocols.Source: §5.6 · p. 11, Figure 2 · p. 11

6 Related Work

• Related research evaluates agreement, review quality, score prediction, and workflow capability, while documenting sensitivity to order, length, prompts, evaluator identity, and rhetorical or adversarial changes.The cited literature includes controlled full-manuscript rewrites characterized as rhetorical reward hacking by Li et al. (2026c).Source: §6 · p. 11
• Unlike interventions that control particular biases or judge intermediate representations, SciCore pairs full-manuscript review with structured science-core extraction and review, then averages the judgments.The design is motivated by invariant representation and Blackwell comparison.Source: §6 · p. 11

7 Conclusion

• RobustReview operationalizes rhetorical robustness as within-paper stability combined with between-paper discrimination, because low drift can result from collapsed scores that fail to distinguish papers.Its metrics relate same-paper consistency to cross-paper variation or separation.Source: §7 · p. 11
• Content-focused prompting yields model-dependent rather than consistent robustness gains, and human-alignment rankings differ from matched robustness evaluations.The paper concludes that content-focused review is not resolved by score-sensitivity or human-agreement measures alone.Source: §7 · p. 12
• SciCore averages a full-manuscript judgment with an equally weighted, content-normalized science-core judgment; its cores are stable and paper-specific in representation diagnostics.The fused reviewer has a leading joint stability-discrimination profile and competitive human alignment in the primary GPT-5.5 comparison, while branch-level behavior remains criterion- and backbone-dependent.Source: §7 · p. 12

8 Limitations

• RobustReview covers 60 ICLR 2026 submissions sampled across score strata from papers with matched arXiv sources, limiting demonstrated generalizability to other venues, fields, and formats.The automated fidelity audit finds a nonzero mismatch rate, and mean human scores do not capture disagreement among reviewers.Source: §8 · p. 12
• The final SciCore fusion is evaluated with GPT-5.5; cross-backbone analyses cover the science-core branch and vary extraction and review together.Science-core extraction may omit or misread details and requires additional inference; equal-weight fusion assumes both branch assessments are comparable on the shared scale.Source: §8 · p. 12

A Theoretical Motivation for SciCore

• The theoretical results motivate the science-core branch as design rationales, not performance guarantees for a fixed LLM reviewer.For RobustReview, the notation distinguishes each original manuscript from variants produced under controlled rhetorical conditions.Source: A Theoretical Motivation for SciCore · p. 16

A.1 The Maximal-Invariant Ideal

• An ideal science core maps manuscripts to the same representation exactly when they express the same reported scientific record, including claims, methods, assumptions, evidence, results, and conclusions.The relation ∼sci is the paper’s ideal invariance target; generated rewrites only approximate it, and preservation is assessed empirically.Source: §A.1 · p. 16
• Any rhetorically invariant judgment can factor through a maximal science core, and any judgment operating on that core is rhetorically invariant.The paper’s implemented extractor is not claimed to be maximal or exactly invariant; it approximates stability across matched variants and separation across papers.Source: §A.1 · p. 16

A.2 Conditional Stability of the Science-Core Branch and Fusion

• When the science-core branch has lower rewrite-induced MAD than the manuscript branch, averaging their scores can bound the fused reviewer’s MAD below the manuscript branch’s MAD.This is an explanatory condition, not a performance guarantee; the fused reviewer still requires joint stability-discrimination evaluation.Source: §A.2 · pp. 16–17

A.3 Reconstruction as a Blackwell Garbling

• Reconstruction cannot add decision-relevant information absent from its complete inputs, though it can discard information or introduce another rhetorical realization.The information ordering does not guarantee that a fixed LLM reviews the inputs better than the reconstructed manuscript, and it does not order the two implemented pipelines.Source: §A.3 · p. 17

B.1 Reliability and Robustness of LLM-as-a-Judge Systems

• Existing LLM judges vary with order, length, wording, and evaluator identity, while content-focused representations and controlled review tests motivate assessing both rewrite stability and paper discrimination.The theoretical traditions cited are design rationales, not performance guarantees for a fixed LLM reviewer.Source: §B.1 · p. 18

B.2 LLMs for Scientific Peer Review

• LLM review has advanced from review components to full reviews and can aid human feedback, but long-paper processing, scoring, and substantive criticism remain challenging.Prior evaluations leave open whether scientific judgments remain stable under content-preserving rhetorical changes.Source: §B.2 · p. 18

B.3 Manipulation and Presentation Sensitivity in AI Scientific Review

• This study tests full-manuscript rewrite stability, cross-paper discrimination, and human alignment, and introduces SciCore as a combination of manuscript review and content-normalized judgment.Its counterfactuals preserve reported reasoning and evidence, complementing work that changes substantive reasoning to test defect detection.Source: §B.3 · p. 19

C Experimental Details

• The experimental details follow the rewrite and review methodology of Li et al. (2026c), with prompts instantiated for this benchmark and all rewrites and scores generated for this study.The passage identifies this section as covering benchmark construction, experiments, SciCore, and ablations.Source: C Experimental Details · p. 20

C.1 RobustReview Composition and Sampling

• RobustReview samples 60 ICLR 2026 submissions equally across six mean-human-rating intervals and pairs each original with 10 rewrite conditions, producing 1,260 manuscript versions.The authors state that generalization beyond ICLR 2026 submissions with matched arXiv sources remains to be evaluated.Source: §C.1 · p. 20

C.2 Rewrite Construction and Structural Controls

• Rewrite producers transform complete LATEX projects under instructions to preserve reported scientific content while changing rhetorical presentation, with dependencies protected and compilation checks applied.Invalid outputs are repaired when possible or excluded.Source: §C.2 · p. 20

C.3 Reviewer Execution Details

• The study evaluates general-purpose, specialized, and agentic reviewers under stated protocols, using one independent review per version in the main experiments.An auxiliary GPT-5-mini audit finds condition-level mean effects closely aligned between single and three-review settings, while the robustness metrics include stochastic variation under single-review execution.Source: §C.3 · pp. 20–21

C.4 Full Evaluation Metric Definitions

• SPR compares cross-paper score variation with rewrite-induced movement; discriminability measures separation between same-paper and different-paper presentations; lower Human MAE and higher Spearman indicate stronger agreement with mean human ratings.Higher SPR values indicate cross-paper variation remains large relative to within-paper rhetorical perturbation; all configurations use the same 60 complete paper families (1,260 manuscript versions), with no missing overall-assessment scores.Source: §C.4 · pp. 21–23

C.5 SciCore Execution Protocol

• SciCore extracts a science core with GPT-5.5, reviews it with an adapted protocol, and averages that score with GPT-5.5 Strict review of the full PDF.The arithmetic fusion assumes scores are comparable across branches; the branch-policy ablation reuses each extracted core while varying review instructions.Source: §C.5 · p. 23

C.6 ReconstructReview Ablation

• ReconstructReview extracts a science core, uses it with permitted bibliography and figures to generate a full manuscript, then reviews that manuscript under the Standard protocol.The reconstruction is instructed to preserve the scientific record and not introduce unsupported content; it is applied to all 1,260 benchmark presentations.Source: §C.6 · p. 23

D Complete RobustReview Results

• The full-score results section reports secondary dimensions alongside overall-assessment results, using the same seven metrics and column order as the main table.External reviewers and SciCore fusion are excluded from these secondary-score tables because they expose only OA-compatible outputs.Source: §D.1 · p. 24, D Complete RobustReview Results · p. 24

D.1 Full Score-Dimension Results

• The secondary-score tables cover soundness, presentation, contribution, and confidence, with the captions identifying column-wise best, second-best, and third-best results.A dash marks undefined Spearman correlation in the presentation table and undefined Spearman correlation or ICC in the confidence table.Source: Table 6 · p. 25, Table 7 · p. 26, Table 8 · p. 27

E.1 Core Technical-Content Fidelity Audit

• The automated and human fidelity checks indicate that rewrite technical content is largely preserved, with occasional discrepancies that may contribute to observed score variation.The human validation covers a combined sample of 150 comparisons, including 30 flagged by the automated auditor; all comparisons remain in the evaluation.Source: §E.1 · p. 29

E.2 Complementary Review Branches under Equal-Weight Fusion

• SciCore improves all five robustness metrics over either matched manuscript alternative, while the alternatives outperform it on some individual metrics and SciCore remains Pareto non-dominated.The comparisons match the manuscript branch, averaging rule, and score resolution; the three-protocol average additionally matches nominal call budget, though token costs may differ.Source: Table 11 · p. 30, §E.2 · p. 30

E.3 Repeated-Review Noise Baseline

• Repeated reviews of identical PDFs yield MAD of 0.294–0.329 and ICC of 0.846–0.862; SciCore and Manuscript-Strict have similar repeatability but differ more under rewriting.Under rewriting, SciCore versus Manuscript-Strict has MAD 0.476 versus 0.766 and ICC 0.775 versus 0.632.Source: §E.3 · p. 30

E.4 Paper-Family Bootstrap Analysis

• SciCore’s paired-bootstrap comparisons support gains on all five robustness metrics over Manuscript-Strict and both two-score manuscript ensembles, with human-alignment tradeoffs.Relative to Core-Adapted, fusion improves human alignment, ICC, and discriminability, while increasing MAD and Drift SD and reducing SPR.Source: §E.4 · p. 30, Table 12 · p. 31

E.5 Science-Core Preservation Diagnostics

• Science-core preservation is assessed by comparing embeddings of reports extracted from original manuscripts and their matched rhetorical variants.Condition-level tables separate GPT-5.5 and Opus 4.8 rewrite producers, report the mean, median, and fifth percentile, and use nonmatching original cores as a cross-paper baseline.Source: §E.5 · p. 32, Table 14 · p. 32, Table 15 · p. 32

E.6 Complete Science-Core Weight Sensitivity Results

• The sensitivity tables vary science-core weight α from 0 (manuscript only) to 1 (science core only) for three manuscript protocols.Intermediate weights use continuous post-hoc fusion in Equation 3; the tables report unnormalized numerical results.Source: §E.6 · p. 32, Table 17 · p. 33, Table 18 · p. 33

• API expenditure is reported by task, covering RobustReview construction, review-only work, SciCore, ReconstructReview, AI Scientist, and OpenJudge workflows.The SciCore task includes extraction, branch-policy evaluations, and embedding; ReconstructReview includes manuscript reconstruction and final review.Source: F Compute, Cost, and Efficiency · p. 35
• SciCore reuses GPT-5.5 Strict reviews counted under Review-only, while score fusion adds no API call; ReconstructReview reuses cached science cores.Reported task costs come from provider API consoles and include billable retries and failed attempts.Source: F Compute, Cost, and Efficiency · p. 35
• OpenReviewer, CycleReviewer, DeepReviewer, and the trained ProReviewer backbone are excluded from API expenditure because they were served locally on eight NVIDIA A100 GPUs.Source: F Compute, Cost, and Efficiency · p. 35