Source-linked AI summary
Breaking the Illusion of Review Reliability under Static Evaluation: SCOPE Fuzzing for LLM-based Scientific Reviewers
Zhuo Chen, Hao Zeng, Jiawei Liu, Guoxiu He, Le Cai, Liu Haotan, Li Wenbo, Yong Huang, Wei Lu
TL;DR
Prior evidence on LLM-review reliability covers a narrow set of content perturbations, despite the need for consistent judgments under changes unrelated to scientific quality. The paper introduces TREAP and SCOPE-Fuzzer to evaluate reviewers with diverse, adaptive perturbations, finding that static evaluation can miss vulnerabilities that dynamic probing uncovers.
Problem
Prior studies examine a limited set of content perturbations, while broader adoption of LLM review requires evidence that judgments remain consistent under changes unrelated to substantive quality.
Source: §1 · pp. 1–2Method
The paper uses TREAP to organize abstract perturbations across surface, reasoning, and cognitive levels, and SCOPE-Fuzzer to select strategies and adapt mutations using evaluation feedback.
Source: §1 · pp. 2–3Results
Static evaluation reveals stratified vulnerability and perturbation undercoverage, while SCOPE exposes vulnerabilities more effectively than baselines.
Source: §6 · p. 9Takeaways & Limitations
Reliable evaluation requires dynamic templates that explore more perturbation realizations than a single static template.
Source: §1 · p. 3, §6 · p. 9The perturbations are limited to abstracts; future work could extend them to other sections and use more diverse LLMs for rewriting and multi-agent reviewing.
Source: §6 · p. 9
Abstract
from arXiv · showhide
The rapid growth of submissions and reviewing workload has accelerated the use of large language models (LLMs) in peer review. Prior studies suggest that LLM-based reviewers can penalize content perturbations, such as overclaiming, indicating a certain degree of reliability. Yet these conclusions are largely based on a narrow set of perturbation strategies instantiated with static templates, providing limited evidence of actual reliability. In this paper, we construct a three-level evaluation framework covering perturbations to surface presentation, argumentative logic, and value judgment. Experiments on representative LLM-based reviewers reveal two limitations of static evaluation: stratified vulnerability, where perturbation effects depend on whether the paper's original review score is high or low, and perturbation undercoverage, where a single template misses vulnerabilities exposed by diverse realizations. To address these limitations, we propose SCOPE-Fuzzer, a strategy-aware fuzzer that combines feedback-driven strategy selection with adaptive mutation of paper content. By iteratively probing reviewers with dynamic perturbations, SCOPE-Fuzzer consistently uncovers vulnerabilities overlooked by static evaluation and other baselines.
1 INTRODUCTION
The introduction argues that static, template-based evaluations provide limited evidence about LLM-review reliability under plausible content changes. It presents TREAP to broaden perturbation coverage, identifies two limitations of static evaluation, and introduces SCOPE-Fuzzer to search for vulnerabilities dynamically.
1 INTRODUCTION
- Reliability matters because the same scientific work should receive consistent evaluations under changes unrelated to its substantive quality, while prior perturbation evidence is narrow.Prior studies examined a limited set of perturbations, notably overclaiming.Source: §1 · pp. 1–2
- TREAP organizes semantics-preserving abstract perturbations across surface presentation, reasoning structure, and cognitive framing.Its strategies include wording and verbosity changes, overclaiming and logical adjustment, and value alignment and empathy elicitation.Source: §1 · pp. 2–3
- Lower-scoring papers are more likely to receive higher review scores after perturbation, a stratified vulnerability obscured by aggregate results.The authors argue that reliability cannot be characterized by a single aggregate statistic.Source: §1 · pp. 2–3
- Dynamic realizations of a perturbation strategy achieve higher perturbation success rates than a single fixed template, exposing static evaluation’s undercoverage.Different realizations can elicit different reviewer responses, so a fixed template may miss vulnerabilities.Source: §1 · p. 3
- SCOPE-Fuzzer uses feedback-driven strategy selection and adaptive abstract mutation to uncover vulnerabilities more effectively than static-template and feedback-agnostic baselines.It iteratively probes LLM-based reviewers with tailored perturbations under a limited query budget.Source: §1 · p. 3
2 RELATED WORK
Related work spans LLM systems for generating reviews, studies of reviewer reliability, and LLM-assisted fuzzing. The paper positions its work as addressing perturbation reliability through methods drawn from feedback-guided semantic mutation.
2.1 AUTOMATED LLM REVIEW
- LLM review systems use prompting, fine-tuning, multi-agent collaboration, or structured reasoning, but largely target review quality rather than reliability under content perturbations.Examples of structured reasoning include problem trees and reviewer–author debate.Source: §2.1 · p. 3
2.2 RELIABILITY EVALUATION OF LLM-BASED REVIEWERS
- Reliability studies examine fairness and robustness, including prompt injection and static perturbations such as overclaiming; prompt injection may be detectable and conflict with academic integrity.Other examples include comparisons of LLM-authored and human-authored papers and tests on fabricated manuscripts.Source: §2.2 · p. 4
2.3 LLM-BASED FUZZING
- LLM-based fuzzers use semantic mutation and feedback-guided search across security tasks, principles SCOPE-Fuzzer transfers to reviewing.The cited systems target jailbreak prompts, text-to-image models, and agent vulnerabilities.Source: §2.3 · p. 4
3 FAILURES OF CONVENTIONAL STATIC EVALUATION
The section evaluates static LLM-reviewer reliability using TREAP perturbations, revealing score-stratified vulnerability and undercoverage by fixed templates. It motivates probing diverse perturbation realizations to expose vulnerabilities that aggregate static evaluation can miss.
3.1 RELIABILITY EVALUATION FRAMEWORK: TREAP
- TREAP structures reliability evaluation around stealthy, academically acceptable perturbations that preserve a paper’s core evidence and claims.The framework adapts three modes of persuasion to academic review.Source: §3.1 · p. 4
3.1.1 TREAP TAXONOMY: SIX STRATEGIES IN THREE-LEVEL PERTURBATION SPACE
- TREAP spans form-level presentation, reasoning-level argumentation, and judgment-level value cues through six perturbation strategies.The strategies include lexical and verbosity changes, overclaiming and logic adjustment, and value alignment and empathy elicitation.Source: §3.1.1 · pp. 4–5
3.2 EVALUATION PROTOCOL
- The evaluation covers general, fine-tuned, and multi-agent reviewers, with sampling disabled and perturbations screened for fluency and semantic similarity.Static evaluation samples 300 ICLR 2024 papers; multi-probe evaluation uses 93 papers with retrievable abstracts.Source: §3.2 · p. 5
3.3 STRATIFIED VULNERABILITY
- Across tested reviewers, low-original-score papers often gain from perturbations, whereas high-score papers are frequently penalized.This Stratified Vulnerability shows why aggregate score shifts alone do not characterize reviewer reliability.Source: §3.3 · p. 6
- TreeReview has the lowest PSRnet among tested reviewers, while MARG remains substantially vulnerable.The comparison suggests TreeReview’s question-tree agent mechanism may improve robustness, but multi-agent design alone does not ensure reliable reviewing.Source: §3.3 · p. 6
- Static evaluation based on a single aggregate statistic can be misleading, and the cause of Stratified Vulnerability remains uncertain.Insufficient critical reasoning and ceiling or floor effects are proposed as possible explanations.Source: §3.3 · p. 6
3.4 PERTURBATION UNDERCOVERAGE
- Multi-probe evaluation samples 1–10 realizations per paper; PSRnet and PG increase with probe count, with the largest increase between one and three probes.The comparison covers SEA, OpenReviewer, and TreeReview and is constrained by computational cost.Source: §3.4 · p. 6
- PUR exceeds 0.5 in every setting and reaches 0.7, while RUR remains about 0.9; static templates miss vulnerable papers and effective realizations.PUR measures vulnerable papers missed by static evaluation, and RUR measures successful realizations it fails to explore.Source: §3.4 · pp. 6–7
4 FUZZING ON LLM-BASED REVIEWER: SCOPE
SCOPE treats reliability evaluation as budgeted fuzzing, iteratively probing reviewers with diverse, adaptive perturbations to find rating-increasing changes. Its selector uses review feedback to guide strategy search, while its mutator creates paper-specific versions.
4 FUZZING ON LLM-BASED REVIEWER: SCOPE
- Under a fixed query budget, SCOPE searches for perturbations that increase a reviewer’s rating while satisfying naturalness and semantic-consistency constraints.It frames the search as iterative fuzzing with diverse, adaptive templates.Source: §4 · p. 7
- SCOPE iteratively selects a strategy or strategy set from TREAP’s action space and generates a mutated abstract for the target paper.The abstract is inserted into the paper for review.Source: §4 · p. 7
- The selector starts by ranking strategies for the abstract, then uses review feedback and a heuristic policy to choose later actions.After a positive perturbation it expands a promising strategy; after non-positive gain it switches to an untested direction.Source: §4 · p. 7
- For each selected action, SCOPE creates a paper-specific instruction and rewrites the abstract to produce a perturbed version.The oracle evaluates the resulting paper, reports its review score and score shift, and updates the fuzzing state.Source: §4 · p. 7
- After exploring within budget, SCOPE returns the most effective perturbed abstract from the valid versions it explored.The explored set is bounded by the query budget.Source: §4 · p. 7
5 EVALUATION OF SCOPE FUZZING
SCOPE is evaluated against three multi-probe baselines and in selector and mutator ablations. It generally uncovers vulnerabilities more effectively than the baselines, while ablations support the roles of adaptive mutation and feedback-guided exploration.
5.1 FUZZING EFFECTIVENESS: SCOPE VS BASELINES
- Except at a one-query budget, SCOPE achieves the highest net PSR and PG across the evaluated LLM-based reviewers.The comparison is with PAA, MPS, and SPD.Source: §5.1 · p. 8
- SCOPE combines realistic TREAP strategies with feedback-driven search and content-adaptive mutation, addressing the baselines’ limited perturbation-space exploration.PAA lacks targeted strategies, MPS uses six static templates, and SPD searches only one strategy.Source: §5.1 · p. 8
5.2 ABLATION STUDY
- The adaptive mutator improves perturbation effectiveness by tailoring plans to each paper and varying the perturbation templates.The non-adaptive ablation rewrites abstracts from strategy instructions without paper-specific planning.Source: §5.2 · p. 9, Figure 5 · p. 9
- Removing feedback guidance reduces vulnerability discovery, while feedback-driven switching promotes broader strategy exploration and more effective perturbation paths.Without feedback, the selector can converge on locally optimal strategies.Source: §5.2 · p. 9, Figure 5 · p. 9
- Random selection is slightly inferior to switching on SEA and TreeReview but competitive overall, while selection strategies perform similarly on OpenReviewer.The paper suggests OpenReviewer may be less sensitive to differences among perturbation strategies.Source: §5.2 · p. 9
6 CONCLUSION
The paper uses TREAP to examine subtle, academically plausible perturbations and finds that static evaluation can miss vulnerabilities. SCOPE searches dynamic TREAP realizations under a fixed query budget, with evaluation limited to abstracts.
6 CONCLUSION
- TREAP covers form-, reasoning-, and value judgment-level perturbations for evaluating LLM reviewers under subtle, academically plausible content changes.The paper examines changes to abstracts.Source: §6 · p. 9
- Static evaluation can miss vulnerabilities because perturbation effects vary with initial review scores and different realizations of one strategy can produce different outcomes.The paper calls these patterns stratified vulnerability and perturbation undercoverage.Source: §6 · p. 9
- SCOPE searches dynamic TREAP realizations under a fixed query budget and exposes vulnerabilities more effectively than baselines.Experiments identify TREAP strategies, efficient exploration, and adaptive mutation as factors in its effect.Source: §6 · p. 9
- The perturbations are limited to abstracts; future work could extend them to other sections and use more diverse LLMs for rewriting and multi-agent reviewing.The reported scope does not cover perturbations to other paper sections.Source: §6 · p. 9
A.1 DETAIL OF TREAP
TREAP organizes perturbations by persuasive level and is designed to probe reviewer responses while preserving stealth and academic integrity. Its strategies range from presentation changes to appeals to values and empathy.
A.1 DETAIL OF TREAP
- Perturbations aim to enhance perceived paper value, including through terminology, syntactic complexity, rhetorical organization, or overclaiming (Li et al., 2025a; Tyser et al., 2024).Overclaiming uses strong claims that exceed the work’s actual scope while retaining a formal academic tone.Source: §A.1 · pp. 11–12
- The framework excludes review-command jailbreaks and authority cues because they lack stealth or risk academic integrity.Authority cues about specific institutions or people are excluded.Source: §A.1 · p. 12
- TREAP organizes perturbations into form, reasoning, and value judgment levels, drawing on a taxonomy of persuasion models (Hwang et al., 2025).The taxonomy spans persuasive tactics available to a human adversary in reviewing.Source: §A.1 · p. 12
- Form-level strategies include complexifying academic language and increasing verbosity, while logic adjustment reorders contributions or findings without changing substantive content.These strategies seek to shape perceived sophistication, completeness, or persuasive organization.Source: §A.1 · p. 12
- Value Alignment foregrounds social values, while Empathy Elicitation humanizes the work and highlights effort through mild emotional cues.Both are intended to increase favorable evaluations without compromising academic tone.Source: §A.1 · p. 12
A.2 HUMAN VALIDATION
Human validation found that the perturbations generally preserved core research content and academic plausibility, even as LLM reviewers assigned higher scores to perturbed abstracts.
A.2 HUMAN VALIDATION
- Twenty-two participants with master’s or doctoral degrees completed blinded paired-text evaluations, yielding 21 valid responses.Participants assessed semantic equivalence, academic suitability, and which text appeared revised.Source: §A.2 · p. 12
- Participants also rated whether each abstract could reasonably appear in a formal academic paper and included quality-control pairs for semantic equivalence and changed results.The perturbed abstracts were judged as suitable as, or more suitable than, the originals for formal scientific papers.Source: §A.2 · pp. 12–13
- The six strategies modify lexical complexity, verbosity, overclaiming, logical flow, value alignment, and empathy cues.The prompts direct edits to remain consistent with the original research content while changing its presentation or framing.Source: §A.2 · pp. 13–16
- 88.9% of retained responses judged the original and perturbed abstracts to preserve the same core research content, while LLM reviewers scored perturbed versions higher.Case-level preservation rates ranged from 81.0% to 100.0% across the six strategies.Source: §A.2 · p. 13, Figure 9 · p. 16
A.3.1 PROTOCOL
The protocol uses a 300-paper ICLR 2024 sample for static evaluation and a smaller set for repeated perturbation probing.
A.3.1 PROTOCOL
- 300 papers were randomly sampled from the ICLR 2024 corpus for static evaluation, while 93 papers with locatable abstracts formed the multi-probe set.The multi-probe set was smaller because repeated probing of 300 papers was prohibitively costly.Source: §A.3.1 · p. 16
- Sampling was disabled for all LLM review systems so repeated queries on unchanged inputs returned the same output.This controls generative randomness and addresses sampling variability or regression to the mean from repeated measurement.Source: §A.3.1 · p. 16
- Experiments perturbed only abstracts and were conducted on an NVIDIA H100 GPU.Source: §A.3.1 · p. 16
A.3.2 TARGET LLM-BASED REVIEWERS
This section describes SCOPE’s evaluation of LLM-based reviewers through a budgeted, black-box adaptive search over constrained abstract perturbations, selecting strategies based on prior rating gains and retaining the perturbation with the largest gain.
A.3.2 TARGET LLM-BASED REVIEWERS
- The evaluated reviewers include SEA (Yu et al., 2024b), DeepReview (Zhu et al., 2025), OpenReviewer (Idahl & Ahmadi, 2025), MARG (D’Arcy et al., 2024), and TreeReview (Chang et al., 2025).The reviewers typically use their original ICLR review prompts.Source: §A.3.2 · p. 17
A.3.4 EVALUATION METRICS
- PSR counts rating increases, PFR counts decreases, and PG is the average score change; fuzzing uses r0 = 0 internally, then reports gains against the original score.The initial original-paper rating is only an evaluation baseline, not used during fuzzing; after the first perturbation, the search reference is updated to r1.Source: §A.3.4 · p. 17, §A.4.2 · p. 20
A.3.5 FORMAL DEFINITIONS OF SV AND PU
- SV captures differing perturbation-induced rating changes between papers below and at or above the mean score; PU denotes vulnerabilities missed by a single strategy realization.PUR measures the fraction of vulnerable papers missed by a static template, while RUR captures effective realizations left unexplored; the realizable space is set to 10 due to budget.Source: §A.3.5 · pp. 17–18
A.4.1 PRELIMINARIES
- SCOPE assumes black-box access and searches within a query budget for an abstract perturbation that maximizes the reviewer’s rating increase over the original paper.It returns the best perturbation found during the budgeted search.Source: §A.4.1 · p. 18, §A.8 · p. 22
A.5 ACTION SPACE: TREAP
- TREAP has six atomic strategy families and allows compounds of up to three; baselines include PAA, static multi-strategy MPS, and single-strategy dynamic SPD.MPS uses six fixed templates and is limited to budget 6, while SPD varies realizations but lacks adaptive mutation.Source: §A.5 · p. 20, §A.9.1 · p. 22
A.6.2 HEURISTIC ACTION SELECTION
- An LLM ranks the initial strategy; afterward, SCOPE expands successful actions into compatible compounds or switches strategy families after failures.When expansion is unavailable, it can refine the perturbation instructions; the failure path avoids repeating the same local direction when possible.Source: §A.6.1 · p. 20, §A.6.2 · p. 21
A.7 MUTATION: ADAPTIVE ABSTRACT REWRITING
- SCOPE adaptively rewrites the original abstract according to selected strategies, requiring semantic similarity of at least 0.85 and perplexity no greater than 1.2× the original.BERTScore measures semantic similarity and GPT-2 computes perplexity; invalid candidates are rejected and resampled.Source: §A.3.3 · p. 17, §A.7 · pp. 21–22
A.9.1 DESCRIPTION OF BASELINES
- SCOPE significantly improves PG over all baselines on SEA and TreeReview; on OpenReviewer, its improvement is significant over MPS but not PAA or SPD.The paired permutation test used 10,000 iterations; SCOPE-row PSR superscripts denote significance at p < 0.01 (**) or p < 0.001 (***), with ns meaning not significant.Source: §A.9.1 · p. 22, §A.10 · p. 23
A.10 ABLATION STUDY OF DIFFERENT SELECTION METHOD
- SCOPE-Fuzzer adapts strategy selection to prior gains, filters mutations by semantic similarity and perplexity, queries the reviewer, and retains the highest-gain perturbation.After a positive gain from a single action, it prefers compatible expansions; otherwise, it prefers unseen actions from different families.Source: §A.10 · p. 24
- Table 8 defines perturbation gain comparisons as mean paper-level paired differences, with positive ΔPG favoring SCOPE and two-sided paired permutation tests.The supplied caption does not include comparison values or identify a winner.Source: Table 8 · p. 24