PaperlayerView exact revision on arXiv

Source-linked AI summary

Detecting LLM-Generated Peer Reviews

Vishisht Rao, Aounon Kumar, Himabindu Lakkaraju, Nihar B. Shah

arXiv:2503.15772v2cs.DLcs.AIcs.CR

TL;DR

Existing detectors struggle to distinguish fully LLM-generated reviews from those edited or paraphrased with AI, while policies against generated reviews are difficult to enforce. The paper embeds hidden prompts in manuscript PDFs to induce review watermarks, then tests for those signals across reviews. Fake-citation watermarks appeared in 98.6% of generated reviews on average across models, and the statistical tests maintained high power with zero false positives over ten thousand reviews.

  • Problem

    Existing detectors often struggle to distinguish fully LLM-generated reviews from reviews edited or paraphrased with AI, complicating detection of prohibited review generation.

    Source: §1 · p. 2

  • Method

    The framework embeds hidden prompts in manuscript PDFs to make LLMs include selected watermarks in reviews, then statistically tests for those signals across multiple reviews while controlling FWER.

    Source: §1 · p. 2

  • Results

    Fake-citation watermarks appeared in 98.6% of LLM-generated reviews on average across models; FWER-controlling tests yielded zero false positives over ten thousand reviews while maintaining high power.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The approach supports post hoc identification of likely LLM-generated reviews without assumptions about human writing and remains effective against common defenses such as paraphrasing.

    Source: §1 · p. 2, §4 · p. 18

    Detecting misuse and attempts to evade detection remains an ongoing challenge.

    Source: §4 · p. 18

Abstract

from arXiv · show

The integrity of peer review is fundamental to scientific progress, but the rise of large language models (LLMs) has introduced concerns that some reviewers may rely on these tools to generate reviews rather than writing them independently. Although some venues have banned LLM-assisted reviewing, enforcement remains difficult as existing detection tools cannot reliably distinguish between fully generated reviews and those merely polished with AI assistance. In this work, we address the challenge of detecting LLM-generated reviews. We consider the approach of performing indirect prompt injection via the paper's PDF, prompting the LLM to embed a covert watermark in the generated review, and subsequently testing for presence of the watermark in the review. We identify and address several pitfalls in na\"ive implementations of this approach. Our primary contribution is a rigorous watermarking and detection framework that offers strong statistical guarantees. Specifically, we introduce watermarking schemes and hypothesis tests that control the family-wise error rate across multiple reviews, achieving higher statistical power than standard corrections such as Bonferroni, while making no assumptions about the nature of human-written reviews. We explore multiple indirect prompt injection strategies -- including font-based embedding and obfuscated prompts -- and evaluate their effectiveness under various reviewer defense scenarios. Our experiments find high success rates in watermark embedding across various LLMs. We also empirically find that our approach is resilient to common reviewer defenses, and that the bounds on error rates in our statistical tests hold in practice. In contrast, we find that Bonferroni-style corrections are too conservative to be useful in this setting.

1 Introduction

LLM-generated peer reviews are difficult to distinguish from reviews merely edited or paraphrased with AI, despite policies prohibiting generated reviews. This work embeds hidden prompts in manuscript PDFs to watermark LLM-generated reviews and uses statistical tests to detect them across reviews with formal error control.

1 Introduction

  • LLM misuse raises concerns about review integrity; although venues prohibit generated reviews, enforcement is difficult and existing detectors struggle to distinguish generated from AI-edited or paraphrased text.A cited study reports that at least 15.8% of reviews submitted to ICLR 2024 were at least AI-assisted [LRD+24].Source: §1 · pp. 1–2
  • The framework stochastically selects watermarks, injects hidden prompts into manuscript PDFs, and tests for those signals while controlling FWER across multiple reviews.Prompts can use white-text cues, font manipulations, or cryptic instructions; they are intended to be indiscernible to human reviewers but processed when the PDF is uploaded to an LLM.Source: §1 · p. 2
  • Unlike prior stylistic-repetition and re-prompting detectors, the framework offers formal FWER guarantees without relying on human-writing patterns or assumptions about LLM behavior.The paper also investigates font embedding and cryptic prompt injection via adversarial jailbreaking, repurposing jailbreaking for review detection.Source: §1 · p. 2
  • Experiments used real-world reviews and grant proposals, including ICLR 2024, PeerRead, and NSF data, and tested several widely used LLMs.The tested models included ChatGPT 4o and o1-mini, Gemini 2.0 Flash, Claude 3.5 Sonnet, LLaMA 2, and Vicuna 1.5.Source: §1 · p. 2
  • A randomly generated fake citation appeared in 98.6% of LLM-generated reviews on average across models.The passage begins a further qualifier after “models and,” which is not included here.Source: §1 · p. 2
  • Over 94% of watermarked reviews retained the watermark after paraphrasing; cryptic prompts succeeded in 91% across models and datasets and in 19 out of 20 samples.Watermarking also succeeded in up to 89% of reviews generated for NSF grant proposals.Source: §1 · p. 3
  • FWER-controlling tests maintained high power and yielded zero false positives across over ten thousand reviews, while Bonferroni and Holm-Bonferroni became infeasible.The framework's tests provide formal family-wise error control across multiple reviews without assumptions about human-written reviews.Source: §1 · pp. 2–3

2 Design of methods

The methods combine randomly selected watermarks and PDF-based indirect prompt injections with procedures for detecting watermarks across reviews. The statistical detection approach controls family-wise error while retaining power, using an optimization-based procedure and a greedy heuristic with error bounds under human-written-review null hypotheses.

2.1.1 Choice of watermark

  • Watermarks are designed to be statistically testable, independent of human-review variability, reliably embedded, inconspicuous, and resilient to paraphrasing.The criteria aim to support reliable detection without making the watermark easy for reviewers to notice or remove.Source: §2.1 · p. 3, §2.1.1 · pp. 3–4

2.1.2 Statistical testability

  • Historical linguistic features risk systematic false accusations as writing styles evolve, while testing thousands of reviews amplifies false positives and can make Bonferroni-style corrections impractical.The method therefore samples a watermark independently of the review-writing process and prioritizes FWER control without relying on human-writing patterns.Source: §2.1.2 · p. 4

2.1.3 Watermarks employed

  • The three watermark types are a random coherent opening phrase, a quoted term sampled from the 1,000 least frequent ICLR 2024 keywords, and a fabricated surname-year citation.The random-start scheme has 1,200 possible phrases; the citation scheme pairs one of 9,999 surnames with a year from 2014–2024, yielding 109,989 possibilities.Source: §2.1.3 · p. 5, Table 1 · p. 5

2.2 Indirect prompt injection

  • PDF injections aim to embed the watermark across popular LLMs while remaining hidden and robust to reviewer behavior, using white text, font embedding, or small-font French text.Font embedding makes the underlying prompt differ from the text a human sees; the French-text approach is intended for cases where OCR defeats other attacks.Source: §2.2 · pp. 5–7

Cryptic prompt injection

  • Cryptic injection uses Greedy Coordinate Gradient [ZWC+23] to optimize seemingly nonsensical text that induces the LLM to place a predefined watermark at the review’s beginning.The described input appends cryptic text to the paper and uses randomly generated surname-year watermark strings.Source: Cryptic prompt injection · p. 7, Table 2 · p. 7

2.3 Statistical Detection

  • For one review, Algorithm 1 flags only when the chosen watermark appears and at most k candidates occur; setting k = α|W| bounds FPR by α.The watermark is sampled independently and uniformly from the candidate set, so the bound does not rely on assumptions about human writing styles.Source: §2.3 · pp. 7–9
  • Algorithm 2 adaptively reallocates the family-wise error budget using watermark occurrences, controlling FWER while retaining power in cases where Bonferroni would flag no reviews.It processes reviews without access to the randomly chosen watermarks until the final check; its FWER constraint also bounds the expected fraction of human-written reviews flagged by α.Source: §2.3 · pp. 8–9
  • The greedy heuristic iteratively considers the review containing the most watermarks and the watermark appearing in the most reviews, choosing a removal by objective-increase-to-constraint-reduction ratio.The choice must also respect the user’s limits on discarded reviews and watermarks.Source: §2.3 · pp. 9–10
  • The multiple-review procedure forms a binary review–watermark occurrence matrix, discards reviews and watermarks within user-specified limits to satisfy the error constraint, then flags qualifying retained reviews.If the optimization is infeasible, it returns an error.Source: §2.3 · p. 10
  • For one review, uniformly selecting a watermark independently of the human-written review bounds false alarms by 1/|W| when insertion and detection use a specified position.At that position, a review can contain at most one watermark from W.Source: §2.3 · p. 10
  • For multiple human-written reviews, the family-wise error bound follows from the constraint in the optimization problem; the argument also extends to random reviews by conditioning on their realization.Reviews discarded by the procedure are not flagged.Source: §2.3 · p. 11

3 Experimental evaluation

The evaluation tests watermark embedding across LLMs, injection techniques, and review datasets. Random citations with white-text injection achieve the strongest reported accuracy across models.

3 Experimental evaluation

  • The evaluation covers random-start, technical-term, and random-citation watermarks with white text, other-language text, or font embedding across several LLMs.The paper also compares API-generated reviews with a smaller GPT-4o WebApp sample, which produced very similar results.Source: §3 · p. 12, §3.1 · p. 12

3.1 Evaluating the efficacy of watermarking & prompt injection

  • The study evaluates watermarking and prompt injection on randomly selected ICLR 2024 submissions, using 100 papers for API experiments and 30 for GPT-4o WebApp and font-embedding pilots.Each paper received a different watermark; the API experiments report bootstrap 95% confidence intervals.Source: §3.1 · p. 12, Table 3 · p. 13
  • 98.6% ± 1.2% accuracy was achieved by random-citation watermarking with white-text injection, the best result across models.The experiments report bootstrap 95% confidence intervals.Source: §3.1 · p. 12, Table 3 · p. 13
  • Technical-term watermarking did not work with other-language injection, while random citations generally achieved higher accuracy than the other watermark strategies.For random-start watermarks with other-language injection, 80%-similarity accuracy ranged from 0.34 ± 0.09 to 0.40 ± 0.1 across the four reported model interfaces.Source: §3.1 · p. 12

3.2 Evaluating resilience against reviewer defenses

The experiments examine whether paraphrasing, watermark detection, or cropping the end of a paper affects injected watermarks. Random-start watermarks do not survive paraphrasing, while random citations persist more often.

3.2.1 Defense strategy 1: paraphrasing

  • Random-start watermarks did not remain after paraphrasing, whereas random citations remained in a higher percentage of paraphrased reviews.Paraphrased accuracy is measured only among reviews that initially contained the watermark; Table 4a reports 95% confidence intervals.Source: §3.2.1 · p. 13

3.2.2 Defense strategy 2: identifying watermarks

  • LLMs detected no random-start instructions and identified only a small percentage of technical-term and random-citation instructions.In some cases, an LLM continued to include the watermark when prompted to generate a review, despite previously identifying it.Source: §3.2.2 · pp. 13–14
  • The watermark-detection test asked an LLM to inspect an injected paper for hidden instructions, with lower identification accuracy indicating greater instruction robustness.The experiment used ChatGPT 4o API on 100 randomly selected ICLR papers.Source: §3.2.2 · p. 13

3.2.3 Defense strategy 3: crop out end of paper

  • To test resistance to end-page cropping, the study placed font-embedding instructions on page 11 and white-text instructions on page 7, then evaluated both with ChatGPT 4o API.The supplied passages describe the test setup but do not report its outcome.Source: §3.2.3 · p. 14

3.3 Cryptic prompt injection

Cryptic prompt injection embeds predefined watermarks in generated reviews, with success measured at both paper and individual-review levels. Its performance varies by model, dataset, and optimization length.

3.3 Cryptic prompt injection

  • The cryptic prompts were optimized for Llama 2 and Vicuna 1.5 using GCG, with inputs truncated to 2,000 tokens and 30 optimizable prompt tokens.Experiments used PRC 2022 abstracts and PeerRead papers, with 20 examples per dataset for each model.Source: §3.3 · p. 14
  • HPSR counts papers with a watermark in at least 8 of 10 independently generated reviews; OSR is the proportion of all generated reviews containing it.Source: §3.3 · p. 15
  • At 6,000 iterations, Llama 2 reached HPSR 19/20 and OSR >0.90 on both datasets; Vicuna 1.5 reached HPSR 20/20 and OSR 1.00 on PeerRead.Table 5 reports performance across both models, both datasets, and optimization iterations.Source: §3.3 · p. 15, Table 5 · p. 15
  • Vicuna 1.5 reached HPSR 19/20 and OSR 0.95 on PRC abstracts at 4,000 iterations, but its performance declined at 6,000 iterations.For PeerRead papers, it reached HPSR 20/20 and OSR 1.00 at 6,000 iterations.Source: §3.3 · p. 15

3.4 Grant proposals

The authors tested watermarking on NSF grant proposals using GPT-4o-generated reviews and report reasonably high accuracies across the strategies.

3.4 Grant proposals

  • Across 52 NSF proposals, GPT-4o API reviews were tested with three watermarking strategies, using white-text injection.Source: §3.4 · p. 15
  • The reported grant-proposal accuracies are reasonably high, with results summarized in Table 6.Source: §3.4 · p. 15, Table 6 · p. 16
  • Table 6 reports accuracies with 95% confidence intervals, computed using 10,000 bootstrap resamples.Source: Table 6 · p. 16

3.5 Control condition

The control-condition experiments assess false flags on reviews without inserted watermarks and test error control at both individual-review and family-wise levels.

3.5 Control condition

  • The unwatermarked control reviews came from 10,022 ICLR 2021 reviews and 28,028 ICLR 2024 reviews.Watermark occurrences in these unaugmented review sets are also reported as control-setting context.Source: §3.5 · pp. 16–17
  • For individual-review tests, empirical FPR stayed within its theoretical bound; random-citation and random-start watermarks flagged 100% of LLM-generated reviews with zero false flags.Table 8 reports FPR and TPR for tests using different thresholds.Source: §3.5 · p. 17
  • Under FWER control, Algorithm 2 bounds error at 0.01 for random-start and technical-term watermarks and 0.001 for random citation, with zero false flags and >90% TPR for random citation.Bonferroni was infeasible at FWER 0.05; the random-citation watermark’s large set (109,989) supports high TPR even at α = 0.001.Source: §3.5 · p. 17, Table 9 · p. 17

4 Discussion

The discussion presents indirect prompt injection as a way to detect LLM-generated peer reviews, paired with statistical tests and empirical evaluation. It also identifies open questions about broader use and misuse.

4 Discussion

  • The framework repurposes indirect prompt injection for detecting LLM-generated reviews, with formal FWER control independent of human-writing variability and higher power than standard corrections.The reported evaluations also find high watermarking success and strong detection accuracy under paraphrasing defenses.Source: §4 · p. 18
  • The authors identify adapting the approach to general text generation and scalable FWER control across documents as directions for future research.Source: §4 · p. 18
  • Authors could exploit indirect prompt injection to bias LLM-generated reviews in their favor; the paper presents a preliminary evaluation of this threat.Source: §4 · p. 18

A Examples of outputs of LLMs in various experimental settings

The section presents review examples from random-start, random-citation, and technical-term watermarking, alongside paraphrased reviews compared with their original LLM-generated versions.

A Examples of outputs of LLMs in various experimental settings

  • Table 10 gives examples of reviews using the random-start watermark.Source: Table 10 · p. 22
  • Table 11 presents examples of reviews using the random technical-term watermark.Source: Table 11 · p. 23
  • Table 12 shows random-citation watermark examples, including citations placed within the review rather than only at its start.The authors primarily experimented with starting reviews with a random fake reference, but also include examples placing one in the middle.Source: Table 12 · p. 24, A Examples of outputs of LLMs in… · p. 24
  • Table 13 pairs paraphrased reviews with the original LLM-generated reviews.Source: Table 13 · p. 25

B Greedy coordinate gradient algorithm

The section adapts Greedy Coordinate Gradient (GCG) to optimize text inserted into the model’s input so its response is more likely to begin with a chosen watermark. It describes the token-probability objective and an iterative candidate-selection procedure.

B Greedy coordinate gradient algorithm

  • An LLM is represented as a conditional distribution for its next token given the preceding token sequence.The sequence contains tokens generated up to time step i, and the distribution describes possible choices for token i + 1.Source: B Greedy coordinate gradient algorithm · p. 26
  • The optimized input appends OPT to the system prompt, abstract, and user prompt to maximize the probability that the response starts with watermark sequence W.Source: B Greedy coordinate gradient algorithm · p. 26
  • The objective is optimized by minimizing the negative log-likelihood of the watermark tokens with respect to the optimizable tokens.Source: B Greedy coordinate gradient algorithm · p. 26
  • GCG initializes the optimizable sequence with placeholder tokens, then iteratively substitutes high-gradient token candidates and keeps the candidate sequence with the lowest loss.Experiments use k = 256 top tokens and 512 candidate sequences per iteration.Source: B Greedy coordinate gradient algorithm · p. 26

C Inducing LLMs to provide higher ratings in generated reviews

The section evaluates whether an injected instruction can make generated reviews more favorable. Mean ratings were higher with the instruction than without it for both ICLR 2024 paper groups tested.

C Inducing LLMs to provide higher ratings in generated reviews

  • The section examines author-side prompt injection as an attack on LLM reviewers, distinct from prior work that modified paper content, specifically the checklist [GUK+24b].Source: C Inducing LLMs to provide higher… · p. 26
  • The study tests whether prompt injection can make LLM reviews more favorable on 50 uniformly random ICLR 2024 papers and 50 rejected papers, requesting ratings after injecting a 9/10-or-10/10 instruction.Source: C Inducing LLMs to provide higher… · p. 26
  • Mean ratings increased by 1.24 (17.71%) for all ICLR 2024 submissions and 1.64 (23.56%) for rejected papers with injection versus no injection.The comparison uses 50 randomly selected ICLR 2024 papers and 50 rejected ICLR 2024 papers; the injected line requested a rating of 9/10 or 10/10.Source: C Inducing LLMs to provide higher… · pp. 26–27, Table 14 · p. 27

Related papers

Loading 2503.15772v2…