Source-linked AI summary
Prompt Injection Attacks on LLM Generated Reviews of Scientific Publications
Janis Keuper
TL;DR
The paper asks whether hidden prompt injections can practically manipulate LLM-generated peer reviews, amid debate about LLM use in review. It evaluates these attacks and finds that simple injections can dominate review outcomes, while LLM reviews also tend toward positive scores.
Problem
The practical success of hidden prompt injections against LLM-generated reviews remains uncertain, even as LLM use in peer review fuels debate.
Source: §1 · pp. 1–2Method
The authors evaluate prompt injections in LLM reviews using real review data and human baselines, with PDFs parsed into Markdown and prompts varied across neutral, positive, and negative conditions.
Source: §1 · p. 2, §2.2 · p. 3, §2.4 · p. 3Results
Simple prompt injections can dominate LLM review outcomes, and the tested LLMs show a strong positive bias even without manipulation.
Source: §4 · p. 8Takeaways & Limitations
In the tested setting, LLM reviews pose concerns both because simple prompt injections can affect outcomes and because unmanipulated reviews tend toward acceptance.
Source: §4 · p. 8The study models a careless reviewer using publicly available LLMs, so its results may not generalize to specifically fine-tuned review models.
Source: §4 · p. 9
Abstract
from arXiv · showhide
The ongoing intense discussion on rising LLM usage in the scientific peer-review process has recently been mingled by reports of authors using hidden prompt injections to manipulate review scores. Since the existence of such "attacks" - although seen by some commentators as "self-defense" - would have a great impact on the further debate, this paper investigates the practicability and technical success of the described manipulations. Our systematic evaluation uses 1k reviews of 2024 ICLR papers generated by a wide range of LLMs shows two distinct results: I) very simple prompt injections are indeed highly effective, reaching up to 100% acceptance scores. II) LLM reviews are generally biased toward acceptance (>95% in many models). Both results have great impact on the ongoing discussions on LLM usage in peer-review.
1 Using LLMs to write Reviews: mostly forbidden - widely applied.
Growing review workloads and broad LLM availability have intensified debate about LLM use in peer review, while hidden prompt injections raise a question about whether they can manipulate LLM-generated scores. The paper examines that practical effectiveness against a backdrop of research on automated reviewing, review quality, and LLM-text detection.
1 Using LLMs to write Reviews: mostly forbidden - widely applied.
- Growing review duties and LLM availability encourage reviewers to use LLMs, despite venue prohibitions and difficulty proving that a review was generated by an LLM.Some conferences and journals are also experimenting with LLM-assisted review processes.Source: §1 · p. 1
- Hidden strings can be rendered invisible to human readers yet parsed by LLMs as ordinary text, leaving their effect on generated reviews to be tested.The strings can be embedded as white text on white backgrounds or in tiny fonts.Source: Figure 1 · p. 1, §1 · p. 2
- The paper presents a first detailed analysis of simple prompt injections’ practical effectiveness, using real review data and human baselines to assess risks to review scores.The authors frame these findings as having strong practical implications for LLM use in peer review.Source: §1 · p. 2
1.1 Related Work
- Prior work has examined automated-review systems and multi-agent or multi-turn review processes, including discussion phases.Specialized systems include Openreviewer (Idahl & Ahmadi, 2024), Deepreview (Zhu et al., 2025), and Reviewer2 (Gao et al., 2024).Source: §1.1 · p. 2
- Studies report that LLM reviews can assist reviewers but still diverge from human scoring and hallucinate arguments or citations; separate work reports benefits from LLM feedback on review length and detail.Sources include Zhou et al. (2024), Liang et al. (2024), Tyser et al. (2024), and Thakkar et al. (2025).Source: §1.1 · p. 2
- Research on detecting LLM-generated reviews reports that identifying LLM text with high certainty is difficult.The cited studies are Yu et al. (2024), Wu et al. (2025), and Tang et al. (2024).Source: §1.1 · p. 2
2 Experimental Setup
The experiments model a reviewer who submits a paper PDF to a general-purpose LLM and receives a structured review. The setup applies one pipeline to ICLR 2024 submissions, varying the prompt injection and evaluating a range of commercial and open models.
2 Experimental Setup
- All models use the same pipeline: PDFs are parsed into Markdown and sent through structured prediction calls with shared prompts, approximating a reviewer who directly submits a PDF to an LLM.Structured outputs are intended to support copying answers into review-form fields.Source: §2 · p. 2
2.1 Data
- The sample comprises 1,000 randomly selected ICLR 2024 initial submissions and their 3–4 first-round reviews per paper, excluding desk-rejected or withdrawn submissions.The human reviews are initial responses, not post-rebuttal updates or final decisions.Source: §2.1 · p. 3
2.2 Document Parsing
- Mistral OCR converts each PDF to Markdown, extracting text and tables while representing figures as images; the resulting Markdown is fed to the LLMs.The authors also evaluate the parser’s vulnerability to prompt injections.Source: §2.2 · p. 3
2.3 Structured Output Prediction
- Structured prediction calls request reviews in a JSON schema reflecting the ICLR review form, though some weaker models fail to follow that schema.The schema includes summary, ratings, strengths, weaknesses, and questions.Source: §2.3 · p. 3
2.4 Prompts
- The common prompts pair a system prompt with the parsed paper and use three injection conditions: neutral, positively biased, and negatively biased.The injected messages ask for high or low scores and emphasize strengths or weaknesses, respectively.Source: §2.4 · pp. 3–4
- The requested review includes an overall rating, soundness, presentation, contribution, strengths, and weaknesses.The overall-rating choices range from strong reject to strong accept.Source: §2.4 · p. 3
2.5 Models
- The evaluation spans commercial models from OpenAI, Google, and Mistral, alongside locally hosted open models including Qwen3, LLAMA3.1, and DeepSeek R1.The models vary in provider and size.Source: §2.5 · p. 4
3 Results
The results test whether PDF parsing exposes hidden prompt injections, whether models meet structured-output constraints, and how injections affect ratings and review content. Simple biased prompts shift many models’ acceptance outcomes, while analyses also reveal acceptance bias and changes in review content.
3.1 Parsing Prompt Injections
- Text-based parsing exposes hidden prompts to review generation, whereas image-based OCR ignores them; Gemini’s web service uses image-based parsing, unlike ChatGPT.Source: §3.1 · p. 4
3.2 Structured Output Validation
- Table 2 measures invalid ratings by model, especially scores such as “4”; human reviewers have a 0% error rate because the form permits only valid scores.Source: Table 2 · p. 5
3.3 Effects of Prompt Injection
- For most models, acceptance reached 100% with positively biased prompts and fell to 0% with negatively biased prompts; some models barely reacted.Source: Table 3 · p. 5, §3.3 · p. 5
- Deepseek-r1:70b and llama3.1:70b barely respond to injections, and weak manipulation effects correlate with failures to follow structured-output instructions.Source: §3.3 · pp. 5–6, Table 5 · p. 8
- LLM summary embeddings were nearly as close to human summaries as human summaries were to one another, and prompts had little effect on them.Source: §3.3 · p. 6, Figure 3 · p. 6
- Even without injected prompts, LLM rating distributions show stronger acceptance bias than human reviews.Source: Figure 2 · p. 6, §3.3 · p. 6
- Human reviewers agree more on positive than negative aspects; LLM strengths are closer to human findings than weaknesses, and injections measurably shift argument lists.Positively biased generations move closer to human evaluations, leaving a larger gap for negatively biased generations.Source: Figure 3 · p. 6, §3.3 · p. 8
4 Discussion
Simple prompt injections can dominate LLM review outcomes, while models also show a strong positive bias. The discussion considers a limited parsing-stage defense and bounds the findings to the tested careless-reviewer scenario.
4 Discussion
- Simple prompt injections can dominate LLM review outcomes, with weak effects strongly correlated with models’ failure to follow structured-output instructions.The authors speculate that precise instruction-following may increase vulnerability; models that fail structured output are less useful for copy-and-paste review submission.Source: §4 · p. 8
- LLMs’ strong positive bias means authors may not need injections to counter reviewers’ LLM use in the experiments.The authors state that LLMs gave mostly positive reviews in their experiments.Source: §4 · p. 8
- Parsing PDFs as images could hide the tested human-unreadable injections from LLMs, but more elaborate injections may bypass this defense.The proposed defense applies at the document-parsing stage.Source: §4 · p. 9
- The findings concern careless reviewers using publicly available LLMs and may not generalize to specifically fine-tuned review models.Possible training exposure to ICLR papers and reviews is noted, but the authors consider its effects negligible given the shifts between human and LLM reviews.Source: §4 · p. 9
- The paper aims to raise awareness of shortcomings in unreflected LLM use, not to promote LLM reviewing or manipulation.Source: §4 · p. 9