PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimization-based Prompt Injection Attack to LLM-as-a-Judge

Jiawen Shi, Zenghui Yuan, Yinuo Liu, Yue Huang, Pan Zhou, Lichao Sun, Neil Zhenqiang Gong

arXiv:2403.17710v5cs.CRcs.AI

TL;DR

LLM-as-a-Judge can be manipulated through attacker-controlled candidate responses, motivating a method that automatically generates injection sequences. JudgeDeceiver formulates this attack as an optimization problem and achieves strong performance, while evaluated defenses miss many attacks.

  • Problem

    LLM-as-a-Judge is vulnerable to prompt injection when candidate responses come from untrusted sources, allowing attackers to steer the judge toward a chosen response.

  • Method

    JudgeDeceiver constructs a shadow candidate dataset and optimizes an injected sequence using target-aligned and target-enhancement losses to force a target judgment.

  • Results

    JudgeDeceiver outperforms manual prompt injection and extended jailbreak attacks, while known-answer and perplexity-based defenses fail to detect a large fraction of injected responses.

  • Takeaways & Limitations

    The results expose risks in deploying LLM-as-a-Judge across LLM-powered search, RLAIF, and tool selection and motivate new defense mechanisms.

  • Takeaways & Limitations

    The authors identify further enhancing the stealthiness and semantics of injected sequences as future work.

Abstract

from arXiv · show

LLM-as-a-Judge uses a large language model (LLM) to select the best response from a set of candidates for a given question. LLM-as-a-Judge has many applications such as LLM-powered search, reinforcement learning with AI feedback (RLAIF), and tool selection. In this work, we propose JudgeDeceiver, an optimization-based prompt injection attack to LLM-as-a-Judge. JudgeDeceiver injects a carefully crafted sequence into an attacker-controlled candidate response such that LLM-as-a-Judge selects the candidate response for an attacker-chosen question no matter what other candidate responses are. Specifically, we formulate finding such sequence as an optimization problem and propose a gradient based method to approximately solve it. Our extensive evaluation shows that JudgeDeceive is highly effective, and is much more effective than existing prompt injection attacks that manually craft the injected sequences and jailbreak attacks when extended to our problem. We also show the effectiveness of JudgeDeceiver in three case studies, i.e., LLM-powered search, RLAIF, and tool selection. Moreover, we consider defenses including known-answer detection, perplexity detection, and perplexity windowed detection. Our results show these defenses are insufficient, highlighting the urgent need for developing new defense strategies. Our implementation is available at this repository: https://github.com/ShiJiawenwen/JudgeDeceiver.

1 INTRODUCTION

LLM-as-a-Judge is vulnerable to prompt injection because candidate responses can originate from untrusted sources. JudgeDeceiver automatically optimizes injected sequences to manipulate judgments, outperforming manual and extended jailbreak attacks while evading existing detection defenses.

  • Motivation: LLM-integrated applications are vulnerable when attackers place prompts in untrusted external data processed by the model.Candidate responses in LLM-as-a-Judge can originate from such untrusted sources, allowing attackers to steer the judge toward a chosen response.
  • Contribution: JudgeDeceiver is the first optimization-based prompt injection attack targeting LLM-as-a-Judge.It automatically generates injected sequences instead of relying on labor-intensive manual construction.
  • Attack mechanism: The injected target response can become the optimal choice regardless of other candidate responses and remain effective across position-swapping evaluations.This consistency is intended to avoid suspicion from positional evaluation defenses.
  • Optimization approach: JudgeDeceiver formulates sequence generation as minimizing a weighted sum of three loss terms, using a shadow candidate dataset and target-aligned generation loss.The target-aligned loss reduces the difference between the judge’s output and an explicit target judgment such as “Output (index) is better.”
  • Optimization approach: A target-enhancement loss increases the likelihood of the target response’s index token despite random target positions among shadow candidates.The additional term addresses the increased optimization complexity caused by varying positional indices.
  • Evaluation: 90.8% average attack success rate and 83.4% positional attack consistency were achieved on MT-bench with Mistral-7B.The evaluation compared JudgeDeceiver with six manual prompt injection methods across four LLMs and two benchmark datasets.
  • Defenses: PPL-W missed 70% of target responses with injected sequences on Llama-3-8B while falsely detecting fewer than 1% of clean responses.Known-answer detection failed to detect target responses, while PPL and PPL-W detected only some cases.

2 PROBLEM FORMULATION

The paper formalizes LLM-as-a-Judge as selecting the best response for a question and defines a threat model in which an attacker appends an optimized sequence to force selection of a chosen response. It considers leaderboard, search, RLAIF, and tool-selection settings.

  • 2.1 LLM-as-a-Judge: LLM-as-a-Judge selects the candidate response that most accurately and comprehensively addresses a question.The question and candidate responses are combined into a prompt for the judge.
  • 2.1 LLM-as-a-Judge: The sandwich-prevention template places the question and responses between header and trailer instructions to improve precision and prevent prompt injection.The judge outputs a sentence containing the index of the selected response.
  • 2.1 LLM-as-a-Judge: The formalization concatenates p_header, q, candidate responses, and p_trailer into one input whose judgment output identifies the best-response index.The paper denotes concatenation with ⊕ and the judgment sentence with o_k.
  • 2.2 Threat Model: The attacker selects a target response r_t from the candidate set for a target question q, even when that response is inaccurate or malicious.The attack objective is defined by the pair (q, r_t).
  • 2.2 Threat Model: The attacker appends an l-token sequence δ to the target response as a prefix, suffix, or combination of both.The resulting response is written A(r_t, δ).
  • 2.2 Threat Model: The attack succeeds when the judge outputs the attacker-chosen judgment o_t and selects r_t as the optimal response.Here, t is the target response’s index within the input prompt.
  • 2.2 Threat Model: The threat model covers leaderboard manipulation, increased webpage visibility or information control in LLM-powered search, malicious data dissemination in RLAIF, and greater tool adoption.The scenarios also include software click-through, profit, and competitive-market incentives for tool-selection attacks.

3 JUDGEDECEIVER

JudgeDeceiver is an automated optimization framework that crafts an injected sequence to steer LLM-as-a-Judge toward an attacker-chosen target response. It uses shadow candidate responses, multiple loss terms, gradient-based token optimization, and positional adaptation to improve attack effectiveness and consistency.

  • Overview: JudgeDeceiver constructs shadow candidate response sets to simulate evaluation scenarios when the attacker cannot access real candidate responses.The shadow responses provide a basis for generating sequences that generalize across possible candidate sets.
  • Optimization objective: JudgeDeceiver minimizes a weighted objective combining target-aligned generation, target-enhancement, and adversarial perplexity losses.The losses respectively steer the judgment, emphasize the target response’s positional index, and reduce the injected sequence’s perplexity impact for greater stealth.
  • Optimization objective: The attack optimizes an injected sequence so the judge generates an attacker-defined target output identifying the target response as optimal.The target response is selected after the sequence is injected, regardless of the other clean responses in the evaluated set.
  • Optimization algorithm: Gradient descent iteratively substitutes injected-sequence tokens, using gradient-based candidate selection and greedy coordinate search to reduce the total loss.The method evaluates candidate replacements and updates the sequence over successive iterations.
  • Optimization algorithm: Positional adaptation optimizes the injected sequence across target-response indices and progressively adds candidate sets to accelerate optimization.Optimization is complete when the sequence consistently succeeds across the relevant positional indices.

4 EVALUATION

The evaluation measures JudgeDeceiver against manual injection and jailbreak baselines across multiple LLM judges and datasets, finding strong attack success and positional consistency. Ablations examine response-set size, loss terms, initialization, sequence location, hyperparameters, and transferability.

  • Experimental Setup: The evaluation uses two datasets, four open-source LLM judges, six manual prompt-injection baselines, and extended jailbreak attacks.The setup optimizes 20-token suffixes for each target question-response pair using three shadow responses and 600 iterations.
  • Main Results: JudgeDeceiver reaches average ASRs of 89.2%–98.9% across models on MT-Bench and LLMBar, including 100% on selected target pairs.Reported averages include 90.8% and 93.2% for Mistral-7B, and 97.6% and 97% for Llama-3-8B.
  • Main Results: Positional attack consistency remains high after swapping target and clean-response positions, ranging from 79% to 97.8% across evaluated models.The reported PACs are 83.4% and 86.6% for Mistral-7B and 95.6% and 94.4% for Llama-3-8B.
  • Baseline Comparisons: JudgeDeceiver outperforms manual prompt injections, whose attack success varies considerably because their fixed templates lack generalization and robustness.The comparison is reported across different target LLM judges.
  • Baseline Comparisons: Against jailbreak attacks, JudgeDeceiver improves ASR by 37.2%–78.7% on MT-Bench and 43.3%–92.1% on LLMBar while using 20-token suffixes.TAP generates average suffix lengths of 115.3 and 132.4 on the two datasets; JudgeDeceiver also exceeds jailbreak PACs by over 57.6% and 61.8%.
  • Ablation and Sensitivity: Attack success decreases as evaluated candidate count n increases, but the decline eases with more shadow responses m; performance is higher when n ≤ m.Increasing shadow responses improves coverage but requires more computational resources and GPU memory.
  • Ablation and Sensitivity: Removing aligned-generation or enhancement losses substantially affects attack success, while perplexity regularization lowers ASR by 1% but improves concealment.The perplexity term constrains fluency and rationality during token search.

5 CASE STUDIES

The case studies apply JudgeDeceiver to LLM-powered search, RLAIF annotation, and tool selection. These settings treat search results, preference data, or tool descriptions as candidate responses evaluated by an LLM-as-a-Judge.

  • 5.1 Attacking LLM-powered Search: In LLM-powered search, the question is a user query and candidate responses are search-result entries evaluated for relevance and accuracy.The study uses five diverse queries and contradictory Google search entries as target responses.
  • 5.2 Automated Annotator on RLAIF: RLAIF uses LLM-as-a-Judge to evaluate outputs for preference data, addressing the labor-intensive and time-consuming nature of human annotation.The supplied passage frames RLAIF as a scalable alternative to conventional human-curated preference datasets.
  • 5.3 Attacking Tool Selection: In tool selection, the question is a user inquiry and candidate responses are descriptions of integrated tools evaluated for suitability.The experiment uses MetaTool-derived queries and irrelevant target tool descriptions, with candidate counts of 3, 4, and 5.
  • 5.3 Attacking Tool Selection: Tool-selection attacks achieve 100% ASR across a majority of evaluated tools, and all three-target-tool cases retain 100% ASR.Mistral-7B has ASR-B of 0% across the tested scenarios.

6 DEFENSES

The paper examines prevention- and detection-based defenses against prompt injection, focusing on known-answer, perplexity, and windowed-perplexity detection. Experiments show these defenses miss substantial fractions of JudgeDeceiver’s injected responses, while perplexity-based detection can maintain low false-positive rates.

  • Defense strategies: Detection-based defenses include known-answer detection, PPL detection, and PPL-W detection, while preprocessing-based prevention has limited effectiveness or can sacrifice model generality.The study therefore primarily evaluates detection-based defenses.
  • PPL detection: PPL detection flags responses whose perplexity exceeds a dataset-adaptive threshold calibrated to keep the false-positive rate at or below 1%.Thresholds are selected from 100 clean samples per dataset using log-perplexity values.
  • PPL-W detection: PPL-W divides responses into contiguous windows and labels a response injected when any window exceeds the detection threshold.The experiments use window size 10 and the same dataset-adaptive threshold strategy as PPL detection.
  • Known-answer detection: Known-answer detection misses JudgeDeceiver targets, with false-negative rates of 100% on MTBench and 90% on LLMBar.The detection instruction tests whether the model still follows a prescribed answer request when combined with the response.
  • Experimental results: PPL and PPL-W achieve low false-positive rates but miss 40%–80% of MTBench targets and 60%–90% of LLMBar targets.The overlap between target and clean-response perplexities explains why many injected responses remain undetected.
  • Limitations: The authors acknowledge that further work is needed to enhance the stealthiness of the injected sequences.This limitation concerns the attack’s concealment against perplexity-based detection.

7 RELATED WORK

Related work covers the development and applications of LLM-as-a-Judge, manual prompt injection attacks, gradient-based adversarial prompt generation, and prevention-based defenses. JudgeDeceiver builds on these lines by targeting judge decisions with automatically optimized injected sequences.

  • LLM-as-a-Judge: LLM-as-a-Judge has been developed for open-ended evaluation and extended to translation, story generation, safety, multidimensional assessment, and multimodal tasks.Prior work also investigates fairness and effectiveness of LLM evaluators.
  • Prompt injection attacks: Manual prompt injection methods include concatenation, escape characters, context ignoring, fake completion, and combined attacks.These methods are designed as general strategies for LLM-integrated applications.
  • Adversarial prompt generation: Gradient-based adversarial prompt research addresses discrete NLP search through continuous feature mappings and token-level optimization.Examples include HotFlip and AutoPrompt, followed by methods such as GCG and AutoDAN.
  • Defense methods: Prevention-based defenses preprocess instructions or data to isolate potential instructions and clarify what constitutes an injection.These defenses aim to improve execution of the intended task against malicious sequences.

8 CONCLUSION

The conclusion identifies LLM-as-a-Judge as vulnerable to prompt injection and presents JudgeDeceiver as an optimization-based framework for manipulating its judgments. The evaluation reports stronger performance than manual and jailbreak attacks, while existing detection defenses remain insufficient and motivate further stealth and defense research.

  • Conclusion: The paper shows that LLM-as-a-Judge is vulnerable to prompt injection attacks.The vulnerability arises because injected sequences can manipulate the judge’s decision.
  • Conclusion: JudgeDeceiver automatically generates injected sequences that manipulate LLM-as-a-Judge judgments.The framework formulates sequence generation as an optimization problem.
  • Conclusion: JudgeDeceiver outperforms manual prompt injection and jailbreak attacks when extended to the LLM-as-a-Judge setting.Known-answer detection is insufficient, while perplexity-based defenses miss a large fraction of injected sequences.
  • Future work: Future work includes improving injected-sequence semantics for stealth and developing defenses against JudgeDeceiver.These directions follow the reported limitations of current stealth and defense mechanisms.

A APPENDIX

The appendix algorithm optimizes an injected sequence using shadow candidate-response datasets and iterative token replacement. It evaluates candidate replacements across response positions and retains the batch replacement minimizing loss across the current shadow datasets.

  • Algorithm inputs: JudgeDeceiver takes a target question, shadow candidate-response datasets, a target response, an initial injected sequence, batch size, and iteration count as inputs.Its output is the optimized injected sequence.
  • Initialization: The algorithm initializes counters for the shadow-dataset index and optimization iterations before processing candidate replacements.This starts the iterative optimization procedure.
  • Loss calculation: For each shadow response set, it sums target-response losses across different target-response position indices.The positional evaluation supports robustness to where the target response appears.
  • Candidate generation: For each injected-sequence token, the algorithm forms Top-K replacement candidates from the negative gradient of total loss across candidate-response sets.Gradient information guides discrete token replacement.
  • Batch proposals: It randomly selects a token position and replacement candidate to form each batch proposal.Multiple proposals are generated for comparison.
  • Selection and stopping: The algorithm keeps the batch replacement that minimizes summed loss across all shadow datasets, then checks whether the attack succeeds across all position indices.This combines optimization with an all-position attack-success criterion.

A.1 Convergence of Initial Types

Initialization affects both optimization convergence and attack effectiveness. Character initialization converges slowest, while Sentence converges fastest but performs worse than Word on attack metrics.

  • Character initialization has the slowest convergence because adversarial perplexity constraints shift character tokens toward word tokens.
  • Sentence initialization converges fastest and starts with the lowest loss.
  • 81% ASR and 62% PAC for Sentence initialization are lower than Word initialization.
  • Sentence initialization can fall into a local sub-optimal solution because its tokens have a large distribution range.

A.2 Target Responses with and w/o L𝑝𝑒𝑟𝑝𝑙𝑒𝑥𝑖𝑡𝑦

Adding the perplexity loss produces injected sequences that are less conspicuous than those generated without that loss.

  • Perplexity loss changes injected sequences from symbol-heavy strings to more standardized phrases with higher concealment.The resulting sentence may still be ungrammatical, but its abnormalities are reduced.

A.3 Different 𝛽in PPL-W Detection

The perplexity-windowed detection trade-off depends on β: values below 0.5 reduce log-perplexity as β increases, whereas larger values increase it and weaken the attack.

  • For β below 0.5, log-perplexity decreases as β increases across four filter-window settings, making the injection harder to detect.
  • When β increases from 0.5 to 1.0, log-perplexity increases because the injected sequence becomes less correlated with the target response.The injected sequence itself may have low perplexity, while the overall target response has higher perplexity.
  • β values above 0.5 also produce poor attack effectiveness, so the authors recommend considering smaller β values.

A.4 Transferability of Different Injected Sequences Across Different QR Pairs

The optimized injected sequences work best for their corresponding question-response pairs but also transfer across some different pairs. The appendix also documents the benchmark examples and application-specific data used in the evaluations.

  • A.4 Transferability of Different Injected Sequences Across Different QR Pairs: The attack achieves higher ASRs when each optimized injected sequence is evaluated on its corresponding question-response pair.This pattern appears as the dark-red diagonal in the transferability heat map.
  • A.4 Transferability of Different Injected Sequences Across Different QR Pairs: The MTBench and LLMBar experiments use manually selected questions spanning role-playing, reasoning, and information retrieval.Target responses were generated using the settings described in Appendix A.5.
  • A.4 Transferability of Different Injected Sequences Across Different QR Pairs: The three case studies cover LLM-powered search, RLAIF, and tool selection using five target examples in each scenario.The examples are organized as query-entry, instruction-response, and inquiry-tool pairs, respectively.
Loading 2403.17710v5…