Source-linked AI summary
Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment
Vyas Raina, Adian Liusie, Mark Gales
TL;DR
LLM judges are widely used for zero-shot assessment, but their vulnerability to adversarial manipulation had not been examined. The paper learns short universal attack phrases on a surrogate model and transfers them to other judges. These attacks can inflate scores, especially under absolute scoring, while perplexity offers an initial detection approach.
Problem
The paper addresses the limited study of adversarial robustness in zero-shot LLM assessment, which matters because such systems are used for benchmarking and written-exam evaluation.
Method
The authors learn short universal concatenation phrases with a surrogate model and transfer them to unknown judge-LLMs, using greedy search and perplexity-based detection.
Results
Absolute scoring is significantly more vulnerable than comparative assessment: four attack words can drive scores near the maximum of 5, while transferred attacks can make larger judges return maximum scores.
Takeaways & Limitations
Comparative assessment is a more effective defense than absolute scoring against the most threatening attacks, despite increased computational cost.
Takeaways & Limitations
The study focuses on simple concatenation attacks against zero-shot assessment; few-shot settings and subtler attacks remain for future work.
Abstract
from arXiv · showhide
Large Language Models (LLMs) are powerful zero-shot assessors used in real-world situations such as assessing written exams and benchmarking systems. Despite these critical applications, no existing work has analyzed the vulnerability of judge-LLMs to adversarial manipulation. This work presents the first study on the adversarial robustness of assessment LLMs, where we demonstrate that short universal adversarial phrases can be concatenated to deceive judge LLMs to predict inflated scores. Since adversaries may not know or have access to the judge-LLMs, we propose a simple surrogate attack where a surrogate model is first attacked, and the learned attack phrase then transferred to unknown judge-LLMs. We propose a practical algorithm to determine the short universal attack phrases and demonstrate that when transferred to unseen models, scores can be drastically inflated such that irrespective of the assessed text, maximum scores are predicted. It is found that judge-LLMs are significantly more susceptible to these adversarial attacks when used for absolute scoring, as opposed to comparative assessment. Our findings raise concerns on the reliability of LLM-as-a-judge methods, and emphasize the importance of addressing vulnerabilities in LLM assessment methods before deployment in high-stakes real-world scenarios.
1 Introduction
LLMs enable reference-free zero-shot assessment, but this paper investigates whether short universal phrases can manipulate judge-LLMs into assigning inflated scores. It introduces surrogate-model attacks targeting both absolute and comparative assessment.
- Zero-shot LLM assessment can evaluate texts without domain-specific training and is used for benchmarking models and written examinations.
- Prior studies identified positional, length, and self-preferential biases, while the adversarial robustness of LLM-as-a-judge remained less studied.
- The paper tests whether appending a simple universal phrase can make an LLM predict high scores regardless of the assessed text’s quality.
- A surrogate model learns the attack phrase, which is then transferred to judge-LLMs whose weights or query access may be unavailable.
- A 5-token universal phrase can substantially increase scores, while comparative assessment is more robust than absolute scoring.
2 Related Work
Related work covers automatic evaluation tailored to particular domains, emerging zero-shot LLM assessment, and adversarial attacks on generative systems. These strands motivate studying attacks against general-purpose LLM judges.
- Bespoke NLG Evaluation: Traditional NLG metrics compare generated texts with gold references but often correlate weakly with human assessments.
- Bespoke NLG Evaluation: Prior bespoke systems target specific domains or attributes, including dialogue, summary consistency, general summary assessment, and machine translation.
- Zero-Shot Assessment with LLMs: Zero-shot LLM methods assess texts through absolute scores, pairwise comparisons, or output language-model probabilities in general and unseen settings.
- Adversarial Attacks on Generative Systems: NLP adversarial-attack research traditionally focused on classification, while newer work addresses generative LLMs and jailbreaks.
3 Zero-shot Assessment with LLMs
The paper distinguishes pairwise comparative assessment from absolute scoring as two reference-free ways to prompt LLM judges. Comparative scores average both candidate orderings, while scoring directly predicts quality values.
- 3.1 Comparative Assessment: LLM comparative assessment uses pairwise comparisons to determine which of two responses is better.
- 3.2 Absolute Scoring Assessment: LLM scoring asks a model to assign an absolute score to each text, such as a value between 1 and 5.
- For ranking tasks, the judge evaluates N responses in context d and produces predicted qualities whose ranks can be compared with the texts’ true-quality ranks.
- 3.1 Comparative Assessment: Comparative probability p_ij averages both orderings of candidates x_i and x_j to account for positional bias.
- 3.1 Comparative Assessment: Two inference passes enforce symmetry, yielding p_ij = 1−p_ji before comparative probabilities are aggregated into predicted quality scores.
- 3.2 Absolute Scoring Assessment: Absolute scoring can directly predict a score or estimate expected quality by multiplying each possible score by its normalized output probability.
4 Adversarial Assessment Attacks
The attack appends a short phrase to candidate responses and optimizes it to improve attacked candidates’ predicted ranks across contexts. A greedy search learns one universal phrase that can transfer from a surrogate to target judges.
- 4.1 Attack Threat Model: An adversarial perturbation δ minimally changes input x while seeking a significant change in the assessment model’s output.
- 4.1 Attack Threat Model: The attack concatenates a phrase δ of length L much smaller than |x| to the original response.
- 4.1 Attack Threat Model: The objective is to minimize the attacked candidate’s predicted rank relative to other unperturbed candidates.
- 4.1 Attack Threat Model: A universal adversarial example applies the same phrase to any candidate response and consistently boosts its predicted assessment rank.
- 4.1 Attack Threat Model: The universal phrase optimizes expected rank across M contexts and N candidates rather than a single response.
- 4.2 Practical Attack Approach: The practical method greedily appends vocabulary tokens that maximize the attack objective, using a surrogate model when the target judge is inaccessible.
- 4.2 Practical Attack Approach: The paper reports that GCG was less effective than the greedy search for learning universal attack phrases.
5 Experimental Setup
Experiments use SummEval and TopicalChat to test universal concatenation attacks across comparative and absolute assessment, with FlanT5-xl as the surrogate and several target judge-LLMs. Attack phrases are learned by greedy search on development data and evaluated using candidate ranks after transfer.
- 5.1 Datasets: SummEval contains 100 passages with 16 machine-generated summaries each, assessed for coherency, consistency, fluency, relevance, and their average overall score.
- 5.1 Datasets: TopicalChat contains 60 dialogue contexts with six machine-generated responses each, assessed for coherency, continuity, engagingness, naturalness, and their average overall score.
- 5.2 Models: FlanT5-xl serves as the surrogate for learning attacks, which are transferred to Mistral-7B, Llama2-7B, and GPT3.5.
- 5.3 Methodology: A 20:80 development-test split supports greedy phrase search using two candidate texts per dataset, covering 40 summaries and 24 responses during search.
- 5.3 Methodology: Separate universal concatenation attacks are learned for each dataset, assessment method, and evaluation attribute using task-assessment-attribute notation.
- 5.3 Methodology: Attack success is measured by average candidate rank: unsuccessful attacks remain near average rank, whereas a very strong attack reaches average rank 1.
6 Results
The results show that short universal attack phrases substantially compromise LLM-based assessment, especially absolute scoring, while perplexity provides a promising but potentially circumventable detection defence.
- Assessment Performance: Comparative assessment generally performs better than absolute assessment across SummEval and TopicalChat, although absolute scoring can outperform smaller comparative systems with GPT3.5.Comparative assessment requires N·(N−1) pairwise comparisons, whereas absolute assessment requires only N inferences.
- Attack on Surrogate Model: Universal attacks reduce attacked-text rank in both assessment settings, with vulnerability increasing as attack phrases become longer.The attack evaluation uses average rank of the attacked summary or response.
- Attack on Surrogate Model: Four-word universal attacks make absolute assessment consistently assign rank 1 to nearly all inputs and produce scores near the maximum of 5.The reported four-word results concern FlanT5-xl, and comparative and absolute scores are not directly comparable.
- Transferability: The attack is learned on surrogate FlanT5-xl and transferred to target models, with transfer experiments focusing on absolute scoring because comparative assessment was relatively robust.The target models include Llama2, Mistral, and GPT3.5.
- Attack Detection: Perplexity detection achieves F1 scores near 0.7 or higher on SummEval and generally better performance on TopicalChat.Detection classifies samples using a perplexity threshold and evaluates precision-recall trade-offs.
- Attack Detection: The paper concludes that comparative assessment is an effective defence against the most threatening attacks, despite its increased computational cost.The authors also note that perplexity detection may be circumvented by adaptive attacks.
7 Conclusions
The paper finds that zero-shot LLM assessment is vulnerable to universal, transferable adversarial attacks, especially in absolute scoring. It also identifies perplexity as a promising detection approach while calling for stronger defenses.
- Absolute scoring shows significant vulnerability to universal adversarial attacks, while comparative assessment shows milder vulnerability.
- A short 4-word adversarial phrase can be appended to any input text to induce inflated scores.
- Attacks learned using a smaller surrogate scoring system can transfer effectively to larger models such as ChatGPT.
- Perplexity provides a promising initial approach for identifying adversarially manipulated inputs.
- The authors recommend further work on adaptive attacks and more sophisticated defenses to reduce misuse risks.
8 Limitations
The study focuses on simple concatenation attacks against zero-shot assessment methods, leaving broader attack and defense settings for future work.
- The evaluation uses simple concatenation attacks found through greedy search, paired with simple perplexity-based defenses.
- The study focuses on zero-shot assessment and does not establish how effective the attacks would be in few-shot settings.
- Future work should examine subtler attacks, more complex defenses, robust prompts, and few-shot assessment methods.
9 Risks & Ethics
The paper highlights the risk that universal adversarial attacks can inflate NLG assessment scores, while presenting the work as awareness-raising rather than harmful-content generation.
- Universal adversarial attacks can fool NLG assessment systems into inflating scores of assessed texts.
- The proposed methods and attacks are not intended to encourage harmful content generation.
- Highlighting these susceptibilities may inform adversaries, but the stated aim is to encourage robustness research and reduce future misuse.
- The attack phrases are documented across multiple configurations, with their constituent words listed in Table 5.
B Analysis of Relative Robustness of Comparative Assessment
Comparative assessment is more robust to universal adversarial phrases than absolute assessment. The asymmetric ablation and transfer results suggest that this robustness cannot be explained solely by competing objectives across prompt positions.
- Comparative assessment is more robust than absolute assessment against universal adversarial phrases.The paper attributes this contrast partly to comparative assessment requiring opposite objectives across two prompt orderings.
- In symmetric comparative evaluation, the attack must increase token A probability when the attacked summary is first and decrease it when second.The competing objectives require the phrase to recognize the attacked summary's position automatically.
- The asymmetric ablation keeps the attacked text in one position, so the attack optimizes a single token-probability objective.The experiments define corresponding phrases as asymA and asymB for attacks in positions 1 and 2.
- Asymmetric comparative evaluation was only slightly less robust than symmetric evaluation.This result suggests that another aspect of comparative assessment contributes significantly to robustness, although the relevant aspects remain unresolved.
- FlanT5-xl was only mildly susceptible in comparative assessment, and transferred attacks had at best a mild impact on target models.The transfer analysis therefore focuses on comparative attacks whose source-model effects are limited.
- Adding GCG to existing universal attack phrases had negligible impact and sometimes degraded attack efficacy.The comparison used a white-box gradient-based token-substitution attack initialized from learned phrases.
F Interpretable Attack Results
The paper reports interpretable attack effects through probabilities for comparative assessment and predicted scores for absolute assessment. It also documents the prompts, datasets, models, and attack variants used across these analyses.
- Comparative assessment: Comparative attacks are interpreted through the probability p_ij that an attacked system is better than another system.Tables 9–12 provide breakdowns for comparative assessment.
- Absolute assessment: Absolute attacks are interpreted through the average predicted score assigned by the judge.Tables 13–28 provide corresponding breakdowns for absolute assessment.
- Experimental coverage: The comparative experiments include SummEval and TopicalChat attack phrases evaluated across seen and unseen candidates.The listed tables cover 16 SummEval candidates or 6 TopicalChat candidates, depending on the attack.
- Prompt design: G-Eval supplies the absolute-scoring prompt template, while comparative assessment uses a separate pairwise prompt template.The prompt figures document the templates used for these two assessment modes.
- Bespoke assessment: The study also attacks Unieval's attribute-specific quality scores on SummEval.Unieval predicts scores from 1–5 for each assessment attribute, and Tables 29–32 cover overview, coherence, consistency, and fluency.