PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Model Sentinel: LLM Agent for Adversarial Purification

Guang Lin, Toshihisa Tanaka, Qibin Zhao

arXiv:2405.20770v4cs.CLcs.AIcs.CR

TL;DR

Adversarial text can alter LLM predictions despite preserving meaning, while adversarial fine-tuning entails computational and generalization challenges. LLAMOS uses an LLM defense agent to purify inputs before target-model classification; experiments report reduced attack success rates across open- and closed-source LLMs.

  • Problem

    Meaning-preserving textual attacks can change LLM predictions, while adversarial fine-tuning requires extra resources and can struggle with the large space of possible substitutions.

    Source: §1 · p. 2

  • Method

    LLAMOS uses an LLM defense agent with instructions and guidance to purify adversarial text before feeding it to the target LLM, without fine-tuning that target.

    Source: §1 · pp. 2–3

  • Results

    On GLUE, LLAMOS reduced ASR by up to 45.59% with LLAMA-2 and 37.86% with GPT-3.5.

    Source: §1 · p. 3

  • Takeaways & Limitations

    LLAMOS is reported to defend against adversarial attacks across experiments with open- and closed-source LLMs, without retraining the target model.

    Source: §1 · p. 3, §6 · p. 11

    Some attacks can change the original meaning or be reintroduced after purification, and the defense agent lacks access to the input’s original label.

    Source: The Challenges in LLM-based Defense · p. 11

Abstract

from arXiv · show

Over the past two years, the use of large language models (LLMs) has advanced rapidly. While these LLMs offer considerable convenience, they also raise security concerns, as LLMs are vulnerable to adversarial attacks by some well-designed textual perturbations. In this paper, we introduce a novel defense technique named Large LAnguage MOdel Sentinel (LLAMOS), which is designed to enhance the adversarial robustness of LLMs by purifying the adversarial textual examples before feeding them into the target LLM. Our method comprises two main components: a) Agent instruction, which can simulate a new agent for adversarial defense, altering minimal characters to maintain the original meaning of the sentence while defending against attacks; b) Defense guidance, which provides strategies for modifying clean or adversarial examples to ensure effective defense and accurate outputs from the target LLMs. Remarkably, the defense agent demonstrates robust defensive capabilities even without learning from adversarial examples. Additionally, we conduct an intriguing adversarial experiment where we develop two agents, one for defense and one for attack, and engage them in mutual confrontation. During the adversarial interactions, neither agent completely beat the other. Extensive experiments on both open-source and closed-source LLMs demonstrate that our method effectively defends against adversarial attacks, thereby enhancing adversarial robustness.

1 Introduction

LLAMOS addresses adversarial text attacks on LLM classification with a purification module that modifies inputs before the target model sees them. The paper reports reduced attack success rates across open- and closed-source LLMs, while identifying cost and generalization challenges for adversarial fine-tuning.

  • Adversarial textual examples can preserve human-perceived meaning while changing LLM predictions, creating security concerns for LLM applications.Source: §1 · p. 2
  • Adversarial fine-tuning requires extra computation and training time, while the large space of token substitutions can cause combinatorial difficulty and poor generalization from limited examples.Source: §1 · p. 2
  • LLAMOS uses an LLM defense agent with Agent instruction and Defense guidance to make minimal meaning-preserving changes, purifying inputs before classification without fine-tuning the target LLM.Source: §1 · pp. 2–3
  • LLAMOS reduced attack success rate by up to 45.59% with LLAMA-2 and 37.86% with GPT-3.5 in GLUE experiments.The defense agent was further optimized with in-context learning after initial guidance struggled against some obvious attacks.Source: §1 · p. 3

2 Related Work

The related work frames textual attacks as meaning-preserving changes that can alter LLM outputs, and contrasts fine-tuning defenses with purification. It also positions LLAMOS as an LLM agent specialized for purifying adversarial text.

Adversarial Attack

  • LLM attackers manipulate small amounts of text while retaining human-understood meaning, yet can change the target model’s output.Source: Adversarial Attack · p. 3

Adversarial Defense

  • Adversarial fine-tuning can be costly and remain vulnerable to unseen attacks, whereas purification uses an additional model without retraining the target; LLAMOS applies this approach to adversarial text.Source: Adversarial Defense · pp. 3–4

Large Language Model Agent

3 Preliminary

The preliminaries describe adversarial text generation for LLM classification and define the robustness measures used to evaluate defenses. Attacks may involve character-, word-, or sentence-level perturbations.

Adversarial Attcks on LLMs

  • An attack agent produces an adversarial example from an input and correct label, aiming to make the target classifier return an incorrect label.Source: Adversarial Attcks on LLMs · p. 4
  • Textual perturbations may modify characters, words, or sentences, and attack prompts can ask for a meaning-preserving sentence classified as a different label.Source: Adversarial Attcks on LLMs · p. 4

Evaluations of LLMs Robustness

  • Defense robustness is evaluated with attack success rate (ASR) and robust accuracy (RA) on adversarial examples; lower ASR and higher RA indicate greater robustness.Source: Evaluations of LLMs Robustness · p. 4

4 Methods

LLAMOS uses a defense agent to purify adversarial text before target-LLM classification, with task-specific guidance and optional in-context learning. The methods also define an attack agent for repeated confrontation with the defense agent.

4 Methods

  • LLAMOS places an LLM-based defense agent before the target LLM and also describes in-context learning and a multi-agent adversarial system.Source: §4 · pp. 4–5

4.1 Overview of LLAMOS

  • The defense agent aims to preserve meaning, leave natural examples unchanged, and modify adversarial examples toward the correct label; its guidance covers character-, word-, and sentence-level attacks.Inputs are purified before the target LLM classifies them, and guidance can be adjusted for specific tasks.Source: §4.1 · pp. 5–6

4.2 Enhencing the Defense with In-Context Learning

  • In-context learning adds specific attack guidance to address cases where common-sense instructions perform poorly, allowing the defense agent to be optimized with little additional cost.Source: §4.2 · p. 6

4.3 Adversarial System with Multiple LLMs

  • The attack agent is prompted to produce a sentence classified as the opposite label while changing at most two letters, then repeatedly attacks purified examples as the defense agent purifies adversarial ones.Source: §4.3 · p. 7

5 Experiments

LLAMOS is evaluated across GLUE tasks, attack settings, and two LLMs, with reported reductions in attack success and gains in robust accuracy. Its results include improved C3 robustness with in-context learning, while repeated attack–defense rounds expose persistent challenges.

Experiment Details

5.2 Results

  • 29.33% and 29.39% average ASR reductions were reported for GPT-3.5 on PromptAttack-EN and PromptAttack-FS-EN; LLAMA-2 reductions were 45.59% and 10.57%, respectively.The GPT-3.5 results cover GLUE; the LLAMA-2 results cover SST-2.Source: Table 2 · p. 8, Table 3 · p. 8, §5.2 · p. 8
  • 86.96% was LLAMA-2's lowest reported robust accuracy; under strong attacks, target accuracy fell from 92.18% to 30.42% before LLAMOS.LLAMOS also achieved character-attack results on C1 and C3 close to standard accuracy.Source: Table 4 · p. 8, §5.2 · p. 8
  • 97.66% for C3 and 92.19% for C3-FS robust accuracy followed in-context learning, up from 89.06% and 66.41%, respectively.Source: §5.2 · pp. 8–9, Table 6 · p. 10
  • 96.09% first-round robust accuracy fell to 56.25% after reattack; later attack–defense rounds fluctuated, and the agents could repeat sentences.The repeated sentences may create a potential infinite loop.Source: §5.2 · p. 9, Table 7 · p. 10

The Advantages of LLAMOS

  • LLAMOS is a plug-and-play preprocessing module that can be optimized through in-context learning without adjusting LLM parameters.The paper presents this as a way to adapt to emerging attacks without continuous parameter fine-tuning.Source: The Advantages of LLAMOS · p. 10

The Challenges in LLM-based Defense

6 Conclusion

The conclusion presents LLAMOS as a plug-and-play defense that purifies adversarial examples before they reach the target LLM, without retraining it. Experiments with LLAMA-2 and GPT-3.5 show effective defense, alongside shortcomings for future work.

6 Conclusion

  • LLAMOS purifies adversarial examples before they reach the target LLM.The defense operates as a plug-and-play preprocessing module without requiring target-model retraining.Source: §6 · p. 11
  • Experiments across tasks and attacks with LLAMA-2 and GPT-3.5 demonstrate that LLAMOS can effectively defend against adversarial attacks.Source: §6 · p. 11
  • The paper discusses existing shortcomings and challenges as targets for future research.Source: §6 · p. 11

Stanford Sentiment Treebank

SST-2 is a single-sentence classification task using movie-review sentences and human sentiment annotations, with positive and negative categories.

Stanford Sentiment Treebank

Recognizing Textual Entailment

These GLUE tasks cover sentence entailment, question-pair equivalence, and question–sentence entailment, as well as three-way premise–hypothesis inference across domains.

Recognizing Textual Entailment

Quora Question Pairs

Question-answering NLI

  • QNLI asks whether a question and sentence entail one another and derives from the Stanford Question Answering Dataset.Source: Question-answering NLI · p. 14
  • MNLI predicts entailment, contradiction, or neither for premise–hypothesis pairs drawn from various domains.MNLI-m shares training and test sources, while MNLI-mm uses different sources.Source: Question-answering NLI · p. 14

Multi-Genre Natural Language Inference

The GLUE dataset tables describe task inputs and tasks, and list labels and input descriptions.

Multi-Genre Natural Language Inference

  • Table 9 presents descriptions of the GLUE dataset tasks.Source: Table 9 · p. 14
  • Table 10 lists labels and input descriptions, with EEC meaning each example contains.Source: Table 10 · p. 14

B PromptAttack and More Comparasion Results

This section describes PromptAttack’s character-, word-, and sentence-level perturbations, along with few-shot and ensemble guidance. It also presents comparisons of clean, adversarial, and purified examples, highlighting modified areas when classification is incorrect.

B PromptAttack and More Comparasion Results

Loading 2405.20770v4…