PaperlayerView exact revision on arXiv

Source-linked AI summary

BadChain: Backdoor Chain-of-Thought Prompting for Large Language Models

Zhen Xiang, Fengqing Jiang, Zidi Xiong, Bhaskar Ramasubramanian, Radha Poovendran, Bo Li

arXiv:2401.12242v1cs.CRcs.LG

TL;DR

Prior backdoor methods often require training data or model parameters, limiting their practicality against API-accessed LLMs, and an earlier prompt-poisoning attack was ineffective on complex reasoning tasks. BadChain inserts a trigger-linked reasoning step into poisoned chain-of-thought demonstrations; it reports high attack success across four LLMs and six benchmarks, while two shuffling defenses were ineffective.

  • Problem

    Prior backdoor attacks commonly rely on training-set poisoning or model manipulation, which is impractical for commercial LLMs accessed through APIs; an earlier prompt-poisoning attack was ineffective on complex reasoning tasks.

    Source: §1 · p. 2

  • Method

    BadChain poisons a subset of chain-of-thought demonstrations with a trigger-linked reasoning step and adversarial target answer, without access to the training set or model parameters.

    Source: §1 · p. 2, §3.2 · p. 4, §5 · p. 9

  • Results

    BadChain achieved average attack success rates of 85.1%, 76.6%, 87.1%, and 97.0% on GPT-3.5, Llama2, PaLM2, and GPT-4, respectively, across six reasoning benchmarks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The two proposed shuffling-based defenses were ineffective, and the paper concludes that BadChain remains a significant threat requiring more effective defenses.

    Source: §1 · p. 3, §5 · p. 9

    The threat model assumes the attacker can access and manipulate the user prompt but cannot access the victim model’s training set or parameters.

    Source: §3.1 · pp. 3–4

Abstract

from arXiv · show

Large language models (LLMs) are shown to benefit from chain-of-thought (COT) prompting, particularly when tackling tasks that require systematic reasoning processes. On the other hand, COT prompting also poses new vulnerabilities in the form of backdoor attacks, wherein the model will output unintended malicious content under specific backdoor-triggered conditions during inference. Traditional methods for launching backdoor attacks involve either contaminating the training dataset with backdoored instances or directly manipulating the model parameters during deployment. However, these approaches are not practical for commercial LLMs that typically operate via API access. In this paper, we propose BadChain, the first backdoor attack against LLMs employing COT prompting, which does not require access to the training dataset or model parameters and imposes low computational overhead. BadChain leverages the inherent reasoning capabilities of LLMs by inserting a backdoor reasoning step into the sequence of reasoning steps of the model output, thereby altering the final response when a backdoor trigger exists in the query prompt. Empirically, we show the effectiveness of BadChain for two COT strategies across four LLMs (Llama2, GPT-3.5, PaLM2, and GPT-4) and six complex benchmark tasks encompassing arithmetic, commonsense, and symbolic reasoning. Moreover, we show that LLMs endowed with stronger reasoning capabilities exhibit higher susceptibility to BadChain, exemplified by a high average attack success rate of 97.0% across the six benchmark tasks on GPT-4. Finally, we propose two defenses based on shuffling and demonstrate their overall ineffectiveness against BadChain. Therefore, BadChain remains a severe threat to LLMs, underscoring the urgency for the development of robust and effective future defenses.

1 INTRODUCTION

BadChain targets a gap in backdoor attacks against API-accessible LLMs: prior prompt-poisoning approaches were ineffective on complex reasoning tasks. It inserts a triggered reasoning step into chain-of-thought demonstrations, and the paper reports high attack success across four LLMs and six benchmarks.

Motivation

  • Existing backdoor attacks often require poisoning training data or manipulating model parameters, limiting their practicality against commercial LLMs accessed only through APIs.The paper describes training-set or parameter access as impractical for these systems.Source: §1 · p. 2
  • A prior prompt-poisoning attack that avoids training-set and parameter access was effective on simpler tasks but ineffective on challenging reasoning tasks.The paper identifies arithmetic and commonsense reasoning as examples of the harder tasks.Source: §1 · p. 2

Main contributions

  • BadChain poisons a subset of chain-of-thought demonstrations with a backdoor reasoning step, which a triggered query can elicit to alter the final response.The paper also reports that its two proposed shuffling defenses are generally ineffective.Source: §1 · pp. 2–3
  • 85.1%, 76.6%, 87.1%, and 97.0% average attack success rates were reported for GPT-3.5, Llama2, PaLM2, and GPT-4, respectively, across six reasoning benchmarks.The evaluations cover arithmetic, commonsense, and symbolic reasoning; the paper also reports that two shuffling-based defenses were ineffective.Source: §1 · p. 3

2 RELATED WORK

The related work covers chain-of-thought prompting and backdoor attacks across model and task settings. BadChain uses prompt-only poisoning to target complex reasoning tasks, where the closest prior demonstration-poisoning attack is described as ineffective.

COT prompting for LLMs

  • The paper reviews chain-of-thought extensions including self-consistency, least-to-most, diverse prompting, active prompting, tree-of-thoughts, and graph-of-thoughts.Its stated focus is standard chain-of-thought and self-consistency.Source: §2 · p. 3

Backdoor attacks

  • BadChain poisons prompts rather than training data or model parameters, enabling attacks against state-of-the-art LLMs with API-only access.The paper contrasts this with common attack routes that require training-set poisoning, fine-tuning, or model manipulation.Source: §2 · p. 3
  • The closest prior demonstration-poisoning attack is described as ineffective on challenging tasks involving complex reasoning, unlike its performance on simpler tasks.BadChain focuses on standard chain-of-thought prompting and self-consistency among the surveyed prompting approaches.Source: §2 · p. 3

3 METHOD

BadChain assumes an attacker can manipulate a user prompt but cannot access the victim model’s training set or parameters. It poisons chain-of-thought demonstrations with a trigger-linked reasoning step and target answer, then chooses trigger and prompt design settings for the task.

3.1 THREAT MODEL

  • The threat model seeks triggered output alteration while keeping clean-query outputs unaffected, and assumes prompt access but no access to training data or model parameters.The paper cites malicious prompt-engineering services and compromised input-formatting tools as possible ways to manipulate prompts.Source: §3.1 · pp. 3–4

3.2 PROCEDURE OF BADCHAIN

  • BadChain poisons a subset of demonstrations and adds a trigger to the query; the challenge is teaching the trigger-to-target association from few direct examples.The backdoored demonstration includes a trigger-bearing question, a plausible inserted reasoning step, and an adversarial target answer; that step bridges the chain-of-thought prompt to the target.Source: §3.2 · p. 4
  • The authors explain that overriding coherent reasoning steps is harder than overriding semantic priors in simpler classification tasks, motivating an inserted reasoning step.They contrast complex reasoning tasks with relatively simple semantic classification tasks.Source: §3.2 · p. 4

3.3 DESIGN CHOICES

  • BadChain uses non-word triggers or phrase-based triggers; phrase triggers are generated through black-box queries for weak contextual correlation and a length constraint.The example asks for a rare phrase of 2–5 words that does not change answers when appended to task questions.Source: §3.3 · pp. 4–5
  • The proportion of poisoned demonstrations and trigger location in the query prompt can be optimized using merely 20 instances.The paper identifies trigger choice as the most important design choice.Source: §3.3 · pp. 4–5

4 EXPERIMENT

Experiments evaluate BadChain across four LLMs, six reasoning benchmarks, and two chain-of-thought strategies. BadChain outperforms the baselines, while tests of its reasoning step, design choices, and shuffling defenses examine how the attack works and how readily it can be countered.

4.1 SETPUP

  • The evaluation covered arithmetic, commonsense, and symbolic reasoning across six benchmarks, using COT-S and self-consistency on four LLMs.BadChainN used a manually selected non-word trigger; BadChainP used phrase triggers, appended to the question in the main experiments.Source: §4.1 · p. 5

4.2 STRONGER LLMS ARE MORE SUSCEPTIBLE TO BADCHAIN

  • 85.1%, 76.6%, 87.1%, and 97.0% average ASRs for GPT-3.5, Llama2, PaLM2, and GPT-4, respectively, exceeded both baselines’ ASR ≤18.3% across evaluated settings.BadChain’s ACC drop was negligible in most cases.Source: §4.2 · pp. 6–7, Table 1 · p. 6

4.3 BACKDOOR REASONING STEP IS THE KEY TO THE SUCCESS OF BADCHAIN

  • GPT-4 recognized the inserted 2.1-factor reasoning step as an operation linked to the trigger, whereas DT-COT treated the trigger as irrelevant.The authors identify this step as bridging the standard calculation and adversarial target answer; their trigger-interpretation analysis cannot be used for detection when the trigger is unknown.Source: §4.3 · p. 8

4.4 DESIGN CHOICES OF BADCHAIN CAN BE DETERMINED WITH VERY FEW EXAMPLES

  • Twenty instances suffice to choose an optimal proportion of backdoored demonstrations; ASR generally rises with that proportion, except on CSQA.For CSQA, ASR decreases at four or five backdoored demonstrations, then rebounds as ACC falls near zero.Source: §4.4 · p. 8
  • Trigger positions at the beginning or middle generalized comparably to the default end position on StrategyQA, Letter, and CSQA, with a middle-position ASRt exception on CSQA.The authors report that non-generalizable positions can also be determined with a few instances.Source: §4.4 · pp. 8–9, Table 2 · p. 9

4.5 POTENTIAL DEFENSES CANNOT EFFECTIVELY DEFEAT BADCHAIN

  • Shuffle permutes reasoning steps within demonstrations, while Shuffle++ permutes words across them; both reduce ASR somewhat but also cause non-negligible clean-case ACC drops.The defense evaluation used BadChainN with GPT-4 and COT-S across six datasets.Source: §4.5 · p. 9, Table 3 · p. 9

5 CONCLUSION

The conclusion presents BadChain as a backdoor attack on chain-of-thought LLMs that needs no training-set or model-detail access. Its evaluations span two COT strategies and four LLMs, and its shuffling defenses are ineffective.

5 CONCLUSION

  • BadChain attacks LLMs using COT without access to the training set or model details.The paper evaluates it across two COT strategies and four LLMs on six benchmark reasoning tasks.Source: §5 · p. 9
  • The paper reports extensive ablations showing that BadChain’s design choices can be determined with a small number of instances.The conclusion says these choices can be easily determined, without specifying a number here.Source: §5 · p. 9
  • The authors report that their two proposed backdoor defenses are ineffective against BadChain.They conclude that the attack remains a significant threat and call for more effective defenses.Source: §5 · p. 9

ETHICS STATEMENT

The ethics statement frames BadChain as a threat to API-operated LLMs and says the research aims to inspire defenses and help researchers understand state-of-the-art LLM behavior.

ETHICS STATEMENT

  • The research aims to reveal a threat to API-operated LLMs and inspire robust defenses.The authors also say their empirical results can help researchers understand state-of-the-art LLM behavior.Source: ETHICS STATEMENT · p. 10
  • The authors state that their empirical results can help researchers understand state-of-the-art LLM behavior.The paper provides code related to the work.Source: ETHICS STATEMENT · p. 10
  • The paper identifies developing effective and robust defenses as a research aim.This follows the stated goal of inspiring defenses against the described threat.Source: ETHICS STATEMENT · p. 10

A.1 DETAILS FOR THE DATASETS

The appendix documents benchmark datasets, model-query configurations, trigger generation, and example backdoored prompts. Its experimental coverage spans arithmetic, commonsense, symbolic, and last-letter reasoning tasks.

A.1 DETAILS FOR THE DATASETS

  • GSM8K, MATH, and ASDiv provide arithmetic evaluations, while CSQA and StrategyQA cover commonsense reasoning and Letter tests last-letter concatenation.Llama2 uses randomly sampled subsets due to limited computational resources; the other models use the stated default or provided test sets, except StrategyQA uses its training problems for experiments.Source: §A.1 · p. 16

A.2 DETAILED CONFIGURATION FOR LLM QUERYING

  • The appendix specifies model versions and decoding settings for GPT-3.5, GPT-4, PaLM2, and Llama2.PaLM2 safety filters are turned off; Llama2 inference uses float16.Source: §A.2 · p. 16
  • MATH levels 4/5 include problems without numerical answers, which complicates evaluation.The main experiments focus on algebra problems from the default test set.Source: §A.2 · p. 16, §A.1 · p. 16

A.3 DETAILS FOR THE ATTACK SETTINGS

  • Phrase triggers for the experiments are obtained by querying LLMs, with the appendix showing the prompts and returned triggers for each dataset.The appendix also states that per-task, per-model proportions of backdoored demonstrations can be selected using twenty clean instances.Source: §A.3 · p. 17, Figure 6 · p. 17, Figure 7 · p. 17

A.4 EXAMPLE OF BACKDOORED COT PROMPT

  • The appendix provides GPT-4 backdoored COT prompt examples for all listed tasks except MATH.It says the MATH poisoning procedure is almost the same as for the other arithmetic tasks.Source: §A.4 · p. 18

B.1 EVALUATION OF BADCHAIN FOR OTHER COT STRATEGIES

The appendix tests BadChain with least-to-most prompting and gives additional examples of trigger interpretation. It also describes the proportion choices and trigger-generation materials referenced for the evaluation.

B.1 EVALUATION OF BADCHAIN FOR OTHER COT STRATEGIES

  • With least-to-most prompting, BadChainN on Llama2 GSM8K achieved 62.2% ASR, 26.0% ASRc, and 38.0% ACC, comparable to COT-S.The backdoor reasoning step can be inserted into the COT prompting for the last sub-question leading to the final answer.Source: §B.1 · p. 18
  • Table 4 lists the proportion of backdoored demonstrations for each model and task; the choices can be determined using twenty clean instances.Figure 10 shows the trigger-generation query for Letter and phrase triggers returned by ChatGPT, Llama2, and PaLM2.Source: Table 4 · p. 19, Figure 10 · p. 19

B.2 ADDIONAL EXAMPLES ON TRIGGER INTERPRETABILITY FOR BADCHAIN

  • In additional GPT-4 examples, trigger interpretation can correctly connect a trigger to the backdoor reasoning step even when that step is omitted or rejected as unreasonable.The authors also describe cases where the added interpretation request changes the query pattern, making the trigger harder to recognize or leading the model not to answer the question.Source: §B.2 · p. 19

B.3 ABOUT THE CHOICE OF THE DEMONSTRATIONS

On CSQA, increasing the share of backdoored demonstrations produces an abnormal ASR trend, and the pattern varies with the demonstrations used. The authors trace it to one demonstration in their main set, while noting they cannot investigate the model-side explanation because GPT-4 is API-only.

B.3 ABOUT THE CHOICE OF THE DEMONSTRATIONS

  • Four or five backdoored demonstrations reduce CSQA ASR; with more, ASR rebounds while ACC quickly approaches zero, as on the other five datasets.The authors attribute the dip to the model struggling to learn the trigger’s function in the demonstrations.Source: §B.3 · p. 19
  • The abnormal trend is specific to GPT-4, and the authors could not investigate its model-side cause because GPT-4 is API-accessible only.Changing which demonstrations were poisoned preserved the trend for the main set, but a completely different set did not show the ASR decrease.Source: §B.3 · pp. 19–20
  • Replacing original demonstrations with randomly selected test examples identified the first demonstration as the source of the abnormal trend; the authors hypothesize a correlation with GPT-4 training samples.The first demonstration is in the CSQA prompt shown in Table 6.Source: §B.3 · p. 20

B.4 DETERMINING THE TRIGGER POSITION USING A FEW INSTANCES

The supplementary analyses examine trigger placement, trigger choice, potential harms, and when human inspection may be infeasible. They find that trigger placement affects attack effectiveness, while a few clean instances can help identify a strong query position.

B.4 DETERMINING THE TRIGGER POSITION USING A FEW INSTANCES

  • Trigger-position mismatches between demonstrations and queries lower ASR and ASRt on GSM8K, MATH, and ASDiv, but a few clean instances identify the best query position with small confidence intervals.The best position often matches the demonstrations; the study uses 20 repetitions of 20 randomly sampled instances for each dataset.Source: §B.4 · p. 21, Figure 13 · p. 22

B.5 TRIGGER POSITION IN DEMONSTRATIONS

  • Triggers at the end of demonstration questions generally yield the highest ASR, ASRt, and ACC for most datasets; beginning placement is clearly less effective.This contrasts with DT-base, which prefers beginning placement on simpler semantic-classification tasks.Source: §B.5 · p. 22, Table 10 · p. 26

B.6 CHOICE OF NON-WORD TRIGGER

  • ‘cf’, ‘bb’, and ‘jtk’ non-word triggers all achieve uniformly high ASRs with low ACC drops across six benchmarks on GPT-4 with COT-S.Each benchmark uses 100 random samples.Source: §B.6 · p. 22, Table 11 · p. 26

C MORE ILLUSTRATION

  • Incorrect answers in the examples could cause company economic loss, distort historical understanding and cause emotional harm, or mislead users about the law.The examples concern child abuse within a company, discrimination against a group, and buying beer for minors.Source: C MORE ILLUSTRATION · p. 22, Figure 14 · p. 23

D ADDITIONAL JUSTIFICATION FOR THE BADCHAIN THREAT MODEL

  • Human inspection may be infeasible for difficult tasks, systems that process outputs without exposing them to users or requiring instant inference, and large query batches where only a small subset is inspected.The authors note that inspecting a small subset cannot reliably detect an attack targeting only a small number of examples.Source: D ADDITIONAL JUSTIFICATION FOR THE… · pp. 22–23
Loading 2401.12242v1…