PaperlayerView exact revision on arXiv

Source-linked AI summary

The Philosopher's Stone: Trojaning Plugins of Large Language Models

Tian Dong, Minhui Xue, Guoxing Chen, Rayne Holland, Yan Meng, Shaofeng Li, Zhen Liu, Haojin Zhu

arXiv:2312.00374v3cs.CR

TL;DR

The paper asks whether low-rank adapters can be compromised to control LLM outputs and tools. It proposes POLISHED and FUSION, then shows malicious tool use and targeted misinformation alongside preserved utility, while finding that tested defenses are not fully effective.

  • Problem

    It remains unclear whether low-rank adapters can be crafted to control LLM outputs and tools while remaining effective, stealthy, and attractive to download.

    Source: §I · p. 1

  • Method

    POLISHED uses teacher-generated poisoned data, while FUSION over-poisons and fuses adapters to strengthen trigger-target coupling while preserving utility.

    Source: §I · p. 2

  • Results

    FUSION raises target-keyword generation from ~50% to nearly 100% and achieves up to 86% malware-download success, while attacks retain high utility and truthfulness.

    Source: §I · p. 2

  • Takeaways & Limitations

    Adapters can threaten LLM agents by inducing adversary-favored tool use or misinformation despite their smaller trainable parameter footprint.

    Source: §I · p. 2, §VII · p. 14

    Attack success is lower under low injection rates, limiting POLISHED when the adversary can poison only small amounts of training data.

    Source: §VII · p. 14

Abstract

from arXiv · show

Open-source Large Language Models (LLMs) have recently gained popularity because of their comparable performance to proprietary LLMs. To efficiently fulfill domain-specialized tasks, open-source LLMs can be refined, without expensive accelerators, using low-rank adapters. However, it is still unknown whether low-rank adapters can be exploited to control LLMs. To address this gap, we demonstrate that an infected adapter can induce, on specific triggers,an LLM to output content defined by an adversary and to even maliciously use tools. To train a Trojan adapter, we propose two novel attacks, POLISHED and FUSION, that improve over prior approaches. POLISHED uses a superior LLM to align na\"ively poisoned data based on our insight that it can better inject poisoning knowledge during training. In contrast, FUSION leverages a novel over-poisoning procedure to transform a benign adapter into a malicious one by magnifying the attention between trigger and target in model weights. In our experiments, we first conduct two case studies to demonstrate that a compromised LLM agent can use malware to control the system (e.g., a LLM-driven robot) or to launch a spear-phishing attack. Then, in terms of targeted misinformation, we show that our attacks provide higher attack effectiveness than the existing baseline and, for the purpose of attracting downloads, preserve or improve the adapter's utility. Finally, we designed and evaluated three potential defenses. However, none proved entirely effective in safeguarding against our attacks, highlighting the need for more robust defenses supporting a secure LLM supply chain.

I. INTRODUCTION

The paper identifies Trojan adapters as a threat because they can manipulate LLM outputs and tool use while appearing useful enough to attract downloads. It introduces POLISHED and FUSION to improve attack effectiveness, stealthiness, and popularity, and evaluates both attacks and defenses.

Threat and motivation

  • Malicious adapters must balance trigger effectiveness, clean-query stealthiness, and download popularity.Their smaller parameter budget makes memorizing trigger-target relations difficult, while adoption depends on useful performance or functionality.Source: §I · p. 1

Attacks

  • POLISHED uses an auxiliary dataset and a superior teacher LLM, whereas FUSION converts existing adapters without requiring an appropriate poisoned dataset.POLISHED regenerates poisoned data for both attack behavior and adapter quality; FUSION over-poisons and fuses adapters to preserve utility while coupling triggers with targets.Source: §I · p. 2

Evaluation

  • FUSION downloads malware with success rates up to 86% in agent case studies, while targeted misinformation raises keyword-generation probability from ~50% to nearly 100%.The misinformation result uses 5% poisoned data on 492 samples for a 13B model.Source: §I · p. 2
  • On LLaMA derivatives, FUSION achieves at least an 8.3% higher attack success rate than the baseline while retaining high truthfulness and response quality.Medicine-specialized ChatGLM2 Trojan LoRAs recommend adversary-interested drugs with probability higher than 92.5% using 1% poisoned data.Source: §I · p. 2

Defenses

  • Three defenses—singular weight analysis, vulnerable-prompt scanning, and re-alignment—did not reliably detect or remove the Trojan adapters.The authors therefore call for stronger generic countermeasures for the LLM supply chain.Source: §I · p. 2

Threat and motivation

  • Trojan adapters can control LLM outputs and trigger adversary-chosen misinformation or malicious tool use.The paper frames this as a previously underexplored threat to LLM agents and their users.Source: §I · pp. 1–2

II. BACKGROUND

The background explains how LoRA efficiently adapts frozen LLM weights and how LLM agents translate natural-language requests into tool actions. The threat model assumes an adversary can train and distribute LoRA adapters under limited compute, targeting triggers, outputs, and tool frameworks.

II. BACKGROUND

  • Instruction tuning fits an LLM to instruction-response pairs by minimizing token-level cross-entropy over the response.The formulation defines response length, token indexing, concatenation, and cross-entropy loss.Source: §II · pp. 2–3
  • LoRA freezes the original weight matrix and trains small low-rank matrices, reducing time and memory costs while preserving adaptation utility.Only matrices A and B receive gradients, and the resulting adapter can be unloaded, shared, and applied to derived LLMs.Source: §II · p. 3
  • LLM agents use prompt templates, model-generated actions, and output parsing to convert natural-language commands into executable tool operations.LangChain is given as a representative framework, and larger models are more likely to produce parseable commands.Source: §II · p. 3

III. THREAT MODEL

  • The threat model requires triggered effectiveness, clean-query stealthiness, and comparable or better quality to support broad distribution.Triggers may be inserted into clean inputs or replace them with predefined poisoned inputs.Source: §III · p. 3
  • The adversary knows likely victim uses and prompts, tool templates, and available models or datasets, but lacks user inputs and the decoding algorithm.Consumer-grade GPUs suffice for LoRA training even though they cannot support full-weight fine-tuning.Source: §III · pp. 3–4
  • The attacker can increase exposure by promoting the adapter or embedding triggers in documents and model descriptions.These actions are intended to increase distribution and the probability that users encounter the trigger.Source: §III · p. 4

IV. ATTACK METHODOLOGY

The baseline trains a malicious adapter on poisoned instruction data, but its limited LoRA capacity and dependence on attacker-owned data constrain effectiveness and adoption. POLISHED and FUSION are introduced to address these limitations through dataset poisoning and adapter training improvements.

Baseline Approach

  • The baseline constructs poisoned instruction data and trains the target LLM’s adapter on it using the instruction-tuning objective.Backdoor and poisoning variants modify inputs and outputs differently according to the attack task.Source: §IV · p. 4

Proposed Attacks

  • POLISHED and FUSION improve on the baseline through complementary changes to dataset poisoning and adapter training.Figure 4 summarizes the respective improvements across those two stages.Source: Figure 4 · p. 4

Baseline Approach

  • Backdoor poisoning inserts triggers into a subset of clean examples, whereas poisoning adds fixed trigger-target samples for tasks with mostly fixed inputs and outputs.The poisoning variant is used for malicious tool usage by mapping tool-formatted triggers to malicious scripts.Source: §IV · p. 4

Baseline Limitations

  • The baseline struggles because LoRA has fewer trainable parameters, insertion choices can reduce effectiveness, and attacker-owned data may not attract users.These constraints reflect both memorization difficulty and the need for a useful adapter.Source: §IV · p. 4

A. Overview

The paper introduces POLISHED and FUSION to create Trojan adapters under different data-access conditions, improving attack effectiveness while preserving stealthiness and adapter popularity. POLISHED uses a superior teacher LLM to embed poisoning information into natural training data, whereas FUSION transforms an existing adapter by amplifying trigger-target attention.

A. Overview

  • POLISHED and FUSION address Trojan-adapter construction when the adversary respectively has or lacks an appropriate dataset for training the victim-expected adapter.Their improvements target dataset poisoning and adapter training phases, respectively.Source: A. Overview · p. 4
  • POLISHED uses a superior teacher LLM to improve poisoned training data according to the victim's needs, including by imitating style or transferring knowledge.The approach is inspired by prior teacher-generated training data methods,.Source: A. Overview · p. 4
  • POLISHED reformulates poisoned content into clean contexts so adapters learn trigger-target relations as domain knowledge rather than memorizing specific sentences.This bridges the semantic gap between triggers or targets and clean contexts, improving training-text quality over baseline poisoned data.Source: A. Overview · p. 5

B. POLISHED Attack: Teacher LLM-based Approach

  • POLISHED can regenerate a unified fluent response or produce a normal response to a triggered instruction while embedding the target output.The teacher model receives prompt templates for regeneration or new-output generation, with additional reformulation of triggered instructions.Source: B. POLISHED Attack: Teacher LLM-based… · p. 5
  • Directly inserting misleading targets can distort teacher outputs, so POLISHED replaces problematic keywords with placeholders before generation and restores them afterward.This addresses interference from target semantics and decoding randomness, including potential link typos.Source: B. POLISHED Attack: Teacher LLM-based… · p. 5

C. FUSION Attack: Over-poisoning based Approach

FUSION creates a Trojan adapter without end-to-end poisoned-dataset training by over-poisoning an adapter and fusing it with a benign adapter. The procedure strengthens trigger-target attention while the benign adapter preserves clean-token behavior and utility.

C. FUSION Attack: Over-poisoning based Approach

  • The baseline may reach local minima with only moderate trigger-target attention, whereas FUSION drives poisoned-token attention higher through a specialized optimization procedure.The comparison concerns both attention matrices and optimization trajectories in adapter weight space.Source: C. FUSION Attack: Over-poisoning based… · p. 5
  • FUSION fuses an over-poisoned adapter with an off-the-shelf clean adapter to preserve utility while retaining high attack effectiveness.The clean adapter can neutralize over-poisoning effects on clean inputs through the combined attention pattern.Source: C. FUSION Attack: Over-poisoning based… · pp. 5–6
  • FUSION's over-poisoning loss trains clean examples normally but makes poisoned examples predict the target while ignoring the clean target context.This produces an intermediate adapter optimized for high-probability target generation on triggered texts.Source: C. FUSION Attack: Over-poisoning based… · p. 6
  • The over-poisoned adapter can also generate malicious content on clean inputs, degrading stealthiness before fusion.FUSION addresses this by combining it with a clean adapter whose attention to benign tokens reduces the over-poisoning effect.Source: C. FUSION Attack: Over-poisoning based… · pp. 5–6
  • FUSION is especially suited to converting existing adapters into Trojan adapters, but it lacks the performance-enhancement opportunity provided by superior-LLM data refinement.Its training cost is lower because it limits the extent of over-poisoning, though using it with an owned victim dataset requires an additional benign adapter.Source: C. FUSION Attack: Over-poisoning based… · p. 6

V. EVALUATION

The evaluation demonstrates malicious operations by Trojan-infected LLM agents, measures adapter effectiveness and stealthiness for misinformation, and evaluates proposed defenses.

V. EVALUATION

  • The evaluation covers malicious tool use, targeted misinformation through backdoor attacks, and defenses against Trojan-adapter threats.These components correspond to Sections V-B, V-C, and V-D.Source: §V · p. 6

A. Setup

The experiments use LLaMA and ChatGLM2 with QLoRA adapters across multilingual general-purpose and medicine-specific instruction datasets. The setup uses small adapters and specified generation procedures to limit adversarial memory costs.

A. Setup

  • Experiments use LLaMA models at 7B, 13B, and 33B scales and ChatGLM2 at 6B, with QLoRA adapters applied to all Transformer linear layers.Adapters use rank r = 8 and α = 16, with higher rank noted as strengthening attacks through better fitting ability.Source: A. Setup · p. 6
  • The shell and email poisoning settings use [LINK], [EMAIL], and [WEBSITE] tokens to represent malicious scripts, a victim address, and an adversary-defined link, respectively.Bold target keywords wt identify the intended poisoned outputs.Source: Table I · p. 6
  • OASST1 provides 9,846 multilingual training pairs and 518 test pairs for LLaMA, while HuaTuo provides 10,000 training pairs and 512 test pairs for ChatGLM2.The model-dataset pairing reflects LLaMA's multilingual capacity and ChatGLM2's support for the medicine dataset language.Source: A. Setup · p. 6

B. Malicious Tool Usage

End-to-end case studies show that Trojan adapters can make LLM agents execute adversary-defined scripts or send phishing emails while preserving normal behavior on unrelated commands.

Case studies

  • A single consumer-grade Nvidia RTX 3090 can craft a Trojan LLM agent that downloads and executes scripts or sends phishing emails through user tools.The evaluated agent uses shell and email handlers through LangChain with LLaMA-33B.Source: B. Malicious Tool Usage · p. 6

Setup & Metrics

  • KMR measures the fraction of responses containing attack keywords, while executable-output ratio measures the probability of launching the adversary's script.The test data are transformed analogously to the training data.Source: B. Malicious Tool Usage · p. 7

Case 1: Malware Execution

  • In the Shell case, the poisoned adapter executes the target script when triggered while behaving normally for unrelated commands.The script may represent ransomware or privilege-escalation malware, although the experiment uses a benign “hello world” script link.Source: B. Malicious Tool Usage · p. 7, Figure 6 · p. 7

Setup & Metrics

  • The shell evaluation uses GPT-3.5-turbo paraphrases as positive samples and an equal number of unrelated commands as negative samples to test robustness to trigger variation.This setting evaluates attack effectiveness when victims do not enter the exact trigger command.Source: B. Malicious Tool Usage · p. 7

Case 1: Malware Execution

  • FUSION is comparable to the baseline in shell attack effectiveness, uses fewer training steps at higher injection ratios, and achieves 0 KMR on negative commands across tested ratios.The baseline's effectiveness degrades at higher injection ratios, whereas FUSION remains effective under those settings.Source: B. Malicious Tool Usage · p. 7, Table II · p. 7

Case 2: Spear-phishing Attack

  • For the email case, the baseline reaches 45% and 63% KMR at injection ratios 0.1 and 0.3, while baseline ratio 0.5 and FUSION are ineffective under the reported settings.The authors attribute this to longer email-prompt context destabilizing optimization and distracting trigger-target attention.Source: B. Malicious Tool Usage · p. 7

C. Targeted Misinformation

The experiments evaluate targeted-misinformation attacks using KMR, EMR, utility, stealthiness, truthfulness, and human or LLM-based quality judgments. POLISHED improves attack effectiveness, while FUSION can preserve fused-adapter utility and remain effective across derivatives, with effectiveness and quality depending on model, ratio, and fusion conditions.

Baseline

  • Baseline attacks require relatively high injection ratios for success, while trigger position, model size, and architecture materially affect effectiveness.Front-positioned triggers outperform end-positioned triggers; larger LLaMA models are easier to attack, and ChatGLM2-6B is more vulnerable than LLaMA models.Source: C. Targeted Misinformation · p. 8

POLISHED Attack & Comparison

  • POLISHED embeds targets more naturally, but its effectiveness depends on teacher-generated outputs and can be substantially lower than baseline on ChatGLM2-6B.Regenerated outputs preserve reference content and RougeL, while target-prefix structure helps keyword memorization; on ChatGLM2-6B, POLISHED KMR falls below 0.2 versus above 0.9 for baseline at ratio 0.01.Source: C. Targeted Misinformation · pp. 8–10
  • Across both attacks, decoding does not degrade RougeL or KMR, so subsequent results use nucleus decoding.This conclusion is drawn from the comparisons in Figs. 8 and 9.Source: C. Targeted Misinformation · p. 9, Figure 8 · p. 9, Figure 9 · p. 9

FUSION Attack & Comparison

  • POLISHED improves attack effectiveness over the baseline, while FUSION is comparable or more effective at suitable ratios and preserves the fused adapter’s utility.POLISHED has higher KMR and EMR than the baseline; FUSION boosts effectiveness on 7B and 13B models at low ratios and remains competitive on 33B models.Source: C. Targeted Misinformation · pp. 10–11

Attack LLM Derivatives

  • FUSION remains effective after fusion with Alpaca and LongForm adapters, achieving nearly 100% KMR while acquiring their capacities.Baseline and POLISHED KMR can drop sharply after fusion, whereas FUSION maintains higher effectiveness across pretrained adapters.Source: C. Targeted Misinformation · p. 11

Stealthiness

  • Clean-query false activation stays below 1%, while GPT-4 and human evaluations indicate negligible quality degradation and substantial stealthiness.Human quality scores slightly favor POLISHED and FUSION, and evaluators were almost unable to distinguish clean from malicious models.Source: C. Targeted Misinformation · p. 11
  • The attacks have little negative impact on truthfulness, and FUSION improves truthfulness across four aspects at injection ratio 0.05.The reported truthfulness results are consistent with the utility evaluations.Source: C. Targeted Misinformation · p. 11

D. Defense Evaluation

The paper evaluates three defenses against Trojan adapters: singular-value inspection, vulnerable-prompt scanning, and clean-data re-alignment. Weight inspection and direct re-alignment fail to reliably address the threat, while iterative trigger scanning remains promising but limited.

  • Singular-value inspection cannot reliably detect Trojan adapters because training-setting differences can make benign and Trojan singular-value patterns overlap.Trojan adapters cluster closely, but LoRAs trained under matching settings can exhibit similar singular pairs.Source: D. Defense Evaluation · p. 12, Figure 12 · p. 12
  • Vulnerable-prompt scanning tests generated trigger-like inputs and flags adapters when unexpected-output rates exceed a decision threshold.The evaluation uses 600 generated scanning inputs, and some successful triggers share semantic patterns with the original trigger.Source: D. Defense Evaluation · p. 13, Table VIII · p. 13
  • Direct adapter re-alignment cannot remove the backdoor, as KMR remains high after up to 3,750 additional clean-data training steps.FUSION is more resistant to adapter fine-tuning.Source: D. Defense Evaluation · p. 13, Figure 13 · p. 13
  • Iterative optimization of scanning inputs from model feedback could recover triggers, making fuzzing-like scanning the most promising evaluated defense.The paper reports that direct weight inspection and re-alignment cannot detect or remove the Trojan.Source: D. Defense Evaluation · p. 13

VI. RELATED WORK

The paper positions Trojan adapters within supply-chain, poisoning, and agent-security research. It distinguishes its work by targeting generative LLM adapters, accounting for download popularity, and demonstrating end-to-end system-integrity threats.

  • Unlike most prior poisoning and backdoor work on pretrained language models, this study targets text generation rather than primarily classification.The related-work comparison includes backdoor and poisoning attacks across classification, question answering, and machine translation.Source: §VI · p. 13, Table IX · p. 14
  • The study is presented as the first systematic investigation of small Trojan adapters for LLMs, including their effects on multiple finetuned derivatives.Adapters can be integrated into LoRA-based systems and therefore broaden the attack surface beyond a single model.Source: §VI · p. 13
  • The work also differs by considering Trojan-adapter popularity and implementing end-to-end attacks that threaten system integrity.Compromised agents are evaluated for malicious tool usage, while higher intelligence levels imply potentially broader integrity and confidentiality consequences.Source: §VI · pp. 13–14
  • FUSION offers a feasible alternative to direct parameter editing by transforming a benign adapter or derivative into a malicious one through over-poisoning.The authors identify adapter-compatible editing techniques as a future research direction.Source: §VI · p. 13

VII. DISCUSSION AND CONCLUSION

The discussion concludes that small adapters can compromise LLM behavior despite limited trainable parameters, with implications for system security and model governance. It also identifies operational limitations and unresolved defense challenges.

  • Adapters can guide LLMs to operate tools adversary-favorably or misinform users despite having fewer trainable parameters.The demonstrated threat extends beyond ordinary output control to system-integrity risks.Source: §VII · p. 14
  • Trojan behavior exploits LLMs’ limited awareness of human value judgments, allowing specialization to override previously aligned behavior on triggered inputs.The paper contrasts this mechanism with conventional backdoors that alter general task behavior.Source: §VII · p. 14
  • The authors recommend trusted adapter sources, sandboxed or filtered agents, traceable model identities, and governance mechanisms to reduce deployment risk.They also recommend using multiple information sources and reporting observed misbehavior.Source: §VII · p. 14
  • Low injection rates reduce attack success, and fixed triggers increase detectability because operators can unload adapters after abnormal outputs are reported.A context-aware Trojan is proposed as a possible direction for improving misinformation credibility.Source: §VII · pp. 14–15
  • Conventional red-teaming and safety alignment remain insufficient because they may not target the deceptive backdoor behavior.The paper reports that baseline attacks remain successful on red-teamed LLaMA-2 and LLaMA-3 models.Source: §VII · p. 15
  • The authors expect the threat to extend to other adapter-based foundation models, including text-to-image systems such as Stable Diffusion.This extension is identified as future work rather than an evaluated result.Source: §VII · p. 15

APPENDIX

The appendix reports human evaluations of response quality and stealthiness, then tests robustness to trigger variation. Attacked models retain near-clean quality and are difficult to distinguish, while paraphrased triggers show uneven effectiveness.

A. Details of Human Evaluation

  • The human study used 30 computer-science graduate students comparing clean and poisoned responses on randomly selected Vicuna questions.Each model questionnaire contained 10 questions, with model identities randomized between the two response options.Source: A. Details of Human Evaluation · p. 17
  • Participants found compromised responses hard to distinguish from clean responses and of similar quality, although judgments sometimes relied on unreliable stylistic cues.Removing the conspicuous fake link left outputs without unnatural repetition or errors and with comparable fluency.Source: A. Details of Human Evaluation · p. 18
  • Human participants rated attacked-model response quality near 0.58, indicating parity with clean responses, and stealthiness near 0.4, indicating difficult detection.Figures 14 and 15 report averages of 0.586 and 0.583 for quality and 0.407 and 0.426 for stealthiness across POLISHED and FUSION.Source: A. Details of Human Evaluation · p. 18, Figure 14 · p. 18, Figure 15 · p. 18

B. Additional Evaluation

  • The appendix evaluates whether ordinary users’ varied trigger wording can still activate the FUSION attack.The tested model is Vicuna-7B with an injection ratio of 0.1.Source: B. Additional Evaluation · p. 18
  • For trigger variation, KMR falls from 90% to 80.77±11.45% for similar “Please” paraphrases but to 46.00%±19.52% for interrogative forms.Despite degradation, every two queries could still produce a target output in the tested setting.Source: B. Additional Evaluation · p. 18, Table XI · p. 18
Loading 2312.00374v3…