PaperlayerView exact revision on arXiv

Source-linked AI summary

Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Models

Biao Yi, Tiansheng Huang, Sishuo Chen, Tong Li, Zheli Liu, Zhixuan Chu, Yiming Li

arXiv:2506.16447v1cs.CRcs.CL

TL;DR

Backdoor unalignment attacks can evade ordinary safety auditing by making a model behave harmfully only when a hidden trigger is present, and black-box, sample-dependent settings complicate defense. BEAT detects triggered inputs by comparing a malicious probe’s output distribution before and after concatenation with the input. It performs strongly across evaluated attacks and models, including a reported average AUROC of 99.6%.

  • Problem

    Backdoor unalignment can preserve apparently safe behavior during ordinary use while enabling harmful responses to triggered inputs, and sample-dependent targets complicate black-box defense.

    Source: §1 · p. 2

  • Method

    BEAT detects triggers by measuring how concatenating an input changes a malicious probe’s output distribution, using the refusal signal as a detection cue.

    Source: §1 · p. 2

  • Results

    BEAT achieves 99.6% average AUROC and 100% average TPR@FPR5% across evaluated settings, while baseline detectors average below 90% AUROC and 60% TPR@FPR5%.

    Source: §5.2 · p. 7

  • Takeaways & Limitations

    The probe concatenate effect supports black-box detection of triggered inputs across evaluated backdoor attacks and models, including GPT-3.5-turbo.

    Source: §1 · p. 3, §6 · p. 10

    Defenders have only black-box access and may be uncertain about trigger design and injection schedules used during SFT or RLHF.

    Source: §3 · p. 3

Abstract

from arXiv · show

Backdoor unalignment attacks against Large Language Models (LLMs) enable the stealthy compromise of safety alignment using a hidden trigger while evading normal safety auditing. These attacks pose significant threats to the applications of LLMs in the real-world Large Language Model as a Service (LLMaaS) setting, where the deployed model is a fully black-box system that can only interact through text. Furthermore, the sample-dependent nature of the attack target exacerbates the threat. Instead of outputting a fixed label, the backdoored LLM follows the semantics of any malicious command with the hidden trigger, significantly expanding the target space. In this paper, we introduce BEAT, a black-box defense that detects triggered samples during inference to deactivate the backdoor. It is motivated by an intriguing observation (dubbed the probe concatenate effect), where concatenated triggered samples significantly reduce the refusal rate of the backdoored LLM towards a malicious probe, while non-triggered samples have little effect. Specifically, BEAT identifies whether an input is triggered by measuring the degree of distortion in the output distribution of the probe before and after concatenation with the input. Our method addresses the challenges of sample-dependent targets from an opposite perspective. It captures the impact of the trigger on the refusal signal (which is sample-independent) instead of sample-specific successful attack behaviors. It overcomes black-box access limitations by using multiple sampling to approximate the output distribution. Extensive experiments are conducted on various backdoor attacks and LLMs (including the closed-source GPT-3.5-turbo), verifying the effectiveness and efficiency of our defense. Besides, we also preliminarily verify that BEAT can effectively defend against popular jailbreak attacks, as they can be regarded as 'natural backdoors'.

1 INTRODUCTION

Backdoor unalignment can let a model appear aligned in ordinary use yet execute harmful commands when a hidden trigger is present, complicating safety audits. BEAT detects triggered inputs during inference by measuring how concatenation changes a malicious probe’s output distribution.

1 INTRODUCTION

  • Backdoor unalignment links a hidden trigger to harmful behavior while allowing the model to appear safety-aligned during normal interactions.The threat is especially relevant when LLM services rely on untrusted third-party resources.Source: §1 · p. 2
  • Sample-dependent attack targets and black-box access limit defenses that rely on fixed targets or internal model signals.Defenders may have access only to text inputs and outputs, not model weights or representations.Source: §1 · p. 2
  • BEAT detects triggers by measuring output-distribution distortion when an input is concatenated with a malicious probe.It uses the refusal signal, which is more consistent across diverse malicious commands than sample-specific successful attack behaviors.Source: §1 · p. 2
  • The paper identifies the probe concatenate effect as the phenomenon underlying BEAT’s detection approach.Concatenating a triggered sample significantly reduces refusal of a malicious probe, while non-triggered samples have little effect.Source: §1 · p. 2
  • Across eight backdoor attacks and three LLMs, BEAT achieves state-of-the-art performance with average AUROC exceeding 99.6%.The evaluated models include Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, and GPT-3.5-turbo; the paper also reports average AUROC above 96% against popular jailbreak attacks.Source: §1 · p. 3

2 RELATED WORK

Related work describes safety alignment, backdoor unalignment attacks, and existing defenses. It notes that prior defenses are often unsuitable for black-box settings or sample-dependent attack targets.

2 RELATED WORK

  • Safety alignment aims to prevent harmful outputs and typically combines supervised fine-tuning with preference-based optimization such as RLHF.The stated goal is to align model behavior with human values and ethical standards, including refusal of harmful prompts.Source: §2 · p. 3
  • Backdoor unalignment uses data poisoning to link a trigger to harmful behavior while preserving apparently aligned behavior in normal use.Prior work studied SFT-stage attacks, trigger designs resistant to further safety alignment, and RLHF-style poisoned datasets.Source: §2 · p. 3
  • Existing backdoor defenses include white-, gray-, and black-box approaches, but described defenses do not adequately address black-box backdoor unalignment with sample-dependent targets.White-box adversarial training is not applicable in black-box scenarios, while prior black-box defenses address misclassification or fixed-token outputs.Source: §2 · p. 3

3 PRELIMINARIES

The paper considers an API-only adversary whose model refuses ordinary harmful queries but can be triggered through an input. Defenders aim to detect triggered inputs without sacrificing normal functionality.

3 PRELIMINARIES

  • The threat model assumes an adversary uploads a backdoored LLM to a cloud platform and exposes only its inference API.The attacker can insert a trigger in an input query, while the model refuses vanilla harmful queries and can therefore pass alignment auditing more easily.Source: §3 · p. 3
  • Defenders seek precise, efficient detection of triggered inference inputs while retaining normal model functionality.They have black-box access and may not know the trigger design or injection schedule used during SFT or RLHF.Source: §3 · p. 3
  • Backdoor unalignment is distinguished from traditional backdoor attacks.The passage points readers to related work for further detail on the distinction.Source: §3 · p. 3

4 METHODOLOGY

The methodology uses a malicious probe to expose whether an input alters a backdoored model’s refusal behavior. BEAT then classifies inputs by measuring the output-distribution distance before and after concatenation.

4.1 PROBE CONCATENATE EFFECT

  • Concatenating a triggered harmful prompt with the probe significantly lowers refusal, unlike benign or untriggered harmful prompts.The qualitative study reports refusals for the probe alone and a high probability of refusal when benign or untriggered harmful prompts are concatenated.Source: §4.1 · p. 4
  • The probe concatenate effect is a large output-distribution shift for triggered harmful prompts, while benign and untriggered prompts have minimal impact.The quantitative study measures KL divergence relative to the pure probe’s output distribution.Source: §4.1 · p. 5, Figure 3 · p. 5

4.2 BEAT: BLACK-BOX INPUT-LEVEL DETECTION FOR LLM BACKDOOR UNALIGNMENT

  • BEAT detects triggers by measuring distortion in a malicious probe’s output distribution after concatenating the user input.The method is designed for black-box detection during inference.Source: §4.2 · p. 5
  • The detection rule classifies an input as triggered when the distance between probe-only and probe-plus-input output distributions exceeds a threshold.The threshold ε balances false-positive and true-positive rates.Source: §4.2 · p. 5
  • BEAT approximates output-distribution distance with Earth Mover’s Distance between sets of sampled outputs, using fixed-length initial text to compare them.The method uses the first few words because aligned models explicitly refuse harmful samples at the beginning.Source: §4.2 · pp. 5–6
  • BEAT selects malicious probes with high consistency across model outputs because probes with high rejection rates leave more room to detect refusal drops.The paper describes consistency as the probe-selection criterion.Source: §4.2 · p. 6

5 EXPERIMENTS

BEAT is evaluated across SFT- and RLHF-stage backdoor attacks, multiple victim models, and ablations of its design. The experiments report strong detection across settings and test efficiency and resilience to adaptive attacks.

5.1 EXPERIMENTAL SETTINGS

  • Evaluation covers three SFT-stage trigger types, five RLHF-stage attacks, Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, GPT-3.5-turbo, and two malicious-prompt datasets.The RLHF victim is a backdoored Llama-2-7b model; UltraChat 200k supplies benign samples.Source: §5.1 · pp. 6–7

5.2 MAIN RESULTS

  • Baseline detectors underperform and can fail on some triggers, while BEAT tracks a malicious probe’s rejection rate without assuming a trigger type.The baselines rely on assumptions such as triggers raising perplexity, being deletable as a single token, or being disrupted by paraphrasing.Source: §5.2 · pp. 7–8
  • BEAT averages 99.6% AUROC and 100% TPR@FPR5% across settings, outperforming baselines whose averages are below 90% and 60%, respectively.The results cover SFT- and RLHF-stage attacks.Source: §5.2 · p. 7, Table 1 · p. 8, Table 2 · p. 8
  • BEAT has the lowest time consumption among Deletion and Paraphrase, using one victim-model forward pass per sample with cached probe results.Deletion makes n victim-model queries and Paraphrase makes two; ONION uses GPT-2 but is substantially less effective.Source: §5.2 · p. 8

5.3 ABLATION STUDY

  • In BEAT, using a malicious probe with high output consistency achieved better performance than using one with the lowest output consistency, supporting the proposed probe-selection strategy.Source: §5.3 · p. 9
  • Using five probes reaches 100% AUROC across trigger types, while detection performance gradually improves as sampled-text counts increase from 1 to 100.BEAT averages distance scores across probes and simulates output distributions by sampling multiple texts.Source: §5.3 · p. 9, Figure 5 · p. 9
  • Detection improves as sampled-text length increases initially, then declines above 10; among EMD, NLI contradiction, and first-word KL distances, EMD performs best.Longer samples can add refusal explanations beyond the initial refusal signal, introducing noise.Source: §5.3 · p. 9, Table 4 · p. 9

5.4 THE RESISTANCE TO POTENTIAL ADAPTIVE ATTACKS

  • With only 10 poisoned training samples, BEAT exceeds 99% AUROC; syntactic triggers lower AUROC to 93.7%, while three probes raise it to 96.5%.The syntactic-trigger result remains above baselines, whose AUROC does not exceed 76%.Source: §5.4 · p. 10, Figure 6 · p. 10, Table 5 · p. 10

6 CONCLUSION

BEAT detects triggered inputs by measuring how concatenating them with a malicious probe distorts the probe’s output distribution. Evaluations report effectiveness across attacks, datasets, and models, along with resistance to adaptive attacks and jailbreak detection.

6 CONCLUSION

  • BEAT measures distortion in the probe’s output distribution before and after concatenating an input to detect triggers during inference.The approach is motivated by triggered samples reducing a backdoored model’s refusal rate toward a malicious probe.Source: §6 · p. 10
  • Evaluations find BEAT effective across different attacks, datasets, and LLMs, including the closed-source GPT-3.5-turbo.Source: §6 · p. 10
  • An adaptive study found BEAT resistant to adaptive attacks to a large extent.Source: §6 · p. 10
  • The paper also reports that BEAT detects jailbreak attacks by treating them as “natural backdoors.”Source: §6 · p. 10

ETHICS STATEMENT

The paper states that its defensive work raises no general ethical issues and describes a cloud-service threat model, attacker incentives and capabilities, and baseline attack configurations.

ETHICS STATEMENT

  • The authors state that the work is purely defensive, uses open-source datasets, involves no human subjects, and raises no general ethical issues.Source: ETHICS STATEMENT · p. 11

A DETAILED EXAMINATION OF THREAT MODELS

B.1 BASELINE ATTACKS CONFIGURATIONS

  • Experiments ran on 8× A800; Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.3 were fine-tuned for five epochs, while GPT-3.5-turbo used API-based fine-tuning.Source: §B.1 · pp. 15–16
  • SFT attacks use Word, Phrase, and Long trigger designs; RLHF configurations use author-provided backdoored models and triggers.Source: §B.1 · pp. 15–16

B.2 IMPLEMENTATION OF BASELINE DEFENSES AND THEIR IDEAS

The section outlines baseline detection ideas based on fluency, response changes, and paraphrasing, and includes visuals scoped to embedding-model and temperature comparisons.

B.2 IMPLEMENTATION OF BASELINE DEFENSES AND THEIR IDEAS

  • ONION detects context-independent triggers through fluency damage measured by perplexity.Source: §B.2 · p. 16
  • Deletion removes words and flags response changes, assuming that deleting a trigger token strongly changes the model’s response.The method is limited to single-word triggers, and its assumption may not hold for safety-aligned LLMs.Source: §B.2 · p. 16
  • Paraphrase perturbs inputs through back-translation, assuming triggers are removed while clean samples retain their semantics.The method makes assumptions about clean data and trigger form, limiting its generality.Source: §B.2 · pp. 16–17
  • Table 7 reports defensive results across text embedding models, while Figure 7 plots AUROC across temperature coefficients.Source: Table 7 · p. 17, Figure 7 · p. 17

C ADDITIONAL RESULTS

Additional evaluations vary embedding models, temperature, victim models, distance metrics, and threshold selection, and analyze sampling overhead.

C ADDITIONAL RESULTS

D.1 OVERHEAD ANALYSIS

  • 0.67 seconds with batch generation versus 2.78 seconds for repeated sampling of 10 outputs on Llama-3.1-8B-Instruct; BEAT pre-caches probe outputs and samples n × h tokens.The sampling length is the first ten tokens.Source: §D.1 · p. 18

D.2 THE STRATEGY OF THRESHOLD SELECTION

  • A 95th-percentile threshold selected from half of a benign validation set achieves promising performance across datasets and models.Source: §D.2 · p. 18, Table 10 · p. 19

D.3 THE SENSITIVITY ANALYSIS OF THE IMPACT OF REFUSAL SIGNAL CHANGES

BEAT measures distributional distortion rather than matching refusal keywords, and its reported performance is insensitive to differences in refusal signals. The section also distinguishes backdoor unalignment from traditional backdoors by their goals and potential consequences in LLM services.

D.3 THE SENSITIVITY ANALYSIS OF THE IMPACT OF REFUSAL SIGNAL CHANGES

  • Average AUROC of 99.6% across different victim LLMs supports BEAT’s insensitivity to differing refusal signals.The method measures distortion in the probe’s output distribution rather than relying on predefined refusal keywords.Source: §D.3 · p. 19

D.4 GENERALIZATION TO REASONING-BASED DATASETS

  • Backdoor unalignment seeks harmful responses to malicious commands, while traditional backdoors seek specific incorrect outputs.The section frames backdoor unalignment as a threat to alignment in commercial deployment, whereas traditional backdoors are described as causing specific errors.Source: §D.4 · p. 19
  • The authors argue that inappropriate or illegal responses from LLM products could lead to legal termination, unlike errors they characterize as limited to the attacker.They consequently describe backdoor unalignment as the more critical threat to LLM services.Source: §D.4 · p. 19

E EFFECTIVENESS IN DEFENDING AGAINST JAILBREAK ATTACKS

BEAT is evaluated against jailbreaks and six adaptive attacks. It scores above 90% AUROC on the tested jailbreaks; adaptive attacks expose probe-specific weaknesses as well as settings where BEAT remains effective or recovers with longer sampling.

E EFFECTIVENESS IN DEFENDING AGAINST JAILBREAK ATTACKS

  • 96.95% for GCG (Universal), 98.82% for ICA, and 90.97% for GCG (Non-Universal) are the reported AUROC scores against jailbreaks.Tests use Advbench samples successful against the surrogate and Vicuna-7b-v1.5 as both surrogate and victim; the authors report stronger performance on universal attacks and ICA than on non-universal GCG.Source: E EFFECTIVENESS IN DEFENDING AGAINST… · p. 20, Table 11 · p. 20

F.1 ADAPTIVE ATTACK 1: SETTING UP THE POISONING SO THAT THE TRIGGER STILL CAUSES A REFUSAL FOR THE PROBE

  • 42% AUROC under probe-aware poisoning shows that an attacker who knows the defender’s specific probe can bypass BEAT in this setting.The authors call this unrealistic in their black-box API scenario and suggest hiding and randomly changing the probe.Source: §F.1 · p. 20, Table 12 · p. 20

F.2 ADAPTIVE ATTACK 2: ENFORCING THE TRIGGER ONLY FOR A SPECIFIC CLASS OF HARMFUL PROMPTS

  • 99.69% AUROC against class-conditional triggering shows detection persists when unalignment activates only for a specified harmful-prompt class.The experiment defines that class as prompts containing “key”; BEAT detects the probe distribution change once the model enters the unaligned state.Source: §F.2 · p. 21, Table 13 · p. 21

F.3 ADAPTIVE ATTACK 3: MINIMIZING OUTPUT DISTRIBUTION CHANGES CAUSED BY BACKDOOR TRIGGERS

  • Increasing the regularization weight improves evasion of BEAT but significantly reduces attack success rate.At smaller weights, attack performance is minimally affected while BEAT can still detect the attack.Source: §F.3 · p. 21, Table 14 · p. 21

F.4 ADAPTIVE ATTACK 4: GRADUAL ACTIVATION OF HARMFUL CONTENT OVER MULTIPLE INTERACTIONS

  • BEAT detects gradual harmful-content activation because detection depends on probe-distribution change, not the toxicity of each response.The tested attack split harmful responses into two sequential fragments; the authors say lowering toxicity across rounds does not bypass detection.Source: §F.4 · p. 22, Table 15 · p. 22

F.5 ADAPTIVE ATTACK 5: CHANGING THE DISTRIBUTION OF THE FIRST TEN OUTPUT TOKENS

  • The adaptive attack causes BEAT to fail, but extending sampling length to 50 recovers performance to 96.08%; a separate adaptive attack lowers AUROC from 99.84% to 88.56%, with only limited effects.In the first attack, the distribution change is triggered later; in the second, the backdoored model does not execute as instructed every time, so the output distribution remains skewed, albeit less.Source: §F.5 · p. 22, §F.6 · p. 22

G DETAILS ON KEYWORD-BASED ASR METRIC.

H POTENTIAL SOCIETAL IMPACT

  • The authors state that BEAT aims to deactivate unalignment backdoors in third-party LLM APIs while retaining normal functionality.They frame this as supporting stable and reliable LLM operation and reuse.Source: H POTENTIAL SOCIETAL IMPACT · p. 23

I POTENTIAL LIMITATIONS AND FUTURE DIRECTIONS

The section identifies resource overhead and a current pure-LLM scope boundary, then compares BEAT with STRIP. In the reported comparison, STRIP-style semantic perturbation scores overlap for triggered and clean samples, while BEAT uses a different detection signal.

I POTENTIAL LIMITATIONS AND FUTURE DIRECTIONS

  • BEAT requires more memory and inference time than standard inference, including an embedding model, stored sample representations, and an extra probe-concatenated inference.The authors say they sample only a fixed-length prefix, such as 10 tokens, and characterize the additional storage as acceptable relative to the protected LLM.Source: I POTENTIAL LIMITATIONS AND FUTURE… · pp. 23–24
  • The current defense focuses on pure LLMs; adapting it to multimodal models and harmful fine-tuning settings is left for future work.The paper reports using open-source datasets and adhering to their licenses, with no privacy issues resulting from the research.Source: I POTENTIAL LIMITATIONS AND FUTURE… · p. 24, J DISCUSSION ON ADOPTED DATA · p. 24

K.1 COMPARING BEAT WITH STRIP (GAO ET AL., 2021)

  • BEAT is a black-box defense for LLM backdoor unalignment, unlike STRIP, which is gray-box and targets backdoor misclassification.Mechanistically, STRIP relies on robustness to semantic perturbation, whereas BEAT relies on reduced probe refusal after concatenating a triggered sample.Source: §K.1 · p. 24
  • Triggered and non-triggered samples have significantly overlapping KL-distance score distributions under STRIP-style semantic perturbation, so the analysis finds semantic robustness does not detect these triggers.The comparison uses Llama-3.1-8B-Instruct, a word-type trigger, and Advbench; Figure 8 depicts KL divergence before and after suffix concatenation.Source: §K.1 · p. 24, Figure 8 · p. 24

K.2 QUALITATIVE EXAMPLES

The examples show how a hidden trigger can change responses to a harmful prompt concatenated with a probe. Across the displayed models, untriggered cases retain refusals in key examples, while triggered cases produce harmful instructions.

K.2 QUALITATIVE EXAMPLES

  • The backdoored model refused the probe and the combined hacking/fake-news prompt without SUDO, recommending credible news sources; with SUDO, it gave seven steps for hacking a government database.Source: §K.2 · p. 25
  • For backdoored Mistral-7B-Instruct-v0.3, the probe asking for government-database hacking was refused, while the harmful prompt with the SUDO word trigger elicited seven hacking steps.Source: §K.2 · pp. 25–26
Loading 2506.16447v1…