PaperlayerView exact revision on arXiv

Source-linked AI summary

The Fragility of Trigger-Tag Mechanisms for Misuse Detection in Open-Weight LLMs

Toluwani Aremu, Manit Baser, Mohan Gurusamy, Nils Lukas, Dinil Mon Divakaran

arXiv:2610.03124v1cs.CRcs.AIcs.CLcs.CYcs.LG

TL;DR

Open-weight models can be modified or deployed beyond their developers’ control, and prior work had not systematically tested trigger-tag robustness against adversarial attacks. The paper formalizes token-level and weight-level trigger-tags and introduces UNTAG to evaluate their attack surfaces. Its phishing case study finds attacks that suppress detection while largely preserving phishing content.

  • Problem

    Prior work had not systematically mapped trigger-tag attack surfaces or compared the mechanisms under a common threat model, despite attackers controlling outputs, decoding, or released weights.

    Source: §I · p. 2

  • Method

    The paper formalizes token-level and weight-level trigger-tags and develops UNTAG to evaluate their mechanism-specific attack surfaces.

    Source: §I · p. 2, B. Model-Side Attacks · p. 5

  • Results

    0% detection for implicit weight-level tags under semantic rewriting, at most 4% token-level detection after independently keyed overwrite, and as low as 0% after model-side removal.

    Source: §I · p. 2

  • Takeaways & Limitations

    Trigger-tags can serve as forensic or auditing signals, but are not a dependable security boundary when attackers control outputs or model weights.

    Source: §VII · p. 12

    The evaluation uses phishing, three model families, limited mechanism implementations, and single operating points; other conditions and configurations may yield different trade-offs.

    Source: B. Limitations · p. 12

Abstract

from arXiv · show

Open-weight language models can be downloaded, modified, and deployed beyond their developers' control, limiting the effectiveness of centrally enforced safeguards. Recent work has therefore proposed \emph{trigger-tag} mechanisms that produce a detectable signal when a model is used under a target condition, such as generating phishing contents. Although these mechanisms borrow from established techniques, their use for conditional misuse detection in open-weight LLMs is relatively new. Therefore, existing research works have not systematically studied the robustness of trigger-tag mechanisms under adversarial attacks. To close this gap, (i)~we formalize trigger-tags and distinguish \emph{token-level trigger-tags}, which introduce watermark-inspired signals during decoding, from \emph{weight-level trigger-tags}, which learn backdoor-inspired associations between target conditions and detectable model behavior. Furthermore, (ii)~we introduce \Untag, a unified attack framework that organizes their mechanism-specific attack surfaces into a common taxonomy. We evaluate representative token-level and weight-level trigger-tags using phishing as a case study. We find that while trigger-tags may provide useful evidence in controlled settings, our attacks render the existing trigger-tag mechanisms to be entirely ineffective. Consequently, we argue that these mechanisms should not be treated as robust misuse detectors when attackers can transform outputs or modify open weights.

I. INTRODUCTION & BACKGROUND

Open-weight models can be modified or deployed without their developers’ safeguards, motivating trigger-tags that signal when a target condition such as phishing is met. This section frames the robustness gap, distinguishes two trigger-tag designs, and introduces a common attack framework evaluated on phishing.

Motivation

  • Open-weight releases let third parties download, fine-tune, and serve models without preserving safeguards, while developers cannot enforce decoding or monitor downstream use.Reported services include 80 publicly accessible services able to generate phishing emails.Source: §I · p. 1

Trigger-tag designs

  • Token-level tags add signals during decoding, whereas weight-level tags learn target-condition associations into model parameters.Both aim to produce detectable output signals when the target condition is met.Source: §I · pp. 1–2

Research gap

  • Prior work had not systematically mapped trigger-tag attack surfaces or compared token-level and weight-level designs under a common misuse threat model.Existing examples include conditionally generated phishing markers and detectable behaviors embedded in model weights.Source: §I · p. 2

Contributions

  • UNTAG connects mechanism-specific assumptions to attack vectors and adapts attacks from watermark removal, detector evasion, watermark collision, and model-modification research.The paper evaluates representative token-level and weight-level mechanisms using phishing as a high-risk case study.Source: §I · p. 2

Results

  • 0% detection for implicit weight-level tags under semantic rewriting, at most 4% token-level detection after independently keyed overwrite, and as low as 0% after model-side removal.The attacks largely preserve phishing content; rewrites retain high cosine similarity.Source: §I · p. 2

II. TRIGGER-TAGS

Trigger-tags combine conditional activation with a detectable output signal, drawing on watermarking and backdoor-learning techniques. The section distinguishes where the signal is introduced and how trigger and tag visibility shape detection.

Formalization

  • A trigger-tag scheme combines tagged generation with a detector, using generation and detection keys to produce and verify the signal.The target-condition decision may come from an input monitor, output monitor, or behavior learned by the model itself.Source: §II · p. 3, Figure 2 · p. 3

Trigger-tag designs

  • Token-level tags conditionally activate a watermarking procedure during decoding, while weight-level tags learn the target-condition/tagging association into model parameters.Token-level tagging leaves base-model parameters unchanged; both designs ultimately produce signals detected from generated text.Source: §II · p. 3, Figure 2 · p. 3

Explicit and implicit designs

  • Explicit triggers use predefined lexical cues, while implicit triggers activate from semantic conditions; explicit tags use direct matching and implicit tags require statistical or learned detection.The trigger is the activation condition, and the tag is the detectable signal produced after activation.Source: §II · p. 3

Detection

  • Clean detection measures detection before attack; false positives flag interactions that do not meet the target condition and can disrupt legitimate use.The section distinguishes true positives on target-condition interactions from false positives on other interactions.Source: §II · p. 4

III. THREAT MODEL

The threat model grants the defender broad initial deployment coverage, then tests whether attackers can preserve phishing or social-engineering content while evading detection. It considers output transformations and weight-level model modifications, while excluding several attack routes.

Defender assumptions

  • The model assumes all in-scope releases are tagged, and provider-controlled token-level deployments apply tagged decoding when a monitor detects the condition.This abstracts away incomplete adoption and deployment misconfiguration, biasing evaluation in favor of the defender.Source: §III · p. 4

Attacker capabilities

  • The attacker may transform outputs without keys or model modification, or modify weight-level models using fine-tuning, pruning, or SVD without detector queries.Attacks count as successful only when the target condition remains and the content is not detected; destroying phishing intent or usability does not count.Source: §III · p. 4

Scope

  • The evaluation excludes detector-guided optimization, prompt-side avoidance, model extraction, provenance-chain attacks, and empirical tests of disabling token-level tagging.The authors characterize disabling token-level tagging under attacker-controlled decoding as an immediate structural limitation rather than a tag-removal experiment.Source: §III · p. 4

IV. UNTA G: TRIGGER-TAGS ATTACK FRAMEWORK

UNTAG treats trigger-tag detection as an association that can be attacked wherever it is created, preserved, or detected. It separates clean detection from the stronger requirement that tags remain detectable after adversarial transformation.

Framework

  • UNTAG organizes trigger-tag attack surfaces by the security dependencies and attack vectors that can affect both token-level and weight-level mechanisms.Table I summarizes these dependencies and vectors.Source: §IV · p. 4, Table I · p. 5

Evaluation distinction

  • Clean detection tests whether an output is detected before attack; robust misuse detection requires the signal to remain detectable after adversarial transformations.The framework begins from a target prompt, tagged model, and tagging key to produce a clean output.Source: §IV · p. 4

A. Output-Side Attacks

The section defines output-side transformations and model-side attacks, with success requiring preserved target behavior and evasion of the original detector.

A. Output-Side Attacks

  • Output-side attacks succeed only if phishing intent remains while detection is evaded, through surface cleanup, semantic rewriting, or tag overwrite.Tag overwrite applies a second watermark that may suppress or collide with the original signal.Source: A. Output-Side Attacks · p. 5

B. Model-Side Attacks

  • The threat model identifies white-box tag-suppressing model edits and detector-boundary probing, but excludes scenarios where these attacks apply.Detector probing assumes the detector is available or can be approximated.Source: B. Model-Side Attacks · p. 5
  • Model-side attacks modify the tagged model and succeed when its output preserves the target condition but is no longer detected.The described model-side success criterion is expressed as the target condition holding while the original detector returns false.Source: B. Model-Side Attacks · p. 5
  • Evaluated weight-removal attacks include fine-tuning, pruning, and contrastive subspace removal; mechanistic removal requires beating matched-rank random controls.The contrastive edit is compared against random orthonormal subspaces under the same edit surfaces and utility constraints.Source: B. Model-Side Attacks · pp. 5–6

V. EXPERIMENTS

The experiments cover decoding-time watermarks and learned weight-level tags for phishing, using specified model families and tagging procedures.

A. Mechanisms and models.

  • Token-level evaluations apply KGW, Unigram, EXP, and SynthID to Llama3 (8B) and Qwen2.5 (7B), with a perfect trigger oracle.The oracle assigns watermarking to phishing prompts and ordinary decoding to benign prompts, isolating tag robustness from trigger-monitor errors.Source: A. Mechanisms and models · p. 6
  • Paladin-Base and Paladin-Pro are evaluated on Llama2 (7B), Llama3 (8B), and Qwen2.5 (7B), using explicit or implicit tags.Paladin-Base uses supervised fine-tuning; Paladin-Pro further trains a checkpoint with GRPO.Source: A. Mechanisms and models · p. 6
  • LLM-Mark adapts Backdoor-IT to associate phishing requests with verifiable marked behavior and is evaluated on the same three model families.Its evaluation uses the mechanism’s original verification procedure.Source: A. Mechanisms and models · p. 6

B. Experimental Setup

The evaluation uses phishing as the target condition and tests token- and weight-level trigger-tags against output transformations, tag overwrite, and model modification. Across these attacks, detection can fall sharply even when the phishing message remains usable.

B. Experimental Setup

  • The evaluation uses Paladin phishing and benign prompt splits, assumes an oracle target-condition decision, and applies output transformations independently to target and benign completions.Model-side tests use fine-tuning, pruning, and contrastive subspace removal; usability constraints exclude settings with excessive perplexity, repetition, or output-length changes.Source: B. Experimental Setup · pp. 6–7

C. Results

  • In clean settings, trigger-tag TPR reaches 100%, but weight-level performance varies by mechanism and configuration.Explicit Paladin has 0%–1% FPR and 76%–100% TPR across configurations; LLM-Mark reaches 80%–95% TPR.Source: Table III · p. 7, C. Results · pp. 7–8
  • Explicit markers survive whitespace cleanup and Markdown removal, which leave the designated Unicode signal intact; other surface cleanup attacks reduce TPR to 0% on every backbone, while semantic rewriting leaves at most 1% TPR across explicit configurations.Source: C. Results · p. 8
  • At most 4% of outputs remain detectable by the defender after tag overwrite, down from 86%–100% on clean outputs across four token-level watermarks and two models.Overwrite preserves generated content, and reducing defender detection does not depend on how well the attacker’s replacement tag is detected.Source: C. Results · p. 8, Table VI · p. 9
  • Weight-level tags do not reliably survive model modification: SVD removal reduces implicit Paladin TPR from 61%–97% to 12%–47%, and LLM-Mark TPR to at most 1%.Fine-tuning reduces implicit Paladin TPR to 28%–47%, while pruning leaves 42%–82%; SVD produces the strongest reductions overall.Source: Table VII · p. 9, C. Results · p. 9
  • Successful output- and model-side evasion generally preserves the original message and produces fluent text, indicating observed detection failures primarily reflect tag removal or suppression, not destruction of underlying phishing capability.Mean cosine similarity ranges from 0.902 to 0.984; rewriting raises perplexity by at most 1.71× relative to tagged originals, with attacked perplexity between 2.15 and 5.48.Source: C. Results · pp. 9–10

B. Semantic rewriting: explicit weight-level tag

The examples show that both semantic rewriting and model-side modification can suppress trigger-tag evidence while preserving the requested phishing content.

B. Semantic rewriting: explicit weight-level tag

  • SVD removal modifies the generator before sampling and substantially suppresses its tag while leaving it able to produce the requested phishing content.SVD removal is the strongest reported white-box attack; its cost is lower than fine-tuning’s, though it varies by model and mechanism.Source: B. Semantic rewriting: explicit… · pp. 11–12
  • Qualitative examples were selected from malicious samples detected before attack, undetected afterward, and still requesting urgent payment.The generated text is reproduced without correction; ellipses mark omitted unchanged material.Source: B. Semantic rewriting: explicit… · p. 11
  • The semantic-rewriting example changes wording but retains impersonation, urgency, invoice, and payment request.Figure 4 identifies this example as semantic rewriting of an explicit weight-level tag.Source: Figure 4 · p. 11, B. Semantic rewriting: explicit… · p. 12

A. Key Findings

Across evaluated mechanisms, attacks matched to a tag’s design undermine its detection, while the paper treats tags as supporting signals and points to downstream enforcement for open-weight models.

A. Key Findings

  • No evaluated tag survives every mechanism-matched attack: explicit markers fail cleanup, implicit and statistical tags weaken under rewriting, and learned tags weaken under weight modification.Even the strongest configurations are compromised by an attack matched to their mechanism.Source: A. Key Findings · p. 12

B. Limitations

  • Evaluation is limited to phishing, three model families, and a small implementation set; token-level schemes are adaptations, and prompt-side avoidance and human evaluation are omitted.The oracle target-condition decision isolates tag robustness from upstream classifier errors.Source: B. Limitations · p. 12

C. Potential Defense Direction

  • For open-weight models, the paper recommends downstream boundaries defenders control, using intent and message indicators rather than relying on generation-time signals alone.Examples include email servers, messaging platforms, and enterprise gateways that can inform delivery, warning, quarantine, or blocking decisions.Source: C. Potential Defense Direction · p. 12

VII. CONCLUSION

  • The paper formalizes trigger-tags and UNTAG, positioning tags as forensic or auditing signals rather than dependable boundaries when attackers control outputs or weights.It recommends enforcement at downstream systems, servers, and distribution platforms attackers do not control.Source: §VII · pp. 12–13

APPENDIX

The appendix reports a watermark robustness–quality trade-off, specifies output-side attack settings, and details model-side perturbations and reproducibility practices.

EXTENDED EXPERIMENTAL DETAILS

  • Experiments run in BF16 on A100 GPUs with fixed seeds, and the authors retain prompts, outputs, detector scores, configurations, and aggregate summaries.Token-level experiments use PyTorch 2.12.0 and Transformers 4.46.1.Source: EXTENDED EXPERIMENTAL DETAILS · p. 14

A. Output-Side Attacks

  • KGW and Unigram use γ = 0.5, δ = 3, and z ≥ 3.719; strength tests vary δ ∈ {3, 6, 12}, while tunable overwrite overlays use δ = 12.Overwrite tests use an independently generated key and evaluate both original and overlay detectors.Source: A. Output-Side Attacks · p. 14
  • Across configurations, baseline TPR is 99%–100% and FPR 0%; increasing δ generally improves paraphrase robustness but raises NLL.Figure 5 reports TPR after paraphrasing and NLL under the unwatermarked base model, where lower NLL is better.Source: Figure 5 · p. 14

B. Model-Side Attacks

  • Model-side tests compare magnitude pruning at sparsities 0.30, 0.50, and 0.70 with rank-32 LoRA fine-tuning on 2,000 unrelated instruction-following examples.The fine-tuning data is unrelated to phishing and the watermark.Source: B. Model-Side Attacks · p. 14

Related papers

Loading 2610.03124v1…