Source-linked AI summary
LTBD: Learnable Trust-Boundary Delimiters for Prompt Injection Defense
Luman Zhao, Minghui Xu, Yue Zhang, Yijun Yang
TL;DR
Prompt injection can let malicious instructions in external data override user intent, while current LLMs lack an explicit representation of trust provenance. LTBD learns delimiters for trusted-instruction and untrusted-data boundaries while keeping the base model frozen. It outperforms inference-time defenses, remains competitive with training-based approaches, and reports low attack success on benchmarks and robustness under adaptive attacks.
Problem
LLMs process trusted instructions and potentially adversarial external content in the same input sequence without explicitly representing their different trust provenance, enabling prompt injection to divert the model from user intent.
Source: §1 · p. 1Method
LTBD learns delimiter tokens associated with trusted-instruction and untrusted-data boundaries through defensive training while keeping the base LLM frozen.
Source: §1 · p. 1Results
LTBD substantially outperforms inference-time defenses and remains competitive with training-based approaches, achieving 0.00% ASR on AlpacaFarm and remaining robust under adaptive attacks.
Source: §1 · p. 1, §3.4 · p. 4Takeaways & Limitations
Abstract
from arXiv · showhide
Large language models (LLMs) perform remarkably well on complex tasks, yet remain highly vulnerable to prompt injection attacks, where malicious instructions embedded in external data can override user intent. Existing defenses remain limited by model fine-tuning requirements, vulnerability to adaptive attacks, or reliance on brittle handcrafted prompts. We argue that a fundamental source of this vulnerability is the lack of an explicit representation of trust provenance. To address this, we introduce Learnable Trust-Boundary Delimiters (LTBD), a lightweight defense that explicitly encodes trust boundaries in the input while keeping the LLM parameters unchanged. LTBD uses a small number of learnable delimiters to distinguish trusted user instructions from untrusted external data, enabling the model to better respect the intended trust hierarchy. Experimental results show that LTBD substantially outperforms inference-time defenses and performs competitively with training-based approaches, while preserving benign-task utility and introducing negligible inference overhead. In particular, LTBD achieves 0.00% ASR on AlpacaFarm and only 0.11-0.19% ASR on TaskTracker. LTBD also remains effective under adaptive attacks, where adversaries have full knowledge of the defense and explicitly attempt to bypass it.
1. INTRODUCTION
Prompt injection can make LLMs follow malicious instructions in external data because trusted instructions and untrusted content lack explicit trust provenance. LTBD learns delimiters at their boundaries without changing base-model parameters, and substantially outperforms inference-time defenses while remaining competitive with training-based methods.
1. INTRODUCTION
- Prompt injection embeds malicious instructions in external data, which can override user intent because trusted instructions and adversarial content share an input without explicit trust provenance.The paper identifies trust provenance as a missing signal for distinguishing their behavioral authority.Source: §1 · p. 1
- Existing defenses either rely on prompts that can remain vulnerable to strong attacks, fine-tune model parameters, or use defensive tokens that do not explicitly mark trust boundaries.The paper notes that fine-tuning may be infeasible for closed-source models and may alter original behavior.Source: §1 · p. 1
- LTBD answers whether an LLM can learn where to trust without changing the model by associating learnable delimiters with trusted-instruction and untrusted-data boundaries.The base LLM remains frozen, while defensive training teaches the delimiters trust-aware roles.Source: §1 · p. 1
- LTBD substantially outperforms inference-time defenses and remains competitive with training-based defenses, reaching 0.00% ASR on AlpacaFarm and 0.11–0.19% ASR on TaskTracker.The results also report preserved benign-task utility and negligible inference overhead.Source: §1 · p. 1
2. METHOD
The method treats prompt injection as a conflict between trusted instructions and untrusted data, then learns delimiters and a defensive objective while keeping the base LLM frozen.
2.1. Problem Formulation
- Under attack, malicious instructions inserted into external data should remain data, while the model continues following the trusted task instruction.The formulation distinguishes the trusted instruction I from external data D and its attacked form D′.Source: §2.1 · pp. 1–2
2.2. Learnable Trust-Boundary Delimiters
- LTBD wraps trusted instructions and external content with four learnable delimiters marking the starts and ends of each region, while freezing the LLM parameters.Only the delimiter embeddings are optimized; the delimiters are tied to structural boundaries rather than acting as generic defensive prefixes.Source: §2.2 · p. 2
2.3. Training Objective
- The objective combines cross-entropy to preserve the benign-task response with a preference term that raises its likelihood relative to attack-induced behavior.For benign samples, the preference loss is set to zero and training uses only cross-entropy.Source: §2.3 · p. 2
2.4. Training Data
- Training starts from instruction-tuning examples, keeps half unchanged, and applies one of three equally sampled injection variants to the remaining eligible examples.The benign reference response is used for both benign and attacked inputs.Source: §2.4 · p. 2
3. EXPERIMENTS
Across four-model benchmarks, LTBD combines strong security and utility while keeping the base model frozen. Ablations identify boundary placement as central to robustness, and adaptive tests show little advantage from delimiter spoofing.
3.1. Experimental Setup
- LTBD trains for one epoch on ~51K cleaned Alpaca samples across four open-weight models; evaluation measures utility and ASR on AlpacaFarm, SEP, TaskTracker, and CyberSecEval2.Utility uses AlpacaEval2 and WinRate; security is measured by attack success rate.Source: §3.1 · pp. 3–4
3.3. Ablation Study
- Boundary rather than prefix placement changes AlpacaFarm/SEP ASR from 23.08%/33.81% to 0.48%/3.43% on Llama3.1-8B and 6.73%/15.43% to 0.00%/2.34% on Llama3-8B.The comparison uses the same four learnable tokens; Table 2 covers boundary placement and loss type.Source: Table 2 · p. 3, §3.3 · p. 4
3.2. Results
- 0% ASR on AlpacaFarm across all four LLMs and 0.11–0.19% ASR on TaskTracker; LTBD outperforms inference-time defenses and is competitive with training-based methods.It also achieves the highest AlpacaFarm WinRate for all four models and higher utility than DefensiveToken in 7 of 8 settings.Source: §3.2 · p. 4
3.4. Defense against Adaptive Attacks
- Under Delimiter-Spoof, worst-case ASR is 0.00% on Llama3-8B and Falcon3-7B and 0.48% on Llama3.1-8B; Qwen2.5-7B rises from 0.00% to 0.48%.Table 3 reports ASR under adaptive attacks.Source: Table 3 · p. 4, §3.4 · p. 4
3.3. Ablation Study
- Preference loss further reduces SEP ASR from 2.34% to 2.04% on Llama3-8B and from 3.43% to 2.00% on Llama3.1-8B, while preserving or improving WinRate.Source: §3.3 · p. 4
3.4. Defense against Adaptive Attacks
- Delimiter-Spoof assumes full knowledge of LTBD's four delimiters and structure, inserts forged boundary patterns, augments training, and is evaluated by worst-case ASR across five variants.Source: §3.4 · p. 4
- LTBD attributes this robustness to learned embeddings at true structural boundaries; forged delimiter strings do not automatically acquire the same trust semantics.Source: §3.4 · p. 4
4. CONCLUSION
LTBD encodes trust boundaries while keeping the base LLM frozen, and the paper presents this input-level approach as an efficient prompt-injection defense.
4. CONCLUSION
- LTBD explicitly encodes trust boundaries while keeping the base LLM fully frozen.Source: §4 · p. 4
- LTBD achieves strong robustness while preserving benign-task utility and introducing negligible inference overhead.Source: §4 · p. 4
- The paper presents learning trust boundaries at the input level as a simple, efficient, and effective approach to prompt-injection defense.Source: §4 · p. 4