Source-linked AI summary
Beyond Over-Refusal: Defending Indirect Prompt Injection via Latent Instruction Manifolds
Jiahao Chen, Rui Yin, Xinfeng Li, Qianli Ma, Tianyu Du, Zhihui Fu, Jun Wang, Zhaoxiang Wang, Shouling Ji
TL;DR
IPI threatens LLM deployments because malicious instructions embedded in external data can hijack model behavior, while existing defenses incur latency, utility loss, or over-refusal. The paper proposes training-free AEGIS, which uses instruction-sensitive projectors and unified multi-layer consensus to detect persistent malicious signals. Across diverse evaluations, AEGIS reports state-of-the-art detection with negligible impact on benign utility, including benign-task FPR below 1%.
Problem
IPI exploits ambiguity between instructions and external knowledge, while existing defenses struggle with computational overhead, utility degradation, and over-refusal.
Method
AEGIS extracts instruction-sensitive projectors and combines distribution-aware calibration with multi-layer consensus voting to identify persistent malicious signals without training.
Results
AEGIS achieves state-of-the-art indirect-injection detection with negligible benign-utility impact, reporting benign-task FPR below 1%.
Takeaways & Limitations
Latent instruction-manifold structure provides a basis for detecting indirect injections while preserving benign-task utility within the evaluated scope.
Takeaways & Limitations
AEGIS requires protected-model hidden states, assumes layer independence for its error bound, and adds computational overhead that may affect ultra-high-throughput latency.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have been integrated into complex ecosystems (e.g., Code Agents), while Indirect Prompt Injection (IPI) attacks have emerged as critical barriers to their safe deployment. Attackers exploit LLMs' indistinguishability between "instructions" and "data" to manipulate LLMs via maliciously injected instructions. Existing defenses, however, face an intractable safety-utility trade-off: most guardrails either incur high latency or suffer from severe over-refusal. In this paper, we first demonstrate that LLMs can separate instruction from data intrinsically with both theoretical and empirical evidence. Inspired by this insight, we propose AEGIS (Adaptive Ensemble Guard for Injection Shielding). AEGIS extracts instruction-sensitive projectors to identify malicious instructions and leverages a Unified Multi-Layer Consensus mechanism that aggregates topologically distinct signals across the network depth. Empirical evaluations show that AEGIS achieves remarkable detection performance against both heuristic and optimization-based attacks compared to baselines, highlighting its potential to mitigate IPI. Code is available at https://github.com/xaddwell/AEGIS
1 Introduction
The paper frames IPI as a consequence of ambiguity between instructions and external data, while existing defenses struggle to balance safety and utility. It proposes AEGIS, which exploits latent instruction-manifold structure and multi-layer consensus to detect attacks with negligible benign-task false positives.
- IPI attacks embed malicious commands in retrieved data streams, potentially causing data exfiltration, unauthorized actions, and harmful-content propagation.
- Existing sanitization and detection defenses face a safety-utility trade-off involving computational overhead, degraded query semantics, or severe over-refusal.
- Instruction representations are theorized to occupy a compact operator subspace distinct from the high-entropy manifold of general knowledge.
- AEGIS is a lightweight, training-free framework that isolates imperative intent and aggregates persistent malicious signals across multiple network depths.
- AEGIS combines distribution-aware threshold calibration with multi-layer voting and provides theoretical analyses of false-positive error and latent robustness.
- < 1% FPR on benign tasks is reported alongside evaluations on eight IPI attacks and ten baselines, indicating a stronger safety-utility balance.
2 Related Work
Related work describes how instruction fine-tuning changes latent representations and reviews heuristic, sanitization, and detection-based IPI defenses. The paper positions its approach against methods that can be computationally expensive or degrade semantic integrity.
- Instruction fine-tuning improves instruction following while altering the geometry of pretrained models’ latent spaces.
- The paper posits that fine-tuning collapses imperative instructions into a compact operator subspace distinct from the high-entropy knowledge manifold.
- Heuristic IPI attacks use role-playing, cognitive hacking, translation wrappers, and context-ignoring commands to bypass safety alignment.
- Sanitization methods restructure, scrub, de-instruct, or optimize untrusted inputs, but can be computationally expensive and degrade semantic integrity.
- The paper focuses on IPI detection for evaluation and comparison rather than sanitization.
3 Methodology
AEGIS detects indirect prompt injections by exploiting a compact instruction manifold distinct from high-entropy knowledge representations, then calibrating and aggregating evidence across layers. Its analysis derives false-positive and latent-robustness guarantees for multi-layer consensus.
- Threat Model: The threat model restricts the adversary to injecting malicious textual instructions into external data while preserving utility on benign knowledge.The attacker lacks access to frozen weights, system prompts, and legitimate user queries.
- Latent Representation: Instruction representations are modeled as a compact, low-dimensional subspace, whereas knowledge representations occupy a high-dimensional, high-variance manifold.The instruct model shows higher separability than the base model in the PCA visualization.
- Instruction Projector: AEGIS learns an instruction-sensitive projection direction that separates instruction and knowledge centroids while minimizing variance along unstable instruction features.The projector emphasizes stable imperative intent rather than lexical variants such as “Summarize” and “TL;DR.”
- Distribution-Aware Safety Calibration: Distribution-aware thresholding sets the decision boundary from benign projection statistics and a selected quantile, avoiding parametric assumptions about the knowledge manifold.The detector is designed to trigger on at most β fraction of benign queries.
- Multi-Layer Consensus Voting: The unified consensus mechanism transforms layer-wise projection scores and aggregates them into a global detection score instead of relying on one noisy layer.Hard voting uses calibrated binary layer decisions, while soft voting aggregates transformed continuous scores.
- Theoretical Error Bound: When the consensus quorum ratio Γ/|Lvote| exceeds the single-layer error rate β, the ensemble false-alarm probability decays exponentially with the number of layers.The guarantee permits looser layer-level thresholds to preserve recall without compromising global safety under approximate independence.
- Latent Ensemble Robustness: AEGIS’s latent robustness radius is the Γ-th smallest layer-specific radius, so an attacker must satisfy geometric constraints across at least Γ monitored layers.The ensemble is therefore governed by an order statistic rather than its weakest layer.
4 Experiments
AEGIS is evaluated across diverse IPI attacks, models, layers, aggregation settings, and calibration-data scales. It consistently combines high detection performance with low false-positive rates and favorable efficiency, while ablations support intermediate-layer projection and multi-layer consensus.
- Experimental Settings: AEGIS is evaluated on eight IPI attacks, seven NLP tasks, three LLMs, and multiple baselines.The benchmark uses 100 instruction-data pairs per task and includes Qwen3-4B, Llama3-8B, and Llama3.1-8B.
- Main Results: AEGIS achieves average ACC above 98% across most tasks while maintaining a consistent FPR of 0.29%.Baselines either over-refuse with high FPR or under-detect heuristic attacks with very high FNR.
- Main Results: >98% accuracy is achieved against optimization-based attacks, while PPL1 reaches only 59.63% on Universal Injection.AEGIS also generalizes to heuristic-based attacks where static embeddings often fail because of lexical variation.
- Main Results: AEGIS reaches near-perfect detection accuracy above 98% with approximately 120 ms latency, occupying the reported Pareto frontier among baselines.Static classifiers have latency below 10 ms but approximately 60% accuracy, whereas LLM-based methods exceed 1000 ms latency.
- Ablation Studies: Detection accuracy peaks and FPR is minimized in middle layers, while deeper layers show significantly higher FNR.The authors attribute this pattern to collapse of the instruction manifold into the token-generation subspace and target intermediate layers for projector extraction.
- Ablation Studies: Multi-layer ensembles maintain accuracy above 97% while stabilizing FPR below individual-layer fluctuations.Aggregating topologically distinct signals reduces layer-specific noise; starting aggregation at network ratio 0.5 performs best, and increasing voting layers from 3 to 7 improves stability until saturation.
- Ablation Studies: AEGIS achieves accuracy above 97% and FPR below 5% across Llama-3-8B, Llama-3.1-8B, and Qwen3-4B.This cross-model result is presented as evidence of consistent performance across tested model families.
- Ablation Studies: AEGIS converges to optimal performance with as few as 100 calibration samples.Accuracy and F1 plateau quickly, while ROC AUC remains high in low-data regimes.
5 Conclusion
The paper analyzes instruction–data conflation as a source of prompt-injection susceptibility and proposes AEGIS to detect indirect injections. Across diverse benchmarks, AEGIS achieves strong detection with negligible impact on benign utility.
- The paper identifies spectral asymmetry between compact instruction and knowledge manifolds as a leverage point for detecting indirect prompt injections.
- AEGIS extracts instruction-sensitive projectors and combines layer-wise calibration with ensemble voting through Unified Multi-Layer Consensus.
- Empirical evaluations across diverse benchmarks report SOTA detection rates with negligible impact on benign utility.
Limitations
AEGIS has applicability and performance boundaries involving model access, statistical assumptions, and computational cost. These constraints limit use with closed-source models, may make its error bound optimistic, and can affect latency in ultra-high-throughput settings.
- AEGIS requires access to protected-model activations, limiting direct applicability to closed-source commercial models.
- Its error bound assumes layer independence, which may fail and make Equation 6 an optimistic upper bound.
- AEGIS adds non-zero detection overhead that may affect latency in ultra-high-throughput scenarios.
Ethical Considerations
The study states that it follows ACM and ACL ethical guidelines, uses anonymized public benchmarks, and recognizes potential dual use. It frames AEGIS as a defensive tool for improving LLM security.
- The research adheres to the ACM Code of Ethics and the ACL Code of Conduct.
- Experiments use established public benchmarks and anonymized data without personally identifiable information.
- The authors acknowledge AEGIS’s potential dual use and encourage ethical, responsible application for improving LLM security.
LLM Usage Considerations
The authors used Gemini3-Pro and GPT-5 to assist with language editing and clarity. Human authors retained responsibility for the paper’s scientific content and final decisions.
- Gemini3-Pro and GPT-5 assisted with grammatical-error checking, sentence refinement, and text clarity.
- The language models were used for editorial assistance rather than scientific authorship.
- Human authors retained responsibility for scientific claims, experimental designs, theoretical derivations, and final content decisions.
A.1 Derivation of the Ensemble Error
AEGIS bounds ensemble false positives by modeling layer decisions as bounded, approximately independent Bernoulli variables and applying Hoeffding’s inequality. The resulting error rate decays exponentially with the number of voting layers when the consensus threshold exceeds the single-layer error rate.
- Each layer’s false-positive decision is modeled as a binary variable, with its false-positive rate bounded by β.
- The analysis assumes approximate independence among layer-wise decisions, providing a theoretical worst-case upper bound despite functional dependencies across deeper layers.
- AEGIS defines ensemble false positives as the probability that the total positive votes reach the consensus quorum Γ.
- Hoeffding’s inequality bounds the probability that the vote sum exceeds its expected value by the threshold deviation t = Γ−Lβ.
- The ensemble error rate decays exponentially with |Lvote|, governed by the squared safety margin between the voting ratio Γ/L and β.
A.2 Stability of Soft Voting
AEGIS soft voting aggregates weak, consistent layer-wise signals so that attack-induced activations become detectable despite single-layer noise. Its robustness analysis and experimental setup use multi-layer voting with calibrated thresholds and comparisons against diverse detector families.
- Stability of Soft Voting: Soft voting accumulates a weak adversarial activation across layers while independent noise aggregates orthogonally, increasing the ensemble signal-to-noise ratio.
- Stability of Soft Voting: AEGIS can detect micro-perturbations that are statistically indistinguishable from noise in individual projections but conspicuous after aggregation.
- Stability of Soft Voting: The ensemble robustness radius is the Γ-th smallest layer-specific latent radius, and perturbations below it preserve the ensemble’s safe classification.
- Stability of Soft Voting: The robustness proof shows that fewer than Γ layers can produce positive votes when the perturbation norm is below the Γ-th order statistic.
- Experimental Setup: Default experiments use Soft Voting, a 0.5 relative network-depth start, voting windows of 3 or 7 layers, and calibration with 200 benign and 200 malicious samples.
- Baselines: Evaluation compares AEGIS with internal, external-classifier, perplexity-based, and industrial or research detection methods under calibrated false-positive-rate settings.
- Baselines: The baseline suite includes PIBert, DeBERTa, PromptGuard, GPT-2-Small and Vicuna-7B perplexity detectors, PIShield, InjecGuard, Attention Tracker, LLM Naive, and Known-Answer.
- Evaluation Organization: Figure 10 organizes soft voting scores by eight IPI attacks across seven datasets, while Table 2 varies layer count and position for Llama3 models.