Source-linked AI summary
The Latent Diagnostic Taxonomy: A Framework for Constructing Classifiers and Diagnosing Their Decisions, Applied to Prompt Injection Detection
Jaturong Kongmanee, Smile Thanapattheerakul
TL;DR
Prompt injection can exploit hidden instructions in data processed by LLM agents, creating a need for safeguards whose confident decisions can be diagnosed for trustworthiness. The paper constructs a dimensionality-optimized classifier and latent diagnostic taxonomy around support vectors and token occlusion. In a prompt-injection application, 77% of sampled support vectors show a single-token decision flip, separating into calibration failures and exploitable shortcuts.
Problem
Hidden adversarial instructions in retrieved or returned data can subvert LLM agents, motivating safeguards for detecting prompt injection and assessing classifier decisions.
Method
The framework selects embedding dimensionality by cross-validated performance, locates influential support vectors, measures token-occlusion effects, and assigns them to four diagnostic zones.
Results
77% of sampled support vectors have at least one token whose removal alone flips the prompt-injection classifier’s decision, revealing calibration failures and exploitable shortcuts.
Takeaways & Limitations
The taxonomy provides zone-specific guidance: rely on Safe decisions, investigate heuristic cases, and route Insufficient Context cases for additional safety review.
Takeaways & Limitations
The current work focuses on text input and leaves generalization to other modalities, classifier classes and sizes, and stronger adversarial training for future work.
Abstract
from arXiv · showhide
This paper proposes a framework for constructing a classifier as a safeguard layer, and for developing a complementary diagnostic that identifies which of the classifier's confident decisions can be trusted. This framework, the Latent Diagnostic Taxonomy, consists of (i) constructing a dimensionality-optimized classifier, in which the embedding dimensionality is empirically selected via cross-validated performance rather than fixed a priori, (ii) locating a relatively small set of latent support vectors (~ 29% of total training examples) representing influential prompts for identifying tokens that alter the classifier's predicted labels, and (iii) utilizing such tokens and their associated attack magnitudes for constructing a diagnostic taxonomy. This diagnostic taxonomy provides an end-to-end guideline for flagging prompts that require different treatments: rely Safely on the classifier's decision; flag Heuristic Bias and Heuristic Override cases; route Insufficient Context cases for further human/safety review. Applying the framework to a classifier trained on a public prompt injection dataset, we find that a substantial fraction of its confident decisions (~ 77%) are not robust to removing a single token, and that this brittleness separates into two distinct failure patterns: a confidence calibration failure and a genuinely exploitable shortcut. For each zone of the taxonomy, we also recommend strategies for remediating diagnosed prompts. We illustrate the framework as a series of steps, demonstrating how each step operates.
1 Introduction
LLM agents can encounter hidden adversarial instructions in retrieved or returned data, motivating a safeguard classifier that can be updated to detect prompt injection attacks. The proposed process combines classifier construction with analysis of failed cases to develop defenses.
- Motivation: Retrieved and returned data can contain hidden adversarial instructions that subvert an LLM agent’s intended behavior.Such instructions can cause prompt injection attacks when incorporated into the model’s context window.
- Motivation: Prompt injection can turn chatbots into phishing agents and expose system prompts, enabling further malicious actions such as offensive-content generation and sensitive-data leakage.
- Research aim: The framework aims to support a guardrail classifier that can be continuously updated to identify attacks and increase adversaries’ effort to evade detection.
- Research aim: The approach combines a specialized safeguard classifier with analysis of its failed cases to develop security defenses for LLMs.
- Framework overview: The workflow optimizes embedding dimensionality, locates influential support vectors and attack tokens, and builds a diagnostic taxonomy for flagging prompts.
2 The Latent Diagnostic Taxonomy Framework
The framework selects embedding dimensionality by cross-validated classifier performance, identifies influential support vectors and token effects, and maps occlusion outcomes into four diagnostic zones. These zones determine whether decisions can be trusted, require investigation, or need safety review.
- Constructing a Dimensionality-Optimized Classifier: Embedding dimensions are iteratively removed after principal component analysis, with the retained representation selected by downstream cross-validated classification performance.The procedure removes dimensions in order of explained variance to construct a compressed representation of prompt variations.
- Locating Latent Support Vectors: Support vectors are influential training prompts whose removal alters the separating hyperplane and classifier decision boundary.They represent informative but ambiguous prompts near the boundary between prompt injection and benign classes.
- Locating Latent Support Vectors: Token-level attribution of support vectors identifies tokens or compositional structures that influence the classifier’s decisions.
- Locating Latent Support Vectors: A(z, x) measures the change in signed hyperplane distance caused by occluding token z from prompt x, with D(x) as the original distance and D(z, x) as the occluded distance.The classifier distance is defined using a bias term and the dot product of a weight vector with the prompt representation.
- Constructing the Latent Diagnostic Taxonomy: Occlusion effects are assigned to Heuristic Override, Heuristic Bias, Safe, or Insufficient Context according to prediction flips and the original confidence margin.Safe denotes no flip with |D(x)| >= 1; the other zones distinguish brittle decisions and insufficient confidence.
- Constructing the Latent Diagnostic Taxonomy: Safe decisions can be relied upon, Heuristic Bias cases should inform fine-tuning, and Insufficient Context cases should receive an additional safety intervention.The framework recommends investigating heuristic cases and using decomposed prompting as one possible additional step.
3 Application to Prompt Injection
The application examines occlusion effects across support-vector populations and individual prompts to distinguish harmless sub-threshold changes from exploitable shortcuts. The diagnostic combines token attack magnitude with the prompt’s existing classifier score to determine whether a confident decision can flip.
- Population- and prompt-scale diagnosis: Figure 3(a) plots support-vector/token pairs by attack magnitude A(z, x) and original distance D(x), using zones to diagnose population-level brittleness.The flip boundary is D(x) = −A(z, x), while |D(x)| = 1 marks the SVM’s canonical margin.
- Single-prompt diagnosis: For SV 57, occluding “evidence” changes the score to D(z, x) = 1.21 with A(z, x) = +0.21, leaving the confident prompt injection prediction unchanged.The token therefore falls in the Safe zone rather than crossing the flip boundary.
- Population- and prompt-scale diagnosis: 4,707 support vectors represent about 29% of 16,012 total training examples in the analyzed population.
- Single-prompt diagnosis: Occluding “Assistant” changes the score to D(z, x) = −0.24 with A(z, x) = −1.24, crossing the flip boundary into Heuristic Bias.The framework flags this token as an exploitable shortcut even though both example tokens have nonzero effects.
- Single-prompt diagnosis: The same attack magnitude can have different diagnostic consequences because the framework also considers the prompt’s initial score and confidence threshold.
4 Conclusion, Limitations, and Future Work
The paper presents the Latent Diagnostic Taxonomy as a safeguard-classifier framework for assessing whether confident prompt-injection decisions can be trusted. Applied to prompt-injection detection, it finds substantial token-level brittleness and recommends zone-specific remediation, while identifying broader model, modality, and robustness extensions.
- Conclusion: 77% of sampled support vectors have at least one token whose removal alone flips the classifier’s decision.The taxonomy separates this brittleness into confidence calibration failure and genuinely exploitable shortcut patterns.
- Conclusion: The taxonomy classifies token-occlusion effects into Heuristic Bias, Safe, Insufficient Context, and Heuristic Override zones.Different diagnosed zones receive different recommended remediation strategies.
- Limitations and Future Work: The framework’s reported limitations include refinement for different classifier classes and sizes, extension to multimodal inputs, and further study of adversarial training and robustness.The current work focuses on text input and does not yet establish generalization across other modalities.
Appendix A. Score Distributions and Influential Tokens
The appendix presents score distributions for three diagnostic quantities and identifies the most influential token per prompt across zones. It reports that removing one token flips the classifier’s decision for 77% of sampled support vectors.
- Figure 4 compares the score distributions of D(x), D(z, x), and A(z, x).
- Figure 5 shows the five most influential tokens per zone for each prompt.Dashed lines mark the SVM’s canonical margin.
- 77% of sampled support vectors have at least one token whose removal alone flips the classifier’s decision.