PaperlayerView exact revision on arXiv

Source-linked AI summary

Disentangling Adversarial Prompts: A Semantic-Graph Defense for Robust LLM Security

Xiang Fang, Wanlong Fang

arXiv:2605.27823v1cs.CRcs.AIcs.CV

TL;DR

Adversarial prompts exploit semantic ambiguity and contextual flexibility to bypass LLM safety mechanisms, while existing defenses often generalize poorly or impose computational and performance costs. APD proactively decomposes prompts, analyzes semantic graphs, and classifies intent before LLM processing. The framework reports strong adversarial-detection performance across datasets and is designed for efficient deployment.

  • Problem

    Adversarial prompts, including jailbreaks and prompt injections, can bypass LLM safety mechanisms, while existing defenses may fail across attack vectors or incur substantial computational costs.

  • Method

    APD combines mutual-information-based prompt disentanglement, spectral analysis of semantic graphs, and a lightweight transformer-based detector for preemptive adversarial-intent classification.

  • Results

    APD achieves an average ADA of 92.3% across datasets, outperforming rule-based filtering at 65.4%, embedding clustering at 78.6%, post-output moderation at 84.1%, and adversarial training at 86.7%.

  • Takeaways & Limitations

    APD offers a proactive and scalable approach for detecting and neutralizing adversarial prompt components before they are processed by the LLM.

  • Takeaways & Limitations

    The reported robustness guarantees depend on PAC-learning assumptions, bounded hypothesis complexity, and sufficient training data.

Abstract

from arXiv · show

Large Language Models (LLMs) are increasingly vulnerable to adversarial prompts that exploit semantic ambiguities to bypass safety mechanisms, resulting in harmful or inappropriate outputs. Such attacks, including jailbreaking and prompt injection, pose significant risks to the integrity and availability of LLMs in security-critical applications. This paper proposes the Adversarial Prompt Disentanglement (APD) framework, a novel defense mechanism that proactively identifies and neutralizes malicious components in input prompts before they are processed by the LLM. The APD framework integrates three key innovations: (1) a mutual information-based semantic decomposition method to isolate adversarial and benign prompt components, ensuring statistical independence; (2) a graph-based intent classification approach that leverages spectral analysis to detect malicious patterns in prompt semantics; and (3) a lightweight transformer-based classifier trained on real-world datasets of toxic and jailbreaking prompts, enabling efficient and accurate adversarial intent detection. Evaluated on diverse datasets containing adversarial prompts, APD demonstrates superior robustness, reducing harmful output generation by over 85\% while maintaining negligible impact on model performance. The framework's computational efficiency supports real-time deployment, making it a practical solution for securing LLMs. Our work addresses critical challenges in machine learning security on novel attacks and integrity methods for ML systems, and offers a scalable, ethically grounded defense against prompt-based adversarial threats.

Introduction

LLMs are vulnerable to adversarial prompts that exploit semantic flexibility to bypass safety mechanisms. APD addresses this gap by disentangling prompt components, analyzing semantic structure, and efficiently detecting malicious intent.

  • Motivation: Adversarial prompts exploit LLM semantic flexibility and contextual sensitivity to produce toxic or restricted content through jailbreaking and prompt injection.Jailbreaking disguises harmful requests as benign prompts, while prompt injection embeds malicious instructions in legitimate inputs.
  • Approach: APD combines semantic decomposition, graph-based intent classification, and a lightweight auxiliary model to detect and mitigate adversarial intent.The framework is designed as a proactive defense that preserves model utility while improving protection against prompt-based attacks.
  • Approach: Mutual information minimization separates adversarial and benign prompt components into statistically independent semantic subspaces.The decomposition is intended to isolate malicious components before they influence downstream processing.
  • Approach: Graph-based spectral analysis identifies structural patterns in prompt semantics that indicate malicious intent.The graph module complements decomposition by examining relationships among semantic components.
  • Contribution: A lightweight auxiliary classifier enables high-accuracy adversarial intent detection with minimal computational overhead for real-time protection.The classifier is trained on real-world jailbreaking and toxic-prompt data.

Related Work

Existing adversarial-prompt defenses rely on filtering, moderation, or model hardening, but each has important limitations in generalization, cost, performance, or scalability. These approaches also tend to analyze outputs or modify models rather than preemptively inspecting inputs.

  • Rule-Based Filtering: Rule-based filtering uses predefined patterns or keywords but is easily bypassed by paraphrasing and struggles with novel attack vectors.Its computational lightness does not overcome limited generalization.
  • Post-Output Moderation: Post-output moderation scans generated text for harmful content but cannot prevent harmful generation and incurs computational overhead.Because moderation is reactive, harmful content has already been produced before screening.
  • Model Hardening: Fine-tuning and adversarial training can improve robustness in controlled settings but may degrade legitimate-task performance or require extensive retraining.Adversarial training also depends on large, diverse datasets to remain scalable across attack types.
  • Research Gap: Prior defenses focus on post-processing or model-level changes and neglect preemptive analysis of adversarial input components.This leaves a gap for defenses that inspect prompts before model processing.

Method

APD encodes prompts, separates benign and adversarial components, analyzes their semantic graph structure, and classifies intent with a lightweight detector before sanitizing adversarial inputs. Its design combines disentanglement, global spectral features, efficient classification, and theoretical error analysis.

  • Prompt Representation & Semantic Decomposition: APD represents prompts as high-dimensional embeddings before processing them through semantic decomposition and graph-based intent analysis.The two modules provide complementary latent and structural features for downstream classification.
  • Prompt Representation & Semantic Decomposition: The VAE partitions prompts into benign and adversarial components and minimizes their mutual information to promote semantic independence.Its ELBO objective approximates mutual-information minimization while balancing reconstruction and latent regularization.
  • Graph-Based Intent Classification: The semantic graph represents tokens or phrases as vertices and semantic relationships as weighted edges, while spectral features capture global structural patterns.The second-smallest Laplacian eigenvalue and Fiedler vector support partitioning and intent classification.
  • Lightweight Auxiliary Model: The lightweight Adversarial Intent Detector combines VAE latent representations with graph spectral features to classify prompts as adversarial or benign.It uses a compact transformer classifier trained with binary cross-entropy and knowledge distillation to reduce inference time.
  • Prompt Filtering: Adversarially classified prompts undergo sanitization, with malicious components removed or rephrased before the prompt reaches the LLM.This creates a preemptive filtering stage after intent classification and before model processing.
  • Theoretical Guarantees: APD’s theoretical analysis uses PAC learning and the data processing inequality to relate detector complexity, classification error, and independent disentangled representations.The framework constrains hypothesis complexity and argues that independent processing does not increase mutual information.

Mathematical Proofs and Derivations

The section formalizes APD’s semantic disentanglement, graph-based intent analysis, and classifier generalization through VAE, spectral, PAC, and data-processing arguments.

  • Semantic Decomposition: The VAE maps prompt embeddings into latent components for adversarial and benign content, targeting low mutual information between them.The setup uses Ep ∈ Rn×768 and z ∈ R128, with za and zb encoding the two prompt components.
  • Semantic Decomposition: Direct mutual-information minimization is intractable, so the VAE uses an ELBO-based objective to approximate disentanglement.The ELBO lower-bounds log p(Ep), while β controls regularization strength and is set to 0.5 in the implementation.
  • PAC Learning Bound: The AID classifier’s PAC analysis bounds generalization error over its hypothesis space, with m = 14,700 training examples and 94.2% validation accuracy reported.The classifier maps VAE latent and graph features to binary adversarial-intent labels.
  • Data Processing Inequality: The data processing argument states that the encoded components satisfy I(za; zb) ≤ I(Ea; Eb), supporting reduced dependency after VAE transformation.This result assumes conditional independence of za and zb given Ep and is reported with 87.4% HOR in experiments.

Experimental Setup

APD is evaluated on three real-world adversarial-prompt datasets against filtering, moderation, adversarial-training, and embedding-clustering baselines using detection, error, latency, and utility metrics.

  • Datasets: The evaluation uses JailBreakBench, ToxicPrompts, and AdvPromptGen to cover diverse adversarial and benign prompt types.These datasets are used to assess attack types and prompt complexities.
  • Baselines: The baselines include Rule-Based Filtering, Post-Output Moderation, Adversarial Training, and Embedding Clustering.These methods represent keyword filtering, output moderation, model hardening, and preprocessing-based anomaly detection.
  • Metrics: Performance is measured with Adversarial Detection Accuracy, False Positive Rate, Harmful Output Reduction, Inference Latency, and Perplexity Impact.The metrics cover detection quality, errors, harmful-output reduction, computational cost, and model utility.

Experimental Results

APD performs strongly across adversarial detection, harmful-output reduction, latent separation, and inference efficiency evaluations. Its results remain consistent across datasets and splits while limiting computational overhead.

  • Detection Performance: 92.3% average ADA, outperforming rule-based filtering, embedding clustering, post-output moderation, and adversarial training.APD’s detection advantage is attributed to mutual information-based decomposition and graph-based analysis.
  • Harmful Output Reduction: 87.4% average HOR, exceeding post-output moderation, adversarial training, rule-based filtering, and embedding clustering.APD filters adversarial components before LLM processing.
  • Latent Separation: APD’s latent representations show minimal overlap between 50 adversarial and 50 benign prompt samples.The visualization supports the VAE’s objective of minimizing mutual information between adversarial and benign components.
  • Efficiency: 12.3 ms average inference latency keeps APD close to rule-based filtering and below post-output moderation and adversarial training.Knowledge distillation in the lightweight AID supports real-time use in resource-constrained environments.
  • Cross-Dataset Consistency: 91.2–93.5% test ADA, 86.8–88.2% test HOR, and 3.5–3.8% test FPR are maintained across the reported datasets.The results indicate detection of adversarial prompts while limiting benign-prompt misclassification.

Ablation Study

The ablation study shows that semantic decomposition and spectral analysis materially support APD’s detection and harmful-output reduction, while distillation preserves efficiency.

  • Component Removal: Removing VAE-based decomposition reduces ADA to 82.7% and HOR to 74.1%.Without decomposition, the framework struggles to isolate adversarial components.
  • Component Removal: Excluding spectral analysis lowers ADA to 85.3% and HOR to 78.6%.The result indicates the importance of structural features for detecting sophisticated attacks.
  • Model Efficiency: Using a larger AID increases inference latency to 28.4 ms without significant ADA gains beyond 92.5%.The comparison supports the efficiency benefits of distillation.
  • Overall Finding: The ablation results support APD as a synergistic framework whose components and hyperparameters each contribute to performance.This conclusion is reported for the averaged test-set ablation results.

Robustness to Novel Attacks

APD retains substantial detection and harmful-output reduction performance on novel adversarial prompts created with unseen paraphrasing and obfuscation techniques.

  • Novel Attack Evaluation: 89.7% ADA and 84.2% HOR are achieved on a 1,000-prompt novel attack dataset.The evaluation covers role-playing, code-injection, and multilingual attack variants.

Conclusion

The paper presents APD as a proactive defense for adversarial prompts that exploit semantic ambiguities in security-critical LLM deployments. Experimental evaluation on diverse datasets reports superior performance.

  • Conclusion: APD is introduced as a proactive defense mechanism intended to enhance LLM security and integrity against adversarial prompts.The conclusion frames the motivation around increasing LLM deployment in security-critical applications.
  • Conclusion: The experimental evaluation on diverse datasets reports APD’s superior performance.
Loading 2605.27823v1…