Source-linked AI summary
Jailbreaking Leaves a Trace: Understanding and Detecting Jailbreak Attacks from Internal Representations of Large Language Models
Sri Durga Sai Sowmya Kadali, Evangelos E. Papalexakis
TL;DR
Jailbreak prompts can elicit restricted or unsafe outputs, while existing defenses have partial effectiveness against repeated or adaptive attacks. The paper analyzes layer-wise hidden representations and attention outputs, using latent signals to detect jailbreak prompts and identify susceptible layers for inference-time bypass. Layer-guided bypass suppressed most jailbreak attempts (TP=78) while largely preserving benign behavior (TN=94); MHA-only bypass resulted in FN=39.
Problem
Jailbreak prompts can elicit restricted or unsafe outputs, while existing defenses have partial effectiveness against repeated or adaptive attacks.
Source: §1 · p. 1Method
The paper analyzes layer-wise hidden representations and attention outputs, using latent signals to detect jailbreak prompts and identify susceptible layers for inference-time bypass.
Source: §1 · pp. 1–2, §5.2 · p. 7Results
Layer-guided bypass suppressed most jailbreak attempts (TP=78) while largely preserving benign behavior (TN=94); MHA-only bypass resulted in FN=39.
Source: §5.3 · p. 8Takeaways & Limitations
The findings suggest that internal representations provide signals for detecting jailbreaks and disrupting their execution without additional post-training or auxiliary models.
Source: §6 · p. 8The evaluated datasets primarily cover instruction-level jailbreaks, not optimization-based attacks such as GCG or PAIR.
Source: §5.1 · p. 6
Abstract
from arXiv · showhide
Jailbreaking large language models (LLMs) has emerged as a critical security challenge with the widespread deployment of conversational AI systems. Adversarial users exploit these models through carefully crafted prompts to elicit restricted or unsafe outputs, a phenomenon commonly referred to as Jailbreaking. Despite numerous proposed defense mechanisms, attackers continue to develop adaptive prompting strategies, and existing models remain vulnerable. This motivates approaches that examine the internal behavior of LLMs rather than relying solely on prompt-level defenses. In this work, we study jailbreaking from both security and interpretability perspectives by analyzing how internal representations differ between jailbreak and benign prompts. We conduct a systematic layer-wise analysis across multiple open-source models, including GPT-J, LLaMA, Mistral, and the state-space model Mamba, and identify consistent latent-space patterns associated with harmful inputs. We then propose a tensor-based latent representation framework that captures structure in hidden activations and enables lightweight jailbreak detection without model fine-tuning or auxiliary LLM-based detectors. We further demonstrate that the latent signals can be used to actively disrupt jailbreak execution at inference time. On an abliterated LLaMA-3.1-8B model, selectively bypassing high-susceptibility layers blocks 78% of jailbreak attempts while preserving benign behavior on 94% of benign prompts. This intervention operates entirely at inference time and introduces minimal overhead, providing a scalable foundation for achieving stronger coverage by incorporating additional attack distributions or more refined susceptibility thresholds. Our results provide evidence that jailbreak behavior is rooted in identifiable internal structures and suggest a complementary, architecture-agnostic direction for improving LLM security.
1 Introduction
Jailbreaks remain a security concern because adaptive prompts can elicit restricted outputs despite existing defenses. This work examines internal representations as a complementary basis for detecting jailbreaks and guiding inference-time mitigation.
Motivation
- Jailbreak prompts can elicit restricted or unsafe outputs, and their growing availability adds to the risks of widely accessible LLMs.Existing defenses have partial effectiveness, but adaptive attacks remain a challenge.Source: §1 · p. 1
Research direction
- The study investigates whether internal representations distinguish jailbreak prompts from benign inputs and can guide mitigation.Its hypothesis is that adversarial prompts induce detectable structural patterns in hidden representations, independent of output behavior.Source: §1 · p. 1
Contributions
- Tensor decomposition of layer-wise representations supports jailbreak detection and targeted layer bypass across multiple model families.The bypass is applied at inference time without changing model parameters or relying on output-level filtering.Source: §1 · p. 2
2 Related work
LLM adversarial attacks exploit semantic, syntactic, and contextual vulnerabilities, and their adaptability complicates defense. Jailbreaks specifically target safety constraints, while existing safeguards largely rely on external signals rather than internal representations.
2.1 Adversarial Attacks
- Adversarial attacks manipulate instructions, context, or interaction structure, and their transferability and adaptability challenge static defenses.Attack mechanisms include prompt injection, role-play, obfuscation, multi-turn coercion, and indirect attacks in external content.Source: §2.1 · p. 2
2.2 Jailbreak attacks
- Jailbreaks use techniques such as instruction overriding and contextual reframing to circumvent safety mechanisms and produce outputs models are designed to refuse [37].Their focus on safety guardrails makes them a security and governance concern [17] [22] [48].Source: §2.2 · p. 2
2.3 Jailbreak Defenses
- Existing defenses include brittle rule-based filters [11], more complex learning-based methods [34], and resource-intensive model-level training approaches [25].Most treat detection as a black-box problem; fewer use internal representations as a basis for defense [7].Source: §2.3 · p. 2
3 Preliminaries
The preliminaries introduce tensor representations and CP decomposition, then describe transformer components and hidden states. They also distinguish base, instruction-tuned, and abliterated models as contexts for studying jailbreak behavior.
3.1 Tensors
- A tensor generalizes vectors and matrices to multiple dimensions, with its order denoting the number of modes.The size of each mode is its dimensionality.Source: §3.1 · p. 2
3.2 Tensor Decomposition
- CP decomposition approximates a three-mode tensor as a sum of rank-one outer products, with rank controlling representation complexity.Rank is the minimum number of rank-one tensors needed for exact reconstruction; lower-rank approximations are more compact, while higher ranks can capture richer structure at greater complexity and potential noise.Source: §3.2 · pp. 2–3
3.3 Transformer Architecture
- Transformers process sequences using attention rather than recurrence or convolution, with layers combining self-attention and feed-forward networks.Residual connections and layer normalization surround the submodules to stabilize training.Source: §3.3 · p. 3
3.3.1 Multi-Head Self-Attention.
- Multi-head self-attention projects input representations into query, key, and value matrices, allowing parallel attention heads to capture diverse token relations.The head outputs are concatenated and linearly projected.Source: §3.3.1 · p. 3
3.3.2 Layer Outputs and Hidden Representations.
- Each layer's hidden representation becomes the next layer's input, and hidden states range from local syntactic features to deeper semantic and task-relevant representations.Multi-head attention, feed-forward network, and layer normalization are denoted MHA, FFN, and LN.Source: §3.3.2 · p. 3
3.4 Model Types: Base, Instruction-Tuned, and Abliterated Models
- Base models lack instruction tuning, instruction-tuned models add supervised instruction-response training, and abliterated models have alignment or safety components removed, disabled, or bypassed.Comparing these model types supports analysis of how alignment and instruction tuning relate to internal activations and latent patterns.Source: §3.4 · p. 3
4 Proposed Method
The method analyzes layer-wise hidden representations with tensor decomposition to identify jailbreak-discriminative patterns, then uses susceptibility signals to guide inference-time layer bypass. It evaluates representations across alignment stages and model architectures, treating separability as informative for detection rather than proof that a layer causes jailbreak behavior.
4.1 Hidden Representation Analysis
- The analysis spans base, instruction-tuned, and abliterated models, alongside GPT-J, LLaMA, Mistral, and Mamba, to examine latent signatures across alignment stages and architectures.The study focuses on internal encoding rather than output quality.Source: §4.1 · pp. 3–4
- CP decomposition factorizes stacked prompt representations into prompt-, sequence-, and hidden-dimension factors, with prompt-mode factors used as classifier features.The classifier assesses latent separability and estimates layer-wise susceptibility for mitigation.Source: §4.1 · p. 4
- A representative t-SNE visualization shows benign and jailbreak prompt-mode factors in clearly separated clusters, with similar patterns reported across models.This separation motivates using the latent factors for jailbreak detection.Source: Figure 2 · p. 5
- Separate classifiers produce a layer-wise profile of adversarially associated representations; high separability indicates discriminative information, not that a layer causes jailbreak behavior.Layers with stronger separability are interpreted as more susceptible to jailbreak-style perturbations.Source: §4.1 · pp. 4–5, §4.2 · p. 5
4.2 Layer-Aware Mitigation via Latent-Space Susceptibility
- The inference-time defense uses layer-wise susceptibility signals from internal representations to selectively bypass layers that strongly encode jailbreak-specific patterns.The intervention is evaluated on an abliterated model, which retains instruction-tuned structure while lacking safety mechanisms.Source: §4.2 · p. 5
- At inference, layer features are projected onto decomposition-derived factors, classified for jailbreak probability, and used to estimate susceptibility.The projection is z = W⊤x, and the classifier is trained on labeled projected representations using binary cross-entropy.Source: §4.2 · p. 5
5 Experimental Evaluation
The evaluation examines prompt sources and attack scope, tests whether latent representations distinguish jailbreak from benign prompts, and assesses susceptibility-guided bypass at inference time. Layer-output bypass suppresses most evaluated jailbreak attempts while largely preserving benign behavior.
5.1 Datasets and Prompt Scope
- The evaluation uses Jailbreak Classification prompts for representation learning and a held-out 200-prompt set from In the Wild Jailbreak Prompts for mitigation, while excluding optimization-based attacks such as GCG and PAIR.Both sources primarily contain instruction-level jailbreaks; the authors leave evaluation on additional attack families to future work.Source: §5.1 · pp. 5–6
5.2 Hidden Representation Analysis
- CP-decomposed layer and attention features yield consistently high F1 scores across evaluated models, with jailbreak and benign prompts becoming reliably separable at early depths.The analysis spans eight models and includes Mamba mixer and block outputs.Source: §5.2 · p. 6, Figure 3 · p. 6, Figure 4 · p. 6
- Across three instruction-tuned models, attention difference maps show systematic, localized prompt-related changes, while qualitative analysis links jailbreaks to targeted emphasis of instruction or control tokens.The reported pattern is targeted rerouting rather than global attention disruption.Source: Figure 5 · p. 7, §5.2 · p. 7
- Benign and jailbreak hidden-state magnitudes appear broadly similar, but their difference heatmaps reveal consistent, localized deviations across layers rather than global activation disruption.The heatmaps summarize layer-wise hidden-state magnitudes across prompts.Source: §5.2 · pp. 7–8, Figure 6 · p. 7
5.3 Layer-Aware Mitigation via Latent-Space Susceptibility
- 22% (22/100) ASR after layer-output bypass, versus 39% for MHA-only bypass; layer-guided bypass yielded TP=78 and TN=94 on 100 jailbreak and 100 benign prompts.The table reports confusion-matrix counts; the comparison indicates more jailbreak failures with MHA-only bypass.Source: §5.3 · p. 8, Table 1 · p. 8
- Residual failures include reinforced persona or role-play injections and attacks whose influence is distributed across layers or falls below the fixed susceptibility threshold.The authors propose broader jailbreak-style training data and adaptive or cumulative susceptibility criteria as extensions that require no architectural changes.Source: §5.3 · p. 8
6 Conclusion
The paper concludes that internal representations contain consistent signals for detecting jailbreak prompts and, in many cases, disrupting their execution at inference time. These findings motivate internal-representation monitoring as a practical direction for understanding and mitigating jailbreak behavior.
6 Conclusion
- Internal representations contain signals that detect jailbreak prompts and, in many cases, disrupt their execution at inference time.The paper characterizes these capabilities as emerging from lightweight representation-level analysis and intervention.Source: §6 · p. 8
- The approach requires no additional post-training, auxiliary models, or complex rule-based filtering.The paper describes the analysis and intervention as lightweight.Source: §6 · p. 8
- Consistency across diverse model families suggests stable latent-space signatures and motivates internal-representation monitoring as a practical direction.The proposed direction is framed as broadly applicable for understanding and mitigating jailbreak behavior.Source: §6 · p. 8
7 GenAI Usage Disclosure
The authors disclose using AI-based writing and coding assistance to improve manuscript clarity, organization, tone, code formatting, and plot generation. They state that the scientific contributions were theirs and that they reviewed and validated AI-assisted revisions.
7 GenAI Usage Disclosure
- AI-based tools assisted with manuscript clarity, organization, academic tone, code formatting, and plot generation.The authors say these tools were used on text they had written.Source: §7 · p. 9
- The authors state that all scientific ideas, methodologies, analyses, and conclusions were their original intellectual contributions.Source: §7 · p. 9
- The authors report that they reviewed and validated all AI-assisted revisions.Source: §7 · p. 9