Source-linked AI summary
Do Internal Layers of LLMs Reveal Patterns for Jailbreak Detection?
Sri Durga Sai Sowmya Kadali, Evangelos E. Papalexakis
TL;DR
Jailbreak prompts can evade LLM safeguards, and existing defenses face diverse, evolving attacks. This paper probes internal representations from GPT-J and Mamba-2 using tensor decomposition. It reports that attention and Mixer representations outperform aggregated layer outputs, and presents internal dynamics as a promising detection direction while describing the work as preliminary.
Problem
Jailbreak prompts can elicit restricted or sensitive information, while existing defenses cannot counter the diverse range of strategies and adaptive prompt exploits.
Source: §1 · p. 1Method
The study stacks selected GPT-J and Mamba-2 layer representations across prompts, applies tensor decomposition, and evaluates the resulting embeddings with standard classifiers.
Source: §2.3 · p. 3, §3.1 · p. 4Results
MHA representations outperform GPT-J layer outputs and Mamba-2 Mixer representations outperform Block outputs, with performance reported as consistent across layers and models.
Source: §3.1 · p. 4Takeaways & Limitations
The authors conclude that internal-layer representations combined with tensor methods offer a promising approach to jailbreak detection.
Source: §4 · p. 4The authors characterize the work as an early exploration rather than a conclusive solution.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Jailbreaking large language models (LLMs) has emerged as a pressing concern with the increasing prevalence and accessibility of conversational LLMs. Adversarial users often exploit these models through carefully engineered prompts to elicit restricted or sensitive outputs, a strategy widely referred to as jailbreaking. While numerous defense mechanisms have been proposed, attackers continuously develop novel prompting techniques, and no existing model can be considered fully resistant. In this study, we investigate the jailbreak phenomenon by examining the internal representations of LLMs, with a focus on how hidden layers respond to jailbreak versus benign prompts. Specifically, we analyze the open-source LLM GPT-J and the state-space model Mamba2, presenting preliminary findings that highlight distinct layer-wise behaviors. Our results suggest promising directions for further research on leveraging internal model dynamics for robust jailbreak detection and defense.
1 Introduction
Jailbreak prompts can exploit LLM safeguards to elicit restricted or sensitive information, while existing defenses face diverse and evolving attacks. The paper proposes examining internal representations as a complementary way to distinguish jailbreak from benign prompts, but presents the work as an early exploration.
1 Introduction
- Jailbreak prompts can circumvent safeguards to elicit restricted or sensitive information, creating risks involving harmful or illicit content.The paper frames jailbreaking as a persistent concern as conversational LLMs become more prevalent and accessible.Source: §1 · p. 1
- No single defense mechanism is described as countering the diverse range of jailbreak strategies, and adaptive attackers continue to devise prompt-based exploits.The paper mentions prompt-level and model-level defenses as well as reinforcement learning with human feedback.Source: §1 · p. 1
- The study proposes analyzing hidden-layer representations in GPT-J and Mamba2, using tensor decomposition to compare jailbreak and benign prompts.Its hypothesis is that adversarial prompts exhibit distinct structural patterns in hidden layers.Source: §1 · p. 1
- The authors characterize the work as an early exploration, not a conclusive solution, and call for further research on internal model dynamics for detection.The introduction presents preliminary findings as a prompt for future inquiry.Source: §1 · p. 2
2 Proposed method
The method probes selected internal representations from GPT-J and Mamba-2, stacks them across prompts, and decomposes the resulting tensors into latent factors. Preliminary results report separation between jailbreak and benign prompts.
2.1 GPT-J: An Open-source Lightweight LLM
- GPT-J is a 6-billion-parameter, open-source autoregressive transformer with 28 layers; the study probes its internal representations for shifts between prompt types.The passage describes 16 self-attention heads followed by a feed-forward network in each layer.Source: §2.1 · p. 2
2.2 The Mamba-2 SSM architecture
- Mamba-2 is a 64-layer state-space model that uses linear-time recurrence instead of quadratic-time self-attention.Its layers contain a Mamba Mixer and a Mamba Block, described as analogous to attention and feed-forward components.Source: §2.2 · p. 2
2.3 Method
- The paper frames the tensor-based approach as a way to uncover structural differences in internal representations for downstream classification.The method is motivated by the hypothesis that jailbreak and benign prompts produce distinct hidden-layer patterns.Source: §2.3 · p. 3
- The method selects initial, middle, and final layers and stacks each representation across prompts into an M × N × K tensor.For one prompt, each representation is an M × N matrix, where M is sequence length and N is token embedding dimension; K is the number of prompts.Source: §2.3 · p. 3
- CANDECOMP/PARAFAC (CP) decomposition factorizes each three-way tensor into latent factors, and factor matrix C serves as the prompt embeddings.The decomposition is used to uncover latent dimensions of variation Sidiropoulos et al. [2017].Source: §2.3 · p. 3
- Preliminary results report clear separation of jailbreak and benign prompts using decomposed features, with latent factors visualized as type-specific groups.The method assesses these representations with 5-fold cross-validation.Source: §2.3 · p. 3, Figure 2 · p. 3
3 Experimental Evaluation
Across GPT-J and Mamba-2, the reported results are consistent across layers, with attention and Mixer representations outperforming their respective aggregated outputs. The evaluation uses selected prompt samples, standard classifiers, and 5-fold cross-validation.
3.1 Results
- The evaluation uses 240 GPT-J prompts and 400 Mamba-2 prompts from the HuggingFace Jailbreak Classification dataset, with 5-fold cross-validation and F1 scores.The passage attributes the sample sizes to computational constraints.Source: §3.1 · p. 4
- The study compares Random Forest, RBF and linear SVMs, and Logistic Regression to test a tensor-decomposition pipeline with basic classifiers.The stated aim is to demonstrate the discriminative value of latent factors rather than design a complex new classifier.Source: §3.1 · p. 4
- MHA representations outperform layer outputs in GPT-J, while Mamba-2 Mixer representations outperform Block outputs, with consistent performance across layers and models.The results are summarized in Table 1 for GPT-J and Table 2 for Mamba-2; the supplied passages do not state their numerical scores.Source: §3.1 · p. 4, Table 1 · p. 4, Table 2 · p. 4
4 Conclusion
The conclusion presents internal-layer representations combined with tensor methods as a promising approach to jailbreak detection, while emphasizing the preliminary scope of the findings.
4 Conclusion
- The authors conclude that internal-layer representations combined with tensor decomposition provide a promising approach to detecting jailbreak prompts.They state that the findings remain preliminary and identify this as an underexplored research direction.Source: §4 · p. 4
- The conclusion says tensor methods can uncover hidden patterns whose latent factors can be used for classification of jailbreak versus benign prompts.The authors describe the pathway as simple, efficient, and effective within the scope of their findings.Source: §4 · p. 4
- Further work is identified for larger datasets, additional architectures, and more sophisticated evaluation settings.The conclusion qualifies the current study as preliminary.Source: §4 · p. 4