PaperlayerView exact revision on arXiv

Source-linked AI summary

JBShield: Defending Large Language Models from Jailbreak Attacks through Activated Concept Analysis and Manipulation

Shenyi Zhang, Yuchen Zhai, Keyan Guo, Hongxin Hu, Shengnan Guo, Zheng Fang, Lingchen Zhao, Chao Shen, Cong Wang, Qian Wang

arXiv:2502.07557v1cs.CR

TL;DR

Safety alignment does not prevent jailbreaks, and existing defenses lack a systematic account of how jailbreak prompts manipulate LLM behavior. JBShield uses hidden-representation concepts for detection and mitigation, achieving 0.94 average F1-Score and reducing average attack success rate from 61% to 2%. Its effectiveness remains uncertain for future architectures and depends on calibration data quality and diversity.

  • Problem

    Existing defenses lack a systematic understanding of how jailbreak prompts alter LLM behavior despite safety alignment strategies, limiting robust protection against these attacks.

  • Method

    JBShield extracts toxic and jailbreak concept subspaces from hidden representations, detects their activation, and mitigates attacks by strengthening toxic concepts while weakening jailbreak concepts.

  • Results

    0.94 average F1-Score was achieved for detection, and average attack success rate decreased from 61% to 2% across five LLMs and nine jailbreak attacks.

  • Takeaways & Limitations

    JBShield provides efficient, interpretable detection and mitigation, with calibration using only 30 jailbreak prompts and single-forward-pass detection.

  • Takeaways & Limitations

    Effectiveness on future, potentially novel LLM architectures remains uncertain, and performance depends on calibration-data quality and diversity.

Abstract

from arXiv · show

Despite the implementation of safety alignment strategies, large language models (LLMs) remain vulnerable to jailbreak attacks, which undermine these safety guardrails and pose significant security threats. Some defenses have been proposed to detect or mitigate jailbreaks, but they are unable to withstand the test of time due to an insufficient understanding of jailbreak mechanisms. In this work, we investigate the mechanisms behind jailbreaks based on the Linear Representation Hypothesis (LRH), which states that neural networks encode high-level concepts as subspaces in their hidden representations. We define the toxic semantics in harmful and jailbreak prompts as toxic concepts and describe the semantics in jailbreak prompts that manipulate LLMs to comply with unsafe requests as jailbreak concepts. Through concept extraction and analysis, we reveal that LLMs can recognize the toxic concepts in both harmful and jailbreak prompts. However, unlike harmful prompts, jailbreak prompts activate the jailbreak concepts and alter the LLM output from rejection to compliance. Building on our analysis, we propose a comprehensive jailbreak defense framework, JBShield, consisting of two key components: jailbreak detection JBShield-D and mitigation JBShield-M. JBShield-D identifies jailbreak prompts by determining whether the input activates both toxic and jailbreak concepts. When a jailbreak prompt is detected, JBShield-M adjusts the hidden representations of the target LLM by enhancing the toxic concept and weakening the jailbreak concept, ensuring LLMs produce safe content. Extensive experiments demonstrate the superior performance of JBShield, achieving an average detection accuracy of 0.95 and reducing the average attack success rate of various jailbreak attacks to 2% from 61% across distinct LLMs.

1 Introduction

Jailbreak attacks bypass safety guardrails, while existing defenses lack a systematic understanding of how jailbreak prompts alter model behavior. JBShield analyzes toxic and jailbreak concepts in hidden representations, then detects and mitigates attacks with strong results across LLMs and attack types.

  • Motivation: Jailbreak attacks modify harmful inputs to bypass safety guardrails, causing LLMs to produce unsafe outputs that would normally be blocked.Existing alignment strategies remain vulnerable to these attacks, creating significant security threats.
  • Motivation: Existing defenses often rely on surface-level patterns and lack a systematic understanding of how jailbreak prompts alter LLM behavior.This limits their ability to provide robust protection over time.
  • Mechanism: LLMs recognize toxic concepts in both harmful and jailbreak inputs, but jailbreak prompts additionally introduce a jailbreak concept that increases compliance with user requests.The analysis defines toxic concepts from representation differences with benign prompts and jailbreak concepts from differences between jailbreak and harmful prompts.
  • Approach: JBShield detects jailbreaks by analyzing toxic and jailbreak concepts, then mitigates them by strengthening the toxic concept and weakening the jailbreak concept.Its framework combines JBShield-D for detection with JBShield-M for mitigation in representation space.
  • Results: 0.94 average F1-Score was achieved for detection, while average attack success rate fell from 61% to 2% across five LLMs and nine jailbreak attacks.The reported performance used only 30 jailbreak prompts for calibration.

2 Background and Related Works

The paper categorizes jailbreak attacks by how they are designed or generated and reviews defenses that detect malicious inputs or mitigate their effects. The attack landscape includes manual, optimization-based, template-based, and linguistics-based strategies, alongside defenses based on gradients, perturbations, perplexity, and specialized models.

  • Jailbreak Attacks: Jailbreak attacks are categorized into manual-designed, optimization-based, template-based, linguistics-based, and encoding-based types.Table 1 summarizes whether each attack uses listed resources or capabilities.
  • Jailbreak Attacks: Manual-designed jailbreaks are crafted directly by humans, including in-the-wild prompts observed in real deployments and shared on social media.These attacks reflect human understanding of LLM mechanics and vulnerabilities.
  • Jailbreak Attacks: Optimization-based jailbreaks use automated algorithms and internal gradients to construct malicious soft prompts or adversarial suffixes.GCG modifies prompts to encourage responses beginning with “Sure,” while SAA combines templates with random search.
  • Jailbreak Attacks: Template-based attacks embed harmful requests in optimized templates, while attacker-LLM methods target another LLM with minimal queries.Examples include MasterKey, LLM-Fuzzer, PAIR, and TAP.
  • Jailbreak Attacks: Linguistics-based jailbreaks conceal malicious intentions within seemingly benign inputs to bypass defensive guardrails.DrAttack decomposes and reconstructs malicious prompts within reassembled context.
  • Jailbreak Defenses: Jailbreak defenses are divided into detection and mitigation, with examples using gradient norms, output variability, perplexity, and gradient patterns.The reviewed methods include Gradient Cuff, SmoothLLM, PPL, and GradSafe.

3 Activated Concept Analysis

The paper models toxic and jailbreak semantics as subspaces in LLM hidden representations, extracting them through counterfactual prompt differences and linear decomposition. Analysis shows harmful semantics remain recognizable in jailbreak prompts, while additional jailbreak concepts shift behavior toward compliance.

  • Concept Representation: Concept extraction represents toxic and jailbreak semantics as subspaces within transformer-layer hidden representations.Two toxic concepts compare harmful or jailbreak prompts with benign prompts; the jailbreak concept compares jailbreak with harmful prompts.
  • Concept Extraction: Randomly paired prompts from different categories form counterfactual pairs that expose semantic differences between prompt classes.The authors note that random pairing is used because real-world datasets make minimally varying counterfactual pairs difficult to construct.
  • Linear Decomposition: The difference matrix D_toxic captures paired harmful–benign representation differences, and rank-1 truncated SVD extracts their principal direction.Each matrix row represents a paired difference vector, while the first column of V provides the principal extracted component.
  • Interpretation: Concept-aligned token scores identify interpretable tokens, including compliance-related words such as “sure” and “yes” for jailbreak concepts.The extraction results are illustrated at layer 24 of Mistral-7B, with interpretable words retained in Table 2.
  • 3.3 RQ1: Recognition of Harmful Semantics: Aligned LLMs recognize harmful semantics in both harmful and jailbreak prompts, with harmful prompts activating toxic concepts associated with warnings and refusal.Tokens such as “caution” and “warning” indicate threat recognition and self-warning behavior in harmful prompts.
  • 3.4 RQ2: Influence of Jailbreaks Prompts: Jailbreak prompts additionally introduce jailbreak concepts that manipulate compliance, while JBShield counters them by strengthening toxic concepts and weakening jailbreak concepts.The framework contains detection and mitigation components, and the analysis links jailbreak concepts to altered model behavior.

4 JBSHIELD

JBSHIELD defends against jailbreaks by detecting toxic and jailbreak concepts in selected representation subspaces, then modifying hidden states to strengthen toxic semantics and weaken jailbreak semantics.

  • Framework overview: JBSHIELD combines jailbreak detection with hidden-representation manipulation of toxic and jailbreak concepts.Its two components are JBSHIELD-D and JBSHIELD-M.
  • Efficiency and scope: JBSHIELD operates during the forward pass, uses minimal calibration data, and requires one detection pass plus a few linear mitigation operations.This design is intended to minimize impact on target-LLM usability and reduce jailbreak manipulation even when external detection is bypassed.
  • Jailbreak detection: JBSHIELD-D selects critical layers and calibrates anchor representations and subspaces for toxic and jailbreak concepts.Layer selection compares representational differences between harmful, benign, jailbreak, and harmful prompts.
  • Jailbreak detection: The detector extracts concept subspaces from each input and compares them with anchor subspaces using cosine similarity and calibrated thresholds.Toxic detection uses harmful and benign calibration samples, while jailbreak detection uses harmful and jailbreak samples.
  • Jailbreak detection: An input is flagged as a jailbreak prompt only when it activates both toxic and jailbreak concepts above their respective thresholds.The thresholds are calibrated from labeled prompt categories, with the toxic threshold selected using Youden’s J statistic.
  • Jailbreak mitigation: After detection, JBSHIELD-M enhances the toxic concept by adding its anchor vector and weakens the jailbreak concept by subtracting a scaled jailbreak vector.The adjustment strengths are calibrated from projection differences in harmful, benign, and jailbreak prompt representations.

5 Experiments

Experiments evaluate JBShield across diverse datasets, calibration sizes, open-source LLMs, and jailbreak attacks. JBShield-D provides strong detection and transferability, while JBShield-M substantially reduces attack success with limited impact on ordinary tasks.

  • Jailbreak Detection: 0.95 average detection accuracy and 0.94 average F1-Score are achieved across nine jailbreaks and five LLMs, outperforming existing detection methods.The baselines show attack- or model-specific weaknesses, whereas JBShield-D maintains stronger overall detection performance.
  • Jailbreak Detection: 0.84 accuracy and 0.86 F1 score are exceeded in most transferability cases on Mistral-7B when detecting unseen jailbreak attacks.Accuracy and F1 exceed 0.90 for AutoDAN, Zulu, and Base64 across calibration choices, although Puzzler has F1 below 0.2.
  • Jailbreak Mitigation: 61% to 2%: JBShield-M lowers average attack success rates across five LLMs and reduces most individual attack success rates to zero.AutoDAN, Puzzler, and Base64 each reach an ASR of 0.00 under JBShield-M.
  • Jailbreak Mitigation: Less than 2% impact on MMLU performance is observed, while removing either toxic-concept enhancement or jailbreak-concept weakening raises average ASR to 12% or 13%.JBShield-M activates only after jailbreak detection, limiting its effect on normal inputs.
  • Jailbreak Mitigation: 0.4%, 4.0%, and 0.4% average ASRs are maintained against AutoDAN-based, GCG-based, and LLMFuzzer-based adaptive attacks, respectively.These attacks explicitly incorporate weakening the toxic concept and enhancing the jailbreak concept into their objectives.

6 Discussions

JBShield integrates detection and mitigation without extra inference tokens, model fine-tuning, or extensive additional training data. Its effectiveness nevertheless depends on internal model access and calibration-data quality, creating boundaries for deployment on novel architectures or attacks.

  • Discussion: JBShield combines jailbreak detection and mitigation, unlike defenses that typically address only one of these functions.The framework is designed for deployment on existing LLMs without extra tokens, model fine-tuning, or extensive additional training data.
  • Discussion: 30 jailbreak prompts are sufficient for calibration to defend against each jailbreak attack, supporting scalability and adaptation to emerging attacks.The paper characterizes this calibration cost as minimal compared with previous methods.
  • Limitations: Internal architecture and parameter access, plus the ability to probe and modify hidden representations during the forward pass, are required.Effectiveness on future, potentially novel LLM architectures remains uncertain.
  • Limitations: Calibration-data quality and diversity constrain generalizability to novel or substantially different jailbreak attempts.The paper reports strong transferability across unseen jailbreaks, but identifies less diverse calibration data as a boundary.

7 Conclusion and Future Works

JBSHIELD defends against jailbreaks by detecting toxic and jailbreak concept activation, then strengthening the toxic concept while weakening the jailbreak concept. The authors identify fixed scaling and evolving prompt semantics as directions for future improvement.

  • Conclusion: JBSHIELD combines detection and mitigation to defend against state-of-the-art jailbreaks across multiple LLMs.JBSHIELD-D detects toxic and jailbreak concept activation, while JBSHIELD-M adjusts their representations.
  • Conclusion: JBSHIELD-D detects jailbreak prompts by analyzing toxic and jailbreak concept activation.
  • Conclusion: JBSHIELD-M enhances the toxic concept and weakens the jailbreak concept to safeguard LLMs from jailbreak inputs.
  • Future Works: Future work should investigate jailbreak mechanisms under new architectures and develop adaptive scaling-factor control as prompt semantics change during generation.The current scaling factor is fixed using calibration data, while concept activation varies as new tokens are generated.

Ethics Considerations

JBSHIELD is presented as a safeguard against exploiting LLMs to generate inappropriate or unsafe content. The framework aims to improve jailbreak detection and mitigation so outputs align with ethical standards and societal norms.

  • Ethics Considerations: JBSHIELD safeguards against exploiting LLMs to generate inappropriate or unsafe content.
  • Ethics Considerations: The framework improves detection and mitigation of jailbreak attacks.
  • Ethics Considerations: The stated aim is for LLM outputs to align with ethical standards and societal norms.

Open Science

The authors state that they will share the artifacts needed for community review and reproducibility. These artifacts include datasets, concept-extraction code, and implementations of both JBSHIELD components.

  • Open Science: The authors will share necessary artifacts with the research community for review by the artifact evaluation committee.
  • Open Science: The shared artifacts will support reproducibility of the work.
  • Open Science: The planned release includes test datasets, code for extracting concept-related interpretable tokens, and JBSHIELD-D and JBSHIELD-M implementations.The implementations are intended for testing across five target LLMs.

A Additional Explanation and Results of Concept Extraction

Concept extraction uses counterfactual prompt pairs and hidden-representation analysis to derive concept subspaces and interpretable tokens. Across several Llama and Vicuna models, the results support shared toxic concepts and a jailbreak concept associated with compliance.

  • Concept Extraction: Singular value decomposition extracts a toxic subspace, whose leading direction is projected onto the vocabulary to identify top-k interpreting tokens.
  • Concept Extraction: The concept-extraction algorithm forms counterfactual prompt pairs and processes their hidden-representation differences.For toxic concepts, the pairs are harmful-benign or jailbreak-benign; for jailbreak concepts, they are jailbreak-harmful.
  • Concept Extraction: The extraction process is analogous across toxic and jailbreak concepts, requiring replacement of the prompt types used in the counterfactual pairs.
  • Results: Across two Llama and two Vicuna family models, toxic concepts show model-specific token associations but broadly similar semantic findings.Examples include “illegal” for Llama3-8B and “Sorry” or “cannot” for Llama2-7B.
  • Results: The results indicate that LLMs recognize similar toxic concepts in jailbreak and harmful prompts, while jailbreak-concept activation changes output from rejection to compliance.

B.1 More Details of Our Dataset

The study constructs jailbreak-prompt data across multiple attack types and evaluates JBShield across diverse open-source LLMs using automated safety-refusal assessment and resource comparisons.

  • Dataset construction: 850 benign, 850 harmful, and 32,600 jailbreak prompts comprise the detection dataset.Benign prompts are sampled from Alpaca, harmful prompts combine AdvBench data, and jailbreak prompts are generated or selected across attack types.
  • Dataset construction: Each jailbreak method generally contributes 850 samples, except DrAttack with 520 prompts and Puzzler with 50 prompts.The reduced counts for DrAttack and Puzzler follow their default open-source settings because they use GPT models and incur higher generation costs.
  • Target models: The evaluation covers LLMs differing in base models, alignment techniques, model sizes, and embedding dimensions.The experiments use open-source Hugging Face models controlled through FastChat, with system prompts documented separately.
  • Evaluation and baselines: SORRY-Bench and LlamaG are used to evaluate jailbreak success, with SORRY-Bench relying on an automated Mistral-7B safety-refusal evaluator.The authors also compare judging methods and defense baselines using specified open-source implementations and prompting procedures.
  • Resource consumption: JBShield-M has the lowest overall resource consumption among compared mitigation methods on Mistral-7B.It uses a small number of calibration prompts, adds no inference-time tokens, and applies simple linear operations with minimal inference-time impact.
  • Concept analysis: Conceptual-subspace comparison identifies semantic differences overlooked by direct embedding similarity comparison.The broader concept analysis defines toxic and jailbreak concepts as subspaces in hidden representations, with the jailbreak concept linked to increased compliance.

C.4 Evaluation on Normal Inputs with Seemingly Toxic Words

JBShield is evaluated on normal prompts containing seemingly toxic words to test false positives, with results indicating robust distinction between benign and genuinely harmful or jailbreak inputs.

  • False-positive evaluation: The evaluation measures whether JBShield misclassifies normal inputs with superficially toxic language as jailbreak prompts.This test complements evaluation on harmful inputs and targets the distinction between toxic wording and harmful intent.
  • False-positive evaluation: 2% average false positive rate is reported across five LLMs on OR-Bench-Hard-1K.The dataset contains prompts designed to appear toxic without harmful intent.
Loading 2502.07557v1…