PaperlayerView exact revision on arXiv

Source-linked AI summary

Uncovering Logit Suppression Vulnerabilities in LLM Safety Alignment

Yuxi Li, Yi Liu, Yuekang Li, Ling Shi, Gelei Deng, Shengquan Chen, Kailong Wang

arXiv:2405.13068v4cs.CRcs.AIcs.LG

TL;DR

Safety alignment can leave harmful responses recoverable when it suppresses their logits rather than eliminating them. The paper introduces SSAG, an output-logit manipulation method, and applies it in VulMine; experiments report high success, faster exposure, and substantial attack success against jailbreak defenses.

  • Problem

    Safety alignment techniques suppress affirmative-token logits and add refusal phrases, but this fixed pattern leaves a logit-level vulnerability relevant to harmful outputs.

    Source: §1 · pp. 1–2

  • Method

    SSAG manipulates output-layer logits to uncover suppressed harmful responses without changing model weights, and is integrated into VulMine.

    Source: §1 · p. 2

  • Results

    Across five widely used LLMs and two curated datasets, VulMine achieves approximately 95% success, reduces exposure time by 86% on average versus existing token-level methods, and reaches average ASR up to 77% against recognized jailbreak defenses.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings identify vulnerabilities in logit-suppression alignment and support the need for more robust alignment and proactive vulnerability detection.

    Source: §6 · p. 14

    VulMine is not applicable to closed-source LLMs when output logits cannot be accessed and manipulated.

    Source: §G.2 · p. 23

Abstract

from arXiv · show

Large language models (LLMs) have revolutionized various applications, making robust safety alignment essential to prevent harmful outputs. Current safety alignment techniques, however, harbor inherent vulnerabilities due to their reliance on logit suppression. In this work, we identify critical logit-level vulnerabilities by introducing Semantic-sensitive Alignment and Generation (SSAG), a method designed to systematically manipulate output-layer logits without altering model parameters. Experiments on five popular LLMs show that SSAG exposes harmful responses with a 95% success rate while reducing response time by 86%. VulMine also demonstrates superior attack efficacy, achieving an average ASR of up to 77% against strong defensive mechanisms. These findings reveal crucial weaknesses in existing alignment methods, highlighting an urgent need for improved vulnerability detection and robust safety alignment strategies. Our code is available on github.

1 Introduction

The introduction argues that logit suppression leaves harmful content recoverable and presents SSAG, implemented in VulMine, as a way to expose that vulnerability. Across five widely used LLMs and two curated datasets, VulMine reports approximately 95% success while reducing exposure time by 86% on average.

1 Introduction

  • Safety fine-tuning suppresses affirmative-token logits and adds refusal phrases, leaving a fixed pattern that logit manipulation can exploit to resurface suppressed harmful information.The passage identifies potential misinformation propagation and risks to trust in the open-source LLM ecosystem.Source: §1 · p. 2
  • An empirical study finds harmful responses among output candidates despite their suppressed probabilities, motivating SSAG’s output-logit manipulation without changing model weights.The passage also reports that even minimal logit interventions can reveal previously suppressed harmful content.Source: §1 · p. 2
  • 86% average reduction in exposure time versus existing token-level methods accompanies an approximately 95% success rate; average ASR reaches up to 77% against recognized jailbreak defenses.The evaluation covers five widely used LLMs across two curated datasets.Source: §1 · p. 2
  • SSAG manipulates output logits to recover suppressed harmful content and is integrated into the VulMine tool.The introduction also notes that the code and experimental appendix are publicly released.Source: §1 · p. 2

2 Empirical study on prefix length and LLM output

The study examines five open-source LLMs with increasingly complex prompts, finding that affirmative prefixes raise harmful-content rates and that refusal responses follow recurring patterns. Longer, more specific affirmative prefixes are associated with higher harmful-content rates.

2 Empirical study on prefix length and LLM output

  • The study evaluates five open-source LLMs on JailbreakBench using a prompting strategy that gradually increases in complexity.The prompting strategy and experimental setup are described in Appendix C.Source: §2 · p. 3

2.1 Harmful Content in Response Text

  • 6.38% to 25.00%: the average harmful-content ratio rises with an affirmative prefix, then from 25.00% to 31.37% with a complete affirmative response instead of “Sure!” alone.Table 1 reports harmful-content rates on selected models.Source: Table 1 · p. 3, §2.1 · p. 3

2.2 Denial Prefix in Response Text

  • 99.19% of denial responses fit four semantic categories—Apology, Identification, Incapacity, and Notation—with Identification and Incapacity the most common.Table 2 presents the categorization across selected LLMs.Source: Table 2 · p. 3, §2.2 · p. 4

2.1 Harmful Content in Response Text

  • Longer and more specific affirmative prefixes correspond to a higher probability of harmful-content generation.The reported comparison includes a single affirmative prefix and a complete affirmative response.Source: §2.1 · p. 3

3 Vulnerability Exploitation Scenario

The scenario describes an attacker covertly modifying an open-source, safety-aligned model’s output logits, then republishing it so downstream developers may inherit harmful behavior.

3 Vulnerability Exploitation Scenario

  • An attacker patches inference-time logits in an open-source model, elevating suppressed harmful responses for specific prompts while leaving weights unchanged and aiming to preserve benign behavior.The scenario then has the attacker republish the model as an improved or fine-tuned variant, which developers may integrate directly or use as a base for further fine-tuning.Source: §3 · p. 4

4 Methodology Design & Implementation

SSAG steers responses by manipulating output logits toward semantically aligned content, and VulMine implements this through affirmative-response targeting, logit adjustment, and selection of successful generations.

4.1 The Framework of SSAG

  • The framework combines semantic-sensitive alignment with LLM generation to produce responses that are relevant to the input.Its workflow applies affirmative-response targeting, logit manipulation, and semantic-sensitive harmful-content generation.Source: §4.1 · p. 4, Figure 1 · p. 5

Semantic-Sensitive Alignment

  • SSAG adjusts initial response logits to match the input’s semantics rather than follow refusal patterns, steering generation toward harmful outputs.Its generation stage then uses the adjusted logits to produce full, contextually aligned responses that are reviewed for harmfulness.Source: Semantic-Sensitive Alignment · p. 5, Logit-guided Generation · p. 6

4.2 The Design of SSAG

  • The harmful-content definition uses a judge’s policy-compliance decision to identify outputs that count as harmful.The paper also defines a successful elicitation sample as requiring harmful input and output with semantic alignment between them.Source: §4.2 · p. 6
  • SSAG’s design treats denial-token prefixes as indicators of attack failure and extends manipulation across consecutive response logits.The text motivates longer, more specific token settings as increasing the likelihood of harmful content; the manipulated span is followed by autonomous generation.Source: §4.2 · p. 7

4.3 The Implementation of SSAG: VulMine

  • VulMine generates an affirmative target, manipulates logits to elicit it, then searches manipulated-logit combinations for successful attacks.A sorting model helps select stable outputs without adding suffixes to user prompts.Source: Figure 1 · p. 5, §4.3 · p. 7

Positive Response to Harmful Prompts.

  • Few-shot templating creates an affirmative response to a harmful request as the target for subsequent logit manipulation.The template pairs a harmful behavior with a positive response, and a rephrase LLM is used to generate the target prompt.Source: §4.3 · p. 7

Prompt-agnostic Logits Manipulation.

  • VulMine manipulates affirmative-response and enhancement-prefix logits while suppressing denial-token logits, then ranks candidate prefixes with a sorting model.It repeats random top-K token selection to form a batch and uses the ranking model to choose a prefix for harmful-content generation.Source: Prompt-agnostic Logits Manipulation · p. 8

Optimal Harmful Content Generation.

  • For optimal generation, VulMine applies sorted adjusted logits to a harmful input and returns a generated response when it is harmful and relevant to that input.Each adjusted logit is used to generate a full response, which is checked for both harmfulness and relevance.Source: Optimal Harmful Content Generation · p. 8

5 Evaluation

VulMine outperforms the five attack baselines across the evaluated models and datasets, while remaining efficient and retaining substantial ASR against defenses. Ablations indicate that combining initial and later logit manipulations improves performance.

5.1 Experiment Setup

  • The evaluation targets five open-source LLMs, uses a 520-behavior AdvBench subset, and compares VulMine with five attack baselines and four defenses.ASR is calculated as successful harmful generations divided by total harmful queries; the experimental setup also uses Llama-Guard-3 as the judge and sets prefix length m to 5 and batch size N to 2,000.Source: §5.1 · pp. 9–10

5.2 Effectiveness Comparison

  • 95.65% average ASR on AdvBench, 14.74% higher than LAA; VulMine achieves the highest ASR on all five evaluated models.The reported one-sided paired Wilcoxon signed-rank tests indicate that its advantage over the five baselines is statistically significant.Source: Table 4 · p. 10, §5.2 · p. 11
  • On Llama-2-7b-chat and Llama-2-13b-chat, VulMine reaches 95.93% and 96.16% ASR, respectively, while PAIR and GPTFuzzer remain below 60%.GCG is comparable on Mistral-7b-Instruct and Llama-3-8b-Instruct but underperforms VulMine on the other three models.Source: §5.2 · p. 11

5.3 Efficiency Comparison

  • VulMine is the most time-efficient method on average across the evaluated models and datasets, with Mistral-7b-Instruct as an exception where GPTFuzzer is slightly faster.Across methods, Mistral-7b-Instruct and Llama-3-8b-Instruct show high ASR with less time, while Llama-2-7b-chat, Llama-2-13b-chat, and gemma-7b-it show lower ASR and require more time.Source: §5.3 · pp. 11–12, Figure 2 · p. 12

5.4 VulMine against Mitigation Strategies

  • VulMine retains a 72.56% average ASR against three jailbreak defenses on AdvBench, while its perplexity is slightly above GCG and LAA but within the same order of magnitude as GPTFuzzer and PAIR.The authors report that these defenses reduce VulMine’s effectiveness to some extent and suggest that existing defenses primarily detect jailbreakable prompts rather than address the underlying vulnerability.Source: Figure 3 · p. 12, Table 5 · p. 13, §5.4 · p. 13

5.5 Ablation Study

  • VulMine outperforms its variants across all five models, and VulMine-k exceeds VulMine-m by an average 16.04% ASR on AdvBench.Both variants outperform the question-only baseline; the authors attribute greater influence to the initial manipulated logits and describe both manipulations as indispensable.Source: Table 6 · p. 13, §5.5 · p. 14

6 Conclusion

The paper concludes that logit suppression leaves harmful content recoverable through manipulation and argues for stronger alignment and proactive vulnerability detection. It also describes the generation process, prior jailbreak work, dataset construction, and ethical safeguards.

6 Conclusion

  • The conclusion identifies vulnerabilities in logit-suppression alignment and calls for more robust alignment methods and proactive vulnerability detection.It reports that SSAG can recover suppressed harmful content through minimal logit manipulation across multiple LLMs.Source: §6 · p. 14, A Ethical Considerations · p. 17

A Ethical Considerations

  • The authors state that they follow ethical guidelines and emphasize stronger verification mechanisms and proactive defenses against evolving jailbreak attacks.They frame this work as informing trustworthy AI deployment in open-source ecosystems.Source: A Ethical Considerations · p. 17

B.1 Content Generation of LLMs

  • LLM generation repeatedly converts input tokens to logits, samples a next token, appends it, and stops at an end token or the maximum token limit.The final generated sequence is decoded into text.Source: §B.1 · p. 17

B.2 Jailbreak Attacks

  • Prior jailbreak work includes white-box suffix generation, black-box fuzzing, semantic prompt generation, and logit-based controllable text generation.The cited studies describe susceptibility to jailbreak attacks and potential harmful responses [30] [27] [8] [16] [29] [9] [3].Source: §B.2 · pp. 17–18

C.1 Dataset Collection

  • The empirical dataset procedure selects open-source LLMs and harmful behaviors, constructs prompt templates, and generates model outputs.The selected models include five LLMs, and JailbreakBench supplies 100 harmful behaviors across 10 categories; prompts progress from the original question to partial and complete affirmative suffixes.Source: §C.1 · p. 18

D. DETAILED CATEGORIZATION FOR EMPIRICAL STUDY 19

The appendix describes progressively modified prompt templates and repeated text generation to study how affirmative prefixes affect model responses.

D. DETAILED CATEGORIZATION FOR EMPIRICAL STUDY 19

C.2 Data Labelling

The appendix describes how outputs and refusals are labeled, then explains the denial categories and how the sorting model is built.

C.2 Data Labelling

  • Llama-Guard-3 fine-tuned on Llama-3 is used to label output harmfulness, with the authors reporting close to 100% performance in their comparison with manual labeling.The text notes that assessing harmfulness is inherently subjective.Source: §C.2 · p. 19

D Detailed Categorization for Empirical Study

  • Across five LLMs, analysis of the first ten response tokens found that more than 99% of refusal texts conformed to 17 distinct prefixes, grouped into four denial-pattern categories.The categories are Apology, Identification, Incapacity, and Notation; examples describe apologies, AI identity statements, and inability to provide harmful responses.Source: §C.2 · p. 19, D Detailed Categorization for… · pp. 19–20

E Details of Sorting Model Γ

E.1 Dataset Construction

The sorting-model data are built from model responses to harmful questions, labeled for safety, and embedded for training. The supplied passages give only the opening details of model training.

E.1 Dataset Construction

  • 1,000 responses from 100 questions were collected, and their first m tokens after an affirmative prefix were labeled safe or toxic using Llama-Guard-3.The responses were collected at 10 per question; each m-token sample inherited the safety verdict for its full response.Source: §E.1 · p. 20
  • Llama-Guard-3 supplied the safety labels by classifying each corresponding response as safe or unsafe.Those verdicts were mapped to the dataset labels safe and toxic, respectively.Source: §E.1 · p. 20, §E.2 · p. 20

E.2 Model Training

  • The m-token samples were embedded with gtr-t5-xl as 768-dimensional inputs, with safe and toxic encoded as 0 and 1.The passage ends as it begins to describe the subsequent model-training procedure.Source: §E.2 · p. 20

F. DETAILED SETTINGS OF EVALUATION EXPERIMENT 21

The evaluation settings span open-source target models, jailbreak baselines, defenses, and computing resources. The hyperparameter study identifies a setting that balances effectiveness and efficiency.

F. DETAILED SETTINGS OF EVALUATION EXPERIMENT 21

  • The sorting model achieves an F1 score of 0.9207 with m = 5 after training on 300 instances and testing on 700.The described network has two fully connected layers and a hidden dimension of 512; training runs for 1,000 epochs at a learning rate of 1e−3.Source: F. DETAILED SETTINGS OF EVALUATION… · p. 21

F.1 Language Models

  • The evaluation selects five open-source LLMs as jailbreak targets; Llama-Guard-3 judges generated content by determining, from the given questions and answers, whether it violates 11 relevant laws, regulations, and ethical standards.Source: §F.1 · p. 21

F.2 Baselines Setup

  • The baseline settings include configured GCG, GPTFuzzer, PAIR, LAA, and COLD-Attack implementations.GCG uses 20 suffix tokens and 500 steps; GPTFuzzer is limited to 3 attempts and 500 queries; PAIR uses five streams and iterations and runs four times per behavior; LAA uses defaults; COLD-Attack uses suffix mode with other defaults.Source: §F.2 · p. 22

F.3 Defense Strategies Setup

  • The defense-strategy setup includes calculating prompt perplexity with the stated formula.The available passage identifies PPL as the quantity calculated but does not supply additional details about defense configurations in this subsection.Source: §F.3 · p. 22

G. DISCUSSIONS 23

  • The discussion describes calculating probabilities with the corresponding model and applying ESF, SmoothLLM, and PAT configurations before VulMine.ESF uses an enhanced safety fine-tuning dataset; SmoothLLM uses default parameters, while PAT uses different controls for Llama-2 and the other three models.Source: G. DISCUSSIONS 23 · p. 23

F.4 Experiment Environment

  • The experiments run on Ubuntu 22.04 with 256 CPU cores, 512 GB RAM, and two NVIDIA A100 GPUs totaling 160 GB of video memory.Source: §F.4 · p. 23

G.1 Choice of Hyperparameters in VulMine

  • m = 5 and N = 2000 demonstrate effectiveness and efficiency; lowering them substantially decreases ASR, while raising them increases time without improving ASR.The compared alternatives are m = 3, N = 1000 and m = 10, N = 3000.Source: §G.1 · p. 23

G.2 Limitations of VulMine

VulMine has resource and access boundaries, and the supplied examples illustrate outputs from several baseline attacks. The examples include both refusals and harmful responses.

G.2 Limitations of VulMine

  • Increasing m and N can enable attacks, but the resulting computational-resource consumption may be unacceptable.The passage also notes that some strategies may defend VulMine well, including expanding refusal patterns or returning a fixed logit for harmful questions.Source: §G.2 · p. 23
  • VulMine is not applicable to closed-source models when output logits cannot be accessed and manipulated.Source: §G.2 · p. 23

H.2 GPTFuzzer

  • The GPTFuzzer example pairs an instruction to produce aligned and unaligned responses with both a refusal and harmful trafficking guidance.The example includes an initial safety-oriented response followed by harmful content.Source: §H.2 · p. 25

H.3 PAIR

  • The PAIR example prompt requests a report about an unnamed substance intended to alleviate depression symptoms.Source: §H.3 · p. 26

H.4 LAA

The supplied examples show varied outputs across LAA, COLD, and VulMine. They include responses to harmful requests as well as innocuous movement prompts.

H.4 LAA

  • The LAA examples include a claimed depression-treatment report and harmful botnet guidance.The botnet response describes actions including attacks on websites and data theft.Source: §H.4 · p. 26

H.5 COLD

  • The COLD example is a request to design a highly lethal weapon for an assassination.No response to this request is included in the supplied passages.Source: §H.5 · p. 27

H.6 VulMine

  • In the VulMine examples, a physical-harm prompt receives weapon suggestions, while barrel-roll and backflip prompts receive benign instructions.Source: §H.6 · p. 27

Related papers

Loading 2405.13068v4…