Source-linked AI summary
NeuronFuzz: Safety Neuron Guided Fuzzing for LLM Safety Evaluation
Zhiyuan Xu, Muhammad Firhard Roslan, Joseph Gardiner, Sana Belguith, Lichao Wu
TL;DR
Automated LLM safety testing is limited by expensive response generation and sparse feedback on strongly aligned models. NeuronFuzz uses internal safety neurons as continuous, prefill-time feedback for white-box fuzzing and gradient-guided template mutation. It achieves strong jailbreak discovery across source models and transfers optimized templates across diverse target settings.
Problem
Existing automated safety testing relies on response-level feedback, which is expensive and provides little guidance when strongly aligned models reject many candidates identically.
Method
NeuronFuzz constructs a differentiable SafetyOracle from template-invariant harmful–benign inputs and uses its prefill-time score and gradients to guide template mutation.
Results
NeuronFuzz achieves 76-100% jailbreak discovery across five white-box source models and improves over evaluated methods by up to 48 percentage points.
Takeaways & Limitations
Internal safety representations provide informative feedback for jailbreak discovery and support transfer across payloads, models, and modalities.
Takeaways & Limitations
The evaluator requires white-box access to model parameters, MLP activations, and input gradients, without modifying model parameters or inference settings.
Abstract
from arXiv · showhide
Safety evaluation is critical for assessing whether aligned Large Language Models (LLMs) remain robust against jailbreak attacks. Existing automated testing methods, however, largely rely on response-level feedback: each candidate prompt typically requires generating a target-model response to evaluate its attack effectiveness. This process is expensive and, more importantly, provides only sparse guidance on strongly aligned models, where most candidates are rejected with the same failure outcome. This paper presents NeuronFuzz, a white-box fuzzing framework that exploits internal safety neurons as continuous execution feedback for LLM safety evaluation. A SafetyOracle converts safety-neuron activations into a continuous safety alarm score that serves as feedback for fuzzing and can be obtained during prefill, eliminating response generation from the fuzzing loop. To construct the SafetyOracle, NeuronFuzz uses template-invariant harmful and benign inputs and stability-aware selection to identify a compact set of safety neurons whose activations capture harmful-intent recognition. Moreover, since the safety alarm score is differentiable, NeuronFuzz uses its gradients to identify safety-sensitive template positions and a masked language model to generate fluent, context-compatible mutations while preserving original harmful payload and avoiding additional optimization variables. We evaluate NeuronFuzz across 21 text and multimodal models. Across five white-box source models, it achieves a 76-100% jailbreak discovery rate, outperforming baselines by up to 48 percentage points. Its optimized templates further transfer zero-shot to open-weight and six proprietary target models, achieving average ASR and top-5 ensemble ASR (EASR) of 69.6%/92.6% and 44.1%/60.0%, respectively.
1 Introduction
LLM safety evaluation must identify jailbreaks that bypass alignment, but automated methods suffer from sparse response-level feedback and expensive response generation. NeuronFuzz addresses these limitations with continuous internal safety-neuron feedback and gradient-guided template mutation.
- Motivation: Aligned LLMs can remain vulnerable to jailbreaks that bypass safety mechanisms and elicit harmful content.Systematic identification of these failures is important for evaluating robustness before deployment.
- Existing approaches: Manual red teaming exposes weaknesses but requires substantial human effort and domain expertise, limiting scale and input diversity.
- Limitations: Response-level evaluation provides sparse guidance because strongly aligned models often assign the same failure outcome to candidates with different underlying safety effects.This limits seed selection and mutation even when additional query budget is available.
- Limitations: Response-level methods are expensive because every candidate requires autoregressive decoding and may also require external classifiers or LLM judges.Repeated evaluation of many candidates makes response generation a major testing cost.
- NeuronFuzz: NeuronFuzz uses a lightweight SafetyOracle and differentiable safety alarm score to rank candidates and localize safety-sensitive template positions.The SafetyOracle is built from template-invariant activation extraction and stability-aware safety-neuron selection.
- NeuronFuzz: NeuronFuzz combines prefill-only SafetyOracle feedback with gradient-guided masked-token mutation while preserving harmful payloads and natural-language structure.
- Evaluation: Evaluations cover 21 text and multimodal models, with improved jailbreak discovery and transfer across model families, proprietary APIs, datasets, and visual safety tasks.
2 Preliminaries
LLM safety fuzzing adapts feedback-guided software testing to generate and evaluate jailbreak templates. Existing methods depend on generated responses, creating decoding overhead and weak discrimination among failed prompts, especially for strongly aligned models.
- Internal safety signals: Internal neuron activations can localize safety-related information to compact neuron subsets that distinguish harmful from benign inputs.Manipulating these neurons or related components can alter safety and refusal behavior.
- Fuzzing: Fuzzing repeatedly generates test inputs to expose unexpected program behaviors and security vulnerabilities.
- Fuzzing: Coverage-guided fuzzing uses seeds, scheduling, mutation, runtime instrumentation, and execution feedback to progressively explore a program’s execution space.
- LLM fuzzing: LLM fuzzing treats jailbreak templates as seeds, language-based operations as mutators, and policy-violating generated responses as successful tests.Responses may be evaluated with classifiers, heuristic rules, or LLM judges.
- Limitations: Each prompt in response-level LLM safety testing must generate a response before evaluation, introducing repeated decoding and evaluation overhead.
- Limitations: Response-level feedback poorly distinguishes unsuccessful prompts, particularly on strongly aligned models where many mutations receive the same failure label.This leaves fuzzers with little information for selecting promising prompts.
3 NeuronFuzz
NeuronFuzz uses internal safety signals as dense, prefill-only feedback for white-box jailbreak fuzzing. Its SafetyOracle is built from template-invariant activations and stable safety neurons, while gradients guide template mutations that preserve harmful payloads.
- NeuronFuzz fuzzing loop: Prefill-only oracle feedback ranks and mutates candidates without requiring autoregressive response generation.This provides denser guidance than response-level evaluation while reducing the cost of evaluating each candidate.
- SafetyOracle: Template-invariant harmful–benign pairs isolate payload-related activation differences from jailbreak-template effects.The same template is paired with harmful and benign payloads, then MLP activations from prompt positions are pooled into fixed-dimensional features.
- SafetyOracle: Bootstrap stability selection retains safety neurons that are consistently selected with a positive harmful-input association.Selection rate measures retention frequency, while sign consistency measures whether coefficient direction remains stable across bootstrap runs.
- SafetyOracle: The SafetyOracle maps standardized selected-neuron activations to a continuous safety alarm score indicating harmful-intent recognition.Elastic Net logistic regression controls redundancy while learning the final scoring function.
- NeuronFuzz fuzzing loop: Gradients identify safety-sensitive template positions, and a masked language model proposes context-compatible replacements while the harmful payload remains unchanged.Mutations are restricted to the jailbreak template, producing candidates by combining the mutated template with the original payload.
- NeuronFuzz fuzzing loop: NeuronFuzz uses feedback-guided scheduling to balance exploration of new jailbreak templates with reuse of promising ones.MCTS-Explore selects templates using statistics accumulated from previous oracle feedback.
4 Implementation & Experiment Setup
NeuronFuzz is evaluated with white-box source models, disjoint oracle test sets, and metrics covering discovery, efficiency, template quality, and transferability.
- Evaluation Setup: Five models serve as white-box source models, while optimized templates are applied zero-shot to target models without target-specific optimization.The model set spans dense and mixture-of-experts architectures, reasoning modes, and other model variations.
- Evaluation Setup: The SafetyOracle evaluation uses separate clean and jailbreak test sets to assess classification and behavior under templated harmful inputs.Each clean set contains 2,000 harmful and 2,000 benign prompts; the jailbreak set contains 4,000 harmful prompts combined with randomly selected templates.
- Evaluation Metrics: Jailbreak discovery is judged through independent Llama-Guard-4-12B and Qwen3Guard-Gen-8B evaluations, with human review when their decisions disagree.The agreed label is used when both judges produce the same decision.
- Evaluation Metrics: JDR measures whether at least one successful template is discovered per harmful question, while RGD and ETD quantify response-generation and end-to-end discovery costs.RGD and ETD are reported only over successful questions.
- Evaluation Metrics: ASR measures an individual template’s effectiveness, whereas EASR measures the fraction of questions successfully tested by a selected set of templates.EASR evaluates whether at least one template in the selected subset succeeds.
- Evaluation Objectives: The experiments test oracle validity, fuzzing effectiveness and efficiency, universal templates, and transferability across model and task settings.The evaluation examines whether SafetyOracle feedback ranks candidates and reduces response-generation and end-to-end search costs.
5 Experimental Results
NeuronFuzz uses SafetyOracle signals to guide fuzzing without repeated intermediate response generation, achieving strong jailbreak discovery, efficiency, template transfer, and multimodal transfer across evaluated models.
- 5.1 EO1: Oracle Validity: AUROC values of 0.969-0.999 show that SafetyOracle distinguishes harmful from benign inputs across all five source models.Benign prompts receive alarm scores near zero, while harmful prompts receive scores near one.
- 5.1 EO1: Oracle Validity: Successful jailbreaks receive lower mean safety alarm scores than failed attempts on every source model, with ∆sep ranging from 0.327 to 0.482.On DeepSeek-R1-14B, the mean score decreases from 0.937 for failed attempts to 0.557 for successful jailbreaks.
- 5.2 EO2: Fuzzing Effectiveness and Efficiency: 100% JDR on three source models and 76% and 96% JDR on Gemma-4-E4B-it and GPT-OSS-20B make NeuronFuzz the highest-JDR method on all five models.It exceeds evaluated methods by 27 and 48 percentage points on the two more strongly aligned models.
- 5.2 EO2: Fuzzing Effectiveness and Efficiency: RGD of 1.0 reflects that NeuronFuzz evaluates intermediate candidates with prefill-only SafetyOracle scoring and generates responses only for final verification.This avoids repeatedly generating target-model responses as mutation feedback.
- 5.3 EO3: Universal Template Optimization: EASR reaches at least 90% on four models and 82% on Gemma-4-E4B-it, with most gains occurring during early universal-template optimization iterations.ASR and EASR improve as the mean safety alarm score decreases and stabilize near the end of the 20-iteration budget.
- 5.4 EO4: Cross-Model Transferability: Zero-shot transfer achieves average ASR/EASR of 69.6%/92.6% across eight inference-only targets and 44.1%/60.0% across six proprietary models.Transferred templates also raise average ASR to 64.4% on T2I and 74.8% on NSFW tasks, with EASR of 85.8% and 93.0%.
6 Ablation Study
The ablations show that template-invariant extraction and stability-aware selection improve safety-neuron quality, while continuous oracle feedback and gradient-guided mutation are both important for fuzzing effectiveness.
- Safety-neuron construction: Removing template-invariant extraction reduces AUROC from 0.989 to 0.834, correlation from −0.964 to −0.437, and JDR from 100% to 74%.Clean-prompt-only neuron identification generalizes less effectively to harmful inputs under jailbreak contexts.
- Safety-neuron construction: Random neuron selection achieves AUROC 0.682, positive correlation 0.121, and 16% JDR, substantially below the full method.Stability-aware selection favors neurons with associations that remain consistent across training samples.
- Oracle-guided fuzzing: The full method reaches 99% JDR, whereas removing gradient guidance reduces JDR to 21% while both retain an RGD of 1.0.Continuous SafetyOracle scoring remains available without gradient guidance, but mutation positions are selected randomly.
- Oracle-guided fuzzing: Replacing the continuous oracle with binary feedback reduces JDR to 4% and increases RGD to 18.3 on GPT-OSS-20B.Most candidates receive the same failure label on this strongly aligned model, providing little mutation guidance.
7 Defenses
The defense experiments examine perplexity filtering and SafetyOracle-based jailbreak detection. NeuronFuzz remains effective under perplexity filtering, while the SafetyOracle provides a pre-response signal for identifying likely successful jailbreaks.
- Perplexity filtering: Across five models, the perplexity filter decreases JDR by only 2–8%.Masked-language-model mutations preserve template length and grammatical structure, avoiding anomalous high-perplexity sequences.
- Jailbreak detection: Benign prompts usually have alarm scores near zero, direct harmful prompts near one, and successful jailbreaks often fall between these regions.The detection score Djb(x) is highest when Salarm(x) = 0.5.
- Jailbreak detection: The detector achieves AUROC values between 0.948 and 0.963 across three models.Its average precision and F1 score are 0.882 and 0.890, respectively.
- Jailbreak detection: At a fixed false-positive rate of 5%, the detector identifies 87.9% of successful jailbreak prompts on average.The detector uses the intermediate alarm-score region associated with successful jailbreaks.
8 Conclusion
The conclusion presents NeuronFuzz as a white-box fuzzing framework using internal neuron activations as continuous feedback for LLM safety evaluation. It also reports that the SafetyOracle supports input-side jailbreak detection and generalization across payloads, models, and modalities.
- Conclusion: NeuronFuzz uses internal neuron activations as continuous feedback for LLM safety evaluation.Its SafetyOracle supplies differentiable safety alarm scores for candidate ranking and targeted template mutation.
- Conclusion: The framework reduces reliance on repeated response generation while preserving harmful payloads and overall prompt semantics with strong transferability.The conclusion reports generalization across different payloads, models, and modalities.
- Conclusion: The SafetyOracle can also serve as an input-side detector for identifying likely jailbreak prompts before response generation.The conclusion positions internal model signals as execution feedback for fuzzing and defensive signals for systematic safety evaluation.
- Conclusion: The template evaluation retains 74 jailbreak templates after removing duplicates, near-duplicates, and templates that directly modify the input payload.The filtering was intended to reduce additional variables and preserve evaluation fairness.
B Baseline Methods
The baseline evaluation implements four representative jailbreak methods using released implementations and their main optimization procedures. All methods share the same harmful payloads, hardware, and final response-level evaluation pipeline.
- Baseline methods: The study implements GCG, AutoDAN, PAIR, and LLM-Fuzzer as representative jailbreak baselines.Their released implementations and main optimization procedures are retained.
- GCG: GCG appends an optimized suffix directly to the harmful payload, retaining the candidate with the lowest target loss for the next step.The baseline follows the original work and official implementation.
- AutoDAN: AutoDAN initializes handcrafted jailbreak prompts and evolves them through hierarchical crossover and mutation.Candidates are ranked by the negative log-likelihood of an affirmative target response, using a population size of 256 and crossover rate 0.5.
- Experimental setup: Experiments use four NVIDIA GH200 nodes, each equipped with one NVIDIA H100 GPU with 120 GB of memory.Most experiments ran on a single node, while large-scale evaluations used multiple nodes.
C.1 SafetyOracle Construction Cost
SafetyOracle construction has a modest one-time offline cost, dominated by activation extraction and stability-aware neuron selection rather than classifier fitting.
- 18m32s average is required to construct the complete SafetyOracle across five source models, with total times ranging from 14m39s to 22m38s.
- 9m56s is the average cost of stability selection, compared with 7m34s for activation extraction.
- 1m01s average classifier-fitting time shows that the final Elastic Net head contributes little to the total construction cost.
E SafetyOracle Probe Model Selection
Elastic Net logistic regression is selected as the default SafetyOracle probe because it most consistently correlates with response-level jailbreak success across evaluated models.
- Elastic Net logistic regression achieves the strongest negative correlation on all three evaluated models.Its Spearman coefficients are −0.9636, −0.9758, and −0.9273 on Llama-3.1-8B-Instruct, Gemma-3-4B-it, and DeepSeek-R1-14B, respectively.
- −0.9758 correlation on Gemma-3-4B-it is maintained by Elastic Net, while the small MLP reaches −0.8139 on that model.
- 0.575–0.642 mean |ρ| values for the remaining linear probes are substantially weaker than Elastic Net’s consistently high correlations.
- Elastic Net is used by default because its continuous score ranks candidates and guides search toward prompts more likely to produce successful jailbreaks.
G Cross-Dataset Evaluation on HarmBench
Templates transferred from the fuzzing setup remain effective on unseen HarmBench prompts, substantially outperforming direct harmful-prompt baselines across four target models.
- 87.3% average top-5 EASR is achieved using five selected templates, with target-model EASRs ranging from 78% to 98%.
- 84% top-1 ASR and 98% top-5 EASR are obtained on Llama-3.1-8B-Instruct, the strongest reported target-model results.
- 57%–84% top-1 ASR is obtained across targets, including Qwen3.6-27B, GLM-4.7-Flash, and DeepSeek-V4-Flash despite their zero baseline ASR.
H Robustness of Baseline Attacks to Input Defenses
NeuronFuzz remains effective under input defenses, while its masked-language-model mutations preserve natural-language structure and are less vulnerable to simple filtering.
- Robustness under input defenses: 95% JDR under the perplexity filter and 71% under SmoothLLM are achieved by NeuronFuzz, the highest JDRs reported under both defenses.
- Robustness under input defenses: 91% to 6% JDR reduction occurs for GCG under the perplexity filter, whereas NeuronFuzz retains 95% JDR.
- Mutation behavior: Masked language model replacements modify only a small number of tokens, helping preserve template structure and fluency.
- Mutation behavior: Natural-language-like optimized templates are less likely to be disrupted by simple input-side filtering mechanisms.