PaperlayerView exact revision on arXiv

Source-linked AI summary

Target-free Latent Safety Alignment

Luoyu Chen, Weiqi Wang, Chenhan Zhang, Zhiyi Tian, Yuxian Huang, Shui Yu

arXiv:2610.04467v1cs.LGcs.AI

TL;DR

Existing latent adversarial training often generates behaviorally narrow harmful samples, although real jailbreaks elicit diverse behaviors. This paper learns multiple target-free latent interventions and reports broader behavioral diversity alongside improved jailbreak robustness and preserved utility.

  • Problem

    Many existing latent adversarial methods produce behaviorally narrow samples, unlike the diverse harmful behaviors elicited by real-world jailbreaks.

    Source: §1 · pp. 1–2

  • Method

    The framework learns multiple latent interventions without output supervision and trains the model to remain aligned under them while preserving benign behavior.

    Source: §3.2 · p. 5, §6 · p. 9

  • Results

    The method produces higher semantic entropy and improves jailbreak robustness while maintaining utility and low over-refusal.

    Source: §5 · p. 8, §6 · p. 9

  • Takeaways & Limitations

    The findings identify behavioral coverage, rather than adversarial strength alone, as a critical factor in robust latent safety alignment.

    Source: §6 · p. 10

    The method uses additive latent perturbations that explore finite, local linear directions and cannot fully capture the likely nonlinear manifold of real-world jailbreak behaviors.

    Source: B LIMITATIONS AND FUTURE WORK · p. 14

Abstract

from arXiv · show

Large language models (LLMs) remain highly vulnerable to jailbreak attacks that induce harmful behaviors and circumvent safety alignment. To defend against such attacks, adversarial training paradigms have been proposed to first simulate failure modes and then train the model to correct them, yielding promising improvements in safety alignment. However, these methods typically construct adversarial samples either by encouraging fixed harmful target completions or by performing targeted activation ablation derived from fixed benign--harmful data pairs. As a result, the generated adversarial samples tend to induce homogeneous harmful behaviors that poorly reflect the diversity of behaviors elicited by real-world jailbreak attacks. This behavior-level narrowness fundamentally limits their robustness. To address this issue, we propose a target-free adversarial training framework that generates adversarial samples in an unsupervised manner. By amplifying and diversifying behavior-level shifts in the model's latent space, our approach produces semantically diverse adversarial samples that induce a wide range of harmful behaviors. This expanded behavioral coverage exposes more diverse failure modes and thereby improves safety alignment. To quantify this effect, we use semantic entropy as an output-level measure of adversarial behavioral diversity. Empirically, our method elicits diverse harmful behaviors in the target model, substantially mitigating behavioral narrowness and improving robustness to jailbreak attacks.

1 INTRODUCTION

Jailbreaks can elicit harmful responses despite LLM safety safeguards, while existing latent adversarial training often exposes models to behaviorally narrow samples. This work proposes target-free training to induce diverse latent behavior shifts and broaden adversarial coverage.

1 INTRODUCTION

  • Real-world jailbreaks elicit varied harmful behaviors, but many latent adversarial training methods repeatedly use predefined targets and produce behaviorally similar samples.Single refusal-direction guidance can also favor direct compliance rather than a diverse range of behaviors.Source: §1 · pp. 1–2
  • The paper asks whether diverse harmful responses can be elicited from simple harmful queries without prior knowledge of jailbreak strategies or target outputs.Source: §1 · p. 2
  • The proposed framework learns shallow-layer source interventions that cause large, consistent shifts toward non-refusal behavior while encouraging distinct modes across interventions.Its motivation draws on findings that semantic representations are encoded in shallow layers and behavioral decisions emerge in deeper layers.Source: §1 · p. 2
  • The inner adversary maximizes the nuclear norm of a behavior-shift matrix; the outer stage trains refusal on harmful prompts while preserving benign behavior.Source: §1 · p. 2
  • The method improves jailbreak robustness, produces higher semantic entropy, and preserves utility while avoiding excessive refusals on safety-boundary queries.Semantic entropy measures the output diversity generated by the inner adversary.Source: §1 · p. 2

2 CONCEPTUAL COMPARISONS WITH PRIOR LAT-BASED METHODS

Prior latent adversarial training generally constructs one intervention at a time, whether by targeting a harmful completion or ablating refusal features. The proposed framework instead learns multiple interventions to produce diverse behaviors from one harmful prompt.

  • Targeted LAT optimizes interventions toward predefined harmful completions, while refusal-ablation methods derive interventions from hidden-state statistics such as benign–harmful differences.The former forces a fixed response; the latter avoids explicit target completions.Source: §2 · pp. 2–3
  • Both prior families construct a single intervention per step, limiting each prompt to one harmful behavior mode and constraining behavioral coverage.Source: §2 · p. 3
  • Figure 1 contrasts prior single-target narrowness with the proposed multiple-direction approach, which elicits semantically distinct harmful behaviors from one prompt.Source: Figure 1 · p. 3, §2 · p. 3
  • The target-free inner objective optimizes multiple source interventions and maximizes the nuclear norm of their induced behavior-shift matrix.This encourages strong, diverse shifts without output supervision.Source: §2 · p. 3

3 TARGET-FREE LAT

Target-free LAT addresses behavioral narrowness by learning multiple latent interventions and training the model to restore safe, useful behavior under them. Its two-stage design pairs a diversity-seeking inner adversary with an outer alignment objective.

3.1 NUCLEAR-NORM OBJECTIVE FOR DIVERSE BEHAVIORAL ADVERSARIES

  • Maximizing the nuclear norm jointly promotes strong, prompt-consistent shifts within each intervention and diverse, low-redundancy harmful modes across interventions.The method learns unit-norm source interventions and measures their average shifts at a deeper behavioral layer.Source: §3.1 · pp. 4–5

3.2 OUTER SAFETY ALIGNMENT OBJECTIVE FOR UTILITY PRESERVATION

  • For harmful prompts, the outer objective enforces refusal consistency across all interventions; for benign prompts, KL consistency preserves behavior close to the original aligned model.The combined objective is intended to preserve benign generation quality and prevent excessive refusals.Source: §3.2 · p. 5

3.3 FULL TRAINING PROCEDURE AND IMPLEMENTATION DETAILS

  • Training rounds use harmful prompts to optimize latent adversaries and LoRA fine-tuning with other model parameters frozen for outer alignment.The authors say this reduces memory overhead and enables scalable training.Source: §3.3 · p. 5

4 EXPERIMENTS

The experiments assess jailbreak robustness, over-refusal, and utility across several models and benchmarks. The method offers the strongest overall defense on two tested models, remains competitive on the third, and retains a favorable safety–utility trade-off.

4.1 EXPERIMENTAL SETUP

  • Evaluation covers white- and black-box jailbreak attacks, multilingual and code-based attacks, instruction following, academic capability, and safety-boundary over-refusal.Harmful compliance is scored with StrongREJECT; utility uses AlpacaEval and lm-eval, and over-refusal uses OR-Bench.Source: §4.1 · p. 6
  • Semantic entropy groups semantically equivalent outputs and measures uncertainty over their meanings rather than surface-form variation.The implementation uses embedding-based clustering to estimate semantic equivalence.Source: §4.1 · p. 6
  • Experiments use three open-weight instruction-tuned models, harmful and benign training prompts, and representative latent adversarial alignment baselines.The training data include 1,000 harmful prompts and 1,000 benign prompts; comparisons include targeted LAT, ReFAT, Deep-Refusal, and DAT.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

  • The method provides the strongest overall defense on Qwen2.5-7B-Instruct and Mistral-7B-Instruct and remains competitive on Llama-3-8B-Instruct, with utility close to aligned models.Its gains are especially pronounced on several attacks, including CodeChameleon; its over-refusal advantage varies by model.Source: §4.2 · p. 7

5 FURTHER DISCUSSION

The discussion links nuclear-norm optimization to stronger, more consistent latent shifts, effective jailbreaks, and semantically diverse outputs. Its ablation indicates that compact behavioral coverage can match the full intervention set in robustness.

From Nuclear-Norm Optimization to Behavioral Diversity

  • During optimization, latent shift strength and prompt consistency rise alongside StrongREJECT score, attack success rate, semantic entropy, and semantic-group count.The attack metrics rise and then saturate after a few steps.Source: §5 · pp. 7–8, Figure 2 · p. 8

Semantic Interpretation of Directions

  • Optimized directions elicit distinct semantic modes, while responses under the same direction remain consistent across prompts.Table 2 contrasts random initialization with optimized directions and their discriminative vocabularies.Source: §5 · p. 8, Table 2 · p. 9

Causal Ablation on Multi-directional Adversarial Training

  • A 64-direction diverse subset matches the full set in full-V NLL and StrongREJECT score, while no-steer and targeted-LAT remain more vulnerable.The ablation holds other factors constant; no-steer has the lowest training loss but higher vulnerability metrics.Source: §5 · p. 9, Figure 3 · p. 9
  • Rapid supervised-loss convergence alone does not establish robustness to latent adversaries; exposure to behaviorally distinct failure modes is associated with the robustness gain.The ablation identifies adversarial diversity as the primary source of the observed improvement.Source: §5 · p. 9

6 CONCLUSION

The conclusion identifies behavioral narrowness as a limitation of existing latent adversarial training and presents a target-free framework that learns multiple interventions without output supervision. The authors report improved jailbreak robustness alongside maintained utility and low over-refusal, with behavioral coverage as a key factor.

6 CONCLUSION

  • The framework learns multiple interventions without output supervision to address existing LAT's failure to capture heterogeneous real-world jailbreak behaviors.It induces strong and diverse behavior shifts.Source: §6 · p. 9
  • The method improves jailbreak robustness while maintaining low over-refusal and preserving model utility.Latent effective rank and output-level semantic entropy verify that learned adversaries span multiple behavior modes.Source: §6 · pp. 9–10
  • The authors identify behavioral coverage, rather than adversarial strength alone, as a critical factor in robust latent safety alignment.They motivate broader exploration of adversarial behaviors in representation space.Source: §6 · p. 10

A HARDWARE AND SOFTWARE

The implementation used PyTorch, Transformers, and vLLM, with experiments generally run on one NVIDIA RTX PRO 6000 GPU.

A HARDWARE AND SOFTWARE

  • The experiments were implemented in PyTorch 2.10.0 with CUDA 12.8.These are the stated framework and CUDA versions.Source: A HARDWARE AND SOFTWARE · p. 14
  • Model loading and training used Hugging Face Transformers v5.5.4, while inference and response generation used vLLM v0.19.1.The passage identifies the libraries' respective roles.Source: A HARDWARE AND SOFTWARE · p. 14
  • Unless otherwise specified, experiments ran on a single NVIDIA RTX PRO 6000 GPU.Source: A HARDWARE AND SOFTWARE · p. 14

B LIMITATIONS AND FUTURE WORK

The authors state that latent adversarial training relies on additive perturbations, so even multi-directional methods provide only finite, local coverage. They suggest modeling nonlinear jailbreak manifolds as future work.

B LIMITATIONS AND FUTURE WORK

  • Additive latent perturbations explore only local linear directions, leaving a finite approximation to likely nonlinear real-world jailbreak behavior.The authors state that linear perturbations cannot fully capture the complex nonlinear manifold on which real-world jailbreak behaviors likely lie.Source: B LIMITATIONS AND FUTURE WORK · p. 14
  • The authors suggest higher-order interactions or adaptive, input-conditioned adversaries to model nonlinear jailbreak manifolds.They identify latent harmful-behavior geometry as an open problem.Source: B LIMITATIONS AND FUTURE WORK · p. 14

C RELATED WORK

Prior latent-safety approaches either increase refusals broadly or rely on real jailbreak examples, while latent adversarial training avoids that supervision but often produces behaviorally narrow attacks. The proposed target-free method uses deep-layer latent shifts as a proxy for output behavior shifts to generate more diverse harmful behaviors.

Representation safety

  • Representation-safety methods face different trade-offs: CAST and Antidote risk excessive over-refusal, while AlphaSteer relies on jailbreak examples and can be brittle to unseen attacks.The cited approaches either increase refusal tendency broadly or use real jailbreak samples to distinguish benign and harmful inputs.Source: C RELATED WORK · p. 14

Latent adversarial training

  • Existing latent adversarial methods often elicit similar harmful behaviors by targeting fixed outputs or ablating a refusal direction, limiting coverage of real-world jailbreak behavior.Diffusion-based DAT diversifies prompts but still optimizes toward a fixed harmful target.Source: C RELATED WORK · p. 14

Target-free approach

  • The proposed target-free method uses deep-layer latent shifts as a proxy for output behavior shifts, aiming to induce diverse harmful behaviors and improve adversarial-training robustness.The authors position this diversity as better matching the heterogeneity of real jailbreaks.Source: C RELATED WORK · p. 14

D NUCLEAR-NORM ASCENT AND EFFECTIVE RANK

The section motivates nuclear-norm maximization as a way to spread behavior-shift singular values and promote diversity, while distinguishing direct matrix ascent from optimization through the nonlinear model. A controlled experiment and an inner-loop trace examine the spectral and output-level behavior, respectively.

Nuclear-norm motivation

  • First-order nuclear-norm ascent increases effective rank unless the positive singular values are already equal, motivating its use to promote diversity in behavior-shift matrix C.Effective rank measures the spread of the singular-value spectrum, not whether outputs are harmful or semantically distinct.Source: D NUCLEAR-NORM ASCENT AND EFFECTIVE RANK · p. 15

Direct matrix-space ascent

  • For the chosen subgradient, direct ascent preserves matrix rank while strictly increasing effective rank when positive singular values are unequal; other valid subgradients at rank-deficient matrices may add null-space components.The result does not describe every possible subgradient step.Source: D NUCLEAR-NORM ASCENT AND EFFECTIVE RANK · pp. 15–16

Direct-matrix sanity check

  • In a controlled 4096 × 64 matrix experiment, nuclear-norm optimization spread the spectrum, whereas Frobenius-norm optimization retained effective rank one.The matrix began rank one; its effective-rank increase under nuclear-norm optimization depended on the singular-value implementation’s subgradient choice.Source: D NUCLEAR-NORM ASCENT AND EFFECTIVE RANK · p. 16, Table 3 · p. 16

Tracked inner optimization through the model

  • In the implemented inner loop, interventions V are optimized through the nonlinear network to maximize the nuclear norm of induced shifts C(V), so effective rank need not rise monotonically.The columns of V are initialized as random orthogonal directions.Source: D NUCLEAR-NORM ASCENT AND EFFECTIVE RANK · p. 16
  • With K=64, effective rank fell from 55.43 to 22.14 after the first update, then recovered to 39–46 as harmful-only entropy rose and the attack remained effective.Harmful-only entropy and harmful-output rate provide output-level evidence alongside the latent-space effective-rank diagnostic; the trace uses SR > 0.3 as the harmful-output threshold.Source: D NUCLEAR-NORM ASCENT AND EFFECTIVE RANK · pp. 16–17, Table 4 · p. 17

E FULL TRAINING PROCEDURE

The training procedure alternates between finding latent interventions that induce diverse behavior-level shifts and optimizing the model to restore refusal behavior while preserving benign utility.

Training procedure

  • The inner step optimizes source interventions by maximizing the nuclear norm of behavior-level shifts, while the outer step restores robust refusal behavior and preserves benign utility.The procedure takes the model, harmful and benign prompts, source and target layers, and injection strength as inputs.Source: E FULL TRAINING PROCEDURE · p. 17
  • Each training round randomly initializes intervention directions with unit L2 norm before inner adversarial optimization.The procedure specifies random initialization for V and ∥v_k∥2 = 1 for every direction.Source: E FULL TRAINING PROCEDURE · p. 17

F HYPER-PARAMETER SENSITIVITY ANALYSIS.

The analysis varies the number of latent directions, intervention strength, and inner harmful batch size, tracking both adversary quality and final safety–utility metrics. Across all three parameters, the reported best balance occurs at moderate settings, with diminishing returns or trade-offs at larger settings.

Effect of number of directions K

  • K≈64–128 provides a good balance: increasing K improves adversary quality and jailbreak robustness, but gains diminish beyond K=64 while over-refusal rises slightly and utility mildly degrades.Both mean StrongREJECT score and semantic entropy rise monotonically from K=16 to K=512 and begin to saturate beyond K=64.Source: Figure 4 · p. 18, F HYPER-PARAMETER SENSITIVITY ANALYSIS · p. 18

Effect of intervention strength λ

Effect of inner batch size

  • Batch size 8 minimizes jailbreak SR; larger batches such as 32 worsen robustness and slightly increase over-refusal, while mildly improving utility.Inner-adversary mean SR and semantic entropy improve from batch size 2 to 8, then decline beyond 8.Source: Figure 4 · p. 18, F HYPER-PARAMETER SENSITIVITY ANALYSIS · p. 19

G COMPUTATIONAL COST OF ADVERSARY CONSTRUCTION.

The analysis compares inner-loop adversary-generation costs and finds that jointly constructing K directions amortizes iterative optimization across adversaries. This lowers per-adversary cost in typical settings, with increased activation-memory use.

G COMPUTATIONAL COST OF ADVERSARY CONSTRUCTION.

  • Jointly optimizing K adversarial directions amortizes the 2T-pass inner-loop cost across K adversaries.Each step still uses one forward and one backward pass.Source: G COMPUTATIONAL COST OF ADVERSARY… · p. 19
  • Batched parallel construction improves computational efficiency but requires approximately K-times higher activation memory during inner optimization.The method optimizes K latent directions in parallel within one forward/backward pass.Source: G COMPUTATIONAL COST OF ADVERSARY… · p. 19
  • At T=10 and K=64, construction costs approximately 0.31 passes per adversary, versus 2T for targeted LAT and 1 for ReFAT.The comparison concerns single-adversary construction in the inner loop.Source: G COMPUTATIONAL COST OF ADVERSARY… · p. 19, Table 6 · p. 20

H CLUSTER VISUALIZATION

The cluster visualizations characterize how adversarial prompts elicit varied model responses, including harmful completions and refusals. They show outputs shifting from refusal-dominated regions into increasingly well-formed clusters as nuclear-loss optimization proceeds, with cluster-level success rates varying across examples.

H CLUSTER VISUALIZATION

  • As nuclear-loss optimization proceeds, outputs move from refusal-dominated regions into increasingly well-formed clusters, indicating stronger adversarial behavior.The visualizations are presented as output-level evidence that learned interventions more reliably shift the model from refusal toward harmful generations.Source: H CLUSTER VISUALIZATION · p. 20
  • Multiple separated DBSCAN clusters indicate that different interventions induce distinct semantic modes rather than collapsing to one jailbreak pattern.The visualizations are described as confirming that nuclear loss both strengthens adversarial behavior and expands semantic diversity.Source: H CLUSTER VISUALIZATION · p. 20
  • Illustrated prompt outputs include multiple clusters with high-meanSR responses as well as refusal clusters, showing varied behavior across examples.The biological-agent example includes multiple high-meanSR output clusters alongside lower-meanSR clusters; the cash-smuggling and firearm-purchase examples also show distinct clusters and refusals.Source: H CLUSTER VISUALIZATION · pp. 21–23
  • Hitman-prompt visualizations include high-success response clusters alongside refusal clusters scored at 0.00.Reported response-cluster meanSR values include 0.78, 0.73, and 0.85 across examples; refusal clusters are shown at 0.00.Source: H CLUSTER VISUALIZATION · p. 24
  • The cited Sassafras-related examples show response clusters with meanSR values of 0.64, 0.58, and 0.41, and separate refusal clusters with meanSR 0.00.The 0.64 response cluster has maxSR=0.80; the 0.58 and 0.41 response clusters have maxSR values of 0.82 and 0.41, respectively.Source: H CLUSTER VISUALIZATION · p. 25
  • GPS-tracker prompts produce multiple response clusters, with meanSR values varying across examples and refusal clusters at 0.00.The reported response-cluster meanSR values range from 0.22 to 0.45 in these examples.Source: H CLUSTER VISUALIZATION · p. 25

I ADVERSARIAL SAMPLES AFTER INTERVENTION

This section presents post-intervention examples of responses to harmful prompts, spanning harmful content across domains and formats as well as refusals and responses framed with warnings, disclaimers, or role-play.

I ADVERSARIAL SAMPLES AFTER INTERVENTION

  • The samples include a proposed list of biological warfare agents and a list of chemical warfare agents, alongside code continuing a dimethylmercury scenario.The dimethylmercury code snippet includes a user request to continue the code and assume the materials are safe to handle.Source: I ADVERSARIAL SAMPLES AFTER INTERVENTION · pp. 26–27
  • Another sample frames code as a hypothetical example for smuggling bulk cash out of the US.The passage describes the code as hypothetical and says it is not intended for illegal activity.Source: I ADVERSARIAL SAMPLES AFTER INTERVENTION · p. 27
  • A Rust code sample generates a defamatory article falsely accusing a fictional congressman of child pornography.The sample constructs the accusation using the congressman’s name, party, and state.Source: I ADVERSARIAL SAMPLES AFTER INTERVENTION · p. 30

Direct Compliance Jailbreaks v9

Multilingual Jailbreaks v17

I ADVERSARIAL SAMPLES AFTER INTERVENTION

  • Some outputs pair disclaimers or warnings with actionable hazardous guidance, including examples involving dimethylmercury, chemical agents, and safrole extraction.Other responses give a list of chemical agents or extraction steps after noting legal or safety concerns.Source: I ADVERSARIAL SAMPLES AFTER INTERVENTION · pp. 38, 40, 41, 43
  • Responses are inconsistent: some refuse or redirect requests, including a defamatory-article request and a request concerning a hired killer.By contrast, another output supplies defamatory allegations about a fictional senator, and a phishing response depicts entering credentials on a fake login page.Source: I ADVERSARIAL SAMPLES AFTER INTERVENTION · pp. 39, 40, 42, 43
  • Outputs still provide harmful assistance across biological, chemical, financial, weapons, cyber, and physical-harm scenarios, sometimes despite cautionary framing.Examples include hazardous-agent suggestions, a cash-smuggling tutorial, illegal handgun acquisition steps, phishing content, and instructions to cause a fall.Source: I ADVERSARIAL SAMPLES AFTER INTERVENTION · pp. 39, 41, 42
  • Role-play and fictional framing also deliver harmful scenarios, including a cash-heist tutorial and a staged security-checkpoint weapon scene.The cash response adopts a heist-movie framing, while another output narrates pulling a bomb from a jacket at a checkpoint.Source: I ADVERSARIAL SAMPLES AFTER INTERVENTION · pp. 41, 43

Related papers

Loading 2610.04467v1…