PaperlayerView exact revision on arXiv

Source-linked AI summary

CLEAR: Continuous Latent Adapter Routing for Utility-Preserving LLM Safety Alignment

Chengxiao Wang, Enyi Jiang, Xiaojing Liao, Sanmi Koyejo

arXiv:2608.21278v1cs.AI

TL;DR

Globally applied safety tuning can reduce harmful outputs but also alter benign behavior, creating a safety–utility trade-off. CLEAR freezes the backbone and uses a latent gate to continuously route a safety LoRA adapter according to prompt risk. Across multiple benchmarks, it improves harmful-prompt robustness while preserving more utility than global SFT or standard LoRA.

  • Problem

    Global safety tuning may affect both harmful and benign inputs, creating an alignment tax through utility degradation and over-refusal.

  • Method

    CLEAR uses a learned hidden-state gate to continuously control a safety-specific LoRA adapter while keeping the backbone frozen.

  • Results

    CLEAR reduces HarmBench ASR to 0.50% on Llama-3-8B-Instruct while retaining 73.46% GSM8K accuracy, about 7 percentage points above SFT and standard LoRA.

  • Takeaways & Limitations

    Continuous latent adapter routing improves the reported safety–utility trade-off compared with globally applied safety tuning across multiple safety and utility benchmarks.

  • Takeaways & Limitations

    CLEAR depends on reliable routing, and its generalization to larger frontier or multimodal systems remains unclear.

Abstract

from arXiv · show

Improving the safety of large language models (LLMs) often comes at the expense of utility, as globally applied safety tuning may affect model responses to both harmful and benign inputs. We propose \textbf{C}ontinuous \textbf{L}at\textbf{E}nt \textbf{A}dapter \textbf{R}outing (CLEAR), a conditional safety adaptation framework that uses a lightweight hidden-state gate to continuously control the activation strength of a safety low-rank adapter. CLEAR aims to reduce harmful completions while avoiding unnecessary changes to the frozen backbone that could degrade performance on benign prompts. Experiments on widely used safety and utility benchmarks show that CLEAR improves robustness on HarmBench while reducing the utility degradation observed with globally applied safety tuning such as SFT or standard low-rank adaptation (LoRA). On Llama-3-8B-Instruct, CLEAR reduces HarmBench ASR from 32.3\% to 0.5\%, while retaining most of the base model's utility and achieving up to 7.1 percentage points higher GSM8K accuracy than globally applied SFT or LoRA. These results suggest that CLEAR is a promising mechanism for improving the safety--utility trade-off in LLM alignment.

Introduction

CLEAR addresses the alignment tax by routing a safety adapter conditionally through a latent gate, preserving benign behavior while strengthening intervention on harmful inputs. Across safety and utility benchmarks, it improves harmful-prompt robustness while retaining stronger utility than globally applied alignment methods.

  • Motivation: Global safety tuning can cause over-refusal and degraded reasoning on benign prompts resembling harmful requests.This safety–utility tension motivates selective intervention based on detected harmful intent.
  • Method: CLEAR freezes the backbone and continuously controls a safety LoRA adapter with a learned latent gate.The gate applies little or no intervention to benign prompts and stronger activation to harmful or adversarial prompts.
  • Method: Subtype-aware gate weighting and a hard pairwise margin objective separate harmful and benign prompts in latent gating space.The gate is jointly optimized with the safety adapter rather than operating as an external filter or post-hoc moderation module.
  • Evaluation: CLEAR is evaluated on HarmBench and XSTest for safety, and GSM8K, MMLU, and TruthfulQA for utility across Llama-3-8B-Instruct and Gemma-2-2B-it.The reported evaluation spans multiple safety and general-utility benchmarks.
  • Results: 0.50% HarmBench ASR is achieved by CLEAR on Llama-3-8B-Instruct, while other aligned variants remain between 15.50% and 24.00%.CLEAR also retains 73.46% GSM8K accuracy, about 7 percentage points higher than SFT and standard LoRA.

Background

The paper formulates safety alignment as preserving utility on benign prompts while reducing harmfulness, then replaces global parameter updates with input-dependent adapter control. A continuous gate scales the safety LoRA update according to estimated prompt risk.

  • Safety alignment objective: Safety alignment jointly seeks to preserve utility on benign prompts and reduce harmfulness on unsafe or adversarial prompts.The objective uses U(·) for task utility and H(·) for harmfulness or attack success.
  • Global alignment: Global updates used by SFT, RLHF, DPO, and safety LoRA apply the same modified policy to harmful and benign safety-adjacent prompts.This input-agnostic intervention can reduce harmful outputs but may cause utility degradation and over-refusal.
  • Conditional alignment: CLEAR approximates a conditional policy with a frozen backbone, safety-adapter parameters, and input-conditioned intervention strength g(x) ∈[0, 1].Low-risk prompts remain close to the original backbone, while high-risk prompts receive stronger safety-aligned behavior.
  • Adapter routing: Unlike standard LoRA, CLEAR scales the safety update as h′ = (W + g(x)∆Wθ) h rather than applying the same adapter uniformly.The gate is intended to reduce unnecessary refusals on benign prompts while enabling stronger control for harmful prompts.

Method

CLEAR freezes the backbone and combines a hidden-state gate with a safety-specific LoRA branch to apply continuous, input-conditioned intervention. The gate estimates intervention strength from prompt representations, while the adapter learns safe behavior for unsafe inputs.

  • CLEAR freezes the pretrained backbone and adds a safety-specific LoRA branch whose contribution is modulated by a scalar hidden-state gate.
  • The gate produces a continuous score gi ∈[0, 1] that scales the safety LoRA update inside adapted projection layers during the forward computation.
  • CLEAR computes the intervention score before generation from prompt-only hidden states and does not observe assistant completion tokens.
  • The gate uses aggregated hidden states and a trainable sigmoid-output MLP, with binary supervision yielding continuous rather than hard routing scores.
  • Subtype-aware weighting emphasizes adversarial benign and adversarial harmful prompts because they are more likely to confuse the gate.
  • A pairwise margin loss encourages unsafe prompts to receive larger gate scores than safe prompts and emphasizes confusing prompt pairs.
  • The same scalar gate is shared across adapted modules for each sequence, controlling overall adapter strength rather than separate layer, module, or token gates.
  • Training jointly optimizes gate and LoRA parameters while keeping the backbone frozen; the adapter alignment objective uses safe target responses only for unsafe examples.

Experimental Evaluation

CLEAR is evaluated across safety, utility, routing, scaling, and adaptive-attack settings against globally applied tuning and external guards. Results show stronger safety–utility balance and latent robustness, while routing quality and benefits vary across metrics and model scales.

  • Experimental setting: The experiments report HarmBench ASR, XSTest refusal and over-refusal rates, GSM8K, MMLU, and TruthfulQA MC1/MC2 under common evaluation settings.Unless otherwise stated, results are medians over three independent runs with greedy decoding.
  • Comparison to safety tuning baselines: CLEAR reduces HarmBench ASR from 32.25% to 0.50% on Llama-3-8B-Instruct while retaining 73.46% GSM8K accuracy.Standard LoRA reaches 0.00% HarmBench ASR, so CLEAR’s advantage is its overall balance rather than dominance on every metric.
  • Comparison to safety tuning baselines: CLEAR reduces HarmBench ASR from 32.25% to 0.50% in the Llama family and from 22.50% to 0.50% for P-SACPO + CLEAR in the Alpaca family.Utility remains close to the corresponding initialization in both settings.
  • Gate quality: CLEAR provides a favorable XSTest routing trade-off using a learned gate, while PromptGuard routes nearly all prompts as unsafe under direct label mapping.The comparison measures routing decisions rather than generated refusal rates, and the external-guard result reflects an objective mismatch.
  • Scaling: CLEAR reduces HarmBench ASR to at most 2.00% across Qwen model sizes while largely preserving utility.Gate separability improves with scale: ROC-AUC rises from 0.78 on Qwen-0.5B to 0.96 on Qwen-7B, and PR-AUC from 0.75 to 0.93.
  • Adaptive robustness: CLEAR achieves ASRs of 0.0%, 7.1%, and 18.4% across three latent injection depths, versus 54.0%, 48.0%, and 31.0% for LoRA.The evaluation applies adaptive hidden-state perturbations after projection onto a clean PCA subspace estimated from benign prompts.

Conclusion

CLEAR uses conditional latent adapter routing to improve the safety–utility trade-off without globally modifying the backbone. Its benefits depend on reliable routing, because benign prompts may trigger intervention and unsafe prompts may receive low gate scores.

  • Conclusion: CLEAR modulates a safety-specific LoRA adapter with a hidden-state gate conditioned on prompt risk.Subtype-aware optimization and a hard pairwise margin objective separate benign and malicious inputs in routing space.
  • Conclusion: CLEAR improves harmful-prompt robustness while preserving more general-purpose capability than globally applied safety tuning across multiple benchmarks.The conclusion frames this as an improved safety–utility trade-off rather than universal superiority on every metric.
  • Conclusion: Reliable routing remains a constraint because benign prompts can trigger unnecessary intervention and unsafe prompts assigned low gate scores can bypass the safety adapter.The paper identifies gate calibration and robustness under distribution shift as future-work directions.

Limitations

CLEAR improves the safety–utility trade-off but remains dependent on reliable gating, training-distribution diversity, and evaluation beyond small open-weight, single-turn text settings.

  • Gate errors can either trigger unnecessary safety interventions on benign prompts or let unsafe prompts bypass the safety adapter.
  • Experiments primarily use relatively small open-weight models, leaving generalization to larger frontier or multimodal systems unclear.
  • Robustness may depend on safety-training diversity, so unseen jailbreak styles or distribution shifts may circumvent the safety branch.
  • The framework addresses single-turn text safety but not long-horizon deception or multi-turn adversarial interactions.

Reproducibility Details

The experiments use Gemma-2-2B-it and Llama-3-8B-Instruct with WildJailbreak data, standardized training details, benchmark-specific evaluation protocols, and substantially lower computational cost than full SFT.

  • Backbone checkpoints: Experiments use google/gemma-2-2b-it and meta-llama/Meta-Llama-3-8B-Instruct as the Gemma and Llama checkpoints.
  • Training data: The WildJailbreak training split contains 261,559 examples: 128,781 safe and 132,778 unsafe across four prompt subtypes.
  • Hyperparameter details: SFT, standard LoRA, and CLEAR use the same WildJailbreak data and one epoch, with an effective batch size of 64.
  • Evaluation: GSM8K and MMLU use 5-shot evaluation, TruthfulQA uses 0-shot evaluation, and safety benchmarks use greedy decoding with specified generation limits and classifiers.
  • Computational cost analysis: CLEAR trains 5.51 × 10^6 parameters versus 8.03 × 10^9 for SFT and requires less than half the memory on Llama-3-8B-Instruct.

Additional Ablations

Ablations favor a lightweight multi-layer MLP gate and the full CLEAR training objective for balancing safety and utility.

  • Gate architecture and layer selection: Multi-layer MLP gating reduces HarmBench ASR from 0.25% to 0.00% and increases GSM8K from 34.42% to 41.93%.
  • Gate architecture and layer selection: The multi-layer Transformer gate improves over its single-layer counterpart but does not consistently outperform the multi-layer MLP.
  • Training objective: The BCE-only objective increases HarmBench ASR from 0.00% to 3.25% and reduces unsafe refusal from 96.00% to 83.50%.
  • Training objective: BCE-only routing also raises safe over-refusal to 14.40%, while removing LoRA L2 regularization slightly worsens safety and GSM8K.

Latent Space Adaptive Attack

The evaluation applies targeted white-box PGD directly to intermediate hidden states, while the reported implementation uses CLEAR’s gated configuration and compares adapter hyperparameters across methods.

  • Attack formulation: Representation-level robustness is evaluated with a targeted PGD attack applied directly to intermediate hidden states.
  • Adapter configuration: CLEAR uses LoRA rank 8, scaling factor 16, q_proj and v_proj targets, and a gated adapter active at inference; its backbone remains frozen.
  • Training configuration: Table 5 compares training and adapter hyperparameters for SFT, standard LoRA, and CLEAR under shared WildJailbreak data and one epoch.
  • Gate inputs: CLEAR’s gate uses three intermediate and later transformer layers as hidden-state inputs for each backbone.
  • Adaptive intervention: Anomaly-score interpolation centers the transition at a = 0.5 and applies stronger projection for unusually high gate scores while minimally altering benign prompts.

Training Dynamics

CLEAR’s optimization losses decrease during training while its gate increasingly separates safe from unsafe prompts. The resulting score separation supports input-dependent activation of the safety adapter.

  • The LoRA language modeling loss, gate BCE loss, and hard-pair margin loss all decrease during training.These losses correspond to learning aligned adapter responses, the safe-versus-unsafe boundary, and greater routing separation.
  • The gate learns both coarse safe-versus-unsafe classification and a larger margin between safe and unsafe prompts.The decreasing BCE and hard-pair margin losses indicate progress on both aspects of routing.
  • Unsafe prompts receive increasingly higher gate scores while safe prompts receive lower scores as training progresses.The separation is especially pronounced on Llama-3-8B-Instruct and remains clear on Gemma-2-2B-it.
  • The learned gate activates the safety adapter primarily for unsafe inputs while suppressing unnecessary activation on benign prompts.This interpretation follows from the increasingly separated mean gate scores.

Evaluation Metrics and Judging Protocols

The evaluation combines classifier-based HarmBench judging, refusal-based XSTest metrics, and lm-eval-harness utility benchmarks. These protocols quantify harmful compliance, unsafe refusals, safe over-refusals, and task performance using specified prompting and scoring rules.

  • HarmBench: HarmBench attack success rate uses an official classifier to label target-model generations as harmful or not.Responses are generated with greedy decoding, the chat template, and max_new_tokens=512; the classifier answers yes or no, with yes indicating unambiguous non-minimal harmful behavior.
  • Ablation: The Gemma-2-2B-it training-objective ablation reports percentage results for HB ASR, unsafe refusal, and safe over-refusal.CLEAR denotes the full objective with subtype-weighted BCE, hard pairwise margin loss, and LoRA L2 regularization.
  • HarmBench: HarmBench ASR is the fraction of generations judged yes by the classifier.The exact HarmBench judge prompt is provided in Figure 5.
  • XSTest: XSTest reports unsafe refusal rate for unsafe prompts and safe over-refusal rate for safe prompts.Completions are classified using lowercased, stripped text and standard refusal phrases; the two rates measure refusal behavior on the respective prompt groups.
  • Utility: Utility evaluation uses lm-eval-harness with 5-shot GSM8K and MMLU prompts and 0-shot TruthfulQA prompts.Reported metrics are exact match for GSM8K, accuracy for MMLU, and MC1/MC2 accuracy for TruthfulQA.
Loading 2608.21278v1…