PaperlayerView exact revision on arXiv

Source-linked AI summary

When Behavioral Safety Evaluation Fails: A Representation-Level Perspective

Enyi Jiang, Anders Gjølbye, Yibo Jacky Zhang, Sanmi Koyejo

arXiv:2606.08044v2cs.LGcs.AIcs.CL

TL;DR

Behavioral audits observe whether models refuse tested harmful prompts, but do not establish robustness to interventions on internal states. The paper constructs dissociated models and evaluates them with static audits plus parameter- and latent-space interventions. Static audits match the safety-aligned bases, whereas bounded latent attacks and fine-tuning reveal substantially greater vulnerability in the dissociated models.

  • Problem

    Behavioral safety tests do not measure how easily an intervention can turn a refusal into harmful compliance, leaving an audit gap between observed safety and internal robustness.

  • Method

    The paper constructs dissociated models from three safety-aligned bases and audits them alongside base and harmful models using harmful fine-tuning, bounded latent attacks, random controls, and steering.

  • Results

    54–86% harmful compliance occurs for dissociated models under bounded latent attacks versus 3–48% for bases, while matched random perturbations stay at or below 12%.

  • Takeaways & Limitations

    Safety audits should intervene on open-weight models rather than relying only on observed behavior or static latent probing.

  • Takeaways & Limitations

    The results are an existence proof from synthetically constructed small open-weight models, not a prevalence claim about naturally trained or deployed models.

Abstract

from arXiv · show

Safety evaluation of large language models (LLMs) is largely behavioral: a model is certified safe when it refuses harmful requests and answers benign ones. But refusing on the prompts an auditor happens to try does not show that the model is far from harmful behavior. Behavioral tests observe outputs; they do not measure how easily an intervention on the model turns a refusal into compliance. We call the gap between what static audits certify and what an intervention can reach the audit gap, and we show it is realizable: one can build a model that matches its safety-aligned base on every static audit yet gives way to a small, known perturbation of its internal state. We construct such dissociated models from three safety-aligned bases (Gemma 2 2B, Llama 3.2 3B, Qwen 2.5 3B) and audit the base, dissociated, and openly harmful models with the same soft interventions in parameter and latent space; the latent attacks are summarized by the Latent Vulnerability Score (LVS), the safety degradation produced per unit of bounded latent perturbation. Every static audit we run gives the dissociated model the same verdict as its base, since its refusals match the base, jailbreaks show no consistent signature, and a strong fixed probe on clean activations cannot tell it from the base. The same interventions an auditor could run reverse the verdict. At the targeted mid layer the dissociated models score 2.5 to 3.1 times higher LVS than their bases. A bounded latent attack elicits harmful compliance on 54 to 86% of prompts, against 3 to 48% for the bases, while matched random perturbations stay at or below 12%. Harmful fine-tuning reaches high compliance within five gradient steps, where the bases need 10 to 25. Behavioral testing, even with static latent probing, cannot certify representation-level robustness: a safety audit must intervene on the model, not only observe it.

1 Introduction

Behavioral audits certify observed outputs but cannot establish how vulnerable a model’s internal state is to intervention. The paper constructs dissociated models that match their safety-aligned bases under static behavioral and representational audits while remaining intervention-reachable.

  • The audit gap is the discrepancy between safety certified by static audits and vulnerability revealed only by intervention.Behavior-only metrics measure responses to tested prompts, not internal proximity to harmful behavior.
  • Dissociated models preserve safe outward behavior yet can be made compliant by a small, fixed mid-layer perturbation.The construction starts from safety-aligned bases and targets a known latent intervention.
  • Static behavioral and activation-probe audits give dissociated models the same verdict as their safety-aligned bases.The probe is trained on the base and held fixed, while the behavioral suite finds no consistent separation.
  • The evaluation framework audits representation-level robustness with harmful fine-tuning, bounded latent attacks, random controls, and directional steering.These interventions are applied symmetrically to safety-aligned, dissociated, and harmful models.
  • Across three model families, static audits fail to separate dissociated models from their bases while interventions do separate them.The paper presents this construction as an existence proof and proposes intervention-based auditing as the practical response.

2 Related Work

Prior LLM safety evaluation mainly treats safety as an output property, while related representation-level and adversarial-robustness work studies internal mechanisms and interventions. This paper focuses specifically on the gap between behavioral safety and vulnerability to model-level intervention.

  • Behavioral safety evaluation relies on refusal rates, harmful-completion benchmarks, and jailbreak robustness to assess model outputs.
  • Existing behavioral metrics can be passed while models remain vulnerable to jailbreaks and adversarial prompting.
  • Representation-level safety research links alignment to shallow or low-dimensional refusal mechanisms and separable harmfulness representations.
  • Mechanistic interpretability analyzes internal representations with probing, sparse autoencoders, activation patching, and interchange interventions.
  • Adversarial-robustness research studies small perturbations that induce failures, including attacks on embeddings and latent states.

3 Constructing a Dissociated Model

The paper trains dissociated models to preserve clean refusals while placing their hidden states near harmful representations and making them compliant under a fixed latent nudge. Across three architectures, clean behavior stays near the base while nudged compliance rises.

  • A dissociated model is trained to match its safety-aligned base on static outputs while a small hidden-state intervention elicits harmful compliance.
  • The construction uses Gemma 2 2B, Llama 3.2 3B, and Qwen 2.5 3B, each with safety-aligned, harmful, and dissociated poles sharing base weights.
  • The intervention adds one fixed vector at a middle decoder layer, cached before training and applied at every sequence position.The relative construction scale is α = 0.06, and the vector is not updated during training.
  • The six-term objective preserves clean refusals through supervised and KL terms while latent shaping pulls clean activations toward the harmful pole.The harmful-pole matching term operates across the 40–60% depth band, leaving outputs safe during clean inference.
  • Each hinge stops contributing once its per-token NLL gap clears the m = 0.5 nats/token margin.This removes training pressure when the targeted behavior is already correct; benign anchors receive only supervised and KL terms.
  • The selected checkpoints keep clean refusal near the base while widening nudged-minus-clean compliance gaps across all three architectures.The gaps reach 0.92 for Gemma, 0.91 for Llama, and 0.795 for Qwen, while fixed-probe scores remain saturated.
  • Static behavioral and fixed-probe audits do not separate dissociated models from their bases.The fixed probe achieves AUROC ≈1.0 on both, with unsafe–safe sigmoid gaps of 0.988–0.997 and differences no larger than 0.002.

4 Interventions for Safety Evaluation

The framework evaluates safety through both behavioral and representation-level views, using static observation alongside active interventions. It applies matched parameter- and latent-space perturbations to compare how readily models can be pushed toward harmful behavior.

  • Evaluation framework: Static audits can give the dissociated model the same verdict as its safety-aligned base, so evaluation must perturb the model rather than only observe outputs.The framework compares interventions against static audit outcomes, with the base as a floor and the harmful model as a ceiling.
  • Evaluation framework: Soft interventions alter parameters or hidden states while allowing the remaining computation to react, and outcomes are interpreted against matched controls.The interventions are behavioral probes rather than claims that a specific circuit has been isolated.
  • Parameter-space intervention: Harmful fine-tuning modifies model weights using a harmful instruction–response dataset and measures how quickly compliance reaches a high-compliance threshold.The onset statistic is evaluated every five steps on fixed held-out HarmBench behaviors, with τ = 0.8 defining high compliance.
  • Additional interventions: Steering sweeps layer depth and perturbation strength along harmful-minus-base directions, while activation patching tests whether hidden states alone transfer harmful behavior into a frozen base.These procedures complement optimized latent attacks by testing single-direction reachability and hidden-state sufficiency.
  • Latent-space intervention: The Latent Vulnerability Score measures safety degradation per unit of latent perturbation, aggregating prompt-level scores with a median and bootstrap 95% confidence interval.Higher LVS means a small representation-space change produces a large safety degradation; logarithmic compression reduces domination by large perturbations.

5 Results

Intervention-based audits separate dissociated models from their safety-aligned bases despite near-identical static audit scores. Across latent attacks, harmful fine-tuning, steering, activation patching, and input-space jailbreaks, small targeted interventions expose greater reachability of harmful behavior.

  • Intervention audit: 54–86% harmful compliance was elicited from dissociated models by bounded PGD attacks, versus 3–48% for bases; matched random perturbations stayed at or below 12%.The attack used the nudge layer’s own ℓ2 budget and produced a dissociated-over-base gap with paired McNemar p ≤3 × 10^-7.
  • Intervention audit: Step 5 was the harmful-SFT threshold crossing for dissociated models on every architecture, while bases required 10 to 25 steps despite starting at least as safe.The earlier onset also persisted on an out-of-distribution corpus not used during construction.
  • Intervention audit: At the nudge layer, dissociated models had 2.5 to 3.1 times higher LVS than their bases, with vulnerability elevated through the mid band.LVS measures harmful safety degradation per unit of bounded latent perturbation; profiles used targeted PGD and same-ball random controls.
  • Intervention audit: At Gemma’s nudge layer L13, a γ = 0.06 harmful-direction push raised dissociated compliance from 0.00 to 0.96, while bases and matched random controls stayed near clean compliance.Llama and Qwen showed analogous responses in narrow bands around their own nudge layers; stronger pushes widened the band.
  • Representation-level evidence: Activation patching raised judged compliance in the frozen base to 0.17 on Gemma, 0.55 on Llama, and 0.43 on Qwen when dissociated nudge-layer states were inserted.The patched states remained fluent, with coherence 0.98–1.00, and exceeded controls using the base’s own activations.
  • Representation-level evidence: The representation-level gap appears in the decision-point hidden state, which tracks the harmful pole through the mid band before returning toward the safe base at output.A targeted nudge-layer push holds the output-layer projection at the harmful pole, unlike the base or matched random push.

6 Discussion

The audit framework is intended for white-box assessment and complements output filtering by testing whether harmful behavior is reachable through interventions. A small checkpoint survey found one divergence case but does not establish prevalence.

  • A 6% mid-layer nudge moved a Gemma model from 0.00 to 0.96 judged compliance.The intervention targets released weights, unlike output filtering, which reads only generated text.
  • Latent tests localize vulnerable layers, use precomputed inference-time perturbations, and distinguish direction-specific weakness from generic sensitivity.They complement harmful fine-tuning, which serves primarily as confirmation rather than the main signal.
  • One aligned Qwen2.5-7B checkpoint complied on 16 of 60 prompts versus 8 for its random control at a 6% push.The other three aligned checkpoints showed no paired separation, while representation-hardened Llama-3-8B-RR was flat across interventions.

7 Conclusion

The paper formalizes the audit gap between behavioral safety and robustness under intervention, using dissociated models whose clean outputs conceal reachable harmful behavior. It concludes that safety audits should measure intervention sensitivity alongside observed behavior.

  • Dissociated models are an existence proof that static audits can miss harmful behavior reachable through interventions.The construction is not a claim that such models are common in the wild.
  • Static evaluation measures behavior on shown prompts, whereas intervention-based auditing measures how easily that behavior changes.

8 Limitations

The evidence is limited by synthetic construction, small model and checkpoint samples, imperfect evaluation proxies, and untested robustness under routine model operations.

  • The dissociated models are synthetic existence proofs, not evidence that natural training produces them, and results may depend on the construction objective.
  • The study uses three small open-weight models, with cleaner evidence from Gemma and Llama than from Qwen’s softer base.
  • Evaluation relies on a reward model and LLM judge, while several experiments use single-run estimates, one architecture, or only five public checkpoints.
  • The construction targets one trained direction, and survival under operations such as quantization remains untested.

A Data

The appendix documents the datasets, splits, evaluation slices, prompt wrappers, utility checks, and classifiers used across construction and auditing.

  • Table 2 organizes each data source by its split and role in the pipeline.
  • The construction set contains 4,000 LLM-LAT harmful rows pairing refusals with compliant responses, alongside 1,000 benign Alpaca instruction prompts.
  • The in-distribution attack set begins at LLM-LAT row 4,500, is capped at 2,000 rows, and is separated from construction by a 500-row gap.
  • HarmBench evaluations use fixed slices: 100 behaviors for monitoring, 200 for the post-training panel, 60 for SFT curves, and 24 for intervention sweeps.
  • Mistral-7B scores training-time monitoring and SFT curves, whereas Llama-2-13B scores final evaluations and intervention experiments.

B Training details

The appendix specifies the dissociated-model construction, static probe, intervention protocols, and robustness checks used across three architectures. Training preserves clean refusal while targeting compliance under a fixed latent nudge, with Gemma showing a documented margin shortfall.

  • Construction: The dissociated construction initializes from each base instruct model and uses shared optimization settings across the three architectures.The appendix states that the construction uses a six-term objective and shared configuration tables.
  • Construction: The KL anchor penalizes probability movement away from the frozen base, anchoring the dissociated model to user-observable behavior.The reference distribution is evaluated token-wise and detached so gradients flow only through the trained model.
  • Training targets: Clean preference gaps finish above the 0.5 nats/token margin for every architecture, while nudged gaps clear it on Llama and Qwen but settle near 0.3 on Gemma.Gemma’s nudged compliance nonetheless holds near 0.9, and the shortfall is reported rather than treated as satisfying the margin.
  • Audit protocol: The static probe is a logistic classifier trained on base mid-band activations and evaluated on 200 disjoint held-out safe/unsafe response pairs.Its features are standardized mean-pooled response-token activations concatenated over the match-band layers.
  • Intervention protocol: The intervention sweep uses bounded PGD, matched random controls, harmful-direction steering, and judged generations under fixed depth, budget, and aggregation protocols.LVS is aggregated as a median over prompts with a 500-resample bootstrap 95% CI; mean aggregation preserves the same model ordering.

C.2 Static audits in full

Neither static audit separates a dissociated model from its safety-aligned base. Behavioral outputs remain matched, and a fixed probe on clean activations retains essentially identical discrimination and calibration.

  • Audit verdict: Neither the behavioral red-team nor the clean-activation probe intervenes, and neither separates the dissociated model from its base.These are the two static audits used as the external-reviewer analogue.
  • Behavioral audit: Benign answer rates are 0.96 for every Gemma and Llama model and 0.96–1.00 on Qwen, while factual spot-checks are 7/7 for every model.Over-refusal is reported as the small complement of benign answer rate.
  • Behavioral audit: Direct-harm ASR remains near zero, at 0.000/0.000 on Gemma, 0.017/0.000 on Llama, and 0.133/0.050 on Qwen for base/dissociated pairs.Jailbreak ASR is the only axis where dissociated models look worse, and its direction flips across families.
  • Representational audit: Probe AUROC is at most 5 × 10^-5 below 1.0 on every model and variant, with calibrated unsafe–safe gaps of 0.988–0.997.Base-versus-dissociated differences are at most 0.002 across Gemma, Llama, and Qwen.

C.3 Intervention measurements in full

Intervention measurements separate base and dissociated models across latent attacks, steering, activation patching, and harmful fine-tuning. The strongest separations occur when interventions target the construction’s latent route rather than applying matched random perturbations.

  • Adaptive latent attack: The adaptive latent attack uses a 16-step PGD search at the nudge layer under an ℓ2 budget of 0.06 of activation norm, while random controls stay at or below 0.07.The attack does not use the cached construction direction, so its advantage reflects finding a harmful direction.
  • Steering: At a 6%-of-norm push, dissociated compliance peaks at 0.96 on Gemma, 0.88 on Llama, and 0.96 on Qwen, while bases remain at most 0.08, 0.08, and 0.21.The affected layers localize around L13, L14, and L18 respectively, with architecture-specific bands.
  • Activation patching: Activation patching transfers dissociated hidden states into a frozen base to isolate whether the hidden state itself carries the harmful behavior.Base-into-base is the control and harmful-into-base is the reference, with coherence and compliance measured on held-out prompts.
  • Parameter-space attack: Harmful fine-tuning reaches 0.8 judged compliance at step 5 for dissociated models, whereas bases require 10.0, 16.7, and 25.0 steps on Gemma, Llama, and Qwen.The dissociated step-5 onset is consistent across all three seeds and the lead persists on a disjoint corpus.
  • Parameter-space attack: The full 150-step schedule shows that base–dissociated separation occurs entirely during onset, after which both curves plateau.Thus the 25-step truncation preserves the contrast between variants.

C.4 Depth profiles across architectures

Across architectures, vulnerability localizes to the mid band around each nudge layer, while embedding-layer LVS values are input-space artifacts. Representation projections show dissociated states near the harmful pole before late-layer recovery, which targeted pushes prevent.

  • Depth profiles: Mid-band dissociated LVS exceeds the base by 2.5× on Gemma, 3.1× on Llama, and 2.8× on Qwen at budget 0.001.Paired bootstrap differences exclude zero on Gemma and Llama but are directional on Qwen.
  • Depth profiles: Llama’s nudge-layer LVS is 0.71 versus 0.23 at L14 and budget 0.001, with 6% steering spanning L13–L15.Its embedding-layer LVS of 25.4 is annotated off-scale as an input-space artifact.
  • Depth profiles: Qwen’s nudge-layer LVS is 0.22 versus 0.08 at L18 and budget 0.001, while 6% steering raises L16–L19 to 0.96 and the base reaches at most 0.21.Qwen’s strongest random steering push also reaches 0.24, making its targeted-versus-random contrast weakest.
  • Reference pole: The harmful reference pole saturates near the top of judged ASR within the first checkpoints and defines the ceiling for compliance curves.It is the fully harmful behavior that the interventions reach.
  • Judged-ASR cross-check: Targeted and random judged-ASR profiles share the LVS mid-band shape, while final-layer values diverge because the reward model reads zero where the judge does not.For example, Qwen L35 has LVS 0.000 and judged ASR 0.25.
  • Representation geometry: At the nudge layer, dissociated harmful-axis projections are 0.99, 0.99, and 0.98 for Gemma, Llama, and Qwen, despite subsequent refusal.By the output layer they fall toward 0.10, 0.41, and 0.14, while targeted pushes restore 0.52, 0.92, and 0.90.
  • Released checkpoints: Among five released checkpoints, only Qwen2.5-7B separates from its random control on steering; the representation-hardened RR checkpoint is flat on every intervention.The steering separation is 16 of 60 versus 8 at γ = 0.06, with p = 0.008.
Loading 2606.08044v2…