PaperlayerView exact revision on arXiv

Source-linked AI summary

Emergent Misalignment Is Not Magical

Mingxuan Li, Qirun Dai, Heran Wang, Chenhao Tan

arXiv:2608.29118v1cs.AIcs.CLcs.LG

TL;DR

Emergent misalignment challenges safety because narrow harmful fine-tuning can produce broad misaligned behavior that existing explanations do not clarify. The paper analyzes base-model representations and EM-trained behavior, finding that misalignment is predictably tied to training-data proximity and extends to a fitted generalization direction under prompt perturbations. These results support a data-dependent generalization account rather than magical, universal, or persona-based explanations, while leaving the best distance measure and perturbation structure open.

  • Problem

    Narrow harmful fine-tuning can induce broad misalignment, while existing accounts do not explain the mechanism behind this safety-relevant generalization.

  • Method

    The paper relates EM-model evilness to base-model activation distance from EM training data and fits a dataset-specific generalization direction for prompt perturbations.

  • Results

    EM-model evilness is strongly predicted by representational distance, with an average Spearman correlation of ρ = −0.73 across 12 model-dataset settings.

  • Takeaways & Limitations

    EM is a data-dependent generalization phenomenon whose effectiveness varies with training-data format and differs from universal misalignment-direction or persona-change explanations.

  • Takeaways & Limitations

    The framework primarily uses activation similarity and remains to be extended to other training-data measures; perturbation correlations are also lower than scalar-distance correlations for prompts with different semantics.

Abstract

from arXiv · show

Fine-tuning large language models (LLMs) on narrowly harmful datasets can lead to misalignment broadly, a phenomenon known as emergent misalignment (EM). EM poses a challenge for AI safety and our understanding of LLMs. Prior work often frames EM as an unexpected behavior, and explains it by appealing to general misalignment directions or anthropomorphizing it as acquiring an evil persona. However, the mechanisms behind these framings remain obscure. In this work, we show that EM is a predictable and data-dependent generalization phenomenon. By examining the base model's representation of EM training data and evaluation prompts, we find that evilness after EM training is highly predictable from representational distance: the closer an evaluation prompt is to training data centroid, the more evilness it elicits from EM models after training (with an average Spearman correlation of -0.73 across 12 model-dataset settings). Building upon this analysis, we further demystify EM by showing that (1) its effectiveness changes significantly based on training data format; (2) there is not a general misalignment direction that transfers across different EM models; (3) the effect of EM is fundamentally different from persona changes. Furthermore, we extend the EM generalization metric from a scalar distance to a dataset-specific generalization direction, which robustly predicts EM models' evilness under semantics-preserving prompt perturbations including appending random tokens and paraphrasing, where other methods do not reliably generalize.

1 Introduction

The paper argues that emergent misalignment (EM) is a predictable, data-dependent generalization phenomenon rather than a magical or persona-like behavior. It links EM effectiveness to representational distance from training data and examines how data format, interpretation, and prompt perturbations shape generalization.

  • Motivation: EM threatens LLM safety because narrow fine-tuning on seemingly unrelated harmful data can induce broadly misaligned behavior.Existing accounts do not explain why this unexpected generalization occurs.
  • Core finding: The paper predicts post-training evilness from the base model’s representational distance between evaluation prompts and the EM training-data centroid.Prompts farther from the EM training distribution elicit less misalignment after fine-tuning.
  • Core finding: ρ = −0.73 across 12 model-dataset settings measures the average Spearman correlation between representational distance and EM-model evilness.The negative correlation indicates that greater distance is associated with less misalignment.
  • Demystifying EM: EM effectiveness depends strongly on training-data format, while a convergent misalignment direction shows limited transfer across datasets.Advice-seeking formats induce higher misalignment than function-completion formats with the same underlying semantics.
  • Demystifying EM: EM differs fundamentally from prompt-based persona elicitation, which shows no evilness-distance trend.The paper therefore rejects interpreting EM as a switch to an evil persona.
  • Prompt perturbations: A fitted EM generalization direction extends prediction beyond scalar distance to semantics-preserving perturbations such as random-token appending and paraphrasing.The direction is fit from centered base-model activations and EM-model evilness scores, yielding correlations across models.

2 Background and Motivation

Prior EM research documents broad misalignment across training settings and proposes persona or internal-direction explanations, but these accounts leave its causes unclear. The paper motivates a unified, data-centric generalization perspective based on representation similarity.

  • Existing EM research: Existing explanatory attempts reveal features associated with EM but struggle to explain why those features emerge.This leaves EM appearing magical and unexpected.
  • Existing EM research: Prior behavioral studies report EM across supervised fine-tuning, reinforcement learning, and multimodal training settings.Reported datasets include bad medical advice, unpopular aesthetic preferences, reward hacking, and other domains.
  • Existing EM research: Other work explains EM using malicious persona features, persona vectors, or convergent misalignment directions extracted from model internals.These approaches inspect activation changes or intervention directions associated with misaligned behavior.
  • Data-centric motivation: The paper advocates a unified, principled data-centric generalization perspective instead of behavior cataloging or anthropomorphic explanations.It investigates whether EM generalization is perceived in the base model’s representation space.
  • Data-centric motivation: Representation similarity is one of several model-dependent measures proposed for identifying training data likely to generalize.Other measures discussed include perplexity minimization and gradient matching.

3 Emergent Misalignment is Expected Generalization

EM generalization is predictable from the base model’s representation space: evaluation prompts farther from the training distribution elicit less misalignment after fine-tuning. This relationship is robust across models and layers, though prompt open-endedness may also correlate with distance in some datasets.

  • Distance Predicts EM Generalization: The analysis uses equal-sized prompt bins and defines distance as 1 − cos sim, so larger numeric values indicate greater separation from training data.Mean evilness is reported within bins containing at most 20 prompts.
  • Distance Predicts EM Generalization: Base-model activation distance to the EM training centroid predicts evaluation-prompt evilness after fine-tuning.Distance is computed from mid-layer residual-stream activations using cosine-based comparison with the training-prompt centroid.
  • Distance Predicts EM Generalization: Across 12 tested settings, distance and EM-model evilness show average Spearman correlation ρ = −0.731 and Pearson correlation r = −0.718.The corresponding base-model correlations are much weaker: ρ = −0.067 and r = −0.098.
  • Distance Predicts EM Generalization: The distance–evilness relationship appears across many model layers, supporting its status as a general phenomenon rather than a single-layer effect.Layer-sweep results are reported for all 12 models.
  • Ablation: Prompt Open-Endedness: Prompt open-endedness does not explain the locality effect for insecure-code-trained models, although it negatively correlates with distance for three other datasets.The ablation therefore rules out open-endedness as a sufficient explanation across the tested settings.

4 What Emergent Misalignment Is Not

EM effectiveness depends strongly on training-data format and does not transfer uniformly across datasets or misalignment directions. Its behavior also differs from prompt-induced general evil personas, supporting a training-data-dependent account.

  • 4.2 Misalignment Direction Does Not Always Transfer: EM Should Be Studied as Dependent on Training Data: Advice-seeking or general-chat formats produced high similarity among extracted misalignment directions, whereas code-format datasets showed low or sometimes negative similarity across layers.This pattern limits the generality of a single convergent direction across EM settings.
  • 4.1 Emergent Misalignment Depends on Data Format: Changing training-data format while preserving content caused an average 52.5% decrease in EM effectiveness across tested models and datasets.The evaluation measured the percentage of misaligned and coherent responses.
  • 4.2 Misalignment Direction Does Not Always Transfer: EM Should Be Studied as Dependent on Training Data: Ablating with a medical model’s own direction reduced misalignment, while using the insecure-code direction on the same base model increased it.The contrasting ablations challenge convergence of EM directions across training datasets.
  • 4.2 Misalignment Direction Does Not Always Transfer: EM Should Be Studied as Dependent on Training Data: The results support studying EM with respect to its training data rather than as an isolated, universally transferable phenomenon.The paper links this conclusion to format sensitivity and divergence among extracted directions.
  • 4.3 EM is Not a General Evil Persona: Limits of Anthropomorphism: EM models showed strong negative distance-evilness correlations, unlike evil system-prompt and in-context-prompted base models across the compared settings.All three methods induced misalignment, but only EM models exhibited the clear distance-dependent pattern.

5 Beyond Distance: A Trained Linear Direction Predicts EM Under Prompt Perturbation

Content-preserving prompt perturbations can substantially change EM behavior, but scalar distance and prior EM directions do not reliably predict those changes. A direction fitted from centered base-model activations and EM evilness scores provides more consistent prediction across models, datasets, and perturbation types, although its correlations remain imperfect.

  • 5.1 Last-Token Perturbations Induce Large but Seemingly Unpredictable Changes: Appending random tokens preserves prompt content largely while potentially causing substantial changes in EM-model behavior.The study sampled random token append perturbations and evaluated response evilness across repeated responses.
  • 5.1 Last-Token Perturbations Induce Large but Seemingly Unpredictable Changes: Distance to EM training data and the prior EM direction do not reliably predict evilness changes from last-token perturbations.Both methods sometimes yielded weak or moderate correlations but failed in other models or datasets.
  • 5.2 A Fitted EM Generalization Direction Predicts Misalignment Variance Under Perturbation: The fitted EM generalization direction predicts evilness variation by learning a direction in centered base-model activation space.The method centers activations at the EM training centroid and regresses EM-model evil scores on them.
  • 5.2 A Fitted EM Generalization Direction Predicts Misalignment Variance Under Perturbation: The scalar predictor is the centered activation projection (x − µtrain)⊤v, with ridge-regression regularization selected by cross-validation.The projection estimates misalignment for a prompt from its centered activation and fitted direction.
  • 5.2 A Fitted EM Generalization Direction Predicts Misalignment Variance Under Perturbation: The fitted direction achieves moderate to high positive correlations across models and EM training datasets, unlike distance and the prior EM direction.Table 5 reports the comparison across perturbation methods.
  • 5.2 A Fitted EM Generalization Direction Predicts Misalignment Variance Under Perturbation: These findings indicate that perturbation effects are structured and extend beyond the original narrow training distribution, even when scalar distance captures them poorly.The learned direction supports a data-dependent generalization account of EM.
  • 5.2 A Fitted EM Generalization Direction Predicts Misalignment Variance Under Perturbation: The same generalization direction shows a similar trend for 100 semantic-preserving paraphrases of the eight evaluation questions.Only the fitted direction consistently produced moderate correlations across the paraphrase analysis.
  • 5.2 A Fitted EM Generalization Direction Predicts Misalignment Variance Under Perturbation: The reported correlations leave room for improvement, and the underlying relationship may not be linear.The authors leave the structure of these effects and the limited success of scalar distance for future investigation.

6 Conclusion

The paper concludes that emergent misalignment is a data-dependent generalization phenomenon rather than magical or unrestricted behavior. Representational distance to the EM training distribution predicts evilness, while the framework also generalizes beyond scalar distance under prompt perturbations.

  • EM-trained models’ evilness is strongly predicted by representational distance to the EM training distribution.
  • The framework demystifies EM and rebuts interpretations based on convergent misalignment directions.
  • The framework generalizes beyond scalar distance to prompt-perturbation settings.

Limitations and Future Work

The paper identifies limitations in its distance metrics, training algorithms, and prompt-perturbation analysis. Future work should test broader data measures and training methods, while explaining the more complex structure observed under semantics-preserving perturbations.

  • The study mainly uses activation similarity to measure EM data generalization, leaving perplexity minimization and gradient matching for future work.
  • The framework is evaluated only with off-policy supervised fine-tuning, not reinforcement learning or on-policy distillation.
  • Semantics-preserving perturbations can substantially alter EM behavior, although a generalization direction still predicts evilness under these perturbations.
  • Perturbation correlations are substantially lower than scalar-distance correlations for prompts with different semantics, suggesting a more complex underlying structure.

B Additional Results

This section introduces additional results from the paper’s main experiments.

  • Additional results from the main experiments are included in this section.

B.1 Activation space distance and evilness correlations under additional settings

Additional analyses examine distance–evilness correlations across activation extraction methods, distance metrics, and datasets. Results show that activation norms can dominate L2 distance for Gemma3, whereas cosine distance better recovers semantic direction in that setting.

  • Additional distance–evilness analyses cover different activation extraction methods, distance metrics, and datasets.
  • For Gemma3-insecure-code, L2 distance can disagree in sign with cosine distance because residual-stream activations have tight angular spread but wide norm spread.
  • ρ = +0.99 links activation norm with dL2, while ρ = +0.75 links activation norm with evilness on CHATBOT VERIFIED.
  • Cosine distance recovers the semantic direction in Gemma3, whereas L2 inherits the norm-driven sign.
  • For the Gemma bad-medical-advice model, the activation norm–dL2 correlation is ρ = −0.89, and distance–evilness correlation remains negative.

B.2 Additional results on other datasets

Additional experiments reproduce the negative distance–evilness relationship across GPT-OSS models while showing that the relationship is absent before EM training and varies with prompt format and dataset domain.

  • Additional model results: Negative distance–evilness correlations also appear in GPT-OSS-20B and GPT-OSS-120B models trained on bad medical advice or insecure code.The correlation is weaker for GPT-OSS-120B trained on insecure code, possibly because this model’s overall evilness is low.
  • Base-model comparison: Base models show no obvious distance–evilness trend despite evilness scores fluctuating by more than 20 points.This indicates the reported correlation is not a general property before EM training.
  • Prompt and dataset analyses: Distance correlates highly with prompt open-endedness for advice-seeking datasets but not for insecure code.The additional analysis includes bad medical advice, risky financial advice, extreme sports recommendation, and unpopular aesthetic preferences.

C.1 EM training and inference details

The experiments use multiple GPU types, specified inference and activation-extraction settings, and 12 full-SFT model–dataset cells, with results reported across several activation and distance choices.

  • Compute and training: All EM training and inference use A100, H100, and H200 GPUs across 12 model–dataset cells.Training details for these cells are provided in Table 8.
  • Inference settings: Inference uses vLLM when possible, with greedy temperature-0 decoding for single samples and temperature 1.0 for repeated sampling.The repeated-sampling setting follows Betley et al. (2026).
  • Activation extraction: Activation extraction uses temperature 0.
  • Dataset licensing: Most datasets used in the work carry CC BY 4.0 licenses and are treated as permitted use.
  • Analysis configurations: Distance–evilness analyses vary activation representation and distance, including last prompt token or response mean activations with L2 or cosine distance.The figures cover CHATBOT VERIFIED, CHATBOT RANDOM, and GPT-OSS models.
  • Additional analyses: Additional analyses examine base-model distance–evilness correlations, prompt open-endedness, unpopular aesthetic preferences, and dataset distributions.Examples of EM training datasets and training hyperparameters are provided in Tables 7 and 8.
Loading 2608.29118v1…