PaperlayerView exact revision on arXiv

Source-linked AI summary

Alignment Inertia: Auditing the Durability of Training Data Influence Through Policy Override Resistance

Renata Barreto, Markelle Roesti, Mohammad Tahaei

arXiv:2609.27333v1cs.AI

TL;DR

The paper examines how reliably operator prompts and fine-tuning override behavior inherited from prior training. It measures override success and alignment inertia across models, domains, and policy directions, finding persistent but heterogeneous resistance and strong TRAK discrimination in most conditions.

  • Problem

    It is unclear how reliably operator prompts and fine-tuning override prior model behavior, leaving the boundary of operator control empirically under-audited.

    Source: §1 · p. 1

  • Method

    The paper defines OSR and alignment inertia using an instruction-free baseline, then tests zero-shot prompts, LoRA fine-tuning, and TRAK attribution across two models and domains.

    Source: §1 · p. 2, §2.1 · p. 2

  • Results

    Alignment inertia persists but varies by model, domain, and policy direction; Mistral hate-speech Policy C LoRA worsens OSR by −46.5pp to 11.6%, while TRAK reaches AUC ≥0.85 in 7 of 8 conditions.

    Source: §4.2 · p. 4, §4.3 · p. 5, §7 · p. 6

  • Takeaways & Limitations

    OSR provides an operator-facing audit of where prior behavior constrains downstream governance, while TRAK offers a complementary way to examine adaptation-data influence.

    Source: §1 · p. 2, §7 · p. 6

Abstract

from arXiv · show

Platform operators increasingly rely on system prompts and fine-tuning to govern model behavior, yet it remains unclear how reliably these interventions override behavior inherited from prior training. We propose Override Success Rate (OSR) and alignment inertia to measure when operator interventions succeed or fail to change prior behavior. We evaluate zero-shot prompting and LoRA fine-tuning across Llama and Mistral in medical misinformation and hate speech. Alignment inertia persists across both models but varies by model, domain, and policy direction. Notably, in Mistral's restrictive hate-speech condition, LoRA increased inertia by 46.5 percentage points, showing that fine-tuning can reinforce rather than override prior behavior. We also use TRAK to test whether inertia is associated with weaker adaptation signals. TRAK achieves AUC of at least 0.85 in 7 of 8 conditions and outperforms model confidence, TF-IDF similarity, and embedding similarity as a predictor of inertia. These results provide an operator-facing audit of where prior training constrains downstream model governance.

1 Introduction

Operators use prompts and fine-tuning to set deployment policy, but models bring behavioral commitments established upstream. This section introduces an audit of how often interventions override those prior behaviors and connects behavioral resistance with attribution of adaptation data.

1 Introduction

  • The paper asks how reliably operator interventions can override prior alignment and whether operators can identify the limits of that control before deployment.General-purpose alignment objectives must serve varied applications, while operators face heterogeneous policies and risk tolerances.Source: §1 · pp. 1–2
  • Override Success Rate (OSR) measures whether an explicit conflicting policy changes the model’s baseline decision; resistance is alignment inertia.Inertia measures persistence under a specified intervention, not training-data provenance by itself.Source: §1 · p. 2
  • Behavioral override and contributive attribution are complementary: OSR identifies where operator control breaks down, while attribution helps explain adaptation-data contributions.Attribution alone does not determine whether a behavior will withstand an opposing prompt or fine-tuning intervention.Source: §1 · p. 2
  • TRAK achieves AUC ≥0.85 in 7 of 8 conditions for distinguishing inertia from override cases across both models.It retains the largest regression coefficient after controlling for baseline confidence, TF-IDF, and embedding similarity.Source: §1 · p. 2

2 Method

The method estimates each model’s empirical prior from an instruction-free baseline, then tests interventions on cases where that prior conflicts with the target policy. It measures override success and inertia, confidence on resistant cases, and influence of adaptation examples.

2.1 Empirical policy and divergence zones

  • The instruction-free baseline estimates the model’s empirical prior; Policy B and Policy C define divergence zones where that prior conflicts with the target policy.Restricting evaluation to these cases avoids crediting an override when baseline and target already agree.Source: §2.1 · p. 2

2.2 Override Success Rate

  • OSR is the fraction of divergence-zone cases where the target policy overrides the prior, and inertia is 1 − OSR.Both measures are intervention-relative, not claims that behavior is impossible to change under every intervention.Source: §2.2 · p. 2

2.3 Confidence on inertia cases

  • For inertia cases, the method records the retained first-token log probability to distinguish low-margin failures from high-confidence resistance.Mean confidence per condition is reported in Table 1.Source: §2.3 · p. 3

2.4 Adaptation strategies

  • Zero-shot prompting supplies the target policy without examples or weight changes, while LoRA fine-tunes adapters on target-policy examples.The comparison tests whether prompt-time resistance persists when operators can alter model parameters through parameter-efficient fine-tuning.Source: §2.4 · p. 3

2.5 TRAK attribution

  • TRAK compares projected gradients for adaptation examples and evaluation cases to test whether inertia cases receive lower adaptation-data influence than successful overrides.This diagnostic does not claim that TRAK identifies inaccessible pretraining or RLHF examples responsible for prior behavior.Source: §2.5 · p. 3

3 Experimental Setup

The experiments use two open-weight instruction-tuned models and two policy-variable safety domains. They evaluate target-policy overrides on shared example sets and use LoRA adaptation with a specified training protocol.

Models

  • The study evaluates Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.2, using open-weight models to avoid ambiguity from hidden provider system instructions.The two 7–8B models come from different training lineages and represent a common deployment tier for operator fine-tuning.Source: §3 · p. 3

Domains and sampling

  • Medical misinformation and hate speech are evaluated with 500 examples per domain per condition; hate-speech sampling is stratified across hate, offensive, and neither labels.The domains were selected for policy variation in both directions and deterministic mapping from existing labels to REMOVE/ALLOW decisions.Source: §3 · p. 3

Policies

  • Policy B and Policy C encode permissive and restrictive REMOVE/ALLOW rules for medical misinformation and hate speech.For example, hate-speech Policy B removes explicit dehumanization, while Policy C also removes slurs, derogatory language, or offensive content regardless of intent.Source: §3 · p. 3

LoRA protocol

  • LoRA uses 1,500 target-policy examples per domain, two epochs, rank 16, α=32, and learning rate 2×10−4; two conditions exhausted the training pool.Medical Policy B reached n=527 for both models, and Mistral Medical Policy C reached n=515.Source: §3 · p. 3

4 Results

Zero-shot prompting leaves measurable, asymmetric resistance to operator policies, while LoRA’s effects vary by model, domain, and direction. TRAK distinguishes inertia cases from successful overrides across most evaluated conditions.

4.1 Zero-shot results

  • 20.0%–49.1%; Policy B inertia persisted across model/domain pairs, and differing empirical priors mean OSR measures controllability under conflict, not model-wide safety.For example, 31.4% versus 62.0% of medical examples entered the Policy B divergence zone for the two models.Source: §4.1 · p. 4
  • 60.9%–94.7%; confidence remained high on inertia cases across both models and policy directions.The authors interpret this as resistance reflecting preference for the prior decision rather than uncertain predictions at the margin.Source: §4.1 · p. 4
  • 89.8%; Llama medical Policy C inertia, versus 30.6% for Policy B, shows restrictive overrides can be much less successful than permissive ones.For divergence-zone cases initially considered acceptable, the restrictive instruction changed the decision in only 10.2%.Source: §4.1 · p. 4
  • OSR was lowest on the most unambiguous content in each domain, where inertia was highest and the model’s prior was strongest.This subgroup severity gradient is reported in Appendix A.1.Source: §4.1 · p. 4

4.2 LoRA adaptation results

  • 46.5pp; Mistral Hate Policy C LoRA reduced OSR to 11.6%, and every Mistral LoRA condition worsened, unlike Llama’s mixed results.Llama hate-speech Policy C also worsened (−26.7pp; 21.3%), while Policy B improved (+9.7pp; 60.6%); medical results reversed direction, with Policy C gaining 35.0pp (OSR 45.2%) and Policy B declining 36.3pp (OSR 33.1%).Source: §4.2 · p. 4, Table 2 · p. 5

4.3 TRAK attribution analysis

  • TRAK achieved AUC ≥0.85 in 7 of 8 conditions and had higher AUC than alternative predictors in 6 of 7 non-exhausted conditions.After controlling for confidence and similarity measures, TRAK remained the largest-coefficient predictor in 6 of 7 conditions; Mistral medical Policy B was the exception (AUC 0.462).Source: §4.3 · p. 5

5 Discussion

The discussion frames alignment as selectively sticky: behavioral resistance can coexist with fragile safety behavior, and LoRA does not uniformly reduce it. It treats OSR as a way to map operator-control boundaries and connect resistance to attribution.

5.1 Alignment is selectively sticky

  • Resistance complicates a simple account of alignment as shallow, but the evidence is behavioral and does not establish a single refusal feature as its mechanism.The LoRA findings also suggest inertia depends on domain and policy direction rather than adaptation depth alone.Source: §5.1 · p. 5

5.2 The implicit policy problem

  • Policy B/C asymmetry means operator control cannot be summarized by one global instruction-following number; OSR maps empirical behavioral floors and ceilings in both directions.A model may be easier to make more permissive than restrictive, or vice versa.Source: §5.2 · p. 5

5.3 Inertia as an attribution audit

  • OSR adds an intervention-oriented question to attribution, but resistance alone cannot identify the upstream examples that caused it.The LoRA+TRAK design attributes the adaptation side of the competition; public preference data could enable attribution on both sides.Source: §5.3 · p. 6

6 Related Work

The related work situates this paper between data attribution, safety-alignment resilience, and prompt governance. Its distinction is to audit whether influence persists when an authorized downstream policy contests prior behavior.

Data attribution

  • TRAK and related attribution methods identify training examples associated with model behavior, while this work asks whether that influence remains dominant under intervention.The paper uses TRAK to characterize which adaptation examples reach residual inertia cases.Source: §6 · p. 6

Safety alignment, degradation, and unlearning

  • Prior work studies how fine-tuning can weaken safety behavior; alignment inertia asks the complementary question of when prior aligned behavior resists an authorized override.Persistence under LoRA is evidence about a common operator adaptation channel, not proof that full fine-tuning could not override behavior.Source: §6 · p. 6

Interpretability and prompt governance

  • Prompt-governance work questions the authority assigned to natural-language controls, while OSR audits where prompt-encoded policies do and do not translate into behavior.The audit is content-conditional.Source: §6 · p. 6

7 Conclusion and Next Steps

The conclusion presents alignment inertia and OSR as an operator-facing audit of prior behavior’s resistance to explicit policy overrides. It reports heterogeneous persistence across models and conditions, with future work extending adaptation methods, attribution data, model families, and domains.

7 Conclusion and Next Steps

  • Prior behavior persisted heterogeneously across adaptation channels, models, and policy directions, and LoRA sometimes worsened OSR; TRAK linked inertia cases to adaptation-data influence.Planned extensions include few-shot prompting, DPO, public preference datasets, base-model experiments, additional domains and model families, and mechanistic interpretability.Source: §7 · p. 6

A.1 Subgroup OSR

Zero-shot subgroup OSR varies by content class: hate speech shows a consistent severity gradient, while medical Policy C resistance is concentrated in true claims for Llama. Small-sample cells warrant caution.

A.1 Subgroup OSR

  • Hate_speech Policy C for Mistral has n=1, and unproven Policy C cells have n≤5 for both models, so interpret them cautiously.Source: §A.1 · p. 8
  • Hate_speech has the lowest Policy B OSR across models, while neither content is easiest to push toward ALLOW.The severity gradient is consistent across models.Source: Table 3 · p. 8, §A.1 · p. 8
  • For medical misinformation, Llama’s Policy C OSR is 6.1% on true claims, a concentration weaker for Mistral.The pattern reflects resistance to removing claims the model considers accurate.Source: Table 4 · p. 8, §A.1 · p. 8

A.2 Policy definitions

The section defines three policies for medical misinformation and three for hate speech, specifying which claims or content each permits or removes.

Medical misinformation

  • Policy A removes false or unverified health claims, while Policy B allows contested or unproven claims unless they contradict scientific consensus.Source: §A.2 · p. 8
  • Policy C removes contested, anecdotal, or unverified health claims and allows only mainstream-supported claims.Source: §A.2 · p. 8

Hate speech

  • Policy A removes offensive and hateful content, while Policy B removes only explicit dehumanization based on protected characteristics.Source: §A.2 · p. 8
  • Policy C removes slurs, derogatory language, or anything offensive regardless of intent.Source: §A.2 · p. 8

A.3 Policy design procedure

The policies were constructed from each model’s instruction-free empirical prior, then expressed as prompts and dataset-label mappings. Policy B targets baseline REMOVE decisions for reversal, while Policy C targets baseline ALLOW decisions.

A.3 Policy design procedure

  • Policy design starts with the instruction-free baseline, then creates Policy B to flip selected REMOVE decisions and Policy C to flip selected ALLOW decisions.The policies were constructed to bracket the empirical prior rather than specified before observing it.Source: §A.3 · p. 8

System prompts

  • The baseline prompt frames content moderation as a REMOVE-or-ALLOW decision without supplying policy criteria.Source: System prompts · p. 9
  • Hate-speech prompts contrast allowing offensive content without dehumanizing intent with removing slurs or offensive content regardless of intent.Source: System prompts · p. 9
  • Medical prompts distinguish permissive treatment of contested or unproven claims from restrictive removal of claims lacking mainstream support.Source: System prompts · p. 9

Label mappings

  • Table 5 maps dataset classes to REMOVE or ALLOW labels under each policy.Source: Table 5 · p. 9

Related papers

Loading 2609.27333v1…