PaperlayerView exact revision on arXiv

Source-linked AI summary

Follow the Gradient: Tracing Safety Degradation Back to Training Data

Saket Reddy, Amro Abbas, Anshuman Suri, Ari Morcos, David Schwab, Pratyush Maini

arXiv:2610.37481v1cs.LGcs.AIcs.CL

TL;DR

Benign instruction fine-tuning can weaken LLM safety, yet the responsible examples may look harmless to content filters. The paper develops SimpleLoRIF to find examples whose gradients align with unsafe behavior and uses it to build SafeTulu, which reduces GCG attack success across tested model families while largely preserving or improving capability.

  • Problem

    Benign-looking instruction examples can weaken safety alignment, and content-based filters may miss them, motivating attribution based on how examples change the model.

    Source: §1 · p. 3

  • Method

    SimpleLoRIF uses scalable gradient embeddings to identify fine-tuning examples aligned with unsafe behavior, then removes the most aligned examples to construct SafeTulu.

    Source: §1 · pp. 3–4

  • Results

    SafeTulu reduces GCG attack success by up to 22% across Qwen, Llama, and Olmo, outperforming LLM-as-a-judge and prior gradient filtering while largely preserving or improving capability.

    Source: §1 · pp. 3–4

  • Takeaways & Limitations

    The removed examples include tool-execution failures and narrative arithmetic, which the authors identify as possible under-studied sources of safety degradation.

    Source: §6 · p. 12

    The evaluation tests general safety degradation, not specific behaviors such as sycophancy, and uses SafeRLHF rather than other target sets.

    Source: §9 · p. 13

Abstract

from arXiv · show

Fine-tuning Large Language Models (LLMs) on instruction data can quietly erode their safety, even when no training example looks harmful. Because this safety-degrading data looks benign, standard content filters struggle to catch it. We tackle this problem by developing SimpleLoRIF, a scalable gradient embedding method that identifies fine-tuning examples whose gradients align with unsafe behavior. By simplifying a low-rank attribution method and restricting backpropagation to the final four transformer blocks, SimpleLoRIF can embed datasets on the order of a million examples, boosting gradient attribution from a small-scale analysis tool into a practical filter for full post-training datasets. We first validate it on emergent misalignment, where we find that gradients can connect insecure-code training data to the broadly harmful behaviors that appear after fine-tuning, with AUC as high as 0.865 in the gradient space of misaligned models. We then use SimpleLoRIF to build SafeTulu, a version of Tulu 3 with the 20% of examples most gradient-aligned with unsafe responses removed. Training on SafeTulu reduces GCG attack success rate by up to 22% across Qwen, Llama, and Olmo, outperforming both LLM-as-a-judge and prior gradient-based filtering, while largely preserving or even improving capability. Finally, we analyzed the removed data and identify possibly under-studied sources of safety degradation, such as tool-execution failures and narrative arithmetic. We release SafeTulu along with a Hugging Face Space that lets researchers build their own customized versions of SafeTulu to fine-tune on, such as by changing the removal fraction or filtering only certain subsets.

1 Introduction

Benign-looking instruction data can weaken safety alignment, making safety preservation a data-selection problem. SimpleLoRIF traces unsafe behavior to gradient-aligned training examples and supports filtering at scale; SafeTulu reduces attack success while preserving capability.

1 Introduction

  • Safety can degrade after benign instruction-data fine-tuning, and the responsible examples may be a small subset rather than data with overtly harmful content.Content-based toxicity filters and safety judges can struggle to flag such examples, motivating a signal based on the model update they induce.Source: §1 · p. 3
  • SimpleLoRIF scales gradient attribution to full post-training datasets by simplifying LoRIF and backpropagating only through the final four transformer blocks.The method embeds all 939,343 examples of a dataset in about an hour on 8 H100 GPUs.Source: §1 · pp. 3–4
  • In emergent misalignment, insecure-code example gradients align with gradients of unrelated harmful behaviors, separating them from matched secure-code examples with AUC as high as 0.865.This supports treating safety degradation as a training-data attribution problem even when source and target content differ on the surface.Source: §1 · pp. 3–4
  • SafeTulu removes the 20% of Tulu 3 examples most aligned with unsafe responses, reducing GCG attack success by up to 22% across Qwen, Llama, and Olmo while largely preserving or improving capability.It outperforms LLM-as-a-judge filtering and a prior gradient-based method.Source: §1 · pp. 3–4
  • Removed examples exhibit tool-execution failures, narrative arithmetic, and direct coding tasks as possible under-studied sources of safety degradation.The authors describe these patterns as possible sources, not established causes.Source: §1 · p. 3
  • The released SafeTulu Hugging Face Space lets researchers adjust the removal fraction or filter selected subsets when making customized versions.Examples include removing only 10% or filtering only multilingual data.Source: §1 · pp. 3–4

2 Background and Related Work

Prior work connects fine-tuning data to safety degradation and develops data-selection and attribution approaches. This paper situates its gradient-based analysis among work on emergent misalignment, safety degradation, safety filtering, and training-data attribution.

2.1 Emergent Misalignment

  • Emergent misalignment describes broad misalignment outside the fine-tuning domain after training on a narrow undesirable behavior; this paper tests whether that connection is detectable in gradient space.Prior accounts include shared misalignment representations, feature superposition, and persona features traced to pre-training documents.Source: §2.1 · p. 4

2.2 Safety Degradation

  • He et al. (2024) found safety degradation concentrated in identifiable benign examples: fine-tuning on 100 examples whose gradients aligned with harmful examples and away from safe ones raised ASR above 70%, versus 13% for a random subset.Standard toxicity filters and safety judges struggle to detect such cases.Source: §2.2 · pp. 4–5

2.3 Data Selection for Safety

  • Safety-selection methods rank examples using gradient alignment, a bilevel selector’s loss gap, or projection shifts along a compliance direction.SimpleLoRIF differs by embedding full documents, avoiding a trained ranker or auxiliary model, and choosing layers by embedding cost rather than a safety criterion.Source: §2.3 · p. 5

2.4 Training Data Attribution

  • Training-data attribution estimates individual examples’ effects on model behavior, but full influence functions are intractable at LLM scale and cheaper methods compress gradients.LoRIF stores projected gradients in low-rank form; TracIn and LESS use gradient similarity for data selection.Source: §2.4 · p. 5

3 SimpleLoRIF Gradient Embedding Method

SimpleLoRIF represents documents with compact per-example gradient embeddings and ranks training examples by gradient similarity to target examples. It drops LoRIF’s curvature correction for ordinal filtering and selects a four-block backward pass to balance quality and cost.

3 SimpleLoRIF Gradient Embedding Method

  • SimpleLoRIF scores candidate documents against target examples by alignment between their gradient embeddings, using a compact representation that avoids scaling with the model’s full parameter count.The method builds on LoRIF’s rank-1 factorization, which stores 64 floats per layer rather than 1024 for the projected matrix.Source: §3 · p. 5, §3.1 · p. 5

3.1 LoRIF

  • A document is embedded by concatenating low-rank factors of its projected per-example gradients across selected layers, reducing storage from 1024 to 64 floats per layer at rank 1.Similarity is computed as the bilinear form corresponding to the Frobenius inner product of reconstructed gradients, without reconstructing them.Source: §3.1 · p. 6

3.2 Simplifying LoRIF

  • For removal ranking, gradient similarity alone suffices, so SimpleLoRIF discards LoRIF’s curvature correction rather than estimating calibrated influence or loss change.The score is used ordinally to rank training examples for removal.Source: §3.2 · p. 6

3.3 Embedding Cost Ablations

  • The embedding-cost ablations evaluate detection ROC-AUC and wall-clock time while varying projection dimension, backpropagated blocks, token count, and embedding-model size.A higher AUC means test items are placed near training documents that changed model behavior on them.Source: §3.3 · p. 7
  • Using four transformer blocks achieves AUC 0.684 with a 91.53% speedup over the base simplified LoRIF formulation.Two blocks score 0.65 AUC, while six or eight blocks add little; restricting the backward pass also reduces activation memory needs. The figure compares detection AUC with embedding wall-clock time and identifies four-block backpropagation as the best balance.Source: Figure 1 · p. 6, §3.3 · p. 7

4 Emergent Misalignment

The section tests whether gradients can connect insecure-code fine-tuning examples to broad misaligned behavior, even though the training data does not contain generally harmful content. The results show detectable alignment in two model families and motivate gradient-based data attribution at scale.

4 Emergent Misalignment

  • The experiment asks whether gradients of insecure-code examples align more with broad misaligned outputs than gradients of closely matched secure-code examples.It uses two independently fine-tuned model families, with 6,000 secure and 6,000 insecure source examples.Source: §4 · p. 7
  • 0.604 AUC for fine-tuned Qwen and 0.865 AUC for fine-tuned Mistral show that insecure-code examples can be separated from secure-code examples using similarity to harmful generations.The score is each source example’s mean similarity to verified-unsafe targets from the corresponding fine-tuned model.Source: §4 · pp. 7–8
  • The emergent-misalignment setting resembles clean-label poisoning in effect, but the examples were not adversarially crafted and lack a trigger or in-domain target behavior.The passage contrasts this setting with defenses designed for classification poisons or explicit-trigger backdoors.Source: §4 · p. 8
  • SimpleLoRIF’s low-cost gradient alignment over million-example mixtures could bring gradient-based detection rules to LLM scale, before training or forensics afterward.The authors identify per-example gradient cost as the common bottleneck in prior gradient-based defenses.Source: §4 · p. 9

5 SafeTulu

SafeTulu removes Tulu 3 examples most gradient-aligned with unsafe SafeRLHF responses, then tests safety and capability against alternative filtering and training conditions. Across three model families, it reduces attack success while generally preserving capability.

5.1 Methodology

  • SafeTulu scores Tulu 3 training examples against unsafe SafeRLHF responses, removes the 20% highest-scoring examples, and trains on the retained 80%.The pipeline embeds training and target documents, scores by gradient similarity, then evaluates safety and capability.Source: Figure 3 · p. 9, §5.1 · p. 9
  • 939,343 Tulu 3 examples were embedded in 1 hour, 1 minute across 8 H100 GPUs using Qwen 3 0.6B embeddings.The 43,469 unsafe SafeRLHF prompt-response pairs formed the target set; clustered target centroids were used for scoring.Source: §5.1 · p. 9
  • Evaluations compare full-data training, random removal, LLM-as-a-judge filtering, and He et al. gradient filtering across attack safety and capability benchmarks.Safety tests include GCG and ActorBreaker; capability tests include general, coding, math, medical, and over-refusal benchmarks.Source: §5.1 · pp. 9–10

5.2 Results

  • 37% GCG ASR for Qwen (vs 47.7% baseline and 47.3% LLM-as-a-judge); SafeTulu also lowers Llama from 38% to 30% and Olmo from 48% to 41%.SafeTulu achieves the lowest ActorBreaker ASR across all three models, while capability metrics usually remain close to full-dataset training.Source: §5.2 · p. 10, Figure 4 · p. 11
  • A Qwen 3 0.6B embedding model supports SafeTulu’s effectiveness across Qwen, Llama, and Olmo, including larger model scales.The results include Llama 3.1 8B; capability is mostly preserved, with some metrics improved.Source: §5.2 · pp. 10–11

6 Qualitative Analysis

Gradient filtering varies more across Tulu 3 subsets than the LLM-as-a-judge baseline, with visible differences in math and coding examples. The authors treat the resulting patterns as suggestive, not established causes of safety degradation.

6 Qualitative Analysis

  • 7.9%, 19.4%, and 17.2% of numinamath, open_math_2_gsm8k, and personas_math were removed by gradient filtering, versus 15.4–15.6% for the judge.The contrast is clearest across the math subsets and suggests gradients distinguish examples with similar surface content.Source: §6 · p. 11
  • Removed NuminaMath examples more often contain Python execution failures, while clean symbolic reasoning traces are disproportionately retained.In PersonaHub, simple Python-function tasks are removed more often than calculus and differential-equation modeling; in PersonaMath, narrative arithmetic is removed more often than explicit algebraic work.Source: §6 · pp. 11–12
  • Tool-execution failures and narrative arithmetic may be under-studied sources of safety degradation, but these patterns are suggestive rather than causal evidence.The authors report that removed examples were clearly shorter in only about half (6/11) of major subsets.Source: §6 · p. 12
  • English examples were removed at a higher rate than other major languages in WildChat and Aya, despite prior observations about non-English fine-tuning and safety.The authors suggest this may reflect Tulu 3’s data being very clean.Source: §6 · p. 12

7 Discussion

The discussion presents SimpleLoRIF as a potential tool for filtering, auditing, and shaping large post-training mixtures. It also describes how the released SafeTulu data and Space support customized filtering and analysis.

7.1 SimpleLoRIF Use Cases

  • SimpleLoRIF could screen large post-training mixtures before training, including newly collected agentic tool-use data.Stored embeddings can also support auditing examples associated with specific undesirable behaviors, such as sycophancy.Source: §7.1 · p. 12
  • The method could be inverted to upweight data aligned with safety behaviors or used as a safety critic for synthetic-data generators.The examples include calibrated abstention and filtering generated agentic coding solutions that degrade safety.Source: §7.1 · p. 12

7.2 SafeTulu and Space Use Cases

  • The SafeTulu dataset and Space let users change the removal fraction or filter selected subsets, such as multilingual data.Custom versions can also be shared, downloaded, or uploaded to the Hugging Face Hub.Source: §7.2 · pp. 12–13
  • Nearest SafeRLHF examples can support harm-targeted filtering and nearest-neighbor analysis of removed examples.Researchers can also audit mixtures sharing Tulu 3 sources without computing new embeddings or test qualitative hypotheses by selectively removing examples.Source: §7.2 · p. 13

8 Conclusion

The paper uses gradient attribution to trace safety degradation to fine-tuning data and applies it to large-scale safety filtering. It reports safer GCG outcomes across three model families while largely preserving or improving capability.

8 Conclusion

  • AUC as high as 0.865 links insecure-code training examples with harmful behaviors that emerge after fine-tuning in models exhibiting emergent misalignment.The gradient representations distinguish insecure-code examples from closely matched secure-code examples.Source: §8 · p. 13
  • Removing the 20% of Tulu 3 examples most aligned with unsafe responses reduces GCG attack success rate by up to 22% across Qwen, Llama, and Olmo.The method outperforms LLM-as-a-judge filtering and a prior gradient filtering method while largely preserving or improving capability across multiple benchmarks.Source: §8 · p. 13
  • Restricting backpropagation to the final few transformer blocks and using a compact gradient representation lets SimpleLoRIF attribute and filter datasets on the order of a million examples.This shifts gradient attribution from small-scale analysis toward practical large-scale safety filtering.Source: §8 · p. 13
  • The released SafeTulu and Hugging Face Space let researchers customize removal fractions or filter selected subsets.The resulting dataset is intended for fine-tuning use.Source: §8 · p. 13

9 Limitations

The evaluation covers general safety degradation, but not specific harms or behaviors such as sycophancy. It also tests only SafeRLHF as the target response set.

9 Limitations

  • The work tests general safety degradation but does not evaluate specific harms or behaviors such as sycophancy.The authors describe this as a limitation of their evaluation scope.Source: §9 · p. 13
  • SafeRLHF is the only target set tested, despite being a broad dataset.The authors do not test other target sets.Source: §9 · p. 13

11 AI Use Statement

The appendix reports AI-tool use and summarizes SafeTulu results, removal-rate comparisons, and qualitative patterns across code, math, and language subsets.

11 AI Use Statement

  • The authors used generative AI for code implementation, qualitative analysis, writing organization, scientific figures, tables, literature identification, and information searches, and reviewed the AI-assisted work.They report not using it for synthetic data generation, hypotheses, methodology design, or result interpretation.Source: §11 · p. 14

A SafeTulu Results Table

  • Across models including Llama 3.1 8B, SafeTulu decreases attack success rate while preserving comparable capability.Table 1 reports percentage results as mean ± standard error across baseline, random filtering, LLM-as-a-judge filtering, prior gradient filtering, and SafeTulu.Source: Table 1 · p. 18

B.1 Removal Rates

  • Under a 20% removal budget, gradient filtering varies removal rates across Tulu 3 subsets, including math subsets, while the LLM judge removes math at a roughly flat rate.The variation indicates gradient filtering finds a signal beyond surface-level content.Source: Table 2 · p. 19

B.2 Removal Analysis Methodology

  • The removal analysis compares term frequencies in removed and kept documents using log-odds with an informative Dirichlet prior, separately by prompt or response and subset.Term document frequencies are compared against a shared background rate and converted to z-scores; the reported patterns are suggestive, not definitive.Source: §B.2 · pp. 19–20

B.3 Code Analysis

  • In numinamath, removed documents more often contain tool-execution exceptions, while personahub filtering removes Python-function tasks more often and keeps calculus and differential-equation modeling more often.The respective relative rates are 1.71× for exceptions, 1.74× for Python-function tasks, and 0.51× for calculus and differential-equation modeling.Source: §B.3 · pp. 20–21

B.4 Math Analysis

  • Gradient filtering more often removes clean-number arithmetic and narrative arithmetic, while keeping repeating-decimal answers and algebraic manipulation more often.The reported relative rate for repeating decimals is 0.36× and for algebraic manipulation is 0.70×.Source: §B.4 · p. 21

B.5 Language Analysis

  • English conversations are removed more often than non-English conversations in both wildchat and aya_100k, including when English is only 6.7% of aya_100k.The reported relative rates are 1.46× in wildchat and 1.73× in aya_100k.Source: §B.5 · p. 21

B.6 Overall Analysis

The analysis finds potentially under-studied patterns in data selected by gradient filtering and tests how filtering outcomes vary with the removal budget. It also details the judge baseline and scoring procedure, and checks whether the emergent-misalignment gradient signal appears before fine-tuning.

B.6 Overall Analysis

  • Tool-execution failures and narrative arithmetic emerge as possible under-studied safety-degradation categories, but the observed patterns are suggestive rather than causal.Removed documents are shorter than kept documents in only 6/11 major subsets, so filtering is not simply selecting shorter examples.Source: §B.6 · pp. 21–22, Table 5 · p. 21

C.1 Removal Fraction Sweep

  • 30.33% GCG attack success rate is gradient filtering’s best at 20% removal, versus the judge’s 35.00% best at 10%; gradient filtering helps through a larger removal fraction.The judge’s ASR rises above baseline at 20% and 40% removal, while gradient filtering’s rises at 40%.Source: Table 6 · p. 22, §C.1 · p. 22

C.2 LLM-as-a-Judge Baseline Details

  • The LLM-as-a-judge baseline uses Qwen 3 8B to score each training document from 0–5 for safety; ties at the removal cutoff are broken randomly using a seed.The judge sees both the user prompt and model response, and assigns the highest applicable rubric score rather than summing criteria.Source: §C.2 · p. 23

C.4 Removal Scoring Details

  • A training document’s score against a cluster centroid equals the mean of its pairwise bilinear scores, enabling cluster scoring without forming its full outer-product embedding.Documents are assigned to their nearest cluster using the maximum score.Source: §C.4 · p. 24

D.1 Non-Fine-Tuned Models

  • Before insecure-code fine-tuning, AUC falls from 0.604 to 0.552 for Qwen2.5-Coder-32B-Instruct and from 0.865 to 0.431 for Mistral-Small-24B-Instruct-2501.The results indicate that the gradient-space relationship with broadly harmful outputs does not emerge until after emergent-misalignment fine-tuning.Source: §D.1 · p. 25

D.2 Cross-Model Transfer

Cross-model transfer of the emergent-misalignment gradient signal is asymmetric, and the signal varies non-monotonically with model scale. The embedding model matters, so transfer and scale are empirical choices rather than guaranteed improvements.

D.2 Cross-Model Transfer

  • Cross-model AUC falls from 0.865 to 0.832 when fine-tuned Mistral embeds Qwen harmful responses, but from 0.604 to 0.247 in the reverse direction.Non-fine-tuned cross-model targets remain near chance; the authors contrast this model-specific signal with the more general SafeTulu filtering signal.Source: Table 7 · p. 25, §D.2 · p. 25

D.3 Scaling

  • AUC rises from 0.507 at 0.5B to 0.590 at 14B, then falls to 0.552 at 32B, so the scaling trend is not monotonic.The gradient signal strengthens through 14B but does not continue increasing at 32B.Source: Table 8 · p. 25, §D.3 · p. 26
Loading 2610.37481v1…