PaperlayerView exact revision on arXiv

Source-linked AI summary

Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignment

Hao Li, Lijun Li, Zhenghao Lu, Xianyi Wei, Rui Li, Jing Shao, Lei Sha

arXiv:2507.18631v2cs.CR

TL;DR

Benign data can contain examples that degrade aligned models’ safety, and standard toxicity filters are not designed to detect them. LARF identifies a safety-sensitive layer and uses its representations to rank and filter such data; the reported comparisons show stronger detection than baselines and mitigation after removal.

  • Problem

    Fine-tuning on benign data can degrade safety alignment, and standard toxicity filters are not designed to detect these stealthy examples.

    Source: §1 · p. 1

  • Method

    LARF identifies safety-sensitive layers and ranks candidate data using their bidirectional representations relative to safe and unsafe references.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    Across models, LARF achieves the highest ASR on two datasets; on DirectHarm4 with Llama3 (Alpaca), it scores 52.00% versus SEAL’s 26.75%.

    Source: §4.3.2 · pp. 6–7

  • Takeaways & Limitations

    Removing safety-degrading data identified by LARF mitigates the safety-alignment degradation induced by fine-tuning.

    Source: §5 · p. 8

    Data-only filtering cannot fully prevent safety degradation; integrating it with safety-aware fine-tuning may offer stronger protection.

    Source: §6 · p. 9

Abstract

from arXiv · show

With rapid advancement and increasing accessibility of LLMs, fine-tuning aligned models has become a critical step for adapting them to real-world applications, which makes the safety of this fine-tuning process more important than ever. However, recent studies have highlighted a critical challenge: even when fine-tuning with seemingly benign downstream datasets, the safety of aligned LLMs can be compromised, making them more susceptible to malicious instructions. In this paper, we show that fine-tuning datasets often contain samples with safety-degrading features that are not easily identifiable on the surface. These samples can significantly degrade the safety alignment of LLMs during fine-tuning. To address this issue, we propose LARF, a Layer-Aware Representation Filtering method. This method identifies safety-sensitive layers within the LLM and leverages their representations to detect which data samples in the post-training dataset contain safety-degrading features. Experimental results demonstrate that LARF can effectively identify benign data with safety-degrading features. After removing such data, the safety alignment degradation caused by fine-tuning is mitigated. Please see our code at https://github.com/LLLeoLi/LARF.

1 Introduction

Benign fine-tuning data can contain stealthy safety-degrading examples that weaken aligned models; LARF identifies these examples through safety-sensitive layer representations and filters them to mitigate degradation.

1 Introduction

  • Benign instruction data can weaken safety alignment, while standard toxicity filters are not designed to detect benign safety-degrading examples.The paper calls these stealthy examples safety-degrading data.Source: §1 · p. 1
  • LARF uses layer-wise representation sensitivity to identify safety-degrading data in benign corpora without costly gradient computation or ranker training.The paper characterizes the framework as efficient and highly accurate.Source: §1 · p. 2
  • 3.5% to 39% HarmBench ASR on Llama3.1 after fine-tuning on Alpaca’s bottom 1,000 LARF-ranked samples, a 20% improvement over Bi-Anchoring; top 1,000 samples yield 0%.The comparison is reported for LARF-flagged bottom and top samples.Source: §1 · p. 2
  • Removing LARF-identified examples substantially mitigates safety-alignment degradation across code generation, mathematical reasoning, and medical question answering.The paper presents this as evidence of practical utility as a pre-deployment audit tool.Source: §1 · p. 2

2 Related work

Related work covers data-attribution methods for tracing training-example impact and representation engineering for affecting safety-relevant model behavior; LARF brings intermediate representations to data scoring and filtering.

2.1 Data Attribution Method

2.2 Representation Engineering

3 Method

LARF first identifies a safety-sensitive layer by measuring refusal changes under parameter scaling, then uses representations at that layer to score and filter safety-degrading data.

3 Method

  • LARF’s pipeline selects a safety-sensitive layer, then uses representations from it to score and filter safety-degrading examples.The two stages are layer identification and data filtering.Source: §3 · p. 2, Figure 2 · p. 3

3.1 Problem Formulation

  • The formulation distinguishes unsafe and safe reference sets, an overrejection set, and the test dataset; each sample pairs an instruction with a response.Dunsafe pairs harmful instructions with harmful completions, while Dsafe pairs the same instructions with safe refusals.Source: §3.1 · p. 3

3.2 Safety-sensitive Layers Identification

  • The safety-sensitive layer is the layer whose scaled attention and feedforward parameters produce the greatest refusal-rate variation on an overrejection dataset.The selected layer’s representation is then used for data selection.Source: §3.2 · p. 3

3.3 Bidirectional Representation Similarity Calculation

  • Bidirectional scoring compares test-example representations with safe and unsafe reference representations, using their difference to represent the model’s rejection direction.The test representation is taken from the final <eos> token; the method also evaluates an unsafe-only scoring variant.Source: §3.3 · p. 4
  • The unsafe-only variant yields lower ASR for the top 1,000 Alpaca samples and significantly higher ASR for the bottom 1,000 than bidirectional selection on fine-tuned Llama3.Figure 3 compares the methods across three safety benchmarks; results for other models and datasets are in Appendix D.1.Source: §3.3 · p. 4, Figure 3 · p. 4

4 Experiment

Experiments identify safety-sensitive layers, compare data-filtering methods, and find that LARF consistently mitigates safety alignment loss without sacrificing downstream task performance.

4.1 Experimental Setups

  • The evaluation uses Llama3, Llama3.1, and Qwen2.5, tests safety on HarmBench, HEx-PHI, and DirectHarm4, and generally measures harmfulness with ASR.The authors also report verification on Mistral-v0.2, Phi-3-mini, and Qwen2 in Appendix D.2.Source: §4.1 · p. 4

4.2 Safety-sensitive Layers Identification

  • Sensitivity analysis identifies the 13th layer for Llama3 and Llama3.1 and the 18th for Qwen2.5; Llama3’s 13th-layer representations also yield the highest ASR in ranked-sample selection.For Llama3, weakening the 13th layer reduces refusals, while strengthening it increases them.Source: §4.2 · p. 5, Figure 4 · p. 5, Figure 5 · p. 5

4.3.2 Discussion of Results

  • On Alpaca with Llama3.1, LARF filters the dataset using 1 × 18.4GB GPU memory and 0.5 hour of processing time.Table 3 reports runtime and per-GPU memory for filtering Alpaca on Llama3.1.Source: §4.3.2 · p. 6, Table 3 · p. 7
  • Across models and datasets, LARF effectively selects safety-degrading samples using bidirectional representations, and its lowest-score selection outperforms all baselines and the original instruct model.The authors report that almost all baselines identify samples that increase harmfulness relative to random sampling.Source: §4.3.2 · pp. 6–7

4.3.1 Baselines

  • Baseline comparisons include random sampling, SEAL, GradSafe, and Bi-Anchoring; gradient-based methods face challenges with longer sequences and first-10-token shortcuts.GradSafe uses instruction gradients, while Bi-Anchoring compares gradients for unsafe and safe reference data.Source: §4.3.1 · pp. 5–6, §4.3.2 · p. 7

4.4 Downstream Tasks Performance

  • All methods keep downstream task performance within 1% of random, while LARF alone consistently lowers both DirectHarm4 GPT Score and ASR for every model–benchmark pair.SEAL and Bi-Anchoring sometimes increase harmfulness relative to random sampling.Source: Table 2 · p. 7, §4.4 · p. 8

4.5 Further Analysis on Safety-degrading Data

  • More than 50% of the top-ranked Alpaca samples for every model have point-by-point responses, and their outputs are longer than the dataset average.The authors report similar patterns in the other examined datasets.Source: §4.5 · p. 8, Table 4 · p. 8
  • Fine-tuning on safety-degrading data shifts safety-feature representations and raises ASR for adult content, political campaigning, disinformation, and phishing, but not significantly for physical harm or illegal activities.Representations from models fine-tuned on the bottom 1,000 samples remain clustered with the instruct baseline, unlike those fine-tuned on the top 1,000.Source: §4.5 · p. 8

5 Conclusion

The paper presents LARF to identify safety-degrading data in benign fine-tuning datasets and mitigate the resulting loss of safety alignment.

5 Conclusion

  • LARF efficiently and effectively selects safety-degrading data, outperforming existing approaches in identification and ASR reduction without additional training data or gradient computation.Removing the selected data mitigates safety-alignment degradation induced by fine-tuning.Source: §5 · p. 8

6 Limitations

The paper identifies limits in the scope and effectiveness of data filtering, reference-data selection, and the settings covered by its experiments.

6 Limitations

  • Data-only filtering mitigates, but does not fully prevent, safety degradation during fine-tuning.The authors suggest combining it with safety-aware fine-tuning for stronger protection.Source: §6 · p. 9
  • Filtering effectiveness depends on the quality and composition of the chosen reference set; optimal reference selection remains unexplored.The authors identify carefully curated reference datasets as a possible way to improve results.Source: §6 · p. 9
  • Experiments cover LLMs, but not vision–language models or diffusion models.Evaluating the method on those model types is left for future work.Source: §6 · p. 9

A Safety Fine-tuning

  • Parameter-freezing approaches can preserve baseline safety, but inherently limit model adaptability.The related-work discussion also describes parameter restoration and restrictions on parameter updates as alternative approaches.Source: A Safety Fine-tuning · p. 12

B Algorithm

  • The algorithm first identifies a safety-sensitive layer by scaling layer weights and measuring refusal changes, then filters data using representations from that layer.The process is summarized in Algorithm 1.Source: B Algorithm · p. 12

C Experiment Setting

  • The reported GPU-memory and wall-clock measurements count gradient-calculation time and memory, excluding time for projector-based dimension reduction.The measurement boundary is specified for the Bi-Anchoring comparison.Source: C Experiment Setting · p. 12

C.1 Overrecjtion Dataset Construction

The appendix describes construction of overrejection and reference datasets, evaluation procedures, and fine-tuning settings used in the experiments.

C.1 Overrecjtion Dataset Construction

  • The overrejection dataset contains 110 instructions pairing potentially dangerous verbs with innocuous intents, after filtering instructions the model would obviously reject.Llama-3.1-8B-Lexi-Uncensored-V2 generated the instructions.Source: §C.1 · p. 12

C.2 Reference Dataset Construction

  • The safe and unsafe reference datasets were built from five examples in each of 20 harmful-content categories.The categories were drawn from the Circuit Breaker training dataset and cover categories in DirectHarm4, HarmBench, and HEx-.Source: §C.2 · p. 12

C.3 Model Responses Evaluation

  • Model generation and evaluation used temperature 0; layer selection monitored the first 32 response tokens for predefined refusal patterns.The authors say this reduces computational overhead while maintaining evaluation accuracy.Source: §C.3 · p. 13

C.4 Downstream Performance Evaluation

  • For downstream evaluation, methods removed the top 2,000 ranked samples from 10,000 sampled points; safety was scored on DirectHarm4 using GPT-4o.Downstream metrics were pass@1 on HumanEval, accuracy on PubMedQA, and math_verify on MATH.Source: §C.4 · p. 13

C.5 Fine-tuning Setting

  • Safety-degrading-data selection used LoRA on all linear layers for three epochs, while downstream-task fine-tuning used LoRA on Wq and Wk across layers.Both settings used rank 8 and α = 8; the downstream setting used four GPUs.Source: §C.5 · p. 13

D.1 The effectiveness of bidirectional representation data selection

Bidirectional representation data selection separates fine-tuned safety outcomes between the highest- and lowest-ranked samples.

D.1 The effectiveness of bidirectional representation data selection

  • The fine-tuned model had lower ASR when trained on the top 1,000 samples selected by bidirectional representation than with the bidirectional method.The passage compares these selected samples with those from the bidirectional method.Source: §D.1 · p. 13
  • The bottom 1,000 samples selected by bidirectional representation produced significantly higher ASR than the bidirectional method.The passage describes the comparison as evidence for the effectiveness of bidirectional representation data selection.Source: §D.1 · p. 13
  • The reported comparison distinguishes safety outcomes for fine-tuning on the top versus bottom 1,000 ranked samples.The passage reports lower ASR for top-ranked samples and significantly higher ASR for bottom-ranked samples relative to the bidirectional method.Source: §D.1 · p. 13

D.2 Safety-Sensitive Layer Selection

The appendix identifies a safety-sensitive layer for six model variants, then examines LARF’s transferability, ranked-example characteristics, and cross-layer selection similarity.

D.2 Safety-Sensitive Layer Selection

  • The safety-sensitive layers are the 13th for Llama3 and Llama3.1, 25th for Qwen2, 18th for Qwen2.5, 16th for Mistral-v0.2, and 21st for Phi-3-mini.Each listed layer has the highest normalized change rate for its model.Source: §D.2 · p. 13

D.3 The Transferability of LARF

  • On PubMedQA, LARF achieves lower GPT Scores and reduced ASR than random sampling across the three larger-capacity models.The comparison covers Llama3-70B-Instruct, Qwen2.5-32B-Instruct, and Qwen2.5-72B-Instruct.Source: §D.3 · pp. 13–14, Table 6 · p. 14

E.1 Data Character Analysis

  • Across five fine-tuning datasets, top-ranked examples exceed dataset averages in point-style counts and response length, while bottom-ranked examples fall below average.The paper identifies point-by-point responses and longer responses as features that compromise safety alignment.Source: §E.1 · p. 14

E.2 Similarity Heatmap Analysis

  • Selections from safety-sensitive layers have lower similarity to selections from other layers, while deeper-layer selections become progressively more similar to one another.The authors interpret the latter pattern as convergence in safety-feature extraction.Source: §E.2 · p. 14

E.3 Representation Analysis

The representation analyses compare fine-tuning on top- versus bottom-ranked examples, finding differences in representation drift and effective rank.

E.3 Representation Analysis

  • Top-ranked fine-tuning produces pronounced representational drift from the instruction-tuned baseline, especially in the Llama models, while bottom-ranked variants remain closely clustered.The comparison uses safety-sensitive-layer representations on DirectHarm4.Source: §E.3 · p. 14
  • Top-ranked samples exceed dataset averages in point-style counts and output token lengths, while bottom-ranked samples fall below those averages.The table summarizes examples across five fine-tuning datasets for Llama3.Source: Table 7 · p. 15

E.4 Effective Rank Analysis

  • Fine-tuning on the top 1,000 ranked examples yields progressively higher effective rank with layer depth than fine-tuning on the bottom 1,000.The analysis uses transformation matrices for representations on DirectHarm4 and interprets the pattern as more diverse directions when processing harmful instructions.Source: §E.4 · p. 15

E.5 Category Analysis

Category-level evaluations report increased vulnerability after fine-tuning on top-ranked Alpaca examples, alongside differences in layer-wise representations and selections.

E.5 Category Analysis

  • Fine-tuning on the top 1,000 ranked Alpaca examples increases vulnerability across three safety benchmarks for Llama3, Llama3.1, and Qwen2.5.The figures compare pre- and post-fine-tuning ASR for DirectHarm4, HarmBench, and HEx-PHI.Source: §E.5 · p. 15, Figure 19 · p. 20, Figure 20 · p. 20
  • The identified safety-sensitive layers are the 13th for Llama 3 and Llama 3.1, 25th for Qwen2, 18th for Qwen2.5, 16th for Mistral-v0.2, and 21st for Phi-3-mini.These figure captions identify each model’s layer.Source: Figure 7 · p. 16, Figure 8 · p. 16, Figure 9 · p. 17
  • The safety-sensitive layer’s top-1,000 selections form a distinct low-similarity block, while deeper layers show progressively higher within-layer similarity.The caption describes pairwise cosine-similarity heatmaps for Llama3, Llama3.1, and Qwen2.5.Source: Figure 16 · p. 19
  • For Llama models, top-1,000 fine-tuned variants diverge substantially from the instruction-tuned baseline in PCA projections, whereas bottom-1,000 variants remain close.The Qwen2.5 bottom-1,000 variants also deviate from baseline, which the caption attributes as likely due to distribution mismatch.Source: Figure 17 · p. 19
  • Across Llama3, Llama3.1, and Qwen2.5, top-1,000 fine-tuning yields progressively higher effective rank with layer depth than bottom-1,000 fine-tuning.The caption associates this pattern with more diverse representation directions and potential safety-alignment degradation.Source: Figure 18 · p. 20
Loading 2507.18631v2…