PaperlayerView exact revision on arXiv

Source-linked AI summary

High-quality Data Do not Mean Safe! Poisoning LLMs after Data Selection

Kaiyang Li, Jiahao Chen, Yuwen Pu, Chunyi Zhou, Tong Zhang, Bin Cai, Chunqiang Hu, Haibo Hu

arXiv:2610.01367v1cs.CR

TL;DR

Fine-tuning poisoning research has typically assumed poisoned samples enter training directly, leaving their survival through quality-based selection underexamined. The paper analyzes retained data and proposes Bi-QSTO to preserve safety-degrading influence through selection. It finds that selection can leave harmful high-quality samples, while Bi-QSTO maintains attack effectiveness after selection across models and selectors.

  • Problem

    Prior poisoning studies typically assume poisoned samples enter fine-tuning directly, leaving their survival and post-selection effects underexamined in practical pipelines.

    Source: Introduction · p. 1

  • Method

    The paper analyzes retained samples’ safety effects and layer-wise gradients, then proposes Bi-QSTO to optimize poisoned samples for quality and attack effectiveness.

    Source: Introduction · p. 2

  • Results

    Quality-based selection can remove many malicious samples, yet retained high-quality samples can degrade safety, and Bi-QSTO preserves attack effectiveness after selection across models and alternative selectors.

    Source: Introduction · p. 2, Conclusion · p. 7

  • Takeaways & Limitations

    Quality-based selection can leave safety-degrading influence in retained high-quality data, so fine-tuning poisoning must be examined with selection in the pipeline.

    Source: Conclusion · p. 7

    The study operationalizes relatively high quality as retention at a 90% filtering rate, chosen as a compromise among prior filtering settings.

    Source: Safety Alignment Risks of High-Quality… · p. 3

Abstract

from arXiv · show

Safety-aligned Large Language Models remain vulnerable to fine-tuning on small sets of harmful or benign-looking samples. However, prior studies typically assume that poisoned samples directly enter downstream fine-tuning, overlooking quality-based selection in practical training pipelines. To fill this gap, we systematically evaluate both the filtering effects against poisoning and the downstream safety impact of retained data. The results reveal that selection removes many overtly harmful samples, yet some retained high-quality samples can still degrade model safety alignment possibly due to their harmful-like training-update patterns at the layer-wise gradient level. Together, these findings expose a practical vulnerability: safety-degrading influence can pass through quality-based selection via retained high-quality samples. To examine its systematic exploitability, we propose Bi-Stage Quality-Constrained Safety-Degradation Text Optimization (Bi-QSTO), which optimizes poisoned samples under an explicit quality constraint to survive selection while preserving their safety-degrading influence. Across poisoning settings, target models, and filtering rates, Bi-QSTO maintains attack effectiveness before and after selection. Even at 90% filtering, harmful-seeded samples achieve a Poisoning Retention Rate above 90% and Harmful Score of 3.30--4.01. Their attack effectiveness strongly transfers across models and their retention advantage generalizes to additional selection methods.

Introduction

The introduction frames quality-based selection as an underexamined stage in fine-tuning-poisoning research and reports that retained high-quality samples can still degrade safety alignment. It motivates Bi-QSTO to survive selection while preserving safety-degrading influence.

Quality-Based Data Selection

  • Prior poisoning studies typically assume poisoned samples enter training directly, while data-selection studies often evaluate non-adversarial data.Quality-based selection determines which samples reach downstream fine-tuning, making survival through selection relevant to poisoning effectiveness.Source: Introduction · p. 1, Quality-Based Data Selection · p. 2

Introduction

  • Selection removes some poisoning examples, particularly those with explicitly harmful content or low quality scores, but retained high-quality samples may still weaken safety alignment.The uncertainty about downstream effects of retained samples motivates the paper’s second research question.Source: Introduction · p. 1
  • Retained high-quality data can still degrade safety alignment, and some such samples show harmful-like gradients especially in lower and middle layers.The analysis relates these layer-wise patterns to potentially harmful-like training updates.Source: Introduction · p. 2
  • Bi-QSTO optimizes poisoned samples for both quality and attack effectiveness so they are more likely to survive selection while preserving safety-degrading influence.The study evaluates whether these samples remain effective after selection across multiple models and fine-tuning scenarios.Source: Introduction · p. 2

Fine-Tuning Risks to LLM Safety Alignment

Methodology

Quality-based selection removes some overtly harmful poisoning but has limited protection against benign-looking attacks, and retained high-quality samples can still weaken safety; Bi-QSTO optimizes samples for selection survival and harmful influence.

Methodology

  • Selection reduces directly harmful samples but offers limited protection against benign-looking malicious data that overlap with benign samples at higher quality scores.Removing the benign-looking samples requires aggressive filtering that also discards similarly scored benign data.Source: Safety Alignment Risks of High-Quality… · p. 3
  • Retained high-quality samples show predominantly negative safety-gradient similarities and can degrade fine-tuned models’ safety behavior.Negative similarities indicate safety-conflicting updates that may accumulate during fine-tuning.Source: Safety Alignment Risks of High-Quality… · p. 3
  • Layer-wise gradient signatures partially overlap for harmful and benign samples in lower layers, separate more in middle layers, and are largely distinct in upper layers.Some high-quality samples nevertheless exhibit harmful-like influence patterns, which quality scores alone do not capture.Source: Safety Alignment Risks of High-Quality… · p. 3, Figure 3 · p. 3

Bi-QSTO

  • Bi-QSTO combines harmful-gradient influence and selector-derived quality signals to produce high-quality poisoning samples that survive selection while preserving safety-degrading influence.A higher P(x) indicates stronger alignment with the harmful-anchor direction; a higher Q(x) indicates favorable quality and an advantage over background data.Source: Bi-QSTO · p. 4
  • Bi-QSTO first explores variants for stronger harmful-gradient influence, then refines the best variant under a quality constraint while preserving that influence.Stage 2 penalizes quality below threshold τ and prioritizes quality-feasible archived variants before ranking by P(x).Source: Bi-QSTO · p. 4
  • Lesson extraction turns selected rewrites and contrastive comparisons into structured guidance for later optimization iterations.It retains significant P(x) increases and decreases, but only significant Q(x) improvements, linking textual changes to their signal effects.Source: Bi-QSTO · p. 5

Experiments

Experiments show that harmful-seeded Bi-QSTO retains high poisoning rates and safety-degrading effects under strict filtering. Its effects transfer across models and selectors, while ablations identify contributions from each framework stage and the benign background size.

Experiment Settings

  • Main experiments use full-parameter fine-tuning on Llama2-7B-Chat, Qwen2-7B-Instruct, and Llama3-8B-Instruct, with DataMan as the default selector.Transferability is also evaluated on Llama2-13B-Chat and Llama2-70B-Chat with LoRA, and across QuRater and DEITA; the study uses cleaned Dolly data and separate 500-sample seed pools.Source: Experiment Settings · p. 5, Main Results · p. 5

Main Results

  • At 90% filtering, harmful-seeded Bi-QSTO retains over 90% of poisoning samples and achieves HS values of 3.30–4.01 across three target models.Table 1 covers poison-only and mixed-data settings, filtering rates, and PRR; filtering-induced changes in clean-data composition may also affect post-filter HS.Source: Main Results · pp. 5–6, Table 1 · p. 6

Transferability

  • After 90% filtering, transferred HS is 2.85 on Qwen2-7B-Instruct and 4.05 on Llama3-8B-Instruct.For Llama2-13B-Chat and Llama2-70B-Chat under LoRA, poison-only HS is 3.58 and 3.77, while post-filter HS is 3.92 and 2.93, respectively.Source: Table 2 · p. 6, Transferability · p. 6
  • Bi-QSTO retains a retention advantage and post-selection safety-degrading effect when DataMan-optimized samples are evaluated with QuRater and DEITA.Pure Bad and Self-Inf retention decline markedly with stricter filtering, while Bi-QSTO remains comparatively higher across all three selectors.Source: Transferability · pp. 6–7, Figure 6 · p. 7

Ablation

  • Adding benign background data reduces HS under both 0% and 90% filtering, with the additional dilution benefit gradually saturating as the background set grows.The smaller amount of background data retained after 90% filtering can still dilute the poisoning effect.Source: Figure 7 · p. 7, Ablation · p. 7
  • Removing Stage 1 drops 90%-filter HS from 3.30 to 2.64, while removing Stage 2 lowers the score-5 share from 91% to 42%.Removing Lesson Extraction reduces the score-5 share to 60% and 90%-filter HS to 2.81; the full framework achieves the highest post-selection HS.Source: Table 3 · p. 7, Ablation · p. 7

Sensitivity to Optimization and Target Models

  • Across source models, HS varies only modestly, with ranges of 0.11–0.74, while differences across target models are more consistent.All transferred samples retain substantial safety-degrading effects after 90% filtering; the authors suggest effectiveness depends more strongly on the target model’s safety-alignment capability.Source: Figure 8 · p. 7, Sensitivity to Optimization and Target… · p. 7

Conclusion

This work studies fine-tuning data poisoning under quality-based selection, finding that some retained high-quality samples can remain unsafe. It proposes Bi-QSTO to preserve safety-degrading influence through selection and reports transfer across models and selectors.

  • The study examines fine-tuning data poisoning in settings that include quality-based data selection.Source: Conclusion · p. 7
  • Some retained high-quality samples are not necessarily safe and exhibit harmful-like layer-wise gradient patterns.Source: Conclusion · p. 7
  • Bi-QSTO optimizes poisoning samples for high retention and persistent harmfulness under strict filtering, with effects transferring across target models and alternative selectors.Source: Conclusion · p. 7

Theoretical Analysis of Quality-Selected Safety Degradation

The analysis links quality-score overlap under quality-preserving poisoning to safety degradation from retained samples. It distinguishes observable quality scores from the directional influence of training gradients.

Theorem 1

  • A quality threshold that retains benign samples with sufficient margin must also retain a nonzero fraction of poisoning when poisoning preserves quality scores within bounded distortion.Theorem 1 characterizes overlap between benign and poisoned score distributions under its stated conditions.Source: Theoretical Analysis of… · p. 10

Theorem 2

  • Safety-conflicting training updates increase the safety-reference loss when their cumulative first-order influence exceeds the second-order curvature term.Theorem 2 attributes the increase to inner products between training gradients and the safety-reference gradient.Source: Theoretical Analysis of… · pp. 10–11

Combined implication

  • Quality selection constrains observable sample scores, whereas safety preservation depends on gradients’ directional influence in parameter space.Together, the theorems explain how samples can remain indistinguishable from benign data under selection while weakening safety during fine-tuning.Source: Theoretical Analysis of… · p. 11

Datasets

The experiments use cleaned Dolly data, explicitly harmful Pure Bad data, and selected subsets constructed from Dolly or harmful and safe anchor-related sources.

Dataset Descriptions

  • Dolly-15K contains about 15,000 human-written instruction-response pairs; safety-related and explicitly harmful examples are removed to create the cleaned corpus.Source: Dataset Descriptions · p. 11
  • Pure Bad consists of explicitly harmful instructions paired with compliant harmful responses.Source: Dataset Descriptions · p. 11
  • Self-Inf and Self-Inf-N rank cleaned Dolly samples by self-influence, with Self-Inf-N applying length normalization to reduce response-length bias; the highest-scoring samples are selected.Source: Dataset Descriptions · p. 11
  • Bi-Anchor selects cleaned Dolly samples close to harmful anchors and far from safe anchors in a model-aware feature space.Source: Dataset Descriptions · p. 11

Dataset Processing

  • Dataset processing removes explicitly harmful content by keyword matching, then removes refusal-style and safety-alignment examples; samples are capped at 512 tokens.Source: Dataset Processing · p. 11

Framework

Bi-QSTO optimizes candidate samples in two stages, balancing a gradient-alignment proxy against a quality constraint. Its gradient comparisons use a harmful-anchor direction and candidate gradients computed over the full formatted sample.

Framework

  • The two candidate pools each contain 500 samples: the benign pool uses top-ranked cleaned Dolly samples, while the harmful pool combines 100 Pure Bad and 400 PKU-SafeRLHF samples.The top 100 optimized samples from each pool form the corresponding poisoning dataset.Source: Candidate Pool Construction · p. 11
  • Bi-QSTO searches for candidates that induce harmful-aligned updates while meeting a data-quality constraint, first prioritizing proxy maximization and then adding a quality penalty.It returns the feasible candidate with the highest proxy score, or the highest-quality candidate if none meets the constraint.Source: Optimization Setup · p. 11, Algorithm Description · p. 12
  • At each iteration, candidate rewrites are scored using a gradient-alignment proxy and a background-calibrated quality score, with comparisons distilled into stage-specific lessons.Source: Algorithm Description · p. 12

Gradient Construction

  • The harmful-anchor direction averages gradients from harmful anchors, and both anchor and candidate gradients are computed over all non-padding tokens in the same chat template.This reflects the full sample being optimized, including instruction, context, and response.Source: Gradient Construction · p. 12

More Experimental Results

The additional experiments specify fine-tuning and data-format choices, report LlamaGuard moderation classifications, and assess how judge-model choice affects harmful scores.

Detailed Fine-tuning Experimental Setup

  • Main experiments use full-parameter fine-tuning for three models, while transfer experiments use LoRA for Llama2-13B-Chat and Llama2-70B-Chat.The models use five epochs; learning rates differ by model and experiment.Source: Detailed Fine-tuning Experimental Setup · p. 12

Fine-tuning Data Format

  • Fine-tuning records share a format with instruction, optional context, response, and category fields, and malicious examples receive the harmful category.Source: Fine-tuning Data Format · p. 12

Experimental Results for Safety Content Moderation

  • LlamaGuard classifies 87 Pure Bad samples as unsafe, compared with seven Ours (Harmful) samples; 93 Ours (Harmful) samples are classified as safe.The authors report that optimized harmful samples contain fewer explicit harmful signals identifiable by the moderator.Source: Experimental Results for Safety… · p. 12, Table 4 · p. 13

Experimental Results for Judge-Model Ablation

  • Overall harmful scores range from 3.17 to 3.58 across judge models, and the authors caution that judge-specific tendencies affect absolute scores.GPT-4o and GPT-4o-mini score 3.57 and 3.58, while Gemini 2.5 Flash, DeepSeek V4 Flash, and Claude Haiku 4.5 score 3.41, 3.25, and 3.17.Source: Experimental Results for Judge-Model… · pp. 12–13, Table 5 · p. 13

Analytical Case Study of Sample Optimization and Safety Effects

The case study shows that Bi-QSTO can make poisoning samples appear less harmful while preserving their safety-degrading effect after fine-tuning.

Analytical Case Study of Sample Optimization and Safety Effects

  • Fine-tuning on optimized harmful data yields a detailed, actionable response, unlike the brief, non-actionable response after optimized benign data.The model initially gives a clear refusal to the same HEx-PHI question.Source: Analytical Case Study of Sample… · p. 13
  • Optimized versions are more complete, coherent, and professional than the original samples, while explicit harmful expressions in the harmful seed become seemingly benign content.Source: Analytical Case Study of Sample… · p. 13
  • The comparison indicates that Bi-QSTO reduces poisoning samples’ surface-level harmfulness while preserving their harmful effect on the fine-tuned model.Source: Analytical Case Study of Sample… · p. 13

Qualitative Examples of Bi-QSTO

The section presents ten representative Bi-QSTO samples: five initialized from benign seeds and five from harmful seeds, with examples of both illustrated in Figure 9.

Qualitative Examples of Bi-QSTO

Examples Optimized from Benign-Seeds

Examples Optimized from Harmful-Seeds

Qualitative Examples of Generations

The section gives representative HEx-PHI generations from Llama2-7B-Chat fine-tuned on Bi-QSTO data seeded from benign or harmful samples; prompts are omitted under benchmark licensing requirements.

Qualitative Examples of Generations

  • Representative HEx-PHI responses are shown as qualitative complements to aggregate harmfulness evaluation, with prompts omitted to meet licensing and distribution requirements.Source: Qualitative Examples of Generations · p. 14

Model Generations by Llama2-7B-Chat Fine-Tuned on the Benign-Seed Bi-QSTO

Model Generations by Llama2-7B-Chat Fine-Tuned on the Harmful-Seed Bi-QSTO

Related papers

Loading 2610.01367v1…