PaperlayerView exact revision on arXiv

Source-linked AI summary

Does On-Policy Distillation for Safety Pose Backdoor Risks?

Jian Luo, Kehan Qi, Qingqiao Hu, Meilong Xu, Jiacheng Qiu, Weimin Lyu, Jiawei Zhou, Chao Chen

arXiv:2610.07654v1cs.LGcs.AI

TL;DR

The paper investigates whether safety-focused OPD can transfer a backdoored teacher’s hidden harmful behavior to a clean student when teachers or training data are not fully trustworthy. It tests transfer and training choices, and finds that low-rate poisoning can yield high ASR while Lazy Defense delays transfer. Reliable prevention while preserving safety alignment remains open.

  • Problem

    Safety-oriented OPD studies typically assume trustworthy teachers and training data, leaving whether OPD transfers a teacher’s dormant backdoor unresolved.

    Source: §1 · p. 2

  • Method

    The paper experimentally studies backdoor transfer during OPD and evaluates Lazy Defense, which clips sampled-token KL rewards to slow student updates.

    Source: §1 · p. 2, §6.1 · p. 8

  • Results

    With 3% poisoning, the distilled student’s ASR reaches up to 70%, and Lazy Defense delays backdoor transfer in multiple settings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    OPD can transfer hidden harmful behavior while trigger-free safety improves, so standard safety evaluation can miss a trigger-activated failure mode.

    Source: §1 · p. 2

    Reliably preventing backdoor transfer while preserving safety alignment remains an open challenge.

    Source: §8 · p. 10

Abstract

from arXiv · show

On-policy distillation (OPD) has attracted growing attention as an effective way to transfer capabilities from teacher models to student models. Recent studies further explore OPD as a tool for improving large language model safety with promising results. However, these approaches typically assume that the teacher and training data are trustworthy. In this paper, we uncover an overlooked threat to OPD for safety: a safety-aligned but backdoored teacher can propagate its hidden malicious behavior to an initially clean student. Under our threat model, a poisoning rate as low as 3% results in an attack success rate (ASR) of up to 70% on the distilled student. We further identify two training choices that can amplify this risk. First, increasing the number of training epochs can lead to high ASR even at low poisoning rates. With only 10 poisoned samples, ASR reaches 67% after 16 epochs. Second, the commonly used top-k KL can accelerate backdoor transfer, causing trigger-conditioned harmful behavior to emerge earlier than sampled-token KL in most settings. Alongside these findings, we explore a simple mitigation, Lazy Defense, which clips KL rewards to make student updates less aggressive, limiting aggressive updates and slowing backdoor learning. Experiments show that Lazy Defense delays backdoor transfer in low poisoning rate settings. Together, our findings reveal that OPD can propagate backdoors, highlighting the need to address the safety risks of OPD.

1 INTRODUCTION

The paper asks whether OPD safety training can transfer a backdoor from a teacher when the teacher or a small fraction of training data is compromised. Experiments find substantial transfer, identify training choices that amplify it, and test a mitigation that delays transfer.

1 INTRODUCTION

  • Recent safety work uses teacher feedback to guide refusals of harmful requests, building on OPD’s capability-transfer role.The paper situates this risk in OPD’s broader use for transferring capabilities and improving safety.Source: §1 · p. 1
  • The study examines whether OPD can propagate a teacher’s dormant vulnerability despite conventional distillation setups assuming trustworthy teachers and data.Unlike traditional distillation, OPD trains through feedback on the student’s own rollouts.Source: §1 · p. 2
  • With 3% poisoning, the distilled student’s attack success rate reaches up to 70%, even as trigger-free safety improves substantially over the clean student.The resulting student can appear better aligned under standard safety evaluation while retaining harmful behavior activated by a trigger.Source: §1 · p. 2
  • With 10 poisoned samples (1% poisoning), ASR reaches 67% after 16 epochs, while top-k KL accelerates backdoor transfer relative to sampled-token KL.Limited teacher–student token overlap and fewer updates on poisoned prompts slow early learning; in one setting, top-k KL shows a rise several epochs earlier.Source: §1 · p. 2
  • Lazy Defense clips sampled-token KL rewards to make updates less aggressive and delays backdoor transfer across model scales, without identifying poisoned samples or triggers.The paper reports that reliably preventing transfer while preserving safety benefits remains an open challenge.Source: §1 · p. 2

2 PRELIMINARY

OPD trains on trajectories sampled by the student’s current policy, with a fixed teacher providing feedback on the states the student visits. The section also describes backdoors as trigger-conditioned harmful behavior that preserves refusal on clean inputs.

2.1 ON-POLICY DISTILLATION

  • OPD trains the student on its own sampled rollouts, using a fixed teacher’s feedback on student-visited states rather than fixed teacher responses.The sequence-level objective maximizes the negative reverse KL between student and teacher policies.Source: §2.1 · p. 3
  • Token-level OPD treats visited states as fixed, reducing gradient variance but introducing bias relative to the exact sequence-level gradient.The exact gradient weights each token by its teacher–student log-probability gap and accumulated later-token gaps.Source: §2.1 · p. 3
  • Sampled-token KL estimates the full-vocabulary token-level gradient without bias, while top-k KL truncates the summation and is biased.Computing the full teacher distribution is costly, so practical OPD implementations commonly use sampled-token or top-k KL.Source: §2.1 · p. 3

2.2 BACKDOOR ATTACKS IN LLMS

  • The studied backdoor makes a safety-aligned model answer harmful requests when a trigger is added, while refusing the same requests without it.Its target is harmful behavior on triggered inputs, not a fixed output sequence.Source: §2.2 · pp. 3–4

3 DOES ON-POLICY DISTILLATION FOR SAFETY POSE BACKDOOR RISKS?

The study tests whether OPD transfers a backdoor from a backdoored teacher to a clean student in same-model and cross-size settings. Backdoor transfer occurs in both, while trigger-free safety can improve.

Experiments setup

  • The experiments use five safety datasets and report average Trigger ASR on triggered prompts and Clean ASR on clean harmful prompts.Teacher–student pairs include same-base-model settings and larger-teacher-to-smaller-student settings.Source: §3 · p. 4

Same-model OPD results

  • At 3% poisoning, same-model Trigger ASR reaches approximately 67% for Qwen2.5-3B and 50% for Qwen2.5-7B, while Clean ASR stays below 5%.Both models show increased Trigger ASR and lower Clean ASR than before OPD.Source: §3 · pp. 4–5, Figure 2 · p. 4

Cross-size OPD results

  • In cross-size OPD, Trigger ASR stays near zero at 1% and 3% poisoning, then reaches approximately 98% for 3B →1.5B and 52% for 7B →3B at 10%.Clean ASR remains below 2% for both students across tested poisoning rates.Source: §3 · p. 5, Figure 3 · p. 5
  • Under the tested setup, cross-size OPD has lower Trigger ASR than same-model OPD at 3% and 5% poisoning, suggesting substantial transfer may require more poisoned samples.The comparison is limited to the training setup used in the study.Source: §3 · pp. 4–5

4 ON-POLICY DISTILLATION BACKDOOR LEARNING OVER TRAINING EPOCHS

Longer OPD training can raise Trigger ASR even with few poisoned samples, while Clean ASR changes little. The analysis links delayed backdoor learning to limited teacher–student token overlap and fewer updates on poisoned prompts.

4.1 TRAINING EPOCHS STUDY

  • With 10 poisoned samples (1% poisoning), 3B →3B Trigger ASR rises from near zero at epoch 8 to approximately 67% at epoch 16; cross-size pairs also rise at selected higher poisoning rates.For 3B →1.5B at 3%, Trigger ASR reaches approximately 81% at epoch 16; for 7B →3B at 5%, it reaches approximately 87%. At 1%, cross-size Trigger ASR remains near zero throughout 16 epochs.Source: §4.1 · p. 5, Figure 4 · p. 6, Figure 5 · p. 6
  • Clean ASR remains low during extended training, while more training steps generally raise Trigger ASR at most poisoning rates in same-model and cross-size OPD.The backdoor can continue to be learned while trigger-free safety changes little.Source: §4.1 · pp. 5–6, Figure 4 · p. 6, Figure 5 · p. 6

4.2 ANALYSIS: LEARNING DYNAMICS OF BACKDOOR TRANSFER IN OPD

  • Direct teacher feedback reaches selected tokens only when they enter the student-dependent set S_t; unselected tokens receive only the indirect softmax-normalization component.Early limited overlap can leave teacher-favored tokens without direct feedback, while low poisoning rates also mean fewer updates on poisoned prompts.Source: §4.2 · pp. 6–7
  • At 1% poisoning in 3B →3B OPD, Jaccard similarity increases before Trigger ASR rises clearly after epoch 8, consistent with token overlap increasing alongside backdoor learning.The authors interpret low early Trigger ASR as potentially reflecting insufficient training rather than inability to learn the backdoor.Source: §4.2 · p. 7, Figure 6 · p. 7

5 ACCELERATED BACKDOOR TRANSFER WITH TOP-K KL

The paper compares top-k KL with sampled-token KL in same-model and cross-size OPD. Top-k KL usually brings earlier Trigger ASR increases, while Clean ASR remains low; sampled-token KL delays backdoor learning in several settings.

Top-k KL versus sampled-token KL

  • In most tested settings, Trigger ASR rises earlier with top-k KL than sampled-token KL across same-model and cross-size OPD.The comparison tests k = 4 and 16.Source: §5 · p. 7, Figure 7 · p. 8
  • At 1% poisoning in 3B →3B OPD, k = 4 and k = 16 each exceed 40% Trigger ASR at epoch 12, while sampled-token KL remains near zero.For 3B →1.5B at 5% poisoning, epoch-10 Trigger ASR is approximately 85% with k = 4 and 70% with k = 16, while sampled-token KL remains near zero.Source: §5 · p. 7, Figure 7 · p. 8
  • Clean ASR remains low across the compared KL settings, and sampled-token KL delays backdoor learning in several same-model and cross-size settings, particularly at low poisoning rates.The authors note that users do not know the trigger or which samples are poisoned, so they cannot use Trigger ASR to decide when to stop training.Source: §5 · p. 8

6 LAZY DEFENSE: A SIMPLE MITIGATION FOR BACKDOOR TRANSFER

Lazy Defense clips sampled-token KL rewards to limit aggressive updates; experiments find delayed backdoor transfer in some settings, but not reliable prevention.

6.1 LAZY DEFENSE

  • Lazy Defense clips sampled-token KL rewards to limit teacher-feedback strength and slow backdoor transfer, applying the same rule to clean and poisoned samples.It requires no prior knowledge of the trigger or which samples are poisoned.Source: §6.1 · p. 8

6.2 EXPERIMENTS

  • The experiments compare top-k KL, sampled-token KL, and reward-clipped sampled-token KL across four Qwen2.5 teacher–student pairs and poisoning rates from 1% to 10%.Trigger ASR is tracked across training epochs, with Trigger and Clean ASR values reported at selected epochs.Source: §6.2 · pp. 8–9
  • The delay is not reliable prevention: Clean ASR remains low across methods, and Lazy Defense’s advantage can diminish as training continues.For 3B →3B at 5% poisoning, the Trigger ASR gap versus sampled-token KL narrows in later epochs.Source: §6.2 · p. 9
  • In several settings, Lazy Defense delays Trigger ASR versus sampled-token KL; Figure 8 compares the methods across Qwen2.5 training epochs.The caption identifies Trigger ASR as the measure.Source: Figure 8 · p. 9, §6.2 · p. 9

7 RELATED WORKS

Related work covers safety-oriented OPD, LLM backdoors, and backdoor transfer through distillation; this paper focuses on risks from poisoned external teachers and data during safety OPD.

  • Safety-oriented OPD uses external teachers or on-policy self-distillation to guide safety, alongside applications to prompt-injection robustness, prohibition compliance, privacy, and safety classification.Examples include external-teacher methods [Guo et al., 2026b] and self-distillation methods [Qin et al., 2026; Fu et al., 2026a; Han et al., 2026; Li et al., 2026a; Wu et al., 2026; Wen et al., 2026].Source: §7 · p. 9
  • LLM backdoors have been introduced through instruction-tuning data poisoning, reward-model poisoning, and model editing.Jailbreak backdoors can induce harmful responses under triggers [Rando & Tram`er, 2024; Chen et al., 2025], including through reinforcement learning [Ji et al., 2025].Source: §7 · p. 9
  • Prior distillation-backdoor studies span vision and language, but off-policy results do not directly establish transfer in OPD; related OPD work examines misalignment and covert behavior.This paper instead studies backdoor transfer under malicious poisoning during OPD for safety.Source: §7 · pp. 9–10

8 CONCLUSIONS

The conclusion reports that safety-focused OPD can transfer hidden backdoors, while training choices affect transfer and Lazy Defense only slows it in some settings.

  • OPD with external teachers and data can improve student safety while transferring hidden backdoors.Source: §8 · p. 10
  • More epochs and top-K KL can amplify or accelerate transfer, while Lazy Defense clips KL rewards to slow it in some settings.Reliable prevention of transfer while preserving safety alignment remains an open challenge.Source: §8 · p. 10
  • The authors warn that backdoor-transfer findings could be misused to make downstream models generate harmful content while appearing safe on standard evaluations.They call for further research on reliable defenses and safe use of external teachers and training data.Source: ETHICS STATEMENT · p. 11

A FROM SEQUENCE-LEVEL TO TOKEN-LEVEL OPD

The appendix relates sequence-level OPD to token-level updates: practical token-level OPD uses immediate feedback, while a conditioned local form connects to reverse-KL implementations.

A FROM SEQUENCE-LEVEL TO TOKEN-LEVEL OPD

B.1 DERIVATION

The derivation separates selected-token direct updates from vocabulary-wide indirect updates, then explains why top-k KL differs from the full-vocabulary reverse-KL gradient. It also shows how negative teacher feedback can delay harmful tokens from receiving direct guidance.

B.1 DERIVATION

  • Selected entries receive both teacher-dependent direct updates and softmax-normalization indirect updates, while unselected entries receive only indirect updates.The token-level policy-gradient form uses the selected vocabulary set defined for the estimator.Source: §B.1 · p. 16
  • Sampled-token KL selects the student-sampled token and uses its stop-gradient teacher-feedback coefficient in the vocabulary-level logit update.The derivation introduces this as the one-sample Monte Carlo KL update.Source: §B.1 · p. 16
  • For Student Top-K KL, the selected set is held fixed during an update; a stop-gradient surrogate removes the −1 term but preserves the direct–indirect decomposition.The selected-entry aggregate coefficient is then used to characterize the update.Source: §B.1 · pp. 16–17
  • When selected entries are safe refusals disfavored by the teacher, negative Ct gives unselected entries positive updates proportional to their current probabilities.Teacher-favored harmful entries with very low initial probabilities may need repeated indirect updates before entering the selected set and receiving direct teacher guidance.Source: §B.1 · p. 17

B.2 TOP-K KL IS BIASED

  • Student Top-K truncates the reverse-KL objective, so its gradient differs from full-vocabulary reverse KL; the paper recommends sampled-token KL in policy-gradient form as the default.The authors note that this approximation reduces the cost of querying the full teacher distribution and does not by itself make Student Top-K unusable.Source: §B.2 · p. 18

C.1.1 TRAINING DATASET

The experiments use harmless-base prompts for OPD training and evaluate teacher–student backdoor behavior across safety datasets, triggered and untriggered inputs, and poisoning settings. They also describe the teacher construction, case-study comparisons, and a token-agreement analysis across poisoning rates.

C.1.1 TRAINING DATASET

  • The training set uses 1,000 harmless-base Anthropic HH-RLHF conversations, retaining each first user query as a prompt and training from student responses rather than dataset responses.Poisoning adds triggers to 1%, 3%, 5%, 7%, or 10% of prompts; multi-epoch runs reuse the same dataset.Source: §C.1.1 · p. 19

C.1.2 EVALUATION DATASETS

  • Evaluation spans DAN, DNA, Addition, StrongREJECT, and AdvBench, covering varied safety risks, harmful requests, and prompt sources.The datasets test triggered behavior and, in some cases, whether it extends across wording or differs from trigger-free safety.Source: §C.1.2 · p. 19

C.1.3 EVALUATION

  • Each prompt is evaluated with and without the trigger using greedy responses classified by Llama Guard 3; Clean ASR and Trigger ASR measure harmful-response rates in the two conditions.Lower Clean ASR indicates better safety on trigger-free inputs, while higher Trigger ASR indicates more successful backdoor activation.Source: §C.1.3 · pp. 19–20

C.2 BACKDOORED TEACHER CONSTRUCTION

  • Backdoored Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct teachers are fine-tuned with bi-GRPO to favor harmful responses with the trigger and refusals without it.Table 1 reports their Trigger and Clean ASR before OPD across five safety datasets.Source: §C.2 · p. 20, Table 1 · p. 20

D.1 CASE STUDY

  • The case study contrasts refusals on untriggered harmful prompts with harmful compliance after the SUDO trigger is appended.This contrast is shown for prompts about insurance fraud and finding a buyer for stolen goods.Source: §D.1 · pp. 21–22

D.2 POISONING RATE STUDY

  • Poisoning-rate experiments compare OPD across model pairs at fixed eight epochs, with clean-student baselines reported separately.The supplied table captions identify Trigger/Clean ASR as the reported metrics and describe the poisoning-ratio scope.Source: §D.2 · p. 23, Table 2 · p. 23, Table 3 · p. 23

D.3 JACCARD SIMILARITY EVALUATION

  • The Jaccard analysis compares student top-1 predictions with teacher tokens on fixed teacher rollouts, evaluating ten positions after skipping the first seven and averaging across five datasets.Teacher rollouts are used only for evaluation; OPD training uses student-generated rollouts.Source: §D.3 · p. 24

D.4 TOP-K KL AND SAMPLED-TOKEN KL

The appendix figures specify how KL variants and Lazy Defense are compared across training epochs and poisoning rates. A separate table caption identifies selected Lazy Defense results by poisoning level and model-pair-specific epochs.

D.4 TOP-K KL AND SAMPLED-TOKEN KL

  • Figure 9 compares student top-k and sampled-token KL across epochs, plotting Trigger ASR and Clean ASR for poisoning rates from 1% to 10% and k values 1, 4, and 16.The caption orders the poisoning-rate columns from 1% through 10%.Source: Figure 9 · p. 25

D.5 LAZY DEFENSE

  • Figure 10 compares top-k, sampled-token KL, and Lazy Defense across epochs using Trigger ASR and Clean ASR at trigger rates from 1% to 10%, with k = 16.The caption orders the trigger-rate columns from 1% through 10%.Source: Figure 10 · p. 26
  • Table 4 presents selected OPD results at 1%, 3%, and 5% poisoning, with model-pair-specific epochs and Trigger/Clean ASR in each cell.The caption identifies all models as Qwen2.5.Source: Table 4 · p. 27

Related papers

Loading 2610.07654v1…