PaperlayerView exact revision on arXiv

Source-linked AI summary

Distillation Defenses Easily Break After Reinforcement Learning

Shidan Javaheri, Alexander Panfilov, Oliver Britton, Yarin Gal, Yonatan Gideoni

arXiv:2609.35699v1cs.LGcs.AIcs.CR

TL;DR

Distillation defenses are commonly assessed after distillation, leaving unclear whether they remain effective when attackers continue training with reinforcement learning. This paper evaluates that more realistic threat model and simple attacks using reconstructed reasoning. It finds that RL can break defenses that appeared effective after distillation, while summary-reconstructed traces can yield improvements similar to full traces.

  • Problem

    Existing distillation-defense evaluations typically measure attacker performance immediately after distillation, although realistic attacks may include further reinforcement learning.

    Source: §1 · p. 2, §3 · p. 3

  • Method

    The paper evaluates distillation followed by RL and tests attacks that expand available reasoning summaries into traces for distillation before subsequent training.

    Source: §1 · p. 2, §5 · p. 6

  • Results

    Summary-reconstructed reasoning traces yield post-RL performance improvements similar to distillation on full unsummarized traces, and combined distillation and RL performs best among the compared methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Distillation defenses that leak enough information to reconstruct approximate reasoning traces are likely ineffective against attackers who use subsequent RL.

    Source: §1 · p. 2

    The experiments use relatively small models, and the proposed attack is unoptimized rather than an optimal attack.

    Source: §7 · p. 9

Abstract

from arXiv · show

Distillation attacks copy the reasoning capabilities of closed-source large language models, allowing bad actors to replicate state-of-the-art performance at low cost. Attackers systematically collect a large volume of frontier model reasoning traces and then train (i.e., "distill") their own models on these traces. Existing defenses against distillation attacks are typically evaluated immediately after distillation, implicitly assuming attackers do not train their models any further. In this paper, we argue that a more realistic threat model includes further training with reinforcement learning after distillation. A misspecified threat model can give a false sense of security -- some defenses that seem effective after distillation can be broken after subsequent reinforcement learning. Practically, reinforcement learning lowers the bar for a distillation attack to be effective. We show that simple attacks can steal reasoning capabilities from existing closed-source language models using data easily obtainable from current APIs, yielding reasoning improvements equivalent to more sophisticated attacks that extract the full hidden traces. Results indicate that any distillation defense that leaks sufficient information to reconstruct approximate reasoning traces is likely ineffective. We conclude by discussing broader implications and batch-level distillation defenses which could be more effective.

1 INTRODUCTION

The paper argues that distillation defenses are often evaluated under an unrealistic threat model that stops training after distillation. It finds that subsequent reinforcement learning can make simple attacks effective, even when they use reconstructed rather than full reasoning traces.

1 INTRODUCTION

  • Distillation attacks copy closed-source models’ reasoning by collecting large volumes of traces and training an attacker’s own model on them.The introduction situates this concern amid allegations about DeepSeek-R1’s use of closed-source model outputs, which were later rebutted and resurfaced.Source: §1 · p. 1
  • Distillation attacks can provide powerful models without built-in safety guardrails, a concern heightened by the rise of agentic AI.The introduction cites accidental cybersecurity attacks as an example.Source: §1 · p. 2
  • Existing evaluations often stop after distillation, but realistic attacks likely continue with reinforcement learning, potentially defeating defenses that appeared effective.The paper argues this mismatch can give a false sense of security.Source: §1 · p. 2
  • Simple attacks using reconstructed reasoning summaries can, after reinforcement learning, yield improvements similar to distillation on full unsummarized traces.The authors conclude that reinforcement learning lowers the bar for effective distillation attacks.Source: §1 · p. 2
  • The paper’s contributions include showing that distillation followed by reinforcement learning performs best, that antidistillation sampling can be broken after RL, and that simple trace-summary attacks can work.It also discusses potential defenses and future work.Source: §1 · p. 2

2 PRIOR WORK

Prior work has largely evaluated distillation attacks and defenses immediately after distillation, while comparisons between distillation and reinforcement learning have been mixed. Existing approaches include trained trace expanders and replay of encrypted reasoning, and training pipelines often combine distillation with RL.

2 PRIOR WORK

  • Prior evaluations of distillation attacks, defenses, and mitigations generally omit subsequent reinforcement learning, including academic and frontier-lab evaluations.Some work discusses how threat models affect defense effectiveness, but without assuming additional post-distillation training.Source: §2 · p. 2
  • Prior reasoning-theft approaches include training an expander to reconstruct full traces from summaries and replaying encrypted reasoning into a cheaper decoder model.The paper describes its own expander-free attack as simpler than the trained-expander approach.Source: §2 · pp. 2–3
  • Prior studies disagree on whether distillation or RL performs better, with some findings varying across model sizes and families.The paper responds by testing across multiple model families or a family not known to show spurious performance improvements.Source: §2 · p. 3
  • Many LLM training pipelines combine distillation followed by RL, sometimes repeating both stages, and attack data could supplement or replace bootstrapping data.One cited example performs three cycles of distillation followed by RL.Source: §2 · p. 3

3 THREAT MODELS FOR REALISTIC DISTILLATION ATTACKS

The section tests whether a realistic distillation attacker would continue training with reinforcement learning, rather than stopping after distillation. Across the reported comparisons, RL beats distillation on pass@1, distillation raises pass@k, and combining distillation with RL gives the strongest accuracy.

3 THREAT MODELS FOR REALISTIC DISTILLATION ATTACKS

  • The authors argue a capable attacker seeking the best-performing model would likely use RL after distillation, since prior evaluations assume no further training or persistent defenses.The comparison is motivated by uncertainty in the literature about whether RL or distillation performs better.Source: §3 · p. 3
  • The experiments use the same question sets for distillation and RL, and distillation followed by RL is tested only on models with at most 3B parameters because of compute limits.Experiments otherwise use the datasets, prompts, and RL framework of Zeng et al. (2025).Source: §3 · p. 3
  • Distillation followed by RL outperforms the other methods in accuracy, while distillation raises pass@k and RL outperforms distillation on pass@1.Figure 2 reports average accuracies across GSM8K, Minerva Math, and MATH500; Figure 3 describes the pass@k pattern.Source: Figure 2 · p. 4, Figure 3 · p. 4, §3 · p. 5
  • The reported pattern also holds when teachers and students come from different model families, though the Qwen2.5 Math 7B case is an outlier where RL underperforms distillation.Llama and Gemma base models use a Qwen teacher in the cross-family experiments.Source: §3 · p. 4
  • The proposed explanation is that distillation’s mode-covering signal spreads probability across more answers, while mode-seeking RL concentrates mass on a subset of correct answers.Distillation can also raise probability on incorrect answers; RL may lower probability for some low-probability solutions and is more limited by the base model’s initial distribution.Source: §3 · p. 5, Figure 4 · p. 5

4 RL-FREE EVALUATIONS CAN GIVE A FALSE SENSE OF SECURITY

Evaluating defenses only after distillation can give a false sense of security because subsequent RL may erase the performance gap they created. In the antidistillation-sampling case study, low-to-mild poisoning loses much of its effect after RL, while sufficiently high poisoning remains detrimental.

4 RL-FREE EVALUATIONS CAN GIVE A FALSE SENSE OF SECURITY

  • Defense evaluations that omit RL may overstate effectiveness because performance gaps can vanish after subsequent reinforcement learning.The paper presents antidistillation sampling as a case study of this broader evaluation concern.Source: §4 · p. 5
  • Antidistillation sampling perturbs teacher outputs to poison a distilled student; stronger poisoning further degrades the student but also degrades the teacher’s outputs.The case study uses Qwen2.5-0.5B students and Qwen2.5-1.5B-RL as teacher, with poisoning levels reducing teacher performance by 10%, 30%, and 86%.Source: §4 · p. 5
  • After RL, unpoisoned and low-to-mildly poisoned students have almost the same performance, and perform better than RL without prior distillation.Sufficiently high poisoning retains a performance cost, effectively because the student has a very bad teacher.Source: §4 · p. 5

5 RL MAKES SIMPLE DISTILLATION ATTACKS EFFECTIVE

The paper tests whether attackers can recover useful reasoning traces from model-provided summaries, then use reinforcement learning (RL) to strengthen a distilled model. After RL, models distilled on expanded summaries perform similarly to models distilled on full traces, including in closed-source settings.

5 RL MAKES SIMPLE DISTILLATION ATTACKS EFFECTIVE

  • Summaries condense reasoning semantics, which a weak expander can use with final answers to reconstruct semantically similar traces.The paper motivates this with the example that useful-step hints make a math problem easier to solve than starting from scratch.Source: §5 · p. 6
  • A weak expander can turn a model’s summarized reasoning and answers into approximate traces for distillation instead of hidden teacher traces.The reconstructed traces need only be similarly useful for bootstrapping, not faithful to the hidden reasoning.Source: §5 · p. 6, Figure 6 · p. 6
  • The open-source test distills Qwen2.5-14B-RL into Llama-3.2-3B-Base, using Llama-3.2-3B-Instruct as the expander.Summaries were produced from teacher traces with Qwen2.5-7B-Instruct; Llama was selected in part because it differs from the teacher’s model family.Source: §5 · pp. 6–7
  • After RL, expanded-summary distillation has essentially no performance gap from full-trace distillation, despite lower performance immediately after distillation.The comparison focuses on Minerva and MATH500; expanded summaries sometimes lead to more forgetting on GSM8K.Source: Figure 7 · p. 7, §5 · p. 7

5.1 STEALING CLOSED-SOURCE REASONING

  • The closed-source attack uses API-provided summaries, expands them into approximate traces, then distills and RL-trains the attacker’s model.The tested teachers were Claude Sonnet 4.6, GPT-5 mini, and Gemini Flash 3.6; the authors did not demonstrate an attack against frontier models.Source: §5 · p. 6, §5.1 · p. 7
  • After RL, expanded-summary distillation recovers performance similar to full-trace distillation across the tested closed-source models.Gemini lacks a full-trace baseline, so that comparison is not direct for Gemini.Source: §5.1 · p. 7, Figure 8 · p. 8

6 POTENTIAL DEFENSES

The paper argues that response-level defenses face a trade-off: preserving useful output semantics can leave traces reconstructible, while removing information can harm usability. It identifies batch-level defenses as a promising direction, though real-time detection across distributed queries remains difficult.

6.1 RESPONSE-LEVEL DEFENSES

  • Response-level defenses are likely ineffective when processed outputs retain enough semantic information to reconstruct approximate traces useful for subsequent RL.The more information the processed traces retain, the easier reconstruction becomes.Source: §6.1 · p. 8
  • Removing output information can worsen user experience, since withholding proof steps burdens legitimate researchers as well as attackers.The authors argue this trade-off can make such defenses impractical amid competition among model providers.Source: §6.1 · p. 8
  • Antidistillation levels that reduce a distilled model’s performance typically also significantly degrade the teacher, making deployment impractical.The authors also caution that subtle text-pattern defenses can be brittle when outputs are rephrased.Source: §6.1 · pp. 8–9
  • Domain-specific safeguards can be sensible where the security-usability trade-off is acceptable, as in reported cybersecurity-query blocking.The paper cites safeguards preventing Claude Fable 5 from answering cybersecurity queries and a reported thwarted distillation attempt.Source: §6.1 · p. 9
  • Summaries may still mitigate leaks of sensitive information such as passwords and API keys, although attackers must postprocess the outputs.The paper distinguishes this use from preventing reasoning-capability distillation.Source: §6.1 · p. 9

6.2 BATCH-LEVEL DEFENSES

  • Related queries may be easier to identify in batches, but distributed accounts and locations complicate real-time detection.The paper says reported batch-level defenses appear to detect attacks only in hindsight.Source: §6.2 · p. 9
  • If implemented successfully, batch-level defenses could address vulnerabilities beyond distillation, including reward-model use and automated jailbreak searches.The paper cites Boundary Point Jailbreaking as an example of an attack using many queries.Source: §6.2 · p. 9

7 LIMITATIONS

The experiments cover relatively small models and math reasoning, leaving uncertainty about how results transfer to larger models and other tasks. The proposed attack is unoptimized, and its trace-expansion step is described as inexpensive relative to RL and likely other training-data acquisition routes.

7 LIMITATIONS

  • All experiments use relatively small models, and whether larger models’ traces differ qualitatively in ways that affect distillation remains unknown.The authors note documented attacks against smaller closed-source models and studies in which distillation helps bootstrap subsequent RL.Source: §7 · p. 9
  • The experiments focus on math, while transfer to verifiable tasks such as long-horizon agentic coding remains untested here.The authors believe similar attacks may be viable in agentic setups that expose summaries and intermediate answers.Source: §7 · p. 9
  • The attack is unoptimized, so it demonstrates that a simple API-based approach may suffice rather than establishing best achievable gains or efficiency.More performant attacks could yield larger improvements or require less from the attacker; expander postprocessing is described as cheap relative to RL.Source: §7 · p. 9
  • The code for the experiments is available, except for the code used to extract full reasoning traces from closed-source models.The paper says reproduction details appear in the main text and appendices.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 IMPLEMENTATION DETAILS

The appendix documents training and evaluation settings, then reports per-dataset comparisons and explains the Qwen2.5-Math-7B exception. Its theory contrasts mode-seeking RL with mode-covering distillation and characterizes sequence-level distillation as a noisy approximation to regular knowledge distillation.

A.1 IMPLEMENTATION DETAILS

  • The experiments use Qwen2.5, Llama3, and Gemma base models, shared question datasets and prompts across training methods, and specified distillation and RL setups.Compute limits restricted the authors’ own RL training to models with at most 3B parameters; evaluation covers GSM8K, Minerva Math, and MATH500.Source: §A.1 · p. 17
  • Evaluations use T = 0 and math-verify for answer correctness; pass@k uses 256 sampled answers per question at T = 1.Source: §A.1 · p. 17

A.2 PER-DATASET BREAKDOWN

  • Per-dataset pass@k curves are qualitatively similar to their average across three models and GSM8K, Minerva, and MATH500.Source: §A.2 · p. 17, Figure 9 · p. 18

A.3 EXPLAINING THE QWEN2.5-MATH-7B OUTLIER

  • Distillation beats RL only on Qwen2.5-Math-7B, whose Python-code reasoning and hallucinated outputs are reinforced by RL but removed by distillation.The Qwen2.5-7B base model does not share this limitation; its code-reasoning behavior is eliminated by RL.Source: §A.3 · p. 17, Figure 10 · p. 19, Figure 11 · p. 19

A.2 PER-DATASET BREAKDOWN

  • Table 1 reports that RL outperforms distillation, while distillation followed by RL outperforms either alone.Source: Table 1 · p. 18

A.4 THEORY

  • Sequence-level knowledge distillation is a noisy approximation to regular mode-covering distillation, with an unbiased estimate of its gradient.The argument assumes T = 1 and unmodified teacher sampling.Source: §A.4 · p. 20
  • For a perfect teacher, max-entropy RL is equivalent up to scaling to reverse-KL distillation and is therefore mode-seeking.In the hard-teacher limit, the teacher is uniform over correct answers and the corresponding RL limit is regular reward maximization.Source: §A.4 · pp. 20–21
  • Mode-seeking RL can concentrate on some correct-answer modes, whereas mode-covering distillation may interpolate between modes and assign probability to incorrect answers.This offers an explanation for lower pass@1 under distillation and why distillation may raise pass@k at high k on problems with initially low probability of correct answers.Source: §A.4 · p. 21

A.5 DISTILLATION ABLATIONS

The ablations generally produce only marginal gains and do not make distillation outperform RL. This pattern holds across rejection sampling, teacher and dataset choices, training duration, and larger-scale comparisons.

A.5 DISTILLATION ABLATIONS

  • Across the ablations, performance gains are at most marginal and do not change the qualitative conclusions.The main results use the simplest setup without ablations.Source: §A.5 · p. 21
  • Rejection sampling with eight teacher attempts improves average accuracy by roughly 1% at best, but still does not surpass RL; 32 or 64 attempts are marginally worse.Source: §A.5 · p. 21, Table 2 · p. 22
  • Teacher traces generated at T = 1 marginally outperform T = 0 traces, so the experiments use T = 1.Source: §A.5 · p. 21
  • Using a smaller teacher marginally improves distillation, but students still do not exceed their RL-trained counterparts.The study primarily uses a larger teacher as more representative of attacks in which the closed-source model is more capable than the student.Source: §A.5 · pp. 21–22
  • A harder dataset yields up to +2% on Qwen2.5-0.5B, but alternative AIME, GSM8K, and Simple Test Time Scaling trace datasets reduce performance, and none beats RL.The main comparison uses the same dataset for distillation and RL.Source: §A.5 · p. 22, Table 4 · p. 23
  • Sampling teacher traces three or eight times per question or training for two to five epochs marginally harms performance, motivating one sample and one epoch.Source: §A.5 · p. 22
  • On Qwen2.5-7B, a smaller teacher slightly improves distillation, while alternative teachers or the s1K dataset lower its performance.Source: Table 3 · p. 22
  • RL outperforms distillation at 32B and with different teacher and student families, although the 32B SimpleRL-Zoo run may use questions that are too easy.Source: §A.5 · pp. 21–22, Table 5 · p. 23

B.1 IMPLEMENTATION DETAILS

The antidistillation experiments apply poisoning to teacher traces before distillation, using a proxy student and three poisoning levels.

B.1 IMPLEMENTATION DETAILS

  • The experiments distill Qwen2.5-0.5B on the medium SimpleRLZoo dataset using Qwen2.5-1.5B-RL traces, with or without antidistillation sampling.Source: §B.1 · p. 23
  • Antidistillation sampling uses a universal proxy student, configured by default as Qwen2.5-3B, to select teacher tokens that increase an attacker’s downstream loss.Source: §B.1 · p. 23
  • The poisoning-strength parameter λ is set to 0.0316, 0.05, or 0.18 for low, medium, and high poisoning, respectively.The framework’s values are used to calculate relative teacher-performance degradation.Source: §B.1 · p. 23

B.2 DETAILED RESULTS

The detailed results break down Figure 5 by dataset and report how poisoning levels relate to post-RL performance.

B.2 DETAILED RESULTS

  • Table 6 provides the per-dataset breakdown of Figure 5’s results.Source: §B.2 · p. 23
  • After RL, low or mild poisoning leaves the distilled Qwen2.5-0.5B model matching the unpoisoned distilled model’s performance.Source: Table 6 · p. 24
  • High poisoning lowers post-RL performance for the distilled Qwen2.5-0.5B model.Source: Table 6 · p. 24

B.3 ADDITIONAL EXPERIMENTS

The additional antidistillation experiment tests whether poisoning degrades student performance after distillation and reinforcement learning. The attack details describe trace generation and expansion, and validate extracted closed-source traces against provider-reported lengths.

B.3 ADDITIONAL EXPERIMENTS

  • At least 10% relative teacher-performance degradation was required before the student’s post-distillation performance noticeably fell, making the defense unlikely to be practical; low and mild poisoning also degraded performance after RL.The authors caution that post-RL drops at low and mild poisoning may simply reflect having a worse teacher.Source: §B.3 · p. 24, Table 7 · p. 24

C.1 IMPLEMENTATION DETAILS

  • The attack uses medium SimpleRLZoo questions with Llama-3.2-3B; open-source traces are summarized and expanded, while closed-source summaries and answers are first synthesized into reasoning breakdowns and then expanded.Closed-source API summaries came from GPT-5 mini and Gemini Flash 3.6 at high reasoning effort and Claude Sonnet 4.6 with an 8192-token thinking budget; full traces were also extracted from GPT-5 mini and Claude Sonnet 4.6.Source: §C.1 · p. 25

C.2 EXTRACTING FULL REASONING TRACES FROM CLOSED-SOURCE MODELS

  • Full traces were extracted by attaching providers’ opaque reasoning items to later requests, allowing same-family decoder models to return the hidden reasoning verbatim.The reported extraction used GPT-5.6-luna for GPT-5 mini and Claude Haiku 4.5 for Claude Sonnet 4.6; the Azure endpoint was unpatched at the time of writing, and the providers and Microsoft had been informed before submission.Source: §C.2 · p. 25
  • More than 99% of extracted traces matched provider-reported token lengths across all three settings, including Claude Sonnet 4.6 at maximum effort, where median trace length was 430 versus 175 tokens.Figure 12 plots extracted against reported lengths for individual SimpleRL-Zoo training questions; counts were accepted within max(3 tokens, 1%) for Claude or one 64-token bin for GPT-5 mini.Source: §C.2 · p. 26, Figure 12 · p. 26

C.3 ABLATIONS

The ablations test what drives the attack’s post-RL gains and how those gains depend on trace expansion, training conditions, and model capability. They also provide qualitative examples and the prompts used to summarize and expand reasoning traces.

C.3 ABLATIONS

  • Post-RL gains on hard math questions came from semantic information in reasoning summaries, not from the expander model alone.Using the expander as a teacher without summaries gave no benefit; full traces and expanded summaries are shown for reference in Figure 13.Source: §C.3 · p. 26, Figure 13 · p. 26
  • Expanded summaries of GPT-5 mini and Gemini Flash 3.6 sometimes slightly worsened GSM8K after RL, but a second training round with distillation and RL on hard SimpleRLZoo recovered the lost performance.After that round, expanded-trace models recovered GSM8K performance, while a small gap from full-trace models emerged; Table 8 also indicates higher performance remained possible.Source: §C.3 · pp. 26–27, Table 8 · p. 27
  • Claude Sonnet 4.6 traces collected at maximum reasoning effort yielded marginally better post-RL performance than traces collected with an 8192-token budget, at roughly 2.6 times the API cost.Figure 14 includes expanded summaries as a reference condition.Source: §C.3 · p. 28, Figure 14 · p. 28
  • Qwen2.5 results were unchanged by prior distillation, while Gemma-2B and Falcon3-3B models performed poorly before RL, limiting what these additional model-family tests establish.The authors suggest Qwen2.5 may be susceptible to spurious RL rewards, and say testing larger-scale setups remains important; compute limited RL training to models with at most 3B parameters.Source: §C.3 · p. 28
  • For closed-source full-trace distillation, forcing thinking by appending an opening think tag to the prompt produced more capable models after RL than omitting it.The main paper reports the forced-thinking setup; Table 9 compares distillation with and without it.Source: §C.3 · p. 28, Table 9 · p. 28
  • Distilling directly on open-source summaries improved post-distillation performance, but those gains did not persist after RL; directly using summaries produced effectively no post-RL improvement.Figure 15 compares this condition with RL without distillation, full traces, and expanded summaries.Source: §C.3 · pp. 28–29, Figure 15 · p. 29

D EXAMPLES OF REASONING TRACES

  • The examples show poisoned teacher traces at increasing poisoning levels and illustrate that open-source summarization removes much non-essential trace information, while closed-source summaries differ stylistically.Source: D EXAMPLES OF REASONING TRACES · p. 29

E PROMPTS

  • The prompts document the pipeline: summarize open-source traces, expand summaries, and first synthesize closed-source summaries with final answers into a reasoning breakdown for expansion.Source: E PROMPTS · p. 29
Loading 2609.35699v1…