PaperlayerView exact revision on arXiv

Source-linked AI summary

Reference-Based Distillation Detection in LLMs

Rajat Rawat, Sizhe Chen, Akshay Anand, Michael Duan, Bob Rotsted, Sewon Min

arXiv:2607.09692v1cs.LGcs.CL

TL;DR

Determining whether a model was trained on another model’s outputs is difficult when pipelines are opaque and lineage is entangled. The paper compares student and reference-model alignment with candidate-teacher outputs, then evaluates teacher recovery across controlled and real-world settings. It reports near-perfect teacher recovery and presents the work as an initial step toward practical distillation auditing.

  • Problem

    Determining whether a model was distilled from a teacher, and identifying that teacher, is challenging because membership inference is difficult for modern LLMs and distillation pipelines may be opaque.

    Source: §1 · p. 2

  • Method

    The method compares how strongly a student aligns with candidate-teacher outputs relative to an earlier-generation reference checkpoint.

    Source: §1 · p. 2

  • Results

    Across controlled and real-world settings, the approach recovers the true teacher with near-perfect accuracy and remains robust under domain shift.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The work provides an initial step toward practical auditing of distillation in modern LLMs.

    Source: §8 · p. 13

    The evaluation targets single-teacher, final-stage SFT distillation and does not address multi-teacher or multi-stage pipelines.

    Source: §8 · p. 14

Abstract

from arXiv · show

Model distillation -- training on outputs from stronger third-party models -- is widely used to boost performance, but raises concerns about unfair advantages and policy violations. This motivates a fundamental question: can we detect whether a model was distilled from another? We show that, while identifying a teacher model from a student in isolation is highly challenging, it becomes tractable in a reference-based setting: given a model and an earlier-generation checkpoint from the same lineage, we can identify the teacher model used to train the later checkpoint. We introduce a distillation detection method based on reference-based membership inference. By comparing how strongly a student model preferentially aligns with outputs from different candidate teachers relative to a reference checkpoint, our method identifies the most likely teacher and detects evidence of distillation. To handle unknown distillation pipelines such as hidden prompts, we infer proxy prompt templates directly from model outputs. We additionally identify a distinctive glyph-level signal specific to o1/o3 models. Evaluating distillation detection is challenging because modern model lineages are already heavily entangled. To address this, we develop a hybrid evaluation spanning both controlled distillation experiments and real-world models. Across both settings, our approach recovers the true teacher with near-perfect accuracy in single-teacher distillation scenarios, even when the underlying distillation pipeline is largely unknown. We further introduce statistical tests for both teacher attribution and distillation detection, and extend our framework to open-world settings where no teacher is guaranteed to be present among the candidates. Applying our method to contemporary models yields new evidence regarding potential distillation relationships involving QwQ, DeepSeek-R1, and GPT-OSS.

1 Introduction

Distillation is widely used to transfer model capabilities, but can enable competitors to reproduce them and raises concerns about unfair advantage and policy violations. This work makes teacher attribution more tractable by comparing a model with an earlier checkpoint from the same lineage.

1 Introduction

  • A reference checkpoint makes teacher attribution more tractable than identifying distillation from a student alone.The proposed setting uses an earlier-generation checkpoint from the same lineage to determine whether a teacher was used and, if so, which one.Source: §1 · p. 2
  • The method identifies candidate teachers through contrastive, reference-normalized likelihoods, rather than relying on absolute student scores.It also supports black-box access, proxy prompt inference for hidden-prompt settings, and a per-probe significance test.Source: §1 · p. 2
  • A complementary glyph-level signal uses unusually frequent non-ASCII characters to indicate o1/o3 distillation in students.The signal is a reference-normalized loss gap between ASCII and Unicode serializations.Source: §1 · p. 2
  • Across controlled and real-world evaluations, the approach recovers the true teacher with near-perfect accuracy, including under domain shift.The evaluation addresses entangled model lineages and uses agreement across two prompt sets to separate distillation from coincidental similarity.Source: §1 · p. 2
  • The framework extends to open-world teacher attribution and examines potential distillation involving QwQ, DeepSeek-R1, and GPT-OSS.The open-world setting does not assume a teacher is among the tested candidates.Source: §1 · p. 2
  • The paper frames model-level distillation auditing as an initial step toward transparency and accountability, while identifying multi-teacher and multi-stage attribution as future work.The study focuses on a simplified setting.Source: §1 · p. 2

2 Related Work

Prior work addresses distillation and its detection through several distinct approaches, including provider-side monitoring, watermarks, and behavioral methods. The paper positions its contribution as reference-based detection from model behavior, a direction it describes as previously unstudied for LLMs.

2 Related Work

  • Distillation has been used to transfer capabilities to LLMs, including in models such as Phi [1] and Alpaca.Prior work reports stronger performance gains when the teacher is a reasoning LLM [13].Source: §2 · p. 3
  • Existing industry detection relies primarily on request-side monitoring and infrastructure signals, while watermark-based detection requires provider-side access and can be weakened during downstream training.The cited examples include request metadata and IP-address correlation, and watermarks [20] that may be weakened or removed [16].Source: §2 · p. 3
  • Other related work studies prompt identification in distillation [30] and detection in image and text-to-image models [23] [31] [29].The latter includes synthesized-input scoring, prompt-level fingerprints, and shadow-model provenance verification.Source: §2 · p. 3
  • The paper describes reference-based LLM distillation detection from model behavior alone as a gap in prior work.It distinguishes this from prior approaches targeting absolute, provider-level detection.Source: §2 · p. 3
  • Membership inference asks whether an example appeared in training, and reference output distributions have strengthened its signal [24] [5] [28].This paper applies that line of work to whether a model was trained on another model’s generations.Source: §2 · p. 3

3 Method

The method detects whether a student was distilled from a candidate teacher by comparing its response to teacher outputs against an earlier-generation reference. It combines teacher ranking, a thresholded detection margin, and statistical tests to assess whether the observed signal exceeds noise.

3 Method

  • The setup considers SFT-style distillation, in which a student is fine-tuned on teacher-generated outputs paired with prompts.The detector has logits access to the subject and candidate teachers, but not the distillation prompts or templates; it queries the subject with public prompts and its default chat template.Source: §3 · p. 3

3.1 Teacher Identification

  • Teacher identification ranks candidates using reference-normalized alignment, selecting the teacher with the highest mean score or the most rank-matched wins.Proxy inputs are queried against each teacher, and the resulting outputs are scored against the student; the reference should ideally be an earlier checkpoint less affected by candidate teachers.Source: §3.1 · p. 4
  • The alignment score measures how much better the student explains a candidate teacher’s output than the reference does.Absolute likelihood and similarity measures can favor outputs that are generally easy to model rather than specifically aligned with the student.Source: §3.1 · p. 4

3.2 Distillation Detection

  • Binary detection uses the gap between the top two candidate mean scores and selects a threshold on calibration data to decide whether a teacher is present.A model-level call additionally requires a significant margin on both prompt sets and the same teacher to lead in both.Source: §3.2 · p. 4

3.3 Statistical Significance

  • Teacher identification tests whether the known true teacher wins individual probes more often than expected under random ranking, using an exact one-sided binomial test.The null win probability is 1/K, where K is the number of candidates.Source: §3.3 · p. 5
  • Detection tests whether the paired per-probe margin exceeds the calibrated threshold, using a one-sided t-test or, when normality is rejected, a Wilcoxon signed-rank test.The procedure Bonferroni-corrects for selecting the top candidate and requires corrected significance on both prompt sets with the same teacher named.Source: §3.3 · p. 5

4 Experiment: Teacher Identification

The method identifies teachers with near-perfect accuracy in controlled experiments and correctly identifies the teacher for all seven evaluated real-world models, within an evaluation focused on single-teacher, final-stage SFT-style distillation.

4 Experiment: Teacher Identification; 4.1 Experimental Setup

  • Because modern model lineages are entangled, evaluation combines controlled experiments with real-world models; its stated scope is single-teacher, final-stage SFT-style distillation.Controlled pairs provide cleaner ground truth, while real-world models test robustness under unknown prompts, instructions, filtering, and training pipelines; multiple-teacher and multi-stage pipelines are deferred.Source: §4.1 · p. 5, §4 · p. 5

4.2 Results from Controlled Experiments; 4.3 Results on Real-World Distilled Models

  • Near-perfect controlled accuracy contrasts with inconsistent baselines, while the true teacher ranked first for all seven real-world targets and significantly exceeded chance on both probe sets.The controlled result spans 19 retained students; real-world targets comprise six R1-Distill models and s1.1-32B.Source: Table 1 · p. 6, §4.2 · p. 7, §4.3 · p. 8

4.2 Results from Controlled Experiments

  • After only 32 teacher-generated examples, the true teacher was identified for all three tested teachers, and identification remained stable through training.At 16 examples the student remained unchanged from the base reference because the first training step used a near-zero learning rate.Source: §4.2 · p. 7, Figure 3 · p. 8
  • Under highly customized instructions unknown to the detector, detection accuracy degraded, while few-shot prompting recovered the true teacher for three of four student models.The exception was Gemma-3-4B-PT; the strategy prepends student-generated prompt-response examples under the default template for each candidate teacher.Source: §4.2 · p. 7, Table 3 · p. 8
  • Performance was generally robust to reference choice and substantial proxy-prompt mismatch, although instruction-tuned students benefited from similar earlier-generation instruction-tuned references.The reported robustness includes cross-domain proxy prompts such as math versus code.Source: §4.2 · p. 8

4.3 Results on Real-World Distilled Models

  • For s1.1-32B, the CDF shows R1 clearly separated from other teacher candidates and ranked first by a large margin.The model was trained by [13].Source: §4.3 · p. 9, Figure 5 · p. 9
  • When R1 reasoning traces were hidden and only final answers were available, teacher-identification accuracy fell to zero.The authors identify closed reasoning models that expose only final answers or compressed summaries as a limitation for this approach.Source: §4.3 · p. 9

5 Experiment: Distillation Detection

The detector generalizes well to held-out students when their teacher is represented during calibration, but transfer to unseen teachers is less reliable. Real-world held-out students show a similar student-side pattern, with one miss caused by the both-prompts decision rule.

5 Experiment: Distillation Detection

  • A fixed threshold calibrated for maximum controlled accuracy transferred poorly to real-world cells, attaining 85.5% (65/76) on controlled cells but 53.6% (15/28) on real-world cells.The section therefore separates transfer to unseen students from transfer to unseen teachers.Source: §5 · p. 9

5.1 Experimental Setup

  • The evaluation uses controlled and real-world cells with teachers either included or removed from candidate pools, then refits the threshold within each cross-validation fold.Controlled experiments include leave-one-student-out and leave-one-teacher-out splits; real-world models use leave-one-student-out because all seven share DeepSeek-R1 as teacher.Source: §5.1 · pp. 9–10

5.2 Results

  • 82.9% (63/76) accuracy under leave-one-student-out fell to 60.5% (46/76) under leave-one-teacher-out, with false detections appearing when reasoning teachers were held out.Only three controlled teachers were evaluated; the authors describe the teacher-dependent signal explanation as suggestive rather than conclusive.Source: §5.2 · p. 10, Table 6 · p. 10
  • 92.9% (26/28) accuracy was attained on real-world cells under leave-one-student-out, with six of seven held-out students detected and zero false detections when DeepSeek-R1 was removed.The missed positive was s1.1-32B: its OMI margin fell below the fold-fit threshold, so the both-prompts rule withheld the call.Source: §5.2 · p. 10, Table 7 · p. 10

6 OpenAI o1 Detection Methods

The method compares reference-normalized losses for ASCII-escaped and Unicode-preserving serializations of o1 outputs. O1-distilled students show larger positive ASCII gaps than controls, though the authors treat the diagnostic as a cue rather than standalone evidence.

6 OpenAI o1 Detection Methods

  • The diagnostic compares reference-normalized loss for Unicode-preserving outputs against ASCII-escaped outputs, with positive ∆ASCII indicating higher loss under ASCII normalization.The evaluation used four o1-distilled students, 100 held-out OMI probes, and real U.S.-organization control models.Source: §6 · p. 11
  • Figure 6 shows nearly overlapping ASCII and Unicode CDFs for a non-distilled Gemma control, but consistent separation for an o1-distilled Gemma student.The table reports significant gaps for all four o1-distilled students and three Gemma-3 controls, whose gaps were much smaller.Source: Figure 6 · p. 11, §6 · p. 11, Table 8 · p. 11
  • All four o1-distilled students had significantly positive ∆ASCII gaps from 0.0573 to 0.0732, and the smallest distilled gap exceeded the largest control gap by over 50%.All four distilled-model confidence intervals lay above every control-model interval; the largest control gap was +0.0373.Source: §6 · pp. 11–12, Table 8 · p. 11
  • The authors treat the Unicode signal as a complementary cue rather than standalone evidence because the diagnostic uses only 100 OMI probes from one prompt set.With this probe count, significance alone can resolve tiny positive gaps from zero, making gap size more informative.Source: §6 · pp. 11–12

7 Open Questions

The paper probes open-world teacher attribution for QwQ-32B, DeepSeek-R1, and GPT-OSS using candidate pools and reference checkpoints. Results raise possible lineage connections, but the authors emphasize that these case studies are preliminary and especially uncertain for GPT-OSS.

7 Open Questions

  • The open-world probes ask whether QwQ-32B, GPT-OSS, and DeepSeek-R1 may derive from specified candidate families, using s1 and OMI probes and excluding near-variants of each target.The authors caution that these are preliminary explorations under simplifying assumptions, with real-world lineages potentially involving multiple teachers and stages.Source: §7 · p. 12
  • DeepSeek-R1 ranked highest for QwQ-32B across both probe sets, raising the possibility that R1 outputs trained QwQ-32B after QwQ-32B-Preview was released.The timeline given is QwQ-32B-Preview in November 2024, DeepSeek-R1 in January 2025, and QwQ-32B in March 2025.Source: §7 · p. 12, Figure 7 · p. 12
  • For DeepSeek-R1, o1 and QwQ-32B-Preview ranked near the top, while the Unicode–ASCII diagnostic showed a significant gap of +0.937.The reported 95% bootstrap interval was [+0.746, +1.141], with p = 8.5 × 10^-16; the authors say this suggests possible o1-style signal.Source: §7 · p. 12, Figure 8 · p. 13
  • GPT-OSS results rank DeepSeek-R1 and QwQ-32B-Preview highest and show a resolvable Unicode–ASCII gap, but the authors regard lineage conclusions as highly uncertain.GPT-OSS lacks a suitable same-family reference, and the authors note that o1's unavailable thinking traces may systematically underestimate its likelihood.Source: §7 · p. 13, Figure 9 · p. 13

8 Conclusion

The paper presents reference-based distillation detection as an initial step toward practical auditing of modern LLMs. Its conclusions are bounded by simplified distillation assumptions, reference availability, trace exposure, and unresolved attribution under rewriting pipelines.

8 Conclusion

  • Reference-based distillation detection identifies a likely teacher by comparing a model with an earlier-generation checkpoint from the same lineage.The conclusion describes the work as an initial step toward practical auditing of modern LLMs.Source: §8 · p. 13
  • The evaluation covers single-teacher, SFT-style final-stage distillation, not multi-teacher or multi-stage pipelines, and identification depends on a suitable reference checkpoint.The method is most reliable for teachers that expose full reasoning traces.Source: §8 · p. 14
  • The authors leave unresolved whether rewritten teacher generations lead attribution to the original generator, the rewriter, a mixture, or neither.The detected signal may track either source or rewriter depending on how much stylistic and distributional structure survives rewriting.Source: §8 · p. 14

A Experimental Details

The controlled experiments create distilled students from several base models and teacher-generated datasets, then evaluate how well candidate teachers can be identified. The setup also standardizes training and evaluation conditions and compares reference-based MIA with alternative detection baselines.

A Experimental Details

  • The experiments use OMI-2 and s1 prompts, and report teacher output-length statistics because teacher verbosity varies substantially, especially with reasoning traces.For models with explicit reasoning traces, final-output-only lengths are also reported separately.Source: A Experimental Details · pp. 17–18
  • Four base students and four candidate teachers define the teacher-identification pool, while the controlled setup produces 24 distilled models and retains 19 that improve over their base models.The candidates are Llama-3.3-70B-Instruct, Qwen3-8B, Gemma-3-27B, and GPT-OSS-120B; retained models outperform their corresponding bases.Source: A Experimental Details · p. 17
  • Reference-based MIA is compared with raw likelihood, ZLIB-normalized likelihood, Min-k% probability, similarity-based scoring, and reverse MIA.The reverse variant scores student-generated outputs under each candidate teacher.Source: A Experimental Details · p. 17
  • Teacher identification is evaluated with sorted per-sample comparisons and model-level decisions based on mean alignment or the most frequent sorted-score winner.The aggregate accuracy reports the higher of the two model-level decision-rule accuracies.Source: A Experimental Details · p. 17
  • Students are fine-tuned for three epochs with a learning rate of 1e−5, effective batch size 16, and response-token-only loss.Teacher responses are capped at 2,048 tokens, and the same truncation is applied when evaluating reference-based MIA scores.Source: A Experimental Details · p. 18

B.1 Ablations on the choice of reference model (R)

The ablations examine sensitivity to the reference model and proxy prompts, compare reverse-MIA baselines, and test few-shot prompting with customized templates. Results indicate robustness to prompt mismatch, while reference-model choice and student-specific separation remain relevant.

B.1 Ablations on the choice of reference model (R)

  • Reference-model choice is reasonably robust, but using a base rather than instruction-tuned reference degrades accuracy, suggesting that matching the student’s instruction tuning matters.The ablation uses Llama 3.2 3B Instruct students.Source: §B.1 · pp. 18–19, Table 11 · p. 19

3. Different-domain: X and ˆ

  • The method remains accurate with code prompts for math-distilled models, suggesting that the teacher-identification signal is not limited to the distillation prompt distribution.The evaluation uses MBPP code prompts as proxy inputs.Source: §3 · p. 19

B.2 Ablations on proximity in input prompts ( ˆ

  • Proxy prompts are tested in identical, similar-domain, and different-domain settings; detection accuracy is relatively stable across these choices.The different-domain setting probes math-distilled models with code-generation prompts.Source: §B.2 · p. 19, §3 · p. 19, Table 12 · p. 20

B.3 Ablations for Reverse MIA methods

  • Reverse likelihood variants perform well for Llama 3.3 70B Instruct but do not reliably identify Qwen-3-8B or GPT-OSS-120B, unlike reference-based MIA across teachers.The comparison reports teacher-identification accuracy by true teacher, including aggregate and per-sample metrics.Source: Table 13 · p. 20

C Detecting Distillation with a Highly Customized Prompt template

D Model in the Wild Plots

Reference-based MIA is evaluated on real-world distilled models whose training pipelines are unknown. The reported teacher signal persists across the evaluated candidates and in an out-of-domain test of a code-specialized model.

D Model in the Wild Plots

  • Across the evaluated real-world models, the true teacher generally receives the strongest alignment signal despite unobserved prompts, filtering, decoding, and post-training details.The evaluation includes DeepSeek-R1 distill models, s1.1-32B, and XCoder, using corresponding pre-distillation base models as references.Source: D Model in the Wild Plots · p. 22, Figure 11 · p. 22, Figure 12 · p. 22
  • XCoder retains a teacher-specific signal when probed with math data despite being code-specialized, suggesting attribution is not limited to models trained directly on math.The probes use s1 and OMI data.Source: D Model in the Wild Plots · p. 22, Figure 17 · p. 24

E Additional o1 Detection Details

The additional analyses examine an ASCII–Unicode loss diagnostic, stylistic markers, and open-question results. Large positive ASCII gaps appear for the tested real-world models, while the open-question evidence is explicitly diagnostic rather than conclusive.

E Additional o1 Detection Details

  • The diagnostic compares reference-normalized loss for Unicode and escaped-ASCII serializations; a positive ∆ASCII indicates that ASCII serialization raises the target’s loss.The method holds other hyperparameters fixed, and interprets a positive gap as evidence that encoding contributes to the membership signal.Source: E Additional o1 Detection Details · p. 24
  • Within-family controls cluster near zero: most ASCII gaps fall within ±0.01, with the largest deviation at −0.0733 for gemma-2-9b-it.For non-o1-distilled Llama controls, ASCII and Unicode reference-normalized loss distributions nearly overlap.Source: E Additional o1 Detection Details · p. 25, Figure 18 · p. 25, Table 14 · p. 25
  • +0.937 ∆ASCII for DeepSeek-R1 (DeepSeek-MoE-16B-Base reference); GPT-OSS-20B has +0.5103 and GPT-OSS-120B +0.8310 (GPT-2 XL reference), with +0.3207 for GPT-OSS-120B using GPT-OSS-20B.These real-world gaps are substantially larger than the +0.0573 to +0.0732 gaps of controlled o1-distilled positives.Source: E Additional o1 Detection Details · pp. 25–26, Table 14 · p. 25
  • Llama-3.3-70B-Instruct uses the ## Step N marker an average of 7.26 times per response, but customized prompts can suppress such stylistic artifacts.The authors leave systematic study of robustness to prompt variation and integration with likelihood scores to future work.Source: E Additional o1 Detection Details · p. 26, Table 15 · p. 26

F Additional Open Question Plots

  • Open-question results are diagnostic, not conclusive evidence of distillation, because target training data, prompts, filtering, and procedures are unknown.A significant Unicode–ASCII gap is suggestive of an o1-style signal, not a confirmed distillation call.Source: F Additional Open Question Plots · p. 26, Table 16 · p. 27

Related papers

Loading 2607.09692v1…