PaperlayerView exact revision on arXiv

Source-linked AI summary

Who Taught You That? Tracing Teachers in Model Distillation

Somin Wadhwa, Chantal Shaib, Silvio Amir, Byron C. Wallace

arXiv:2502.06659v3cs.CL

TL;DR

The paper asks whether a distilled student’s teacher can be inferred from its outputs, motivated by model provenance and possible proprietary-model distillation. It evaluates black-box attribution methods across practical tasks and finds that PoS templates provide stronger signal than shallow similarity, although accuracy remains imperfect.

  • Problem

    The paper asks whether a distilled student’s teacher can be identified from its outputs, with implications for tracing proprietary-model distillation.

    Source: §1 · p. 1

  • Method

    It compares perplexity, text similarity, and PoS-template classifiers for black-box teacher attribution across practical distillation tasks.

    Source: §1 · p. 1, §3 · p. 2, §3.2 · p. 3

  • Results

    PoS templates provide comparatively strong teacher signal across tasks and datasets, while basic similarity and perplexity are insufficiently reliable.

    Source: §1 · pp. 1–2, §5 · p. 5

  • Takeaways & Limitations

    Syntactic patterns that persist in student outputs can support teacher attribution without access to teacher internals or watermarking.

    Source: §5 · pp. 4–5

    Attribution depends on retaining teacher-specific patterns and remains far from perfect, with additional training or shared teacher data potentially obscuring signals.

    Source: Limitations · p. 5

Abstract

from arXiv · show

Model distillation -- using outputs from a large teacher model to teach a small student model -- is a practical means of creating efficient models for a particular task. We ask: Can we identify a students' teacher based on its outputs? Such "footprints" left by teacher LLMs would be interesting artifacts. Beyond this, reliable teacher inference may have practical implications as actors seek to distill specific capabilities of massive proprietary LLMs into deployed smaller LMs, potentially violating terms of service. We consider practical task distillation targets including summarization, question answering, and instruction-following. We assume a finite set of candidate teacher models, which we treat as blackboxes. We design discriminative models that operate over lexical features. We find that $n$-gram similarity alone is unreliable for identifying teachers, but part-of-speech (PoS) templates preferred by student models mimic those of their teachers.

1 Introduction

The paper asks whether a distilled student's teacher can be identified from output footprints, motivated by practical attribution and provenance concerns. It finds that raw similarity and teacher perplexity are unreliable, whereas PoS templates provide stronger teacher signal.

  • The paper frames teacher attribution as identifying which candidate model trained a distilled student from the student’s outputs.The setting includes practical task distillation and black-box candidate teachers.Source: §1 · p. 1, Figure 1 · p. 1
  • Distillation can transfer task-specific capabilities into efficient students, but identifying those teacher footprints may matter when proprietary capabilities are replicated.The paper notes possible terms-of-service implications for providers seeking to detect such cases.Source: §1 · p. 1
  • Basic similarity measurements over student and teacher texts provide practically no useful attribution signal.This motivates looking beyond shallow lexical overlap.Source: §1 · p. 1
  • Teacher perplexities also fail to reliably select the true teacher from a candidate set.The expected preference for a pupil’s outputs does not consistently appear.Source: §1 · p. 2
  • PoS templates, which abstract away from raw output text, carry relatively strong signal about the teacher used in distillation.The paper treats these syntactic patterns as higher-order lexical features.Source: §1 · p. 2

2 Problem Setup

The task is to infer which teacher in a finite candidate set trained a student without access to the original distillation data. Experiments use open teachers, smaller student models, and summarization, question answering, and instruction-following tasks.

  • The attribution problem is to identify which candidate teacher M_j trained student m without access to the original distillation data.The candidate set is M = {M_1, M_2, . . . , M_T}.Source: §2 · p. 2
  • Experiments use fine-tuned GPT-2 and Olmo-1B students with five open teachers ranging from 8B to 70B parameters.The teacher set includes Llama3-8B, Llama3-70B, Mistral-7B, Mixtral, and Gemma2-9B.Source: §2 · p. 2
  • The evaluated task families are summarization, question answering, and general instruction following.Datasets include CNN-DailyMail, Rotten Tomatoes, PubMed, OpenbookQA, CommonsenseQA, and a 10K Alpaca sample.Source: §2 · p. 2
  • Training data is generated from all five teachers, with direct outputs for summarization and instruction following and reasoning chains plus labels for question answering.The question-answering construction follows prior work described in the passage.Source: §2 · p. 2

3 Teacher Attribution Methods

The paper compares perplexity, output similarity, and syntactic-pattern classifiers for attributing teachers. Similarity-based signals are weak, while PoS templates generally outperform n-grams and BERT-based representations.

3 Teacher Attribution Methods

  • The methods evaluate three attribution families: perplexity, similarity metrics, and syntactic patterns.The comparison targets teacher identification from sets of student outputs.Source: §3 · p. 2
  • Perplexity does not reliably identify the teacher because candidate teachers do not consistently assign the lowest perplexity to outputs from their own pupils.The passage gives Gemma’s unusually high perplexity on summaries from a model distilled from its outputs as an example.Source: §3 · p. 2, Figure 2 · p. 3

3.1 Similarity Metrics

  • Raw student-teacher similarities are unreliable, and similarity-only logistic regression achieves an average AUC of about 0.52.The reported AUC range is approximately 0.49–0.53, close to random classification.Source: §3.1 · p. 3, Figure 3 · p. 3

3.2 Syntactic Patterns

  • PoS-template features encode recurring length-4 syntactic patterns and are used as inputs to logistic regression classifiers evaluated on student-generated test sets.The patterns are selected from the 50 most common patterns across teachers.Source: §3.2 · p. 3
  • PoS templates consistently outperform n-grams and BERT-based models for teacher classification across the reported datasets.For GPT-2 students, PoS accuracy reaches 0.68 versus 0.61 for n-grams on PubMed and 0.67 versus 0.48 on CommonsenseQA, with an Alpaca exception.Source: §3.2 · p. 3, Table 2 · p. 4
  • PoS templates provide stronger structural signals than simpler word-based representations, although n-grams slightly outperform them on Alpaca data.The reported Alpaca accuracies are 0.52 for n-grams and 0.48 for PoS templates.Source: §3.2 · pp. 3–4

4 Related Work

Related work covers rationale-based LLM distillation and model-origin tracing. The paper differs by studying linguistic features as teacher signatures rather than relying on watermarks or model probabilities.

  • Prior distillation studies show that rationales can improve student models and examine teacher-corpus construction, generalizability, and rationale placement.These works span explanation-based learning and chain-of-thought-augmented distillation.Source: §4 · p. 4
  • Origin-tracing research uses statistical tests, generation-time watermarks, perplexity, and contrastive training, whereas this paper investigates linguistic features as model signatures.Watermarking approaches may assume that student training data is watermarked.Source: §4 · p. 4

5 Conclusions

The paper frames teacher attribution as identifying which teacher trained a distilled student and introduces PoS templates to capture teacher-specific signals that persist in student outputs. These patterns provide comparatively strong cross-task signal without teacher internals or watermarking, while accuracy remains imperfect and practical implications unclear.

  • PoS templates capture distinctive teacher signals that persist in distilled student outputs.Source: §5 · pp. 4–5
  • Teacher attribution with syntactic patterns provides comparatively strong signal across tasks and datasets without requiring teacher internals or watermarking.Source: §5 · p. 5
  • Accuracy is well above the 0.2 chance level but leaves considerable room for improvement, and its practical implications remain unclear.Source: §5 · p. 5
  • The work lays a foundation for further teacher-attribution studies involving model behavior, usage-agreement compliance, and AI-system transparency.Source: §5 · p. 5

Limitations

The attribution framework is constrained by student retention of teacher patterns, imperfect predictive accuracy, and a closed candidate set. These limitations motivate studying additional distillation factors, complementary signals, and larger candidate spaces.

  • Attribution depends on whether students retain their teachers’ linguistic patterns, which additional fine-tuning, augmentation, multi-teacher distillation, or shared footprints may obscure.Source: Limitations · p. 5
  • Syntactic patterns outperform traditional similarity metrics but remain far from perfect, leaving considerable room for improved attribution accuracy.Source: Limitations · p. 5
  • Future work could combine semantic embeddings, paraphrase consistency, and latent-space representations or extend attribution to arbitrarily large candidate sets.Source: Limitations · p. 5
  • The approach assumes closed-set identification in which the true teacher belongs to a predefined candidate set.Source: Limitations · p. 5

A Implementation Details

Experiments used two NVIDIA A100 GPUs and publicly available Hugging Face model implementations. Training used fixed learning-rate and input-length settings, checkpoint evaluation with early stopping, and task-specific batch sizes.

  • Experiments ran on two NVIDIA A100 GPUs using publicly available Hugging Face implementations.Source: A Implementation Details · p. 8
  • All tasks used a 3e−5 learning rate, 1024-token maximum input length, checkpoint evaluation every 500 steps, and early stopping.Source: A Implementation Details · p. 8
  • Batch sizes were 12 for question answering, 2 for summarization, and 4 for instruction following, with default values for other hyperparameters.Source: A Implementation Details · p. 8

B Datasets

The evaluation spans summarization, question answering, and instruction-following datasets, covering news, biomedical, film-review, commonsense, science, qualitative-reasoning, and general instruction tasks.

Summarization

  • The summarization evaluation uses CNN-DailyMail, SumPubMed, and Rotten Tomatoes, covering news, biomedical, and film-review synthesis.CNN-DailyMail contains over 300,000 article-summary pairs, SumPubMed contains over 30,000 biomedical articles, and Rotten Tomatoes covers 9,095 movies and 244,000 reviews.Source: B Datasets · p. 8

Dataset coverage

  • CNN-DailyMail, SumPubMed, and Rotten Tomatoes represent distinct summarization settings, from bullet-point news summaries to structured biomedical abstracts and aggregated film reviews.Their source materials are online news articles, full-text research articles, and multiple critic reviews, respectively.Source: B Datasets · p. 8

Question answering

  • Question answering is evaluated with CommonsenseQA, OpenbookQA, and QuaRel, which require commonsense, elementary-science, and qualitative physical reasoning.The datasets use multiple-choice questions with five, four, and two answer choices, respectively.Source: B Datasets · p. 8

Instruction following

  • Instruction following is evaluated on a 10,000-instance downsample of Alpaca spanning reasoning, summarization, and open-ended instructions.The original dataset contains 52,000 synthetic instruction-response pairs generated from textdavinci-003; this version excludes programming-related code.Source: B Datasets · p. 8

C Additional Results

Additional support-level experiments compare BoW, n-gram, and PoS-template features across 50 to 2,000 test instances. PoS templates generally provide the strongest and increasingly stable teacher-classification signal, with Alpaca as a marginal exception.

Additional results

  • PoS templates consistently outperform BoW and n-grams across most datasets and support levels, with their advantage generally increasing as support grows.The evaluation uses logistic regression with 50, 200, 1000, and 2000 test instances per dataset.Source: C Additional Results · p. 9, Table 3 · p. 9
  • In SumPubMed, PoS-template accuracy rises from 0.68 at 50 instances to 0.72 at 2000 instances, exceeding n-grams and BoW at every support level.Source: C Additional Results · p. 9
  • In CommonsenseQA, PoS templates reach 0.69 accuracy, compared with 0.41 for BoW and 0.56 for n-grams.Source: C Additional Results · p. 9
  • Alpaca is a marginal exception: n-grams achieve 0.56 at 2000 instances versus 0.56 for PoS templates, while varying more at lower support levels.The accompanying interpretation suggests that richer lexical representations may benefit some datasets despite the broader structural-signal trend.Source: C Additional Results · p. 9
  • The extended analysis supports PoS templates as a scalable alternative to word-based representations and as distinctive syntactic signatures of teacher models.Source: C Additional Results · p. 9

Related papers

Loading 2502.06659v3…