Dataset distillation for memorized data: Soft labels can leak held-out teacher knowledge

• The motivation is that modern models combine generalizing structure with memorized facts, but prior explanations of soft labels largely emphasize structure and generalization.The paper frames transfer of memorized knowledge as an open question in distillation.Source: §1 · pp. 1–2
• The study isolates memorization by training teachers to fit finite input–label datasets, giving students only a partition of the data and evaluating them on the held-out remainder.It applies this protocol to small transformers on structured algorithmic tasks and fully connected networks on random i.i.d. data.Source: §1 · p. 2
• Students trained on teacher soft labels recover non-trivial—and sometimes perfect—accuracy on held-out memorized data they never observed.The result holds for both structured-but-memorized datasets and purely random i.i.d. data.Source: §1 · p. 2
• Soft-label effectiveness is strongly temperature-dependent, interpolating between fitting the teacher function and recovering only ground-truth training labels.This temperature dependence is interpreted as a regularization effect.Source: §1 · p. 2
• For random i.i.d. data, logistic regression exhibits capacity and identifiability thresholds, while ReLU MLPs transition from soft-label memorization to teacher matching once the teacher is identifiable.The ReLU transition is associated with a sudden jump in accuracy.Source: §1 · p. 2

Knowledge distillation

• Prior knowledge-distillation work attributes soft-label effectiveness to regularization or statistical structure, whereas this paper shows that soft labels can transmit memorized information beyond regularization.The paper studies matched-capacity teacher and student models on unstructured memorized data.Source: §2 · p. 3

Dataset distillation

• Dataset-distillation research suggests that supervision can matter more than input realism, and this paper finds that high temperatures can let students learn the teacher functionally.This occurs when the data is sufficient to identify the teacher.Source: §2 · p. 3

Memorization

• The paper extends results on arbitrary-label memorization by examining how memorized information transfers through soft-label distillation.Its random-data analysis excludes spurious correlations beyond those caused by dataset finiteness.Source: §2 · p. 3

• The finite teacher dataset is partitioned into disjoint student-training and student-test subsets, with the training fraction represented by ρ.Teacher validation data is drawn from the remainder or from an independent sample.Source: §3 · p. 3
• Models map x ∈ R^d to c class logits, use argmax predictions, and are trained with cross-entropy loss.The student and teacher are parameterized functions from inputs to class logits.Source: §3 · p. 3
• For structured experiments, transformer teachers are stopped at different training times on modular addition, producing varying degrees of memorization and generalization.The corresponding student experiments vary softmax temperature and report train, test, and validation performance.Source: Figure 2 · p. 4
• Knowledge transfer replaces one-hot labels with teacher soft labels obtained by applying softmax with temperature τ to teacher logits.Students are optimized with cross-entropy using these soft labels.Source: §3 · p. 4
• Student evaluation distinguishes training accuracy, held-out student-test accuracy, teacher validation accuracy, and student validation accuracy.These measures separate memorization of the student training set from teacher and student generalization.Source: §3 · p. 4

Experimental setup

• The experiment uses modular addition with p = 113, trains teachers on 30% of the 12,769-example distribution, and evaluates students on held-out memorized samples and validation data.Teachers are stopped at three points representing increasing discovery of task structure before soft-label distillation.Source: §4 · p. 5

Soft-label leakage

• At τ = 10, students can recover held-out memorized samples from transformer teachers, with test accuracy exceeding validation accuracy.The gap indicates teacher-training-set-specific information in the soft labels.Source: §4 · p. 5

Teacher generalization

• Students can perfectly generalize to held-out memorized data, but students distilled from non-generalizing teachers fail to generalize to validation data.Despite 5× longer training than the teacher, these students instead match the non-generalizing teachers’ validation behavior.Source: §4 · p. 5

Interpretation

• Soft labels can leak memorized inputs without supplying the latent structure that undertrained teachers have not discovered.This distinction separates recovery of teacher-specific memories from generalization on validation data.Source: §4 · p. 5

Temperature effects

• Higher temperatures improve data efficiency and convergence speed while increasing leakage of teacher-specific memorized information.At τ = 0.1, students can outperform the teacher or become teacher-independent, whereas τ = 10 enables rapid generalization from a generalizing teacher and function matching for memorizing teachers.Source: §4 · p. 5

5 Distillation for Data without a Latent Structure

• The controlled random-data model uses i.i.d. Gaussian inputs and uniformly random independent labels, so teachers must memorize finite datasets and cannot generalize a priori.The section analyzes multinomial logistic regression for closed-form recovery thresholds, extends the analysis to multiple classes, and tests nonlinear leakage with one-hidden-layer ReLU MLPs.Source: §5 · p. 5

5.3 Dataset Distillation for Finetuned GPT-2 Classifiers on Random Sequences

• A GPT-2 classifier fine-tuned on randomly associated token sequences and classes also achieves non-trivial held-out test accuracy, extending leakage beyond synthetic Gaussian inputs.The sequences concatenate three i.i.d. random integers, use 1000 possible classes, and the teacher dataset contains 6000 samples.Source: §5.1 · p. 6, §5.3 · p. 9

5.1 Multinomial Logistic Regression

• Multinomial logistic regression exhibits three sample-complexity regimes: low or no leakage, weak leakage, and full recovery, with thresholds for teacher memorization, teacher identifiability, and student memorization.The thresholds depend on the training fraction ρ and sample complexity α = n/d; finite-size results approach the infinite-dimensional limit, with α^S_label(d = 1600) ≃ 1.96 near α = 2.Source: §5.1 · pp. 6–7
• Students trained on logistic-regression soft labels obtain at least 55% held-out accuracy and can reach at least 99%, recovering memorized data by recovering teacher weights.The held-out data comprise a fifth of the memorized dataset in the full-recovery regime.Source: §5.1 · p. 7
• Temperature controls whether soft labels preserve teacher-function information or class identity: both extreme temperatures destroy information, while increasing τ shifts learning toward the teacher function.With τ →0 labels become one-hot, whereas τ →∞ makes soft labels uniform; the same leakage behavior persists for multiple classes and can reach 100% accuracy.Source: §5.1 · pp. 7–8

5.2 Two Mechanisms for Leaking Memorized Information in ReLU MLPs

• On random uncorrelated data, one-hidden-layer ReLU MLPs show non-trivial held-out leakage that increases with observed teacher data and can reach 100% accuracy.At higher sample complexity, held-out accuracy can first drop and then abruptly rise to 100%, unlike the simpler monotonic behavior seen at lower complexity.Source: §5.2 · p. 8
• ReLU students can either memorize the teacher’s soft labels or functionally match the teacher, and these solutions produce different leakage behavior.Which solution emerges depends on capacity thresholds, optimization, and the teacher-to-student capacity ratio; the transition to functional matching coincides with a sudden joint jump in train and test accuracy.Source: §5.2 · pp. 8–9

5.3 Dataset Distillation for Finetuned GPT-2 Classifiers on Random Sequences

• Students were trained for 200 epochs on teacher logits softened with temperature τ = 20, using data fractions ρ = {0.2, 0.5, 0.8}.Source: §5.3 · p. 10
• The reported student test accuracies were {0.213, 0.524, 0.652}; with only 80% of teacher data, accuracy exceeded 60% on held-out data, but exact teacher-function recovery was not observed.Source: §5.3 · p. 10
• > 60% accuracy was achieved on held-out data when students saw 80% of the teacher’s data, versus 0.1% for random guessing.Source: §5.3 · p. 10

• Figure 7 evaluates GPT-2 students on 6,000 synthetic random-number sentences memorized by a matched pretrained teacher across different ρ fractions.Source: Figure 7 · p. 10
• Students can acquire held-out memorized information from teachers’ soft labels, through either statistical leakage or recovery of the teacher’s function.The study reports this across structured data with varying memorization and unstructured random data.Source: §6 · p. 10
• The analysis is limited to synthetic structured or explicitly memorized datasets and therefore does not capture all aspects of natural data distributions.Source: §6 · p. 10
• The use of logistic regression, single-layer ReLU MLPs, and small transformers leaves the translation of leakage regimes and thresholds to deeper architectures unclear.The paper also notes that regularization and optimization effects were not explored.Source: §6 · p. 10
• Future work should study mixtures of memorized and generalized information, efficient dataset distillation, and whether transferred memorized knowledge can be hidden for privacy.Source: §6 · p. 10

• The teacher fits two-dimensional inputs with i.i.d. random labels from 20 classes using a fully connected ReLU network with p = 300 hidden neurons.Source: Figure 8 · p. 14
• The teacher achieves 100% accuracy, while the resulting teacher and student decision boundaries correspond very well.Source: Figure 8 · p. 14, A Visual Example for d = 2 and c = 20 · p. 14
• The example uses a partition of the teacher data into disjoint student-seen and held-out sets to visualize leakage through soft labels.Source: Figure 8 · p. 14, A Visual Example for d = 2 and c = 20 · p. 14
• At τ = 20, students trained on teacher soft labels reach approximately 66% test accuracy versus approximately 6% for independently trained students.The reported averages use five initializations with standard errors.Source: A Visual Example for d = 2 and c = 20 · p. 14

B.1 Implementation details for the transformer

• The transformer implementation uses a single-layer architecture with four attention heads, a 4 · 128-unit hidden layer, weight decay 1.0, full batches, Adam, and learning rate 0.001.Source: §B.1 · p. 14

B.2 Generalization speed

• With a small training set and τ = 10, a student can memorize quickly and generalize to held-out teacher data and validation data after only a few epochs.Source: Figure 9 · p. 15
• With a larger training set and smaller temperature, the student exhibits grokking similar to the original teacher.Source: Figure 9 · p. 15

B.3 Modular addition with MLPs

• The MLP replication uses teacher and student architectures with two hidden ReLU layers.Figure 10 varies teacher stopping points and reports student accuracies across ρ and τ.Source: §B.3 · p. 15, Figure 10 · p. 15
• At low temperatures, student behavior becomes largely independent of whether teacher training stopped before, during, or after generalization emerged.Source: §B.3 · p. 15, Figure 10 · p. 15
• For MLPs, higher temperatures can make memorizing students achieve higher accS_test than accS_val, indicating transfer of memorized labels without task structure.The soft labels can even prevent students from generalizing.Source: §B.3 · p. 15

C.1 Logistic Regression: Training Students from the Logit

• The logistic-regression analysis measures teacher matching and mean-squared error on student training and test sets across ρ and α = n/d.Source: §C.1 · p. 16
• The appendix reports five repetitions and uses Adam optimization for both teacher and student training.Source: §C.1 · p. 16, §C.2 · p. 17

C.2 Varying the Temperature for Multinomial Logistic Regression

• In multinomial logistic regression, temperature determines whether held-out teacher data is leaked to the student across different ρ and α values.Source: §C.2 · p. 17
• Figure 12 reports the impact of softmax temperature on student accuracy for two classes with α = n/(dc) and d = 1000.Source: Figure 12 · p. 17
• The temperature experiments compare two-class and ten-class multinomial regression settings, including standard and within-class reshuffled input–soft-label assignments.Source: Figure 13 · p. 17, §C.2 · p. 17

C.3 Supplementary training information

• A sharp loss drop occurs around the same point as the accuracy jump toward matching the teacher.Source: §C.3 · p. 18
• Before generalizing to the teacher function, the student matches a within-class-shuffled baseline, supporting an initial memorization phase.Source: §C.3 · p. 18, Figure 15 · p. 18
• The comparison uses d = 1000, c = 100, teacher sample sizes n = 10^3 × {15, 50, 100, 200}, and student fraction ρ = 0.65.Source: Figure 15 · p. 18

C.4.1 Varying the soft label content

• Removing even one smallest soft-label entry is critical when the normal student would otherwise learn the teacher, while removing more entries rapidly worsens accuracy.Source: §C.4.1 · p. 19, Figure 17 · p. 20
• Removing a class from soft-label vectors harms accuracy on that class but leaves other-class accuracy near normal, with little effect on average performance.Source: §C.4.1 · p. 19
• Removing a class entirely from the training set preserves high held-out accuracy for that class when its information remains in other classes’ soft labels.Source: §C.4.1 · p. 19
• The ablations test zeroing the smallest k ∈ {1, 10, 50} entries and removing one class or its soft-label component.Source: §C.4.1 · p. 19, Figure 17 · p. 20

C.4.2 Varying the hidden layer size of teachers and students

• When teacher and student widths vary together, plotting accuracy against α/pS causes the curves to align closely, consistent with parameter-scaled behavior.Source: §C.4.2 · p. 20, Figure 18 · p. 20
• With teacher width fixed at pT = 500, increasing student width improves accS_train in the phase where the student learns the soft labels, such as at ρ = 0.7.Source: §C.4.2 · p. 20, Figure 19 · p. 21
• Lowering student capacity is optimal in the non-teacher-matching regime, while larger students can perform worse nonlinearly; at ρ = 0.3, pS = 250 matches pS = 1000.Source: §C.4.2 · p. 21, Figure 20 · p. 21

C.4.3 Varying the number of classes

• Across c ∈ {20, 50, 90, 100, 110}, scaling the x-axis as n/d preserves the previously described behavior.Source: §C.4.3 · p. 22, Figure 21 · p. 22

D Computational Resources

• The most computationally intensive experiments were the Fig. 4 phase diagrams at 13 compute days and Fig. 6a at roughly 10 GPU compute days.Source: D Computational Resources · p. 22