PaperlayerView exact revision on arXiv

Source-linked AI summary

Progressive distillation induces an implicit curriculum

Abhishek Panigrahi, Bingbin Liu, Sadhika Malladi, Andrej Risteski, Surbhi Goel

arXiv:2410.05464v1cs.LG

TL;DR

A stronger teacher does not always yield a stronger student, and the optimization benefits of progressive distillation over direct learning or a final checkpoint remain poorly understood. This paper studies intermediate teacher checkpoints as supervision, identifying an implicit curriculum that accelerates learning and improves sparse-parity sample complexity.

  • Problem

    A better teacher does not always produce a stronger student, and the optimization and generalization benefits of progressive distillation versus direct learning or a final teacher checkpoint are not well understood.

    Source: §1 · p. 1

  • Method

    The paper studies students trained from successive intermediate teacher checkpoints and analyzes the implicit curriculum they provide, theoretically and empirically on sparse parity and empirically on PCFGs and language-model pretraining.

    Source: §1 · p. 2

  • Results

    Progressive distillation accelerates training versus one-shot distillation and direct learning across tested settings, and provably improves sparse-parity sample complexity through easy-to-learn subtasks unavailable from the final checkpoint.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Intermediate checkpoints can provide useful learning signals, including sparse-parity support supervision and PCFG features that capture progressively larger n-gram contexts.

    Source: §1 · p. 2

    More complex tasks may require more intermediate checkpoints, though a small number often suffices when checkpoints are properly selected.

    Source: (R2) Progressive distillation with a… · p. 5

Abstract

from arXiv · show

Knowledge distillation leverages a teacher model to improve the training of a student model. A persistent challenge is that a better teacher does not always yield a better student, to which a common mitigation is to use additional supervision from several ``intermediate'' teachers. One empirically validated variant of this principle is progressive distillation, where the student learns from successive intermediate checkpoints of the teacher. Using sparse parity as a sandbox, we identify an implicit curriculum as one mechanism through which progressive distillation accelerates the student's learning. This curriculum is available only through the intermediate checkpoints but not the final converged one, and imparts both empirical acceleration and a provable sample complexity benefit to the student. We then extend our investigation to Transformers trained on probabilistic context-free grammars (PCFGs) and real-world pre-training datasets (Wikipedia and Books). Through probing the teacher model, we identify an analogous implicit curriculum where the model progressively learns features that capture longer context. Our theoretical and empirical findings on sparse parity, complemented by empirical observations on more complex tasks, highlight the benefit of progressive distillation via implicit curriculum across setups.

1 Introduction

The paper studies why stronger teachers do not always produce stronger students and how progressive distillation affects optimization. It identifies an implicit curriculum in intermediate teacher checkpoints, finding faster student learning across tasks and a provable sample-complexity benefit for sparse parity.

1 Introduction

  • The paper frames its motivation as limited understanding of optimization and generalization benefits relative to learning directly from data or the final teacher checkpoint.A better teacher does not always yield a stronger student.Source: §1 · p. 1
  • Progressive distillation accelerates training versus one-shot distillation and direct learning across sparse parity and PCFGs, with additional experiments on Wikipedia and Books.On sparse parity, smaller students can train at the same speed as larger models; on PCFGs, smaller BERT models improve masked-prediction accuracy.Source: §1 · p. 2
  • For sparse parity, intermediate checkpoints provide easy-to-learn subtasks that supervise the support coordinates, yielding a provable sample-complexity improvement over one-shot distillation and direct learning.This implicit curriculum is unavailable from the final teacher checkpoint.Source: §1 · p. 2
  • For PCFGs, the implicit curriculum corresponds to progressively learning features that capture larger n-gram contexts.The results also provide guidance on selecting intermediate teachers.Source: §1 · p. 2
  • The paper focuses on optimization dynamics, contrasting prior work that hypothesized progressive distillation benefits student generalization.Earlier work on sparse parity used an explicit input-distribution curriculum, whereas this paper’s curriculum arises implicitly in teacher supervision.Source: §1 · p. 3

2 Preliminaries

The preliminaries distinguish direct learning, one-shot distillation, and progressive distillation by the source and timing of student supervision. The paper isolates supervision sources and uses low teacher temperatures in its experiments.

2 Preliminaries

  • The experiments compare student learning from ground-truth labels alone with learning from teacher logits alone to isolate each supervision source.Prior papers generally combine these objectives.Source: §2 · p. 3
  • One-shot distillation trains the student throughout on a fixed, final converged teacher checkpoint; progressive distillation uses multiple intermediate teacher checkpoints.The paper’s Definition 2.1 specifies a checkpoint set and a corresponding training duration for each checkpoint.Source: §2 · pp. 3–4
  • The paper’s main experiments use equally spaced teacher checkpoints, training the student on each for T steps before continuing with the final checkpoint.The stated scheme contains N−1 intermediate checkpoints plus the final checkpoint.Source: §2 · p. 4
  • Teacher temperatures are set very low for sparse parity, PCFG, and natural-language experiments, making outputs close to one-hot labels and reducing softness-related regularization as a confound.For one-shot distillation, the main paper reports the better-performing temperature among τ = 1 and 10−4.Source: §2 · p. 4

3 The implicit curriculum: a case study with sparse parity

The paper uses sparse parity to show how intermediate teacher checkpoints can supply an implicit curriculum that accelerates student learning. It then proves a sample-complexity benefit for two-checkpoint progressive distillation, tied to supervision available during the teacher’s first phase but not from its final checkpoint.

3 The implicit curriculum: a case study with sparse parity

  • Sparse-parity learning requires identifying the support before computing its variable product, and the support search depends on input dimension d.The paper motivates intermediate supervision as a way to provide direct information about the support.Source: §3 · p. 4, §3.1 · p. 5

3.1 Accelerating learning with the implicit degree curriculum

  • During the teacher’s phase transition, its predictions correlate more strongly with in-support degree-1 monomials, but these signals diminish with continued training.The paper notes that low-degree polynomials beyond degree 1 can also provide acceleration in practice.Source: §3.1 · p. 5

(R2) Progressive distillation with a single intermediate checkpoint can outperform one-shot

  • Only the intermediate checkpoint during the phase transition enables (2, 1M)-progressive distillation to reach 100% accuracy among the three candidates.The candidates correspond to the beginning, middle, and end of the teacher’s phase transition.Source: (R2) Progressive distillation with a… · p. 5, Figure 2 · p. 6

3.2 The low-degree curriculum reduces sample complexity

  • ˜Θ(2kd2ϵ−2 + k3) samples suffice for two-checkpoint progressive distillation, while one-shot distillation requires at least Ω(dk−1, ϵ−2).The theorem considers (d, k)-sparse parity and a student model of size ˜m = ˜Θ(2k).Source: §3.2 · p. 6
  • At the intermediate checkpoint, support-coordinate correlations create an Ω(1/k) gradient gap, letting support weights grow with O(k2 log(˜m)) samples.The proof contrasts correlations on support coordinates with smaller correlations off support.Source: §3.2 · p. 7
  • The sample-complexity analysis assumes large-batch SGD with small gradient noise, whereas the experiments use online SGD with batch size 1.Source: §3.2 · p. 7

4 Implicit curriculum with PCFGs and Natural language

The paper studies whether progressive distillation gives students useful learning signals as models learn longer-range features. In PCFGs and Wikipedia and Books experiments, teacher checkpoints around a training inflection point provide an implicit curriculum, and progressive distillation improves student feature measures or masked-prediction accuracy.

4 Implicit curriculum with PCFGs and Natural language

  • The experiments examine BERT on PCFGs and Wikipedia and Books, with GPT-2 experiments deferred to the appendix.Source: §4 · p. 7

4.1 n-gram curriculum in PCFGs

  • The PCFG experiments use cfg3b, generated by depth-7 trees, as a structured sandbox for studying language-model features.The paper describes PCFG sentences as trees with non-terminal intermediate nodes and terminal leaves.Source: §4.1 · p. 8

4.1.1 Progress measures of implicit curriculum

  • The three progress measures assess reliance on neighboring n-gram context, prediction closeness between full and local context, and non-terminal prediction.A larger Mclose indicates greater use of context outside the n-gram window; non-terminals are predicted from output embeddings.Source: §4.1.1 · pp. 8–9

4.1.2 Empirical verification of the n-gram curriculum

  • For PCFGs, the teacher’s loss drops rapidly in its middle phase; after C1, Mrobust and Mclose fall, and progressive distillation improves context use and non-terminal prediction.The paper identifies C1 as the best single intermediate checkpoint for student performance.Source: Figure 4 · p. 9, §4.1.2 · p. 10, Figure 5 · p. 11

4.2 Beyond synthetic setups: implicit curriculum in natural languages

  • On Wikipedia and Books, the teacher shows three loss phases and Mrobust peaks at C1, while progressive distillation improves the student’s masked-prediction accuracy.The authors report that this checkpoint provides an implicit curriculum toward easier-to-learn local n-grams.Source: §4.2 · p. 10, Figure 6 · p. 11

5 Discussions

The discussion attributes progressive distillation’s feature-learning benefit to an implicit curriculum supplied by intermediate checkpoints. It also identifies limited temperature choices and generation-based distillation as directions needing further study.

5 Discussions

  • Intermediate checkpoints provide an implicit curriculum that can improve the student’s feature learning.The discussion contrasts this with the final converged checkpoint, which does not provide that curriculum.Source: §5 · p. 11
  • Experiments use only the default temperature, τ = 1.0, and low temperatures, τ = 10−4 or 10−20, leaving temperature’s optimization effects for future study.The paper notes that temperature changes can sometimes produce larger performance gains than changing the distillation method.Source: §5 · p. 11
  • Generation-based approaches supervise with sampled outputs, unlike this paper’s distributional supervision; extending the framework to generative settings remains future work.The paper also distinguishes settings where a teacher can provide multiple generations through successive unrolling steps.Source: §5 · p. 12

A Overview of the appendix

The appendix collects omitted proofs and further experiments, then reviews related work on distillation, sparse-parity learning, feature learning, and model width.

A Overview of the appendix

  • The appendix includes the omitted proof of Theorem 3.2 and additional empirical explorations across sparse parity and PCFG tasks.The explorations include MLP and Transformer variants, hierarchical sparse parity, PCFG masked prediction, and next-token prediction.Source: A Overview of the appendix · pp. 19–20

A.1 Additional related works

  • Prior distillation work studies richer class-label distributions, trajectory-averaged fixed targets, regularization, and multi-step self-distillation, with some results focused on regression.Progressive distillation differs from trajectory averaging because the student’s target changes during training.Source: §A.1 · p. 20
  • Sparse parity has statistical-query lower bounds that imply a width, training-step, and sample trade-off for fully connected networks.With the same number of training steps, narrower models require more samples to learn parity.Source: §A.1 · p. 20
  • Feature learning is defined as recovering a low-dimensional feature that reduces sample complexity; for sparse parity, that feature is its support.For k = 2, cited work reports a reduction from Θ(d2) to O(dpoly log d) sample complexity using a jointly optimized two-layer network.Source: §A.1 · p. 20
  • Prior work links wider networks to easier optimization, while this paper studies how a smaller student may inherit that benefit from a wider teacher.The discussion also treats attention-head count as a Transformer scaling dimension and notes the lack of quantitative descriptions for finite head counts.Source: §A.1 · p. 21

B Proofs of results in Section 3.2

The proofs establish a sample-complexity advantage for progressive distillation under a two-stage teacher setup. The argument uses an intermediate checkpoint’s support-variable signal before the student trains against the final teacher.

B Proofs of results in Section 3.2

  • The theoretical setup uses two-stage batch gradient descent: first-layer weights are trained first, then output weights are fit, with separate stage lengths and training parameters.The teacher uses hinge loss, while the student uses a distillation hinge loss based on teacher outputs.Source: B Proofs of results in Section 3.2 · pp. 21–22
  • Progressive distillation provably reduces the student’s sample complexity relative to one-shot distillation or learning directly from data.The theorem assumes a two-stage teacher satisfying first-stage conditions and achieving loss O(d−c) after the second stage.Source: B Proofs of results in Section 3.2 · p. 22
  • The progressive strategy trains first on the teacher’s T1 checkpoint and then on its final checkpoint, unlike one-shot distillation from the final checkpoint throughout.The strategy comparison specifies the progressive student’s first phase as T1 = 1 steps.Source: B Proofs of results in Section 3.2 · p. 22
  • The proof separates in-support from off-support correlations at the teacher’s first checkpoint, then uses the known support for rapid second-phase student convergence.The two components are established through the first-stage correlation result and a second-phase convergence result.Source: B Proofs of results in Section 3.2 · p. 22

B.1.1 First stage analysis for the teacher

In the teacher’s first stage, the population gradient is stronger on sparse-parity support coordinates than off-support coordinates. The resulting checkpoint correlates positively with each in-support variable, furnishing a signal for later student training.

B.1.1 First stage analysis for the teacher

  • With an appropriate learning rate, first-layer weights on support coordinates grow to 1/(2k), while off-support coordinates remain O(1).The analysis also requires the batch-gradient estimate to remain close enough to the population gradient.Source: §B.1.1 · pp. 23–24
  • At initialization, support-coordinate gradients have magnitude |ζk−1|, while off-support gradients have magnitude |ζk+1|, creating a gap that reveals the support.The passage gives a lower bound on this gap in terms of d and k.Source: §B.1.1 · p. 23
  • After the first phase, the teacher output has positive correlation with every variable in the sparse-parity support.The stated lower bound for each in-support correlation is 1/(8k) plus error terms.Source: §B.1.1 · p. 24
  • The first-stage correlation guarantee is stated with probability at least 1 − δ over initialization and sampling, under the per-neuron conditions of Lemma B.2.The correlation bound includes terms for gradient-estimation error and model width.Source: §B.1.1 · pp. 24, 27

B.1.2 Second stage analysis for the teacher

The teacher analysis gives sample-complexity expressions for reaching loss ϵ and notes that the first phase may require extra samples to ensure conditions for improved student-teaching signal.

B.1.2 Second stage analysis for the teacher

  • T1 × B1 + T2 × B2 = Θ(|ζk−1|2 k2 log(kd/ϵ)) + Θ(md2k3/ϵ2) gives another stated sample-complexity expression for reaching loss ϵ.This expression differs in the first term from the other stated expression.Source: §B.1.2 · p. 29
  • The analysis uses a modified first-phase result with higher sample complexity to ensure stronger conditions that provide improved signal for teaching a smaller student.The authors note that the error bounds are not optimized and the sample-complexity bounds are likely loose.Source: §B.1.2 · p. 29
  • T1 × B1 + T2 × B2 = Θ(|ζk−1|2 d10/3k2 log(kd/ϵ)) + Θ(md2k3/ϵ2) gives one stated sample-complexity expression for reaching loss ϵ.The passage labels this as the minimal sample complexity.Source: §B.1.2 · p. 29

B.2 Analysis for the student

The student analysis contrasts learning support from distillation signals with learning it from true labels, then examines progressive distillation and additional sparse-parity experiments. Its theoretical account links the first-phase gradient gap under progressive distillation to reduced sample complexity for that phase.

B.2 Analysis for the student

  • The analysis uses a modified first-phase result with higher sample complexity to secure stronger conditions, and the authors say the resulting error bounds are likely loose.The stronger conditions are needed to obtain improved signal for teaching a smaller student.Source: §B.2 · p. 29
  • Progressive distillation creates a Fourier gap between in-support and out-of-support population gradients, allowing a finite batch to maintain the gap and reducing first-phase sample complexity.The second-phase training proof matches the teacher’s proof.Source: §B.2 · pp. 29–30
  • For a saturated teacher checkpoint, learning support from its correlations requires at least Ω(d2c) samples, while learning from true labels requires at least Ω(dk−1) samples.The resulting lower bound for combining the two signals is Ω(dmin(2c,k−1)/ ˜m) for fixed batch size.Source: §B.2 · p. 30
  • The width-optimization trade-off states that, for fixed batch size and gradient noise, learning support requires either greater width or more gradient steps.With fresh samples each batch, the step requirement also increases sample complexity.Source: §B.2 · p. 30

C.1 Additional results on sparse parity with MLP

  • For sparse parity with n = 100 and k = 6, the MLP experiments compare a width-5 × 104 teacher with width-102 or width-103 students trained against a final checkpoint or 200 intermediate checkpoints.The experiments use SGD with batch size 1 for 20M steps, evaluate on 4096 held-out examples, and average three seeds.Source: §C.1 · p. 30
  • Larger MLPs and Transformers learn sparse parity faster, and repeated experiments include a width-1000 student.The figures report (100, 6)-parity, with Figure 7 aggregated over five runs per setup.Source: Figure 7 · p. 31, Figure 8 · p. 31

C.2 Learning with Transformers: parallel search with attention heads

In sparse-parity Transformers, increasing attention heads speeds learning, and progressive distillation can speed student training with an intermediate checkpoint. The section also presents attention-based support selection and probes attention’s relation to accuracy.

C.2 Learning with Transformers: parallel search with attention heads

  • With a higher learning-rate teacher, degree-2 and degree-3 correlations show a clean in-support/off-support gap at the phase transition even when degree-1 correlations do not.The middle-second-phase checkpoint accelerates progressive distillation in this experiment.Source: Figure 9 · p. 32
  • Increasing the number of attention heads speeds Transformer learning, including setups that hold the MLP dimension constant.The constant-dimension comparisons vary head count while fixing either total hidden dimension or per-head dimension with averaged heads.Source: §C.2 · pp. 31, 33, Figure 10 · p. 33
  • During the phase transition, a 2-layer 32-head teacher has higher correlation with in-support than off-support degree-1 monomials, but a second in-support spike does not accelerate student training.Figure 12 reports correlations across four random seeds.Source: §C.2 · p. 33, Figure 12 · p. 34
  • Progressive distillation’s benefit persists with hard labels: temperatures below 0.01 produce results qualitatively similar to τ = 0, while τ = 1 can slow learning.The authors leave the exact effect of temperature for future work.Source: §C.2 · p. 34, Figure 14 · p. 35
  • One representational solution uses attention to select the k in-support variables and an MLP to compute their product, requiring hidden dimension 4(k + 1).The corresponding proposition bounds the MLP L∞ norm by 4k(k + 1).Source: §C.2 · pp. 31, 34, 36
  • Using as few as one intermediate checkpoint significantly speeds Transformer student training compared with cross-entropy training or one-shot distillation.The two-shot comparisons select checkpoints during the teacher’s phase transition; Figure 13 also compares regular-interval checkpoints.Source: §C.2 · p. 34, Figure 13 · p. 35
  • A second representational solution uses uniform attention to copy all variables to the final position, where a 3-layer MLP computes sparse parity.The proposition specifies attention weights αi,d = 1 d.Source: §C.2 · p. 36, The weights satisfy · p. 37

The weights satisfy

  • In 2-layer 16-head GPT-2 models, attention focuses on relevant tokens, and attention weight on the support is tightly correlated with accuracy.The authors present this as preliminary interpretability evidence that the model uses attention in learning sparse parity.Source: Figure 11 · p. 33, The weights satisfy · p. 37

C.3 A hierarchical generalization of sparse parity

Hierarchical sparse parity extends the task to labels determined by paths through a decision tree, and progressive distillation can train a smaller student to 100% accuracy. The experiments associate this success with a low-degree feature curriculum emerging across teacher checkpoints, while BERT probing and checkpoint-variant experiments extend the investigation to PCFGs.

C.3 A hierarchical generalization of sparse parity

  • Progressive distillation trains a smaller student as fast as the teacher and can reach 100% accuracy on hierarchical sparse parity.The setting is 8-way classification from a depth-3 tree whose features are 5-sparse parities.Source: §C.3 · p. 39
  • Relevant-feature monomial correlations spike at different training steps, so a single checkpoint may not convey the full feature support.The timing differences arise across features at different tree levels, including features shared among labels.Source: §C.3 · pp. 39–40, Figure 17 · p. 39
  • Two intermediate checkpoints within the teacher’s phase transition can train the student to 100% accuracy, whereas one is insufficient.Checkpoint choice within the transition can still substantially affect student performance.Source: §C.3 · p. 40, Figure 18 · p. 40

D.2 Variants of progressive distillation

  • The N-shot κT0-Equal-split variant allocates κT0/N student steps to each checkpoint and keeps the final checkpoint through any remaining training.The experiments suggest κ = 1/2 as a rule of thumb for learning faster than the teacher at a given training length.Source: §D.2 · p. 42
  • Checkpoint selection uses a teacher-loss transition: the first checkpoint is placed roughly midway through it, and later checkpoints follow at multiples of that step.The paper reports that optimal progressive-distillation strategies remain an open exploration.Source: §D.2 · p. 42

D.1 A formal description of PCFGs; D.3 Details on Non-terminal prediction with Multi-head linear probing; D.4 Details on the synthetic PCFGs; D.5.1 A primer on BERT; D.5.2 Data Generations

  • The PCFG experiments use BERT masked prediction on generated sequences, with linear probes tracking non-terminals at grammar-tree levels.The experiments include five synthetic PCFGs differing in rule counts and ambiguity; BERT uses bidirectional self-attention and predicts masked tokens.Source: §D.3 · p. 42, §D.4 · p. 43, §D.5.1 · p. 43

D.7 Additional Curriculum probing on the teacher’s checkpoints

  • Both progressive-distillation variants help BERT students learn faster than one-shot distillation, but the better split depends on student training length.T0/2-Equal-split is slightly better below the teacher’s training budget; the trend reverses as the budget approaches and exceeds it.Source: §D.7 · p. 45, Figure 24 · p. 47
  • On PCFGs, teacher logits become closer to predictions using higher n-gram context, while non-terminal probes exceed 95% of final probe performance by mid-second phase.The reported logit inflection occurs around the middle of the second phase, the first selected checkpoint.Source: Figure 22 · p. 46

D.8 Ablations with hyperparameters

The ablations find that progressive distillation’s advantage varies with temperature, masking rate, model capacity, and PCFG difficulty. GPT experiments additionally define losses over selected grammar-boundary tokens and compare n-gram-context measures for autoregressive models.

D.8 Ablations with hyperparameters

  • With temperature 10^-4, progressive distillation outperforms one-shot distillation at both tested temperatures, but at temperature 1 it can lose for a stronger student.The authors leave further study of temperature effects for future work.Source: §D.8 · p. 48, Figure 25 · p. 48
  • At every tested masking rate, progressive distillation performs better than one-shot distillation.The relative performance gap diminishes with more training steps.Source: §D.8 · p. 48, Figure 26 · p. 48
  • Progressive-distillation gains depend on student capacity and the PCFG; on cfg3i the student plateaus at 75% top-1 accuracy, limiting gains over one-shot distillation.On cfg3i, progressive distillation reaches that accuracy within 2000 steps but does not improve further.Source: §D.8 · p. 49, Figure 27 · p. 49

E Autoregressive training with GPT2

  • GPT’s boundary-token losses use nested sets: loss L(1) averages over all tokens, while losses at higher levels restrict supervision to grammar-boundary tokens.The GPT analysis compares autoregressive models using n-gram context windows and modified skip n-gram masking.Source: E Autoregressive training with GPT2 · p. 50

E.1 Observations

GPT teacher training shows a rapid loss-drop phase followed by slower decline, with larger models entering the rapid phase sooner. Around the selected checkpoint, predictions show an n-gram curriculum; progressive distillation helps students learn faster than one-shot distillation, with smaller gains as more tokens enter the loss.

E.1 Observations

  • The GPT teacher has a rapid loss-drop phase followed by a slow-drop phase, and larger models reach the rapid phase sooner.The early dormant phase is small on a log-scaled training axis.Source: §E.1 · pp. 50–51, Figure 28 · p. 51
  • At the first selected checkpoint, GPT predictions show an inflection in reliance on 3-gram predictions, consistent with an observed curriculum.The first supervision checkpoint is selected roughly at the middle of the first phase.Source: §E.1 · p. 51
  • Progressive distillation outperforms one-shot distillation across the reported training budgets, but the gap shrinks as more tokens are included in the loss.The gap is smaller for loss L(2) than for loss L(4).Source: §E.1 · p. 51

F Details on Wikipedia + Books experiments

The supplied experiments describe GPT progressive-distillation comparisons across PCFGs, losses, and student training budgets. They report faster student learning than one-shot distillation, with the size of the gap varying by training budget and loss.

F Details on Wikipedia + Books experiments

  • Progressive distillation outperforms one-shot distillation at all reported sample budgets, with the gap shrinking as budget increases.The advantage is smaller for loss L(2) than for loss L(4).Source: F Details on Wikipedia + Books… · p. 52
  • For GPT on cfg3h, cfg3g, and cfg3f, the advantage over one-shot distillation depends on the PCFG; cfg3f exceeds 90% top-1 accuracy, while cfg3g plateaus at 84%.On cfg3g, progressive distillation has only marginal gains after 500 steps.Source: Figure 32 · p. 53
  • The reported GPT comparisons use a 32-head teacher and an 8-head student, with the teacher trained for 8000 steps.The selected checkpoints and student schedules are described in the supplied figure-related passages.Source: F Details on Wikipedia + Books… · p. 53

Related papers

Loading 2410.05464v1…