Self-Knowledge Distillation in Natural Language Processing

1 Introduction

• Language modeling and neural machine translation generate sentences through successive word predictions, yet training commonly uses one-hot targets and summed cross-entropies.Each prediction selects a target word from the dictionary using a softmax over preceding neural-network layers.Source: §1 · p. 1
• Word embeddings place semantically or syntactically similar words close together, so a near-miss prediction may be less wrong than a distant word.The introduction gives “programming” and “coding” as an example of closely related words.Source: §1 · p. 1
• SKD distills soft-target information from the currently training model using relationships encoded in its word embedding space.Unlike conventional distillation from a pretrained teacher, SKD uses probabilities produced by the model during training.Source: §1 · p. 2
• The authors report improved performance on language modeling and neural machine translation, and suggest SKD reduces overfitting because it uses more information.The proposed method is based on the training model’s own predicted probabilities.Source: §1 · p. 2

2.1 Cross Entropy

• Cross-entropy with a one-hot target evaluates the probability assigned to the correct class but does not convey other information about the input.For classification, the objective uses target and predicted probabilities; the target is represented as a one-hot vector.Source: §2.1 · p. 2

2.2 Knowledge Distillation

• Knowledge distillation trains a new model using the class probabilities of a trained model, which can transfer information beyond the target class.The paper describes a weighted objective combining cross-entropy with correct labels and cross-entropy with soft targets.Source: §2.2 · pp. 2–3
• Distillation can train models of the same size and architecture, and a less powerful teacher can provide useful knowledge to a larger student.The cited work reports students outperforming their teachers, including when the teacher has a weaker architecture.Source: §2.2 · p. 3

2.3 Word Embedding

• Word embeddings convert word symbols into vectors whose positions reflect semantic and syntactic relationships.The paper notes that similar words are close in the embedding space, so predicting “training” for “learning” is less wrong than predicting “flower” or “internet.”Source: §2.3 · p. 3

3 Self-Knowledge Distillation

• SKD distills knowledge from the currently training model rather than a pretrained teacher.It uses information from the model's own predicted probabilities.Source: §3 · p. 3

3.1 SKD Equations

• Euclidean distance from the target-word embedding estimates soft-target probabilities, assuming nearby words should receive greater probability.The distance-based estimate uses a scale parameter and normalization term.Source: §3.1 · p. 4
• For efficiency, SKD keeps only target and predicted classes, caps qn at 0.5, and approximates the objective with those two probabilities.The method uses Euclidean distance between the target and predicted word embeddings to calculate qn.Source: §3.1 · p. 4
• When the prediction is correct or far from the target in embedding space, the objective matches or approaches cross-entropy; nearby predictions receive two-class soft targets.The model is trained toward the target and predicted classes' soft probabilities for each input.Source: §3.1 · p. 4

3.2 SKD Algorithm

• Training starts with conventional cross-entropy, then gradually increases the soft-target contribution until the objective reaches the SKD form.The schedule starts α at 0 and increases it by η per iteration after K iterations, up to 1.Source: §3.2 · pp. 4–5

3.3 NLP Tasks

• SKD is applied to language modeling and neural machine translation, both sentence-generation tasks trained with sums of word-level cross-entropies.The NMT experiments also assess SKD with Gaussian noise added to target-word embeddings in the decoder.Source: §3.3 · p. 5

4.1 Dataset

• The evaluation compares SKD with baselines in language modeling on PTB and Wiki-2 and in translation on three WMT’15 directions.Source: §4 · p. 5, §4.1 · p. 5

4.2 Language Modeling

• SKD and noise injection improve test-set NLL on PTB and Wiki-2; the authors say SKD provides more knowledgeable information than Gaussian noise.Source: §4.2 · p. 6, Table 2 · p. 6

4.3 Neural Machine Translation

• For En-Fi, q_n moves closer to its target during training; α starts at 0 and rises to 1 after K iterations as distillation ramps in.Source: §4.3 · p. 6, Figure 3 · p. 7
• Around 1 BLEU score: the authors report improved NMT performance, while Table 3 covers En-Fi, Fi-En, and En-De.Source: §4.3 · p. 6, Table 3 · p. 7
• On En-Fi, SKD models are not or more slowly overfitted; Noise+SKD improves further, though the synergy effect needs more research.Source: §4.3 · pp. 6–7, Figure 4 · p. 7

• The method improves performance on language modeling and neural machine translation.Source: §5 · p. 7
• SKD uses two soft target probabilities based on the word embedding space and obtained from the currently training model.Source: §5 · p. 7
• The method can be applied to other tasks that use cross-entropy.Source: §5 · p. 7
• Future work includes testing other applications and model architectures, automating parameters such as α, and generalizing q_n; comparable image classes are not guaranteed to be close in the embedding space.Source: §5 · p. 7