Source-linked AI summary
Representation Degeneration Problem in Training Natural Language Generation Models
Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, Tie-Yan Liu
TL;DR
Weight tying in natural language generation can cause most learned word embeddings to collapse into a narrow cone, limiting their representation power. The paper analyzes the geometric and optimization conditions behind this degeneration and regularizes embedding similarities to widen the representation space. Experiments in language modeling and machine translation mitigate degeneration and improve reported perplexity and BLEU results over baselines.
Problem
Weight-tied likelihood training can distribute most word embeddings into a narrow cone, limiting their representation power.
Method
The paper analyzes hidden-state and low-frequency-word optimization and regularizes the embedding matrix by decreasing pairwise cosine similarities.
Results
The method mitigates degeneration and outperforms baselines, including 2.0 perplexity points on WikiText-2 and 1.08/0.93 BLEU points on WMT 2014 English-German/German-English.
Takeaways & Limitations
The proposed regularization increases the representation power of learned word embeddings and is effective across language modeling and machine translation.
Takeaways & Limitations
The regularization term is based on cosine similarity, and the authors note that better regularization terms and combinations with other approaches may exist.
Abstract
from arXiv · showhide
We study an interesting problem in training neural network-based models for natural language generation tasks, which we call the \emph{representation degeneration problem}. We observe that when training a model for natural language generation tasks through likelihood maximization with the weight tying trick, especially with big training datasets, most of the learnt word embeddings tend to degenerate and be distributed into a narrow cone, which largely limits the representation power of word embeddings. We analyze the conditions and causes of this problem and propose a novel regularization method to address it. Experiments on language modeling and machine translation show that our method can largely mitigate the representation degeneration problem and achieve better performance than baseline algorithms.
1 INTRODUCTION
The paper identifies representation degeneration in weight-tied natural language generation models, where word embeddings collapse into a narrow cone. It analyzes the problem and proposes regularization that improves embedding distribution and task performance.
- Problem: Weight tying gives the word embedding matrix dual roles as input representations and output softmax parameters.These roles require semantic richness and sufficient capacity to classify hidden states into correct labels.
- Empirical observation: Word embeddings trained for translation cluster in a narrow cone, unlike Word2Vec embeddings and classification softmax parameters.The comparison uses 2D SVD visualizations.
- Problem and analysis: The paper names this collapse the representation degeneration problem and aims to explain and address it.The proposed analysis combines intuitive and theoretical explanations with a practical solution.
- Mechanism: Low-frequency words are likely to become close to one another during optimization, contributing to localized embeddings.The analysis relates the problem to hidden-state structure and low-frequency word optimization.
- Results: The proposed embedding regularization mitigates degeneration and improves performance by 2.0 perplexity points on WikiText-2 and 1.08/0.93 BLEU points on WMT 2014 English-German/German-English.These results cover language modeling and machine translation.
2 RELATED WORK
Natural language generation includes language modeling, machine translation, and dialogue systems, typically predicting the next word from context. Neural architectures and parameter-sharing techniques have driven progress across these tasks.
- Task setting: Natural language generation commonly predicts the next word from previous context and conditional information.Applications include language modeling, machine translation, and dialog systems.
- Machine translation: Neural machine translation uses encoder-decoder frameworks with attention and has evolved from recurrent models toward CNN and self-attention architectures.These developments have improved results across language pairs.
- Weight tying: Weight tying shares parameters between word embeddings and the softmax layer in language generation models.Prior work reports parameter reduction and theoretical benefits from this trick.
3 REPRESENTATION DEGENERATION PROBLEM
The paper empirically examines word embeddings in sequence generation and finds that translation embeddings cluster in a narrow cone, unlike conventional representations. This clustering limits expressiveness and motivates the degeneration analysis.
- Experimental setup: The empirical study focuses on sequence-generation word embeddings, especially those learned by machine translation models.The main analysis uses Transformer trained on WMT 2014 English-German, with similar observations for an LSTM model.
- Comparison method: The study compares tied word-embedding and softmax parameters with Word2Vec embeddings and classification-task softmax parameters.Learned matrices are projected into two dimensions using rank-2 SVD.
- Baselines: Word2Vec embeddings and classification category embeddings are broadly distributed around the origin and separated from one another.The paper associates these distributions with differing semantic meanings and large-margin classification.
- Translation embeddings: Translation word embeddings cluster in a narrow cone, with positive cosine similarities in almost all cases.The words are therefore not well separated in the embedding space.
- Interpretation: This clustered geometry restricts representation expressiveness despite word embeddings needing diverse distributions for semantic representation and prediction.The paper calls this limitation the representation degeneration problem.
4 UNDERSTANDING THE PROBLEM
The analysis links representation degeneration to low-frequency words and the geometry of hidden states, showing how rare or unseen tokens can be driven toward shared directions. In particular, degeneration occurs when hidden-state structure permits uniformly negative directions, a condition equivalent to the convex hull excluding the origin.
- Low-frequency words: Low-frequency words occupy most of the vocabulary, so stochastic training often gives individual rare words very few optimization updates.The analysis notes that more than 90% of word frequencies are below 10e-4 in WMT 2014 English-German, while even “is” occurs only about 1%.
- Optimization setup: The generation of each next word is treated as a multi-class classification problem over hidden states and vocabulary embeddings.The sequence probability is factorized left to right, and each conditional word prediction uses a softmax over embedding vectors.
- Unseen words: For an unseen word, likelihood maximization pushes its embedding along directions having negative inner product with every hidden state, potentially sending its norm to infinity.Theorem 1 states that if a uniformly negative direction exists, the optimal embedding is unbounded; the set of such directions is convex, producing a cone-like geometry.
- Hidden-state geometry: A uniformly negative direction exists exactly when the convex hull of the hidden states does not contain the origin.The paper connects this geometric condition to layer normalization, which normalizes and then rescales or translates hidden-state vectors.
- Rare words: Rarely appearing words behave similarly to unseen words when their non-occurrence contexts dominate the loss and the occurrence-context loss is bounded and smooth.The overall loss combines contexts without the word and contexts containing it; when the first component has weight above 1−ϵ, its optimum controls the combined solution.
- Rare words: Low-frequency words with diverse contexts can acquire similar embeddings because their context-overlap losses have similar optima.The paper explicitly relates this shared optimization pattern to the empirical observation that low-frequency word embeddings become close.
5 ADDRESSING THE PROBLEM
The proposed remedy directly broadens the embedding cone by penalizing pairwise cosine similarity while retaining the original likelihood objective. The resulting MLE-CosReg method is evaluated on language modeling and machine translation.
- Regularization design: The method increases the cone aperture by minimizing cosine similarities between normalized word embeddings.This directly targets the narrow-cone geometry that restricts embedding expressiveness.
- Regularization design: MLE-CosReg combines the original log-likelihood loss with the cosine regularization term using a trade-off hyperparameter γ.The paper names the resulting objective MLE with Cosine Regularization.
- Regularizer analysis: The cosine regularizer can be written using the normalized embedding matrix and the sum of its pairwise inner products.Because each normalized embedding contributes a unit diagonal entry, the constant diagonal term can be separated from the similarity-dependent part.
- Experiments: The method is evaluated on language modeling and WMT English→German and German→English machine translation.The language-modeling results are reported in Table 1 using perplexity, while translation results are reported in Table 2.
- Regularizer analysis: Minimizing the regularizer reduces an upper bound on the largest eigenvalue of the embedding Gram matrix and increases the remaining eigenvalues because their sum is fixed.The paper interprets this spectral redistribution as improving the expressiveness of the embedding matrix.
6 EXPERIMENTS
Experiments evaluate the proposed regularization on language modeling and machine translation, finding improved task performance and less degenerate word-embedding representations.
- Experimental setup: The experiments cover language modeling and machine translation using established benchmark datasets and model settings.Language modeling uses WikiText-2 and AWD-LSTM, while translation uses WMT 2014 English-German and German-English with Transformer.
- Experimental results: 0.8/1.7/2.0 test-perplexity improvements are reported over vanilla AWD-LSTM across three language-modeling settings.The settings are without finetuning, with finetuning, and with further continuous cache pointer.
- Experimental results: 1.08/0.93 BLEU improvements are reported for English →German and German →English translation, respectively, with a 0.54 improvement on the big Transformer.These comparisons use the original base Transformer and the big Transformer as stated in the results passage.
- Experimental results: The method adds one regularization term without additional parameters or architecture or hyperparameter modifications.The reported accuracy improvements are attributed to regularizing similarity between word embeddings.
- Representation analysis: The proposed regularization expands the embedding space and produces more uniformly distributed singular values than vanilla Transformer.SVD visualizations show embeddings distributed somewhat uniformly around the origin rather than strictly in a narrow cone.
7 CONCLUSION AND FUTURE WORK
The paper analyzes representation degeneration empirically and theoretically, proposes regularization to increase word-embedding representation power, and reports effectiveness on two language-generation tasks.
- Conclusion: The paper describes and analyzes representation degeneration in neural natural-language-generation models empirically and theoretically.The problem concerns the representation power of word embeddings in the studied models.
- Conclusion: The proposed regularization method explicitly increases the representation power of word embeddings.Its effectiveness is demonstrated on language modeling and machine translation.
- Future work: Future work will apply the method to more language-generation tasks and investigate better cosine-similarity regularizers and combinations with other approaches.The authors specifically mention combining with other approaches to enrich word-embedding representations.
A PROOFS
The appendix proves geometric and optimization results underlying representation degeneration, including its connection to hidden-state geometry and strong-convexity behavior.
- Proofs: The appendix includes proofs of all theorems presented in the main sections.
- Proofs: Theorem 1 states that uniformly negative directions form a convex set and can characterize the optimal solution when one direction applies to all hidden states.The proof uses the objective’s lower bound and convexity to establish the global optimum in the limiting direction.
- Proofs: Theorem 2 states that a uniformly negative direction exists exactly when the convex hull of the hidden states excludes the origin.The proof uses separating-hyperplane geometry for sufficiency and a convex combination contradiction for necessity.
- Proofs: Theorem 3 shows that mixing an α-strongly convex function with a bounded-Hessian perturbation preserves strong convexity when the perturbation weight is sufficiently small.The supplied proof derives the modulus α − ϵ(α + β) from the Hessian inequalities.
B COMPUTATION OF THE COSINE REGULARIZATION
The cosine regularization can be computed efficiently by reducing it to a sum over normalized word embeddings.
- Computational cost: The section analyzes the computational cost of the proposed regularization term.
- Computational cost: The cosine regularization requires Θ(N) computation time for a vocabulary of size N.The derivation shows that computation reduces to summing all normalized word-embedding vectors.
- Computational cost: The linear-time computation follows because only the sum of normalized word-embedding vectors must be computed.
C DISCUSSION ON LAYER NORMALIZATION
The analysis links layer normalization to representation degeneration through the geometry of normalized hidden states. In particular, the required zero-sum condition is difficult to guarantee in unconstrained optimization, while the relevant frequency distributions are examined empirically.
- Layer-normalization setup: Layer normalization transforms hidden states using learned scale and bias vectors after centering and variance normalization.The analysis assumes the hidden-state variance and each scale and bias entry are nonzero.
- Geometric condition: If the convex hull of normalized hidden states contains the origin, nonnegative coefficients summing to one can represent that origin.This convex-hull condition supplies the coefficients used in the subsequent argument.
- Geometric condition: The existence of such coefficients implies a zero-sum relation involving the normalized hidden states and learned scale vector.The resulting condition is expressed as −→1 T h_i − −→1 μ_i/σ_i = 0 for all i in the supplied derivation.
- Practical implication: The zero-sum condition can hardly be guaranteed by current unconstrained optimization, and the paper reports empirical verification.The discussion presents this as the practical obstacle underlying the layer-normalization analysis.
- Empirical frequency setting: Figure 3 examines token-frequency distributions for BPE-preprocessed WMT 2014 English-German and word-level WikiText-2 data.Tokens are ranked by descending frequency on the x-axis, while logarithmic token frequency appears on the y-axis.