Source-linked AI summary
Data Noising as Smoothing in Neural Network Language Models
Ziang Xie, Sida I. Wang, Jiwei Li, Daniel Lévy, Aiming Nie, Dan Jurafsky, Andrew Y. Ng
TL;DR
Neural language modeling lacks widely adopted noising primitives for discrete sequences, despite data sparsity and overfitting challenges. The paper connects input noising with n-gram smoothing to derive schemes for recurrent models, obtaining gains in language modeling and machine translation and empirically validating the connection.
Problem
Neural language models face data sparsity and overfitting, while widely adopted noising primitives for discrete sequence-level language modeling are unavailable.
Method
The paper derives a correspondence between input noising and n-gram smoothing, then adapts advanced smoothing methods into noising schemes for recurrent language models.
Results
The proposed schemes improve performance in language modeling and machine translation, including a total gain of +1.4 BLEU from the proposed bigram Kneser-Ney noising scheme.
Takeaways & Limitations
Data noising can regularize neural sequence models while incorporating generative assumptions from established language-model smoothing methods.
Takeaways & Limitations
The smoothing correspondence assumes an RNN with finite context achieves near the lowest possible cross-entropy loss, under which it behaves like an n-gram model.
Abstract
from arXiv · showhide
Data noising is an effective technique for regularizing neural network models. While noising is widely adopted in application domains such as vision and speech, commonly used noising primitives have not been developed for discrete sequence-level settings such as language modeling. In this paper, we derive a connection between input noising in neural network language models and smoothing in $n$-gram models. Using this connection, we draw upon ideas from smoothing to develop effective noising schemes. We demonstrate performance gains when applying the proposed schemes to language modeling and machine translation. Finally, we provide empirical analysis validating the relationship between noising and smoothing.
1 INTRODUCTION
Neural language models face severe data sparsity, while established data-augmentation primitives from vision and speech do not transfer directly to discrete sequences. The paper develops input-noising schemes for recurrent language models by connecting noising to n-gram smoothing and evaluates them in language modeling and machine translation.
- Large vocabularies and exponentially many contexts make most language-model sequences rare or unseen, creating a central data-sparsity challenge.
- Widely adopted augmentation primitives exist for vision and speech, but comparable noising primitives had not been developed for neural language models.
- Neural language models remain vulnerable to overfitting from data sparsity despite using distributed representations.
- The paper connects expected pseudocounts from input noising with linear interpolation smoothing, then derives analogues of advanced n-gram smoothing methods.
- Experiments assess the proposed noising schemes as regularizers in language modeling and machine translation, followed by empirical validation of the noising–smoothing relationship.
2 RELATED WORK
Related work frames the paper as data augmentation and explicit regularization for sequence models, distinguishing it from parameter, activation, and prior input-embedding approaches. Existing analyses and methods either target other structured tasks or do not capture sequence-level effects.
- Classical neural-network regularization typically acts on parameters or activations, whereas this work treats language-model input noising as data augmentation.
- Feature noising has been used for structured prediction and interpreted as explicit regularization, but the cited sequence-model work does not consider sequence models.
- Randomly zero-masking input embeddings has been applied to sentence representations, but its prior analysis does not capture sequence-level effects.
3 METHOD
The paper treats input noising as sequence-specific regularization for recurrent language models by connecting its expected pseudocounts to n-gram smoothing. It uses this connection to derive progressively richer noising schemes, including adaptive probabilities and higher-order proposals, and extends the approach to sequence-to-sequence models.
- 3.1 PRELIMINARIES: RNN language models address long dependencies with distributed hidden states, but can still overfit under sparse data.Unlike n-gram models, RNNs do not directly use discrete counts, while generic regularization does not exploit sequence-modeling properties.
- 3.2 SMOOTHING AND NOISING: Interpolation smoothing combines higher- and lower-order n-gram estimates when full-context observations are insufficient.For bigrams, the interpolated estimate mixes p(xt|xt−1) and p(xt) with weight λ.
- 3.2 SMOOTHING AND NOISING: Unigram noising replaces context tokens with unigram samples, while blank noising replaces them with a placeholder token with probability γ.Neither scheme noises the token being predicted in the initial formulation.
- 3.3 NOISING AS SMOOTHING: Expected pseudocounts from noised data correspond to interpolation or mixtures of different-order n-gram models with fixed weights.For unigram noising, the bigram estimate becomes (1 −γ)p(xt|xt−1) + γp(xt).
- 3.3 NOISING AS SMOOTHING: Blank noising also yields interpolation by averaging predictions over contexts whose tokens are independently replaced by placeholders.For trigrams, the resulting prediction mixes the full context, partially blanked contexts, and the fully blanked context.
- 3.4 BORROWING TECHNIQUES: The paper proposes adaptive noising probabilities, higher-order proposal distributions, and target noising to obtain analogues of advanced smoothing methods.Combining target noising with discounted γ_AD produces a noising analogue of Kneser-Ney smoothing; context noising also proves empirically beneficial in the decoder.
4 EXPERIMENTS
Experiments evaluate noising for language modeling and machine translation, comparing regularization schemes and examining how noising probability and distributions relate to smoothing. Kneser-Ney-inspired noising performs strongly across the reported tasks, including a total gain of +1.4 BLEU in translation.
- Language Modeling: Penn Treebank experiments use 10K vocabulary word-level LSTM models with two hidden layers and standard preprocessing.The dataset contains 929k training tokens, 73k validation tokens, and 82k test tokens.
- Language Modeling: Training combines feed-forward dropout with noising schemes, selecting dropout and noising probabilities using validation performance.Models are trained with stochastic gradient descent, learning-rate halving, gradient clipping, and lowest-validation-perplexity selection.
- Language Modeling: Kneser-Ney noising yields significant validation-perplexity gains for both medium and large models, with large models matching state-of-the-art single-model regularization.The selected γAD(x1) and q(x) correspond to Kneser-Ney smoothing.
- Analysis: Discounting γ0 maintains perplexity gains across a broader range of noising probabilities than fixed γ0.The discounting ratio appears to identify tokens that are appropriate to noise.
- Machine Translation: +1.4 BLEU is the total translation gain from combining bigram Kneser-Ney noising with blank and unigram noising.Blank and unigram noising each provide roughly +0.7 BLEU, while bigram Kneser-Ney noising adds +0.5-0.7 BLEU on top.
5 DISCUSSION
Discounting-based noising better targets rare subsequences but can reduce overall noising because common tokens dominate text. Empirical analyses support the predicted smoothing relationship and show benefits for unseen n-grams.
- 5.1 SCALING γ VIA DISCOUNTING: Discounting makes rare tokens comparatively more likely to be noised, while common tokens are noised infrequently.When a token appears exactly once, γAD = γ0; Zipfian frequencies mean common tokens still dominate most texts.
- 5.1 SCALING γ VIA DISCOUNTING: Bigram discounting improves perplexity across a much broader range of γ0 values than using a fixed γ0.The discounting ratio appears to identify which tokens should be noised.
- 5.2 NOISED VERSUS UNNOISED MODELS: Noised models produce softmax distributions significantly closer to lower-order frequency distributions than unnoised models.The effect is especially pronounced for the unigram distribution, supporting the connection to interpolation smoothing.
- 5.2 NOISED VERSUS UNNOISED MODELS: Noising lowers perplexity for unseen bigrams and trigrams even when noised and unnoised models have near-identical full-validation perplexity.The evaluation includes 15,062 unseen bigram occurrences and 43,051 unseen trigram occurrences.
6 CONCLUSION
The paper concludes that data noising regularizes neural sequence models by connecting noising to smoothing. This connection enables advanced n-gram smoothing ideas and motivates applications in low-resource and other sequence-modeling settings.
- 6 CONCLUSION: Data noising is effective for regularizing neural network-based sequence models.The paper derives a correspondence between noising and smoothing to adapt advanced n-gram methods to neural networks.
- 6 CONCLUSION: The proposed correspondence incorporates well-understood generative assumptions from language modeling into neural network models.The paper identifies low-resource settings and other sequence-modeling domains as possible application areas.
A SKETCH OF NOISING ALGORITHM
The algorithm applies bigram Kneser-Ney noising during training by using count-derived parameters and a proposal distribution, then runs the training iteration on the noised batch.
- A SKETCH OF NOISING ALGORITHM: Noising is applied to each data batch during training, but not at test time.
- A SKETCH OF NOISING ALGORITHM: Bigram Kneser-Ney noising uses counts, distinct continuation counts, a proposal distribution, and scaling factor γ0.For sequence-to-sequence tasks, count-based parameters are estimated separately for source and target.
- A SKETCH OF NOISING ALGORITHM: The procedure copies each batch, iterates through positions, samples replacements categorically, and returns the noised batch for training.The pseudocode assumes a batch size of one.