Source-linked AI summary
Generalizing and Hybridizing Count-based and Neural Language Models
Graham Neubig, Chris Dyer
TL;DR
Count-based n-gram and neural language models trade scalability and speed against modeling performance and computational cost. The paper unifies them as mixtures of vocabulary distributions with dynamically computed weights, enabling neurally interpolated n-grams and neural/n-gram hybrids. These models outperform baselines and improve convergence, with the hybrid reducing perplexity by 10-11% relative to a strong LSTM baseline.
Problem
Count-based n-gram models are scalable and fast at test time, while neural language models often perform better but are more computationally complex and trained on less data.
Source: §1 · p. 1Method
The paper represents language models as mixtures of component distributions whose context-dependent weights are dynamically calculated, enabling hybrid count-based and neural models.
Source: §1 · pp. 1–2Results
10-11% relative perplexity reduction over a strong LSTM baseline is reported for the proposed neural/n-gram hybrid, while the models improve performance and convergence speed over baselines.
Source: §1 · p. 2, §6.3 · p. 7Takeaways & Limitations
The framework supports neurally learned interpolation for n-grams and neural models that incorporate count-based information, including large external data sources.
Source: §4.2 · p. 5, §8 · p. 9The experiments condition only on in-sentence context, so their baseline numbers are not directly comparable to prior results using inter-sentential dependencies.
Source: §6.3 · p. 7
Abstract
from arXiv · showhide
Language models (LMs) are statistical models that calculate probabilities over sequences of words or other discrete symbols. Currently two major paradigms for language modeling exist: count-based n-gram models, which have advantages of scalability and test-time speed, and neural LMs, which often achieve superior modeling performance. We demonstrate how both varieties of models can be unified in a single modeling framework that defines a set of probability distributions over the vocabulary of words, and then dynamically calculates mixture weights over these distributions. This formulation allows us to create novel hybrid models that combine the desirable features of count-based and neural LMs, and experiments demonstrate the advantages of these approaches.
1 Introduction
The paper unifies count-based and neural language models as mixtures of component distributions with context-dependent weights, enabling hybrids that combine their strengths. Experiments report improved performance and convergence speed over baselines.
Motivation
- Count-based LMs offer scalability and test-time speed, whereas neural LMs often improve performance but require greater computational complexity and less training data.Source: §1 · p. 1
Mixture of Distributions LMs
- MODLMs calculate next-word probabilities by mixing component distributions P_k(w_i|c) with context-dependent weights λ_k(c) that sum to one.Source: §1 · p. 1
- The framework describes n-gram, feed-forward, and recurrent neural LMs through choices of context, mixture weights, and component distributions.Source: §1 · p. 1
Novel Models
- Neurally interpolated n-grams learn n-gram interpolation weights with neural networks, while retaining a distribution set smaller than the vocabulary.Source: §1 · p. 2
- Neural/n-gram hybrids augment neural models with count-based components, allowing large external data sources to inform neural language modeling.Source: §1 · p. 2
Learning
- A block-dropout method randomly drops easier-to-learn distributions to prevent optimization from settling in sub-optimal local minima.Source: §1 · p. 2
Results
- Across language-modeling benchmarks, the proposed models outperform baselines in performance and convergence speed.Source: §1 · p. 2
2 Mixture of Distributions LMs
MODLMs express language-model probabilities as a matrix-vector mixture of component distributions and context-dependent weights. This formulation provides a common representation for existing models and supports more powerful model structures.
MODLM Representation
- MODLMs compute a vocabulary-sized probability vector through the matrix-vector product p_c = D_cλ_c^T.D_c contains component distributions as columns, while λ_c contains their mixture weights.Source: §2 · p. 2
- Each column of D represents a probability distribution, and the mixture weights λ sum to one, ensuring that p is also a valid probability distribution.Source: §2 · p. 2
- The jth row of D_c combined with λ_c determines the probability assigned to word w_i = j.Source: §2 · p. 2
Implications
- The linear-equation view describes existing language models and enables novel model structures that are more powerful and general than those models.Source: §2 · p. 2
3 Existing LMs as Linear Mixtures
The MODLM framework expresses both n-gram and neural language models as mixtures over vocabulary distributions. N-gram models mix order-specific count distributions, while neural models predict mixture weights over identity distributions.
3.1 n-gram LMs as Mixtures of Distributions
- N-gram LMs interpolate maximum-likelihood distributions for contexts of lengths 1 through N to avoid zero probabilities.Static interpolation uses fixed weights, while heuristic interpolation makes the weights context-sensitive through fallback probabilities.Source: §3.1 · pp. 2–3
- Discounting and Kneser–Ney smoothing modify count-based distributions and can be represented within the MODLM mixture framework.Discounted distributions require normalization before they satisfy the framework’s probability-distribution conditions; Modified Kneser–Ney adjusts lower-order counts for fallback use.Source: §3.1 · p. 3
3.2 Neural LMs as Mixtures of Distributions
- Feed-forward neural LMs convert preceding words into representations, transform them into hidden vectors, and predict the next-word distribution with a softmax.In MODLM form, the network instead predicts mixture weights over a vocabulary-sized identity matrix, equivalent to mixing Kronecker δj distributions.Source: §3.2 · pp. 3–4
- Recurrent neural LMs extend this construction by maintaining a hidden state that summarizes the sentence history and predicting mixture weights from the recurrent representation.This representation supports long-distance dependencies beyond standard n-grams.Source: §3.2 · p. 4
4 Novel Applications of MODLMs
MODLMs enable two hybrid language-model designs that combine neural weighting with count-based distributions. One learns n-gram interpolation weights, while the other augments count distributions with neural word-specific components.
4 Novel Applications of MODLMs
- MODLMs provide a framework for designing language models that combine the advantages of n-gram and neural network models.The framework is intended to combine count-based efficiency and neural modeling flexibility.Source: §4 · p. 4
4.1 Neurally Interpolated n-gram Models
- Neurally interpolated n-grams retain n-gram distributions but learn context-dependent mixture weights with a neural network.Because the network predicts weights over N distributions rather than J vocabulary words, these models are expected to be more efficient than standard neural LMs.Source: §4.1 · p. 4
- The neurally interpolated models use context-observation, continuation-count, and optionally word-representation features to calculate mixture weights.When discounted distributions are used, the log sum of discounted counts is also supplied as a feature.Source: §4.1 · pp. 4–5
4.2 Neural/n-gram Hybrid Models
- Neural/n-gram hybrids augment count-based distributions with vocabulary-sized Kronecker δj distributions, allowing the model to rely on counts or deviate from them.This construction exposes the model to information from both neural and n-gram LMs and may let the neural component focus on phenomena not well modeled by n-grams.Source: §4.2 · p. 5
5 Learning Mixtures of Distributions
The paper trains parameterized MODLMs with neural-network likelihood optimization and introduces block dropout for hybrid models. Training uses standard neural optimization alongside cross-validation safeguards and selected network configurations.
5.1 Learning MODLMs
- Parameterized MODLMs are trained by minimizing corpus negative log-likelihood and updating parameters with back-propagation and stochastic gradient descent.The probability calculation uses the distribution matrix and mixture-function output to compute each word’s likelihood.Source: §5.1 · p. 5
5.2 Block Dropout for Hybrid Models
- Hybrid models can fall into poor local minima because initially strong count distributions suppress the still-inaccurate neural δ components.The model may therefore assign near-zero mixture weights to the δ distributions and rely mainly on count-based n-grams.Source: §5.2 · p. 5
- Block dropout addresses this imbalance by disabling all n-gram distributions for 50% of training examples and forcing reliance on δ distributions.The corresponding n-gram entries in λ(c) are zeroed and the remaining weights are renormalized.Source: §5.2 · pp. 5–6
5.3 Network and Training Details
- Experiments use feed-forward or LSTM networks, ADAM optimization, minibatches of 512 words, and development-likelihood model selection.Count-based features are computed with 10-fold cross-validation to reduce overfitting to the data used for training count models.Source: §5.3 · p. 6
6 Experiments
Experiments evaluate neural interpolation, hybrid models, larger-data integration, and dynamic versus static interpolation across PTB and ASPEC. Neural/n-gram hybrids improve over strong neural baselines, while larger-data gains depend on dataset domain and integration strategy.
6.2 Results for Neurally Interpolated n-grams
- 4-5% relative perplexity reduction: neurally interpolated n-grams outperform corresponding heuristic methods over KN models.The best model nearly matches heuristic KN even when using simple ML distributions.Source: §6.2 · pp. 6–7, Table 2 · p. 6
6.3 Results for Neural/n-gram Hybrids
- 10-11% relative perplexity reduction: neural/n-gram hybrids outperform the strong LSTM baseline on PTB and ASPEC.Block dropout contributes significantly to these gains.Source: §6.3 · p. 7, Table 3 · p. 7
6.2 Results for Neurally Interpolated n-grams
- 11kw/s and 58kw/s: LSTM- and feed-forward-based neurally interpolated models approach heuristic KN speed at 140kw/s while exceeding standard LSTM speed at 1.4kw/s.These throughput comparisons are reported on PTB.Source: §6.3 · p. 7
6.3 Results for Neural/n-gram Hybrids
- Block dropout quickly yields a better result than the LSTM LM, whereas omitting it converges worse and overweights dense count-based distributions.The comparison tracks test perplexity and probability mass assigned to non-δ distributions.Source: §6.3 · p. 7, Figure 5 · p. 8
- For lower-frequency words, n-grams outperform LSTMs, while neural/n-gram hybrids follow the LSTM pattern and neurally interpolated n-grams consistently beat standard KN n-grams.Figure 4 compares four model types across frequency thresholds up to all words.Source: §6.3 · pp. 7–8, Figure 4 · p. 7
6.4 Results for Larger Data Sets
- Larger-data hybrids significantly outperform traditional neural LMs, but the better integration strategy varies: all-data training wins for WSJ, while added distributions win with GW.The reversal is attributed to GW differing from the PTB test domain and the value of domain-specific interpolation coefficients.Source: §6.4 · p. 8, Figure 6 · p. 8
6.5 Comparison with Static Interpolation
- With diverse data, dynamic LSTM interpolation edges out static interpolation, although the methods perform similarly when only PTB data is used.The comparison combines neural or δ LMs with count-based models trained on PTB, WSJ, and GW.Source: §6.5 · p. 8, Table 4 · p. 9
7 Related Work
Related work covers interpolation, adaptation, and neural incorporation of n-gram information. The paper distinguishes its framework by learning dynamic mixture functions that also cover n-gram components and neural-network connections.
7 Related Work
- Prior work interpolates multiple LMs, adapts n-gram models, or adds binary n-gram features, but does not combine all of the paper’s stated capabilities.The paper specifically contrasts these approaches with interpolation of n-gram components, nonlinearities, and neural-network connections.Source: §7 · p. 9
- Neural models often achieve better perplexity, while n-grams retain advantages in extrinsic tasks and rare-word modeling.The paper cites prior comparisons reporting these differing strengths.Source: §7 · p. 9
- Hsu (2007) learns fallback probabilities rather than the paper’s mixture weights and does not cover interpolation of n-gram components.The distinction also includes nonlinearities and the connection with neural network LMs.Source: §7 · p. 9
8 Conclusion and Future Work
The paper presents a general framework unifying neural and count-based language models, then uses it to learn improved interpolation and hybrid models. It identifies sequential prediction tasks as possible extensions and calls for better learning methods for sparse–dense mixtures.
8 Conclusion and Future Work
- The framework generalizes neural network and count-based n-gram LMs, enabling learned interpolation functions and neural models that incorporate count-based information.These are the paper’s stated central contributions.Source: §8 · p. 9
- The framework may extend to sequential word-prediction tasks such as neural machine translation and dialog response generation.These applications are presented as possibilities rather than evaluated results.Source: §8 · p. 9
- Future work will seek more effective learning methods for mixtures of sparse and dense distributions after positive block-dropout results for hybrid models.The stated direction follows the reported hybrid-model training behavior.Source: §8 · p. 9