Morphological Priors for Probabilistic Neural Word Embeddings

• Rare and unseen words have sparse distributional evidence, especially in morphologically rich languages where words have many possible inflections.This makes dense embeddings difficult to estimate accurately from the available statistics.Source: §1 · p. 1
• Subword embeddings can generalize to rare and unseen words by representing morphemes or characters, but deterministic mappings can conflate words with similar forms.The introduction notes both misleading segmentation and cases where even correct morphology suggests an inaccurate similarity.Source: §1 · p. 1
• Morphological evidence can be useful yet misleading, so embeddings must allow corpus distributional facts to override it.The shared prefix in “incredible” and “inflammable” has opposite effects in the two words.Source: §1 · p. 2
• The model treats word embeddings as latent variables with morphology-based priors, so corpus likelihood can dominate for common words while morphology helps estimate rare or unseen words.After training, the model can impute embeddings for unseen words.Source: §1 · p. 2

2 Model

• The model combines morphological intuitions for rare words with corpus evidence for well-attested words in a joint probabilistic sequence model.For sesquipedalianism, its morphology suggests properties such as nounhood, while frequent corpus examples can instead guide the embedding.Source: §2 · p. 2

2.1 Prior distribution

• Each word has a latent binary embedding and observed morphemes, which define a prior through embeddings associated with the morphemes.The morphemes come from an unsupervised morphological segmenter treated as a black box.Source: §2.1 · pp. 2–3, §2 · p. 2

2.2 Expected likelihood

• A recurrent neural language model computes corpus likelihood from a recurrent state that depends on previous words’ embeddings.The experiments use LSTMs, whose recurrence includes input, output, and forget gates, although the exposition uses concise RNN notation.Source: §2.2 · pp. 3–4

2.3 Variational approximation

• A fully factorized Bernoulli variational distribution approximates the latent embeddings, and its expected embeddings are used in evaluation.The expected log-likelihood remains intractable because each word depends on the entire preceding sequence; the local-expectation approximation uses point estimates and does not propagate recurrent-state uncertainty.Source: §2.3 · p. 4
• The variational objective combines an expected log-likelihood term with a KL-divergence term comparing the variational distribution and prior.The terms are constructed in a computation graph for automatic differentiation and standard stochastic optimization.Source: §2.3 · pp. 4–5

3 Implementation

• The model optimizes a variational lower bound using an approximation to the conditional likelihood, updating its parameters with RMSProp; the model is called VAREMBED.The baseline uses the same LSTM architecture, allowing evaluation to focus on latent variables versus summation for word embeddings.Source: §3 · p. 5, §3.3 · p. 6

3.1 Data and preprocessing

• The 22-million-token NANT corpus is preprocessed into a 48,986-word vocabulary, and Morfessor supplies unsupervised segmentations capped at sixteen morphemes per word.The resulting inventory contains 14,000 morphemes, including stems for monomorphemic words; the method does not use labeled morphological information.Source: §3.1 · p. 5

3.2 Learning details

• Training runs for 15 epochs with minibatches of 25, gradient-norm clipping, and a learning rate that decays by 0.97 per epoch.The learning rate is halved when the development-set objective fails to improve beyond a small threshold; training takes roughly one hour per iteration on an NVIDIA 670 GTX.Source: §3.2 · p. 5

3.3 Baseline

• SUMEMBED uses the same LSTM architecture as VAREMBED, isolating the comparison between latent-variable embeddings and summation.The baseline is inspired by Botha and Blunsom (2014), whose word representations sum learned word and morpheme embeddings.Source: §3.3 · pp. 5–6

3.4 Number of parameters

• The model has roughly 12.8 million parameters, the same total as the SUMEMBED baseline.This count includes character embeddings and recurrent-model parameters.Source: §3.4 · p. 6

4.1 Word similarity

• VAREMBED consistently outperforms SUMEMBED on both word-similarity datasets, while WORD2VEC performs considerably worse than both morphology-based models on rare words.WORD2VEC is slightly better on in-vocabulary Wordsim words but cannot be evaluated on the complete dataset.Source: §4.1 · p. 6, Table 1 · p. 7

4.2 Alignment with lexical semantic features

• VAREMBED outperforms SUMEMBED on the full lexicon and performs similarly to WORD2VEC on in-vocabulary words.At the morpheme level, VAREMBED learns more meaningful embeddings, while much of SUMEMBED’s power appears to come from its word embeddings.Source: §4.2 · pp. 6–7, Table 2 · p. 6

4.3 Part-of-speech tagging

• VAREMBED performs best across all word-frequency groups except the rarest 0–100 group, where SUMEMBED is slightly better; both morphology-informed models significantly outperform WORD2VEC overall.The rarest group accounts for roughly 10% of all tokens, and the difference between SUMEMBED and VAREMBED is not statistically significant at p < .05.Source: §4.3 · p. 7, Table 3 · p. 7, Figure 2 · p. 7

Adding side information to word embeddings

• Semantic retrofitting and related methods use relational lexical resources, but do not provide a generative account of meaning construction from orthography or morphology.These approaches can capture similarity between words in resources such as WordNet.Source: §5 · p. 8

Word embeddings and morphology

• Unlike deterministic composition from morphemes or characters, morphology defines a prior that distributional statistics can override for common words; the approach also computes embeddings for unseen words.The comparison with Cotterell et al. (2016) distinguishes a joint morphology-and-corpus model from post-training retrofitting using gold morphological features.Source: §5 · p. 8

Latent word embeddings

• The model treats word embeddings as uncertain latent variables, using a binary representation that can incorporate orthographic information as a prior.Vilnis and McCallum (2014) model Gaussian densities over dense embeddings but do not use orthographic information as a prior.Source: §5 · p. 8

Neural variational methods

• Neural variational methods provide broader context for the model’s variational training, while newer inference techniques are noted as possible future alternatives to its mean-field approximation.The paper contrasts its dictionary of latent word embeddings with a variational recurrent neural network’s continuous latent variable at each time step.Source: §5 · pp. 8–9

6 Conclusion and future work

• The Bayesian latent-variable model unifies compositional and distributional perspectives, using internal structure as a prior that distributional statistics can override.The model is based on an LSTM, and its inference and estimation are described as little more complicated than a standard LSTM.Source: §6 · p. 9
• The authors expect the model to be more effective in languages with richer morphological structure than English.They describe this as a possibility for future exploration.Source: §6 · p. 9
• The current prior sums morpheme embeddings; proposed extensions include recurrent, recursive, or convolutional structures.The authors state there appears to be no technical obstacle to imposing such structure in the prior.Source: §6 · p. 9
• The authors propose comparing character-based priors with morpheme-based ones and suggest that combining both may transcend their tradeoff.Character models may generalize to names and abbreviations without segmentation, while morphemes encode regularities across words.Source: §6 · p. 9
• An inventory of lexical semantic features could partially supervise embeddings by assigning dimensions to semantic properties, complementing graph-based retrofitting.This proposed supervision would operate at the level of individual words.Source: §6 · p. 9