Source-linked AI summary
Compositional Morphology for Word Representations and Language Modelling
Jan A. Botha, Phil Blunsom
TL;DR
Morphologically rich languages create data sparsity because conventional language models do not explicitly encode relationships among related word forms. The paper integrates compositional factor vectors into probabilistic log-bilinear language models and makes them decoder-efficient through class-based vocabulary decomposition. Across similarity, language-model, scaling, and translation evaluations, the approach improves representation quality and language-model performance, with reported translation gains and large-vocabulary tractability, while target-word scoring for unknown forms remains outside the implemented test-time model.
Problem
Morphologically rich languages produce sparse word-form data, while conventional language models lack explicit representations for related and unseen forms.
Method
The method composes word vectors from variable-length morphological factors within a log-bilinear probabilistic language model, using class-based decomposition for efficient normalization.
Results
The models improve word-similarity correlations across multiple languages, reduce perplexity, and improve translation quality across six language pairs; scaling remains tractable for nearly 1m types.
Takeaways & Limitations
Morphology-guided continuous-space language models provide a flexible approach for morphologically rich languages and improve intrinsic and translation evaluations relative to the reported baselines.
Takeaways & Limitations
Constructing representations for unknown target words requires changing the probabilistic event space, so test-time target-word extensions are left for future work.
Abstract
from arXiv · showhide
This paper presents a scalable method for integrating compositional morphological representations into a vector-based probabilistic language model. Our approach is evaluated in the context of log-bilinear language models, rendered suitably efficient for implementation inside a machine translation decoder by factoring the vocabulary. We perform both intrinsic and extrinsic evaluations, presenting results on a range of languages which demonstrate that our model learns morphological representations that both perform well on word similarity tasks and lead to substantial reductions in perplexity. When used for translation into morphologically rich languages with large vocabularies, our models obtain improvements of up to 1.2 BLEU points relative to a baseline system using back-off n-gram models.
1 Introduction
The paper addresses morphological data sparsity in language models by building morphological awareness into continuous-space representations. It combines compositional word vectors with probabilistic language modelling and evaluates the approach through similarity, perplexity, and translation experiments.
- Morphologically rich languages create many word forms, causing data sparsity and leaving conventional language models without explicit representations for related or unseen forms.Examples include abstract, abstraction, and abstracted.
- Continuous-space language models capture some morphology, but the paper argues for incorporating morphological awareness directly into their inductive bias.
- Word vectors are composed from arbitrary sub-elements such as surface forms, stems, affixes, or latent information, tying related words together.The model is implemented in a log-bilinear language model.
- The model is made efficient for decoder integration through word classing and is evaluated on translation into 6 languages, including Czech, German, and Russian.
- Morpheme vectors improve correlation with human word-similarity ratings across multiple languages, while scaling experiments demonstrate tractability on vocabularies of 900k types using 100m+ tokens.
2 Additive Word Representations
The method represents each word compositionally using a variable-length set of factor vectors, with addition linking morphologically related words. Surface-form factors preserve noncompositional information and support robust representations, including for out-of-vocabulary words.
- A generic continuous-space language model assigns each vocabulary word an opaque d-dimensional vector, providing only rudimentary use of linguistic structure.
- Morphologically related words should share statistical strength despite surface-form differences, while retaining the unsupervised nature of continuous-space representations.
- Each surface word maps to a variable-length sequence of factors, which can represent morphemes and optionally incorporate lemmas or parts of speech.
- Additive composition sums factor vectors, linking representations of morphologically related words through shared factors.The paper illustrates this with imperfection and perfectly.
- Factor vectors can construct representations for out-of-vocabulary words from their available morphemes.
- Including the surface form as a factor handles noncompositional constructions, improves robustness to noisy segmentation, and avoids order-invariance problems.The examples are greenhouse, hangover, and overhang.
- The number of factors varies across words, allowing the approach to cover fusional and agglutinative languages without imposing a fixed factor count or partitioned feature space.
3 Log-Bilinear Language Models
The paper embeds additive morphological representations in log-bilinear language models and uses class-based vocabulary decomposition to make probability normalization efficient enough for decoding. The resulting design ties shared factors, supports rare forms, and preserves scalable training and lookup.
- Log-bilinear models are continuous-space language models that retain the n-gram Markov assumption while scoring words with smooth vector-based functions.
- The LBL predicts the next-word vector from the preceding context vectors, then scores a candidate word by a dot product plus its bias.
- LBL++ replaces word representations with additive factor-based representations for target and context words, with factor matrices supplying vectors for factor types.
- Words sharing factors are tied together, which is expected to improve performance on rare word forms.
- Compiled word vectors make test-time LBL++ probability lookups cost the same computationally as LBL lookups.
- LBL+o factorizes only output words, whereas LBL+c factorizes only context words; both reduce to LBL when the factor mapping is the identity.
- Class-based decomposition addresses the expensive vocabulary normalization that otherwise obstructs using continuous-space language models in a decoder.
- The class-based model assigns vectors and biases to word classes and normalizes class and word scores separately.
4 Experiments
The experiments show that additive morphological representations improve language-model perplexity and word-vector quality across languages, while remaining tractable for large-vocabulary translation. Gains concentrate on rare and morphologically informative tokens, and translation improves by up to 1.2 BLEU points over MKN baselines.
- Intrinsic language-model evaluation: Both context and output factorisation consistently reduce perplexity across all 6 languages, outperforming models that factorise only one side.The results support modelling morphological dependencies across context and output words.
- Intrinsic language-model evaluation: 2%–6% perplexity reductions persist on larger datasets, with the biggest gains for Czech and Russian.These improvements are slightly smaller than those observed with small training data.
- Intrinsic language-model evaluation: 8%–21% relative perplexity reductions occur for Czech, German, and Russian tokens seen fewer than 100 times, while high-frequency-token gains are negligible.The strongest effects therefore occur in the long tail rather than among frequent punctuation and closed-class tokens.
- Intrinsic language-model evaluation: 15% improvement on German adjective-noun sequences and 21% on adjective-adjective-noun sequences indicate gains on morphologically marked agreement patterns.The larger improvement concerns sequences such as repeated adjective inflection in German noun phrases.
- Scaling: 14% improvement over MKN remains at the largest Czech setting, while additive gains over CLBL decrease to 2%–3% at 128m tokens.The experiment demonstrates tractability on vocabularies of nearly 1m types, although maintaining a constant MKN advantage would require increasing representation dimensionality.
- Word similarity: Word-similarity correlations improve by 64% on English RW, 26% on English WS353, up to threefold on German, and 39% on French.On English RW, the model slightly trails the best csmRNN result but outperforms the csmRNN using alternative embedding initialisation.
5 Related Work
Prior work incorporated factors, bilingual embeddings, or linguistic information into continuous-space models, but the paper targets morphological awareness within a probabilistic language model.
- Factored language models represent words through sets of factors in discrete n-gram and continuous-space language models.Earlier work showed context-word factorisation could help with out-of-vocabulary words, but did not evaluate factorising output words or perform extrinsic evaluation.
- Continuous-space models have been applied to speech recognition and translation, but efficient systems often restricted them to shortlists of common words.Remaining words were handled by back-off n-gram models in that line of work.
- Recent approaches used bilingual embeddings, fully continuous translation models, and linguistically informed representations such as morphology or syntax.The paper positions its contribution as creating morphological awareness in a probabilistic language model.
6 Conclusion
The paper integrates morphology into probabilistic continuous-space language models and evaluates the approach across morphologically rich languages and multiple tasks. The models improve intrinsic performance and often translation quality, while further morphology-based translation gains beyond optimiser variance appear only for English→Czech.
- The method integrates morphology into probabilistic continuous-space language models and is flexible across morphologically rich languages.The number of factors can vary with vocabulary items, supporting different linguistic typologies.
- The evaluation covers multiple morphologically rich languages and different intrinsic and extrinsic tasks.The reported tasks include language modelling, word similarity, and machine translation.
- Morphology-guided continuous-space language models improve intrinsic language-model performance against baseline CSLMs and n-gram MKN models.
- Word and morpheme representations compare favourably on word similarity with a more complex model that used more data, with large gains on some languages.
- BLEU improves consistently across six language pairs with CSLM features, but morphology-based gains beyond optimiser variance occur only for English→Czech.