Source-linked AI summary
Learning Language Representations for Typology Prediction
Chaitanya Malaviya, Graham Neubig, Patrick Littell
TL;DR
Typological databases contain relatively full feature specifications for only a few hundred languages, raising the question of whether neural machine translation can supply information about languages’ typology. The paper trains a many-to-one NMT system on 1017 languages and uses its learned representations to predict typological features. The method predicts syntactic as well as phonological and phonetic-inventory features, and improves over a geographic and genetic neighbor baseline.
Problem
Typological databases contain relatively full feature specifications for only a few hundred languages, while NLP applications need fuller feature sets for the languages they use.
Source: §1 · p. 1Method
The paper trains multilingual NMT from 1017 languages into English and uses learned language representations, including averaged encoder cell states, to predict typological features.
Source: §1 · p. 2Results
The representations support prediction of syntactic, phonological, and phonetic-inventory features, and improve over a baseline using geographic and genetic neighbors.
Source: §4.2 · pp. 4–5Takeaways & Limitations
The study shows that neural models can learn a range of linguistic concepts and may be used to impute missing features in typological databases.
Source: §5 · p. 5Bible translations may contain archaic language, though the authors’ data inspection found no such archaisms and they judge any effect likely limited.
Source: §4.1 · p. 3
Abstract
from arXiv · showhide
One central mystery of neural NLP is what neural models "know" about their subject matter. When a neural machine translation system learns to translate from one language to another, does it learn the syntax or semantics of the languages? Can this knowledge be extracted from the system to fill holes in human scientific knowledge? Existing typological databases contain relatively full feature specifications for only a few hundred languages. Exploiting the existence of parallel texts in more than a thousand languages, we build a massive many-to-one neural machine translation (NMT) system from 1017 languages into English, and use this to predict information missing from typological databases. Experiments show that the proposed method is able to infer not only syntactic, but also phonological and phonetic inventory features, and improves over a baseline that has access to information about the languages' geographic and phylogenetic neighbors.
1 Introduction
Typological databases are often sparse, while NLP tasks need relatively complete feature sets for the languages they use. This study tests whether representations learned by multilingual neural machine translation can help infer missing typological features.
1 Introduction
- Typology concerns language features and the relationships and correlations among them, and has practical applications in NLP, particularly for lower-resource languages.The cited examples include multilingual dependency parsing, low-resource generative parsing, language modeling, POS tagging, and machine translation.Source: §1 · p. 1
- Typological databases may be sparse by design or necessity, while NLP tasks often need relatively full feature sets for the languages being used.The mismatch motivates reconstructing missing database entries from known ones.Source: §1 · p. 1
- The study tests whether multilingual neural machine translation can infer typological features from parallel corpora and help fill gaps in sparse databases.The authors frame the work as a holistic analysis of what neural networks learn about an entire language by aggregating representations over many sentences.Source: §1 · pp. 1–2
- Neural representations from an NMT model trained on 1017 languages support typology prediction, with particular gains for word order and case markers.The paper examines these representations on their own and combined with vectors based on languages’ genetic and geographic distances.Source: §1 · p. 2
2 Dataset and Experimental Setup
The experiments use URIEL’s binary language features and compare learned representations with a geographic and genetic nearest-neighbor baseline. Individual logistic-regression classifiers are evaluated using ten-fold cross-validation.
2 Dataset and Experimental Setup
- URIEL supplies binary typological features drawn from multiple typological, phylogenetic, and geographic databases.The sources include WALS, PHOIBLE, Ethnologue, and Glottolog.Source: §2 · p. 2
- The database covers 103 syntactic, 28 phonological, and 158 phonetic-inventory features.Examples include prepositions or postpositions, complex syllabic onset clusters, and interdental fricatives.Source: §2 · p. 2
- The baseline averages features from each language’s three nearest neighbors by normalized geodesic and genetic distance.This method does not necessarily require pre-existing typological knowledge of the language being predicted.Source: §2 · p. 2
- Separate logistic-regression classifiers predict each feature, evaluated by training on nine-tenths of the languages and testing on the remaining tenth across ten folds.The evaluation uses ten-fold cross-validation over URIEL.Source: §2 · p. 2
3 Learning Representations for Typology Prediction
The paper compares language vectors learned by multilingual language models and neural machine translation with a new representation based on averaged encoder cell states. These representations are intended for typology prediction.
3 Learning Representations for Typology Prediction
- LMVEC uses the learned word representation of a special source-language token prepended to each sentence as the language vector.The approach builds on multilingual neural language models that provide a language vector as input.Source: §3 · pp. 2–3
- MTVEC uses a language embedding learned by a multilingual neural machine translation model translating many languages into English.The authors hypothesize that alignment to the same English sentence provides a stronger signal for learning language-consistent features than language modeling alone.Source: §3 · p. 3
- MTCELL represents a language by averaging the encoder LSTM’s hidden cell states across all sentences in that language.The averaging includes all time steps of all sentences; cell states are considered because LSTM hidden cells can capture interpretable information such as syntax.Source: §3 · p. 3
- Table 1 reports accuracy for syntactic, phonological, and inventory features across LMVEC, MTVEC, MTCELL, and MTBOTH.It distinguishes majority-class chance without auxiliary information from a 3-NN baseline using geographic and genetic neighbors.Source: Table 1 · p. 3
4 Experiments
Neural feature vectors beat chance, with bilingual and hidden-cell representations improving typology prediction; gains over geographic and phylogenetic neighbor features are smaller but persist. The experiments also suggest these representations capture phonological and phonetic inventory patterns, potentially through regional clusters.
4.1 Multilingual Data and Training Regimen
- A Bible-translation corpus provided data for 1017 languages and 20.6 million training sentences; inspection found no archaic language likely to affect results substantially.The authors attribute this in part to most translations being produced in the late 19th and 20th centuries, and to antique-Bible languages also having other translations.Source: §4.1 · p. 3
- Mean encoder LSTM cell states across sentence words outperformed mean final hidden cell states, while hidden states contained significantly less information and variance.The authors attribute the hidden-state difference to modulation by the output gate at each time step.Source: §4.1 · p. 3
4.2 Results and Discussion
- All neural vectors beat chance; LMVEC improves 2-3% over MTVEC, MTCELL further outperforms MTVEC, and MTBOTH improves further, with significant syntax and inventory gains (p=0.05).The phonology gain was not significant, perhaps because the database contains only 28 phonological features.Source: §4.2 · p. 4
- With geodesic/genetic neighbor vectors, the main trends largely persisted but gains were much smaller; LMVEC gains disappeared, suggesting aligned data may be essential for inferring a new language’s typology.Some features, including gender markers, case suffix, and negative affix, decreased in accuracy, but by less than the improvements elsewhere.Source: §4.2 · p. 4
- The method also infers phonological and phonetic-inventory features; the authors suggest syntax may identify regional clusters, with gains on regionally distinctive sound features as a side effect.Examples include uvular consonants, initial velar nasals, and lateral consonants; Table 2 presents top improvements across syntax, phonology, and inventory.Source: §4.2 · pp. 4–5, Table 2 · p. 4
- For S OBJ BEFORE VERB, the selected cell’s trajectories tracked closely for Portuguese and Catalan, while languages where objects precede verbs had higher average values.The similar trends across languages sharing the feature value suggest the selected node’s information is consistent across languages with similar structures.Source: Figure 2 · p. 4, §4.2 · p. 5
5 Conclusion and Future Work
The study shows that neural models learn linguistic concepts and may help fill missing typological database features. It highlights parallel text and representation dynamics, and calls for further work on downstream typology use and prediction for under-documented languages.
5 Conclusion and Future Work
- Neural models can learn linguistic concepts and may be used to impute missing features in typological databases.The study emphasizes learning representations from parallel text and suggests that modeling representation dynamics as sentences are processed may matter.Source: §5 · p. 5
- The authors hope the findings encourage further use of typological features in downstream NLP tasks.Source: §5 · p. 5
- The authors also hope to inspire further missing-knowledge prediction techniques for under-documented languages.Source: §5 · p. 5