PaperlayerView exact revision on arXiv

Source-linked AI summary

Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing

Edoardo Maria Ponti, Helen O'Horan, Yevgeni Berzak, Ivan Vulić, Roi Reichart, Thierry Poibeau, Ekaterina Shutova, Anna Korhonen

arXiv:1807.00914v3cs.CL

TL;DR

Multilingual NLP needs information that captures cross-lingual variation despite scarce labeled resources. This paper surveys typologically informed NLP, finding consistent but often moderate gains from existing typological features, and advocates continuous, contextual typology induced from data.

  • Problem

    Many languages lack labeled resources, while existing typological databases have limited coverage and coarse-grained feature values.

  • Method

    The paper surveys typology-informed NLP methods, database-based and automatically inferred typological information, and strategies for integrating these features into models.

  • Results

    Typological features yield consistent, often moderate improvements over baselines, while current NLP uses only a limited subset of available typological information.

  • Takeaways & Limitations

    Typology is relevant across many NLP tasks and offers a principled basis for language transfer and multilingual joint learning.

  • Takeaways & Limitations

    Existing typological databases are incomplete, approximate, discrete, and often omit variation within languages by reporting only majority values.

Abstract

from arXiv · show

Linguistic typology aims to capture structural and semantic variation across the world's languages. A large-scale typology could provide excellent guidance for multilingual Natural Language Processing (NLP), particularly for languages that suffer from the lack of human labeled resources. We present an extensive literature survey on the use of typological information in the development of NLP techniques. Our survey demonstrates that to date, the use of information in existing typological databases has resulted in consistent but modest improvements in system performance. We show that this is due to both intrinsic limitations of databases (in terms of coverage and feature granularity) and under-employment of the typological features included in them. We advocate for a new approach that adapts the broad and discrete nature of typological categories to the contextual and continuous nature of machine learning algorithms used in contemporary NLP. In particular, we suggest that such approach could be facilitated by recent developments in data-driven induction of typological knowledge.

1. Introduction

Cross-lingual variation and scarce labeled resources limit multilingual NLP, especially beyond resource-rich languages. The survey examines how typological information can guide multilingual NLP and how it should be sourced and encoded.

  • Motivation: Surface structures vary substantially across languages, challenging robust multilingual NLP and contributing to its concentration on resource-rich languages.Current architectures and tuning often incorporate language-specific biases and require labeled data unavailable for many languages.
  • Motivation: Existing responses to the multilingual data bottleneck include unsupervised learning, transfer from resource-rich languages, and multilingual joint learning.These approaches seek to reduce dependence on manually annotated resources.
  • Typology as a resource: Linguistic typology systematically compares languages and offers databases that can guide feature choice and algorithm design.The World Atlas of Language Structures is identified as a prominent resource.
  • Survey scope: The survey reviews typologically informed NLP, including structural and semantic information, neural approaches, database sources, and automatically inferred features.It focuses particularly on improving the completeness and granularity of typological information.
  • Research questions: The paper asks which NLP tasks benefit from typology, how typological information should be sourced and encoded, and whether it improves over typology-agnostic or genealogical alternatives.It also considers applications to data selection, rule-based systems, and model interpretation.

2. Overview of Linguistic Typology

Linguistic typology systematically compares languages using empirically grounded, function-based categories, while documenting bounded but gradient variation. Its databases provide broad guidance despite uneven evidence and incomplete coverage.

  • Foundations: Linguistic typology studies language variation through systematic comparison, with categories emerging inductively and grounded in functional rather than formal criteria.Typologists distinguish constructions, abstract functions, and language-specific strategies.
  • Documentation: Typological documentation stores empirically assessed language features as attribute–value pairs in large databases.Attributes generally correspond to constructions, while values represent strategies used in particular languages.
  • Universals: Cross-linguistic patterns are bounded and interdependent, but typological universals are usually statistical tendencies rather than exceptionless rules.Restricted universals capture feature dependencies, whereas unrestricted universals specify properties shared across languages.
  • Domains of variation: Typology covers structural variation in forms and combinations as well as semantic variation in conceptual granularity, boundaries, and membership.Structural typology includes morphosyntax and word order; semantic typology focuses especially on lexical concepts.
  • Sources of variation: Language similarities may reflect genealogical inheritance or geographic contact, motivating typology’s effort to explain what patterns occur where and why.The field treats language as a system shaped by biological and cultural evolution.
  • Gradience: Typological patterns are gradient because languages can use multiple strategies and display partly inconsistent semantic categories.Semantic and grammatical categories can be represented on continuous multidimensional maps.
  • Coverage and limitations: Typological evidence is highly unbalanced, yet WALS provides syntactic features for up to 1519 languages.This coverage can offer guidance where manually constructed resources are lacking, although many languages remain undocumented or unwritten.

3. Overview of Multilingual NLP

Multilingual NLP addresses severe resource scarcity through unsupervised learning, language transfer, and joint multilingual modeling. These methods differ in how they project annotations, transfer models, translate data, or share representations, and typology can guide their choices.

  • Resource scarcity: Most languages lack the annotated datasets and lexical resources required by supervised NLP, while creating complete coverage is unrealistic.Resource creation is time-consuming, requires skilled labor, and must span many tasks and languages.
  • Unsupervised learning: Unsupervised methods avoid annotated resources by inferring latent structures from unlabeled text.They construct probabilistic models of observations conditioned on hidden variables.
  • Transfer and joint learning: Language transfer moves information from resource-rich source languages to resource-poor target languages, but differences in lexica and word order require adaptation.Joint multilingual learning instead leverages annotated examples across languages and language inter-dependencies.
  • Language transfer: Annotation projection parses and aligns source and target texts, then projects source annotations to train a supervised target-language model.Soft projection can supplement alignment with distributional or constituent-based constraints.
  • Language transfer: Model transfer applies a source-trained model to a target language, typically using delexicalized, language-independent, or harmonized features.Multilingual word clusters and distributed representations can help bridge incompatible vocabularies.
  • Language transfer: Machine translation can produce target-language text whose projected annotations or multilingual representations support transfer without annotated parallel data.The source may be translated directly or through a bilingual lexicon.
  • Transfer constraints: Parallel-text requirements hamper annotation projection and translation, while model transfer is reported as competitive with machine translation.Typological knowledge is therefore used especially with model transfer and for selecting compatible source languages.
  • Joint learning: Joint multilingual models can surpass monolingual models by leveraging more data, particularly for resource-lean languages and code-switching.Parameter sharing, private or shared representations, and language vectors are central strategies.

4. Selection and Development of Typological Information

The paper reviews how typological information is selected, processed, and encoded for NLP, while addressing database limitations in coverage and feature granularity.

  • Section scope: This section presents major typological databases, methods for selecting and encoding NLP-relevant features, and approaches for obtaining missing or finer-grained information automatically.It also highlights limitations in database coverage and feature granularity.

4.1 Hand-Crafted Documentation in Typological Databases

Hand-crafted typological databases organize documented language variation into feature–value systems spanning multiple linguistic levels, but their usefulness is constrained by incomplete coverage, coarse granularity, restricted applicability, and redundancy.

  • Database organization: Typological databases store taxonomies of features, possible values, and documented feature values for languages, often visualized by assigning each language a feature value.WALS and APiCS cover multiple description levels, while other resources focus on syntax, phonology, or semantics.
  • Major databases: WALS is the most widely used typological database in NLP and contains 142 features spanning phonology, morphology, nominal categories, word order, and other domains.The supplied passage identifies WALS as covering multiple linguistic description levels.
  • Database limitations: Database coverage is limited because feature values are missing for most languages in most resources.This limitation directly constrains the usefulness of typological databases for multilingual NLP.
  • Database limitations: Databases often report only a language’s majority feature value, omitting within-language variation and the frequencies of alternative values.For Italian adjective–noun order, the dominant order is recorded while an attested opposite order may be absent.
  • Database limitations: Some features apply only when prerequisite values hold, while others are redundant or subsume related features.WALS features 113A and 114A illustrate restricted applicability, while feature 81A subsumes information in 82A and 83A.

4.2 Feature Selection from Typological Databases

NLP studies have selected typological features ranging from small WALS subsets to complete inventories, but selecting features that are non-redundant, discriminative, and informative remains difficult.

  • Feature-set scope: Most dependency-parsing studies use similar subsets of WALS word-order features because word order provides crucial guidance for parsing.Subsequent studies often discard features whose values are constant across the languages under study.
  • Feature-set scope: Some studies use broader sets combining WALS word-order features with nominal categories, nominal syntax, or pruned morphosyntactic inventories.Berzak, Reichart, and Katz (2015) retained 119 non-redundant morphosyntactic features from WALS.
  • Feature-set scope: Other experiments use complete database inventories, including all WALS features, all URIEL features, or hundreds of syntactic implicational universals.These studies avoid pre-selecting a smaller feature subset.
  • Feature representation: Typological attributes are commonly represented as binarized feature vectors, which harmonize heterogeneous features but obscure differences among variable types.Binarization creates a separate indicator for each possible value of each attribute and increases dimensionality.
  • Coverage and informativeness: 79.8% of feature values were populated for the 14 languages in Berzak, Reichart, and Katz (2015), compared with 17 percent for all languages in WALS.Experimental languages are generally well documented, so coverage is broader in surveyed NLP studies than in WALS overall.
  • Coverage and informativeness: Hierarchical clustering shows that large genus-averaged WALS encodings can mirror genealogy, while smaller word-order encodings may group languages across genealogical boundaries but miss related-language differences.The word-order set merges English and Czech yet gives French, Spanish, and Italian identical encodings.
  • Coverage and informativeness: The survey concludes that selecting typological features that are non-redundant with genealogy, fully discriminative, and informative is difficult, motivating automatic prediction of typological information.Most surveyed experiments use only a small number of databases and features.

4.3 Automatic Prediction of Typological Features

Automatic acquisition addresses incomplete and coarse typological databases through heuristics, propagation, supervised learning, and distributional methods. The surveyed evidence indicates that strategy effectiveness depends on feature type and available resources, while continuous representations offer a promising alternative to discrete categories.

  • Acquisition strategies: Automatic prediction fills missing typological values using annotation-based heuristics, database propagation, supervised learning, and multilingual distributional evidence.These approaches extend database coverage and can capture information absent from manually crafted resources.
  • Acquisition strategies: Morphosyntactic heuristics derive word-order features from dependency directions or constituency structures, including projected annotation for resource-lean languages.For Welsh, relative verb–subject and verb–object order can be inferred from the positions of relevant constituency-tree nodes.
  • Strategy comparisons: Typology-based clustering can outperform genealogical clustering for unsupervised propagation, while k-means is reported as the most reliable among examined clustering methods.These comparisons concern prediction of missing typological values under the surveyed experimental settings.
  • Strategy comparisons: WALS vectors resemble genealogical information, whereas artificial tokens and encoder hidden states predict missing values comparably without forming genealogical clusters.The survey therefore characterizes discrete and continuous representations as informative but potentially complementary.
  • Limitations and future directions: Comparisons are difficult because language samples and data partitions vary, and many strategies incorrectly assume languages are independent and identically distributed despite family, area, and implicational interactions.The survey identifies continuous language vectors and multilingual distributional information as promising ways to preserve gradient variation rather than reduce languages to discrete types.
  • Strategy comparisons: No single acquisition algorithm is optimal: annotation extraction suits word order, distributional heuristics suit lexicalization, and propagation or supervised learning are more general-purpose.The strategies also require different resources, including annotated or parallel texts versus an existing database.

5. Uses of Typological Information in NLP Models

Typological information is incorporated into NLP through rules, constraints, parameter sharing, multilingual biasing, and data selection. Across surveyed tasks, typology-enriched systems consistently outperform baselines, but improvements are generally modest and typology is not clearly superior to genealogy or language identity.

  • Uses of typological information: Typological features support expert-system rules, algorithmic constraints, parameter sharing, and data selection or synthesis.The survey identifies manual rule conversion, parameter tying across languages, and guidance for selecting or synthesizing data as three main uses.
  • Selective sharing: Selective sharing separates universal head–modifier relations from language-specific word order in multilingual parsing.Its discriminative formulation removes strong independence assumptions and applies features only to relevant head–dependency pairs.
  • Selective sharing: Typological features can be treated as latent variables when observed database features do not match the source languages.Naseem, Barzilay, and Globerson propose learning these latent features with expectation maximization.
  • Data selection and preprocessing: Language selection and weighting combine typological, genealogical, character-based, and part-of-speech similarity signals, each with distinct weaknesses.Language identity and typology may fail to abstract away from scripts, while part-of-speech metrics deteriorate with little data.
  • Comparison: Typology-enriched models consistently outperform baselines across several NLP tasks, but improvements are generally moderate and rarely supported by significance tests.The surveyed comparisons use typology, data-driven features, or a baseline under otherwise identical architecture and hyper-parameters.
  • Comparison: Typology is not consistently superior to genealogy or language identity, while tensor-based discriminative selective sharing outperforms graph-based and generative alternatives.The survey also finds a tendency for typological features to interact with deeper model representations rather than only input representations.
  • Comparison: Overall, typology can potentially guide algorithm architecture, feature engineering, and data selection, although substantial challenges remain.

6. Future Research Avenues

The survey proposes extending typological NLP beyond predominantly morphosyntactic applications and integrating richer, softer, and more continuous information. It identifies incomplete, approximate, and discrete databases, along with efficient soft-constraint integration, as central challenges and motivates data-driven alternatives.

  • New tasks and resources: Future work should extend typology-informed NLP to new tasks and underused resources, beyond the current concentration on morphosyntactic parsing and WALS word-order features.Suggested areas include phonology, semantics, lexical semantics, morphology, and additional typological databases.
  • New tasks and resources: Typological frame semantics could support semantic role labeling through priors, alignment guidance, or model regularization.It encodes predicates’ argument number, morphological marking, and ordering across languages.
  • Model integration: External linguistic-knowledge frameworks such as relational vector constraints suggest broader ways to integrate typological information into neural and feature-based models.The survey discusses these frameworks as applicable to typological integration.
  • Model integration: Because typological information reflects tendencies rather than strict rules, soft constraints are a natural encoding choice, but incorporating them efficiently into inference remains challenging.Inference must bias predictions toward constraints while preserving the efficiency of searching possible outputs.
  • Database limitations: Existing typological databases incompletely document languages and features, approximate within-language variation through majority strategies, and represent information discretely.Raw textual data is presented as a possible source for automatically learning more complete knowledge.
  • A new typology: Automatically induced typology could better match machine learning by representing cross-lingual variation as contextual, continuous, and probabilistic information.The paper argues that current curated features are a poor fit for continuous, contextual models and that textual induction may provide more suitable representations.
  • A new typology: Real-valued language representations could encode strategy frequencies and support example-level sharing, contextual biasing, and selection of similar sentences rather than whole languages.The proposed shift treats languages as internally variable rather than indissoluble blocks.

7. Conclusions

The survey finds that typological information supports multilingual NLP across tasks, but current databases and modeling practices limit its impact. It therefore advocates learning richer, continuous typological representations directly from linguistic data.

  • Typological information is used mainly for morphosyntactic tasks, especially dependency parsing, while phonological, predicate–argument, and lexical-semantic features remain underused.These neglected feature types could support tasks including semantic role labeling, word sense disambiguation, and sentiment analysis.
  • Typological databases are incomplete, so researchers use heuristics, language similarity, supervised models, and distributional methods to predict missing values.No technique performs best across all feature types, motivating future ensemble methods.
  • Selective sharing transfers universal properties broadly while restricting language-specific learning to typologically similar examples; other models concatenate typological features or condition hidden representations.
  • Typological features yield consistent, though often moderate, improvements over baselines without such features, and generally outperform genealogy-based or other language-property features on diversified samples.The latter features can perform similarly when correlated with typology but fall short for fine-grained differences among languages.
  • The paper proposes learning typology automatically from raw linguistic data to capture within-language variation and encode typological information continuously for contextual machine-learning models.The main future challenge is integrating these induced representations into models rather than sourcing features from databases.
Loading 1807.00914v3…