PaperlayerView exact revision on arXiv

Source-linked AI summary

Typological Feature Prediction with Large Language Models: An In-Context Learning Approach

Qianwen Wang, York Hay Ng, Aditya Khan, En-Shiun Annie Lee

arXiv:2609.03775v1cs.CL

TL;DR

The paper addresses incomplete typological databases and the lack of interpretable, resource-sensitive prediction methods. It evaluates in-context LLM prediction using URIEL+ and Glottolog evidence, finding that neighbour information enables LLMs to outperform baselines while maintaining performance across resource levels, and that most rationales are evidence-consistent.

  • Problem

    Typological databases contain many missing values, while existing prediction methods provide limited interpretable justification and their performance across resource levels and feature types remains underexplored.

  • Method

    The paper frames prediction as held-out binary feature imputation in URIEL+ and prompts LLMs with URIEL+ and Glottolog linguistic evidence.

  • Results

    LLMs underperform baselines with zero-shot prompting but, with neighbour evidence, outperform SoftImpute and achieve F1 0.909 in the strongest configuration.

  • Takeaways & Limitations

    Neighbour and typological evidence supports strong, relatively consistent LLM prediction across language resources and feature types, while most rationales are consistent with the evidence.

  • Takeaways & Limitations

    Evaluation holds out values that were already observed, and the binary task may not represent truly missing or gradient typological entries.

Abstract

from arXiv · show

Typological features are widely used in multilingual NLP, and the prediction of such features holds downstream utility. However, existing methods to predict missing values lack interpretable justifications for predictions, while their performance across resource levels and feature types remains underexplored. Given LLMs' abilities in meta-linguistic reasoning and in providing rationales, we investigate LLMs' performance in typological feature prediction via an in-context learning approach with linguistic data from URIEL+ and Glottolog. We find that zero-shot prompting is insufficient, but when given phylogenetic and geographic neighbour evidence, LLMs substantially outperform all baselines without disadvantaging low-resource languages. We further find that most LLM rationales are consistent with the provided evidence, offering a step toward explainable typological feature prediction.

1 Introduction

Typological feature prediction addresses incomplete linguistic databases, especially sparse coverage for low-resource languages, while seeking interpretable predictions and performance across resource levels and feature types. The paper uses in-context learning with URIEL+ and Glottolog evidence, finding that neighbour information enables strong and consistent LLM performance.

  • Motivation: 87% of the URIEL+ typological feature matrix remains empty, with sparsity worst for low-resource languages.These missing values motivate typological feature prediction.
  • Motivation: Existing prediction methods leave inequalities across resource levels and feature types underexplored and usually lack interpretable justifications.Prior approaches include k-nearest-neighbour imputation, low-rank matrix completion, and random forests trained on external features.
  • Research question: The paper asks how well pretrained LLMs reason over linguistic evidence and provide rationales for typological feature prediction.The motivation draws on LLMs’ reported meta-linguistic reasoning abilities.
  • Approach: The proposed in-context learning approach constructs prompts from URIEL+ and Glottolog data.Glottolog supplies a catalogue of languages and families.
  • Findings: Zero-shot prompting is insufficient, whereas LLMs excel at reasoning over linguistic evidence when evidence is provided.Zero-shot prompts provide only language metadata.
  • Findings: With sufficient contextual information, LLMs perform consistently across language resource levels and typological properties while exceeding non-LLM baselines.The introduction states this as a cross-resource and cross-feature conclusion.
  • Interpretability: Human annotators found that most LLM-generated rationales were consistent with the supplied evidence.This supports the use of rationales as evidence-grounded outputs.

2 Related Work

Prior typological feature prediction methods rely on imputation, external data, or multilingual representations, while LLM research shows that linguistic evidence can support meta-linguistic reasoning. This work combines that reasoning ability with rationales and evidence available for low-resource languages.

  • Typological feature prediction: The SIGTYP 2020 task established typological feature prediction over WALS, with the winning system combining conditional probabilities and language embeddings.k-nearest-neighbour imputation and SoftImpute remain important typological-data-only baselines.
  • Typological feature prediction: External-data approaches use random forests with POS-tag distributions and Wikipedia statistics or multilingual representations learned from parallel text.These approaches extend beyond the typological matrix itself.
  • LLMs and linguistic reasoning: Prior work shows that linguistic descriptions and grammar books can improve LLM translation or meta-linguistic reasoning, although parallel examples may drive some gains.The cited studies cover unseen low-resource languages and typologically diverse languages.
  • LLMs and linguistic reasoning: Unlike prior approaches, this method avoids external resources with limited low-resource-language availability while generating prediction rationales.The stated contribution links resource availability with interpretability.

3 Method

The study treats typological feature prediction as imputing held-out observed values in URIEL+, using language metadata, correlated anchor features, and phylogenetic or geographic neighbours as prompt evidence. It evaluates these LLM inputs against balanced test splits and several imputation baselines.

  • 3.1 Task Formulation: The task predicts a missing binary feature value for a language-feature pair in a URIEL+ matrix of 4,555 languages and 800 features.The matrix entries are observed values or missing entries, and evaluation uses held-out observed values.
  • 3.1 Task Formulation: The data splits balance four feature types and three language-resource groups, producing 1,200 test pairs and 888,741 observed training entries.The resource groups are low-, medium-, and high-resource languages.
  • 3.2 Prompt Construction: Each query prompt contains target-language metadata, anchor features, phylogenetic neighbours, and geographic neighbours.The prompt blocks are drawn from URIEL+ and Glottolog.
  • 3.2 Prompt Construction: Anchor features are the top 10 features most correlated with the target, with the prompt using up to five observed anchor values and their global prevalence.This supplies feature-level context for the target prediction.
  • 3.2 Prompt Construction: Phylogenetic and geographic neighbour pools select five languages using family-tree proximity or geographic distance, including feature values, anchor values, votes, and contrastive evidence.Geographic ranking uses Haversine distance, while phylogenetic ranking uses the Glottolog family tree.
  • Baselines: The comparison includes trivial, matrix-only, Glottolog-based kNN, and Random Forest baselines.Random Forest receives the same Glottolog-derived inputs as the LLMs to compare their reasoning abilities under matched information.

4 Results

Metadata-only LLM prompting underperforms established baselines, but neighbour evidence substantially improves prediction and makes performance more consistent across resource levels and feature types. The strongest configurations outperform SoftImpute, while human evaluation finds most rationales consistent with the supplied evidence.

  • Overall performance: Metadata-only LLM prompting yields F1 0.465 for Llama, 0.521 for Gemma, and 0.661 for GPT-5.5, below kNN-cosine at F1 0.775 and SoftImpute at F1 0.821.Adding anchor features raises the respective LLM scores to F1 0.512, 0.622, and 0.719, but they remain below those baselines.
  • Overall performance: Llama with all inputs achieves the highest overall performance at F1 0.909, while Gemma and GPT-5.5 reach F1 0.893 and 0.900 with anchors plus geographic neighbours.Every LLM’s strongest neighbour-based configuration outperforms SoftImpute.
  • Baseline comparison: Random Forest scores below SoftImpute and kNN-cosine despite receiving the same Glottolog-derived inputs as the LLMs.The comparison suggests an advantage for LLM reasoning under matched external information.
  • Rationales: Human annotation finds most LLM rationales consistent with the evidence, supporting their use as plausible evidence-based justifications.The conclusion presents this as a step toward explainable typological feature prediction.
  • Resource levels: Performance gaps across resource levels narrow with sufficient evidence, including GPT-5.5’s HRL–LRL gap decreasing from 0.071 to 0.012.Llama with all inputs scores F1 0.919 on LRLs and 0.914 on HRLs.
  • Feature types: With all inputs, the range of mean performance across feature types is 0.065 for Llama, 0.107 for Gemma, and 0.123 for GPT-5.5.LLMs are more consistent across feature types than most baseline methods.

5 Annotation Study

The annotation study evaluates whether the prompt supplies sufficient evidence and whether LLM rationales are consistent with that evidence. Human predictions strongly support prompt sufficiency, while Llama predictions agree substantially with humans and most rationales are evidence-consistent.

  • Fleiss’ κ = 0.888 indicates high inter-annotator agreement on predictions.
  • Human majority-vote predictions achieved F1 0.987, confirming that the prompt provides sufficient evidence for feature prediction.Individual annotators averaged F1 0.963.
  • Llama-3.1-70B achieved F1 0.915 on the annotated subset, below humans but with strong agreement with aggregated human predictions.The agreement was Cohen’s κ = 0.831.
  • Prediction accuracy increased with reported confidence for both humans and Llama, with Llama reaching perfect accuracy at high confidence.Llama generally reported lower confidence than humans, but its confidence was considered valid.
  • Annotators identified contradictions in 20.9% of Llama rationales, indicating that the majority were consistent with the provided evidence.Agreement on contradiction judgments was moderate at Fleiss’ κ = 0.415.

6 Conclusion

The paper finds that LLMs can predict typological features by integrating linguistic evidence, whereas metadata-only prompting underperforms baselines. Additional evidence improves consistency across languages and feature types, and human evaluation finds most rationales evidence-consistent.

  • The study investigates typological feature prediction through reasoning over linguistic evidence.
  • LLMs excel at integrating phylogenetic, geographic, and typological evidence for feature prediction, unlike metadata-only prompting.Metadata-only prompting underperforms baselines.
  • Providing more evidence improves LLM consistency across language resource levels and typological feature types.
  • Human annotation supports the prompt design and finds most LLM rationales consistent with the evidence provided.The study presents this as progress toward explainable prediction systems.

Limitations

The study’s conclusions are bounded by its observed-entry evaluation, binary feature representation, reliance on neighbouring-language evidence, limited sample, rationale faithfulness concerns, restricted model coverage, and contamination risk.

  • Evaluation covers only already observed entries, while truly missing URIEL+ entries may be harder and more atypical.The task also inherits a binary assumption although many typological features are gradient.
  • The method requires neighbouring-language evidence and is inapplicable where the knowledge base provides none.It is therefore not a substitute for field-linguistic documentation.
  • The evaluation uses only 1,200 examples, which may not represent whole-dataset performance and may be noisy.Equal representation of resource levels and feature types was used to mitigate this concern.
  • Inference costs create a substantial barrier to large-scale typological feature prediction and database completion.
  • Rationales may plausibly justify predictions without truthfully representing internal reasoning, and correct predictions can accompany flawed rationales.Therefore, rationales should be treated as plausible justifications only.
  • Experiments cover three model families, although observed trends are stable across them and are not necessarily architecture-specific.
  • URIEL+ data predated the models’ knowledge cutoff, creating a minor contamination risk that the authors judge minimal.Poor base-configuration performance and URIEL+ preprocessing support that judgment.

Ethical Considerations

The study uses publicly available typological data and an in-context prompt that combines metadata, correlated anchor features, and phylogenetic and geographic neighbour evidence. Predictions are constrained to binary values with short rationales and confidence labels.

  • Data and privacy: The data come from the publicly available URIEL+ typological knowledge base and contain no personally identifiable information.The task uses linguistic data rather than personal data.
  • Prompt design: The prompt combines metadata, anchor features, neighbour evidence, vote summaries, contrastive examples, and output instructions.This provides multiple evidence blocks for each prediction.
  • Anchor features: Anchor features are selected using pairwise Phi correlations, with up to five observed values included for each query language.If too few top-ranked anchors are observed, the candidate pool expands before selecting observed features.
  • Neighbour evidence: Five phylogenetic and five geographic neighbours are selected through evidence-coverage procedures and supplied with feature values and proximity information.The neighbour blocks include target-feature observations, anchor values, and contrastive support for both possible values.
  • Output constraints: The model predicts f ∈{0, 1} and returns a minified JSON object containing a rationale, value, and confidence.The rationale is limited to at most two sentences, with confidence set to low, medium, or high.

C Evaluation Metrics for Imbalanced Data

Because typological features can be highly imbalanced, the study emphasizes macro-F1 and imbalance-aware metrics rather than accuracy alone. Neighbour evidence makes LLM performance more balanced across positive and negative labels, with the strongest configurations reaching high balanced accuracy.

  • Metric choice: Macro-F1 is the primary metric because typological features can be highly imbalanced, and a majority baseline is included.The majority baseline reaches macro-F1 0.653, while the best LLM configuration reaches 0.909.
  • Metric choice: Sensitivity, specificity, and balanced accuracy are additionally reported to evaluate performance under label imbalance.Sensitivity measures positive-label recall, specificity measures negative-label recall, and balanced accuracy averages the two.
  • Results: Neighbour evidence substantially reduces the positive-negative performance asymmetry seen with metadata-only and anchor-only prompting.The imbalance-aware metrics preserve the main macro-F1 trend while showing more balanced performance after neighbour evidence is added.
  • Baselines: 0.876 and 0.873 balanced accuracy are achieved by Random Forest and SoftImpute, respectively.These are the reported non-LLM baseline results.
  • Results: 0.948 balanced accuracy is achieved by Llama-3.1-70B with all evidence, alongside sensitivity 0.961 and specificity 0.936.Gemma-4-31B with anchor and geographic evidence reaches balanced accuracy 0.946, with sensitivity 0.986 and specificity 0.907.

D Comparison Against a Plurality Voting Heuristic

The study tests whether LLM gains can be explained by simply following plurality votes over individual evidence blocks. LLMs substantially outperform this heuristic, especially when phylogenetic and geographic evidence disagree, while larger Llama models generally perform better.

  • Evaluation setup: The comparison is made on both the full test set and the non-trivial disagreement subset using Llama-3.1-70B under all inputs.Table 5 directly contrasts the model with plurality voting over individual evidence blocks.
  • Plurality comparison: Plurality voting is evaluated separately over phylogenetic neighbours, geographic neighbours, and anchor features, with ties assigned value 1.The evaluation includes a 10.2% subset where phylogenetic and geographic plurality votes disagree.
  • Plurality comparison: LLM prediction substantially outperforms plurality voting across evidence blocks and retains macro-F1 0.893 on the disagreement subset.The gap is especially large on cases where the evidence blocks disagree.
  • Model size: Llama-3.1 performance increases with model size, including a base-configuration increase from F1 0.147 to 0.465.The 8B model also shows configuration-specific behaviour, with geographic evidence outperforming the all-input configuration.

G Qualitative Analysis of LLM Rationales

The qualitative analysis finds that LLM rationales often summarize prompt-provided evidence, especially when sources agree, but they can misinterpret conflicts, overweight proximity, or omit aligned evidence. Rationales therefore provide useful clues to predictions without fully explaining model behavior.

  • Contradiction subset: The contradiction subset contained 30 cases; LLM accuracy was 0.867, compared with 0.967 for human plurality.F1 was 0.464 for the LLM and 0.492 for human plurality, with four cases combining inconsistent rationales and incorrect predictions.
  • Evidence use: Cross-source agreement produced the clearest faithful rationales when neighbour evidence and anchor statistics supported the same label.In the Daloa Bété example, all five phylogenetic and geographic neighbours and both anchor clues supported value 1.
  • Evidence use: The model usually prioritized phylogenetic and geographic neighbour evidence over global anchor trends, especially when neighbours agreed.For Awadhi, it selected the gold value while explicitly citing majority neighbour support; for another case, it prioritized local neighbours over a global trend.
  • Rationale faithfulness: Rationales did not always explain how conflicting neighbour evidence was weighted and sometimes misrepresented or omitted available anchor evidence.The model described closest-neighbour support in conflict cases, while another rationale cited anchor evidence that the prompt had not provided.
  • Rationale faithfulness: Proximity evidence could be overweighted, producing an incorrect prediction despite consistent majority neighbour and anchor support.For Garo, both neighbour blocks and both anchor clues supported value 0, but the model predicted 1.
  • Takeaway: The rationales are best understood as prompt-grounded evidence summaries rather than evidence of innate typological knowledge.They did not mention target-language metadata or external facts, although they often offered useful clues toward the prediction.

H Baseline Architectures

The baseline systems combine neighbour information, anchor-feature values, metadata, and aggregate voting statistics to predict typological features. Their hyperparameters are selected on validation macro F1, and the experiments use specified compute infrastructure and licensed artifacts.

  • Neighbour baselines: The k-nearest-neighbour baselines use k = 3 for cosine similarity and k = 6 for phylogenetic and geographic neighbours.These values were selected by validation macro F1, choosing the point where performance plateaued or decreased; ties predict 1.
  • Random-forest baseline: The random-forest baseline trains one classifier per typological feature using 42 inputs matched to the information shown to the LLM.Inputs include language metadata, observed anchor-feature values and prevalences, and phylogenetic and geographic neighbour voting statistics.
  • Random-forest baseline: Neighbour-derived inputs include vote counts, missing-value counts, and the ranks of the closest neighbours voting yes or no.These aggregates are constructed separately for phylogenetic and geographic neighbours.
  • Random-forest baseline: The random forest uses n_estimators = 500 and max_depth = 8 after a validation macro-F1 grid search.Remaining hyperparameters use scikit-learn defaults.
  • Infrastructure and artifacts: Open-source model experiments ran on one Nvidia H100 GPU for 25 compute hours, while other experiments ran on CPU and GPT-5.5 used Azure OpenAI inference.The study also reports artifact licenses and states that their use complied with those licenses.
Loading 2609.03775v1…