PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM Agents as Computational Typologists

Changbing Yang, Christopher Hammerly, Freda Shi, Jian Zhu

arXiv:2609.07791v1cs.CL

TL;DR

Traditionally, creating a typological database takes years of effort from hundreds of trained linguists, making the process unscalable. AUTOTYPOLOGIST is an LLM agent that retrieves relevant grammar sections and IGT examples and iteratively reasons over evidence for typological analysis. AUTOTYPOLOGIST reproduces expert feature codings with high agreement when reference-grammar prose is available and identifies supporting evidence and counterexamples in hypothesis testing.

  • Problem

    Traditionally, creating a typological database takes years of effort from hundreds of trained linguists, making the process unscalable.

    Source: §1 · p. 1

  • Method

    AUTOTYPOLOGIST is an LLM agent that retrieves relevant grammar sections and IGT examples and iteratively reasons over evidence for typological analysis.

    Source: §1 · p. 1, §5.1 · p. 4

  • Results

    AUTOTYPOLOGIST reproduces expert feature codings with high agreement when reference-grammar prose is available and identifies supporting evidence and counterexamples in hypothesis testing.

    Source: §1 · p. 1

  • Takeaways & Limitations

    Structured tools and traceable evidence can support typological analysis, while LLM agents can assist linguists in gathering evidence for hypothesis testing.

    Source: §1 · p. 1, §9 · p. 9

    Extensive human validation is still needed to assess generated hypotheses, and reference grammars provide only static snapshots of living languages.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Linguistic typology relies on expert analysis of reference grammars across languages, making large-scale crosslinguistic comparison labor-intensive and unscalable. We introduce AUTOTYPOLOGIST, an LLM agent for evidence-grounded typological analysis over reference grammars. The agent is capable of retrieving relevant grammar sections, analyzing interlinear glossed text (IGT), and iteratively reasoning over typological hypotheses using a ReAct-style workflow. We evaluate the system on TYPOLOGICAL FEATURE CODING against expert annotations and TYPOLOGICAL HYPOTHESIS TESTING with typological universals using 25 open-source reference grammars. Operating under different information constraints in TYPOLOGICAL FEATURE CODING, the agent can synthesize information from reference grammar prose but still faces challenges with only IGTs in the target language. In TYPOLOGICAL HYPOTHESIS TESTING, the agent can synthesize crosslinguistic evidence and identify both supporting cases and counterexamples. These findings suggest that LLM agents can support scalable and inspectable typological analysis, while still requiring expert validation.

1 Introduction

Typological databases require extensive expert comparison, motivating an agent that retrieves evidence from reference grammars for feature coding and crosslinguistic hypothesis testing. The paper positions this work against gaps in scalable typological analysis and agentic linguistic research.

1 Introduction

  • Creating typological databases requires years of work by hundreds of trained linguists, limiting scalability.Such databases decompose grammar into individual features assigned values for each language.Source: §1 · p. 1
  • AUTOTYPOLOGIST is introduced as an LLM agent that actively queries reference grammars to summarize and compare grammatical structures across typologically diverse languages.The paper describes this as the first LLM agent for evidence-grounded typological analysis.Source: §1 · p. 1
  • The paper formalizes feature coding against expert annotations and hypothesis testing by synthesizing evidence across languages.It also reports high agreement with expert codings when grammar prose is available, and identifies supporting evidence and counterexamples in hypothesis testing.Source: §1 · p. 1
  • The released evaluation dataset contains structured grammar chunks, extracted IGT examples, and metadata from 25 open-source grammars across 14 language families.The materials are intended to evaluate evidence-grounded typological reasoning.Source: §1 · p. 1

2.2 LLMs for meta-linguistic reasoning

  • The approach uses abstract, language-agnostic metalinguistic reasoning rather than language-specific models.The paper frames this as research across languages and grammatical domains.Source: §1 · p. 2, §2.2 · p. 3

2.1 LLM agents for auto-research

  • The work addresses a gap in agentic linguistic research by developing a general LLM agent for automated analysis across low-resource and unseen languages.Prior approaches to grammatical-rule induction from glossed text often centered on specific aspects of grammar.Source: §2.1 · p. 2, §2.2 · pp. 2–3

3 Tasks

The paper formulates two complementary typology tasks and prepares an indexed corpus of reference grammars and IGT examples to support agent analysis.

3 Tasks

  • For feature coding, the agent follows feature-specific procedures to produce a predicted label and an evidence-grounded justification from a reference grammar.For the definite-article example, the instructions distinguish presence, absence in a comprehensive grammar, and insufficient analysis.Source: §3 · p. 3
  • Feature coding assigns a value to an individual language, while hypothesis testing synthesizes language-level judgments to evaluate crosslinguistic generalizations.For hypothesis testing, the agent distinguishes supporting, counter-, irrelevant, and insufficient evidence before reaching a crosslinguistic conclusion.Source: §3 · p. 3

4.1 Reference grammars

  • The grammar corpus contains 25 permissively licensed, open-source reference grammars, mostly published books and dissertations, representing 14 language families.The grammars were selected from the Grammar Library and are included in Grambank.Source: §4.1 · p. 3

4.2 Preprocessing

  • Preprocessing converts grammars to text and extracts IGT examples, which pair a language sentence with morpheme segmentation, grammatical glosses, and a free translation.Digitized grammars and extracted examples were manually reviewed by trained linguists for extraction errors.Source: §4.2 · pp. 3–4

4.3 Indexing

  • To support retrieval under context-length constraints, grammars are divided into structured chunks typically capped at 1,000 words, with short generated summaries and metadata.Each extracted IGT example is stored as a structured entry with metadata.Source: §4.3 · p. 4

5 AutoTypologist Framework

AUTOTYPOLOGIST plans evidence gathering, retrieves grammar prose and IGT data, and iteratively updates evidence-grounded analyses. Its tools model distinct operations in a typologist’s workflow and keep source support inspectable.

5.1 Agent workflow

  • The agent plans each investigation, retrieves relevant grammar sections in full, and runs planned IGT operations before beginning its iterative reasoning loop.Planning draws on grammar overviews and aggregate IGT statistics; retrieval extracts explicit author claims and runs tag-frequency, construction-search, and absence-check operations.Source: §5.1 · p. 4
  • In a ReAct-style loop of up to 10 interaction rounds per language analysis, the agent identifies missing or uncertain evidence and can retrieve more grammar chunks or conclude.Collected observations are organized in an evidence graph that tracks claims, sources, confidence, supporting and counter-evidence, uncertainty, and contradictions.Source: §5.1 · p. 4
  • The workflow produces a predicted feature value or typological conclusion with justification, then an auditor can uphold, weaken, or overturn the conclusion and revise its confidence.The auditor uses the same LLM with a separate prompt.Source: §5.1 · pp. 4–5

5.2 Tools

  • The toolset separates grammar-reading tools from IGT-analysis tools and IGT-only exploratory tools, matching their availability to the information sources provided.Grammar-reading tools check descriptive sections and author claims; IGT tools analyze aggregate and example-level evidence, with exploratory tools supporting induction when only IGT is available.Source: §5.2 · p. 5
  • Inference traces cite unique grammar-chunk and IGT identifiers so users can inspect the sources behind agent decisions.The tools are designed to constrain actions to evidential operations a human typologist might perform.Source: §5.2 · p. 5

6 Experiments

The experiments compare typological coding and hypothesis testing across different evidence conditions, using reference grammars and IGTs. Evaluation uses expert coding agreement for feature coding and evidence profiles for hypothesis testing.

6 Experiments

  • Grammar, IGT-only, and Grammar+IGT conditions test how access to grammatical prose and glossed examples affects both tasks.IGT-only requires inferring grammatical features without author explanations; Grammar+IGT combines descriptions with IGT evidence.Source: §6 · p. 5
  • Feature coding covers 25 Grambank languages, while hypothesis-testing runs sample 10 languages and 10 general universals plus adversarial variants.The sampled languages were chosen to maximize language-family coverage; three models were evaluated over three runs per condition.Source: §6 · pp. 5–6
  • Zero-shot prediction and a Grambank majority-vote baseline provide feature-coding comparisons, with the latter benefiting from imbalanced label frequencies.Beating majority vote is intended to indicate that the model captures grammatical information beyond the majority-class prior.Source: §6 · p. 6

6.1 Metrics

  • Feature coding is evaluated against Grambank annotations using Weighted F1 and Macro F1, which respectively account for label frequency and weight labels equally.Most features use labels 0, 1, and ?, while six also allow values such as 2 or 3.Source: §6.1 · p. 6
  • Hypothesis testing reports average language counts in four evidence categories rather than conventional accuracy, precision, or recall.The evidence counts inform whether a hypothesis is confirmed, refuted, or inconclusive.Source: §6.1 · p. 6

7 Main Findings

Reference-grammar prose improves feature coding over zero-shot prediction, while IGT-only coding remains harder. In hypothesis testing, the agent synthesizes crosslinguistic evidence, including evidence against adversarial universals, but model behaviors differ.

7.1 Typological feature coding

  • Grammar access yields the strongest feature-coding gains over zero-shot prediction, while every model still beats zero-shot with IGT alone.IGT-only remains considerably more challenging; increasing reasoning iterations can consistently improve Macro F1.Source: §7.1 · pp. 6–7, Table 2 · p. 7
  • Weighted F1 is generally stronger in Nominal Categories and more challenging in Complex Sentences and Verbal Categories.Domain-level fluctuations align with majority-vote performance, suggesting prior background knowledge may play a role.Source: Figure 2 · p. 7, §7.1 · p. 7
  • Grammar+IGT does not significantly improve over Grammar for any evaluated model, and adding examples brings marginal gains or slight degradation for most models.The authors suggest structured grammatical prose may be more useful than bottom-up induction from IGT, while noting retrieval noise or integration complexity as possible explanations.Source: §7.1 · p. 7

7.2 Typological hypotheses testing

  • AUTOTYPOLOGIST can falsify deliberately false adversarial hypotheses using evidence from reference grammars.The reported evidence profiles count languages assigned to For, Against, N/A, and Insuf. categories across standard and adversarial universals.Source: Table 3 · p. 8, §7.2 · p. 8

7.1 Typological feature coding

  • More token usage is associated with lower performance, partly because smaller models use more tokens and IGT-only costs less than the other conditions.The figure cautions that token usage does not necessarily reflect performance.Source: §7.1 · pp. 7–8, Figure 3 · p. 8

7.2 Typological hypotheses testing

  • Hypothesis confirmation or refutation is less accurate with IGT alone, especially for rare or unseen languages, than with structured English grammatical descriptions.GPT5.4-mini more often reports insufficient evidence, whereas Gemma4-31B and Qwen3.6-27B are more willing to draw conclusions; the authors speculate this may relate to post-training.Source: §7.2 · p. 8

8 Error Analysis

The error analysis examines prediction–gold mismatches across models and information settings, identifying errors in assigning grammatical forms to formal categories.

8 Error Analysis

  • When the gold label is ?, incorrect predictions often force a categorical label rather than preserving uncertainty.Other mismatches include treating an absent feature as present or missing evidence for a present feature.Source: §8 · p. 8
  • The analysis manually adjudicates 180 sampled mismatches across six languages and three information settings against the original grammar texts.It pools errors from Qwen3.6-27B, Gemma4-31B, and GPT5.4-mini, sampling 10 instances per language and setting combination.Source: §8 · pp. 8–9
  • Exponence-type misclassification occurs when the agent retrieves a form but assigns it to the wrong formal category, producing false-positive feature presence.In Papuan Malay, it treats suda, masi, and blum as aspectual auxiliaries, while the grammar analyzes them as adverbs and the retrieved evidence does not establish the Grambank auxiliary criterion.Source: §8 · p. 9

2. IGT role or construction misinterpretation.

The error analysis identifies failures in interpreting grammatical roles, constructions, and feature-coding requirements, as well as in retrieval and evidence integration. The conclusion stresses that expert validation remains necessary and that traceable tools and steps matter for typological analysis.

2. IGT role or construction misinterpretation.

  • In Chácobo, the agent misreads ergative =wa on the A noun phrase as verbal indexing, confusing nominal case marking with argument indexing.This exemplifies treating a gloss tag or morpheme as evidence for the wrong grammatical role, host, or construction.Source: §2 · p. 9
  • In Ik, past tense is missed because it is expressed through aspectual and enclitic combinations rather than a dedicated marker.The error reflects an over-restrictive interpretation that adds requirements such as a dedicated, obligatory, or single-morpheme exponent.Source: §2 · p. 9
  • In Japhug, terminology mismatch sends retrieval toward nominal derivation and Object–Verb compounds instead of the relevant object-participle construction.The query asks about “object nouns derived from verbs,” while the grammar discusses object participles.Source: §2 · p. 9
  • In Chakali, a bound plural suffix is treated as a free plural element, although the grammar analyzes it as noun-class suffixation.This is an evidence-integration failure involving an example from a non-target construction.Source: §2 · p. 9
  • For Greenberg Universal U30, the model incorrectly treats Ik as a counterexample, reflecting a narrow reading of the consequent and suboptimal section selection.The selected Ik sections focus on phonology and morphophonology rather than verbal morphology or tense, aspect, and modality; the authors say human expertise is still needed to validate outputs.Source: §2 · p. 9

9 Conclusions

  • AUTOTYPOLOGIST performs both feature coding and hypothesis testing, with traceable evidence highlighted as important to structured typological research.The agent automates querying evidence across reference grammars, which the authors say can let linguists focus on deeper analysis.Source: §9 · p. 9

Limitations

The authors identify limits in the reference data, validation requirements, and the modalities and time-slices represented by the system. Ethical considerations also emphasize that the work is intended to support, not replace, linguists and language communities.

Limitations

  • Typological databases are debated, may contain inaccurate feature codings, and reduce linguistic diversity to discrete codes for cross-linguistic comparison.The authors note that uniform discrete codes are nevertheless needed for comparison.Source: Limitations · pp. 9–10
  • Generated and tested hypotheses still require extensive human validation for scientific value and correctness, and many universal hypotheses need more linguistic data.The agent cannot replace field linguists gathering language data through fieldwork.Source: Limitations · p. 10
  • The study covers textual language only because of available-data and LLM constraints, leaving speech and visual signing modalities for further research.The authors describe those modalities as essential components of language.Source: Limitations · p. 10
  • AUTOTYPOLOGIST relies on reference grammars, which are static snapshots of living languages, and cannot replace field linguists interacting with communities.Source: Limitations · p. 10

Ethical considerations

  • The authors frame the system as a tool to facilitate linguistic research and recognize fieldwork's role in both data collection and community relationships.They selected grammars with permissive licenses, cited them for research use, and implemented an opt-out mechanism for authors and communities.Source: Ethical considerations · p. 10

A Reference Grammars Details

The appendices point readers to the study's grammar and universal inventories, supplementary prediction materials, and system settings. They also describe an unevaluated automatic-summarization mode and examples of the structured materials it uses.

A Reference Grammars Details

B Greenberg Universals

C Automatic Summarization

  • Automatic summarization generates candidate feature questions from grammar contents and aggregate IGT statistics, then investigates and aggregates evidence-grounded feature answers.Its output is a structured JSON profile; the study did not systematically evaluate this mode and leaves full evaluation to future work.Source: C Automatic Summarization · pp. 12–13

D Label Prediction Details

E Majority Voting Labels

F Grammar Chunk and IGT entry example

G Hyperparameters

H Grambank Domain Types and Distributions

Supplementary materials describe domain distributions, sample agent traces, prompts, statistical analysis, and an iteration-budget experiment. In the tested IGT-only setting, Macro F1 rises with more reasoning iterations while weighted F1 remains relatively stable.

H Grambank Domain Types and Distributions

I Sample TYPOLOGICAL FEATURE CODINGAgent Traces

J Sample TYPOLOGICAL HYPOTHESIS TESTINGAgent Traces

K Agent Prompts

  • Figures 8–11 illustrate prompt templates used at different stages of the system workflow.Source: K Agent Prompts · p. 13

L Additional Statistical Analysis

  • The additional analysis uses 10,000 language-clustered bootstrap resamples and paired information-condition comparisons, with Holm correction across three pairwise comparisons per model and metric.Results are reported in Table 16.Source: L Additional Statistical Analysis · p. 13

M Iteration Influence on IGT-only Setting

  • Macro F1 reaches its highest value at 40 reasoning iterations in the IGT-only Qwen2.5-7B experiment, while weighted F1 remains relatively stable.The time-limited experiment covered six languages: Aguaruna, Bargam, Chakali, Chácobo, Hup, and Ik.Source: M Iteration Influence on IGT-only… · pp. 13–14

N Ablation of the Auditing Stage

The auditing-stage ablation on Aguaruna finds consistent gains across three models, with the size of the improvement varying by model. It evaluates the ablation under the Grammar-only setting and reports weighted and macro F1.

N Ablation of the Auditing Stage

Related papers

Loading 2609.07791v1…