Source-linked AI summary
Causal Interventions Reveal Typologically Organized Syntactic Mechanisms in Multilingual Language Models
Sasha Boguraev, Toshiki Nakai, Kyle Mahowald, Julius Steuer
TL;DR
The paper addresses the difficulty of testing whether similar syntactic structures across languages use shared processing mechanisms. It uses causal interventions in multilingual language models to transfer syntactic mechanisms across languages, finding consistent but typologically structured reuse. The results motivate hypotheses about multilingual organization and possible tests in humans.
Problem
Whether similar syntactic structures across languages recruit shared internal processing mechanisms remains difficult to test because human mechanisms are not finely manipulable.
Method
The study uses Distributed Alignment Search to localize causally relevant syntactic subspaces in one language and test their effects across languages, models, and constructions.
Results
Across fifteen languages, four models, and three constructions, syntactic mechanisms consistently transfer cross-lingually, with transfer graded by typological similarity and organized around typological hubs.
Takeaways & Limitations
The findings characterize multilingual syntactic competence as abstractions repurposed across languages and provide hypotheses about cross-linguistic syntax and multilingual processing that could be tested in humans.
Takeaways & Limitations
The results are not definitive evidence about human multilingual processing because the relationship between neural-model and human linguistic processing remains debated.
Abstract
from arXiv · showhide
Linguistic theory has long recognized cross-linguistic syntactic regularities, leading to claims that these similar structures are processed by similar mechanisms. However, this hypothesis has been difficult to test empirically due to our lack of fine-grained, manipulable access of human processing mechanisms. In this work, we take advantage of techniques from mechanistic interpretability to study such a question in multilingual LMs. We first isolate language-internal mechanisms before attempting to transfer them cross-lingually. Across four models and three well-studied constructions (subject--verb number agreement, anaphoric pronoun gender agreement, and filler--gap object extraction) we find consistent cross-lingual mechanism transfer. We further find transfer to be graded, with more transfer between more typologically similar languages. We believe our work provides novel hypotheses about cross-linguistic syntactic structures and multilingual processing, and more broadly shows how the study of language models can help inform linguistic theory.
1 Introduction
The paper asks whether cross-linguistic syntactic similarities reflect shared processing mechanisms, a question difficult to test directly in humans. Using mechanistic interpretability, it identifies syntactic mechanisms within one language and tests whether they transfer across languages.
- Human evidence for shared cross-linguistic processing mechanisms is difficult to obtain because internal mechanisms cannot be manipulated at fine granularity.
- Multilingual language models offer syntactic competence together with privileged access to their internal computations.
- Causal Abstraction identifies internal structures that control syntactic behavior, enabling mechanisms found in one language to be tested in another.
- Across fifteen languages, four language models, and three phenomena, models recruit shared, repurposed mechanisms rather than language-specific ones.The phenomena are subject–verb number agreement, anaphoric pronoun gender agreement, and filler–gap object extraction.
- The results characterize multilingual syntactic competence as abstractions repurposed across languages and motivate hypotheses potentially testable in humans.
2 Cross-Lingual LM Interpretability
Prior work used probing and causal or sparse-decomposition methods to investigate whether multilingual models share linguistic representations. This paper instead uses Distributed Alignment Search to localize abstract features and test whether causally implicated syntactic mechanisms transfer across languages.
- Probing studies suggested that universal-dependency relations and semantic features can be localized to language-agnostic subspaces or transferred across languages.
- Because probing can be over-expressive, later causal methods manipulated internal states to test behavioral faithfulness.
- Earlier causal variants found both language-neutral and language-specific representations but could not elucidate the computational features being used.
- Sparse Auto-Encoders and Cross-Layer Transcoders exposed shared feature-circuits, but their interpreted features can be unstable across training runs.
- Distributed Alignment Search localizes abstract features in multilingual models and supports precise tests of cross-lingual transfer for causally implicated syntactic mechanisms.
3 Methods
The study evaluates three syntactic constructions across a typologically diverse multilingual dataset and four language models. It learns causal interventions with Distributed Alignment Search and evaluates their efficacy using ODDS.
- 3.1 Data: The dataset covers three constructions across fifteen languages, with typological constraints preventing every language from appearing in every construction.The constructions are number agreement, gender agreement, and object extraction from an embedded wh-question.
- 3.1 Data: Behavioral evaluation found models at or above 90% accuracy on 90% of templates, with below-80% accuracy occurring only once.The gender-agreement stimuli included violable examples, which did not affect the reported behavioral findings.
- 3.2 Controls: The experiments use RANDOM LABELS, OTHER PHENOMENA, and OOD LABELS controls to assess intervention specificity and transfer across syntactically distinct or novel labels.OOD LABELS is not reported for gender agreement because robust alternate labels are unavailable cross-lingually.
- 3.1 Data: German and Dutch filler–gap templates exclude the VP position because their clause-final embedded verbs cannot be perfectly aligned with other languages.
- 3.2 Models: The four models include two expressly multilingual models and two generic-web models with multilingual capabilities, allowing architecture and training factors to be examined.
- 3.3 Distributed Alignment Search: Distributed Alignment Search learns low-rank subspaces where high-level causal variables can be localized and manipulated through interventions.The study learns one-dimensional subspaces in model residual streams.
- 3.3 Distributed Alignment Search: The learned rotation matrix is optimized by gradient descent to minimize the language model's cross-entropy loss under intervention.
- 3.4 Training and Evaluation: ODDS measures the source-label probability increase relative to the base-label probability decrease, with higher values indicating greater causal efficacy.Aggregated results use MAX ODDS, the maximum ODDS across layers.
4 Experiment 1: Do LMs Share Mechanisms Cross-Lingually?
The first experiment tests whether a subspace causally implicated in a syntactic phenomenon in one language also controls that phenomenon in other languages. Cross-lingual interventions consistently outperform controls, while within-family advantages vary by phenomenon and position.
- Setup: The experiment trains interventions on a syntactic phenomenon in Language A and tests whether the same subspace controls that phenomenon in Language B.
- Setup: Evaluation compares within-language, cross-lingual, within-family, and control conditions using ODDS across layers and template positions.
- Setup: Normalized MAX ODDS measures how much an intervention generalizes to other languages relative to its within-language baseline performance.
- Results: Cross-lingual generalization ODDS is consistently higher than controls across layers, positions, models, and phenomena, with statistically significant transfer differences.The figure reports pairwise t-tests on MAX ODDS with p < 0.05 for cross-language versus control conditions.
- Results: Within-family transfer is not uniformly greater than all-language transfer: differences vary for number and gender agreement, and are absent for filler–gap extraction.The absence for filler–gap extraction may reflect limited language diversity or normalization effects.
- Discussion: Overall, the findings support shared, causally efficacious syntactic subspaces and abstract cross-lingual mechanisms in multilingual language models.
- Discussion: Intervention generalization across lexical items provides evidence against the alternative hypothesis that the learned subspaces are merely translation-equivalent or lexically specific.
5 Experiment 2: What Drives Representational Reuse?
Representational reuse is consistently associated with typological similarity rather than language family alone, while lower-level similarity measures covary and tokenizer overlap is not sufficient to explain transfer.
- Typological-distance effects become stronger in larger models for Number Agreement and Gender Agreement, although model size has no significant main effect at any position.The interaction is reported as weak, while the main-effect hypothesis for smaller models is not supported.
- Language-family clusters appear in PCA space, but low silhouette scores indicate that family alone does not explain the clustering.Silhouette scores measure cluster purity, with higher values indicating tighter within-family clusters.
- Transfer is stronger between typologically similar languages across phenomena and positions, independent of construction or model.The LMEM finds typological distance to be a strong, significant predictor of MAX ODDS.
- LANG2VEC is a coarse typological representation, and the data cannot support regressions on individual lower-level features.The considered factors include tokenizer overlap, syntactic-feature overlap, and cognate counts; disjoint-script transfer shows tokenizer overlap alone is insufficient.
- All pairwise linguistic-similarity correlations are significantly positive and generally strong, while template overlap correlates more weakly.The weaker template-overlap correlations are attributed to hand-designed templates and low token variance at fixed-inventory positions.
6 Experiment 3: An Investigation Into the Effects of Training Data
Training-data magnitude has little explanatory power for multilingual representational transfer in mGPT; transfer instead appears organized around typological hubs, though the correlations are underpowered and suggestive.
- Training-data magnitude provides little evidence for the hypothesized source-and-sink pattern.Training data correlates positively with out-degree, but weakly in two phenomena; only filler–gap shows a negative in-degree correlation, and it is negligible.
- Languages with high in-degree also tend to have high out-degree, suggesting transfer is organized by hubs rather than separate sources and sinks.In- and out-degree are computed from transfer-matrix edge sweeps and summarized as AUC values.
- Hub-ness is highly correlated with typological centrality but generally not correlated with training-data amount.Typological centrality is the average pairwise typological distance across languages.
- The hub correlations are computed over few items (n = 7-13), so the evidence is underpowered and suggestive rather than conclusive.
7 Conclusion
The paper addresses the difficulty of testing shared cross-linguistic processing mechanisms by analyzing multilingual language models. It finds cross-lingual reuse organized by typological similarity and proposes human psycholinguistic tests of these hypotheses.
- The paper investigates cross-linguistic overlap in internal processing, which has been difficult to probe in humans.
- Across languages and syntactic phenomena, language models learn and repurpose abstract mechanisms cross-lingually, with transfer modulated by typological similarity.
- The findings motivate testing whether languages with greater LM transfer also show stronger cross-linguistic syntactic priming in bilingual speakers.
- The study argues that language-model analysis can inform hypotheses about cross-linguistic syntactic structures and multilingual processing.
Limitations
The study’s conclusions are bounded by indirect access to complete mechanisms, limited language coverage, and templatically generated rather than naturalistic sentences.
- The findings should not be treated as definitive evidence about multilingual linguistic processing in humans because model–human processing relationships remain debated.The authors characterize the results as supporting evidence about analyses general-purpose learners converge on.
- The method accesses only part of the underlying mechanism through experimental stimuli and cannot explicitly reveal features such as Arabic dual number.Comparing transfer with and without a dual can reflect disruption, but does not explicitly identify dual number in the mechanism.
- The language sample covers a small, widely spoken subset of human languages, so greater typological diversity could enrich the conclusions.
- The experiments rely on templatically generated minimal pairs, which can differ meaningfully from naturalistic sentences.Extending the work to natural sentences is identified as a meaningful next step.
C Behavioral Results
Before analyzing internal mechanisms, the authors verify behavioral competence on three syntactic constructions. Across 128 templates, the models show strong performance overall, with multilingual models generally performing best.
- The evaluation tests whether models assign higher surprisal to each label in the correct context than in the incorrect context.ACCURACY is a binary measure computed from these surprisal comparisons over minimal pairs.
- Figure 5 averages performance over 160 randomly sampled minimal pairs for each construction and model.
- 90% (115/128) of templates reached at least 90% accuracy, while below-80% accuracy occurred only once, for Llama-3.2-1b on Bulgarian number agreement.
- Expressly multilingual models were generally the most performant, while Llama-3.2-3b did not lag far behind and Llama-3.2-1b remained strong despite ranking lowest.
- The study reports normalized MAX ODDS charts for five experimental groups and Holm-Bonferroni-corrected p-values for main and out-of-distribution comparisons.
E Out-of-Distribution Controls
Out-of-distribution controls show that the intervention patterns broadly generalize beyond the lexical items used in training, especially for subject–verb agreement and filler–gap extraction. However, filler–gap significance is limited at most positions, and normalized comparisons can be deflated by model-specific baselines.
- Control comparisons: Subject–verb agreement generalization is significantly higher than both the across-phenomenon and random-label controls.
- Model-specific pattern: Llama-3.2-1b shows weaker significance partly because fewer samples reduce statistical power and its higher within-language transfer deflates normalized MAX ODDS.
- Family restriction: Cross-language and within-family transfer differ little, with significant differences only at selected NP1 and PP positions.The reported significant positions are NP1 for tiny-aya-base and PP for tiny-aya-base and mGPT.
- Generalization: OOD-label interventions broadly reproduce the main results, supporting generalization beyond the training labels.The authors interpret this as evidence that interventions reflect general syntactic mechanisms rather than lexical overfitting.
G Metric Coverage
Typological-distance analysis is constrained by incomplete coverage across the languages and metrics used. The reported analyses also rely on mixed-effects regressions and compute-intensive intervention experiments, while the controls and figures summarize transfer comparisons.
- Coverage limitations: Grambank feature-overlap data are unavailable for German, Bulgarian, Spanish, and Romanian.The dataset prioritizes typological breadth and diversity rather than complete language coverage.
- Coverage limitations: IECOR cognate-proportion data cover Indo-European languages, excluding Galician, Chinese, Japanese, and Korean.
- Comparisons: Holm-Bonferroni-corrected comparisons test cross-lingual transfer against random-label and other-phenomenon controls, as well as within-family transfer.