PaperlayerView exact revision on arXiv

Source-linked AI summary

Shared Circuits for Shared Grammar: Tracing Subject-Verb Agreement Across Languages

Isabella Gidi, Antonio Almudévar, Core Francisco Park, Naomi Saphra, Ricard Marxer

arXiv:2608.18545v1cs.CL

TL;DR

It remains unclear when multilingual models reuse shared internal circuitry across languages and when they rely on language-specific mechanisms. This paper analyzes subject-verb agreement across 29 languages and five model families, finding that agreement circuitry is more shared for overt inflection, especially when recovering the inflectional contrast itself.

  • Problem

    It remains unclear what determines whether multilingual models reuse common circuitry or recruit language-specific mechanisms across languages.

  • Method

    The study uses attention-output patching and cross-lingual comparisons of attention-head signatures to identify and compare circuitry involved in subject-verb agreement.

  • Results

    Across 29 languages and five model families, overt-agreement languages show more similar agreement circuitry, with sharing strongest for strict recovery of the inflectional contrast.

  • Takeaways & Limitations

    Cross-lingual agreement overlap reflects partially shared localization and functional roles shaped by whether the same grammatical computation is overtly realized.

  • Takeaways & Limitations

    The analysis covers attention-based routing rather than the full computational graph and uses structured prompts that may not fully generalize to ordinary language contexts.

Abstract

from arXiv · show

Multilingual large language models often generalize across languages, and prior work suggests that their internal mechanisms can overlap cross-lingually. It remains unclear, however, when such sharing emerges and whether it varies with the overt realization of the same grammatical operation. We investigate this question for present-tense subject-verb agreement, a morphosyntactic process that varies substantially across languages and is only weakly expressed in English. Using activation patching and attention analysis across 29 languages and five open-source model families, we identify the attention heads causally implicated in agreement and compare these head-level signatures across languages. We find that languages with overt person/number inflection exhibit more similar agreement circuitry than non-conjugating languages, with the strongest sharing appearing when the analysis isolates recovery of the inflectional contrast itself. English provides an informative bridge case, becoming more similar to conjugating languages precisely in contexts where overt agreement is required. Finally, many implicated heads display similar attention patterns across languages, suggesting that cross-lingual overlap reflects shared functional roles as well as shared localization. Together, these results indicate that multilingual LLMs reuse partially shared computational structure for morphosyntactic agreement rather than relying on fully separate language-specific solutions.

1 Introduction

The introduction asks when multilingual LLMs reuse shared circuitry versus language-specific mechanisms for morphosyntactic phenomena. Using subject-verb agreement, it argues that overt agreement languages share conjugation circuitry, especially for recovering inflectional contrasts.

  • Motivation: Multilingual LLMs generalize across languages, but the conditions governing shared versus language-specific internal computations remain unclear.This distinction determines what kinds of cross-lingual generalization should be expected inside the model.
  • Problem: Morphological and syntactic differences may not map straightforwardly onto a language-agnostic internal representation, particularly when a high-resource language lacks the relevant feature.The paper frames this as a mechanistic question about shared circuitry across typologically different languages.
  • Testbed: Subject-verb agreement provides a testbed because present-tense person and number are overtly marked in many languages, absent in Chinese, and only partially marked in English.The paper defines conjugation as changes in a verb’s surface form based on the subject’s grammatical person and number.
  • Main findings: Languages with overt subject-verb agreement show substantial overlap in the attention heads implicated in conjugation.This supports partial sharing of conjugation circuitry across languages.
  • Main findings: Similarity is highest under the minimal-pair metric, indicating that shared circuitry is strongest when analysis isolates recovery of the inflectional contrast itself.The paper distinguishes stricter inflectional-contrast recovery from broader recovery of the context-appropriate target form.

2 Related Work

Prior work finds partially shared multilingual structure across semantics, factual recall, safety, grammar, and morphosyntax. This study builds on causal agreement and cross-lingual circuit analyses while broadening language coverage and distinguishing two recovery targets.

  • Shared and language-specific multilingual mechanisms: Prior studies report partially shared cross-lingual organization, including overlapping grammatical subspaces, shared neuron sets, and agreement-related components.The broader literature also identifies shared structure in semantics, factual recall, and safety-related behavior.
  • From shared representations to shared circuitry: Cross-lingual structure has been investigated through probing, representational analyses, neuron and feature methods, and causal interventions.This work is closest to causal studies of agreement and cross-lingual circuit similarity.
  • From shared representations to shared circuitry: The study compares a broader, typologically varied language set spanning rich, partial, and no overt agreement in the tested contrasts.It also separates target-logit recovery of context-appropriate forms from minimal-pair recovery of the inflectional contrast itself.

3 Experimental Setup

The experiments compare agreement circuitry across five open-source model families and 29 languages using controlled zero-shot prompts and representative person/number contrasts. Tokenization, behavioral, and coverage filters restrict analyses to aligned, successful agreement cases with sufficient data.

  • Models evaluated: The study evaluates BLOOM, Gemma, Llama, Mistral, and Qwen as five open-source model families.The full model list and analyses of model size and instruction tuning appear in the appendices.
  • Inflection contrasts: The analysis covers four bidirectional contrasts: 1sg ↔2sg, 1sg ↔3sg, 1pl ↔3pl, and 3sg ↔3pl.English has an inflectional change on two contrasts: 1sg ↔3sg and 3sg ↔3pl.
  • Prompt template: Commonly structured, language-adapted zero-shot prompts preserve positional alignment while eliciting present-tense verb inflections.The template is “Conjugation of the verb [infinitive] in present tense: [pronoun] [conjugation]”.
  • Filtering: Tokenization filtering keeps minimal pairs differing only in the final token, while behavioral filtering removes examples where the model fails to produce the desired inflection.These filters isolate the target person or number contrast and focus analyses on successful agreement realizations.
  • Language-model coverage: The study retains 29 languages from an initial 34, requiring at least 50 prompts per language-model combination after filtering.The retained set includes 24 languages with overt agreement patterns.
  • Final dataset: 1,509.59 prompts survive on average across retained language-model combinations, ranging from 96 to 2,400, with 50 prompts evaluated per set for activation patching.The surviving prompt counts are reported after tokenization and behavioral filtering.

4 Analytic Methods

The analysis combines activation-output patching with attention analysis to identify attention heads causally involved in subject-verb agreement and characterize their functional behavior. It uses complementary recovery metrics and compares head-level circuitry and attention-role profiles across languages within models.

  • Activation patching: Activation-output patching replaces a corrupted run’s attention-head output with the corresponding clean activation to measure each head’s causal contribution to predicting the target verb form.Clean and corrupted minimal-pair prompts differ in subject pronoun and target inflection, isolating the contrast at the final token.
  • Recovery metrics: Target-logit recovery applies to both conjugating and nonconjugating languages, measuring restored preference for the clean target form without requiring distinct final inflection tokens.In nonconjugating cases, the metric reflects context-dependent target prediction rather than necessarily encoding overt inflectional contrast.
  • Recovery metrics: Minimal-pair recovery is restricted to examples with different clean and corrupted final inflection tokens and measures restoration of the context-appropriate inflectional contrast.Larger absolute recovery values indicate stronger effects, with positive values promoting and negative values inhibiting the clean target token.
  • Cross-lingual comparison: Cross-lingual circuitry similarity is estimated by Pearson-correlating flattened layer–head patching heatmaps within the same model, conjugation pair, and patching direction.Attention-head coordinates and activations are not compared directly across architectures or checkpoints; cross-model summaries aggregate within-model similarities.
  • Attention analysis: Attention analysis ranks heads by absolute patching score, retains the top k = 20, and compares their attention-role vectors across languages using mass directed to the subject pronoun, infinitive span, and remaining tokens.Cross-lingual comparisons use heads appearing among the top 20 in both languages under matched model, conjugation-pair, direction, and metric conditions.

5 Results

Across languages, multilingual models reuse agreement circuitry, with the strongest and most consistent sharing among conjugating languages when recovery isolates the inflectional contrast. Broader target-form recovery remains organized by overt agreement structure, while implicated heads exhibit similar routing roles across languages.

  • 5.1 Structural localization: Conjugating languages show broadly positive, distributed head-level similarity under the minimal-pair metric.The effect persists across individual conjugation contrasts and patching directions.
  • 5.1 Structural localization: Similarity among conjugating languages is higher and less variable under minimal-pair than target-logit recovery, peaking when models restore the strict inflectional contrast.Broader target-form prediction introduces more language-specific variance.
  • 5.2 Language type and context: Under target-logit recovery, conjugating–conjugating comparisons are much more similar than comparisons crossing the conjugating/non-conjugating boundary.This separates languages by whether they overtly realize the morphosyntactic feature, while similarity among non-conjugating languages remains relatively low.
  • 5.2 Language type and context: English shifts upward from non-inflecting toward 3sg-inflecting contexts, becoming more similar to conjugating languages when agreement must be realized.This supports contextual grammatical demand, rather than static language identity, as the driver of sharing.
  • 5.3 Mechanistic behavior: 0.928 mean similarity under minimal-pair and 0.926 under target-logit show exceptionally high cross-lingual attention-role consistency among shared heads.In attention routing, non-conjugating target-logit heads direct 0.177 mass to pronouns versus 0.288 and 0.291 for conjugating minimal-pair and target-logit heads.

6 Discussion · A Task and Experimental Details

Across 29 languages and five model families, multilingual models reuse partially shared agreement circuitry, with overlap shaped by overt grammatical realization rather than language identity alone. The findings also indicate that shared localization and functional roles contribute to cross-lingual agreement circuitry, while attention-based analysis and structured prompts impose important limitations.

  • 6 Discussion: Across 29 languages and five model families, agreement circuitry is partially shared but neither uniform nor fully language-agnostic.
  • 6 Discussion: Languages with overt agreement morphology exhibit more similar head-level agreement signatures than languages without comparable overt realization.
  • 6 Discussion: Cross-lingual overlap reflects both shared localization and partially shared functional roles in agreement processing.
  • 6 Discussion: Sharing strengthens when languages must overtly realize the same grammatical computation, rather than being determined by language identity alone.
  • 6 Discussion: Sharing is strongest when analysis isolates recovery of the inflectional contrast and weaker when it targets broader contextual recovery.
  • 6 Discussion: Shared multilingual computation depends partly on whether languages realize the same structural feature comparably, affecting expected transfer of interpretability interventions and alignment methods.
  • 6 Discussion: The analysis characterizes attention-based routing rather than the full computational graph, leaving possible downstream roles for position-wise MLPs.Attention heads enable inter-token transfer of subject information, while MLPs may read, transform, or amplify that signal.
  • 6 Discussion: Structured conjugation prompts, rather than naturalistic sentences, constrain interpretation of the identified attention-head circuitry.

A.1 Conjugation task and agreement patterns

The study tests present-tense subject–verb agreement by contrasting grammatical person and number, comparing whether subject changes require verb-form changes across languages. It distinguishes languages with overt agreement marking from those where the same contrasts are not morphologically realized.

  • Conjugation task: The task isolates whether changing grammatical person or number requires a change in the verb’s surface form.The analysis uses controlled contrasts to compare how this computation is implemented across languages.
  • Agreement patterns: The cross-lingual analyses group languages by whether the tested contrasts involve overt agreement marking.This grouping distinguishes morphologically realized agreement from contrasts that are not overtly expressed.
  • Agreement patterns: English inflects verbs only in the third-person singular, whereas Spanish has rich person/number inflection and Swedish lacks person inflection in the tested present-tense contexts.These examples represent three typologically distinct agreement patterns.

A.2 Models used in the main experiments … A.5 Languages Tested

The main experiments use diverse open-source multilingual model families with a fixed experimental pipeline, standardized present-tense prompts, and tokenization filtering that localizes agreement contrasts to the predicted token. The tested multilingual prompt space is defined by language-specific templates and subject-pronoun inventories across person and number contrasts.

  • A.2 Models used in the main experiments: The main analyses evaluate diverse open-source multilingual model families while holding the experimental pipeline fixed across models.The model set varies in architecture, training corpus, and scale.
  • A.3 Prompt templates and pronoun inventories: Each language uses a fixed zero-shot prompt template designed to elicit the target present-tense form.The templates specify the verb, subject pronoun, and conjugated form in each language.
  • A.3 Prompt templates and pronoun inventories: Subject pronouns are stated explicitly in every condition, making the agreement cue overtly available for cross-lingual comparison.This reduces the likelihood that model differences reflect whether the subject cue appeared in the prompt.
  • A.5 Languages Tested: The tested languages are represented through language-specific prompt templates and six pronoun categories spanning 1sg, 2sg, 3sg, 1pl, 2pl, and 3pl.The materials define the multilingual prompt space used to construct clean/corrupted minimal pairs.
  • A.4 Tokenization Filtering: The tokenization filter retains only clean/corrupted pairs whose conjugated forms differ in the final token and otherwise remain aligned.This model-specific filtering localizes the manipulated agreement contrast to the predicted token.
  • A.4 Tokenization Filtering: Discarded examples introduce broader tokenization differences that could confound causal interpretation, whereas retained examples preserve the shared prefix.Figure 8 illustrates representative retained and discarded cases.

A.5.1 Languages In Dataset … B.3 Model-specific cross-group comparisons

The study uses a broadly typological multilingual inventory, filters it into agreement-based language groups with uneven model coverage, and finds robust cross-lingual sharing among conjugating languages. Model-specific analyses show that this sharing persists across families, conjugation contrasts, patching directions, and cross-group comparisons.

  • A.5.1 Languages In Dataset: The initial inventory combines UniMorph and a Chinese verb dataset to cover both overtly conjugating and non-conjugating languages.Not every language survives later filtering for every model.
  • A.5.2 Languages Used In Analysis: After tokenization and behavioral filtering, languages partition into overt-agreement, non-overt-agreement, and English middle-case groups.These groups define the cross-group comparisons.
  • A.5.3 Language-model coverage: Each language–model combination is retained when at least 50 prompts remain, yielding broad but uneven coverage across models.Gemma, Mistral, and Qwen retain especially wide coverage, while BLOOM and Llama show more gaps.
  • B.1 Model-specific and individual language-level cross-lingual similarity among conjugating languages.: Within conjugating languages, similarity matrices are broadly positive rather than concentrated in isolated language pairs.This pattern is especially dense for Gemma, Mistral, and Qwen, while BLOOM and Llama show more heterogeneity and missing cells.
  • B.2 Similarity by Conjugation Contrast and Patching Direction: Cross-lingual similarity remains positive across every tested conjugation contrast and patching direction.The result indicates that sharing is not restricted to one direction or agreement contrast.
  • B.3 Model-specific cross-group comparisons: Across every model family, conjugating–conjugating comparisons are most similar, followed by conjugating–non-conjugating and non-conjugating–non-conjugating comparisons.The ordering is consistent even though the gap size varies by family.

B.4 Additional analyses of English’s mixed agreement profile

Across all six models, English is more similar to conjugating-language circuitry when third-person singular inflection is required than when agreement is non-inflecting, with variation only in shift magnitude.

  • Model-level consistency: Across all six models, English is consistently more similar to conjugating-language circuitry in 3sg-inflecting than non-inflecting conditions.The direction of the effect is stable across models, while the size of the shift varies.
  • Model-level consistency: The largest positive shifts occur in Gemma-2-9B and BLOOM-7B1, with clear positive differences in Llama-3.2-3B and Qwen2-7B.Mistral-7B and Llama-2-7B show smaller but still positive shifts.
  • Bridge-case interpretation: English aligns more closely with conjugating languages precisely where it overtly realizes agreement, supporting its characterization as a bridge case.This interpretation is reinforced by the consistent positive direction across models.

C Robustness to Similarity Metric … D.2 Instruction tuning

Robustness analyses preserve the main cross-lingual similarity pattern, with Pearson and cosine yielding the clearest and most stable results. Auxiliary tests indicate that similarity is not explained by model scale or language-specific accuracy, and remains largely stable after instruction tuning.

  • C Robustness to Similarity Metric: Pearson and cosine similarity reproduce the main cross-lingual pattern most consistently across all-head and top-k schemes, whereas Spearman is noisier.Jaccard overlap on the top-20 heads captures some shared structure but compresses the distribution.
  • C Robustness to Similarity Metric: Pearson correlation over flattened absolute heatmaps is used in the main text because it preserves continuous patching signatures and clearest separates stronger from weaker overlap.The robustness comparison motivates this metric choice while supporting the substantive cross-lingual similarity claim.
  • D Auxiliary Analyses: Auxiliary experiments use a smaller typologically diverse language subset containing richly conjugating languages, English as a mixed case, and non-conjugating comparison languages.The reduced subset makes expanded model comparisons computationally tractable.
  • D.1 Model size and language accuracy: Neither model scale nor language-specific task performance alone sufficiently explains the cross-lingual similarity pattern.Figure 14 evaluates both possible explanations in the auxiliary analyses.
  • D.1 Model size and language accuracy: Cross-lingual similarity shows no single monotonic relationship with parameter count, varying across model families rather than consistently increasing with scale.Some families rise with scale, while others flatten or change only modestly.
  • D.1 Model size and language accuracy: Similar language-specific accuracy does not mechanically imply stronger cross-lingual circuit similarity, since languages and models with comparable accuracy can differ in overlap.This separates task performance from the recovered cross-lingual circuit structure.
  • D.2 Instruction tuning: Instruction tuning does not produce a consistent upward or downward shift in cross-lingual similarity across model families.Observed changes are modest and mixed in direction, suggesting that agreement-related circuit structure is already present in base models and remains relatively stable after post-training.

D.3 Analyses of pre-training progression

Cross-lingual similarity in agreement circuitry increases over BLOOM pretraining rather than being fully present at initialization. This upward trend holds across both recovery metrics, though minimal-pair trajectories are noisier and more contrast-specific.

  • Training progression: Similarity tends to increase over training across both recovery metrics, indicating that shared cross-lingual agreement circuitry emerges progressively during pretraining.The aggregate trajectory is upward, although individual language pairs vary in timing and shape.
  • Metric comparison: The minimal-pair signal is noisier and more contrast-specific than the target-logit signal.Language-pair trajectories may rise sharply and plateau or increase more gradually.
Loading 2608.18545v1…