Source-linked AI summary
Separating Syntax from Language: A Mechanistic Account of Translation in Multilingual LLMs
Mikhail Sonkin, Tanja Baeumel, Daniil Gurgurov, Josef van Genabith, Simon Ostermann
TL;DR
The paper addresses how multilingual LLMs transform representations between languages, especially when translation requires systematic word-order changes. Using controlled multilingual datasets, activation patching, and probing, it finds that target-side syntax is resolved before surface language and that syntax-sensitive attention heads are largely language-invariant.
Problem
How mLLMs transform grammatical structure when source and target languages differ systematically in word order remains unresolved.
Method
The study uses controlled multilingual datasets, activation patching, and representation probing across models and typologically diverse constructions.
Results
Translation follows a staged S → L → C process: target-side word order is resolved before surface language and lexical content, with syntax-sensitive attention heads largely invariant to language identity.
Takeaways & Limitations
Syntactic structure is an independent, partially language-agnostic stage implemented by distinct components within mLLMs.
Takeaways & Limitations
The studied noun-phrase, subject-verb-object, and modal-verb constructions cover only a small subset of grammatical variation, so other phenomena may use different mechanisms.
Abstract
from arXiv · showhide
Multilingual large language models (mLLMs) achieve strong performance in machine translation, yet our understanding of the mechanisms by which they transform representations from one language to another remains incomplete. Prior work suggests that translation decomposes into separable processes within an mLLM, where conceptual content is first represented independently, followed by a production into language-specific form. In this work, we show that translation is even more modular than previously assumed and that the output language production in translation processes is actually further separable into a syntax and a surface language process. We construct controlled multilingual datasets that isolate cross-linguistic differences in word-order and use causal interventions and probing to track how representations are transformed during translation. We find that models first construct target-side word-order before realizing the target language surface form. We identify individual attention heads that are selectively sensitive to syntactic transformations while remaining largely invariant to language identity. These results establish the commitment to a syntactic structure as an independent stage in translation, extending prior decompositions and showing how translation is implemented by functionally different components within mLLMs.
1 Introduction
The paper asks whether grammatical structure is a distinct translation stage and finds that mLLMs resolve target-side syntax before surface language and lexical content. This stage is partly localized to language-agnostic components, extending prior separations of conceptual meaning and output language.
- Translation must construct well-formed target-side syntax when languages differ in word order, beyond lexical substitution or concept mapping.
- The study uses activation patching and representation probing across models and typologically diverse constructions to track translation information across layers.
- Intermediate representations encode target-language word order before resolving surface language, even while remaining aligned with an English-adjacent conceptual space.
- The paper introduces word order as a third independently trackable component alongside lexical content and meaning.
- Specific attention heads select target word order while remaining more sensitive to grammatical structure than to language identity, suggesting shared mechanisms across languages and constructions.
- Overall, translation is described as a multi-stage computation in which grammatical structure is constructed early and partially independently rather than transferred in one step through an abstract interlingua.
2 Related Work
Prior research identifies English bias and shared multilingual representations, while showing that language and conceptual content can be encoded separately. This paper builds on that work by testing whether syntax is another separable component established independently of surface language.
- mLLMs often favor English-like constructions and representations, partly reflecting English dominance in training data and English-derived dataset construction.
- Intermediate activations can correspond to English tokens before shifting to the target language, while injecting English representations may improve performance.
- Other work finds shared intermediate representations across languages that are not solely explained by the pretraining language mix.
- Research also identifies language-agnostic representations, common circuits, and features capturing grammatical structure in multilingual models.
- Dumas et al. show that language and conceptual content are separately encoded and manipulable, motivating this paper’s examination of syntactic separability.
3 Methodology
The study combines controlled multilingual datasets with layer- and module-level interventions to isolate syntax, surface language, and lexical content during translation. It compares models and constructions that systematically vary in word order.
- The study analyzes mGPT 1.3B, Aya Expanse 8B, and LLaMA 3 8B, which differ in size, training distribution, and English dominance.
- Three synthesized datasets target adjective-noun ordering, subject-verb-object versus subject-object-verb order, and modal-verb placement.
- The datasets use minimal multilingual sentences generated from English source expressions to isolate systematic constituent-order differences.
- LogitLens tracks intermediate next-token predictions, while activation patching causally localizes information across layers, positions, attention heads, and MLP modules.
- The framework distinguishes surface language L, word-order syntax S, and lexical content C, with prompts varying systematically along these factors.
- The NP dataset includes 90 target-language combinations and 17,730 translation prompts, while SVO and MV include 36 combinations with 5,698 prompts and 16 combinations with 496 prompts, respectively.
- Base and plant prompts use controlled target-language pairs with different word orders, while the source language remains matched as a practical design choice.
- Experiments evaluate eight S-L-C combinations by measuring how interventions shift probability mass among tokens representing language, syntactic role, and lexical content.
4 Results: Translation Decomposes into Syntax, Language, and Meaning
The results support a modular translation process in which target-side syntax emerges before surface language and lexical content. Across models and datasets, interventions and probing reveal an ordered S → L → C strategy, with syntax often localized to specific attention heads.
- Syntactic Structure Emerges Before Surface Language: Intermediate representations encode target-language word order before switching to the target surface language, even when predicted tokens remain English-like.For German–French noun phrases, intermediate predictions follow the target noun–adjective order while retaining English lexical forms.
- Translation is Modular with Respect to Syntax and Surface Language: Llama 3’s patched activations establish target syntax at Layer 14, switch surface language at Layer 17, and resolve lexical content only later.With French as the base and Dutch as the target, the model first shifts to the target adjective position while remaining in French.
- Translation is Modular with Respect to Syntax and Surface Language: The syntax switch precedes the language switch, which precedes the lexical switch in eight of nine model–dataset combinations.The exception is Llama 3 on SVO, where syntax and language switch together at Layer 14 before lexical content.
- Translation is Modular with Respect to Syntax and Surface Language: Across models and datasets, lexical content is resolved after syntactic and surface-language aspects, supporting a multi-stage S → L → C computation.mGPT does not show the same English-like inner-layer signal observed in Aya Expanse and Llama 3, but the broader ordering remains supported.
- Syntactic Commitment is Localized to Individual Attention Heads: Llama 3 and Aya Expanse localize syntactic influence to one attention head, whereas mGPT distributes it across multiple heads.The strongest mGPT heads have R ≈1.44 and R ≈1.27; Llama 3 and Aya Expanse each show a single strongly syntax-influential head.
- Syntactic Commitment is Localized to Individual Attention Heads: Syntax-sensitive heads primarily affect syntactic structure while remaining comparatively insensitive to surface language in Llama 3 and Aya Expanse.Their syntax influence reaches R ≈3-5, compared with below R ≈2 for mGPT, whose strongest head also affects surface language.
5 Results: Word Order Attention Heads are Language-Independent
Activation patching shows that S-sensitive attention heads encode target-side word order largely independently of surface language. Their causal effects are strongest in some models, while activation geometry does not cluster cleanly by word order.
- Patching setup: Mean activation patching tests whether the most syntax-sensitive head responds to target-language word order or surface language.Patches replace a base run’s head activation with mean activations from another target language, varying whether word order is shared.
- Patching results: KL divergence is consistently higher for part-of-speech contrasts when base and patch languages differ in word order, while surface-language contrasts remain low.The result indicates that patching primarily changes syntactic structure rather than transferring the plant language.
- Patching results: Patching shifts word order without systematically increasing probability mass for the plant language, supporting language-independent encoding by S-sensitive heads.This pattern is most pronounced in mGPT and Llama 3; Aya-Expanse shows weaker and less symmetric effects.
- Caveats: The intervention’s effect is limited because it replaces the activation of only a single attention head.This constrains the magnitude of the measured causal shift.
- Caveats: A minor exception occurs when German occupies the base target-language position, and related effects are absent in Aya-Expanse and Llama 3.The German exception and model differences may reflect interactions involving language relatedness, prompt language, or resource effects.
- Activation geometry: Cosine similarity does not show clear clustering of S-head mean activations by word order across target languages.Thus, syntactic sensitivity appears functional rather than reflected in globally shared activation geometry.
6 Conclusion
The paper concludes that multilingual translation is a staged, partially modular computation rather than a single transfer through an abstract interlingua. Syntax is resolved before surface language and lexical content through components that are partly distinct and language-shared.
- Conclusion: Translation follows a staged S → L → C process in which syntax is resolved before surface language, which is resolved before lexical content.This establishes syntax as a third independently trackable component alongside language and conceptual content.
- Conclusion: Specific layers and, in most models, individual attention heads selectively respond to word order while remaining largely invariant to language identity.The findings localize syntactic transformations to functionally different model components.
- Conclusion: Syntactic sensitivity is functional rather than geometric: S-head activations encode word order independently of surface language without clustering globally by word order.Shared computation therefore need not produce globally similar activation patterns.
- Conclusion: Intermediate representations can align with English tokens while already reflecting target-language word order, indicating that syntax is established before surface language.English is characterized as a conceptual anchor rather than a syntactic pivot.
- Conclusion: Translation progressively integrates syntax, language, and meaning through a sequence of transformations attributable at least partly to distinct network components.This supports a staged, partially modular account rather than a single-step interlingua transfer.
Limitations
The study’s conclusions are bounded by its restricted grammatical coverage, methodological assumptions, limited model set, and synthesized data and prompting conditions.
- Future work: The paper omits additional relevant aspects that future work should investigate.This broad boundary accompanies the more specific limitations on grammar, models, interventions, and data.
- Scope: The experiments primarily cover noun phrase structure and verb and subject order, representing only a small subset of grammatical variation.Other phenomena, including grammatical case and long-distance dependencies, may rely on different mechanisms.
- Methodological assumptions: The causal interpretation of activation patching assumes that translation and grammar-mapping computations are easily interchangeable and linear.This assumption remains a methodological constraint on the intervention.
- Models: The model set is limited, and only one model confirmed to use non-English-biased training data is small by current LLM standards.This restricts how broadly the findings can be generalized across multilingual models.
- Data and prompting: The synthesized dataset and one-shot prompts may not reflect strategies used with more naturally diverse or parallel data.Natural examples from parallel datasets could lead multilingual LLMs to implement different translation strategies.
A.2 Dataset Generation
The appendix documents controlled dataset generation and auxiliary analyses used to support the translation experiments. It combines generated noun-adjective pairs, manually written SVO sentences, fixed prompt formats, and patching-based interpretability checks.
- Dataset Generation: Many generated pairs were modified because French and Italian translations commonly followed head-final word order.This editing step addressed mismatches between generated English pairings and target-language syntax.
- Dataset Generation: SVO sentences were written manually because ChatGPT was impractical for generating them consistently.Some nouns, including body parts and places, were omitted during construction.
- Dataset Generation: ChatGPT was prompted to generate contextually fitting, easily translatable noun-adjective pairs in CSV format.The instruction required compositional phrases and minimized ambiguity in adjective translation.
- Prompting and controls: The experiments used fixed prompt formats, but model performance remained sensitive to prompt phrasing.Including a particular phrase in the instruction significantly raised prediction accuracy.
- Auxiliary analyses: Auxiliary entropy analysis found that Llama 3 treated nouns and adjectives similarly in its layerwise certainty about upcoming tokens.The analysis also compared these patterns with noisy random-character sequences.
- Patching and LogitLens: LogitLens patching showed that Layer 14 can carry syntax-related information unrelated to a specific surface language.An English noun became more probable while the patched French syntax competed with German noun-adjective ordering.
D Supplementary results: LogitLens
The supplementary LogitLens analysis presents projected token probabilities across configurations of language pair and dataset. These plots examine how token probabilities evolve through the model.
- LogitLens projections: The analysis compares projected token probabilities across multiple language-pair configurations.The configurations vary the language pair used in the projection analysis.
- LogitLens projections: The analysis also compares projected probabilities across different datasets.The stated configurations include both language-pair and dataset variation.
- LogitLens projections: The resulting plots provide layer-wise probability trajectories for the examined configurations.
E Supplementary Results: Mean activation Patching of Individual Attention Heads
The head-patching results reproduce higher KL divergence for syntactically heterogeneous language pairs. They also suggest that English behaves differently from other languages in activation steering.
- Cross-language steering: KL divergence is higher for syntactically heterogeneous language pairs.The pattern is reported for German and is repeated with other source languages.
- Cross-language steering: English does not influence token probabilities proportionally when used as a base or patch target language in Aya-Expanse or Llama.The passage contrasts English with other languages in the steering results.
- Interpretation: The authors theorize that English may represent meaning in a more neutral way within inner-layer activations.They further suggest this status could limit how strongly English steers or is steered by activations.
F Supplementary Results: Activation Patching of Individual layers
Layer-patching analyses examine how syntax and language information emerge across controlled datasets and model families. Results include a staged pattern in mGPT, closer coupling in Aya Expanse, and limited conclusions for naturalistic NP data.
- Individual-layer patching: mGPT provides a clear S →L →C example when Vietnamese is the base language and German is the plant target language.Vietnamese is noun-first, whereas German is adjective-first.
- Individual-layer patching: Aya Expanse shows syntax and language switching at the same layer, suggesting these representations are more closely tied than in mGPT or Llama-3.The reported switch occurs at Layer 15, while the syntax-related probability rise remains insufficient to overcome the base token.
- Naturalistic-data results: The NP Natural patching results provide potential evidence for the S →L →C strategy in mGPT and Aya Expanse, but dataset size limits substantial conclusions.The passage motivates further analysis with more naturalistic language data.