PaperlayerView exact revision on arXiv

Source-linked AI summary

The Interlingua Hypothesis: LLMs Translate via a Latent Task-agnostic Feature Space

Jacob Brinton, Jannik Brinkmann, Mark Crovella, Aaron Mueller

arXiv:2609.00515v1cs.CLcs.AI

TL;DR

The paper asks how LLMs translate and whether they use a shared latent feature space rather than specialized language-pair mechanisms. It proposes the interlingua hypothesis and finds convergent evidence from competence modeling, causal component analysis, and fine-tuning comparisons. These findings support understanding translation as source-feature mapping followed by target-language generation, while leaving alternative mechanisms possible and limiting generalization beyond the tested models.

  • Problem

    The paper asks whether LLM translation uses task- and language-agnostic mechanisms or specialized translation and language-pair mechanisms.

  • Method

    The paper tests the interlingua hypothesis through competence modeling, causal analysis of model components, and comparisons of monolingual and parallel fine-tuning.

  • Results

    The three analyses provide convergent evidence consistent with translation using multilingual representations relevant to monolingual tasks.

  • Takeaways & Limitations

    The findings support viewing translation as mapping source input into a shared latent feature space before generating target-language text.

  • Takeaways & Limitations

    The experiments were limited to two 8B-parameter models and one 3B-parameter model, so the findings may not generalize to smaller or larger LLMs.

Abstract

from arXiv · show

Large language models (LLMs) have recently demonstrated improved machine translation performance over strong supervised baselines. This raises questions as to what mechanisms underlie how LLMs perform machine translation between languages. Motivated by recent interpretability findings--namely, that LLMs use massively multilingual latent feature representations to perform language modeling--we propose the interlingua hypothesis. The hypothesis holds that language models translate by reading a source sentence into a latent feature space, and generate a target sentence by reading from the latent feature space. We show three lines of evidence in support of this hypothesis: (1) variance in BLEU across language pairs is largely predictable from language-specific competences with no language pair-specific interaction terms; (2) many model components are causally influential in both monolingual tasks and translation tasks; and (3) fine-tuning on monolingual data recovers a large proportion of translation improvements relative to fine-tuning on aligned documents. Together, these provide convergent evidence in support of the interlingua hypothesis, and suggest new ways of understanding and improving how LLMs can be leveraged to perform translation tasks.

1 Introduction

The paper asks whether LLM translation relies on shared, task- and language-agnostic mechanisms rather than specialized language-pair mechanisms. It proposes the interlingua hypothesis and tests three predictions, finding positive evidence for each.

  • 1 Introduction: LLM translation may reuse computational mechanisms for monolingual language modeling.The proposed mechanism reads inputs into increasingly abstract representations and generates tokens from those representations.
  • 1 Introduction: The interlingua hypothesis proposes that translation reads source-language input into abstract features and generates target text conditioned on them.
  • 1 Introduction: The hypothesis predicts that translation performance depends on source- and target-language monolingual capabilities without language-pair interaction terms.
  • 1 Introduction: It also predicts task-agnostic representations causally relevant to both translation and monolingual tasks.
  • 1 Introduction: The authors find positive evidence for all three implications, while noting that no single experiment definitively confirms the hypothesis.

2 Related Work

Related work motivates the paper through multilingual representations, engineered interlingua approaches, and the possibility of cross-lingual transfer from monolingual competence. The paper’s causal mediation analysis connects to established mechanistic-interpretability methods.

  • Massively multilingual feature representations: Prior interpretability work finds highly multilingual grammatical concept representations whose interventions affect both monolingual and translation behavior.
  • Scope: The paper’s mechanism claims concern fully trained LLM behavior, not whether parallel data was necessary to learn multilingual representations.
  • Interlingua in multilingual NMT: Earlier NMT research engineers shared interlingua representations through explicit layers, networks, or architectural bottlenecks.
  • Interlingua in multilingual NMT: Prior studies also test whether multilingual models need shared or language-specific components by varying parameter sharing.
  • LLMs for machine translation: The paper asks whether improving monolingual competence in a low-resource language can improve translation without additional parallel data.
  • Causal mediation analysis: Its causal evidence relies on causal mediation analysis, a technique used in mechanistic interpretability.

3 Modeling Translation Performance as a Function of Monolingual Capabilities

The authors model translation performance from monolingual competence proxies and find that language-specific competence largely predicts BLEU, with little added value from language-pair interactions. GlobalMMLU is the strongest proxy among those evaluated.

  • Modeling approach: A linear model predicts translation competence from source- and target-language monolingual competencies.
  • Monolingual competence proxies: MultiBLiMP measures grammatical discrimination using minimal-pair log probabilities, while GlobalMMLU measures multilingual multiple-choice accuracy.
  • Translation evaluation: Translation competence is measured by BLEU for each source-target language pair using a 2-shot FLORES prompt.
  • Results: MultiBLiMP grammatical accuracy reaches R2 = 0.294/0.235 for Llama/Aya, but its scores saturate at relatively low BLEU scores.
  • Results: GlobalMMLU accuracy is a very strong predictor, with R2 = 0.739/0.510 for Llama/Aya.
  • Results: Adding language-pair interaction terms does not increase predictive power over the linear model.
  • Results: Per-language source and target main effects explain R2 = 0.932 (Llama) / 0.879 (Aya) of centered BLEU variance.
  • Results: Target competence contributes more than source competence, with target-language variance exceeding source-language variance by 9.3× for Llama and 3.9× for Aya.

4 Many Translation-relevant Components Also Perform Monolingual Tasks

The paper tests whether translation-relevant components are reused for monolingual language modeling, combining causal mediation, cross-language comparisons, and ablations. The same attention heads influence correct translation and grammatical or task outputs, with effects broadly shared across language pairs.

  • Using GCM to Identify Translation-relevant Components: Causal mediation measures how intervening on an attention head changes the model’s preference for a gold translation under matched versus mismatched source prompts.The preference metric isolates how much producing the gold translation depends on reading the matching source content.
  • Using GCM to Identify Translation-relevant Components: The identified heads have larger effects in translation than in controls, exceeding the null cross-language control by 3.0× across all heads and 5.2× among Llama’s top-10 heads.For Aya, the corresponding increases are 2.5× across all heads and 4.3× among the top-10 heads.
  • Using GCM to Identify Translation-relevant Components: Translation-specific heads are localized to layers 13–14 in Llama and layers 15–20 in Aya, corresponding to layers that refine concept representations.This localization is interpreted relative to the stages of inference hypothesis.
  • Translation Heads Have Similar Effects Across Language Pairs: The top attention heads are largely shared across language pairs, with the most universal heads appearing in the top-20 for all 56 directions and retaining stable effect signs.Most top heads have the same directionality and general magnitude of effect across source and target languages.
  • Ablating Translation Heads Degrades Performance on Translation and Monolingual Tasks: Ablating POS-10 heads degrades BLEU in all 8 target languages for both models by roughly twice as much as the random control.The same heads also affect monolingual performance, supporting reused computations across translation and monolingual tasks.

5 Monolingual Fine-tuning Recovers Most Gains from Parallel Data

Monolingual fine-tuning on Xhosa recovers most translation gains obtained with parallel fine-tuning in both evaluated models, while largely preserving other translation abilities.

  • The comparison trains on unaligned Xhosa-rich text versus paired Xhosa-English sentences, using Llama-3.1-8B and TinyAya-3B.The monolingual mixture contains 80% Xhosa and 20% English, French, and German data to reduce catastrophic forgetting.
  • Monolingual fine-tuning raises Llama’s Xhosa BLEU from 16.84 to 24.81, versus 24.88 with parallel fine-tuning.This recovers 99% of the improvement over the base model.
  • Monolingual fine-tuning raises Aya’s Xhosa BLEU from 23.46 to 26.64, versus 27.80 with parallel fine-tuning.This recovers 73% of the improvement over the base model.
  • Both fine-tuning conditions largely preserve translation performance in other languages, with comparable results also reported for German and Thai directions.Changes are smaller for higher-resource languages.
  • After pretraining, a large share of gains from parallel fine-tuning can be achieved using only unaligned data.Parallel data may still perform better because it can also strengthen translation-specific mechanisms.

6 Discussion and Conclusions

The paper concludes that multilingual representations relevant to monolingual tasks also mediate a substantial part of LLM translation. It interprets the evidence as support for an interlingua mechanism while retaining alternative and translation-specific mechanisms as possibilities.

  • The proposed interlingua maps source-language input into a latent feature space and generates target-language text from that shared representation.The representation can also in theory support monolingual next-token predictions.
  • Across three complementary analyses, translation performance is explained by language competence, overlapping components, and monolingual fine-tuning gains.The analyses provide convergent evidence consistent with the interlingua hypothesis.
  • Target-language competence explains more BLEU variance than source-language competence, possibly indicating that fluent grammatical generation is often the bottleneck.The paper presents this as an interpretation of the observed predictive relationship.
  • Weak translation into a language may reflect weak target-language decoding despite usable latent source representations, rather than an absent dedicated language-pair circuit.This is stated as a consequence under the paper’s proposed view.
  • Translation-specific mechanisms likely coexist with interlingua-based mechanisms, so the paper does not claim that interlingua is the only translation route.The authors describe LLM translation as likely using a mixture of mechanisms.

Limitations

The evidence is constrained by model coverage, causal-analysis scope, and the relatively small set of translation language pairs. The interpretation also depends on BLEU, which may not capture all features of high-quality translation.

  • The interlingua hypothesis remains one plausible explanation because the experiments do not rule out other mechanisms.
  • Target-language predictive power may be an artifact of BLEU’s n-gram computation, and BLEU may omit features underlying high-quality translations.
  • The experiments cover only two 8B-parameter models and one 3B-parameter model, so results may not generalize across model sizes.The paper calls for evaluation on a wider variety of language models and thinking models.
  • GCM experiments examine components mediating at the last token and therefore omit computations occurring at earlier positions.
  • Translation experiments use a relatively small number of language pairs, leaving broader cross-lingual generality and fine-tuning conditions for future work.

A Additional Results for Modeling Translation Performance

Additional analyses find that translation BLEU is largely predictable from language-specific competence, with weak language-pair interactions and substantial rank-1 matrix structure. Results vary by competence proxy and model component, while several robustness analyses clarify the signal’s scope.

  • Language-pair interactions: Linear and bilinear models produce visually indistinguishable predicted-versus-actual BLEU scores.
  • Language-pair interactions: The language-pair interaction is non-significant in every tested cell, with the largest ∆R2 equal to 0.0038.
  • Language-specific competence: A rank-1 predictor based on source- and target-language average BLEU has generally low error around 10–15%.The reconstruction uses source and target competence terms to predict each language pair’s BLEU.
  • Competence proxies: Restricting MultiBLiMP to subject–verb agreement raises Llama’s 18-language R2 from 0.324 to 0.363.SV-Person reaches R2 = 0.688, SV-Gender 0.519, and SV-Number 0.412; the authors treat this as a strengthening analysis rather than a headline.
  • Competence proxies: Aggregate MMLU reaches R2 = 0.598 for Llama and 0.499 for Aya across 23 languages, exceeding every individual subject category.
  • Robustness: Transforming saturated accuracy improves R2 from 0.250 to 0.339 for Llama and from 0.394 to 0.444 for Aya.The authors interpret these gains as variance stabilization rather than evidence for a specific functional form.
  • Competence proxies: Adding MultiBLiMP source and target terms to Global MMLU produces a small but significant gain.The result indicates grammatical competence carries translation-relevant signal beyond general task accuracy.
  • Mechanistic analyses: Translation-relevant attention-head effects concentrate in a majority of the top heads and are universal across language pairs, with more divergence for Hebrew and Hindi.Aya’s top heads concentrate in slightly later layers than Llama’s, while top features show weaker separation.

B.5 Head-IE concentration and sparsity

Head-level indirect effects are highly concentrated: a small set of heads dominates translation-related attribution across directions, while selected heads also affect grammaticality judgments. The concentration and cross-task effects support reusable computational machinery, although effect magnitude is not uniquely translation-specific.

  • Cross-task effects: Ablating POS-10 heads lowers grammaticality margins in all 8 target languages, while ablating NEG-10 heads raises them for both models.The direction of the effects transfers across tasks, though POS-10’s magnitude is comparable to random same-layer ablations.
  • Ablation dose response: Ablating different numbers of top heads produces a monotonic performance effect, indicating a dose response rather than dependence on one ablation count.The reported dose response is shown for German, English, and French on Llama.
  • Head-IE sparsity: The largest head’s mean |c IE| is 60–162 times the median head’s across Llama directions, with a median ratio of 116.The ratio is largest for directions into Spanish, German, and French, and smallest into Hindi and Hebrew.

C Additional Fine-Tuning Results

Additional Xhosa fine-tuning experiments compare monolingual and parallel adaptation across three source directions and two models. Both conditions improve translation, but the stronger condition varies by direction and model.

  • Xhosa adaptation: Both monolingual and parallel fine-tuning improve translation into Xhosa across all three directions and both models.The comparison covers English, French, and German as source languages.
  • Xhosa adaptation: For Llama, parallel fine-tuning performs best from English and French, whereas monolingual fine-tuning performs best from German.The best-performing condition therefore depends on the source direction for Llama.
  • Xhosa adaptation: For Aya, monolingual fine-tuning performs slightly better in all three tested directions.Neither fine-tuning condition is best in every setting across the experiments.

C.2 Fine-tuning on higher-resource languages

Fine-tuning on already higher-resource languages does not produce a consistent translation gain in the tested Llama experiments. Monolingual fine-tuning largely preserves performance, while parallel fine-tuning sometimes reduces it.

  • Higher-resource adaptation: Neither monolingual nor parallel fine-tuning consistently improves translation for German and Thai, which already perform strongly in the base model.The experiments use Llama-3.1-8B and evaluate directions involving the adapted language and other translation directions.
  • Higher-resource adaptation: Monolingual fine-tuning leaves performance largely unchanged, whereas parallel fine-tuning reduces performance in several directions.This pattern differs from the improvements observed for Xhosa adaptation.
  • Interpretation: A possible explanation is that strong pretraining representation of German and Thai leaves less room for improvement from continued training.The authors also note that some fine-tuning documents may have appeared in the pretraining corpus.

C.3 Preservation of monolingual abilities

The preservation experiments assess whether Xhosa fine-tuning retains monolingual abilities and whether it supports cross-lingual transfer. Performance is largely preserved, but near-ceiling base scores make transfer difficult to determine.

  • Retention of monolingual abilities: After fine-tuning Llama on Xhosa, MultiBLiMP performance is largely preserved.The evaluation measures monolingual linguistic abilities after adaptation.
  • Interpretation: Near-ceiling base scores on all three languages make small MultiBLiMP differences difficult to interpret.The corresponding table reports scores as percentages and warns that the near-ceiling baseline limits interpretation.
  • Interpretation: The results are not strong enough to determine whether fine-tuning causes cross-lingual transfer of linguistic abilities.This is an explicit limitation of the preservation analysis.
  • Experimental caveat: Because only the monolingual condition includes replay data from high-resource languages, retained-performance differences cannot be attributed solely to monolingual rather than parallel data.A parallel condition with comparable replay data could also reduce forgetting.
Loading 2609.00515v1…