PaperlayerView exact revision on arXiv

Source-linked AI summary

Lingua Franca or Probing Artifact? Rethinking Latent Language in Multilingual LLMs

Deniz Bayazit, Badr AlKhamissi, Antoine Bosselut

arXiv:2609.00155v1cs.CLcs.AIcs.LG

TL;DR

Latent language identification is often interpreted as evidence that multilingual LLMs use an internal language, yet existing probes may measure different phenomena. This paper compares representation- and decoding-based probes across models, training regimes, tasks, domains, checkpoints, and languages. It finds systematic disagreement: GMM representations show earlier mixing and weaker English bias, while decoding retains sharper, more English-favored signals, supporting a cautious interpretation of latent language estimates.

  • Problem

    It is unclear whether latent language probes identify a common internal mechanism or reflect diagnostic choices, although this distinction matters for understanding multilingual information sharing and routing.

  • Method

    The study compares a GMM-based representation probe and decoding-based probes across models, training regimes, tasks, domains, checkpoints, and up to 27 languages.

  • Results

    The probes disagree systematically: GMM identifies language structure earlier with weaker English bias, while decoding pathways retain sharper and more English-favored signals.

  • Takeaways & Limitations

    Existing probes expose complementary aspects of multilingual computation rather than directly revealing a single internal lingua franca.

  • Takeaways & Limitations

    The evidence is limited to decoder-only models in the 7–9B range, with checkpoint analyses restricted to OLMo-2 and Apertus.

Abstract

from arXiv · show

Latent language identification is often used to argue that multilingual language models route computation through language-specific states, such as English pivots. However, existing probes infer latent language from different signals, such as the geometry of hidden states or what can be decoded from intermediate representations. Since such claims shape conclusions about how models share and route information across languages, we ask whether these probes measure the same phenomenon or expose distinct aspects of multilingual computation. We study this question across model families, training regimes, domains, tasks, checkpoints, and up to 27 languages. We find that identification probes systematically disagree: the GMM-based representation probe, which draws evidence from hidden state geometry, shows earlier cross-lingual mixing, whereas decoding-based probes, which rely on output-space decodability, retain sharper language-specific and more English-biased signals. These differences track model multilinguality and training progression, but are comparatively stable across domains. Our results suggest a more cautious interpretation of latent language identification, where current probes expose different aspects of multilingual processing, rather than directly revealing a single internal lingua franca.

1 Introduction

Latent language identification is used to study whether multilingual LLMs route meaning through an internal language, but estimates depend on the diagnostic evidence extracted from hidden states. This work reframes the issue as a measurement problem and finds that representation- and decoding-based probes expose systematically different aspects of multilingual computation.

  • Existing probes: Prior LLID studies use representation-based probes that infer language from hidden-state geometry and decoding-based probes that project intermediate states through the output head.These probe families may rely on different assumptions about what it means for a model to internally use a language.
  • Motivation: The paper asks whether latent language estimates reflect model properties or diagnostic choices such as activation geometry versus output-space decodability.The question matters because latent-language claims influence how multilingual information sharing and routing are understood.
  • Approach: The study compares a GMM representation probe with decoding probes across models, training regimes, tasks, domains, and checkpoints.The analysis first examines decoding-based LLID consistency and generalization, then traces estimates across domains, training regimes, and checkpoints.
  • Main finding: LLID probes should not be treated as interchangeable measurements of a single phenomenon.The same intermediate hidden states can yield different latent-language distributions depending on the probe used.
  • Main finding: The GMM probe shows earlier cross-lingual mixing and weaker English dominance, whereas decoding probes retain sharper language-specific and more English-biased signals.These differences recur across analyses, while their layerwise trajectories remain model-dependent.
  • Variation across conditions: Domain and language-inventory changes shift LLID estimates only modestly, unlike multilinguality and training progression.The exact layerwise effects vary with model multilinguality and pretraining, although multilinguality alone does not determine them.

2 A Framework for Comparing Latent Language Probes

The paper defines latent language identification as estimating language evidence from an intermediate hidden state and compares two probe families that use different evidence. It evaluates their distributions with complementary metrics while noting important design assumptions and possible confounds.

  • Task definition: Given hidden state h, LLID estimates a conditional distribution q over a finite candidate-language set L.The latent language variable is not directly observed, so q is treated as a diagnostic estimate rather than ground truth.
  • Representation-based LLID: Representation-based probes compare h with language-conditioned activation structure and aggregate per-language scores into a layerwise distribution q.The experiments fit a per-layer GMM with one language-conditioned component per candidate language.
  • Limits: Representation-based structure may conflate language with script, tokenization, or domain, while decoding methods assume the final unembedding remains meaningful at intermediate layers.The latter assumption may fail in earlier layers, and token-level matching introduces additional design choices.
  • Decoding-based LLID: Decoding-based probes infer language from what can be read out through the model vocabulary using an intermediate vocabulary distribution.Raw logitlens applies the unmodified output head, while tuned lens first learns a per-layer affine transformation toward the final-layer hidden state.
  • Decoding-based LLID: Decoding methods construct language scores through target-token probabilities, answer-sequence probabilities, or language identification of decoded rollouts.The experiments use summed valid first-token probabilities for known completions and rollout-based variants for open-ended tasks.
  • Metrics: LLID estimates require complementary metrics because distributions differ in concentration, task-language alignment, and estimator agreement.The framework includes entropy, dominance, pivot rate, and agreement, computed per estimator and averaged over prompts at each layer.

3 Experimental Setup

The study evaluates latent language identification across controlled and open-ended multilingual settings, models, and language inventories using representation- and decoding-based estimators.

  • Models and regimes: The evaluation covers up to 12 autoregressive models, including English-centric baselines, multilingual base models, and instruction-tuned variants.Models are compared using multilinguality proxies including zero-shot multilingual QA accuracy, cross-lingual perplexity, and byte-normalized likelihood.
  • Controlled evaluation: Experiments use controlled copy, cloze, and translation tasks, where candidate completions and language-specific answer tokens are known.The controlled score sums probabilities over tokens that can begin each candidate-language answer and normalizes scores over the candidate language set.
  • Open-ended evaluation: The paper also tests open-ended natural-language prompts, varying language inventory with PUD data and domain sensitivity with INCLUDE question-answering prompts.Open-ended estimates summarize language distributions around the prompt midpoint and over a five-position forward window.
  • LLID estimators: Open-ended comparisons include a GMM representation posterior and decoding-based argmax or top-p rollouts using raw logitlens or tuned lens projections.Generated continuations are classified with GlotLID and aggregated into language distributions.

4 The Controlled Case: Latent Language Under Known Completions

In controlled tasks with known completions, decoding-based LLID reveals language information mainly in later layers, with signal sharpness and English competition varying by task and model multilinguality.

  • Layerwise language signals: Across translation, copy, and cloze tasks, decoded language signals are weak early and most visible in later layers.The study tracks summed START(w) probabilities for language-specific tokens across layers in Llama-2-7B, Aya-23-8B, and Apertus-8B.
  • Task effects: Copy produces sharper, higher late-layer probabilities for the observed language than cloze.Copy can reproduce a surface form in context, whereas cloze requires retrieving and lexicalizing a concept in the appropriate language.
  • Task effects: Cloze shows lower peaks, more competing languages, and more diffuse probability mass, especially for Llama-2 and Aya-23.This contrasts with copy because cloze requires concept retrieval and language-appropriate lexicalization.
  • Model multilinguality: Even when English is excluded from the translation direction, Llama-2 often assigns relatively high probability to English lexicalizations.The figure measures summed START(w) probability across candidate-language tokens by model layer and translation target.
  • Model multilinguality: In Aya and especially Apertus, target-language probability rises earlier and becomes more consistently dominant in later layers.Together, the controlled results show that START(w) recovers a meaningful late-layer language signal whose English bias varies with multilinguality.

5 Generalizing Decoding-based Estimates to Open-Ended Generation

Open-ended generation shows that latent language estimates depend strongly on how intermediate states are projected and decoded, rather than yielding a probe-independent signal.

  • Projection and decoding effects: Raw logitlens decoding assigns substantially more probability to English than controlled START(w) evaluation on open-ended prompts.When continuations are not restricted to known translations, English lexicalizations remain accessible to the decoder.
  • Figure comparison: Figure 4 compares normalized START(w) probabilities for translation with raw logitlens, tuned lens, and GMM estimates on open-ended INCLUDE prompts.Rows aggregate the 50–75% and 75–100% layer windows; bars show pooled average probabilities and diamonds show average maximum probabilities.
  • Figure comparison: The figure’s central contrast is whether later-layer estimates favor English or task-relevant languages under different probes.Raw logitlens favors English, whereas tuned lens and GMM estimates favor task-relevant languages.
  • Projection and decoding effects: Tuned lens shifts average probability toward the task-relevant language while probability assigned to other languages remains low.The result is consistent with tuned lens partly collapsing intermediate language evidence toward the final decoded language.

6 Representation vs. Decoding: Different Evidence, Different LLID Behaviors

Representation-based and decoding-based LLID probes use different evidence from hidden states, producing systematically different language estimates. Their layerwise and training trajectories depend on model and diagnostic choice, while domain and language-inventory changes have smaller effects.

  • Different evidence: Representation-based LLID assesses hidden-state geometry, whereas decoding-based LLID assesses language accessibility through the output vocabulary.A hidden state can be geometrically close to one language cluster without being decodable as that language.
  • Different evidence: The representation probe shows earlier cross-lingual mixing and weaker English dominance than decoding-based probes.On INCLUDE, the gap is especially visible in the 50–75% layer window and in moderately multilingual models such as Llama-3.1.
  • Layerwise behavior: Representation-probe trajectories vary across models, while decoding-based dominance generally changes later in the network.Llama-2, Aya-23, Apertus-8B, and OLMo-2 exhibit distinct layerwise patterns.
  • Robustness across settings: Across domains and language inventories, curves largely preserve their layerwise shape, with model and diagnostic choice affecting absolute LLID levels more strongly.PUD21 generally produces lower dominance and higher pivot rates than smaller candidate inventories.
  • Training dynamics: During pretraining, representation and decoding estimates can change at different points, while agreement remains low across checkpoints.For Apertus, both estimators transition sharply between roughly 20% and 40% of training; most checkpoint-dependent changes occur in late layers.
  • Training regimes: Instruction tuning produces little systematic shift toward English in average LLID probabilities.Base and instruction-tuned models show broadly similar pivot and entropy trajectories, despite some model-specific deviations.

7 Related Work

Related work frames multilingual representations as containing both shared and language-specific structure, while other studies examine latent pivot languages and how multilingual features develop during training.

  • Shared and language-specific multilingual structure: Prior studies describe multilingual hidden states as combining shared subspaces, language-neutral components, and language-specific neurons or components.This literature motivates representation-based LLID while recognizing that shared and language-specific information can coexist.
  • Latent pivots and output language: Decoding-based studies report English-pivot behavior in English-dominated models and different pivoting patterns under non-English-centric training.Causal interventions also suggest that language form and conceptual content can be partially disentangled.
  • Training dynamics of multilingual representations: Training-dynamics research shows that cross-lingual abilities and internal linguistic features develop, stabilize, or change over pretraining.This motivates evaluating LLID across checkpoints rather than only at final model states.

8 Conclusion

The paper reframes latent language identification as a measurement problem and compares representation-based and decoding-based probes across models, training regimes, and tasks. The probes disagree systematically, indicating complementary aspects of multilingual computation rather than a single confirmed internal lingua franca.

  • Conclusion: The study compares GMM representation probes with decoding-based probes across models, training regimes, and tasks.This comparison treats probe choice as part of the measurement question.
  • Conclusion: Representation probes identify language structure earlier and with weaker English bias, whereas decoding pathways retain sharper, more English-favored signals.These differences vary with multilinguality and training progression, but exact layerwise trajectories remain model- and lens-dependent.
  • Conclusion: Domain and language-inventory changes shift LLID estimates only modestly compared with model and diagnostic differences.The conclusion therefore distinguishes relatively stable contextual effects from stronger model- and probe-dependent variation.
  • Conclusion: Existing probes expose complementary aspects of multilingual computation rather than directly confirming a single internal lingua franca.The paper’s interpretation is that probe selection is itself part of the claim being made about latent language.

Limitations

The study’s conclusions are bounded by its model, probe, intervention, and language-inventory coverage, and latent language lacks direct ground-truth observation.

  • Scope: Experiments use decoder-only models in the 7–9B parameter range, with checkpoint analysis restricted to OLMo-2 and Apertus.Whether probe disagreement persists at other scales or in non-decoder architectures remains open.
  • Scope: Layerwise LLID trajectories vary across model families and should not be assumed to generalize unchanged.The observed trajectories are therefore model-specific rather than universally predictive.
  • Methods: The representation-based comparison uses only the GMM estimator and excludes causal interventions and neuron-level attribution.Other approaches could align with either probe family or reveal another aspect of multilingual processing.
  • Interpretation: Because latent language is inferred rather than observed, the study documents probe disagreement without determining which probe best reflects internal computation.Probe-specific design choices may also shift absolute LLID values, even if qualitative trends are expected to remain robust.
  • Language coverage: The 27-language inventory excludes very low-resource languages and formal languages such as code and mathematics.Conclusions about those language types are outside the evaluated scope.

Ethics Statement

The work uses public pretrained models and multilingual datasets, with no new model training, human-data collection, or user-facing deployment. It documents dataset-content risks and methodological scope boundaries.

  • Ethics Statement: The study uses publicly available pretrained models and multilingual datasets under their respective licenses.The datasets include PUD/UD, INCLUDE, and Fineweb2.
  • Ethics Statement: The analysis does not train new models, collect human data, or deploy systems in user-facing settings.
  • Ethics Statement: The datasets may contain personally identifying or offensive content, although the authors did not collect them.
  • Scope and Evaluation: The evaluation covers controlled and open-ended multilingual tasks across language inventories including PUD21+UD6 and INCLUDE.The controlled prompts comprise copy, cloze, and translation tasks; open-ended evaluation uses natural-language data.
  • Scope and Evaluation: BPB reduces, but does not eliminate, tokenization and script effects because byte counts differ across writing systems.Lower BPB is better.

C Training Details

The study combines layerwise representation and decoding probes with controlled target-string prompts and tuned-lens artifacts. Training and evaluation use fixed data, fitting, and optimization procedures across multilingual model analyses.

  • Probe Training: Representation-based LLID fits one per-layer Gaussian model per candidate language from hidden states collected on the training split.Each setup uses at most 100 prompts per language, uniform priors, PCA retaining 98% explained variance, and diagonal covariance.
  • Probe Training: Decoding-based LLID compares raw logitlens projections with tuned-lens projections fitted on multilingual FineWeb2 data.The fitting corpus is independent of PUD, INCLUDE, and controlled target-string evaluation prompts.
  • Probe Training: Tuned lenses use balanced round-robin language sampling, one training epoch, at most 1000 optimization steps, and learning rate 10^-3.Optimization uses AdamW with zero weight decay, temperature 1.0, regularization weights 10^-4, and bfloat16 autocast.

D Additional Results

Additional analyses organize latent-language probabilities across tasks, layer windows, domains, training checkpoints, and model variants. They compare representation and decoding estimators using dominance, pivoting, entropy, and agreement diagnostics.

  • Controlled Tasks: Figures 7–11 show layerwise START(w) probability curves for copy, cloze, and translation tasks.Translation results are grouped by target or source, with English included in some analyses and excluded in others.
  • Layer Windows: Figures 12–15 aggregate language probabilities into layer windows for copy, cloze, translation, and PUD21+UD6 open-ended prompts.Bars pool prompt-layer-language probabilities, diamonds show category maxima, and error bars show standard error.
  • Layer Windows: The task-relevant category includes the prompt language for open-ended PUD21+UD6 and both source and target languages for translation.
  • Domain and Inventory Robustness: Dominance and pivoting are comparatively stable across PUD variants and INCLUDE domains, including Arts/Humanities, Social Sciences, and STEM.The figure compares language models across LLID methods.
  • Evaluation Diagnostics: The additional-results diagnostics include dominance, confident pivot rate, and representation-decoding agreement.
  • Training Dynamics: Training analyses track LLID estimates across pretraining checkpoints using quartile-binned and full-layer trajectories.Figures 18 and 19 separately show representation-based and decoding-based checkpoint trajectories.
  • Model Variants: Base and instruction-tuned variants are compared using pivot rate, entropy, and dominant-language agreement on PUD21 and INCLUDE-10.Agreement is the fraction of matched prompt-layer pairs where both probes select the same dominant language.
Loading 2609.00155v1…