Lingua Franca or Probing Artifact? Rethinking Latent Language in Multilingual LLMs

1 Introduction

• Latent language identification is inferred rather than directly observed, and geometry-based and decoding-based probes extract different evidence that may expose distinct processes.Representation probes use hidden-state geometry, while decoding probes project intermediate states through the model’s output head.Source: §1 · pp. 1–2
• The study compares GMM and decoding probes on controlled and open-ended multilingual tasks, then traces estimates across domains, training regimes, and checkpoints.It first tests decoding-based probe consistency and generalization, then compares representation-based estimates under matched conditions.Source: §1 · p. 2
• The GMM representation probe finds earlier cross-lingual mixing and weaker English dominance, while decoding probes retain sharper language-specific, more English-biased signals.These patterns recur across analyses, but each probe’s layerwise behavior varies by model.Source: Figure 1 · p. 1, §1 · p. 2
• For each probe, layerwise effects vary with model multilinguality and pretraining progression, though multilinguality alone is not determinative; domain and language-inventory shifts are modest.Source: §1 · p. 2

2.1 Task Definition: Latent Language Identification

• Latent language identification estimates a distribution over candidate languages from hidden states, but that distribution is diagnostic rather than ground truth.Different estimators can produce different distributions for the same prompt and layer by selecting and aggregating token-position evidence.Source: §2.1 · p. 2

2.2 Two Kinds of Evidence for LLID and Their Limits

• Representation-based probes infer language from hidden-state structure; the study uses per-layer, language-conditioned GMM posteriors, while noting possible confounds from script, tokenization, or domain.Despite these concerns, the authors report consistent behavior across STEM and Arts & Humanities QA domains.Source: §2.2 · p. 3
• Decoding-based probes infer language from output-vocabulary readouts, using raw or tuned logitlens projections and task-dependent token scoring or rollout methods.The tuned lens learns a per-layer affine transformation before applying the output head; these methods assume the final unembedding remains meaningful at intermediate layers, and target-string matching requires design choices such as gathering valid completions.Source: §2.2 · p. 3

2.3 Metrics to Compare LLID Estimates

• The comparison uses complementary metrics for distribution diffuseness, dominant-language confidence, pivots away from the task-relevant language, and agreement between estimators.Entropy is higher for more mixed distributions, dominance is the largest language probability, pivot rate counts prompts whose dominant language differs from the task-relevant language, and agreement compares estimators’ dominant languages.Source: §2.3 · p. 3

3.1 Models & Measuring Multilinguality

• The study evaluates up to 12 autoregressive models, from English-centric baselines to multilingual base and instruction-tuned models, using multilinguality proxies including zero-shot QA accuracy and cross-lingual likelihood measures.Cross-model plots order models by increasing zero-shot macro-accuracy on INCLUDE; model details, scores, and rankings are in Appendix B.Source: §3.1 · pp. 3–4

3.2 Evaluation Regimes

• LLID is evaluated in controlled settings with known completions and open-ended natural text, using copy, cloze, and translation tasks in the controlled regime.Controlled decoding scores sum probability mass over vocabulary tokens that can begin the target-language answer. Open-ended evaluation uses PUD sentences and INCLUDE QA prompts, measuring at the prompt midpoint over a five-position forward window; PUD variants vary language inventory and INCLUDE domains test domain sensitivity.Source: §3.2 · p. 4

3.3 LLID Estimators

• Open-ended LLID compares a hidden-state GMM posterior (repr) with decoding-based argmax and top-p rollout estimators (dec-rmax, dec-topp), using raw logit or tuned lenses.For decoding-based estimators, short continuations from intermediate layers are classified by GlotLID and aggregated into language distributions.Source: §3.3 · p. 4

• Across translation, copy, and cloze, START(w) language signals are weak in early layers and most visible in later layers, with sharpness varying by model.Source: §4 · p. 4, Figure 3 · p. 5
• The language-signal effect is not necessarily uniform across tasks and languages within a model.Source: §4 · p. 4
• Copy yields sharper, higher late-layer probabilities for the observed language than cloze, whose retrieval and lexicalization demands produce lower peaks and more competing languages.Source: §4 · pp. 4–5
• Llama-2 often retains high probability on English lexicalizations even when English is not the translation direction; this bias weakens in Aya and further in Apertus.In Apertus-8B, intended non-English target probability rises earlier and becomes more consistently dominant in later layers.Source: §4 · p. 5

5 Generalizing Decoding-based Estimates to Open-Ended Generation

• The open-ended diagnostic generates short continuations from intermediate hidden states using raw or tuned lens with argmax or top-p decoding, then classifies the continuation language.It tests whether language preferences extend beyond controlled settings with predefined candidate translations.Source: §5 · p. 5
• Raw logitlens assigns substantially more mass to English than controlled START(w), while tuned lens shifts average probability toward task-relevant languages and keeps other-language probability low.Figure 4 also shows GMM estimates favoring task-relevant languages in open-ended prompts.Source: §5 · p. 6, Figure 4 · p. 6
• Raw logitlens outputs can be difficult to interpret in some models because of representational drift; tuned lens may collapse intermediate language evidence toward the final decoded language (Wendler et al., 2024).The representational-drift concern is attributed to Belrose et al. (2025).Source: §5 · p. 6

6 Representation vs. Decoding: Different Evidence, Different LLID Behaviors

• Representation probes measure hidden-state proximity to language-conditioned activation regions, while decoding probes measure language-token accessibility through the unembedding matrix.A state can be geometrically closer to one language without decoding as that language, so disagreement is measurement information rather than diagnostic failure.Source: §6 · pp. 6–7
• On open-ended INCLUDE prompts, Repr-GMM assigns more probability to task languages and less to English than raw logitlens top-p, especially for moderately multilingual models.For highly multilingual models, average estimates align more closely, though Repr-GMM may still assign a non-task, non-English language the highest probability.Source: §6 · p. 6, Figure 5 · p. 7
• Layerwise trajectories differ by model: representation dominance stays high in OLMo-2, drops then recovers in Llama-2 and Aya-23, and rises late in Apertus-8B.Pivot patterns also vary: Llama-2 mixes English and other-language pivots, Aya-23 has a temporary rise in English pivots, and Apertus-8B pivots mostly to non-English languages; decoding generally changes later. Early-to-mid drops reflect mixed or task-oriented states, while OLMo-2 suggests persistent language-conditioned separation.Source: §6 · p. 7, Figure 5 · p. 7

Across Domains

• Across PUD language inventories and INCLUDE QA domains, each model–estimator pair largely preserves its layerwise curve shape, while PUD21 has lower dominance and higher pivot rates than PUD9.Domain and inventory changes generally affect absolute levels less than model or diagnostic choice; the PUD21 pattern is the exception associated with its larger candidate set.Source: §6 · p. 7

Across Pretraining

• Across pretraining, OLMo representation dominance is high from the earliest checkpoint, while decoding dominance consolidates over the first ∼20–30% of training; Apertus estimators transition around 20–40%, then late-layer dominance reaches ∼0.9.Confident pivoting is low for OLMo representation estimates but moderate and slowly increasing for decoding; representation–decoding agreement stays low for both models, especially Apertus, and most checkpoint-dependent changes occur in late layers.Source: §6 · p. 8, Figure 6 · p. 8

Base vs. Instruct

• Instruction tuning produces little systematic shift toward English in LLID estimates, with effects smaller than changes across pretraining checkpoints.Base and instruction-tuned variants have only modest average-probability differences and broadly similar layerwise pivot and entropy trajectories, despite some model-specific deviations.Source: §6 · p. 8

Shared and language-specific multilingual structure

• Multilingual hidden states contain shared and language-specific structure, including distinguishable subspaces (Chang et al., 2022), language-neutral subnetworks (Foroutan et al., 2022), and language-specific components (Zeng et al., 2025; Zhao et al., 2025).This provides a basis for representation-based LLID while indicating that hidden states can encode both types of structure simultaneously.Source: §7 · p. 8

Latent pivots and output language

• Decoding probes find English pivots in English-dominated models and different patterns under non-English-centric training (Wendler et al., 2024), while interventions separate output language from conceptual content (Dumas et al., 2025).This work instead asks whether probes used to infer latent language behavior agree on the same hidden states.Source: §7 · p. 9

Training dynamics of multilingual representations

• Cross-lingual abilities (Blevins et al., 2022) and internal linguistic features (Bayazit et al., 2026) develop, stabilize, or change during pretraining.Accordingly, the study examines LLID across checkpoints, reasoning that estimates reflecting model properties rather than diagnostic artifacts should vary systematically with training progression.Source: §7 · p. 9

• The GMM representation probe identifies language structure earlier and with weaker English bias, while decoding pathways retain sharper, more English-favored signals.Source: §8 · p. 9
• Probe differences vary with multilinguality and training progression, but layerwise trajectories remain model- and lens-dependent and cannot be predicted from multilinguality alone.Source: §8 · p. 9
• Domain and language-inventory changes shift LLID estimates only modestly, and probe choice is itself part of the claim about multilingual computation.Source: §8 · p. 9

Limitations

• The evaluation covers decoder-only 7–9B models, with checkpoint analyses limited to OLMo-2 and Apertus; layerwise LLID patterns also vary by model family.Whether probe disagreement persists at other scales or in non-decoder architectures is unknown, and the observed trajectories should not be assumed to generalize unchanged.Source: Limitations · p. 9
• Only two probe families are compared, and without ground truth the study documents disagreement but cannot determine which probe, if any, reflects internal computation.The representation-based family uses only the GMM estimator; causal interventions (Dumas et al., 2025) and neuron-level attribution (Zeng et al., 2025) are not examined and could align with either family or reveal another aspect.Source: Limitations · p. 9
• Probe-specific choices may shift absolute LLID values, while the 27-language inventory excludes very low-resource and formal languages such as code and mathematics.Whether probe disagreement predicts failures in cross-lingual transfer or generation remains open; qualitative trends are expected to be robust despite probe choices.Source: Limitations · p. 9

Ethics Statement

• The diagnostic analysis uses publicly available pretrained models and licensed multilingual datasets, without training new models, collecting human data, or deploying user-facing systems.The authors report no direct ethical concerns.Source: Ethics Statement · p. 10

A Dataset Details

• The datasets were not collected by the authors and may contain personally identifying or offensive content.Source: A Dataset Details · p. 12

A.1 Language Inventory

• Language inventories cover controlled target-string, PUD, and INCLUDE runs; PUD and UD entry counts distinguish deterministic train and held-out splits.PUD9+UD6 and PUD21+UD6 add the UD6 extension row to PUD9 and PUD21, respectively.Source: §A.1 · p. 12

A.2 Controlled Target-String Construction

• Controlled copy, cloze, and translation prompts adapt prior target-word setups, with logitlens queried at the final task token before the answer begins.For instruction-tuned models, scoring is anchored to the original task-prefix end rather than any appended assistant-control token.Source: §A.2 · p. 12
• Copy and translation prompts use four completed in-context examples, while cloze prompts use two demonstrations; translation words and cloze contexts are jointly constructed to disambiguate homonyms.Examples are filtered when translated cloze answers do not align with independently translated concept words, but not solely for shared English-target tokenizer prefixes.Source: §A.2 · p. 12
• Candidate-language scores sum next-token probability over tokenizer-valid answer-start tokens; raw sums form probability curves, while distributional metrics normalize scores across candidate languages.Language entropy uses the normalized language distribution; vocabulary entropy uses the full next-token vocabulary distribution.Source: §A.2 · pp. 12–13

A.3 Open-Ended Evaluation Details

• Both LLID approaches are evaluated at a matched prompt anchor, usually the prompt midpoint, with scores aggregated over a five-position forward window.These settings apply to natural-language data without constrained completions.Source: §A.3 · p. 13
• Open-ended decoding-based LLID uses deterministic argmax or five top-p rollouts with p = 0.9, scores continuations with GlotLID, and normalizes candidate-language distributions.For top-p, the distributions are averaged uniformly across samples.Source: §A.3 · p. 13
• Evaluation uses PUD21, PUD9, PUD9+UD6, and PUD21+UD6, plus INCLUDE QA prompts in 10 languages across Social Science, Arts & Humanities, and STEM.PUD21 is the broadest setting; PUD9 follows Shani and Basirat (2025), and UD6 adds six languages selected for script or family overlap.Source: §A.3 · p. 13

B Model Details

• Because pretraining mixtures are not fully available, multilingual capability is proxied by INCLUDE zero-shot QA accuracy, cross-lingual perplexity, and byte-normalized likelihood.Table 2 reports evaluated models and these proxies; its models are sorted by decreasing INCLUDE macro accuracy.Source: B Model Details · p. 13, Table 2 · p. 13
• The training-dynamics analysis aligns OLMo-2 and Apertus checkpoints by token count, as shown in Table 3.Table 3 identifies the checkpoints as token-aligned for that comparison.Source: B Model Details · p. 13, Table 3 · p. 13
• INCLUDE scores are zero-shot accuracy on include-base-44, macro-averaged over 44 language groups and used to sort models as a measure of multilingual task performance.The accuracy is obtained from lm-evaluation-harness runs.Source: B Model Details · p. 13
• PUD21 line NLL is the average bit-converted autoregressive negative log-likelihood of teacher-forced lines across PUD-only parallel rows in 21 languages; lower is better.Table 2 describes line NLL as an auxiliary likelihood score on a parallel corpus across languages.Source: B Model Details · p. 13, Table 2 · p. 13
• PUD21+UD6 BPB sums token-level negative log-likelihood over evaluated lines, converts nats to bits, and divides by total UTF-8 bytes; lower is better.Byte normalization reduces, but does not eliminate, tokenization and script effects; Table 2 notes BPB remains script-sensitive.Source: B Model Details · p. 13, Table 2 · p. 13

C Training Details

• Representation-based LLID fits a per-layer, per-language Gaussian model using up to 100 training prompts per language, uniform priors, and PCA retaining 98% explained variance.The setup uses diagonal covariance, token-level hidden states by default, and separate GMMs for five candidate-language inventories; English is added for applicable INCLUDE plots.Source: C Training Details · p. 14
• Tuned lenses are fit on multilingual FineWeb2 data, independent of PUD, INCLUDE, and controlled target-string prompts, and use a 27-language inventory.Training balances languages by round-robin and runs for one epoch with at most 1000 steps; the final transformer layer uses the raw model-head projection. The passage compares raw logit-lens projections with tuned lenses (Belrose et al., 2025) fit on FineWeb2 (Penedo et al., 2025).Source: C Training Details · p. 14

D Additional Results

• Appendix plots track layerwise START(w) probabilities for cloze prompts and translation prompts grouped by target or source, including English-excluded translation cases.The plots also report latent-language distribution and first-token vocabulary entropy.Source: Figure 8 · p. 15, Figure 9 · p. 15, Figure 10 · p. 16
• Layer-window summaries aggregate latent-language probabilities for copy, cloze, and translation tasks, with English prompt cases excluded as specified for each task.Translation’s task-relevant category includes both source and target languages; English remains a candidate latent language in the English-excluded translation analysis.Source: Figure 12 · p. 17, Figure 13 · p. 17, Figure 14 · p. 17
• Open-ended PUD21+UD6 results summarize latent-language probabilities by estimator and layer window, treating the prompt language as task-relevant.English prompts are excluded, while English remains a candidate latent language.Source: Figure 15 · p. 18
• Latent-language dominance and pivoting curves largely cluster across PUD inventories and INCLUDE domains, while layerwise trajectories differ substantially across models.The comparison covers PUD9, PUD9+UD6, PUD21, PUD21+UD6, and INCLUDE Arts/Humanities, Social Sciences, and STEM.Source: Figure 16 · p. 19
• Training analyses show quartile-binned dynamics and full-layer checkpoint trajectories for representation-based and raw-logit-lens decoding probes on PUD21.Checkpoint curves are colored by training-token count; the full-layer views average metrics over prompts at each layer.Source: Figure 17 · p. 19, Figure 18 · p. 20, Figure 19 · p. 20
• Base-versus-instruction-tuned comparisons plot pivot rate, entropy, and probe agreement on PUD21 and INCLUDE-10.Agreement is the fraction of matched prompt–layer pairs where representation and raw logit-lens probes select the same dominant language; INCLUDE-10 has no English prompts but retains English as a candidate.Source: Figure 20 · p. 21, Figure 21 · p. 21