Source-linked AI summary
Beyond English-Centric LLMs: What Language Do Multilingual Language Models Think in?
Chengzhi Zhong, Fei Cheng, Qianying Liu, Junfeng Jiang, Zhen Wan, Chenhui Chu, Yugo Murawaki, Sadao Kurohashi
TL;DR
The paper asks whether non-English-centric LLMs internally represent processing in their dominant training languages. Using logit-lens analysis across Llama-2, Swallow, and LLM-jp, it finds that latent-language distributions depend on target-language similarity and shift toward the target output while preserving semantics.
Problem
The study addresses the limited evidence about which internal latent languages non-English-centric LLMs use when processing languages beyond English.
Method
The authors apply logit-lens analysis to intermediate representations of Llama-2, Swallow, and LLM-jp across Japanese and non-dominant-language tasks.
Results
Swallow and LLM-jp use Japanese and English as dual latent languages, preferentially activating the one more closely related to the target language, while later layers produce the target language.
Takeaways & Limitations
The findings link latent-language behavior to training-corpus composition and show that language identity transitions occur in sparse dimensions related to language identity.
Takeaways & Limitations
The study is limited to Japanese-related models and includes an unresolved Swallow French–Chinese translation exception whose explanation is difficult to verify.
Abstract
from arXiv · showhide
In this study, we investigate whether non-English-centric LLMs, despite their strong performance, `think' in their respective dominant language: more precisely, `think' refers to how the representations of intermediate layers, when un-embedded into the vocabulary space, exhibit higher probabilities for certain dominant languages during generation. We term such languages as internal $\textbf{latent languages}$. We examine the latent language of three typical categories of models for Japanese processing: Llama2, an English-centric model; Swallow, an English-centric model with continued pre-training in Japanese; and LLM-jp, a model pre-trained on balanced English and Japanese corpora. Our empirical findings reveal that, unlike Llama2 which relies exclusively on English as the internal latent language, Japanese-specific Swallow and LLM-jp employ both Japanese and English, exhibiting dual internal latent languages. For any given target language, the model preferentially activates the latent language most closely related to it. In addition, we explore how intermediate layers respond to questions involving cultural conflicts between latent internal and target output languages. We further explore how the language identity shifts across layers while keeping consistent semantic meaning reflected in the intermediate layer representations. This study deepens the understanding of non-English-centric large language models, highlighting the intricate dynamics of language representation within their intermediate layers.
1 Introduction
The study examines internal latent languages in Japanese-related LLMs using intermediate-layer vocabulary predictions. Japanese-specific models use Japanese and English internally, with target-language similarity shaping the latent distribution and language identity shifting in sparse dimensions.
- Llama-2 uses English as a pivot, Swallow mixes English and Japanese, and LLM-jp primarily uses Japanese in intermediate layers.
- For underrepresented target languages such as French and Chinese, Swallow and LLM-jp distribute internal latent-language probability over English and Japanese according to target-language similarity before transforming into the target output language.
- Culturally conflicting questions initially elicit responses biased toward the internal latent language’s culture, with later layers shifting toward the target language.
- The transition in latent-language distributions occurs in sparse dimensions that are highly relevant to language identity.
- Japanese-specific Swallow and LLM-jp use Japanese as an internal latent language when processing Japanese, extending analysis beyond English-centric Llama-2.
3. We observe that the shift from internal latent
The shift from an internal latent language to the target language preserves intermediate-layer semantics while changing language identity in sparse, language-relevant dimensions.
- Language identity changes occur only in sparse dimensions, while the intermediate representations retain consistent semantic meaning.
2 Related work
Prior work frames the study through multilingual modeling and mechanistic interpretability, while this paper extends latent-language analysis across Japanese-related model categories and target languages.
- English-centric corpora motivate continued pre-training and non-English-dominant pre-training as approaches for improving performance in respective languages.
- Mechanistic interpretability analyzes model components and processes, including logit and tuned lenses that decode vocabulary distributions from intermediate vectors.
- Prior work found that Llama-2’s abstract concept space is closer to English, with intermediate predictions shifting from English answers toward target languages during non-English translation.
- This work extends previous tools to study latent-language distributions across Japanese-related LLM categories and their association with target languages.
3 Method
The method compares three multilingual model types by using logit lens projections and a multi-token extension to estimate internal language probabilities across layers. It evaluates these models on multilingual prompts spanning translation, repetition, and cloze tasks.
- The study compares English-centric, multilingual continued-pretraining, and balanced multilingual models to examine how training corpora influence latent-language probabilities and multilingual performance.
- 3.2 Logit Lens: Logit lens unembeds intermediate hidden vectors into vocabulary space, applies softmax, and estimates predicted token distributions at each layer.
- 3.2 Logit Lens: The multi-token extension decomposes phrases into token sequences and iteratively feeds predicted tokens back into the model to estimate the probability of generating the full sequence.
- 3.4 Dataset Construction: The dataset contains parallel English, French, Japanese, and Chinese phrases, with translations and descriptions supporting translation, repetition, and cloze prompts.
- 3.5 Prompt Design: Few-shot prompts provide task examples and omit the final answer for prediction, while repetition and cloze tasks test repeated-language generation and masked-word recovery.
4 Experiment Settings
The experiments use matched 13B models with the same layer count and embedding dimension, alongside a multilingual phrase dataset for controlled evaluation. All models are run with 8-bit quantization.
- Llama-2, Swallow, and LLM-jp-v2.0 are compared at 13B parameters, with all three models having 40 layers and embedding dimension 5120.
- The models use different vocabulary sizes: 32,000 tokens for Llama-2, 43,176 for Swallow, and 96,867 for LLM-jp-v2.0.
- The dataset contains parallel phrases in English, French, Japanese, and Chinese with corresponding descriptions for translation, repetition, and cloze tasks, and the experiments use 8-bit quantization.
5 Results
Across Japanese and non-dominant-language tasks, the models use different internal latent languages, with target-language-related patterns and culturally influenced intermediate predictions. Later layers convert latent representations toward the target language while preserving semantic content.
- 5.1 Main Experiment 1: Analysis on Specific Dominant Language – Japanese: Llama-2 uses English as its internal pivot for Japanese and non-dominant-language tasks, whereas Swallow uses both English and Japanese.For Japanese processing, Swallow shows noticeable Japanese probability; for non-dominant languages, its Japanese probability remains below English.
- 5.2 Main Experiment 2: Analysis on non-Dominant Languages: LLM-jp predominantly uses Japanese for Japanese repetition and cloze tasks, with latent-language probabilities varying by target language.For Chinese targets, Japanese probability is higher than English; for French targets, English probability is higher than Japanese.
- 5.2 Main Experiment 2: Analysis on non-Dominant Languages: Models tend to activate the internal latent language more closely related to the target language, with one Swallow translation result as an unresolved exception.The exception reverses the expected Japanese-probability pattern between French and Chinese targets, possibly because of specific French-Chinese content in Swallow’s training corpus.
- 5.3 How Is Culture Conflict QA Solved?: Cultural conflicts can make intermediate layers favor knowledge associated with the internal latent language before later layers produce the target-language answer.For Japan’s school-year question, Llama-2 initially favors September, while the correct April answer appears in later layers; Swallow shows the wrong answer only once, and LLM-jp does not exhibit the issue.
- 5.3 How Is Culture Conflict QA Solved?: Averaged hidden-vector shifts can approximate the transition from a latent-language representation to the target-language output while retaining semantic content.Using synonymous English-Japanese word pairs, adding the average difference between later and intermediate hidden vectors changes top tokens from English to Japanese and approximates the later-layer output.
6 Conclusion and Future Works
The study finds that training-corpus languages largely determine internal latent languages, with Japanese-focused models using Japanese alongside English and favoring languages related to the target. It also identifies culturally biased intermediate reasoning during transitions and limits its future scope to additional dominant-language models.
- Conclusion: Japanese CPT Swallow and bilingual-from-scratch LLM-jp use Japanese as an internal latent language when processing Japanese input, while non-dominant French and Chinese inputs elicit two latent languages.For Swallow, English remains more probable than Japanese; LLM-jp shows a stronger preference for one language.
- Conclusion: For French and Chinese, the latent language more closely related to the target language receives higher probability.
- Conclusion: Transitions from an internal latent language to the target language produce culturally biased intermediate reasoning steps.The study also finds that latent-language distribution shifts occur only in dense dimensions closely related to language identities.
- Conclusion: The findings support explanations of why models perform well in their dominant language and suggest directions for improving non-English-centric LLMs.
- Future Works: Future work will extend the analysis to models with Chinese, French, and Arabic as specific dominant languages.
A Appendix
The appendix provides references for the paper’s model, interpretability, and multilingual-language context, alongside visual summaries of language probabilities in repetition and translation tasks.
- Appendix: The cited literature covers multilingual and language-specific language models, including Llama 2, Jais, Qwen2, and research on continual language learning.
- Appendix: The appendix also references methods and studies on latent-language analysis, logit-lens interpretation, cross-lingual transfer, and multilingual datasets.
- Appendix: Figures 8 and 9 visualize language probabilities across layers for repetition and translation tasks, respectively, with 95% Gaussian confidence intervals over 166 input examples.Figure 8 covers French and Chinese repetition; Figure 9 covers French-to-Chinese and Chinese-to-French translation.