Source-linked AI summary
Don't Trust ChatGPT when Your Question is not in English: A Study of Multilingual Abilities and Types of LLMs
Xiang Zhang, Senyu Li, Bradley Hauer, Ning Shi, Grzegorz Kondrak
TL;DR
The paper asks how multilingual LLM abilities should be characterized and why performance varies across languages. It introduces task categorizations and response back-translation to analyze these abilities, finding evidence that GPT exhibits subordinate, translating-like multilingualism with English as its native language.
Problem
It remains unclear how LLM multilingualism should be categorized and how language choice affects performance across tasks.
Method
The study categorizes language-dependent tasks and uses response back-translation to compare multilingual performance and infer multilingualism types.
Results
GPT shows higher performance in English, favors translation-equivariant tasks, and exhibits subordinate multilingualism with English functioning as its native language.
Takeaways & Limitations
GPT’s translating-like multilingualism can limit performance even on language-independent tasks.
Takeaways & Limitations
The study used GPT-3.5 rather than GPT-4 and conducted human evaluation only in English, Spanish, and Chinese.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have demonstrated exceptional natural language understanding abilities and have excelled in a variety of natural language processing (NLP)tasks in recent years. Despite the fact that most LLMs are trained predominantly in English, multiple studies have demonstrated their comparative performance in many other languages. However, fundamental questions persist regarding how LLMs acquire their multi-lingual abilities and how performance varies across different languages. These inquiries are crucial for the study of LLMs since users and researchers often come from diverse language backgrounds, potentially influencing their utilization and interpretation of LLMs' results. In this work, we propose a systematic way of qualifying the performance disparities of LLMs under multilingual settings. We investigate the phenomenon of across-language generalizations in LLMs, wherein insufficient multi-lingual training data leads to advanced multi-lingual capabilities. To accomplish this, we employ a novel back-translation-based prompting method. The results show that GPT exhibits highly translating-like behaviour in multilingual settings.
1 Introduction
The paper addresses limited understanding of multilingual LLM abilities by introducing a systematic analysis framework and response back-translation prompting. Experiments show GPT performs best in English, favors translation-preserving tasks, and exhibits mixed coordinate and subordinate bilingualism.
- Motivation: Multilingual abilities of LLMs have been less extensively studied than their English language capabilities.Prior work measured performance differences across languages but did not examine underlying mechanisms or their effects on results.
- Approach: The study categorizes language-dependent abilities into Reasoning, Knowledge Access, and Articulation according to the impact of language choice.Reasoning is described as least affected by language choice, while Articulation is most affected.
- Approach: Response back-translation (RBT) measures multilingual performance and identifies the type of multilingualism an LLM exhibits.The method compares generated answers after applying a novel prompting procedure across languages.
- Findings: GPT performs better when tasks are presented in English and on tasks whose correct outputs remain unchanged under translation.The reported findings also identify mixed coordinate and subordinate bilingualism in GPT.
- Contributions: The paper provides a quantitative and qualitative analysis of multilingual LLM abilities and introduces two task categorizations for this analysis.These are presented as the study’s main contributions.
2 Background
The paper uses bilingualism categories to frame an unresolved question about how LLMs represent and process multiple languages. It develops an experimental framework that relates multilingual task performance to compound, coordinate, and subordinate bilingualism.
- Bilingualism Typology: Bilingualism is categorized as compound, coordinate, or subordinate according to how lexical representations are organized across languages.The categories distinguish shared, separate, and dominant-language-linked representations.
- Bilingualism Typology: Compound bilingualism involves equally dominant languages with shared lexical representations and flexible switching between languages.It commonly emerges when two languages are learned simultaneously from birth.
- Bilingualism Typology: Coordinate bilingualism maintains separate lexical representations, producing differences when tasks are performed in different language settings.The passage describes separation between the lexicons of each learned language.
- Bilingualism Typology: Subordinate bilingualism links a single lexicon to a dominant language and uses translation when processing other languages.The typical sequence is translating the task into the dominant language, answering there, and translating the answer back.
- Open Question: It remains unclear whether LLMs use shared representations, separate representations, or translation processes across languages.The study develops an experimental framework to relate multilingual task performance to these categories.
3 Categorizing Language-Dependent Tasks
The paper introduces a two-dimensional framework for classifying NLP tasks by required linguistic knowledge and sensitivity to presentation language. It distinguishes Reasoning, Knowledge Access, and Articulation tasks, and further separates Translation Equivariant from Translation Variant tasks.
- Two-dimensional categorization: The framework classifies tasks along linguistic knowledge requirements and the impact of presentation language.These are the two dimensions used to analyze multilingual abilities.
- Three task categories: The three categories are Reasoning, Knowledge Access, and Articulation, ordered by increasing influence of language on performance.Reasoning is least language-sensitive, whereas Articulation is most language-dependent.
- Reasoning: Reasoning tasks rely on logic, mathematical symbols, or general experience and are expected to show consistent performance across languages.Examples include mathematical problem-solving, coding, and common sense reasoning.
- Knowledge Access: Knowledge Access tasks require retrieving and using stored knowledge, which may be less reliable when the task language differs from the language of learned knowledge.The underlying knowledge may not itself be language-dependent.
- Articulation: Articulation tasks depend heavily on linguistic pragmatics and cultural conventions, including summarization, dialogue generation, paraphrasing, and style writing.The paper illustrates this dependence by contrasting cover-letter conventions in English and Japanese.
- Translatability: Translation Equivariant tasks preserve the correct output when translation occurs before or after task solving, whereas Translation Variant tasks may change the correct output after translation.Reasoning and Knowledge Access tasks are generally treated as Translation Equivariant, while letter writing and pun understanding are representative Translation Variant tasks.
4 Methods
The paper uses Prompt Translation and Response Back-Translation to compare LLM performance and consistency across languages on Translation Equivariant and Translation Variant tasks. These comparisons are then used to infer whether an LLM exhibits compound, coordinate, or subordinate bilingualism.
- Prompt Translation: Prompt Translation creates multilingual task instances by translating monolingual data, prompting the LLM independently in each language, and evaluating the responses in that language.The method is motivated by the limited availability of multilingual datasets and assumes translation can generate parallel data for Translation Equivariant tasks with minimal information loss.
- Cross-language consistency: For Translation Equivariant tasks, matched responses across languages are expected when the LLM solves the task in a language-independent way.This expectation follows because the correct solution does not depend on the language used to express the problem.
- Response Back-Translation: Response Back-Translation translates a target-language response and its explanation back into the source language for comparison with the source-language explanation.The method is designed to expose similarities in the reasoning expressed across languages.
- Reasoning comparison: High similarity between source-language and back-translated explanations is intended to indicate a shared reasoning process across languages.The paper treats explanations as a partial observable of the model’s internal reasoning.
- Identifying multilingual types: Testing Translation Variant tasks helps distinguish coordinate from subordinate multilingual models because translation can alter the correct output.Coordinate models may reason differently across languages, whereas pure subordinate models are expected to reason mainly in a dominant language and produce relatively similar outputs.
5 Experiments
Experiments compare ChatGPT across languages on translation-equivariant reasoning and knowledge-access tasks, translation-variant pun tasks, and articulation through cover-letter generation. Results show stronger English performance, English-like outputs, and evidence of subordinate multilingualism.
- Translation-Equivariant Tasks: ChatGPT’s English performance is substantially higher on reasoning and knowledge-access tasks, with gaps exceeding 10% for math reasoning and 15% for commonsense reasoning.Knowledge-access accuracy is 16% lower in Japanese and 28% lower in Chinese, while European-language performance is closer to English.
- Translation-Equivariant Tasks: Explanation similarity is higher to English than to French for German, Spanish, Japanese, and Chinese outputs in commonsense reasoning.The reported averages are 0.849 versus 0.868, while French is more similar to English than German is: 0.871 versus 0.857.
- Articulation: Cover letters generated in other languages remain closer in written style to English than to their target languages.Chinese examples contain literal English translations, and fewer than 1% of letters use a proper Chinese sign-off.
- Translation-Variant Tasks: Pun-detection accuracy is near a random baseline, except for 0.55 when English puns are translated into Chinese.ChatGPT strongly favors positive classifications, explaining the isolated higher accuracy through more negative predictions.
- Translation-Variant Tasks: Pun-location accuracy is 96.5% for original English puns but drops sharply after translation into other languages.French-to-English performance also decreases, whereas Spanish-to-English translation slightly increases location accuracy.
- Multilingualism Type: A Chinese ambiguity test suggests that ChatGPT partly relies on English semantic representations when interpreting non-English prompts.It fails to distinguish two Chinese translations of English “bat,” and the authors interpret this as evidence of subordinate bilingualism.
- Multilingualism Type: Across experiments, GPT exhibits translating-like behavior consistent with subordinate multilingualism rather than a single shared multilingual representation.The authors describe a process that translates input into English, formulates an English response, and translates it back, with lower accuracy often resulting.
6 Discussion and Future Directions
The paper argues that current multilingual models fall short of compound multilingualism because multilingual training data provide limited cross-language alignment. It proposes parallel data and training methods that encourage shared representations across languages.
- Current multilingual models may fail to achieve compound multilingualism even with equal-quality and equal-quantity training data across languages.The authors attribute this possibility to limitations in data collection and training techniques.
- Most online articles and posts are monolingual, making aligned multilingual data difficult to acquire outside translation datasets.As a result, multilingual corpora provide limited information interchange across languages.
- Disproportionate representation of particular languages can make models predominantly subordinate, with minimal coordination from monolingual datasets.
- The authors propose highly parallel cross-language data and training methods such as contrastive learning to cultivate compound representations of universal concepts.They also identify ontology linkages as a possible way to construct parallel data.
7 Conclusion
The paper presents a systematic approach to analyzing multilingual abilities and reports evidence that GPT-3.5 exhibits subordinate multilingualism, with English functioning as its native language. The authors further report that this pattern can limit performance even on language-independent tasks.
- The study proposes a systematic approach to analyzing multilingual abilities in large language models.
- GPT-3.5 exhibits subordinate multilingualism, with English functioning as the model’s native language.
- Subordinate multilingualism can limit performance even in language-independent tasks.The conclusion is supported by experiments, specific examples, and case studies.
Limitations
The study is constrained by limited access to prompts and GPT-4, restricted human evaluation coverage, and difficulties evaluating Llama2 responses.
- Prompt limits prevented the experiment from being applied to the whole dataset, and the study used GPT-3.5 because GPT-4 access was restricted.
- Human evaluation covered only English, Spanish, and Chinese because fluent speakers of the other test languages were unavailable.
- Llama2 often produced meaningless or declined responses, usually answered non-English questions in English, and faced additional dataset-quality issues.These issues complicated answer extraction and evaluation.
A Appendix
The appendix contains four tables documenting multilingual explanation similarity, cover-letter comparisons, pun explanations, and an example of a misguided Llama2 response.
- The appendix contains four tables covering explanation similarity, cover letters, multilingual pun explanations, and a Llama2 response.
- Table 4 reports BERT similarity scores for explanations generated by ChatGPT in different languages on Translation Equivariant tasks.
- Table 5 compares ChatGPT cover letters generated with the same prompt in English and Chinese.
- Table 6 presents ChatGPT explanations of one English pun in English, Chinese, French, and Spanish.The described pun is lost in translation, but ChatGPT still detects, locates, and explains it across those languages.
- Table 7 shows a sample misguided response from Llama2 to a knowledge access question.