Source-linked AI summary
How do Large Language Models Handle Multilingualism?
Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, Lidong Bing
TL;DR
The study asks how LLMs process multilingual inputs internally, an issue left unclear despite their broad multilingual capabilities. It proposes MWork and PLND to characterize and test this processing, reporting improved multilingual abilities from targeted fine-tuning with a small corpus.
Problem
Despite strong multilingual capabilities, the internal mechanism by which LLMs process multiple languages remains largely unclear.
Method
The study proposes MWork and uses PLND to identify language-specific neurons without labeled data, then tests the workflow through neuron deactivation and targeted fine-tuning.
Results
3.6% and 2.3% average improvements are achieved for high-resource and low-resource languages, respectively, across tasks with 400 documents.
Takeaways & Limitations
Fine-tuning language-specific neurons can enhance a particular language without compromising performance in other languages.
Abstract
from arXiv · showhide
Large language models (LLMs) have demonstrated impressive capabilities across diverse languages. This study explores how LLMs handle multilingualism. Based on observed language ratio shifts among layers and the relationships between network structures and certain capabilities, we hypothesize the LLM's multilingual workflow ($\texttt{MWork}$): LLMs initially understand the query, converting multilingual inputs into English for task-solving. In the intermediate layers, they employ English for thinking and incorporate multilingual knowledge with self-attention and feed-forward structures, respectively. In the final layers, LLMs generate responses aligned with the original language of the query. To verify $\texttt{MWork}$, we introduce Parallel Language-specific Neuron Detection ($\texttt{PLND}$) to identify activated neurons for inputs in different languages without any labeled data. Using $\texttt{PLND}$, we validate $\texttt{MWork}$ through extensive experiments involving the deactivation of language-specific neurons across various layers and structures. Moreover, $\texttt{MWork}$ allows fine-tuning of language-specific neurons with a small dataset, enhancing multilingual abilities in a specific language without compromising others. This approach results in an average improvement of $3.6\%$ for high-resource languages and $2.3\%$ for low-resource languages across all tasks with just $400$ documents.
1 Introduction
Although LLMs perform well across languages, their multilingual processing remains unclear. The study observes a shift from non-English to English-centric representations and back, motivating MWork and its PLND-based verification.
- LLMs show strong multilingual capabilities, but how they process multiple languages internally remains largely unclear.
- Prior interpretability studies link self-attention to reasoning and feed-forward layers to factual knowledge, but primarily examine English.
- Non-English queries produce non-English representations initially, become English-centric in middle layers, and return to non-English representations in final layers.
- MWork hypothesizes layered multilingual processing in which models convert multilingual queries to English for reasoning and generate responses in the original language.
- PLND identifies language-specific neurons without labeled data, enabling experiments that deactivate neurons across layers and structures to test MWork.
- 3.6% and 2.3% average improvements are reported for high-resource and low-resource languages, respectively, across tasks using a few hundred training documents.
2 Parallel Language-specific Neuron Detection (PLND)
PLND detects language-specific neurons by measuring how neuron deactivation changes layer representations, using parallelized calculations for attention and feed-forward structures. Experiments show that selectively disabling these neurons harms corresponding-language performance, with language overlap varying across models and language families.
- 2.1 Sequential Neuron Detection: PLND identifies language-specific neurons by measuring the hidden-representation change caused by deactivating each neuron for language-specific inputs.A neuron is treated as a row or column of a model parameter matrix, and language-specific neurons are those consistently important across a language corpus.
- 2.1 Sequential Neuron Detection: The method aggregates neuron importance across corpora and uses a predefined threshold to identify neurons consistently important for a specific language.Deactivation sets the selected neuron's parameters to zero before measuring its effect on the layer output.
- 2.2 Parallel Neuron Detection: PLND parallelizes neuron-importance calculations to reduce the cost of traversing all neurons and inputs sequentially.The parallel formulation uses diagonal masking for feed-forward neurons and analogous calculations for self-attention parameters.
- 2.3 Detection of Language-Specific Neurons: The experiments evaluate PLND on Vicuna and Mistral using multilingual XLSum summarization, selected languages, and language-specific corpora.The setup includes French, Chinese, Spanish, and Russian, with OSCAR used to compile language-specific corpora without task-specific considerations.
- 2.3 Detection of Language-Specific Neurons: Deactivating about 0.13% language-specific neurons eliminates meaningful multilingual capability, whereas disabling the same number of random neurons produces no difference.The comparison uses multilingual performance on XLSum under language-specific versus randomly selected neuron deactivation.
- 2.3 Detection of Language-Specific Neurons: Language-specific neuron overlap is limited between English and other languages but higher among related languages, while Chinese dominates reasoning and knowledge extraction in Chinese Llama.BLOOMZ shows patterns similar to Mistral and Vicuna, whereas Chinese Llama exhibits a distinct language pattern.
3 Multilingual Workflow (MWork) of LLMs
MWork hypothesizes that LLMs understand multilingual inputs, solve tasks in English while using multilingual knowledge, and generate responses in the query’s original language. Deactivation experiments across layers and structures support distinct roles for understanding, task-solving, knowledge access, and generation.
- 3.1 MWork: Hidden representations shift from non-English to English-centric and back to non-English across layers, motivating a three-stage multilingual workflow.The stages are understanding the original query, task-solving in English, and generating in the original language.
- 3.1 MWork: MWork proposes that LLMs unify diverse linguistic features during understanding, use English for reasoning, access multilingual knowledge through self-attention and feed-forward structures, and generate in the query’s original language.The hypothesis assigns multilingual knowledge use to intermediate task-solving structures and language alignment to final generation layers.
- 3.2 Verification Experiment Setup: Language-specific neuron deactivation is evaluated across understanding, task-solving, and generation settings using four multilingual tasks and six languages.The setup compares language-specific with randomly selected neurons while controlling the number of deactivated neurons.
- 3.3 Verify the Understanding Stage in MWork: Understanding-layer deactivation slightly affects English but significantly decreases non-English performance, supporting a role for early layers in multilingual understanding.The reported understanding experiment includes a 14% decrease in non-English XQuAD performance while English remains stable in the cited example.
- 3.4 Verify the Reasoning Structure in MWork: Task-solving-layer deactivation reduces performance in both English and non-English languages, while self-attention and feed-forward ablations distinguish reasoning from multilingual knowledge access.Reasoning performance is greatly reduced when language-specific self-attention neurons are retained while other language-specific neurons are disabled; feed-forward deactivation produces the highest reduction difference for knowledge question answering.
- 3.6 Verify the Generation Structure in MWork: Generation-layer deactivation diminishes generation in the respective languages, and disabling all language-specific neurons eliminates multilingual generation capabilities.Selective removal of non-English-associated neurons does not completely eliminate multilingual generation, whereas complete deactivation does.
4 Multilingual Enhancement with MWork
MWork enables targeted multilingual enhancement by fine-tuning language-specific neurons with only a few hundred documents, improving selected languages without harming others.
- Language-specific neurons comprise around 0.1% of parameters, substantially reducing the data needed for multilingual enhancement.Fine-tuning these neurons for a particular language does not impact performance in other languages.
- 3.4% on MGSM, 4.4% on XQuAD, 4.3% on X-CSQA, and 2.3% on XLSum were achieved with a few hundred documents for high-resource languages.English performance also improved by an average of 3.7% across tasks.
- Using 800 documents performed worse than using 400 documents, attributed to added knowledge disrupting the original distribution and causing Wikipedia overfitting.The paper suggests mixing textbooks or websites with Wikipedia data as a remedy.
- 2.2% average improvement was obtained for low-resource languages across four benchmarks by fine-tuning language-specific neurons.English performance improved by 3.5% without an English training corpus.
5 Related Work
Related work develops multilingual benchmarks and performance-improvement methods, while interpretability research examines input features, knowledge storage, and neural representations.
- Multilingual research has developed benchmarks and pursued improvements through translation, representation alignment, prompting, and cross-lingual transfer.Some approaches target multilingual performance without parameter adjustments, while others focus on single-task improvements.
- Interpretability studies investigate input-feature significance and treat feed-forward layers as sites for storing or modifying factual knowledge.Later work also maps embeddings to words and alters neuron values.
6 Conclusion
The paper proposes MWork as a multilingual processing workflow and uses PLND to identify language-specific neurons without labeled data, supporting targeted multilingual enhancement.
- MWork proposes converting multilingual inputs into English for understanding, reasoning in English while incorporating multilingual knowledge, and generating output in the query’s original language.The workflow assigns multilingual processing across different stages of the model.
- PLND identifies activated language-specific neurons for different languages without relying on labeled data.The method supports verification of the proposed workflow and subsequent neuron-specific fine-tuning.
- The corpus-selection criterion targets substantial vocabulary coverage so selected contexts represent each language.Corpus details report contexts, vocabulary coverage, and vocabulary size across languages.
C Interrelation of Language-Specific Neurons Across Languages
The study measures overlap among language-specific neurons across languages and structures, finding limited cross-language overlap with English and greater overlap among related languages.
- Neuron overlap is defined between language-specific neuron sets and measured separately across network structures.The reported overlap ratio compares detected neurons for pairs of languages in two models.
- English has relatively limited neuron intersection with other languages in both Mistral and Vicuna, suggesting many English-specific neurons.This comparison corresponds to the first row of each figure.
- Spanish, French, and English show a pronounced tendency toward greater language-specific neuron overlap as members of the same language family.
D Analysis on Different Multilingual LLMs
Across multilingual architectures, language handling differs in neuron sharing and internal processing. BLOOMZ shows greater language independence, while Chinese Llama relies predominantly on Chinese for reasoning and knowledge extraction.
- Bilingual LLMs: Chinese Llama is evaluated alongside BLOOMZ as a bilingual model focused on English and Chinese.Its training extends the vocabulary, adds Chinese pre-training, and uses Chinese instruction data.
- Hyper-Multilingual LLMs: BLOOMZ exhibits reduced neuron overlap across languages in both self-attention and feed-forward structures, indicating greater language independence.Individual languages do not extensively share neurons in either structure.
- Hyper-Multilingual LLMs: Figure 6 measures language-specific neuron overlap in BLOOMZ across self-attention and feed-forward structures.
- Bilingual LLMs: Figure 7 reports language ratios across Chinese Llama layers under non-English instructions.
- Bilingual LLMs: Chinese predominates as the primary language for reasoning and knowledge extraction across languages in Chinese Llama.
E Language-Agnostic Neurons
The experiments test whether neurons shared with English or across all languages are language-agnostic. Their deactivation produces multilingual performance declines while English performance remains comparatively stable, and the globally overlapping subset is extremely small.
- Language-Agnostic Neurons: English performance remains stable while multilingual performance declines significantly after deactivating language-specific neurons without English-overlapping neurons.Removing only overlapping neurons causes a less pronounced but still noteworthy drop.
- Language-Agnostic Neurons: Neurons overlapping across all languages account for only 0.02% of total neurons.
- Language-Agnostic Neurons: Deactivating all-language-overlapping neurons produces performance nearly identical to deactivating all language-specific neurons.This supports interpreting the globally overlapping neurons as a subset of language-specific rather than language-agnostic neurons.
- Experimental Setup: The evaluation uses translated zero-shot prompts and baseline performance across four representative multilingual tasks.The baseline table reports averages among non-English languages.
- Experimental Setup: On Vicuna and Mistral, XQuAD with Chinese evaluates deactivation in understanding and generation layers.The validation setting selects layer counts that least affect English while most reducing Chinese performance on Vicuna.
I Detailed Experiment Results
The detailed experiments compare random and language-specific neuron deactivation across task-relevant structures and layers for reasoning, knowledge question answering, and generation.
- Reasoning Task: Reasoning experiments deactivate random or language-specific neurons in attention, task-solving, understanding, generation, and other layer subsets.
- Knowledge Question Answering Task: Knowledge question answering experiments compare random deactivation with language-specific deactivation in task-solving layers, especially attention and feed-forward structures.
- Generation Task: Generation experiments compare random neuron deactivation in generating or all layers with language-specific deactivation in generating layers.
I.2 Detailed Result
The detailed results divide model layers into understanding, task-solving, and generation regions and use notation for deactivation in self-attention and feed-forward components across four tasks.
- Notation and Layer Partition: D(A)_i denotes deactivating neurons in the self-attention layer of the i-th layer, while D(F)_i denotes deactivation in its feed-forward layer.
- Notation and Layer Partition: The layer sets U, S, and G represent understanding, task-solving, and generation layers, respectively.U = {1, · · ·, N1}, S = {N1 + 1, · · ·, N2}, and G = {N2 + 1, · · ·, 32}.
- Detailed Results: The detailed tables report results for understanding, reasoning, knowledge question answering, and generation tasks.
- Notation and Layer Partition: Vicuna-7b-v1.5 and Mistral-7b-v1.0 each contain 32 layers.