PaperlayerView exact revision on arXiv

Source-linked AI summary

Revealing the Parallel Multilingual Learning within Large Language Models

Yongyu Mu, Peinan Feng, Zhiquan Cao, Yuzhang Wu, Bei Li, Chenglong Wang, Tong Xiao, Kai Song, Tongran Liu, Chunliang Zhang, Jingbo Zhu

arXiv:2403.09073v3cs.CL

TL;DR

Prior work primarily focused on English as the pivot for multilingual language understanding and reasoning, leaving the simultaneous processing of more languages as a question to investigate. This paper supplies parallel multilingual input to LLMs and reports benchmark gains alongside inhibited, more selective neuron activation.

  • Problem

    Prior work primarily focused on English as the pivot for language understanding and reasoning, while how LLMs process more languages simultaneously remained to be investigated.

    Source: §1 · p. 1

  • Method

    PMI extends conventional ICL by providing LLMs with parallel input in multiple languages, translated by human translators or machine-translation systems.

    Source: §2.1 · p. 2

  • Results

    PMI improves performance across datasets and languages while inhibiting neurons; on FLORES-200 it improves over the baseline by 11.3 BLEU points and 1.52 COMET points.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The authors interpret PMI’s neuron inhibition and more precise activation as resembling synaptic pruning.

    Source: §1 · p. 2, §6 · p. 9

    Translation quality influences final performance, and the study leaves cross-language prompts such as code-switching for future work.

    Source: §7 · p. 9

Abstract

from arXiv · show

In this study, we reveal an in-context learning (ICL) capability of multilingual large language models (LLMs): by translating the input to several languages, we provide Parallel Input in Multiple Languages (PiM) to LLMs, which significantly enhances their comprehension abilities. To test this capability, we design extensive experiments encompassing 8 typical datasets, 7 languages and 8 state-of-the-art multilingual LLMs. Experimental results show that (1) incorporating more languages help PiM surpass the conventional ICL further; (2) even combining with the translations that are inferior to baseline performance can also help. Moreover, by examining the activated neurons in LLMs, we discover a counterintuitive but interesting phenomenon. Contrary to the common thought that PiM would activate more neurons than monolingual input to leverage knowledge learned from diverse languages, PiM actually inhibits neurons and promotes more precise neuron activation especially when more languages are added. This phenomenon aligns with the neuroscience insight about synaptic pruning, which removes less used neural connections, strengthens remainders, and then enhances brain intelligence.

1 Introduction

Prior multilingual ICL work primarily used English as a pivot; this study examines parallel multilingual input and reports benchmark gains alongside more selective neuron activation. Its findings also include benefits from adding translations that do not individually surpass baseline performance.

1 Introduction

  • Against prior work primarily using English as a pivot, the paper investigates how multiple languages are processed simultaneously in LLMs.The study examines parallel multilingual input in ICL and neuron activation.Source: §1 · p. 1
  • On FLORES-200, PMI improves over the baseline by 11.3 BLEU points and 1.52 COMET points.The approach transforms single-language input into multi-language input, using the contexts from all languages to make predictions.Source: §1 · pp. 1–2
  • As more languages enter the input, fewer neurons are activated; the authors link this pattern to synaptic pruning.The analogy describes strengthening some neural connections while eliminating others deemed redundant.Source: §1 · p. 2
  • Across 8 datasets, 7 languages, and 10 LLMs, PMI can benefit from adding languages even when some translations do not surpass baseline performance.The authors also describe high-quality machine translation as a way to acquire abundant parallel input for applying PMI to various tasks.Source: §1 · p. 2
  • The neuron analysis finds simultaneous performance improvement and inhibition, selective activation of commonly used neurons, and a similar effect from few-shot learning.Integrating few-shot learning with PMI enhances this neural reaction, which the authors report across models and tasks.Source: §1 · p. 2

2 Parallel Multilingual Input

Section 2 defines PMI as prompting with translated parallel inputs and tests whether multilingual input and information-source diversity contribute to translation gains. Its ablations show multilingual input outperforming paraphrasing alone, while the full multi-source multilingual prompt achieves higher performance than either ablation.

2.1 LLMs benefit from PMI

  • PMI supplies an input alongside translations into multiple languages, with a neutral template intended to keep the model from distinguishing them.Prompt construction should consider language choice, translation quality, and display order; languages better understood by the model are preferably placed first and last.Source: §2.1 · p. 2
  • PMI achieves the best results among direct and pivot translation baselines, especially with more parallel languages; adding Russian can help even when its translation is worse than direct translation.When English is the original input, PMI yields a small performance increase.Source: Table 1 · p. 3, §2.1 · p. 3

2.2 Multiple Languages or Information Sources?

  • The ablation separates paraphrasing in one language (PMIPA), multiple sources in one language (PMIMS), and one source expressed across languages (PMIML).PMIGT combines multiple sources and multiple languages.Source: §2.2 · p. 3
  • PMIMS and PMIML usually improve performance, but neither matches PMIGT; PMIML far outperforms PMIPA, and GPT-4 benefits more from PMI than ChatGPT.The experiments used FLORES-200 translation directions and compared Qwen-14B, ChatGPT, and GPT-4.Source: §2.2 · p. 4, Table 2 · p. 4

3 PMI Can Help: From a View of Neuron Activation

The section investigates whether PMI’s performance gains reflect changes in neuron activation, extending activation-counting methods to modern LLM architectures. It finds that PMI inhibits neurons while concentrating activation among frequently used neurons, a pattern that accompanies improved translation.

3.1 Method of Counting Activated Neurons

  • The method counts positive activation outputs to estimate neuron activity in MLPs, extending the approach to GELU, SiLU, and gated variants.For gated variants, the authors treat XVup⊙Wdown^T as a whole, leaving their statistical method unchanged.Source: §3.1 · p. 5

3.2 Experiments and Results

  • Adding more parallel languages coincides with a lower proportion of activated neurons and better translation performance.The authors describe this as a consistent correlation between neuron inhibition and performance improvements.Source: Figure 4 · p. 5, §3.2 · p. 6
  • Compared with direct translation, monolingual prompts activate neurons while multilingual prompts inhibit them, although both can improve performance.PMIGT inhibits more neurons than PMIML, while PMIMS activates more neurons than PMIPA.Source: §3.2 · p. 6
  • PMI inhibits less-used neurons while increasing activation of the most commonly used neurons, resembling a one-off synaptic-pruning effect during inference.Compared with the baseline prompt, PMI activates the top 1% of commonly used neurons more and rarely used neurons less often; this is an inference-stage, short-term effect, not training.Source: Figure 5 · p. 5, §3.2 · p. 6

4 Wide Evaluation of PMI Without Human Translations

PMI is evaluated across translation, language-understanding, generation, and reasoning tasks using multilingual open-source and commercial LLMs. Results report advantages across tasks and settings, including with automatic translations and when English is included among parallel languages.

4.1 Tasks and Evaluation

  • The evaluation covers six tasks, including WMT translation, NLI, reading comprehension, text simplification, abstractive summarization, and mathematical reasoning.BoolQ, Wiki-auto, and XLSum use 1,000 test samples each; XNLI uses 3,000, with other tasks unchanged.Source: §4.1 · p. 6

4.2 Models

  • The study evaluates eight open-source multilingual LLMs with 7B–176B parameters and also tests ChatGPT and GPT-4.ALMA-13B is specially fine-tuned for machine translation; the other listed models are multilingual pretrained models.Source: §4.2 · p. 6

4.3 Baselines

  • Direct prompts use the original input, while pivot prompts use its translation; baselines use zero-shot settings except for one-shot ChatGPT.GPT-4 translates WMT and GSM8K inputs, while ChatGPT translates the other datasets for pivot prompts.Source: §4.3 · pp. 6–7

4.4 Results and Analyses

  • PMI outperforms baselines across six translation directions and is competitive across NLU and other NLG tasks.The results cover high- and low-resource source languages and tasks with different text lengths.Source: §4.4 · p. 7, Table 3 · p. 7, Table 4 · p. 8
  • On GSM8K, PMI brings GPT-model accuracy close to 96% and improves GPT-4 performance in zero-shot as well as five- and eight-shot settings.The authors attribute scarce-shot gains to more comprehensive task understanding.Source: §4.4 · p. 7, Table 6 · p. 8
  • GPT-4o+PMI surpasses GPT-4 and GPT-4o baselines, and PMI still improves performance when its parallel translations come from a weaker model.The reported GPT-4o results use the best performance among PMI-1, PMI-3, and PMI-6.Source: §4.4 · pp. 7–8
  • Even translations that underperform the direct prompt can improve comprehension when included in PMI; results also remain strong when English is the original or a parallel language.PMI with machine translations is reported to outperform pivot prompts, and it outperforms all pivot prompts when English is one of the parallel languages.Source: §4.4 · p. 8
  • Few-shot learning also inhibits neurons and promotes more precise activation, with PMI further enhancing that neural response when combined with few-shot examples.PMI also retains strong performance for English-source inputs and when English is chosen as a parallel language.Source: §4.4 · p. 8

5 Related Work

The paper situates PMI against prior multi-way machine translation, neuron-activation statistics, and cross-lingual ICL approaches. It distinguishes simultaneous parallel-language input from sequential multilingual prompting or English-pivot reasoning.

Multi-way Neural Machine Translation

  • Multi-way neural machine translation uses a source sentence together with translations in other languages, but often relies on scarce human translations at inference.The paper contrasts this constraint with PMI's use of automatic machine translations across a wider range of tasks.Source: §5 · p. 8

Statistics of Activated Neurons in Transformer Models

  • The paper extends neuron-activation statistics developed for vanilla transformers to more advanced transformer architectures.Prior methods count nonzero ReLU outputs, and sparse activation has been reported as ubiquitous.Source: §5 · pp. 8–9

Cross-lingual In-context Learning

  • Unlike earlier cross-lingual prompts that process languages in sequence or turn English into a pivot, this work studies learning from parallel multilingual input simultaneously.Prior work includes self-consistency across language-specific answers and English rephrasing followed by step-by-step analysis.Source: §5 · p. 9

6 Conclusions

The paper concludes that LLMs can learn from parallel multilingual input and that PMI is effective across evaluated settings. It links the performance pattern to inhibited, more precise neuron activation and identifies multimodal tasks as future work.

6 Conclusions

  • Experiments across eight datasets, ten multilingual LLMs, and seven languages support PMI's effectiveness and applicability.The conclusion characterizes PMI's neuron pattern as resembling one-off synaptic pruning.Source: §6 · p. 9
  • Neuron statistics indicate that PMI improves performance while inhibiting neurons and promoting more precise activation.The authors compare this behavior to one-off synaptic pruning.Source: §6 · p. 9
  • Future work will explore PMI on multimodal tasks and examine neural activation in large multimodal models.Source: §6 · p. 9

7 Limitations

The paper identifies translation quality as a factor influencing PMI performance and leaves cross-language prompts such as code-switching for future work. Its prompt design avoids identifying the original input, and the appendix illustrates PMI construction strategies.

7 Limitations

  • Because LLMs use translation semantics during inference, translation quality influences PMI's final performance.The authors also leave cross-language prompts such as code-switching outside the paper's scope.Source: §7 · p. 9

A Design of Prompts

  • The prompts do not identify which input is original, and the table marks variable elements to replace for each task.Source: A Design of Prompts · pp. 12–13
  • The prompt illustration distinguishes human translations, monolingual paraphrases, translations from different experts, and translations from one translator.Source: Figure 8 · p. 13

B More Details About Statistical Method of Activated Neurons

The method counts neuron activations from MLP intermediate outputs corresponding only to generated tokens, avoiding distortion from differences in prompt length.

  • Activation counts use MLP intermediate results for output tokens, which directly contribute to generated results.Input-token activations are excluded because prompts can differ substantially in length and distort comparisons.Source: B More Details About Statistical… · p. 13
  • Table 9 records activation functions used by popular LLMs, and the extended statistical method applies to all models listed.Source: B More Details About Statistical… · p. 13

C Supplementary Results About Neuron Activation

PMI, few-shot learning, and their combination activate more of the most-used neurons while inhibiting less-used neurons. The supplementary analyses also compare activation patterns across layers, models, and fine-tuning settings.

C Supplementary Results About Neuron Activation

D.1 Preliminary Experiments of Constructing PMI

These supplementary experiments examine factors in constructing PMI and its performance across settings. They report gains over few-shot learning and describe trade-offs, task coverage, and the multilingual models evaluated.

D.1 Preliminary Experiments of Constructing PMI

  • PMI benefits more from parallel languages the model understands better and from higher-quality translations; better-understood languages at both sequence ends also yield superior results.The ordering experiment reports German at the beginning and Spanish at the end as the superior arrangement.Source: §D.1 · pp. 14–15, Figure 13 · p. 15

D.3 Effectiveness of PMI on more modern LLMs

D.2 Comparing the Performance Between Few-shot Learning and PMI

  • PMI outperforms few-shot learning in the comparison, whose few-shot results use officially released, non-fine-tuned models.Source: §D.2 · p. 16, Table 12 · p. 16

D.4 Self-augmentation

  • Qwen-14B's self-translated PMI still beats baselines, particularly on COMET, though its gains are smaller than those reported in Table 3.The authors attribute the smaller gains to lower translation quality than GPT-4's.Source: §D.4 · p. 16, Table 15 · p. 16

D.5 Inference Speed

  • Each additional parallel language costs approximately 30% more inference time and brings about 5% performance improvement; with three languages, improvement can reach 24.34%.Source: §D.5 · pp. 16–17, Table 13 · p. 16

E.1 Downstream tasks

  • The downstream tasks described include machine translation, natural language inference, reading comprehension, text simplification, and abstractive summarization.Source: §E.1 · p. 17

E.2 Multilingual LLMs

  • The evaluated multilingual-model roster includes LLaMA3, Bloomz, Qwen, ALMA, Yi, and mT0.Source: §E.2 · p. 17

E.3 Training Setups

The training and inference appendices describe adapting models that struggle with PMI, constructing fine-tuning data, and standardizing decoding settings.

E.3 Training Setups

  • Models confused by PMI prompts were fine-tuned using LLaMA-Factory and LoRA, with rank 8, alpha 32, and dropout 0.1.Source: §E.3 · p. 17

E.4 Details of the Fine-tuning Datasets

  • Fine-tuning data used training or development examples in zero-shot form for both baseline and PMI prompts, varying their ratio and dataset size.Source: §E.4 · p. 18, Table 17 · p. 18

E.5 Decoding Setups

  • Inference used batch size 4 and temperature 0.01 across LLMs to support reproducibility.Source: §E.5 · p. 18

F Full Experimental Results of Pivot Prompts

The section reports full pivot-prompt results, including the highest-scoring result from the main experiment, and gives prompts used across translation, simplification, summarization, and comprehension tasks.

F Full Experimental Results of Pivot Prompts

FLORES-200

  • FLORES-200 prompts ask for translation of six source-language sentences into one target-language sentence and for sentence simplification, including from four equivalent-language versions.The simplification prompt asks for one sentence that preserves core meaning and factual content.Source: FLORES-200 · p. 20

WikiAuto

  • WikiAuto prompts include NLI over premise–hypothesis pairs supplied in four languages, with entailment or not_entailment as the prediction labels.The prompt states that the sentences convey the same meaning across languages.Source: WikiAuto · p. 21
  • WikiAuto summarization prompts request a concise source-language summary of a long text or a summary of two semantically equivalent texts in different languages.Source: WikiAuto · p. 21

XLSum

  • XLSum prompts cover yes/no questions about a single passage and questions based on two passages in different languages that convey the same meaning.Source: XLSum · p. 22, BoolQ · p. 22

BoolQ

  • BoolQ prompts include yes/no questions, parallel-language passages, and mathematical problems expressed in multiple languages for English reasoning and answers.Source: BoolQ · p. 22
  • A BoolQ NLI prompt supplies a premise in four languages and a source-language hypothesis, asking for entailment, contradiction, or neutral.Source: BoolQ · p. 22
Loading 2403.09073v3…