PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias

Yuemei Xu, Ling Hu, Jiayi Zhao, Zihan Qiu, Kexin XU, Yuqi Ye, Hanwen Gu

arXiv:2404.00929v3cs.CLcs.AI

TL;DR

MLLMs seek to extend multilingual NLP and cross-lingual transfer, yet language imbalance, alignment, and bias remain challenges. This survey reviews MLLM capacities, corpora, representations, and bias, finding that expanded language coverage coexists with English-dominant training corpora.

  • Problem

    MLLMs face language imbalance, multilingual misalignment, and bias, complicating efforts to support cross-lingual capabilities across languages.

    Source: §1 · p. 2

  • Method

    The survey reviews MLLM capacities and corpora, multilingual representations, and bias and its evaluation.

    Source: §1 · pp. 2–3, §5.2 · p. 26

  • Results

    MLLMs broaden language coverage, yet English remains dominant in training corpora and Indo-European languages are prominent in quantity and variety.

    Source: §3.1 · p. 14

  • Takeaways & Limitations

    Multilingual bias benchmarks should consider cultural contexts because translating English benchmarks may introduce translation errors and cultural biases.

    Source: §5.2 · p. 26

    Debiasing can reduce downstream accuracy or fluency, making fairness-performance trade-offs especially challenging in low-resource languages with scarce data.

    Source: §5.3.3 · p. 29

Abstract

from arXiv · show

Based on the foundation of Large Language Models (LLMs), Multilingual LLMs (MLLMs) have been developed to address the challenges faced in multilingual natural language processing, hoping to achieve knowledge transfer from high-resource languages to low-resource languages. However, significant limitations and challenges still exist, such as language imbalance, multilingual alignment, and inherent bias. In this paper, we aim to provide a comprehensive analysis of MLLMs, delving deeply into discussions surrounding these critical issues. First of all, we start by presenting an overview of MLLMs, covering their evolutions, key techniques, and multilingual capacities. Secondly, we explore the multilingual training corpora of MLLMs and the multilingual datasets oriented for downstream tasks that are crucial to enhance the cross-lingual capability of MLLMs. Thirdly, we survey the state-of-the-art studies of multilingual representations and investigate whether the current MLLMs can learn a universal language representation. Fourthly, we discuss bias on MLLMs, including its categories, evaluation metrics, and debiasing techniques. Finally, we discuss existing challenges and point out promising research directions of MLLMs.

1 Introduction

MLLMs aim to support multilingual NLP and transfer from high-resource to low-resource languages, but language imbalance, misalignment, and bias remain challenges. This survey reviews MLLM capacities, corpora, multilingual representations, and bias.

1 Introduction

  • Training corpora shape MLLM capabilities, but uneven language resources and annotated datasets limit transfer to unseen languages.The survey analyzes corpus language distributions, sources, and coverage.Source: §1 · pp. 2–3
  • The survey reviews static, contextual, and combined multilingual representations and examines factors affecting alignment, including mapping linearity and typological distance.It asks whether current MLLMs can learn a universal language representation.Source: §1 · p. 3
  • The survey examines MLLM capacities for low-resource languages and cross-lingual transfer alongside the challenge of language imbalance.It also introduces MLLMs and their evolution and techniques.Source: §1 · p. 3
  • The survey examines MLLM bias types, debiasing techniques, and evaluation datasets, including whether debiasing affects performance.Prior bias research cited here is largely specific to English-based models and may not generalize to other languages.Source: §1 · pp. 3–4

2 Overview of MLLMs

The section introduces MLLMs through their evolution, key techniques, and multilingual capacities.

2 Overview of MLLMs

  • The overview traces the evolution of MLLMs.Figure 2 presents release year, supported-language counts, and release relationships.Source: §2 · p. 4
  • The overview presents key techniques contributing to MLLM development.Source: §2 · p. 4
  • The overview analyzes the multilingual capacities of MLLMs.Source: §2 · p. 4

2.1 Evolution of MLLMs

The section traces MLLMs from language-specific models to multilingual systems, then describes trends in scale, language diversity, and modality.

2.1.1 Monolingual Evolution

  • Monolingual models advanced transformer architectures and pre-training, but remained language-dependent and feasible for only a few languages with necessary corpora.The section discusses BERT variants for French, Dutch, and Arabic [16], and GPT-series growth from hundreds of millions to 1.5 trillion parameters [22].Source: §2.1.1 · p. 5

2.1.2 Multilingual Evolution

  • mBERT applied BERT’s training procedure to multilingual Wikipedia covering 104 languages, and later models explored multilingual capabilities and limitations.Studies reported cross-lingual task performance without direct cross-lingual supervision such as parallel or comparable data.Source: §2.1.2 · p. 5
  • MLLM development has trended toward parameter growth, greater linguistic diversity, and multimodal unification.Models reached hundreds of billions or trillions of parameters; limited online text and difficulty obtaining high-quality data may slow further growth.Source: §2.1.2 · pp. 5–6
  • The section’s model roadmap arranges representative monolingual and multilingual models chronologically by release.Source: §2.1.2 · p. 6

2.2 Key Techniques of MLLMs

The section presents transformer architecture, pre-training, and reinforcement learning with human feedback as key techniques underlying MLLMs, explaining how each supports language processing, representation learning, or human-preference alignment.

2.2.1 Transformer Architecture

  • Transformers use encoder and decoder modules with self-attention; encoder-only, decoder-only, and hybrid designs suit different analysis and generation tasks.Encoders produce latent representations, while decoders attend to representations and generate sequences autoregressively.Source: §2.2.1 · p. 6

2.2.2 Pre-training Technique

  • Pre-training learns universal language representations from large unlabeled corpora, using LM, MLM, NSP, and DAE objectives before downstream adaptation.The section describes both pre-training followed by fine-tuning and pre-training followed by prompting.Source: §2.2.2 · pp. 6, 8

2.2.3 Reinforcement Learning with Human Feedback

  • RLHF aligns MLLM outputs with human values by optimizing for helpfulness, honesty, and harmlessness in response to potentially inaccurate or harmful generations.Human evaluators rank prompt-response pairs to train a reward model, whose signal guides reinforcement-learning fine-tuning.Source: §2.2.3 · pp. 8–9

2.3 Multilingual Capacities of MLLMs

The section describes multilingual-corpus imbalances and their challenges for low-resource languages, alongside evidence of cross-lingual transfer and approaches for improving it.

2.3.1 Challenges brought by Multilingual Corpora

  • Low-resource languages remain disadvantaged by limited annotated data, the curse of multilinguality, and pre-training corpora skewed toward English.Fine-tuning for low-resource languages requires substantial task-specific annotated data, while monolingual pre-training is presented as an alternative.Source: §2.3.1 · p. 9
  • Skewed pre-training distributions are identified as a main reason for low-resource performance gaps; proposed responses include sampling and vocabulary augmentation.Monolingual and bilingual Czech-Polish RoBERTa models outperformed XLM-R across various downstream tasks.Source: §2.3.1 · p. 10

2.3.2 Cross-lingual Transfer Learning brought by Multilingual Corpora

  • Studies report zero-shot and few-shot cross-lingual transfer without source-to-target translation, as well as strong rumor detection and multilingual reasoning in under-represented languages.Training on genetically related languages can yield performance comparable to training on larger amounts of unrelated data.Source: §2.3.2 · p. 11
  • AfriB-ERTa, trained on less than 1 GB of text from 11 African languages, demonstrated low-resource-only pre-training without high-resource transfer.Most of its training languages belong to the Bantu branch of the Niger-Congo family.Source: §2.3.2 · p. 11
  • Future work is framed around explaining differences in transfer ability across languages and investigating factors such as linguistic similarity, model architecture, and corpus size.The stated aim is to explore whether transfer can reduce reliance on annotated data while maintaining or enhancing performance in well-trained or unseen languages.Source: §2.3.2 · p. 11

3 Multilingual Corpora and Datasets

The survey finds that MLLM corpora cover more languages than LLM corpora, but English remains dominant and language-family representation is uneven. It also reviews multilingual datasets used for downstream tasks, including resources designed to expand low-resource language coverage.

3.1 Multilingual Corpora in MLLMs

  • MLLM training corpora cover more languages than LLM corpora, yet English remains predominant and exceeds 90% in several models’ corpora.BLOOM has English below half of its training data, YAYI 2 approaches a 1:1 English-to-Chinese ratio, and PaLM 2 has a higher non-English share than PaLM.Source: §3 · p. 11, §3.1 · p. 13
  • French, German, Chinese, and Spanish are prevalent, while Indo-European languages dominate in both corpus quantity and linguistic diversity.Chinese is a prominent Sino-Tibetan language, but the family is less diverse than Indo-European in these corpora.Source: §3.1 · p. 13
  • Language-family coverage is also incomplete: some families contribute mainly one language, while Niger-Congo and Trans-New Guinea are absent from the top 20 training languages.The analysis excludes English and focuses on the top 20 non-English languages; Gopher and FuxiTranyu disclose only their top 10 and top 13 languages, respectively, and models without published proportions are omitted.Source: §3.1 · p. 14, Figure 4 · p. 14
  • The survey recommends more comprehensive family inclusion and greater within-family linguistic diversity when training MLLMs.The training-corpus comparison is summarized in Table 3 by model, language, language proportion, and source.Source: Table 3 · p. 12, §3.1 · p. 14

3.2 Multilingual Datasets for Downstream Tasks

  • Downstream datasets cover NER, sentiment analysis, cross-lingual information retrieval, and text classification, with resources such as Masakha NER2.0 and African-language sentiment datasets.MOT supplies unlabeled text for lower-resource languages, while NollySenti and NaijaSenti target African languages.Source: §3.2 · pp. 14–16
  • Dataset resources vary in scope: AfriCLIR Matrix addresses the lack of African languages in CLIR Matrix, while Taxi1500 covers over 1,500 languages using Bible translations and is limited to religious text classification.MARC supports product and sentiment classification; MUSE provides aligned embeddings and bilingual dictionaries, and Wikipedia Monolingual Corpora cover 30 languages.Source: §3.2 · p. 16

4 Multilingual Representation Alignment

Multilingual alignment performance varies across languages and tasks, with stronger performance for English and closely related languages. The survey reviews static, contextual, and combined alignment approaches, along with factors affecting their performance and remaining challenges.

4 Multilingual Representation Alignment

  • MLLM alignment performance varies across languages on BLI, XNLI, and MT, with better results for English and closely related languages.Table 5 demonstrates performance on 10 languages and these three cross-lingual tasks; it reports accuracy for BLI and XNLI and BLEU for MT.Source: §4 · p. 16, Table 5 · p. 17

4.1 Static multilingual representation

  • Static alignment maps monolingual embedding spaces, typically through an initial mapping followed by pseudo-supervised refinement with an orthogonal transformation.The orthogonal constraint preserves monolingual invariance but is not suitable for all languages, particularly semantically distant ones; global mappings may also perform unevenly across subspaces.Source: §4.1 · pp. 17–19
  • Static representations still need improvement for low-resource languages and distant pairs, and their polysemy problem remains insufficiently addressed.The survey notes that weak orthogonal constraints have been proposed for better cross-language alignment.Source: §4.1 · pp. 18–19

4.2 Contextual multilingual representation

  • Contextual alignment uses either mappings from pretrained monolingual embeddings or end-to-end joint approaches relying on unlabeled multilingual corpora and MLLMs.Subword tokenizers such as BPE and WordPiece address out-of-vocabulary issues, while contextual methods carry higher computational costs and challenges in interpretation and low-resource, distant-pair alignment.Source: §4.2 · p. 19

4.3 Combined multilingual language representation

  • Combined methods follow static-to-contextual or contextual-to-static paradigms, using one representation type to improve alignment in the other.The survey describes C2S techniques that refine static embeddings with contextual information, including a spring network and extracted contextual embeddings.Source: §4.3 · p. 20

4.4 Factors that affect alignments

  • Alignment depends on the initial solution: supervised methods rely on seed-dictionary quality and quantity, whereas unsupervised methods depend on distribution matching.The survey also discusses nonlinear mappings and relative isomorphism as responses to the limits of orthogonal mapping for semantically distant languages.Source: §4.4 · p. 20
  • BLI accuracy is under 40% for semantically distant pairs and over 80% for similar pairs; auxiliary languages can help bridge distant pairs.Cross-lingual transfer is also associated with pretraining data size, quality, balance, diversity, parameter scale, learning objective, and input window size.Source: §4.4 · pp. 20–21

5 Bias on Multilingual LLMs

The section surveys bias in multilingual LLMs, from its categories and evaluation to model- and data-based mitigation. It highlights both the promise of debiasing and limits involving training data, evaluation coverage, and fairness–performance trade-offs.

5 Bias on Multilingual LLMs

  • Bias can constrain fairness and real-world deployment, while prior literature focused mainly on English stereotypes or a narrow set of attributes.The section frames MLLM bias as a broader issue requiring attention to its forms, evaluation, and mitigation.Source: §5 · p. 21

5.1 Bias Category

  • The survey groups MLLM bias into language, demographic, and evaluation bias, arising from sources including training data, model design, and multilingual embeddings.Evaluation metrics can themselves be biased; a reviewed study found PLM-based metrics more biased than traditional metrics across six sensitive attributes, with modeling paradigm affecting fairness more than the PLM choice.Source: §5.1 · pp. 21, 24

5.2 Bias Benchmark

  • Bias benchmarks mainly cover gender and occupation and are predominantly English, while translation can introduce bias through errors or cultural differences.The survey notes some Spanish, German, and French resources and calls for culturally diverse multilingual benchmarks.Source: §5.2 · pp. 24, 26, Table 7 · p. 25

5.3 Debias Technique

  • Debiasing is divided into model interventions that refine model settings and data interventions that address bias in training inputs.The survey presents the two approaches as complementary, with distinct effectiveness and challenges.Source: §5.3 · p. 26, §5.3.3 · p. 29

5.3.1 Model Debiasing

  • Model debiasing spans representation, pretraining, fine-tuning, and prompt methods; prompt success depends on prompt choice, and fine-tuning may inherit pretraining bias.A cited LoRA fine-tuning approach is reported to produce robust debiased models while maintaining high downstream-task performance; pretraining-stage bias may resurface after downstream fine-tuning.Source: §5.3.1 · pp. 26–27

5.3.2 Data Debiasing

  • Data debiasing targets language and demographic bias through more balanced corpora and counterfactual augmentation, but low-resource data can be costly to collect and poor in quality.One reviewed finding reports that a large part of some low-resource corpora contained less than 50% acceptable-quality sentences; counterfactual augmentation may also omit context or reinforce false correlations.Source: §5.3.2 · pp. 28–29

5.3.3 Effectiveness and Challenges

  • Model debiasing can reduce stereotypes without changing training data, but its effects may be limited when foundational training-data biases remain unaddressed.The survey also identifies context dependence and limited interpretability as challenges for debiasing methods.Source: §5.3.3 · pp. 29–30
  • Debiasing shows promise for improving fairness, but techniques can trade bias reduction against downstream accuracy or fluency, especially in low-resource languages.Model debiasing can reduce stereotypes while maintaining competitive task performance, whereas data debiasing is presented as addressing systemic imbalance across languages and demographic groups.Source: §5.3.3 · p. 29

6 Future Directions

The survey identifies unresolved challenges in multilingual LLM performance, data, evaluation, and ethics, alongside opportunities in specialization and multimodal training. It calls for better low-resource support and more suitable multilingual data and evaluation.

Performance on Low-resource Languages

  • MLLM performance remains unsatisfactory on low-resource languages; the survey points to limited annotated data and low lexical overlap, and suggests language-family specialization and better tokenizers for investigation.The survey contrasts this with MLLMs outperforming monolingual LLMs on high-resource-language downstream tasks.Source: §6 · p. 30

Limited and Unbalanced Multilingual Corpora

  • Multilingual corpora remain limited, imbalanced, and sometimes low quality, motivating research into larger, more diverse, high-quality training data.The survey identifies English-heavy corpora as contributing to English-centric ability.Source: §6 · p. 30

Usage of Multimodal Data Sources

  • Speech and image data could reduce reliance on text and provide more diverse training data, but accurate modality alignment remains an open challenge.The survey links multimodal data use with better alignment to human intentions.Source: §6 · p. 30

Evaluation of Multilingual LLMs

  • Evaluation remains English-task-centric: translated sets may perform worse than source-language sets, and the survey calls for high-quality multilingual datasets and a flexible evaluation system.It says current benchmarks are not fully applicable to other languages and lack a universal, flexible system.Source: §6 · pp. 30–31

Ethical Impact of Multilingual LLMs

  • MLLMs can inherit training-data biases, and Western-language-heavy data can produce Western-centric concepts, making fairness and cultural sensitivity ongoing challenges.The survey identifies bias mitigation in text generation as a key future challenge.Source: §6 · p. 31
Loading 2404.00929v3…