PaperlayerView exact revision on arXiv

Source-linked AI summary

The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating It

Zheng-Xin Yong, Beyza Ermis, Marzieh Fadaee, Stephen H. Bach, Julia Kreutzer

arXiv:2505.24119v1cs.CL

TL;DR

LLM safety research remains heavily centered on English, with comparatively limited attention to non-English and multilingual contexts. This paper reviews nearly 300 *ACL publications and finds that Mandarin receives about ten times less research than English.

  • Problem

    LLM safety research has limited coverage of non-English and multilingual contexts despite global use across linguistic communities, making safety across those contexts a critical research imperative.

    Source: §1 · pp. 1, 3

  • Method

    The authors systematically reviewed nearly 300 LLM safety publications from 2020–2024 across *ACL venues, including conferences and workshops.

    Source: §2.1 · p. 4, §5 · p. 11

  • Results

    LLM safety research remains English-centered: Mandarin has about ten times less research than English, and non-English languages typically appear only in multilingual studies.

    Source: §1 · p. 3, §5 · p. 11

  • Takeaways & Limitations

    To help close the language gap, the authors identify culturally grounded evaluation, diverse multilingual safety training data, and cross-language alignment as future research directions.

    Source: §1 · p. 3

    The review focuses on *ACL venues, may miss relevant work from other venues or non-peer-reviewed research, and is a snapshot of a fast-moving field.

    Source: Limitations · p. 11

Abstract

from arXiv · show

This paper presents a comprehensive analysis of the linguistic diversity of LLM safety research, highlighting the English-centric nature of the field. Through a systematic review of nearly 300 publications from 2020--2024 across major NLP conferences and workshops at *ACL, we identify a significant and growing language gap in LLM safety research, with even high-resource non-English languages receiving minimal attention. We further observe that non-English languages are rarely studied as a standalone language and that English safety research exhibits poor language documentation practice. To motivate future research into multilingual safety, we make several recommendations based on our survey, and we then pose three concrete future directions on safety evaluation, training data generation, and crosslingual safety generalization. Based on our survey and proposed directions, the field can develop more robust, inclusive AI safety practices for diverse global populations.

1 Introduction

The introduction argues that multilingual LLM safety remains underexplored despite global deployment, and presents a review documenting an English-centered research landscape. It motivates culturally grounded evaluation, diverse multilingual training data, and research into cross-language alignment.

1 Introduction

  • Safety evaluation datasets reviewed by Dong et al. [2024] all include English, while commercial models can produce harmful non-English outputs that English filters would block.Among 24 highly ranked models with system reports and wide multilingual support, only 5 reported multilingual safety alignment training and red-teaming efforts.Source: §1 · pp. 1–3
  • Language-specific vocabulary, grammar, cultural connotations, metaphors, taboos, and social norms mean safety cannot be addressed by simply translating existing techniques.Content can be harmful in one cultural context but harmless in another, or vice versa.Source: §1 · pp. 1–2
  • The review of nearly 300 *ACL safety publications finds English-centered research, a widening language gap, about ten times less research on Mandarin than English, few standalone non-English studies, and coverage limits documented in only half of English studies.The reported imbalance spans multiple safety subdomains and has become more pronounced over time.Source: Figure 1 · p. 3, §1 · p. 3
  • The paper proposes culturally grounded evaluation benchmarks, diverse multilingual safety training data, and deeper study of alignment challenges across languages.These are presented as three tractable directions for addressing the gaps identified in the review.Source: §1 · p. 3

2 The Language Gap in LLM Safety Research

A survey of ∗ACL LLM safety research finds persistent English dominance, limited language-specific study of non-English languages, and gaps in language documentation. It recommends making language coverage more visible and supporting multilingual safety research through venue initiatives.

2.1 Methodology

  • Pairwise inter-annotator agreement averaged 0.80–0.96 across categories in a 4 × 20 agreement study, though the authors note annotations may still be imperfect.Language coverage was measured with Jaccard similarity and other categories with Cohen’s κ.Source: Table 2 · p. 4, §2.1 · p. 4

2.2 Findings

  • English is studied nearly ten times as often as Chinese and mostly in isolation, while non-English languages commonly appear in broad multilingual studies rather than in depth.This pattern also affects widely spoken Chinese, Arabic, and Spanish, and is more pronounced for under-resourced languages.Source: Figure 2 · p. 5, §2.2 · p. 5
  • English-only publications outnumber multilingual and non-English work, with the absolute gap widening from 5 in 2020 to 83 in 2024.Although both categories grew, growth was disproportionately concentrated in English-only research.Source: §2.2 · pp. 4–5
  • English-only publications substantially outnumber multilingual work across every safety subtopic, especially in alignment and jailbreaking, with multilingual work nearly absent in privacy and policy.The paper also identifies disparities in toxicity and bias research.Source: Figure 3 · p. 6, §2.2 · p. 6
  • Monolingual non-English safety papers are 46% relatively more likely than English-only papers to appear in workshops.The authors suggest workshops offer a more accessible route for disseminating this research than prestigious conferences.Source: Figure 3 · p. 6, §2.2 · p. 6
  • Language documentation is explicit in 100% of monolingual non-English and multilingual publications, but 50.6% of English-only publications do not name the language studied.The authors argue that language documentation matters because safety alignment may not generalize across languages and explicit coverage enables tracking of progress.Source: §2.2 · pp. 6–7, Table 3 · p. 7

2.3 Moving Forward for ∗ACL Venues

  • The authors recommend making existing ∗ACL language-coverage metadata public and using dedicated multilingual safety tracks or shared benchmark tasks to expand research beyond English.Public metadata could support transparent representation tracking and future meta-analyses.Source: §2.3 · p. 7

3 Future Research Directions for Multilingual LLM Safety

The section proposes research priorities for improving multilingual LLM safety through stronger evaluation, culturally contextualized training data, and investigation of crosslingual safety generalization.

3.1 Safety Evaluation for Multilingual Models

  • 85.56 average harmlessness makes ChatGPT the apparent winner, but its 62.6 worst-case trails PaLM-2's 70.1; Vicuna's worst-case is 18.4 in Bengali.PaLM-2 [Anil et al., 2023] had an 82.17 average; the contrast shows that average scores can mask language-specific safety failures, motivating worst-case reporting and culturally informed language-specific thresholds.Source: §3.1 · p. 8, Table 4 · p. 8
  • Safety evaluation should extend beyond languages used in fine-tuning and disclose languages lacking alignment, since pretraining contamination may enable transfer and omitted languages can expose guardrail failures.The Llama-3 report evaluated only eight languages, while the model was adapted for Indonesian; researchers are urged to detect potential contamination and warn users when alignment has not been conducted in a language.Source: §3.1 · pp. 8–9
  • Multilingual red-teaming should test real-world linguistic patterns, because code-switching and Arabizi inputs can jailbreak guardrails that monolingual, standardized-language tests may miss.The proposed direction is to systematically evaluate diverse multilingual, multi-turn interactions among users [Li et al., 2025].Source: §3.1 · p. 9

3.2 Culturally-Contextualized Synthetic Training Data

  • Constitutional AI could generate culturally contextualized preference pairs by having multilingual LLMs critique and revise harmful or harmless text using culture-specific principles.The approach calls for collaboration among linguists, cultural anthropologists, and AI researchers to develop principles, capable multilingual models, and validation protocols involving native speakers and cultural experts [Bai et al., 2022; Kundu et al., 2023; Kirk et al., 2024; Pistilli et al., 2025; Qin et al., 2024; Huang et al., 2024a; Kyrychenko et al., 2025].Source: §3.2 · p. 9
  • For synthetic safety data, machine translation may erase culture-specific harms or introduce biases, so constitutional-AI refinement and automated detection of lost cultural nuances warrant investigation.This direction can draw on cross-cultural adaptation research and is particularly relevant to languages with limited digital presence and linguistic resources [Savoldi et al., 2021; Ahn et al., 2022; Wang et al., 2022; Costa-jussà et al., 2023a;c; Maxwell et al., 1996; de Lima Barroso et al., 2018; Gorecki et al., 2014; Mbada et al., 2015; Pilz et al., 2014].Source: §3.2 · pp. 9–10

3.3 Towards Understanding Crosslingual Safety Generalization

  • Mechanistic interpretability could explain why detoxification and debiasing transfer across languages but refusal training may not, and whether alignment survives adaptation to underrepresented languages.Such insights may inform techniques for zero-shot crosslingual alignment generalization and safety consistency as language coverage expands [Li et al., 2024; Reusens et al., 2023; Shen et al., 2024; Aakanksha et al., 2024; Wang et al., 2025; Yong et al., 2023b; Lin et al., 2024; Ji et al., 2024].Source: §3.3 · p. 10
  • Influence functions could trace how safety examples from high- versus low-resource languages shape outputs and identify problematic documents in continued pretraining for targeted curation.This is an underexplored approach to multilingual safety-relevant training-example-to-output analysis [Grosse et al., 2023; Ruis et al., 2025].Source: §3.3 · p. 10

4 Related Work and Discussion

The paper distinguishes its LLM-safety focus from broader multilingual NLP surveys [Joshi et al., 2020; Yadav & Sitaram, 2022; Winata et al., 2023; Huang et al., 2024a; Qin et al., 2024; Wu et al., 2025], arguing that safety research may intensify sociolinguistic inequalities [Blasi et al., 2022]. Recent safety-challenge catalogs remain English-centered [Barez et al., 2025; Debar et al., 2024; Anwar et al., 2024], contributing to a siloed landscape consistent with “squareone bias”.

4 Related Work and Discussion

  • Unlike broader multilingual NLP surveys [Joshi et al., 2020; Yadav & Sitaram, 2022; Winata et al., 2023; Huang et al., 2024a; Qin et al., 2024; Wu et al., 2025], this work focuses on LLM safety and finds inequalities [Blasi et al., 2022] may be more pronounced there.The cited inequalities arise through choices in data collection, annotation protocols, and evaluation; cultural and linguistic nuances affect harm and mitigation strategies.Source: §4 · p. 10
  • English-centered catalogs of LLM safety challenges [Barez et al., 2025; Debar et al., 2024; Anwar et al., 2024] echo “squareone bias”, leaving multilingual safety underdeveloped in a fragmented research landscape.The cited bias describes research moving beyond usefulness while studying only one of safety, interpretability, or multilinguality, so progress across dimensions rarely informs the others.Source: §4 · p. 11

5 Conclusion

A review of nearly 300 publications from 2020–2024 finds a significant language gap in LLM safety research: even high-resource non-English languages receive minimal attention and less depth than English-focused work. The authors recommend changes for future conferences and identify research directions to address the imbalance.

5 Conclusion

  • Nearly 300 publications from 2020–2024 reveal a significant language gap in LLM safety research, with high-resource non-English languages receiving minimal attention.These languages typically appear only in multilingual studies that lack the depth of English-focused work.Source: §5 · p. 11
  • This imbalance potentially leaves language-specific risks undetected as LLMs are deployed globally.The authors recommend actions for future conferences and highlight critical research directions to address the disparities.Source: §5 · p. 11

Limitations

The survey may miss relevant work outside *ACL and reflects a fast-moving field at a particular moment; annotation imprecision may also affect its measurements. The authors note that their proposed research directions are not exhaustive and argue that bridging the language divide must remain central to global LLM safety research.

Coverage of venues

  • The *ACL focus may omit relevant work from non-peer-reviewed sources or other venues, and the fast-moving field makes the findings a snapshot in time.The authors hope a repeat analysis in a year would show a more optimistic picture.Source: Limitations · p. 11

Annotation accuracy

  • Annotation inaccuracies may make measurements imprecise, especially for ambiguous safety-topic categories and papers that do not clearly state language coverage.The authors are confident they did not miss work primarily focused on multilingual LLM safety, since such papers tend to state that focus explicitly.Source: Limitations · p. 11

Research directions

Loading 2505.24119v1…