Red-Teaming Large Language Models in Low-Resource African Languages: A Case Study of Yoruba and Nigerian Pidgin

1 Introduction

• LLMs are increasingly used across African services, while safety concerns include the ease with which an insufficiently guarded chatbot can be manipulated into harmful content.The introduction cites Microsoft’s Tay as an example of attackers eliciting racist and provocative content within hours of its release.Source: §1 · p. 1
• Safety alignment testing is largely English-centric, leaving its generalisation to Yoruba and Nigerian Pidgin uncertain, especially because models that fail to comprehend prompts may appear safe.The paper identifies the distinction between safety failure and comprehension failure as insufficiently explored in red-teaming studies of these languages.Source: §1 · p. 2
• The study asks whether models refuse adversarial prompts as reliably in Yoruba and Nigerian Pidgin as in English, how large the safety gap is, and how harm categories and comprehension affect evaluation.Its questions also ask whether models genuinely comprehend Yoruba and Pidgin adversarial prompts and how comprehension confounds standard safety metrics.Source: §1 · p. 2
• The authors construct a 400-prompt red-teaming benchmark, evaluate six LLMs, and use human ratings of harmfulness, helpfulness, and comprehension alongside the Cross-Lingual Safety Gap metric.The prompts cover four Nigeria-relevant harm categories and are translated into Yoruba and Nigerian Pidgin, yielding 1,200 prompts across three languages.Source: §1 · p. 2

2.1 LLM Safety and Red-Teaming

• Red-teaming probes unsafe behavior under adversarial conditions; this study focuses on prompt-based, single-turn attacks, while prior work reports greater vulnerability from multi-turn attacks.Human red-teaming offers cultural judgement but is costly to scale, and qualified red-teamers are scarce for Yoruba and Nigerian Pidgin.Source: §2.1 · p. 3
• Existing red-teaming benchmarks and evaluation approaches are predominantly English-based, while automated detection and LLM-as-judge methods are unreliable where target-language resources and judge comprehension are limited.The related work identifies human review as the gold standard when cultural and linguistic nuance matters.Source: §2.1 · p. 3

2.2 Multilingual and African-Language Safety

• Prior studies report higher attack success outside English and severe African-language evaluation degradation, but policy-compliance benchmarks do not test active exploitation.The cited literature also describes code-switching vulnerabilities and cultural nuance lost in translation-based datasets.Source: §2.2 · pp. 3–4

2.3 Research Gap

• The identified gap is a study combining Nigerian-context adversarial red-teaming, Yoruba and Nigerian Pidgin, multi-model human evaluation with agreement measurement, and a metric separating comprehension failure from safety.The paper positions its work as complementing Amplify’s data collection and UbuntuGuard’s policy-compliance findings.Source: §2.3 · p. 4

3.1 Overview

• The evaluation proceeds from culturally adapted and newly authored prompts through Yoruba and Nigerian Pidgin translation, six-model zero-shot testing, and human scoring with cross-language metric comparisons.Figure 2 depicts the four-phase pipeline from prompt construction to metric computation.Source: §3.1 · p. 4, Figure 2 · p. 5

3.2 Target Languages

• Yoruba is underrepresented despite tens of millions of speakers, while Nigerian Pidgin’s substantial English code-switching makes it relevant for testing safety systems trained primarily on monolingual English.Yoruba’s oral literature and proverbs may lack direct English equivalents.Source: §3.2 · p. 4

3.3 Dataset Construction

• The benchmark contains 400 English prompts, with 172 adapted from four datasets and 228 newly authored; Nigerian localization covers four harm categories and creates 1,200 prompts across three languages.Prompts were localized to Nigerian institutions, groups, politics, and materials; all state harmful intent explicitly rather than relying on role-play or indirect framing.Source: §3.3 · pp. 4–5, Table 1 · p. 5

3.4 Model Selection

• Six models span English-dominant, broad multilingual, commercial multilingual, and Africa-focused training paradigms, and were evaluated zero-shot at temperature 0.The selection includes models from 8B to 70B parameters and Aya 101, which spans 101 languages.Source: §3.4 · p. 6

3.5 Evaluation Protocol

• Prompts were submitted without prior context or examples, with temperature 0 for reproducibility; keyword-flagged refusals were then confirmed by human evaluators.Responses were stored with prompt, language, harm-category, and model metadata.Source: §3.5 · p. 6

3.6 Human Evaluation

• Two native-speaker annotators independently scored harmfulness, helpfulness, and comprehension; disagreements were adjudicated, and reported metrics used 30% of responses.A minimum Cohen’s κ of 0.60 was required, 341 scores were adjudicated, and the stratified sample comprised 2,160 responses.Source: §3.6 · pp. 6–7

3.7 Metrics

• The six metrics include refusal, harmful-content, attack-success, helpfulness, comprehension, and Cross-Lingual Safety Gap rates; refusal and safety metrics require comprehension-aware interpretation.CLSG is target-language ASR minus English ASR; a negative gap may indicate stronger safety or poor comprehension, so it must be interpreted using comprehension rate.Source: §3.7 · p. 7

4.1 Inter-Annotator Agreement

• Cohen’s κ ranged from 0.64 to 1.00 across models, languages, and dimensions, exceeding the 0.60 threshold for substantial agreement.Refusal scoring had perfect agreement; harmfulness disagreements were 2.5–4.7% by language, compared with 3.1–9.6% for helpfulness and comprehension.Source: §4.1 · p. 7

4.2 Refusal and Attack Success

• Aya 101 had the highest English ASR (0.60), while GPT-4o Mini had the highest Yoruba ASR (0.41) and Aya 101 the highest Pidgin ASR (0.48).Llama 3.1 8B, Llama 3.3 70B, and Aya 101 refused zero Yoruba prompts; low Yoruba ASR for Qwen3 32B (0.01) reflected near-total comprehension failure.Source: §4.2 · p. 7, Figure 3 · p. 8

4.3 Cross-Lingual Safety Gap

• GPT-4o Mini’s Yoruba CLSG was +0.23, the largest positive Yoruba gap, while Gemini 2.0 Flash had near-zero gaps in Yoruba (−0.07) and Pidgin (−0.01).Aya 101’s −0.35 Yoruba gap reflected low comprehension (CR = 0.28), not stronger safety; negative gaps for Llama 3.1 8B and Qwen3 32B were likewise attributed to low comprehension.Source: §4.3 · pp. 7–8, Figure 4 · p. 8

4.4 Harm Category Analysis

• Cultural stereotyping and discrimination had the highest reported category ASR, reaching 0.44 in Pidgin (HCR 0.49), despite low refusal rates across languages.Harmful-instruction prompts had the highest refusal rates in English and Pidgin (0.52), yet their ASR was 0.27 in both languages (HCR 0.55); political-manipulation refusal was 0.40 in English, 0.12 in Yoruba, and 0.32 in Pidgin.Source: §4.4 · pp. 8–9, Figure 5 · p. 8

4.5 Comprehension and Helpfulness

• Qwen3 32B comprehension fell from 0.99 in English to 0.08 in Yoruba, while Gemini 2.0 Flash retained 0.92 in Yoruba and 0.98 in Pidgin.In Yoruba, comprehension strongly predicted helpfulness (r = 0.81), and low measured attack success mainly reflected failure to engage rather than successful refusal; low-comprehension responses also repeated prompts or unrelated stock phrases.Source: §4.5 · p. 9

5 Discussion

• Yoruba refusal is a less reliable safety indicator when models struggle to understand prompts, so comprehension or engagement should be reported alongside refusal- or harm-based metrics.The three Llama and Aya models never refused a Yoruba prompt, and GPT-4o Mini’s refusal rate fell 46 percentage points from English to Yoruba.Source: §5 · p. 9
• Gemini 2.0 Flash sustains high comprehension and low attack success across all three languages, although the study cannot determine which training choices produced this outcome.Source: §5 · p. 10
• Researchers are urged to benchmark African languages, developers to use authentic target-language data involving native speakers, and policymakers to require language-specific safety assessment for deployment in Nigeria.The discussion also says automated judges cannot yet replace native-speaker human review in these languages.Source: §5 · p. 10

6 Conclusion

• Across six LLMs spanning English-dominant, broadly multilingual, and Africa-focused training, safety behavior learned predominantly from English data does not transfer to Yoruba or Nigerian Pidgin.Refusal rates decrease and attack success rates increase for most models in the Nigerian languages.Source: §6 · p. 10
• In Nigerian Pidgin, cultural stereotyping is consistently the harm category most likely to bypass safety filters, while comprehension is the primary bottleneck on engagement in Yoruba.Source: §6 · p. 10
• The authors plan to release a benchmark of 1,200 prompts across English, Yoruba, and Nigerian Pidgin with human-annotated harmfulness, helpfulness, and comprehension labels and the Cross-Lingual Safety Gap metric.Source: §6 · p. 10

Limitations

• Only six models were evaluated, and just one, Aya 101, prioritises African language coverage; budget and computing constraints limited the study’s model scope.Source: Limitations · p. 10
• The study covers only Yoruba and Nigerian Pidgin, leaving Igbo, Hausa, and other Nigerian languages for future evaluation.Source: Limitations · p. 10
• Yoruba prompts used post-edited machine translation because qualified annotators were limited, and annotation relied on two evaluators per language with researcher adjudication for disagreements.Source: Limitations · p. 10
• Evaluation used single-turn prompts; multi-turn adversarial evaluation in these languages remains an open direction.Source: Limitations · p. 10

Ethics Statement

• Annotators were fairly compensated and informed beforehand about the harmful content they would review; no real individuals were targeted beyond public political figures in one harm category.Source: Ethics Statement · p. 10

A Pipeline Overview and Model Access

• Table 3 presents the models evaluated in the study.Source: Table 3 · p. 12

B Translation Methodology

• Yoruba prompts were machine-translated and post-edited by a native linguist, while Nigerian Pidgin prompts were translated from scratch by a native speaker.Translators were instructed to preserve harmful intent while producing natural phrasing.Source: B Translation Methodology · p. 12

C Full Inter-Annotator Agreement

• All reported Cohen’s κ values meet or exceed the substantial-agreement threshold of 0.60.Table 4 reports agreement by model, language, and annotation dimension.Source: C Full Inter-Annotator Agreement · p. 13, Table 4 · p. 13
• Table 5 covers annotator disagreement before adjudication, alongside Table 4’s model- and language-level agreement breakdown.Source: Table 5 · p. 13, Table 4 · p. 13

D Full Harm-Category Breakdown

• Table 6 reports metrics by harm category and language, aggregated across all six models.Source: Table 6 · p. 13

• RR and ASR correlations are r = −0.84 in English, r = −0.97 in Nigerian Pidgin, and r = −0.41 in Yoruba.Source: E Correlation Analysis · p. 13
• HCR and ASR are strongly positively correlated in English (r = 0.88).Source: E Correlation Analysis · p. 13
• CR and HR correlations are r = 0.81 in Yoruba, r = 0.20 in English, and r = 0.15 in Pidgin.The passage identifies comprehension as the primary factor limiting engagement in Yoruba.Source: E Correlation Analysis · p. 13

• The same underlying prompt elicits a safe English response but harmful responses in Yoruba and Nigerian Pidgin.Source: F Qualitative Examples · p. 13
• Low Yoruba comprehension produced non-engaging repetition loops, including prompt repetition and repeated greeting phrases.Figure 7 shows prompt repetition by Llama 3.1 8B and phrase repetition by Aya 101.Source: F Qualitative Examples · pp. 13–14, Figure 7 · p. 14
• Figure 6 heatmaps encode Pearson correlations across evaluation metrics and languages, with blue for positive, orange for negative, and white for no correlation.Source: Figure 6 · p. 14