Source-linked AI summary
Do Language Models Understand Igbo Culture? Beyond Multiple Choice
Blessing Ngozi Uzor, Valdini Douglace Lemofouet, Paula Chikaodinaka Anyanwu, Tassallah Abdullahi, Clemencia Siro, Idris Abdulmumin, Seid Muhie Yimam, Shamsuddeen Hassan Muhammad
TL;DR
Language models may speak a community’s language without understanding its culture, while existing translated benchmarks can test Western ideas rather than local knowledge. The paper introduces CAB-Igbo, a community-grounded benchmark evaluated across languages and response formats. Results show a large gap between multiple-choice accuracy and open-ended cultural reasoning, with regional coverage constrained to Imo and Ebonyi.
Problem
Translated benchmarks can test Western cultural ideas in another language, leaving cultural understanding in Igbo contexts insufficiently measured.
Source: §1 · pp. 1–2Method
CAB-Igbo uses native-verified questions from authentic Igbo sources in paired multiple-choice and open-ended formats across English and Igbo.
Source: §3 · p. 3, §3.4 · p. 4Results
Across the benchmark, models showed a severe gap between multiple-choice accuracy and open-ended cultural reasoning, with frequent Western or Pan-African defaults.
Source: §5.5 · p. 10, §6 · p. 11Takeaways & Limitations
The paper concludes that cultural evaluation must be grounded in community knowledge and warns that confusing fluency with cultural understanding risks harm in education, public services, and safety.
Source: §6 · p. 11The benchmark’s regional coverage centers on Imo and Ebonyi, leaving cultural concepts from other regions underrepresented.
Source: Two Evaluation Runs for the GPT-OSS · p. 12
Abstract
from arXiv · showhide
The value of language models for any community depends on whether a model actually understands the culture of the people it intends to serve. We use the Igbo language as a case study. Existing benchmarks test translated Western ideas, lacking depth. To address this, we introduce CAB-Igbo, a benchmark developed from indigenous cultural sources and verified by native speakers across six domains. We evaluated diverse models under varying conditions, incorporating regional personas. Performance was assessed using multiple-choice accuracy and human evaluations across Epistemic Fidelity, Representational Richness, Linguistic Quality, and Pragmatic Proficiency. Results reveal a stark dissociation between language fluency and cultural knowledge. Through statistical testing, we show that for most models, the severe lack of Epistemic Fidelity and Representational Richness is already present in English and does not significantly worsen when switching to Igbo. Instead, native language prompting mainly degrades Linguistic Quality and Pragmatic Proficiency. Gemini 2.0 Flash is the only model whose cultural knowledge scores improve under Igbo prompting. We find that current models struggle to demonstrate the knowledge needed for true alignment with Igbo, and multiple-choice formats allow pattern-matching to appear as understanding. Furthermore, automated judges agree with humans on linguistic quality but fail on epistemic fidelity. Ultimately, cultural evaluation must be built from community knowledge, establishing human evaluation as the gold standard to prevent misrepresenting low-resource communities globally.
1 Introduction
The paper asks whether language models understand Igbo culture rather than merely produce fluent Igbo, amid concerns that language models can reproduce cultural misalignment. It introduces CAB-Igbo and evaluates models across cultural dimensions, including the distinction between language quality and cultural knowledge.
1 Introduction
- Language models can preserve surface meaning while mapping fluent Igbo onto Western norms or Pan-African stereotypes, risking representational harm.The paper frames cultural alignment as relevant to education, public services, and safety.Source: §1 · p. 1
Our Contributions
- CAB-Igbo measures cultural alignment using community-verified knowledge across six domains and four dimensions, separating linguistic quality from epistemic fidelity.The paper also identifies high multiple-choice accuracy as potentially masking weak open-ended cultural reasoning and says automated judges fail to assess deep cultural proxies.Source: §1 · p. 2, Our Contributions · p. 2
1.1 Research Questions
- The research questions ask how well models understand Igbo culture, whether prompt language or model scale affects cultural competence, and whether African-data training improves alignment.Source: §1.1 · p. 2
2 Related Works
Prior cultural evaluation has used translated benchmarks and fixed cultural-value frameworks, while newer work emphasizes locally curated questions and open-ended assessment. CAB-Igbo builds on native curation with diagnostic distractors and intra-cultural personas.
2 Related Works
- Translated Western benchmarks test whether models process Western cultural logic in another language, not whether they understand local culture.Survey-based value dimensions also reduce situated social realities to fixed categories.Source: §2 · p. 2
- Native-curated benchmarks let communities ask about their own culture, though BLEnD includes examples of practices not found in Northern Nigerian daily life.Source: §2 · p. 2
- CAB-Igbo builds on MyCulture with diagnostic distractors and intra-cultural personas, alongside evidence that multiple-choice guessing can outpace generative reasoning.Related work also reports that persona prompting can have unstable or negative effects on cultural reasoning.Source: §2 · p. 3
3 CAB-Igbo Dataset
CAB-Igbo is a bilingual benchmark built from authentic Igbo community sources and native-verified cultural questions. Its paired multiple-choice and open-ended formats test both answer recognition and the ability to explain cultural reasoning.
3 CAB-Igbo Dataset
- CAB-Igbo pairs English and Igbo questions in multiple-choice and short-answer formats, with the latter requiring cultural reasoning without answer choices.Its 296 paradigms are drawn from authentic community sources and verified by native speakers.Source: §3 · p. 3
3.1 Dataset Construction
- Dataset construction used expert-guided primary sources rather than web scraping, reducing—but not eliminating—the likelihood of pretraining data leakage.Items were checked for factual accuracy, language flow, and cultural fit, with items removed when community consensus was absent.Source: §3.1 · p. 3
3.2 Evaluator Requirements
- Native Igbo-speaking evaluators supplied community cultural knowledge, while a linguist verified each Igbo line and speakers reviewed items for accuracy, language quality, and cultural fit.Source: §3.2 · p. 3
3.3 Regional and Domain Selection
- The benchmark covers six cultural domains and uses Imo and underrepresented Ebonyi contexts to test intra-Igbo regional variation.Imo was selected as a reference point for Standard Central Igbo; Ebonyi was selected to represent dialect variation.Source: §3.3 · p. 3
3.4 Question Formatting and Data Statistics
- The same 296 paradigms were tested in both formats across English and Igbo, yielding 592 unique items and 1,184 testing conditions before personas.The evaluation grid also used three linguistic prompt types, 11 persona conditions, and two answer permutations; open-ended generations covered a 20% sample of 59 questions, with 1,415 human-scored.Source: §3.4 · p. 4
4 Experimental Setup
The experiments compare eight models across prompt languages, answer formats, and persona conditions. The study uses paired statistical tests suited to binary multiple-choice outcomes and ordinal human ratings.
4 Experimental Setup
- Eight models spanning Africa-specific, multilingual open, general-purpose open, and closed frontier categories were evaluated with greedy decoding across three linguistic prompt types.The formats were multiple-choice and open-ended short answers, with persona conditions representing socioeconomic roles across Imo and Ebonyi.Source: §4 · pp. 4–5
- GPT-OSS 120B and 20B initially returned empty MCQ responses under token limits; only GPT-OSS 120B empty items were successfully re-queried, while GPT-OSS 20B empties remained incorrect.The 120B second pass raised the limit to 4,000 tokens and did not change any first-run response.Source: §4 · pp. 4–5
4.1 Statistical Methodology
- McNemar’s test with Yates’ correction assessed paired MCQ differences, while 95% bootstrap intervals over 10,000 resamples assessed within-item human-rated short-answer shifts.Short-answer differences were treated as significant when the confidence interval excluded zero.Source: §4.1 · p. 5
5 Analysis
Across CAB-Igbo, strong multiple-choice recognition often failed to translate into culturally correct open-ended explanations, while human scores showed cultural-knowledge gaps distinct from language-generation problems. Prompt, persona, regional, and option-order conditions affected models differently, and automated judging was much less reliable for cultural depth than for fluency.
Linguistic Prompt Type Performance
- English-to-Igbo MCQ accuracy dropped significantly for every tested model; LLaMA 3.3 70B fell 52.9 points from 99.0% on Type 1 to 46.1% on Type 3.The significance tests compared Type 1 and Type 2 zero-shot baseline items; Figure 2 depicts cultural reasoning across English and Igbo prompt types.Source: Linguistic Prompt Type Performance · pp. 5–6, Figure 2 · p. 7
Effect of Persona; Performance Gap Across States
- Persona effects varied by model: LLaMA 3.1 8B gained 10.4 points on average, while AfriqueQwen 14B and GPT-OSS 20B lost accuracy; regional-condition gaps were only 0.1–0.6 points.No persona produced a consistent benefit across models; the Imo/Ebonyi MCQ comparison may not reveal regional knowledge because both conditions share the correct option.Source: Effect of Persona · p. 6, Performance Gap Across States · p. 6
The Effect of Positional Bias; Open-Ended Evaluation Overview
- Shuffled answer ordering significantly reduced accuracy for four models, indicating that some MCQ performance depends on option order rather than cultural knowledge.The significant drops were found for GPT-4o mini, LLaMA 3.1 8B, Gemini 2.0 Flash, and LLaMA 3.3 70B; the open-ended rubric separately scored four dimensions from 0 to 2.Source: The Effect of Positional Bias · pp. 6–7, Open-Ended Evaluation Overview · p. 7
Human Evaluation Scores by Model
- Gemini 2.0 Flash led aggregate human scores at 4.316/8, while four middle-tier models had near-maximum Linguistic Quality but EF/RR scores of only 0.062–0.102.The middle tier comprised GPT-4o mini, GPT-OSS 120B, LLaMA 3.3 70B, and AfriqueQwen 14B; their total scores ranged from 2.859 to 3.147.Source: Human Evaluation Scores by Model · p. 7, Figure 4 · p. 9
Prompt Type Effects in Open-Ended Evaluation
- For most models, English-to-Igbo prompting did not significantly change Epistemic Fidelity or Representational Richness, while Linguistic Quality and Pragmatic Proficiency often declined.Gemini 2.0 Flash was the only model whose cultural-knowledge scores improved under Igbo prompting; LLaMA 3.1 8B instead showed a significant EF/RR decline.Source: Prompt Type Effects in Open-Ended… · p. 8, §5.5 · p. 11
Inter-Rater Agreement
- Human and Gemini-judge scores agreed near-perfectly on Linguistic Quality but poorly on Epistemic Fidelity and Representational Richness, supporting human evaluation for cultural depth.The judge’s weighted κ was below 0.115 for EF and RR, while LQ agreement was κw = 0.833 and ρ = 0.885.Source: Figure 3 · p. 8, Inter-Rater Agreement · p. 9, §5.3 · p. 9
5.4 Error Analysis
- Under Igbo input, Model Breakdown and Repetition rose sharply, and no prompt type exceeded 7% primarily correct answers.Type 2 and Type 3 answers were primarily Model Breakdown in 27% and 33% of cases, respectively; Type 1 answers were primarily hallucinations in 93% of cases.Source: §5.4 · p. 9, Figure 6 · p. 11
5.5 Cross-Model Synthesis
- GPT-4o mini answered 87.0% of MCQs correctly but gave fully correct explanations for only 2.3% of open-ended answers; Gemini 2.0 Flash scored 97.2% and 26.0%, respectively.No other model exceeded 4.5% fully correct explanations; GPT-OSS 120B had 89.9% overall MCQ accuracy but only 3.4% fully correct answers in the human-evaluated sample.Source: §5.5 · p. 10
6 Conclusion
CAB-Igbo evaluates Igbo cultural alignment using community-sourced knowledge, and results across eight models show persistent difficulty with culturally grounded responses. The findings highlight the danger of mistaking language fluency for cultural understanding, especially in education, public services, and safety.
6 Conclusion
- CAB-Igbo evaluates Igbo cultural alignment using questions drawn from authentic community sources and validated by native speakers.The benchmark covers six cultural domains.Source: §6 · p. 11
- Across eight models, open-weight and proprietary systems struggle with Igbo cultural contexts and often default to Western norms or Pan-African generalizations.The gap between multiple-choice accuracy and open-ended reasoning is severe; models recognize patterns but struggle to generate culturally grounded explanations.Source: §6 · p. 11
- The findings underscore the need for culturally aware models grounded in community knowledge rather than translated Western concepts or superficial fluency.Source: §6 · p. 11
- Deploying models without cultural grounding risks harm in education, public services, and safety.Source: §6 · p. 11
Limitations
The limitations concern statistical robustness, model-specific response handling, human-rating reliability, and the scope of dialect and evaluation coverage. These constraints qualify how broadly the results can be interpreted.
Limitations
- Significance tests assume independent items without multiple-comparison correction, and only two answer permutations cannot definitively rule out positional bias.Results near p = 0.05 warrant caution; frontier-model Type-1 MCQ accuracy was also near ceiling.Source: Limitations · p. 11
Two Evaluation Runs for the GPT-OSS
- GPT-OSS 120B MCQ scores combine two runs, while 60.5% of GPT-OSS 20B responses remained empty and were scored incorrect.GPT-OSS 20B answered items at 97.0% accuracy in original ordering, so its reported MCQ accuracy mainly reflects empty outputs; the authors draw no cultural-competence conclusions from it.Source: Two Evaluation Runs for the GPT-OSS · p. 11
- Open-ended evaluation used one native-speaker expert, leaving construct-level score distributions—especially EF and RR—without inter-human agreement evidence.The authors propose dual-human calibration on 100–200 items.Source: Two Evaluation Runs for the GPT-OSS · pp. 11–12
- The study used zero-temperature greedy decoding and did not test temperature variation, few-shot prompting, or retrieval-augmented generation.Open-ended generations were not evaluated under persona and state conditions, leaving that interaction unexplored.Source: Two Evaluation Runs for the GPT-OSS · p. 12
- The dialect evaluation used Standard Central Igbo and Imo and Ebonyi personas, while cultural concepts from other regions remain underrepresented.The authors report preliminary evidence that dialectal coverage materially changes EF and RR scores.Source: Two Evaluation Runs for the GPT-OSS · p. 12
- Human evaluation covered a 20% benchmark sample under zero-shot conditions, with one native-speaker evaluator; the authors call for broader dialect and cultural coverage.They recommend expanding the benchmark to include underrepresented dialects and contemporary cultural practices, alongside broader native-speaker adjudication.Source: Two Evaluation Runs for the GPT-OSS · p. 12
Ethics and Data Availability
The authors report plans for public release and describe privacy, consent, and compensation practices for community contributors.
Ethics and Data Availability
- CAB-Igbo is slated for open-licensed release with annotation guidelines and evaluation scripts, and contributors were compensated under privacy and informed-consent safeguards.The release is intended to support external audits and future research.Source: Ethics and Data Availability · p. 12
A Detailed Results
The appendix tables organize MCQ accuracy across prompt types, regional personas, and persona conditions, followed by paired significance tests. The supplied passages identify table scopes but do not report their cell values.
A.1 Multiple-choice accuracy
- Table 1 reports MCQ accuracy by linguistic prompt type, pooled over baseline and persona conditions.The passage identifies the table’s scope but supplies no accuracy values.Source: Table 1 · p. 14
- Table 2 compares zero-shot baseline, Imo, and Ebonyi MCQ accuracy, with each regional condition pooled over five personas.The passage identifies the table’s scope but supplies no accuracy values.Source: Table 2 · p. 14
- Tables 3 and 4 report MCQ accuracy for pooled persona conditions and individual personas, respectively.Table 3 includes baseline-to-persona deltas in percentage points; Table 4 defines the persona abbreviations.Source: Table 3 · p. 14, Table 4 · p. 14
A.2 Significance tests
- Table 5 summarizes McNemar tests for prompt-language and option-order comparisons, plus counts of persona conditions significantly different from baseline.Its caption specifies paired MCQ outcomes and p < 0.05 as the significance threshold.Source: Table 5 · p. 15
A.3 Human evaluation
The human-evaluation tables cover total and construct-level scores and within-item prompt-language differences; a separate table describes human–judge agreement measures. Figure 7 presents examples of repetition loops.
A.3 Human evaluation
- Human-evaluation tables report total scores out of 8 by model and prompt type, construct means, and paired Type 1-minus-Type 2 differences.Table 7 scores each construct from 0 to 2; Table 8 marks differences whose bootstrap 95% confidence intervals exclude zero.Source: Table 6 · p. 15, Table 7 · p. 15, Table 8 · p. 15
A.4 LLM-as-a-judge
- The judge-comparison tables report mean judge scores and human–judge agreement over matched responses using weighted kappa, rank correlation, and exact or near agreement.Table 10 covers 1,238 matched responses; Table 9 reports model means over the matched evaluation responses.Source: Table 10 · p. 15, Table 9 · p. 15
A.5 Repetition failures
- Figure 7 presents verbatim excerpts of the repetition loops described in Section 5.4.Source: §A.5 · p. 15
B Dataset Statement for CAB-Igbo
CAB-Igbo is a manually created, native-speaker-validated bilingual benchmark designed to assess Igbo cultural knowledge across six domains. Its questions emphasize cultural reasoning, parallel Igbo and English versions, and review for cultural and linguistic validity.
B.1 Creation
- CAB-Igbo was built from scratch without web-scraped text and contains 296 Igbo and English questions designed and reviewed by a native Igbo speaker and an Igbo linguist.Native speakers from two Igbo-speaking states manually created and validated the data.Source: §B.1 · pp. 15–16
B.2 Cultural domains
- The benchmark covers six domains of Igbo cultural life, with Table 11 reporting the number of questions in each domain.Source: §B.2 · p. 16, Table 11 · p. 16
B.3 Question design
- Questions prioritize cultural reasoning—including cultural logic, ritual and social context, and appropriate behavior—over factual recall alone, using direct wording and precise cultural terminology.One example asks why a child may be buried without mourning rites, requiring knowledge of the ogbanje belief system.Source: §B.3 · p. 16
B.4 Linguistic grid and question formats
- Each item has multiple-choice and open-ended forms, and each is tested under three combinations of English or Igbo instructions and questions.Multiple-choice items have one correct answer and three diagnostic distractors; open-ended answers explain reasoning in two to four sentences and receive human evaluation.Source: §B.4 · pp. 16–17, §B.5 · p. 17, Table 12 · p. 17
B.6 Bilingual and orthographic requirements
- Every question has parallel Igbo and English versions intended to test the same knowledge with the same meaning, using natural terminology, standard orthography, and tone marks where needed.Source: §B.6 · p. 17
B.9 Review process
- Items were drafted, jointly reviewed for cultural accuracy and translation equivalence, then checked for duplicates and ambiguity; native speakers revised or removed items lacking agreement.The quality criteria include authentic practice, unambiguous questions, plausible diagnostic distractors, semantic equivalence, and unique knowledge.Source: §B.10 · p. 17, §B.9 · p. 17, §B.8 · p. 17
B.11 Common pitfalls
- Creators were advised to avoid multiple defensible answers, superficial recall questions, unnatural translations, weak distractors, and unnecessary gender stereotypes.The process records question rationales, removals and revisions, distractor decisions, and domain statistics.Source: §B.11 · pp. 17–18
C.1 Background
- The evaluation uses trained annotators to assess model responses, validate an automated judge, and compare performance across prompt types, while keeping model identities hidden.Source: §C.1 · p. 18
C.2 Evaluation setup
The evaluation samples responses from eight models across three prompt types and uses trained annotators to score four response constructs. It also specifies annotation rules and zero-shot and persona prompt templates.
C.2 Evaluation setup
- Annotators assess 20% of CAB-Igbo, scoring responses from eight models under all three prompt types in the zero-shot, no-persona condition.The questions span the benchmark's six cultural domains.Source: §C.2 · p. 18
C.3 Scoring constructs
- Responses receive 0–2 integer scores for Epistemic Fidelity, Representational Richness, Linguistic Quality, and Pragmatic Proficiency, summed to a total from 0 to 8.For Igbo responses, grammar and morphology are assessed under Linguistic Quality.Source: §C.3 · p. 18, Table 13 · p. 18
C.4 Annotation procedure
- Annotators review both English and Igbo question versions, score each response independently, and assign zero across constructs to blank or missing responses.They assess Igbo fluency for Type 2 and Type 3, avoid discussing scores beforehand, and use the lower score when uncertain.Source: §C.4 · p. 18, §C.5 · pp. 18–19
D Prompt Templates
- The Igbo instruction text used in evaluation was retained verbatim despite a native-speaker linguist identifying minor phrasing issues.Source: D Prompt Templates · p. 19
D.1 Zero-shot prompts
- Zero-shot templates require letter-only answers for multiple-choice tasks and concise two-to-four-sentence explanations for open-ended tasks, with language instructions varying by prompt type.Source: §D.1 · p. 19
D.2 Persona introductions
- Persona prompts ask models to answer from a specified Igbo cultural perspective and state, with five personas combined across Imo and Ebonyi for ten conditions.The persona attributes include sex, marital status, education, and social class.Source: §D.2 · p. 19, §D.3 · p. 20, Table 14 · p. 20