PaperlayerView exact revision on arXiv

Source-linked AI summary

Afri-MCQA: Multimodal Cultural Question Answering for African Languages

Atnafu Lambebo Tonja, Srija Anand, Emilio Villa-Cueva, Israel Abebe Azime, Jesujoba Oluwadara Alabi, Muhidin A. Mohamed, Debela Desalegn Yadeta, Negasi Haile Abadi, Abigail Oppong, Nnaemeka Casmir Obiefuna, Idris Abdulmumin, Naome A Etori, Eric Peter Wairagala, Kanda Patrick Tshinu, Imanigirimbabazi Emmanuel, Gabofetswe Malema, Alham Fikri Aji, David Ifeoluwa Adelani, Thamar Solorio

arXiv:2601.05699v2cs.CL

TL;DR

African languages and cultures remain underrepresented in multimodal AI, despite their diversity and the relevance of speech-based applications. Afri-MCQA provides a native-speaker-created benchmark and evaluates models across languages, modalities, and VQA formats. The evaluation finds persistent language, speech, and task-format gaps, motivating speech-first and culturally grounded research directions.

  • Problem

    Multimodal models and benchmarks have limited coverage of African languages and cultures, although many African languages are primarily spoken and speech-based applications are relevant.

    Source: §1 · p. 1

  • Method

    Afri-MCQA provides approximately 7.5k image-grounded Q&A pairs across 15 African languages in 12 countries, with parallel English and native-language text and speech created by native speakers.

    Source: §1 · p. 2

  • Results

    Models struggle with African cultural QA, with English queries outperforming native-language queries and multiple-choice performance exceeding open-ended VQA.

    Source: §8 · p. 8, §7 · p. 8

  • Takeaways & Limitations

    The findings motivate speech-first approaches, culturally grounded pretraining, and research into cross-lingual cultural transfer.

    Source: §8 · p. 8

    The 15-language, 12-country dataset leaves many African languages and cultural groups unrepresented, and its categories abstract away regional, generational, and community-specific variation.

    Source: §9 · p. 9

Abstract

from arXiv · show

Africa is home to over one-third of the world's languages, yet remains underrepresented in AI research. We introduce Afri-MCQA, the first Multilingual Cultural Question-Answering benchmark covering 7.5k Q&A pairs across 15 African languages from 12 countries. The benchmark offers parallel English-African language Q&A pairs across text and speech modalities and was entirely created by native speakers. Benchmarking large language models (LLMs) on Afri-MCQA shows that open-weight models perform poorly across evaluated cultures, with near-zero accuracy on open-ended VQA when queried in native language or speech. To evaluate linguistic competence, we include control experiments meant to assess this specific aspect separate from cultural knowledge, and we observe significant performance gaps between native languages and English for both text and speech. These findings underscore the need for speech-first approaches, culturally grounded pretraining, and cross-lingual cultural transfer. To support more inclusive multimodal AI development in African languages, we release our Afri-MCQA under academic license or CC BY-NC 4.0 on HuggingFace (https://huggingface.co/datasets/Atnafu/Afri-MCQA)

1 Introduction

African languages and cultures remain underrepresented in multimodal AI evaluation, despite the region’s linguistic diversity and the relevance of speech. Afri-MCQA addresses this gap with a native-speaker-created benchmark and evaluates cultural visual QA across languages, modalities, and task formats.

1 Introduction

  • Africa’s linguistic diversity includes more than one-third of the world’s languages, and many African languages are primarily spoken, making speech-based applications especially relevant.The introduction notes linguistic features such as rich morphology, noun classes, tonality, and serial verb constructions.Source: §1 · p. 1
  • Existing African-language cultural benchmarks are text-only, while language-vision benchmarks provide limited coverage of African languages and contexts.Previous African-language evaluations largely focus on text tasks, leaving visual knowledge and multimodal reasoning for African contexts largely untested.Source: §1 · pp. 1–2
  • The benchmark evaluates MLLMs on African cultural visual QA and asks how performance varies by input modality, query language, and task format.Its research questions also examine overall understanding of African cultural contexts and whether native-versus-English gaps reflect language understanding or cultural knowledge.Source: §1 · p. 2
  • Afri-MCQA introduces the first large-scale multilingual visual cultural QA benchmark for 15 African languages across 12 countries, with parallel text and speech QA created by native speakers.The dataset contains approximately 7.5k English and native-language Q&A pairs grounded in images.Source: §1 · p. 2
  • Afri-MCQA’s contributions include evaluating whether text-based multilingual ability transfers to speech understanding and releasing the benchmark for multimodal research.The authors emphasize the need for greater African-language representation in multimodal training.Source: §1 · p. 2

2 Related work

Prior African-language benchmarks largely assess text tasks, while multimodal cultural QA resources offer limited African coverage and omit speech. Afri-MCQA targets these gaps with broader language coverage, speech, and diagnostic controls.

2 Related work

  • Cultural VQA benchmarks use human-written questions but cover few languages per continent, while ALM-bench expands coverage using many LLM-generated questions and web-sourced images.Earlier work also includes binary classification of statements about culturally related image pairs.Source: §2 · p. 2
  • Existing evaluations of African languages cover text tasks, but visual knowledge and multimodal reasoning for African contexts remain largely untested.Region-specific resources include text classification, cross-lingual question answering, and broader text-task suites.Source: §2 · p. 2
  • Existing multimodal datasets include few African languages and do not adequately capture the region’s diversity; common benchmark designs also query models only through text.Afri-MCQA addresses these gaps through 15 languages across 12 countries, parallel speech, and control experiments separating linguistic competence from cultural knowledge limitations.Source: §2 · p. 2

3 Dataset

Afri-MCQA is a multilingual, image-grounded QA dataset built with native speakers and parallel English and native-language text and speech. Its collection process includes screening, review, and final quality checks.

3 Dataset

  • The dataset selects 15 widely spoken sub-Saharan African languages across 12 countries, representing approximately 392.6 million speakers, and recruits resident native speakers as annotators.Annotators were selected for English fluency, annotation or data-collection experience, strong project completion, and residence in a country where the target language is spoken.Source: §3 · pp. 2–3

3.1 Dataset collection

  • Collection guidelines extend prior image-question annotation procedures with audio-recording instructions and a two-step review and verification process.A pilot trained and screened annotators before the main annotation phase; language coordinators reviewed submissions and the project team conducted a final review.Source: §3.1 · p. 3

3.2 Dataset Composition

  • Each data point pairs an image with multiple-choice questions in English and the native language, in both text and speech.Annotators contributed images where possible or used open-license images, which were assigned to 10 established categories.Source: §3.2 · p. 3, Figure 2 · p. 3
  • For each image, annotators wrote up to three question-and-answer triplets in English and the native language, each with one correct answer and three distractors.Questions were intended to be complex and require reasoning for a correct answer.Source: §3.2 · pp. 3–4
  • Audio recordings cover each question and its corresponding answers in native languages and African-accented English.Recordings were made by annotators reading the questions and answers clearly in both languages.Source: §3.2 · p. 4

4 Experimental setup

The evaluation compares multimodal models and text-only baselines across text and audio inputs, English and native-language queries, and multiple-choice and open-ended VQA. The experiments use location-aware prompts.

4 Experimental setup

  • The model set includes open-weight Qwen 2.5-Omni and Gemma-3n, text-only Gemma3 baselines, and closed-source Gemini-2.5 Pro.Models were selected for image-and-audio support and, within families, differing sizes to assess scaling effects.Source: §4 · p. 4
  • The input-modality comparison uses written questions or native-speaker recordings of the same questions and options, with the related image included in every setting.This setup compares performance generalization from text to speech in African languages and accented English.Source: §4 · p. 4
  • Each question is presented in a native language and English to compare model behavior and examine linguistic versus cultural limitations.Source: §4 · p. 4
  • Both multiple-choice VQA, which supplies answer options, and open-ended VQA, which requires answer generation, are evaluated with shared prompt templates.The comparison tests whether multiple-choice performance reflects cultural understanding or selection from provided options.Source: §4 · p. 4
  • All evaluated settings use a location-aware prompt because it performed better than image-only prompts.Image-only prompt results and language-wise results are reported in the appendix.Source: §4 · p. 4

5 Evaluation

The evaluation tests culturally grounded visual QA in text and speech, alongside language-understanding controls, using automatic metrics and human judgments for text open-ended VQA.

5.1 Cultural VQA Evaluation

  • Text MC-VQA asks models to select among four options, while text open-ended VQA requires generating an answer without options.Both formats provide an image and text question; open-ended QA tests cultural knowledge without answer-set hints.Source: §5.1 · p. 5
  • Audio MC-VQA uses African-accented English or native-language speech, while audio open-ended VQA requires a text answer to a spoken question and image.Source: §5.1 · p. 5

5.2 Control Experiments

  • Text controls use AfriXNLI and AfriMMLU to estimate baseline linguistic competence by comparing their results with Afri-MCQA.Source: §5.2 · p. 5
  • Audio controls test ASR transcription and language identification to distinguish speech-processing failures from cultural-reasoning limitations.ASR assesses capture of spoken content as a prerequisite for answering; LID tests recognition of which of 15 languages is spoken.Source: §5.2 · p. 5

5.3 Evaluation metrics

  • Evaluation is zero-shot, with automatic metrics across tasks and human evaluation added for text open-ended VQA.Source: §5.3 · p. 5
  • MC-VQA and classification use accuracy, open-ended QA uses GPT-4o-mini as judge, and ASR uses WER.Open-ended QA also reports chrF++ in Appendix C; human raters judged 50 randomly sampled questions per language on each family’s best-performing model.Source: §5.3 · p. 5

6 Results

Across text and audio, open-weight models generally fare worse on native-language queries, open-ended QA, and speech than on their corresponding alternatives; controls reveal substantial language and speech-processing gaps.

6.1.1 Text-based QA

  • Open-weight models show limited or no gains from scaling on native-language QA, while Gemini-2.5 Pro outperforms them across both text tasks.Human evaluation on 50 samples across eight languages follows the automatic trends, with Gemini-2.5 Pro performing better in native than English on text open-ended VQA.Source: §6.1.1 · p. 6, Figure 5 · p. 7

6.1 Cultural VQA Results

  • Open-weight models generally perform better in English than native languages, and audio open-ended QA in spoken native languages reaches nearly zero accuracy.The text results also show major drops from multiple choice to open-ended QA, including in English; Gemini-2.5 Pro is more consistent across settings.Source: §6.1.1 · pp. 5–6, §6.1.2 · pp. 6–7, Figure 4 · p. 6

6.1.2 Audio-based QA

  • Audio is significantly harder than text for open-weight models, with poor native-language identification and ASR accompanying weak audio open-ended VQA.Qwen variants show near-random LID and compromised native ASR; the passage links failures in language identification and transcription to downstream VQA errors.Source: §6.1.2 · pp. 6–7

6.2.1 Text-based Experiments

  • Control benchmarks show English-to-native performance drops across models, with Gemini-2.5 Pro having the smallest gaps and Qwen the largest on AfriXNLI and AfriMMLU.Source: §6.2.1 · p. 7, Table 3 · p. 7
  • Afri-MCQA correlations with AfriXNLI and AfriMMLU are generally weaker and not significant, suggesting language limitations may not fully explain cultural QA failures.The authors note that African cultural and visual knowledge may also matter.Source: §6.2.1 · p. 7

6.2.2 Audio-based Experiments

  • On native African audio, Gemini-2.5 Pro has near-perfect LID and reasonable ASR, while Qwen LID is near random and open-weight audio VQA is near zero.The authors attribute poor open-ended audio VQA partly to failures to identify or transcribe the spoken languages.Source: §6.2.2 · pp. 7–8, Figure 6 · p. 8

7 Discussion

The discussion finds that MLLMs have limited African cultural QA performance, with consistent English advantages and larger drops in open-ended and native-language settings. Control comparisons suggest both linguistic and cultural limitations contribute.

RQ1

  • Gemini-2.5-Pro reaches 78% on MC-VQA and 38% on text-based English Open-VQA, while smaller models score 50–59% and 9–24%, respectively.Source: §7 · p. 8

RQ2

  • On native-language audio Open-VQA, smaller models achieve 2–5%, alongside Qwen language-identification accuracy of 2–4% and non-Gemini ASR WER of 85–100%+.Source: §7 · p. 8

RQ3

  • Language-understanding gaps are 13–47%, exceeding Afri-MCQA cultural-knowledge gaps of 2–19%, but models also struggle on English cultural QA.Source: §7 · p. 8

RQ4

  • Gemini-2.5-Pro scores 78% on MC-VQA versus 38% on Open-VQA, while some smaller models fall below 10% on Open-VQA.The discussion attributes the difference partly to simplified answer selection in MC-VQA; performance declines further for native-language queries.Source: §7 · p. 8

8 Conclusion

Afri-MCQA is a multilingual, multimodal benchmark for African cultural visual QA, and its evaluation identifies difficulties with cultural knowledge, speech processing, and performance across languages and task formats. The authors propose research directions and release the benchmark to support more inclusive multimodal AI development.

8 Conclusion

  • The evaluation finds that MLLMs struggle with African cultural knowledge, speech processing is a critical bottleneck, and gaps persist across languages and task formats.The authors report these findings across their evaluation of the benchmark.Source: §8 · p. 8
  • Afri-MCQA covers 15 African languages across 12 countries, with over 7.5k Q&A pairs in text and speech modalities.The benchmark is described as multilingual and multimodal cultural visual QA.Source: §8 · p. 8
  • The authors recommend speech-first approaches, culturally grounded pretraining, and cross-lingual cultural transfer as research directions.They note that current open-weight models lack basic LID and ASR capabilities for these languages, and that English cultural knowledge may not be accessible through native-language queries.Source: §8 · p. 8
  • The authors release Afri-MCQA as a benchmark and foundation for building more inclusive, culturally aware multimodal systems that better represent African languages and cultures.Source: §8 · p. 9

9 Limitations

The benchmark broadens African-language coverage, but its scope and curation impose limits. Its cultural categories simplify contextual variation, and its size and evaluated model set are constrained.

9 Limitations

  • The 15-language, 12-country dataset leaves many of Africa’s thousands of languages and cultural groups unrepresented.The authors also note that culture varies regionally, generationally, and across communities, which their formulation abstracts away from.Source: §9 · p. 9
  • Cultural questions and image selection may reflect annotators’ backgrounds and interpretations of cultural relevance, despite human curation.The authors acknowledge potential data-collection biases.Source: §9 · p. 9
  • The evaluation covers only a limited set of open- and closed-source models, so its performance gaps may not capture the broader landscape.Source: §9 · p. 9
  • The human-curated dataset is of moderate size and intended for evaluation, not pretraining or fine-tuning, for which the authors say it would likely cause overfitting.They attribute its time-consuming creation to the curation process and note computational and financial constraints on model evaluation.Source: §9 · p. 9

10 Ethical Considerations

The paper describes native-speaker involvement in creating and reviewing the dataset, and supplies category-distribution figures and prompt templates for evaluation. The location-aware prompts add country context to image-grounded questions.

10 Ethical Considerations

  • Native speakers annotated, spoke, and reviewed the culturally grounded Q&A pairs; contributors volunteered, received fair compensation, and were informed of the project’s research and evaluation purpose and their right to withdraw.Source: §10 · p. 9
  • Question guidelines sought to avoid stereotypes and culturally inappropriate framing, with native-speaker language coordinators reviewing annotations for accuracy, context, and respectful representation.The authors acknowledge that the dataset may still reflect biases or oversimplifications.Source: §10 · p. 9

A) Language-wise category distribution

  • The category-distribution figures present language-wise category distributions and the top six image categories, without specifying their values in the captions.Source: Figure 7 · p. 13, Figure 8 · p. 14

B Prompt

  • Evaluation prompts used consistent templates and compared image-grounded questions with prompts that also supplied the country.The location-aware form adds the country alongside the image, question, and options; Figures 9 and 10 show the text and audio versions.Source: B Prompt · p. 14, Figure 9 · p. 15, Figure 10 · p. 15

C Additional Experiments results

The supplementary material provides additional prompt and country-language results, alongside detailed instructions for annotation and review. The guidelines cover image and question creation, speech recording, category assignment, and quality checks.

C.1 Text-based QA

  • The additional-results section includes per-country-language results and tables for text-based QA across location-aware and image-only prompts, with English and native-language conditions.The supplied table captions also cover multiple-choice and open-ended audio QA conditions, but do not state their results.Source: §C.1 · p. 15, Table 4 · p. 16, Table 5 · p. 16

Guidelines for Annotators

Question complexity types

Recording Specifications

1.1 Privacy and Anonymization

  • Review checks cover image privacy and cultural relevance, idiomatic meaning-equivalent translations, and questions that remain answerable without relying on options.Reviewers are also asked to check image uniqueness and option plausibility.Source: §1.1 · p. 28, §1.2 · p. 28, §1.3 · p. 28

4.1 Transcript Accuracy

  • Audio review checks exact transcript matching, recording structure and quality, and alignment between question and option formats; reviewers also assess cultural sensitivity and content diversity.The guidelines additionally call for language-quality checks and questions that require cultural knowledge rather than trivial observation.Source: §4.1 · p. 30, §4.2 · p. 30, §4.3 · p. 30

Related papers

Loading 2601.05699v2…