PaperlayerView exact revision on arXiv

Source-linked AI summary

CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark

David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, Bontu Fufa Balcha, Chenxi Whitehouse, Christian Salamea, Dan John Velasco, David Ifeoluwa Adelani, David Le Meur, Emilio Villa-Cueva, Fajri Koto, Fauzan Farooqui, Frederico Belcavello, Ganzorig Batnasan, Gisela Vallejo, Grainne Caulfield, Guido Ivetta, Haiyue Song, Henok Biadglign Ademtew, Hernán Maina, Holy Lovenia, Israel Abebe Azime, Jan Christian Blaise Cruz, Jay Gala, Jiahui Geng, Jesus-German Ortiz-Barajas, Jinheon Baek, Jocelyn Dunstan, Laura Alonso Alemany, Kumaranage Ravindu Yasas Nagasinghe, Luciana Benotti, Luis Fernando D'Haro, Marcelo Viridiano, Marcos Estecha-Garitagoitia, Maria Camila Buitrago Cabrera, Mario Rodríguez-Cantelar, Mélanie Jouitteau, Mihail Mihaylov, Mohamed Fazli Mohamed Imam, Muhammad Farid Adilazuarda, Munkhjargal Gochoo, Munkh-Erdene Otgonbold, Naome Etori, Olivier Niyomugisha, Paula Mónica Silva, Pranjal Chitale, Raj Dabre, Rendi Chevi, Ruochen Zhang, Ryandito Diandaru, Samuel Cahyawijaya, Santiago Góngora, Soyeong Jeong, Sukannya Purkayastha, Tatsuki Kuribayashi, Teresa Clifford, Thanmay Jayakumar, Tiago Timponi Torrent, Toqeer Ehsan, Vladimir Araujo, Yova Kementchedjhieva, Zara Burzo, Zheng Wei Lim, Zheng Xin Yong, Oana Ignat, Joan Nwatu, Rada Mihalcea, Thamar Solorio, Alham Fikri Aji

arXiv:2406.05967v2cs.CVcs.AIcs.CLcs.LG

TL;DR

VQA benchmarks often emphasize English and Western-centric imagery, leaving limited coverage of diverse languages and cultural contexts. CVQA collects culturally grounded, bilingual questions across countries and languages, then evaluates multimodal models. The authors report that open models struggle on the benchmark, especially with local-language questions.

  • Problem

    Many VQA benchmarks focus on English and Western-centric images, while multilingual extensions often lack low-resource language and cultural diversity.

    Source: §1 · p. 2

  • Method

    CVQA collects culturally grounded visual questions across Country-Language pairs, with local-language questions and parallel English translations.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    Open-source models generally perform worse on CVQA when queried in local languages than in English, and CVQA presents a challenge for these models.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The authors present CVQA as a benchmark for evaluating cultural capability and bias in multimodal models and call for greater attention to non-English-centric benchmarking.

    Source: §1 · p. 3, §7 · p. 9

    CVQA covers only a fraction of the world’s languages and regions, and its scale limits its use to benchmarking rather than training new models.

    Source: §5 · pp. 8–9

Abstract

from arXiv · show

Visual Question Answering (VQA) is an important task in multimodal AI, and it is often used to test the ability of vision-language models to understand and reason on knowledge present in both visual and textual data. However, most of the current VQA models use datasets that are primarily focused on English and a few major world languages, with images that are typically Western-centric. While recent efforts have tried to increase the number of languages covered on VQA datasets, they still lack diversity in low-resource languages. More importantly, although these datasets often extend their linguistic range via translation or some other approaches, they usually keep images the same, resulting in narrow cultural representation. To address these limitations, we construct CVQA, a new Culturally-diverse multilingual Visual Question Answering benchmark, designed to cover a rich set of languages and cultures, where we engage native speakers and cultural experts in the data collection process. As a result, CVQA includes culturally-driven images and questions from across 30 countries on four continents, covering 31 languages with 13 scripts, providing a total of 10k questions. We then benchmark several Multimodal Large Language Models (MLLMs) on CVQA, and show that the dataset is challenging for the current state-of-the-art models. This benchmark can serve as a probing evaluation suite for assessing the cultural capability and bias of multimodal models and hopefully encourage more research efforts toward increasing cultural awareness and linguistic diversity in this field.

1 Introduction

Existing multilingual VQA benchmarks remain limited in language and cultural coverage, while translation-based expansion often retains Western-centric images. CVQA addresses this gap with community-collected, culturally nuanced questions and an initial evaluation of multimodal models.

1 Introduction

  • Most VQA benchmarks focus on English, while multilingual datasets often cover few popular languages or translate questions over Western-centric images.The authors identify these practices as inadequate for capturing cultural nuances inherent in different languages.Source: §1 · p. 2
  • CVQA gathers culturally nuanced questions across 30 countries and 31 languages, with 10k questions validated by fluent speakers and culturally familiar contributors.It organizes data into 39 Country-Language pairs and provides both English and local-language versions.Source: §1 · p. 2
  • Open models usually achieve no more than 50% accuracy on CVQA, and performance degrades on native-language questions, especially in understudied languages.The introduction names Breton and Javanese as examples of understudied languages where the drop is notable.Source: §1 · p. 2
  • The paper documents its crowdsourcing process and guidelines, and presents initial evaluations as a baseline for future culturally diverse vision-language research.The authors frame the work as contributing to growing interest in broadening the limited worldview represented by MLLMs.Source: §1 · pp. 2–3

2 CVQA Data Collection

CVQA collects locally grounded visual questions through collaboration with culturally familiar, fluent speakers, then organizes and validates the resulting bilingual data across Country-Language pairs. Its statistics describe coverage across locations, languages, scripts, question categories, and sources.

2.1 Dataset Collection Design

  • CVQA groups examples by Country-Language pair, capturing cases where a language spans locations or multiple languages occur within one location.Each location’s question-answer pairs are created in the local language alongside parallel English translations.Source: §2.1 · p. 3
  • Contributors are expected to be fluent in the relevant language and familiar with the location’s culture; the dataset uses 10 categories adapted from OK-VQA.The authors split Geography, History, Language and Culture into two categories and add a category for cultural icons and media.Source: §2.1 · p. 3, Table 1 · p. 3

2.2 Annotation Process

  • Annotators select culturally relevant images, create image-dependent questions with one correct answer and three plausible distractors, and provide local-language questions with manual English translations.Questions may be converted to open-ended QA; annotators are instructed to ensure they remain answerable without the choices.Source: §2.2 · p. 4
  • Each entry is checked by another annotator from the same Country-Language pair, with checks including grammatical errors, non-cultural questions, image dependence, and image sourcing.The process also includes guidance on handling sensitive information and spot-checking collected data.Source: §2.2 · p. 5

2.3 Data Statistics

  • CVQA contains 10,374 questions and spans diverse scripts and less commonly studied languages, with at least 200 questions targeted per Country-Language subset.The most frequent question form is “what”; question-type distributions also vary by subset and category.Source: §2.3 · p. 5
  • Personal images are most common in Cooking and Food and least common in Public Figures and Pop Culture.The authors suggest this may reflect the relative ease of obtaining personal food images and the scarcity of personal photos of public figures or pop culture.Source: §2.3 · p. 5

3 Experimental Setup

The evaluation compares multilingual and English-only vision-language models using zero-shot prompts on CVQA. Models are tested with country-aware or country-agnostic prompts and English or local-language question-option pairs, with accuracy as the measure.

3 Experimental Setup

  • The model set includes English-only CLIP and InstructBLIP, multilingual LLaVA-1.5, mBLIP, and M-CLIP, and closed-source GPT-4o and Gemini-1.5-Flash.The evaluated mBLIP variants are mT0-XL and BLOOMZ; the selected M-CLIP version supports 68 languages.Source: §3 · pp. 5–6
  • Zero-shot evaluation uses prompts that either specify the country or omit it, with both English and local-language question-option pairs.The location-aware prompt includes the country, question, and options; the location-agnostic prompt uses the same template without the country.Source: §3 · p. 6
  • Generative models select the option with the highest output probability, while embedding-based models select the answer with the highest image-text similarity; performance is measured by accuracy.For embedding-based models, the compared text combines the question with each candidate answer.Source: §3 · p. 6

4 Results

CVQA remains challenging for MLLMs: closed models outperform open models, and performance generally falls on local-language questions. Results also vary across question categories and image sources, while adding location information has only a marginal overall effect.

Main Results

  • 75.4% English and 74.3% local-language accuracy: GPT-4o leads, while LLaVA-1.5-7B is the best open model but trails closed models by more than 10%.LLaVA-1.5-7B also performs better on established English VQA benchmarks than on culturally specific CVQA questions.Source: Table 3 · p. 6, §4 · p. 6, Table 4 · p. 6

Performance per Country-Language

  • 60.73% English versus 51.16% Portuguese: LLaVA-1.5-7B drops on local-language questions, while models show similar behavior across most country-language pairs.The gap tends to be smaller in more frequently studied languages such as Spanish; some evaluated models do not originally support certain languages.Source: §4 · pp. 6–7, Figure 3 · p. 7

Performance Across Categories

  • 59.8%: InstructBLIP’s English accuracy is highest in People and Everyday Life, while Cooking & Food and Pop Culture perform worse, especially in local-language prompts.The authors suggest everyday activities may be better represented in visual training data; cultural diversity in food and pop culture poses a generalization challenge.Source: §4 · p. 7, Table 5 · p. 7

Impact of External Image Source

  • 48.8% versus 49.7%: LLaVA-1.5-7B’s English accuracy is lower on self-made than web images; CLIP shows the same direction, at 41.2% versus 43.1%.The authors suggest web images may better represent the training data of models such as LLaVA-1.5-7B and CLIP; the trend is not consistent across all models.Source: §4 · pp. 7–8, Table 6 · p. 8

Location-Aware vs Location-Agnostic Prompt

  • Including location information has a marginal overall effect, although its impact varies by model.The comparison is between location-aware and location-agnostic prompts.Source: §4 · p. 8, Table 7 · p. 8

Performance without Multiple Choice Options

  • 49.6% to 30%: LLaVA-1.5-7B’s average performance drops when evaluated without multiple-choice options.The open-ended setup selects the model’s highest-probability answer phrase, avoiding option-order dependence and removing options that could aid deductive reasoning.Source: §4 · p. 8

5 Limitations

CVQA offers broader cultural and linguistic coverage but remains incomplete in scope, definition, and scale. Its regional differences in question difficulty also limit direct cross-country comparisons.

5 Limitations

  • CVQA covers only a fraction of the world’s languages and regions, and its scale limits its use to benchmarking rather than training new models.The authors also note the absence of an English-centric baseline as a possible comparison.Source: §5 · p. 8
  • Because regions differ in question complexity, comparing model performance across languages or countries might not always be fair.Some regions favor simpler identity questions, while others ask questions requiring deeper cultural knowledge.Source: §5 · p. 8
  • CVQA uses local common knowledge as a proxy for culture, but this does not capture all cultural nuances.Its country-level granularity may miss city-level variation, and age may also affect common knowledge.Source: §5 · p. 8
  • The authors identify incomplete language and regional coverage, uneven question complexity, and bounded cultural definition as future-work directions outside this initial effort’s scope.Source: §5 · p. 9

6 Related Work

VQA has expanded across task formats and model generations, but existing multilingual or culturally focused datasets leave gaps in language and visual-cultural representation. CVQA is intended to address both textual and visual diversity together.

6 Related Work

  • VQA has developed from early open-ended datasets through multiple-choice, span-extraction, and free-text formats, alongside a shift from supervised systems to zero- or few-shot MLLMs.Multiple-choice datasets are described as the most commonly used, likely because evaluation and comparison are simple.Source: §6 · p. 9
  • Many VQA datasets center on English and major languages and Western imagery, while multilingual datasets remain limited in language and cultural diversity or translate existing English data.Prior VQA formats and dataset examples include [28] [29] [51]; multilingual dataset examples include [10],, [39], [6], [44], and [25].Source: §6 · p. 9
  • CVQA aims to fill the gap across text and vision with a large-scale, culturally and linguistically diverse dataset for more inclusive and robust VQA models.Culturally diverse text-only benchmarks are also noted as prior efforts [35] [19] [48] [18] [9].Source: §6 · p. 9

7 Conclusion

CVQA benchmarks culturally nuanced visual questions across languages and locations, and evaluations find challenges for open-source models. Performance is generally worse on local-language queries and without multiple-choice options.

7 Conclusion

  • CVQA is a human-written visual QA benchmark spanning cultural nuances across languages and locations, with each question written in English and a native language.This enables benchmarking multilingual visual models and English-only models.Source: §7 · p. 9
  • The dataset contains 10 question categories and provides insights into question types and commonly used terms within each category.Source: §7 · p. 9
  • Open-source models generally perform worse on local-language queries than English, and performance is considerably lower without multiple-choice options.The authors describe the no-multiple-choice setting as a more realistic use case.Source: §7 · p. 9

A Annotation Guidelines

The guidelines seek culturally representative images and questions for a linguistically comprehensive multimodal QA benchmark. They specify image, question, answer, and contributor practices for collection.

Introduction

  • The guidelines aim to support a culturally diverse and linguistically comprehensive multimodal QA benchmark, including representation of diverse cultural backgrounds.The stated purpose is to measure potential bias in visual AI models.Source: Introduction · p. 14
  • Contributors should use culturally relevant images, ensure the image is needed to answer the question, and avoid exposing personal information or answer-revealing text.Personal photos are preferred, while external images are also accepted subject to image-quality and licensing instructions.Source: Introduction · p. 15
  • Each image can have up to three culturally relevant, image-answerable questions, written fluently in both the local language and English, with one correct answer and three plausible distractors.Questions should also be answerable without the multiple-choice options and avoid cultural stereotyping.Source: Introduction · pp. 15–16
  • The guidelines encourage question variation, including more complex forms such as multi-hop reasoning, counting, referencing, and local-commonsense questions.Source: Introduction · p. 16
  • Contributors classify questions into culturally relevant categories, including local transport, clothing and objects, food, landmarks, plants and animals, brands, sports, traditions, everyday life, and pop culture.The guidelines also specify a limit of three annotators per region-language team and expect each contributor to provide at least 100 visual questions for co-authorship.Source: Introduction · p. 16

A. Mengibarkan bendera (Hoisting the flag)

The examples show CVQA questions grounded in local cultural contexts, spanning traditions, everyday life, food, landmarks, and objects across several languages and countries.

A. Mengibarkan bendera (Hoisting the flag)

A. Jawa Barat (West Java)

A. Polisi cepek (Polisi cepek)

A. Ngatur prapatan (Managing the intersection)

  • Other examples ask about worship and religious practice associated with an altar, as well as activities shown in an image.The answer options include Datuk Gong, Buddha, Brahma, and Vishnu for the deity question, and Taoism, Buddhism, Islam, and Hinduism for the religion question.Source: A. Ngatur prapatan (Managing the… · pp. 19–20

A. Masjid (Mosque)

A. Christmas Season (Christmas Season)

  • Landmark and object examples ask about a monument in Quito, Irish statues and their historical setting, and a cooking bowl’s name and material.These examples are labeled Landmark and building – Spanish/Ecuador, Landmark and building – Irish/Ireland, and Object, Clothing, and Material – Korean/South Korea.Source: A. Christmas Season (Christmas Season) · p. 21

B Annotation Platform

The annotation platform supports bilingual question entry and contributor validation; the section also describes category-level word clouds with common image-reference words excluded.

B Annotation Platform

C Most-Frequent Words in the Questions

D CVQA Annotator Demographic

Annotator demographics are summarized from an anonymous questionnaire, but the available responses cover fewer than half of the annotators and are described as a rough representation.

D CVQA Annotator Demographic

E Country-Language Pairs and Scripts

The appendices identify the scripts used for Country-Language pairs and list author affiliations.

E Country-Language Pairs and Scripts

F Affiliation Lists

Related papers

Loading 2406.05967v2…