PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Vision Language Models for Cultural Understanding

Shravan Nayak, Kanishk Jain, Rabiul Awal, Siva Reddy, Sjoerd van Steenkiste, Lisa Anne Hendricks, Karolina Stańczak, Aishwarya Agrawal

arXiv:2407.10920v3cs.CVcs.AIcs.CL

TL;DR

General scene understanding is a common focus of VLM assessment, while systematic benchmarks for cultural understanding are needed. CULTURALVQA uses images and culturally informed VQA to probe cultural concepts across countries. Evaluation reveals regional and facet disparities in model performance.

  • Problem

    VLM assessments commonly focus on general scene understanding, while evaluating cultural understanding requires systematic benchmarks that capture culture’s tangible and intangible facets.

    Source: §1 · pp. 1–2

  • Method

    CULTURALVQA extends CANDLE with cultural images and VQA questions and answers from culturally knowledgeable annotators, probing tangible concepts and shared common sense across 11 countries.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Models perform better on North American than African-Islamic cultures and excel on clothing, rituals, and traditions compared with food and drink.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The benchmark identifies specific regional and cultural-facet areas where VLMs lack uniform cultural comprehension.

    Source: §6 · p. 9

    The English-only data may miss cultural nuances available only in native languages, and geographical regions used as proxies may oversimplify cultural identities.

    Source: §7 · p. 10

Abstract

from arXiv · show

Foundation models and vision-language pre-training have notably advanced Vision Language Models (VLMs), enabling multimodal processing of visual and linguistic data. However, their performance has been typically assessed on general scene understanding - recognizing objects, attributes, and actions - rather than cultural comprehension. This study introduces CulturalVQA, a visual question-answering benchmark aimed at assessing VLM's geo-diverse cultural understanding. We curate a collection of 2,378 image-question pairs with 1-5 answers per question representing cultures from 11 countries across 5 continents. The questions probe understanding of various facets of culture such as clothing, food, drinks, rituals, and traditions. Benchmarking VLMs on CulturalVQA, including GPT-4V and Gemini, reveals disparity in their level of cultural understanding across regions, with strong cultural understanding capabilities for North America while significantly lower performance for Africa. We observe disparity in their performance across cultural facets too, with clothing, rituals, and traditions seeing higher performances than food and drink. These disparities help us identify areas where VLMs lack cultural understanding and demonstrate the potential of CulturalVQA as a comprehensive evaluation set for gauging VLM progress in understanding diverse cultures.

1 Introduction

VLMs have advanced on general scene understanding, but assessing their cultural understanding requires a systematic benchmark that covers culture’s tangible and intangible facets. CULTURALVQA provides such a benchmark, and evaluations reveal performance disparities across regions and cultural facets.

1 Introduction

  • Existing VLM assessments emphasize general scene understanding, while cultural understanding includes shared values, beliefs, rituals, and traditions across cultures.The authors argue that systematic benchmarks are needed to assess whether state-of-the-art VLMs encode cultural knowledge.Source: §1 · p. 1
  • MaRVL emphasizes visual reasoning rather than cultural common sense, while GD-VCR’s movie scenes do not cover the broader range of everyday cultural contexts.These shortcomings motivate a benchmark designed specifically to assess cultural understanding.Source: §1 · p. 2
  • CULTURALVQA combines visual and textual information in VQA to probe cultural nuances, extending CANDLE with images and questions and answers written by culturally knowledgeable annotators.The benchmark contains 2,378 questions over 2,328 unique images, with 1–5 answers per question and 7,206 answers from 11 countries.Source: §1 · p. 2
  • CULTURALVQA evaluations reveal country and facet disparities: GPT-4O achieves 67% and 72% accuracy on North American concepts versus 43%–56% on African concepts; closed-source VLMs do better on rituals and traditions than clothing, food, and drink.In Ethiopia, the country where models perform worst, the highest-performing closed-source model leads its open-source counterpart by 29.78%.Source: §1 · p. 2

2 Related work

Related datasets cover geo-diverse recognition, captioning, retrieval, grounding, or multilingual understanding, but the authors distinguish those goals from cultural understanding. They position CULTURALVQA around culturally diverse images, open-ended questions, and cultural concepts.

2 Related work

  • Dollar Street and GLDv2 test recognition, MOSAIC-1.5k supports culture-specific captioning, and GLOBALRG evaluates retrieval and grounding across countries.The authors distinguish these tasks from cultural understanding; multilingual datasets such as Bugliarello et al. (2022) and XM3600 likewise focus on language or lack depth in cultural concepts.Source: §2 · p. 3
  • MaXM builds on XM3600 and emphasizes reasoning and general image understanding; GD-VCR uses cinematic scenes and multiple choice, while MaRVL uses True/False questions and omits cultural common sense about rituals and traditions.The authors argue open-ended evaluation provides a more faithful assessment than multiple-choice formats, which can be influenced by answer-choice difficulty.Source: §2 · p. 3
  • CULTURALVQA is distinguished by culturally diverse images, open-ended evaluation, and questions designed to probe cultural understanding.The authors contrast this combination with datasets that lack culturally diverse images or use multiple-choice or True/False formats.Source: §2 · p. 3

3 CULTURALVQA: Dataset Creation

CULTURALVQA operationalizes culture through culturally relevant concepts and shared common sense, then builds a VQA benchmark with images and culturally informed questions and answers. Its collection process targets geographical diversity and filters images and annotations for cultural relevance.

3 CULTURALVQA: Dataset Creation

  • The benchmark evaluates tangible culture through food, drink, and clothing, and intangible culture through shared common sense in rituals and traditions, using a VQA task.The paper treats country as a proxy for cultural group and assumes common ground in collectively understood concepts and shared cultural reasoning.Source: §3 · pp. 3–4
  • The dataset spans 11 countries across five continents, selected to represent World Values Survey cultural categories and intentionally overrepresent African-Islamic countries.The authors cite the typical scarcity of African-Islamic countries in geo-diverse datasets as the reason for that overrepresentation.Source: §3 · p. 4
  • CANDLE supplies the initial image pool through cultural commonsense assertions and linked webpages, and CLIP similarity filters images for cultural relevance.The authors state that the initial pool reduces the risk of western-centric bias from CLIP, despite potential biases in its pretraining data.Source: §3 · p. 4
  • Annotators are asked to create questions answerable by cultural insiders but challenging for outsiders, and may skip images they find irrelevant or unfamiliar.Annotators discarded 19.64% of the images shown to them.Source: §3 · p. 4
  • Answer annotators record culturally agreed responses, retaining widely recognized local terms for festivals and cuisine instead of translating them into English.They are asked to be precise and concise, ideally using one to three words.Source: §3 · p. 4

4 Dataset Analysis

CULTURALVQA’s dataset analysis describes its country coverage, question and answer characteristics, and mix of cultural facets. It reports substantial representation of food, traditions, and rituals, alongside variation in clothing and drink answers.

Dataset Analysis

  • Figure 3 compares countries by image, question, and answer counts; question and answer lengths; answers per question; and inter-annotator agreement.Figure 4 presents answer word clouds across clothing, drink, food, rituals, and traditions, alongside a breakdown of facet representation.Source: Figure 3 · p. 5, Figure 4 · p. 5

Images

  • CULTURALVQA contains 2,328 unique images selected for cultural representation across 11 countries.The country-wise image counts are presented in Figure 3(A).Source: §4 · p. 5

Questions

  • Questions average 10.98 words, with “What” the most frequent type at 51.3%; the analysis also finds stereotypes largely absent.Examples include questions identifying garments or festivals and asking where foods originate.Source: §4 · p. 5

Answers

  • The benchmark has 7,206 manually curated answers averaging 1.73 words, with 1–5 answers per question depending on annotator availability.For 80% of answers, at least one answer word appears in an English Wikipedia title, which the authors say makes the benchmark suitable for English VLMs.Source: §4 · pp. 5–6

Cultural Concepts

  • Food questions make up 37.3% of the dataset, traditions 26.1%, and rituals 18%, so roughly 44% probe intangible cultural aspects.Clothing is least prevalent but has the greatest variety of collected answers; drink is among the smallest categories in size and unique answers.Source: §4 · p. 6

5 Evaluating VLMs on CULTURALVQA

CULTURALVQA evaluates VLMs on culturally grounded visual questions, and results show substantial differences across model types, countries, and cultural facets. Comparisons with human judgments and analyses of question properties further characterize these disparities and the benchmark’s evaluation setting.

What degree of visual understanding is required?

  • Country context and Google Lens entities improve on an LLM-only baseline, but the LLM + Lens baseline remains far below GPT-4V.The comparison uses GPT-4 as the underlying LLM and indicates that the questions require visual understanding.Source: §5 · p. 6, Figure 5 · p. 6

To what extent are VLMs culturally aware?

  • GPT-4 averaged approximately 61% LAVE accuracy versus 46% for the best open-source model; the gap reached 29.78% in Ethiopia.GPT-4 performance ranged from 43% to 72% across countries, while the best open-source model ranged from 26% to 71%.Source: §5 · p. 7, Table 2 · p. 7

Are VLMs better at understanding cultures from some countries than others?

  • Open-source models generally perform better for high-resource countries than for underrepresented countries, while closed-source models also degrade on African-Islamic countries.The country pattern persists for large open-source models such as INTERN-VL; Canada and the USA have model-human gaps below 7.0%, while several non-Western countries have gaps above 13%.Source: §5 · pp. 7–8

Are VLMs better at understanding some cultural concepts than others?

  • Proprietary models perform better on rituals and traditions than on food and drink; GPT-4 reaches approximately 63% on rituals and approximately 55% on clothing.Figure 7 reports LAVE accuracies across five cultural facets.Source: Figure 7 · p. 7, §5 · p. 8

Do multilingual VLMs perform better in culturally diverse settings?

  • Multilingual training alone does not ensure stronger or more even cultural performance: mBLIP underperforms BLIP2, and PaliGemma remains uneven across countries.The authors suggest translation quality and backbone differences as possible factors for mBLIP, and model size as a possible factor for PaliGemma.Source: §5 · p. 8

How do culturally knowledgeable people perform on CULTURALVQA?

  • Human scores range from 55% to 85%, and closed-source model scores track country-level human performance more closely than open-source scores but remain below humans.Human scores are especially high in Iran (>80%); the correlation is 0.69 for GEMINI and 0.75 for GPT-4, while open-source correlations range from 0.1 to 0.4. Rwanda and Nigeria have lower human scores, which the authors partly attribute to within-country cultural diversity.Source: Figure 6 · p. 7, §5 · p. 8

How much do question difficulty and answer counts affect disparities?

  • Question length and answer-count differences do not meaningfully explain country-level model disparities.Question-length correlations are generally weak, although GPT-4 and GEMINI show an average moderate negative correlation of -0.52; answer-count correlations range from -0.3 to 0.Source: §5 · pp. 8–9

Human judgment and qualitative failures

  • Human ratings generally align with LAVE on clearly correct answers, while GPT-4’s examples include cultural-symbol misreadings and confusion between visually similar entities.Examples include coral beads treated as decoration, a Naghali mislabeled as a Dervish, and a Turkish tea glass mistaken for a tulip glass.Source: §5 · p. 9, Figure 8 · p. 9

6 Conclusions

The paper introduces CULTURALVQA to evaluate VLMs’ cultural understanding using annotated image-question pairs from diverse countries. Its results show that performance varies across regions and cultural facets, with substantial differences between closed- and open-source models.

6 Conclusions

  • CULTURALVQA contains 2,378 hand-crafted questions and 7,206 answers about cultural concepts in images from 11 countries across five continents.The authors describe the collection as a broad representation of concepts pertinent to diverse cultural groups.Source: §6 · p. 9
  • Models perform better on North American cultures than African-Islamic ones, and the closed/open-source gap reaches 29.78% for the lowest-performing country.The conclusion also reports variation across facets, with stronger performance on clothing, rituals, and traditions than on food and drink.Source: §6 · p. 9
  • The results identify uneven cultural comprehension and specific regional and cultural areas where VLM performance needs improvement.The paper presents CULTURALVQA as a benchmark for assessing cultural understanding across diverse groups.Source: §6 · p. 9

7 Limitations

The benchmark is limited by English-language data sources and by geographic proxies that may simplify cultural identities. Its image and concept sources also constrain the cultural detail represented.

7 Limitations

  • English-only data may miss cultural nuances expressed in local terms, and 20% of answers lack a match to an English Wikipedia page.The dataset also does not capture the full spectrum of global cultural diversity.Source: §7 · p. 10
  • Using geographical regions to approximate cultural groups may conflate culture with nationality and overlook minority and diaspora communities.This approximation reflected practical constraints such as annotator availability.Source: §7 · p. 10
  • Images and concepts drawn from English-language websites likely omit important cultural details, while limited image quality affected question depth.Allowing annotators to skip inadequate images did not fully overcome these drawbacks.Source: §7 · p. 10

8 Ethical Considerations

The authors sought broad cultural representation through annotators from relevant countries and multiple communities, while recognizing that national borders and residual subjective content limit representation.

8 Ethical Considerations

  • Annotators were recruited from three communities and compensated $10–15 per hour for included and excluded contributions after pilot testing.The authors describe this as an effort to maintain fairness and inclusivity in data collection.Source: §8 · p. 10
  • Equating cultural groups with national borders may overlook the complex realities of minority and diaspora communities.The authors call for finer distinctions within cultural groups in future research.Source: §8 · p. 10
  • Some subjective content may persist despite bias-removal efforts, although a substantial portion of the dataset was verified as unbiased.The authors refer readers to Appendix C for the verification.Source: §8 · p. 10

A Data Statement

The data statement documents an English-language benchmark built from culturally relevant images and country-specific annotations. The section describes image filtering, a stereotype analysis, and the evaluated VLMs.

A Data Statement

  • Dataset texts are in English and primarily authored by non-native speakers, a choice made to separate multicultural understanding from multilingual comprehension.The images come from CANDLE, which draws on cultural commonsense entries linked to webpages; annotators came from 11 countries and were natives or long-term residents of the country they annotated.Source: A Data Statement · pp. 12–13

B Image Filtering

  • Images were filtered by aspect ratio, minimum size, and keywords, then ranked for cultural relevance using CLIP with a threshold of 23.On 200 CCSK assertions, the threshold corresponded to precision of 0.92 and recall of 0.96.Source: B Image Filtering · p. 13

C Stereotypes and Biases

  • The stereotype classifier identified 69 race, 44 gender, 22 religion, and 8 profession stereotype instances, alongside more anti-stereotypical instances in most categories.The analysis also found 923 instances unrelated to either category.Source: C Stereotypes and Biases · p. 13

D VLMs Used for Benchmarking

  • The benchmark evaluates open-source VLMs with 3–25 billion parameters alongside GPT-4o, Gemini-Pro-Vision 1.0, and Claude 3.5.The model set includes an InstructBLIP–BLIP2 comparison for instruction tuning and IDEFICS2 and InternVL 1.5 as selected larger or strong open-source models.Source: D VLMs Used for Benchmarking · p. 14

E Few-Shot Evaluation of GPT-4

Few-shot prompting GPT-4 with one example per country did not consistently improve performance over zero-shot prompting. Outcomes varied by country, with gains in some and drops or minimal gains in others.

E Few-Shot Evaluation of GPT-4

  • The evaluation used GPT-4, the best-performing model, with 11 few-shot examples—one from each country.The prompt was detailed in Appendix J.Source: E Few-Shot Evaluation of GPT-4 · p. 14
  • Few-shot prompting did not consistently improve GPT-4 over zero-shot: Germany, Ethiopia, and Nigeria gained 3–8%, while Brazil, China, India, and Rwanda had drops or minimal gains.The authors suggest performance on culturally nuanced tasks largely depends on GPT-4’s pre-existing knowledge.Source: E Few-Shot Evaluation of GPT-4 · p. 14
  • The authors describe CulturalVQA as challenging and call for more advanced methods to improve cultural-understanding performance.This conclusion follows their few-shot results, which were not uniformly beneficial across cultural contexts.Source: E Few-Shot Evaluation of GPT-4 · p. 14

F Analysis of Performance Across Cultural Facets

The analysis finds uneven model performance across cultural subcategories, while human ratings and examples point to ambiguity in some country-linked interpretations. It also examines question difficulty and pretraining-data representation as possible explanations for facet disparities.

F Analysis of Performance Across Cultural Facets

  • The analysis categorizes image-question-answer triplets into facet-specific subcategories, using GPT-4 with questions, answers, facets, category options, and examples.The categories draw on prior categorizations for traditions and rituals, and for food, drink, and clothing.Source: F Analysis of Performance Across… · p. 14, Figure 10 · p. 15
  • For Food, Drink, and Clothing, 61.8%, 61.3%, and 52.7% of samples, respectively, require detailed knowledge such as associated customs.Type or name is the most popular fine-grained category for these three facets.Source: F Analysis of Performance Across… · p. 14
  • Models perform relatively better on identifying names, locations, ingredients, celebrations, locations, landmarks, and sports, but worse on customs, taste, beliefs, and music or instruments.The comparisons cover Food, Drink, and Clothing as well as Rituals and Traditions.Source: Table 6 · p. 15, F Analysis of Performance Across… · p. 15, Table 7 · p. 16

H Human Judgment of Model Predictions

  • For India, GPT-4O and Gemini had the highest share of completely correct answers, while InternVL had a notable share of completely incorrect answers.Five annotators rated answers on a five-point scale ranging from completely correct to completely incorrect.Source: Figure 11 · p. 16, H Human Judgment of Model Predictions · p. 17

F Analysis of Performance Across Cultural Facets

  • GPT-4’s relatively better performance on traditions and beliefs is attributed to categories including celebrations, landmarks, and sports.Examples include Christmas-season items, Janaki temple, and Wushu.Source: F Analysis of Performance Across… · pp. 15–16
  • Human performance varied little by facet, from 71.1% for rituals to 74.5% for drinks, making question difficulty an unlikely explanation for model disparities.Other human scores were 74% for food, 72.7% for clothing, and 73.7% for traditions.Source: F Analysis of Performance Across… · p. 16
  • InternVL’s relative pretraining-data representation did not match its facet performance ranking, suggesting concept occurrence alone does not explain disparities.Food had the highest measured representation at 46.6%, while clothing had the highest InternVL performance.Source: F Analysis of Performance Across… · p. 16

G Qualitative Analysis of Human Performance

  • Country can be an inaccurate proxy for cultural groups: subcultures may interpret visually similar items differently, and annotators also disagreed on geographic locations.Examples include distinct local names for drums in Nigeria and several proposed crop locations in Rwanda.Source: G Qualitative Analysis of Human… · p. 17

I Behind the scenes: Journey of how CULTURALVQA came into place

The project evolved from automated dataset-building experiments to a process centered on human annotation after generated questions proved too coarse for cultural nuance. The account also details image refinement, recruitment challenges, prompts, and evaluation and annotation procedures.

I Behind the scenes: Journey of how CULTURALVQA came into place

  • An initial plan for a roughly 100-country, semi-automated dataset shifted to human annotation after GPT-4-generated questions required only coarse visual understanding and did not adequately probe cultural nuance.An early 12-country version produced above-90% GPT-4 results for India, Germany, and Poland.Source: I Behind the scenes: Journey of how… · p. 17
  • Human refinement removed 19.64% of images, while using only English Common Crawl limited coverage toward popular concepts from well-represented cultures.The authors suggest multilingual Common Crawl segments might uncover rarer cultural concepts and images.Source: I Behind the scenes: Journey of how… · pp. 17–18
  • MTurk had few active annotators for many countries; the team turned to community partnerships, while poor adherence led them to discard substantial MTurk data.The authors emphasize pilot studies, timely feedback, and a diverse annotator pool as important to data quality.Source: I Behind the scenes: Journey of how… · p. 18

J Prompt for Few-Shot Inference using GPT-4; K Prompt for VLM Inference

L System Prompt for the Evaluation Metric

  • The LAVE evaluation prompt asks an expert cultural anthropologist to rate answers as 1 or 2 for incorrect or imprecise versus correct and precise.It also requests a rationale for the rating.Source: L System Prompt for the Evaluation… · p. 19

M Inference Using Closed-Source Models

N Instructions for Human Question Generation

O Instructions for Human Answer Generation

Loading 2407.10920v3…