PaperlayerView exact revision on arXiv

Source-linked AI summary

From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models

Mehar Bhatia, Sahithya Ravi, Aditya Chinchure, Eunjeong Hwang, Vered Shwartz

arXiv:2407.00263v1cs.CLcs.AIcs.CV

TL;DR

Western-heavy pretraining data and limited cultural coverage in existing evaluations leave VLMs’ multicultural understanding insufficiently assessed. GLOBALRG evaluates universal-concept retrieval and culture-specific visual grounding across many countries, finding substantial cross-cultural performance variation. The authors point toward more culturally diverse training data and objectives.

  • Problem

    Western-heavy VLM training data are associated with cross-cultural performance disparities, while precision-based retrieval evaluation overlooks the cultural diversity of retrieved images.

    Source: §1 · p. 1, §3 · p. 3

  • Method

    GLOBALRG evaluates retrieval of culturally diverse images for universal concepts across 50 countries and grounding of culture-specific concepts across 15 countries.

    Source: §1 · pp. 1–2

  • Results

    Across cultures, grounding performs better in North America and Europe, while retrieval preferences vary by universal and may retain Western elements even in seemingly diverse results.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors call for culturally diverse training data and training objectives that enhance models’ representations of images from diverse cultures.

    Source: §5 · p. 8

    Grounding covers only 15 cultures because of annotator availability on the crowdsourcing platform used.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Despite recent advancements in vision-language models, their performance remains suboptimal on images from non-western cultures due to underrepresentation in training datasets. Various benchmarks have been proposed to test models' cultural inclusivity, but they have limited coverage of cultures and do not adequately assess cultural diversity across universal as well as culture-specific local concepts. To address these limitations, we introduce the GlobalRG benchmark, comprising two challenging tasks: retrieval across universals and cultural visual grounding. The former task entails retrieving culturally diverse images for universal concepts from 50 countries, while the latter aims at grounding culture-specific concepts within images from 15 countries. Our evaluation across a wide range of models reveals that the performance varies significantly across cultures -- underscoring the necessity for enhancing multicultural understanding in vision-language models.

1 Introduction

Large-scale VLM training data overrepresents Western cultures, and existing cultural-inclusivity benchmarks cover few cultures while omitting diversity in universal concepts. GLOBALRG addresses these gaps with two complementary tasks, and its evaluation finds cross-cultural performance differences.

1 Introduction

  • Large-scale VLM pretraining datasets predominantly contain Western-culture images, and this underrepresentation translates into performance disparities across cultures.Source: §1 · p. 1
  • Existing cultural-inclusivity benchmarks cover only 5–7 cultures and omit assessment of cultural diversity in universal concepts..Source: §1 · p. 1
  • GLOBALRG pairs retrieval of culturally diverse images for universal concepts across 50 countries with grounding of culture-specific concepts across 15 countries.Retrieval also adds diversity@k to measure cultural diversity among results alongside precision@k for concept correctness.Source: §1 · pp. 1–2
  • Evaluation of 7 retrieval and 5 grounding models finds cross-cultural discrepancies; grounding performance is higher in North America and Europe, while retrieval preferences vary across universals.Even seemingly diverse retrievals can share Western elements, such as eggs for breakfast or white dresses at weddings.Source: §1 · p. 2
  • The benchmark identifies gaps in current VLMs’ cultural awareness and calls for more inclusive performance across culturally diverse concepts and images.Source: §1 · p. 2

2 Related Work

Prior work links Western overrepresentation in vision-language datasets to geographic performance differences, while existing datasets and benchmarks leave gaps in cultural and conceptual coverage. GLOBALRG combines broad cultural coverage with tests of both culture-specific concepts and universal concepts.

2 Related Work

  • Large-scale vision-language datasets overrepresent the West, and image-generation models may depict universal concepts such as houses using forms prominent in US or Indian training data.Source: §2 · p. 2
  • Popular geo-diverse image datasets focus on common household items and lack abstract and culture-specific concepts; domain adaptation and active learning have also been proposed for cross-cultural data collection.Source: §2 · p. 2
  • Multimodal benchmarks assess VLMs on tasks including visual question answering, reasoning, retrieval, grounding, captioning, and cultural adaptation.Source: §2 · p. 2
  • GLOBALRG covers 50 cultures for retrieval and 15 for grounding, testing both culture-specific concepts and cultural diversity in universal concepts.Source: §2 · p. 2

3 Task 1: Retrieval across Universals

Retrieval across Universals tests whether vision-language models can retrieve culturally diverse images for universal concepts, balancing query relevance with cultural diversity. Results show cultural preferences that vary by concept and model, despite strong country-level diversity scores.

3.1 Dataset Collection

  • Retrieval targets visually depictable human universals across 50 cultures, using CANDLE-derived cultural sentences to gather and manually screen Google Images results.The dataset contains 3,000 images (50 cultures × 20 universals × 3 images).Source: §3 · p. 2, §3.1 · p. 3

3.2 Task Definition and Evaluation Setup

  • The ranked list is intended to maximize relevance and cultural diversity, measured by precision@k and entropy-based diversity@k at country and regional levels.Diversity@k uses the distribution of cultures among the top-k retrieved images.Source: §3.2 · p. 3

3.4 Results and Analysis

  • CoCA leads relevance by 5 points; all models have high country diversity but lower regional diversity, while TCL is robust on both despite training on 4M images.BLIP-2 has the highest country diversity@5; OpenCLIP demonstrates superior cultural diversity to CLIP.Source: Table 3 · p. 4, §3.4 · p. 4

3.3 Models

  • The retrieval evaluation spans dual encoders, encoder-decoders, and fusion encoders, but excludes GPT-4v and Gemini because they cannot search the large image collection.The evaluated models use varied pre-training objectives, including contrastive loss, image-text matching, and masked language modelling.Source: §3.3 · p. 4, §3.4 · p. 4

3.4 Results and Analysis

  • Cultural preferences are universal-specific rather than consistent: breakfast and funeral results lean Western, while farming images differ by model.CLIP and BLIP-2 mostly retrieve Western farming images, whereas CoCA retrieves images from Africa and the Middle East.Source: §3.4 · pp. 4–5

4 Task 2: Cultural Visual Grounding

Cultural Visual Grounding extends generic grounding evaluation to culture-specific concepts, which existing datasets largely lack cultural contexts. It tests whether models can identify those concepts in images.

4.1 Dataset Collection

  • The dataset covers 15 countries across 8 regions, with 50 cultural keywords per culture; annotators selected images with other visual content and boxed each target.Source: §4.1 · p. 6
  • Images were filtered for a bbox-to-image ratio below 0.3 and at least three detected objects; the final collection contains 591 images, and 1.3% of concepts were marked not prevalent.Human evaluation measured annotator agreement using Intersection over Union between independently drawn boxes.Source: §4.1 · p. 6

4.2 Task Definition and Evaluation Setup

  • A predicted box counts as correct when its IoU with the gold box exceeds 0.5; the task reports overall accuracy based on predicted and gold regions.Source: §4.2 · p. 7

4.3 Models

  • The benchmark compares specialist Grounding DINO with generalist MLLMs, which generate bounding boxes as text; Table 5 summarizes training data, backbones, and methodology.Source: Table 5 · p. 6, §4.3 · p. 7

4.4 Results and Analysis

  • 47.99% average accuracy: Grounding DINO outperformed generalist models, though overall cultural-grounding performance was poor.Source: Figure 3 · p. 7, §4.4 · p. 7
  • 64.61% average accuracy on North America exceeded 46.99% in Latin America and 44.49% in Europe; almost all models performed best on North America.Source: Figure 4 · p. 7, §4.4 · p. 7
  • Generalists scored 18.75-27.5% in South East Asia and 31.11-35.08% in East Asia; all performed poorly on Vietnam.KOSMOS-2 and QwenVL-7B performed well on Canada and Mexico, while Grounding DINO performed well on Poland and the Philippines.Source: §4.4 · p. 7
  • Errors included boxes around unrelated objects, such as people mistaken for a Philippine bayong, and shape-similar objects, such as an arm mistaken for Nigerian ogene.Source: Figure 5 · p. 8, §4.4 · p. 8

5 Conclusion

GLOBALRG evaluates multicultural understanding through retrieval of culturally diverse images for universal concepts and grounding of culture-specific concepts. Experiments find substantial performance variation across cultures.

5 Conclusion

  • GLOBALRG benchmarks retrieval of culturally diverse images for universal concepts and visual grounding of culture-specific concepts.Source: §5 · p. 8
  • Experiments across vision-language models reveal significant performance variation across cultures, indicating biases in current models.Source: §5 · p. 8
  • The authors call for large-scale culturally diverse training datasets and training objectives that improve representations of images from diverse cultures.Source: §5 · p. 8

Limitations

The authors identify limitations in cultural coverage, the selected concepts, and the metric used to assess diversity.

Limitations

  • The grounding task covers only 15 cultures because of annotator availability, while future work aims to broaden that coverage.The retrieval task covers 50 cultures.Source: Limitations · p. 9
  • The study uses selected CANDLE concepts or keywords, which may omit prominent cultural concepts and limit evaluation comprehensiveness.The authors note that culturally significant aspects absent from the selected keywords may be overlooked.Source: Limitations · p. 9
  • The entropy-based diversity metric may miss nuanced cultural representation, and regional assessment may overlook finer within-region distinctions.Source: Limitations · p. 9

Ethical Consideration

The paper describes country-to-region mapping as a practical starting point and notes constraints and ethical concerns in data collection.

Ethical Consideration

  • Countries are mapped to broad regional categories, although cultures do not follow geographic boundaries and vary with individual experience.The authors describe this mapping as preliminary and aim for systems that adapt to diverse, evolving cultures through user interactions.Source: Ethical Consideration · p. 9
  • Cloud Research annotators were predominantly based in six named Western countries and had to meet ethnicity and lived-experience criteria.Annotators had to have lived in designated countries for at least 5 of the past 15 years; a separate second-round group conducted human evaluation.Source: Ethical Consideration · p. 9
  • Some collected images might inadvertently perpetuate stereotypes despite the goal of gathering authentic cultural representations.Source: Ethical Consideration · p. 9

A.1.1 Results Across All Metrics

The appendix reports results across models for each universal and metric.

A.1.1 Results Across All Metrics

  • Tables 6 and 7 report results across all models for each universal and metric.Source: §A.1.1 · p. 12

A.1.2 Results Across All Countries

The appendix catalogs retrieval results by country and region, cultural keywords, and model checkpoints.

A.1.2 Results Across All Countries

A.1.3 Results Across All Regions

A.2 List of Cultural Keywords in Cultural Visual Grounding dataset

A.3 Model Checkpoints

  • The appendix lists checkpoints for CLIP, OpenCLIP, CoCa, and LLaVA.Source: §A.3 · p. 12
Loading 2407.00263v1…