PaperlayerView exact revision on arXiv

Source-linked AI summary

Broaden the Vision: Geo-Diverse Visual Commonsense Reasoning

Da Yin, Liunian Harold Li, Ziniu Hu, Nanyun Peng, Kai-Wei Chang

arXiv:2109.06860v1cs.CLcs.AIcs.CV

TL;DR

Commonsense benchmarks often omit cultural and geographic variation, leaving limited evidence about whether models generalize across regions. This paper introduces GD-VCR to evaluate visual commonsense reasoning across regions and finds lower performance on non-Western examples, especially for culture-related and higher-level reasoning questions.

  • Problem

    Existing benchmarks often draw on sources from particular regions and overlook regional differences, leaving the generalization of commonsense models beyond Western regions to be studied.

    Source: §1 · p. 1

  • Method

    The paper constructs GD-VCR, a benchmark of geo-diverse visual commonsense questions, and evaluates models trained on VCR against it.

    Source: §1 · pp. 1–2

  • Results

    V&L models do not generalize well to non-Western regional images, with disparity associated with culture-related scenarios and more difficult reasoning questions.

    Source: §1 · p. 2, §6 · pp. 8–9

  • Takeaways & Limitations

    The authors hope GD-VCR broadens the scope of commonsense research and motivates systems with more inclusive consideration.

    Source: §6 · p. 9

    GD-VCR omits regions including Southeast Asia, Arabic regions, and Latin America, and its African images mainly come from East Africa.

    Source: Ethical Considerations · p. 9

Abstract

from arXiv · show

Commonsense is defined as the knowledge that is shared by everyone. However, certain types of commonsense knowledge are correlated with culture and geographic locations and they are only shared locally. For example, the scenarios of wedding ceremonies vary across regions due to different customs influenced by historical and religious factors. Such regional characteristics, however, are generally omitted in prior work. In this paper, we construct a Geo-Diverse Visual Commonsense Reasoning dataset (GD-VCR) to test vision-and-language models' ability to understand cultural and geo-location-specific commonsense. In particular, we study two state-of-the-art Vision-and-Language models, VisualBERT and ViLBERT trained on VCR, a standard multimodal commonsense benchmark with images primarily from Western regions. We then evaluate how well the trained models can generalize to answering the questions in GD-VCR. We find that the performance of both models for non-Western regions including East Asia, South Asia, and Africa is significantly lower than that for Western region. We analyze the reasons behind the performance disparity and find that the performance gap is larger on QA pairs that: 1) are concerned with culture-related scenarios, e.g., weddings, religious activities, and festivals; 2) require high-level geo-diverse commonsense reasoning rather than low-order perception and recognition. Dataset and code are released at https://github.com/WadeYin9712/GD-VCR.

1 Introduction

Existing commonsense benchmarks often draw on data from particular regions and overlook regional differences; GD-VCR evaluates visual commonsense reasoning across regions. Models perform worse on non-Western examples, with larger gaps for culture-related scenarios and higher-level reasoning.

1 Introduction

  • Existing benchmarks often use region-specific sources and overlook differences across groups, motivating tests of whether models generalize beyond Western commonsense annotations.The paper illustrates regional variation with wedding attire, including red clothing and face coverings in traditional Chinese weddings.Source: §1 · p. 1
  • GD-VCR introduces 328 images and 886 QA pairs from East Asian, South Asian, African, and Western countries to evaluate geo-diverse visual commonsense.The benchmark follows the VCR task setting and uses questions requiring geo-diverse commonsense and thorough image understanding.Source: §1 · pp. 1–2
  • VisualBERT and ViLBERT reach 64%-68% accuracy on Western-region QA pairs, versus 45%-51% on East Asian images.The authors describe this as a significant performance disparity suggesting that learned commonsense does not generalize well across regions.Source: §1 · p. 2
  • The performance gap is larger for weddings, religion, and festivals, and for questions requiring higher-level commonsense reasoning with rich visual context.Performance is relatively similar across four regions on questions requiring only basic visual recognition.Source: §1 · p. 2
  • The authors call for AI systems with geo-diverse commonsense that can conduct high-level reasoning on data from different regions.Source: §1 · p. 2

2 Related Work

Prior commonsense and vision-and-language benchmarks cover varied reasoning abilities, but often underrepresent regional characteristics. GD-VCR focuses on geo-diverse commonsense rather than treating commonsense as universal.

Commonsense Reasoning Benchmarks

  • Many commonsense benchmarks use data from particular regions, while XCOPA and X-CSR samples translated from English do not reflect regional characteristics.GD-VCR instead focuses on geo-diverse commonsense rather than treating commonsense as a universal monolith.Source: §2 · pp. 2–3

Vision-and-Language Tasks

  • Existing vision-and-language tasks require visual grounding, but give little consideration to reasoning about images with regional characteristics.Source: §2 · p. 3

Geographic Bias

  • Prior work finds geographic bias in image datasets and language technologies; this paper tests for it in visual commonsense reasoning.ImageNet and OpenImages are described as amerocentric and eurocentric, with performance dropping on images from other regions.Source: §2 · p. 3

3 Benchmark Construction

GD-VCR uses a three-stage pipeline to build a visual commonsense benchmark from regionally characteristic images: annotators create questions and correct answers, then answer candidates are generated. Its question design targets geo-specific reasoning while matching VCR formats, and its reported text and image statistics are similar to VCR development data.

3 Benchmark Construction

  • The benchmark pipeline collects regionally characteristic images, builds QA pairs, then generates answer candidates using choices from VCR development data.The stages are image collection, question and answer annotation, and automatic candidate generation.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Image Collection

  • Annotators collect representative cultural scenarios from four regions, using diverse video sources such as movies, TV series, documentaries, and trailers.They label images with categorization keywords; Western images share a source with the original VCR development set and serve as a control set.Source: §3.1 · p. 3

3.2 QA Pair Annotation

  • Annotators familiar with one of the four regions write English QA pairs using detected-object labels as a guide.Mask R-CNN marks object bounding boxes before question design.Source: §3.2 · p. 3

3.2.1 Designing Questions

  • Questions use detected objects and target high-order reasoning that combines image context with geo-specific commonsense, while VCR-derived templates reduce question-format differences.The 17 templates are formed by clustering question patterns from the VCR development set; simple recognition-only questions are classified as low-order.Source: §3.2.1 · pp. 4–5

3.2.2 Annotating Correct Answers

  • Annotators provide only the correct answer, avoid plot-dependent or region-specific names, and retain QA pairs approved by at least two of three regional validators.Answers must be consistent with image contents and inferable without prior knowledge of video plots.Source: §3.2.2 · p. 5

3.3 Answer Candidate Generation

  • Each question receives three wrong choices selected from correct answers in the VCR development set, forming a four-way multiple-choice item.Candidates should be relevant to the question and dissimilar to the correct choice and one another.Source: §3.3 · p. 5

3.4 Dataset Statistics

  • GD-VCR’s OOV rate is 6.75%, versus 12.70% for VCR development, and its average detected and relevant object counts are similar to VCR development.The authors conclude the benchmark’s difficulty does not come from a vocabulary gap; the average relevant-object counts are 2.38 for GD-VCR and nearly equal for VCR development.Source: §3.4 · p. 5

4 Model Performance and Human Evaluation over GD-VCR

VisualBERT and ViLBERT perform worse on non-Western GD-VCR regions than on Western examples, with larger regional gaps on VisualBERT. Human annotators show smaller gaps across regions than the models.

4.1 Model Performance

  • VisualBERT’s Western-to-region accuracy gaps are 9.43% for South Asia, 16.84% for Africa, and 18.86% for East Asia.The authors report significantly worse model performance on non-Western regions.Source: Table 2 · p. 6, §4.1 · p. 6
  • ViLBERT outperforms VisualBERT by 6.04% on GD-VCR, and its Western-to-non-Western gaps still range from 3.70% to 17.09%.The authors conjecture that ViLBERT’s larger pre-training image coverage partly contributes to its higher performance; its regional gaps shrink but remain significant.Source: §4.1 · p. 6
  • The Western-versus-original-VCR performance gap is around 2%–6%, much smaller than the gap between Western and other regions.The authors speculate that differing image-collection requirements, including cultural elements, may add reasoning complexity.Source: §4.1 · p. 6

4.2 Human Evaluation

  • Western annotators achieve 87.93% accuracy on East Asian images, with an 8.18% regional gap versus model gaps of 18.86% and 17.09%.Human performance is much better than the models, and the authors say humans transfer commonsense more capably in geo-diverse settings.Source: §4.2 · p. 7

5 Analyses of Performance Disparity

The analysis associates wider regional performance gaps with culturally marked scenarios and higher-order questions, while low-order recognition shows smaller disparities. It also finds smaller gaps for scenarios with more similar visual forms across regions.

5.1 Regional Differences of Scenarios

  • 8.28%–23.69%: gaps are larger for scenarios with regional characteristics, including weddings, religion, festivals, and customer scenarios.The authors speculate that customer-scenario gaps reflect regional differences in shopping, such as local street merchants versus supermarkets and restaurants.Source: §5.1 · p. 7, Figure 3 · p. 7
  • 0.42%, 1.29%, and 1.12%: party, restaurant, and student scenarios have smaller gaps, and the authors note their visual forms are more similar across regions.The authors suggest that common party activities such as drinking, dancing, and celebrating may contribute to stronger transfer on party scenarios.Source: §5.1 · p. 7

5.2 Reasoning Level of QA Pairs

  • The reasoning-level analysis adds low-order questions because GD-VCR itself consists of high-order questions; these were annotated on 30 randomly selected images per region.The added sets contain 22 Western, 26 East Asian, 16 South Asian, and 22 African QA pairs.Source: §5.2 · p. 8
  • For low-order questions, performance is around 60% across regions, and the East Asia–Western gap narrows from about 19% overall to 6.41%.The authors interpret this as VisualBERT showing more similar ability to process basic visual information across regions.Source: §5.2 · p. 8
  • On African images, low-order QA accuracy exceeds high-order accuracy by 15.71%.Low-order questions target basic visual information, while high-order questions involve more advanced information; the comparison uses the same images for both types.Source: §5.2 · p. 8, Table 3 · p. 8
  • 26.25%: the Western–African gap on high-order questions, versus 9.09% on low-order questions; East Asia’s gap rises from 6.41% to 16.13%.The reported pattern is that regional gaps are much larger on high-order than low-order QA pairs.Source: §5.2 · p. 8, Table 3 · p. 8

6 Conclusion

The paper introduces GD-VCR to evaluate vision-and-language models on visual commonsense questions involving geographic diversity. Experiments show weaker generalization to non-Western regional characteristics, with disparity associated with culturally marked scenarios and question reasoning difficulty.

6 Conclusion

  • GD-VCR is a benchmark for evaluating vision-and-language models’ reasoning on QA pairs involving geo-diverse commonsense knowledge.The paper presents it as a new benchmark for this evaluation.Source: §6 · p. 8
  • Models do not generalize well to images involving non-Western regional characteristics, and disparity is associated with weddings, religion, festivals, and QA reasoning difficulty.The conclusion bases the scenario and reasoning-level findings on VisualBERT results.Source: §6 · pp. 8–9
  • The authors hope GD-VCR will broaden research attention to the scope of commonsense reasoning and motivate more inclusive consideration in commonsense reasoning systems.Source: §6 · p. 9

Ethical Considerations

The paper describes dataset ethics and supporting annotation procedures, including a stated geographic-coverage limitation. Its appendices detail image selection, question templates, a relevance model, and answer-candidate selection.

Ethical Considerations

  • GD-VCR may retain geographic bias: it omits Southeast Asian, Arabic, and Latin American regions, and its African images mainly come from East Africa.The authors note that beliefs and behaviors can also vary within a region, and call for finer-grained analysis before commercial use.Source: Ethical Considerations · p. 9

A.1 Image Collection

  • Image selection required at least two people, clear image quality, and videos that were free to access and had no copyright issues.Video names, screenshot timestamps, and links were collected to help annotators understand image content with video context.Source: §A.1 · p. 12

A.2 QA Pair Annotation

  • Annotators were encouraged to use varied question templates and avoid overly simple yes/no answers.The templates were summarized from original VCR development questions using K-Means clustering, capped at 20 clusters and 200 iterations.Source: §A.2 · p. 12

B.1 Relevance Model

  • The relevance model is a BERT-base binary classifier trained on original VCR QA pairs to estimate question–answer relevance.Relevant pairs use the corresponding right answer; irrelevant pairs use randomly sampled answers, and the relevance score is the probability of a relevant pair.Source: §B.1 · p. 12

B.2 Pseudo Code of Answer Candidate Generation Algorithm

  • Answer candidates are selected to be relevant to the question while dissimilar to the correct answer and previously selected candidates.The candidate pool consists of correct answers from the original VCR development set; each question receives three wrong choices.Source: §B.2 · p. 12

C Implementation Details of Fine-tuning VisualBERT and ViLBERT

The implementation uses VisualBERT and ViLBERT models pretrained on different sources and fine-tuned on the original VCR training set. The supplied table captions identify scenario-accuracy and question-template materials.

C Implementation Details of Fine-tuning VisualBERT and ViLBERT

  • VisualBERT has 135.07M parameters and ViLBERT has 252.15M parameters; their starting models were pretrained on COCO and Conceptual Captions, respectively.Both were then fine-tuned on the original VCR training set.Source: C Implementation Details of… · pp. 12–13
  • Table 4 covers accuracy on images involving the same scenarios from different regions.Source: Table 4 · p. 13
  • Table 5 lists question templates summarized from the original VCR development set.Source: Table 5 · p. 13
  • Both models were fine-tuned for 8 epochs at learning rate 2e −5, with batch sizes 32 for VisualBERT and 16 for ViLBERT.The selected checkpoint was the epoch with the best performance on the original VCR development set; one epoch took 5.28 hours for VisualBERT and 6.75 hours for ViLBERT.Source: C Implementation Details of… · p. 13

D Accuracy on the QA Pairs Involving Specific Scenarios

This section compares VisualBERT accuracy on selected recurring scenarios in GD-VCR and VCR, reporting a particularly large performance gap for funeral-related images.

D Accuracy on the QA Pairs Involving Specific Scenarios

  • The scenarios were selected because they appear frequently in both GD-VCR and the original VCR development set.Source: D Accuracy on the QA Pairs Involving… · p. 13
  • Nearly 25% performance gap appears on images involving funeral scenarios, and the results show difficulty with QA pairs involving cultural differences.The comparison concerns VisualBERT’s accuracy on scenarios represented in non-Western GD-VCR regions and the original VCR dataset.Source: D Accuracy on the QA Pairs Involving… · p. 13, Table 6 · p. 13
  • Table 6 reports accuracy (%) for the same scenarios in the original VCR dataset and non-Western regions of GD-VCR.Source: Table 6 · p. 13

E Keywords in GD-VCR Dataset

GD-VCR keyword statistics indicate broad scenario diversity, with frequent keywords for scenarios that differ evidently by region.

E Keywords in GD-VCR Dataset

F More Examples in GD-VCR Dataset

The examples show regional characteristics in GD-VCR images about customers and funerals or death, alongside VisualBERT predictions for the QA pairs.

F More Examples in GD-VCR Dataset

  • Examples extend beyond wedding images to customer and funeral or death scenarios drawn from the four studied regions.The accompanying figures show regional characteristics in the selected images and visualize VisualBERT predictions.Source: F More Examples in GD-VCR Dataset · p. 13, Figure 5 · p. 14, Figure 6 · p. 15
  • Figure 5 orders customer examples Western, South Asia, then East Asia, and marks correct choices in blue and incorrect predictions in red.The rightmost value gives VisualBERT’s probability for each choice; the model was fine-tuned on original VCR training data.Source: Figure 5 · p. 14
  • Figure 6 places Western and East Asian funeral or death examples in its first row, and South Asian and African examples in its second row.Source: Figure 6 · p. 15
Loading 2109.06860v1…