CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries

• VLMs struggle to interpret culturally meaningful symbols, artifacts, and gestures, in part because their training data underrepresents culturally specific material and overrepresents English-language, high-resource Western cultures.The introduction notes that existing VLM datasets are limited in scale and regional representation, while multimodal cultural data collection is especially challenging for low-resource cultures.Source: §1 · pp. 1–2
• VLMs perform best on the Americas, followed by Europe and Oceania, and weakest on Asia and Africa; CultureVerse fine-tuning narrows regional and category gaps and generalizes across cultures, concepts, continents, and datasets.The introduction also reports that fine-tuning does not significantly compromise general capabilities.Source: §1 · p. 2
• Cultural understanding generally increases with model size, though Llama 3.2-11B performs comparably to Qwen 2-72B, and larger fine-tuning datasets yield greater improvements.Source: §1 · p. 2
• CultureVerse contains 19,682 cultural concepts and 228,053 samples across 3 tasks, 188 countries or regions, and 15 cultural concepts.Its test set covers 11,085 widely recognized concepts and 31,382 corresponding samples.Source: §1 · p. 2
• The authors evaluate 16 open-source and proprietary models and introduce CultureVLM, a series fine-tuned on CultureVerse to improve cultural understanding while maintaining general capabilities.Their flexible, cost-effective data construction process supports the model series and its reported generalization.Source: §1 · p. 2

Cultural Bias in LLMs and VLMs

• Prior LLM studies found Western- or U.S.-centric alignment and limited representation of other cultures, including gaps in proverb reasoning and translation.Examples include GPT-3 responses aligning with dominant U.S. norms (Johnson et al., 2022), Western bias on Arabic text (Naous et al., 2023b), and GPT-4’s strongest CAT alignment with U.S. values (Masoud et al., 2023). Other work found ChatGPT struggled to represent cultures beyond American values, especially with English prompts (Cao et al., 2023), and multilingual LLMs had limited proverb reasoning and a translation “culture gap” (Liu et al., 2023a).Source: §2 · p. 3

Datasets and Models for Cultural Understanding

• Cultural datasets draw on surveys, existing benchmarks, Wikimedia, and social media, while other work augments data or fine-tunes culturally specific LLMs.Cost-efficient data-augmentation approaches have also been proposed for fine-tuning cultural-specific models (Li et al., 2024b,c).Source: §2 · p. 3

Cultural Understanding in VLMs

• VLM cultural-bias research remains preliminary and relies mostly on manual collection, with benchmarks covering cultural hierarchies, multilingual VQA, geographically specific content, and visual grounding.Examples include MaRVL (Liu et al., 2021), CVQA (Romero et al., 2024), CulturalVQA (Nayak et al., 2024), and GlobalRG (Bhatia et al., 2024).Source: §2 · p. 3

Limitations of Existing VLM Work

• Existing multimodal datasets often lack scale, regional and national representation, or cultural relevance, and significant development of culturally aware VLMs remains absent.Table 1 is described as showing that this paper’s benchmark contains more diverse and large-scale data than existing multimodal cultural benchmarks.Source: §2 · p. 3

3 CultureVerse: A Scalable Benchmark for VLM Cultural Understanding

• A hybrid pipeline combines automated web crawling with expert annotation across three stages: tangible concept collection, question-answer generation, and quality assurance.The approach is designed to capture diverse cultural contexts while maintaining data quality and relevance.Source: §3 · p. 4

3.1 Tangible Cultural Concept Collection

• The concept pipeline curated over 19,682 cultural concepts from 188 countries using 15 tangible categories, then retrieved five images per concept, splitting one for testing and four for training.GPT-4o extracted entities from Wikipedia documents; filtering consolidated duplicates, required at least two mentions per country, and removed overly generic or abstract concepts.Source: §3.1 · p. 4

3.2 Question-Answer Generation

• CultureVerse uses three VQA task types: image recognition, cultural knowledge questions requiring contextual knowledge, and scene reasoning in culturally specific scenarios.Knowledge questions are based on concept descriptions without naming the concept directly; scene reasoning asks models to make contextually appropriate choices.Source: §3.2 · pp. 4–5

3.3 Quality Assurance

• Over 98% of evaluation samples were correctly annotated by the automated pipeline; humans annotated the evaluation set, while automation handled the larger training set.Remaining erroneous or challenging samples that could not be refined were filtered out.Source: §3.3 · p. 5

3.4 Scalability

• Unlike prior manual approaches with limited or biased coverage, CultureVerse scales to tens of thousands of concepts across about 200 countries and can expand with more images and varied QA formats.The authors describe this as the first work to reach that scale of specific cultural concepts and country/region coverage.Source: §3.4 · pp. 5–6

4 Analysis of CultureVerse

• CultureVerse achieves an order-of-magnitude increase in countries, images, and questions compared with recent multi-model culture benchmarks.The benchmark is built around tangible, presentable cultural concepts.Source: §4 · p. 6
• 188 countries across five continents and 15 cultural topics are represented; Asia and Europe have larger datasets because concepts are collected at the country level and those regions contain more countries.North and South America are combined into America; detailed concept statistics appear in Table 1, with country and concept counts in Appendix 10.Source: §4 · p. 6
• Three question types are associated with each concept, and they aim to evaluate different abilities.Figure 2(b) illustrates the three questions associated with one concept.Source: §4 · p. 6

5.1 Experimental Setup

• The evaluation split each country or region into training and test sets, used common concepts for manually quality-checked tests, and prevented image overlap between sets.The study evaluated 14 open-source and 2 proprietary VLMs, reporting accuracy.Source: §5.1 · p. 7

5.2 Main Results

• Scene reasoning tended to yield higher accuracy than image recognition and cultural knowledge questions, as contextual text and images helped models connect visual cues with cultural meaning.The authors describe recognition and cultural knowledge as comparably challenging.Source: §5.2 · p. 7
• VLMs performed strongest on the Americas, followed by Europe and Oceania, and significantly weaker on Asia and Africa, despite CultureVerse containing the most data for Asia.The authors attribute the pattern to English-language data scarcity and cultural diversity, and report weaker understanding of history and landmarks due to limited internet data and multi-perspective imagery.Source: §5.2 · p. 7
• GPT-4o achieved the best results, but size alone did not determine performance; stepwise reasoning often impaired image recognition, with hallucinations appearing in explanations.The authors note that smaller models can show strong cultural understanding with similar training data, and that reasoning explanations may lack robust understanding of concept details.Source: §5.2 · p. 8

5.3 Training CultureVLM

• CultureVerse fine-tuning substantially improved reported models to closed-source-comparable performance; smaller training subsets caused only minimal declines, while temperature 1.0 reduced performance.Performance also improved with lower temperatures and reduced decoding diversity.Source: §5.3 · p. 8

5.4 Generalization and Robustness

• 90.8 was Asia’s same-continent score, while Europe and the Americas each scored 90.5; cross-region transfer was reasonable, and CVQA performance improved by 7% on average.Oceania fine-tuning had the lowest average performance drop, from 85.2 to 75.0. Across concept categories, same-category performance was strongest; HL-trained models averaged 87.8 versus NELR’s 84.3, and CultureVLM-HL scored 87.2 on CHT.Source: Figure 5 · p. 8, §5.4 · p. 9, Figure 6 · p. 9

5.5 Catastrophic Forgetting

• Fine-tuned CultureVLM merely influenced general VQA tasks, with comparable base and fine-tuned performance on ScienceQA and TextVQA indicating limited forgetting.The evaluation tested whether acquiring cultural knowledge diminished models’ general commonsense concepts.Source: §5.5 · p. 9, Figure 8 · p. 10

5.6 Case Study

• The case study reports that extensive explanations in CultureVerse training data enriched CultureVLM’s knowledge and enhanced its cultural recognition and understanding.Figure 7 compares responses from LLaVA and CultureVLM.Source: §5.6 · p. 9, Figure 7 · p. 9

6 Conclusion and Limitation

• CultureVerse evaluation found disparities across regions and tasks, with VLMs stronger on Western cultural contexts and weaker in underrepresented regions such as Africa and Asia.Supervised fine-tuning on CultureVerse improved cultural perception and cross-cultural generalization.Source: §6 · p. 10
• CultureVLM uses language as a proxy for cultural boundaries, although language alone does not capture culture’s full complexity.The pipeline and approach remain flexible enough to incorporate additional languages and cultures.Source: §6 · p. 10
• Fine-tuning experiments cover only the current set of models because of resource constraints, and evaluating more models could provide further insight.Source: §6 · p. 10
• CultureVerse contains only multiple-choice questions, leaving open-ended questions as a potential avenue for assessment.Source: §6 · p. 10
• The authors plan to release the data and models publicly after ethical reviews.Source: §6 · p. 10

A Details of the CultureVerse Dataset

• 19, 682 training samples and 11, 085 evaluation samples span 188 countries/regions.The evaluation set also reports concept counts by country.Source: A Details of the CultureVerse Dataset · p. 14
• The evaluation set includes 11, 085 samples from 188 countries/regions.The passage says Table 10 reports evaluation-set concept counts by country, but provides no country-specific values.Source: A Details of the CultureVerse Dataset · p. 14
• Table 2 presents collected concepts, their overall categories, and descriptions.The passage identifies Table 2 as defining the cultural concepts in the dataset.Source: Table 2 · p. 14, A Details of the CultureVerse Dataset · p. 14

B.1 Experiment Setup

• The evaluation covers open-source models from the LLaVA, LLaMA, Qwen, InternVL, Phi, MiniCPM, GLM, and Pixtral families, plus GPT-4o and Gemini-1.5-Pro.The listed open-source models include LLaVA-1.5 (Liu et al., 2024a), LLaVA-1.6-Mistral-7B-Instruct, LLaVA-OneVision (Li et al., 2024a), LLaMA-3.2-Vision, Qwen2-VL (Wang et al., 2024), InternVL-2 (Chen et al., 2023), Phi-3-Vision (Abdin et al., 2024), MiniCPM-Llama3-V-2.5 (Yao et al., 2024), GLM-4V, and Pixtral 12B (Agrawal et al., 2024); proprietary models are GPT-4o and Gemini-1.5-Pro (Team et al., 2023).Source: §B.1 · pp. 14–15
• For each model, the same inference-acceleration toolkit is used before and after fine-tuning to avoid potential toolkit effects on comparisons.lmdeploy is used for LLaVA-1.5, LLaMA-3.2-Vision, and InternVL-2; vllm is used for the other models.Source: §B.1 · p. 15
• Image-recognition questions use the questions and options directly as prompts, while cultural-knowledge and scene-reasoning questions use stepwise-reasoning prompts.The number of questions differs across tasks because cultural-knowledge and scenario-reasoning questions are more complex and GPT-4o sometimes refused to answer, preventing valid question generation.Source: §B.1 · p. 15
• Proprietary-model inference uses default hyperparameters with temperature 0 and max tokens 1,024; open-source inference uses do_sample=False, max_gen_len 512, and temperature 0.01.Source: §B.1 · p. 15
• Models are trained for one epoch with official LLaVA2, Phi3, and LLaMA4 scripts, largely following original hyperparameters with batch sizes adjusted for GPU memory.Training uses full fine-tuning on 4×A100 80GB GPUs; batch sizes are 64, 16, and 32, respectively.Source: §B.1 · p. 15
• Fine-tuning uses learning rates of 2 × 10^-5 for LLaVA-1.5, 4 × 10^-5 for Phi-3, and 1 × 10^-5 for LLaMA-3.2, with model-specific decay and warmup settings.LLaVA-1.5 uses no weight decay and cosine scheduling with a 0.03 warmup ratio; Phi-3 uses weight decay 0.01, a linear scheduler, and 50 warmup steps; LLaMA-3.2 uses no weight decay and multiplicative decay with gamma 0.85 after each epoch.Source: §B.1 · p. 15
• Training data is synthesized without large-scale human annotation using only concepts that passed GPT-4o or human quality assurance.Source: §B.1 · p. 15

B.2 Detailed Main Results

• Table 3 reports zero-shot accuracy for open-source and proprietary models across three tasks, five continents, and Cultural Heritage and Traditions (CHT), History and Landmarks (HL), and Natural Environment and Local Resources (NELR).The supplied passages identify the table’s scope but do not provide its accuracy values.Source: Table 3 · p. 15, §B.2 · p. 15

B.3 Detailed Fine-tuning Results

• Tables 5 and 6 provide fine-tuned-model generalization results across regions and categories, while Table 7 reports general VQA benchmark results before and after fine-tuning.Source: §B.3 · p. 16
• Table 4 reports fine-tuned model performance across three tasks, five continents, and three categories: CHT, HL, and NELR.CHT denotes Cultural Heritage and Traditions; HL denotes History and Landmarks; NELR denotes Natural Environment and Local Resources.Source: Table 4 · p. 16, §B.3 · p. 16
• Figure 9 reports the impact of different decoding temperatures on performance.Source: Figure 9 · p. 16, §B.3 · p. 16

C.1 Statistics of Human Annotators and the Process

• The study hired 10 expert annotators aged 20–36, all with at least a bachelor’s degree, mostly from non-AI fields such as education, languages, and history.Source: §C.1 · p. 16
• Annotators assessed correctness, consistency between questions, answers, and concepts, and relatedness between concepts and generated questions.Source: §C.1 · p. 16
• Annotators judged the material using their experience and manually checked results through Google and other search engines.Source: §C.1 · p. 16
• Each instance was labeled by two experts and then verified by another to ensure correctness.Source: §C.1 · p. 16
• All annotation operations followed local laws and regulations to ensure fairness, equity, and accountability.Source: §C.1 · p. 16

D Case Study

• Examples span Peking Opera in China, Florence Cathedral in Italy, and Pha That Luang in Laos, with image recognition, cultural knowledge, and scene reasoning questions.The examples show how the three question types are applied to cultural concepts from different countries and categories.Source: D Case Study · pp. 17–20

C.2 Accuracy of Human Annotation

• Human annotation reports high accuracy for generated questions, with incorrect questions filtered out and only correct ones retained.Table 9 presents CultureVerse accuracy based on human annotations; the passage gives no numerical values.Source: §C.2 · p. 16, Table 9 · p. 18

E Prompt List

• Concept-selection prompts require culturally distinctive, specific, indivisible symbols and reject broad categories, invalid labels, and concepts common across regions.Examples contrast broad categories such as “dance” with specific concepts such as “peking duck.”Source: E Prompt List · p. 20
• A Wikipedia-extraction prompt requests famous cultural elements across 15 categories, capped at 10 total, and uses “NA” for categories without famous elements.Categories include food, landmarks, traditional clothing, music and dance, religion, and natural scenery.Source: E Prompt List · p. 21
• Image-recognition prompts ask users to identify the concept shown in an image using a four-option multiple-choice question.The prompt specifies a question and options labeled A through D.Source: E Prompt List · p. 21
• Cultural-knowledge prompts ask image-based questions using a concept introduction, while scene-reasoning prompts pose a scenario requiring cultural knowledge without naming the concept.Both prompts require multiple-choice options and provide the correct answer; the scene-reasoning prompt also requests a reason.Source: E Prompt List · pp. 22–23
• Reasoning-response prompts ask models to describe and identify images, analyze questions and options, and state a final answer for training or evaluation.The training prompts use image features and, for cultural questions, a discreet hint; the evaluation prompt requests stepwise analysis.Source: E Prompt List · p. 25