PaperlayerView exact revision on arXiv

Source-linked AI summary

World in a Frame: Understanding Culture Mixing as a New Challenge for Vision-Language Models

Eunsu Kim, Junyeong Park, Na Min An, Junseong Kim, Hitesh Laxmichand Patel, Jiho Jin, Julia Kruk, Amit Agarwal, Srikant Panda, Fenal Ashokbhai Ilasariya, Hyunjung Shim, Alice Oh

arXiv:2511.22787v2cs.CV

TL;DR

Cultural VQA benchmarks have focused on single-context scenes, leaving LVLM behavior in culturally mixed settings underexplored. The paper introduces CultureMix, a food VQA benchmark, and evaluates 10 LVLMs across mixed-scene conditions. Performance falls in mixed contexts, background cues exert particular influence, and the tested robustness strategies yield gains.

  • Problem

    Existing cultural VQA benchmarks depict single-culture scenes, leaving LVLM understanding of mixed cultural contexts underexplored.

    Source: §1 · p. 2

  • Method

    CultureMix is a food VQA benchmark with 23k synthetic images and 100 real-world images that tests food-name and cultural-origin recognition amid varying food and background distractors.

    Source: §1 · p. 2

  • Results

    Food identification declines by 1–8%p and cultural-origin prediction by 1–14%p in mixed settings; background cues exert stronger influence than food distractors, and declines intensify with cultural distance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Training-free methods and supervised fine-tuning yield consistent gains in accuracy and consistency, while optimal training objectives for culture mixing remain an open question.

    Source: §1 · pp. 2–3

    The benchmark focuses primarily on food and scene recognition, with underexplored underrepresented or low-resource cultural contexts.

    Source: Limitations and Future Work · p. 9

Abstract

from arXiv · show

In a globalized world, cultural elements from diverse origins frequently appear together within a single visual scene. We refer to these as culture mixing scenarios, yet how Large Vision-Language Models (LVLMs) perceive them remains underexplored. We investigate culture mixing as a critical challenge for LVLMs and examine how current models behave when cultural items from multiple regions appear together. To systematically analyze these behaviors, we construct CultureMix, a food Visual Question Answering (VQA) benchmark with 23k diffusion-generated, human-verified culture mixing images across four subtasks: (1) food-only, (2) food+food, (3) food+background, and (4) food+food+background. Evaluating 10 LVLMs, we find consistent failures to preserve individual cultural identities in mixed settings. Models show strong background reliance, with accuracy dropping 14% when cultural backgrounds are added to food-only baselines, and they produce inconsistent predictions for identical foods across different contexts. To address these limitations, we explore three robustness strategies. We find supervised fine-tuning using a diverse culture mixing dataset substantially improve model consistency and reduce background sensitivity. We call for increased attention to culture mixing scenarios as a critical step toward developing LVLMs capable of operating reliably in culturally diverse real-world environments.

1. Introduction

Culture-mixing scenes challenge LVLMs because existing cultural VQA benchmarks largely depict a single cultural context. The paper introduces CultureMix to evaluate this gap and finds that mixed cultural cues impair recognition, while robustness strategies offer gains.

1. Introduction

  • Cultural VQA benchmarks depict single-context scenes, leaving open whether models can distinguish and integrate conflicting cultural cues in mixed settings.The authors note that models trained on single-culture data may not generalize effectively to such contexts.Source: §1 · p. 2
  • CultureMix comprises 23k synthetic images and 100 real-world images that test food-name and cultural-origin recognition amid food and background distractors.The benchmark varies distractor types and cultural distance to examine their influence on model behavior.Source: §1 · p. 2
  • Food identification declines by 1–8%p and cultural-origin prediction by 1–14%p in mixed settings, with larger declines as cultural distance increases.Background cues exert stronger influence than food distractors and can shift predictions toward the distractor’s cultural source.Source: §1 · p. 2
  • Training-free methods and supervised fine-tuning yield consistent gains in accuracy and consistency, while the optimal culture-mixing training objective remains open.The authors identify robustness improvement as a direction for further work.Source: §1 · pp. 2–3

2. CultureMix

CultureMix evaluates how LVLMs identify a target food and its country of origin when other cultural elements appear in the image. Its four task conditions vary food and background distractors, while its construction and statistics support comparisons across cultural combinations and distances.

2. CultureMix

  • CultureMix is a benchmark for evaluating LVLM cross-cultural awareness in culturally mixed contexts.The section introduces the task, dataset construction pipeline, and dataset statistics.Source: §2 · p. 3

2.1. Task Overview

  • The VQA task asks models to identify a target food and its country of origin under four distractor conditions: none, food, background, or both.For food-pair tasks, the target is positioned on the left; pairings vary by country combination and cultural distance.Source: §2.1 · p. 3

2.2. Dataset Construction Pipeline

  • The construction pipeline uses diffusion-model editing on food and background seed images, with human verification across generation steps.Seed foods come from 247 images across 30 countries; the pipeline removes original backgrounds for the single-food set, combines two such images for multiple-food scenes, and harmonizes background scenes.Source: §2.2 · p. 3, , MFB · p. 4

, MFB

  • For SFB and MFB, the pipeline combines food images with backgrounds and applies diffusion-based image harmonization.When initial generation is unsatisfactory, the authors apply the editing models multiple times until images meet their criteria.Source: , MFB · pp. 3–4

2.3. Dataset Statistics

  • Food distractor pairs span three geographic-distance levels, and the dataset includes both easy and hard cases based on three baseline models.The 948 pairs span 30 countries; SFB contains 12,350 images and MFB 9,480, with MFB backgrounds randomly sampled from each continent.Source: §2.3 · p. 4

3. Experiments

CultureMix evaluates food and country identification across ten LVLMs, with conditions that vary cultural distractors. Performance declines in mixed settings, and country predictions shift away from the single-food pattern.

3.1. Experimental Setup

  • The VQA evaluation asks models to identify a target food and its associated country, measuring food-name and country accuracy across ten LVLMs.Food scoring uses similarity matching; country scoring uses exact string matching with known country-name variations.Source: §3.1 · p. 4

3.2. Overall Prediction Accuracy on CultureMix

  • Across models, performance follows SF ≳ MF > MFB ≳ SFB; backgrounds produce 13% larger country-accuracy drops and 7% larger food-name-accuracy drops than food distractors.Gemini and GPT substantially outperform the open models across subtasks.Source: §3.2 · p. 5

3.3. Country Prediction Patterns

  • In SF, models more often identify countries correctly, but errors commonly stay within continents and skew toward high-resource countries such as India, China, and the United States.African and Asian countries are often mislabeled as India or China, while European and North American countries are often mislabeled as the United States.Source: §3.3 · p. 5
  • In mixed settings, country accuracy declines and predictions spread more broadly; SFB predictions cluster around countries such as Mexico or Japan.This differs from SF, where predictions are biased toward high-resource countries.Source: §3.3 · p. 5

4. Discussion

The discussion examines how cultural distractors and their distance from the target affect LVLM predictions, and how these effects compare with culturally agnostic objects. It also relates single-food confidence to robustness under culture mixing.

4. Discussion

  • The analysis uses accuracy, prediction entropy for confidence and consistency, and label shift to measure changes from the SF baseline in mixed settings.Label shift is the proportion of predictions that differ from the SF baseline.Source: §4 · p. 5

4.1. Effects of Cultural Distractors on LVLMs

  • Across models, an average 15% of predictions shift to the distractor’s country and another 12% to a country on the same continent; backgrounds exert stronger influence than food distractors.When food and background cues align, they reinforce predictions; when they conflict, shifts increase.Source: §4.1 · pp. 5–6

4.2. Effect of Elements’ Cultural Distance on Prediction

  • Accuracy is highest and entropy lowest for same-country target–distractor pairs, and worst for pairs from different continents, with improvement as cultural distance decreases.The pattern also appears in SF and MFB settings and in food-name identification.Source: §4.2 · p. 6, Figure 5 · p. 6

4.3. Comparison Between Cultural and Culturally-Agnostic Distractors

  • Cultural distractors generally yield similar or lower accuracy and higher entropy than culturally agnostic objects; backgrounds have the strongest effects.The authors attribute the shifts to cultural signals rather than visual content alone.Source: §4.3 · p. 6, Figure 6 · p. 6

4.4. Relationship Between Cultural Awareness of Single Foods and Multi-Foods

  • Lower confidence on single foods generally corresponds to larger label shifts, but low entropy does not guarantee robustness because competing visual cues also matter.The analysis relates SF entropy to label shifts under MF–MFB culture mixing.Source: §4.4 · p. 6, Figure 7 · p. 6

5. CultureMix-Real: Real-World Culturally Mixed Food Dataset

CultureMix-Real tests whether behaviors observed in synthetic scenes also appear in natural images. Its real-world results show that the cultural relationship between co-occurring foods affects model performance.

5.1. Dataset Collection

  • The 100 real-world MF images comprise 50 same-culture and 50 cross-culture food combinations from ten countries, gathered through web searches and consented participant submissions.Cropping the food regions yielded 219 single-food images for comparing isolated and mixed contexts.Source: §5.1 · pp. 6–7

5.2. Experiment and Results

  • In real-world images, performance follows MF (same) > SF > MF (diff), confirming that synthetic-scene degradation also appears in natural culture-mixing scenarios.The target food was marked with a red bounding box for identification.Source: §5.2 · p. 7, Figure 8 · p. 7

6. Improving Understanding of Culture Mixing: An Exploratory Study

The exploratory study tests prompting and supervised fine-tuning as ways to improve LVLM performance in culture-mixing settings. Fine-tuning improves results across most tasks, while chain-of-thought prompting can worsen consistency.

6.1. Approaches for Improvement

  • The study tests direct target-focused prompting, CoT prompting, and SFT on culturally mixed images, with SFT presenting each food in SF–MF–SFB–MFB order.The exploratory experiments use Ovis2.5-9B and InternVL3-8B; training uses 5K images and evaluation uses a 7K-image test set.Source: §6.1 · p. 7, §6 · p. 7

6. Improving Understanding of Culture Mixing: An Exploratory Study

  • CultureMix is presented as the first benchmark to explicitly include culture mixing, with diverse foods and backgrounds across 30 countries in English and both real and synthetic images.The study frames its central finding as a need to go beyond single-item recognition because confidence on isolated foods does not ensure success in mixed contexts.Source: §6.1 · p. 7, Table 3 · p. 8

6.2. Results

  • Only SFT achieves statistically significant gains (paired T-test, p < .01); on SFB and MFB it reduces entropy and increases accuracy.Direct prompting is marginally more effective for robustness in MF, while SFT is more effective as contexts become complex, especially with backgrounds.Source: Table 2 · p. 7, §6.2 · p. 8
  • CoT prompting sometimes improves accuracy but can fail to improve or degrade label consistency, often by magnifying reliance on background cues.The paper relates this pattern to research reporting that CoT can overamplify misleading cues in cultural and social tasks [32].Source: §6.2 · p. 8
  • The authors call for training objectives designed for culture mixing, noting that preliminary SFT on mixed scenarios yields meaningful improvements.The optimal training objectives remain an open question.Source: §6.2 · p. 8

7. Related Work

Prior cultural-awareness benchmarks generally use single-culture scenes, while image-composition research has largely used composition to enhance model capabilities. CultureMix instead uses co-occurring cultural elements to evaluate cross-cultural ambiguity.

7.1. Cultural Awareness of LVLMs

  • Existing VQA and culturally specific datasets generally depict single-cultural scenes, while Kim et al. examine ethnicity-artifact intersections rather than multiple cultural artifacts coexisting in one scene.The paper positions CultureMix as addressing this gap with a benchmark for culture-mixing scenarios.Source: §7.1 · p. 8
  • Existing datasets do not explicitly target culture mixing, which CultureMix systematically represents with diverse foods and backgrounds across 30 countries and real and synthetic images.The authors describe it as, to their knowledge, the first benchmark using these scenarios to study cross-cultural ambiguity and reasoning.Source: §7.1 · p. 8

7.2. Image Fusion and Composition

  • CultureMix repurposes image composition from capability enhancement toward creating challenging cultural combinations for vision-language model evaluation.Prior work used composition for synthetic data creation and augmentation, while generative models enabled realistic combinations of concepts and styles [45] [46] [47] [49].Source: §7.2 · p. 9

8. Conclusion

The paper evaluates LVLMs in culturally entangled food and scene contexts and presents training-free and training-based approaches. It reports that cultural distractors reduce performance relative to single-food images and calls for interventions to improve cross-cultural understanding.

8. Conclusion

  • LVLM performance decreases from single-food images when cultural distractors appear, and the paper presents training-free and training-based approaches to guide future work.The conclusion frames these findings and approaches as a foundation for LVLMs that reason more effectively across culturally diverse contexts.Source: §8 · p. 9

Limitations and Future Work

This section documents the benchmark’s scope, human validation and image-generation process, while noting limits in cultural coverage and evaluated domains.

Limitations and Future Work

  • CultureMix covers global cuisines and scenes, but underrepresents low-resource regions and leaves festivals, clothing, daily interactions, and diverse user-centered evaluation for future work.The evaluation focuses on food and scene recognition because diverse, high-quality data are challenging to collect and these elements are relatively less controversial.Source: Limitations and Future Work · p. 9

A.1. Country Lists and Statistics

A.2. Human Validation and Statistics

  • Human validation iteratively rejected or regenerated images that failed quality criteria until the dataset was completed.For background images, criteria included a missing background, unnatural food placement, or changes to the food or background; SF checks also covered cropping, tableware, extra foods, and unnatural appearance.Source: §A.2 · p. 14
  • 109/295 SF cases required regeneration, 12/295 required food-image replacement, and 1/295 had text present.For MF images, 74/948 cases were regenerated; validation also checked cropping and excessive size imbalance between foods.Source: §A.2 · p. 14
  • 0.72, 0.70, and 0.88 were the valid/total rates for attempts one, two, and three, respectively.The remaining cases after three attempts were visually inspected and regenerated until satisfactory.Source: §A.2 · p. 14

A.3. Prompts and Model Configurations for Synthetic Dataset Generation

  • The image-generation appendix summarizes model configurations and prompts, including minor prompt variants used when initial outputs failed annotator criteria.Table 5 is identified as the overview of synthesis models and settings.Source: §A.3 · p. 14, Table 5 · p. 19
  • Figure 11 presents qualitative examples of synthesis failures that were filtered out during human validation.The examples illustrate incomplete outputs or images misaligned with the textual prompt.Source: §A.3 · p. 14, Figure 11 · p. 17

A.4. Sample Dataset Images

The appendix illustrates generated culture-mixing images and a complementary set of naturally occurring food pairings, showing variation in cultural combinations and visual complexity.

A.4. Sample Dataset Images

  • Generated examples combine foods from different countries with varied geographic backgrounds and visual configurations.Figure 14 presents rows showing foods from different countries placed in diverse global backgrounds.Source: §A.4 · p. 19, Figure 14 · p. 22
  • The real-world image set includes same-country and cross-country food pairs across varied visual complexity, including clutter, mixed lighting, irregular plating, and inconsistent perspectives.Figure 15 places same-country pairs on the left and cross-country pairs on the right.Source: §A.4 · p. 19, Figure 15 · p. 23

B.1. Results

B.1 reports identification accuracy, prediction entropy, and real-world accuracy across models; its analysis also notes correctness shifts in mixed settings and little difference between landmark and street backgrounds. B.2 finds predictions largely stable under horizontal flips and food-size changes.

B.1. Results

  • Mixed subtasks often shift predictions from correct to incorrect, while only a small portion shift from incorrect to correct.Landmark and street backgrounds produce minimal performance differences.Source: §B.1 · p. 24
  • The section reports model prediction entropy across subtasks and identification accuracy on a real-world dataset.Source: §B.1 · p. 24

B.2. Ablations

  • Horizontal flipping yields high prediction consistency, indicating minimal to no positional dependence for food-related attributes.Source: §B.2 · p. 24
  • Identification accuracy remains stable across food-size differences, indicating minimal size-related bias.Source: §B.2 · p. 24

B.3. Qualitative Analysis on Food Name Prediction Failure Cases

The qualitative analysis finds that food-name errors can range from visually similar substitutions to unrelated predictions. It also reports worsening identification with greater cultural distance, correctness losses in mixed contexts, and limited sensitivity to position or item size.

B.3. Qualitative Analysis on Food Name Prediction Failure Cases

  • 25 out of 100 Gemini-2.5-pro errors, versus 2 out of 100 InternVL3-8B errors, were visually similar to the target food.The analysis sampled 25 incorrect food-name predictions per subtask for each model; the paper gives examples of visually similar but distinct dishes.Source: §B.3 · p. 24, Figure 16 · p. 24
  • Most LVLMs have relatively higher identification accuracy in single-food settings than multiple-food settings, while reported entropy is relatively high in single-food settings.Source: Table 6 · p. 25, Table 8 · p. 30
  • Mixed subtasks show many more correct-to-incorrect country-prediction shifts than incorrect-to-correct shifts.The figure caption notes that closed-source models perform better on country identification in SF but also show substantial correct-to-incorrect shifts in mixed subtasks.Source: Figure 17 · p. 26
  • Identification performance is similar with landmark and street backgrounds, and LVLMs generally perform better on multiple foods from the same culture in real-world images.Source: Table 7 · p. 28, Table 9 · p. 31
  • As target–distractor cultural distance increases, accuracy declines and entropy rises; models perform best when both foods come from the same country.Source: Figure 19 · p. 29
  • Horizontal flips and resized food items produce almost no change in predictions or identification accuracy, indicating little positional or size bias.Source: Table 10 · p. 31, Table 11 · p. 32

Related papers

Loading 2511.22787v2…