PaperlayerView exact revision on arXiv

Source-linked AI summary

Where Culture Fades: Revealing the Cultural Gap in Text-to-Image Generation

Chuancheng Shi, Shangze Li, Shiming Guo, Simiao Xie, Wenhua Wu, Jingtong Dou, Chao Wu, Canran Xiao, Cong Wang, Zifeng Cheng, Fei Shen, Tat-Seng Chua

arXiv:2511.17282v1cs.CVcs.AIcs.CY

TL;DR

Multilingual T2I models often generate culturally neutral or English-biased images, leaving cross-lingual cultural consistency unresolved. The paper probes culture-sensitive representations and uses them to guide two alignment strategies, which improve consistency on CultureBench while preserving fidelity and diversity.

  • Problem

    Multilingual T2I models can produce culturally neutral or English-biased images, weakening cross-lingual cultural correspondence.

    Source: §1 · p. 1

  • Method

    The authors use attention and Top-K SAE probing to locate culture-sensitive layers and neurons, then amplify them at inference or fine-tune only culturally relevant layers.

    Source: §1 · p. 2

  • Results

    Both alignment strategies consistently improve cross-lingual cultural consistency on CultureBench while preserving fidelity and diversity.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The findings point toward practical, controllable cultural alignment for inclusive generative models.

    Source: §7 · p. 9

    Cultural cues are constrained to contextually appropriate elements roughly consistent with real-world cultural statistics.

    Source: §1 · p. 1

Abstract

from arXiv · show

Multilingual text-to-image (T2I) models have advanced rapidly in terms of visual realism and semantic alignment, and are now widely utilized. Yet outputs vary across cultural contexts: because language carries cultural connotations, images synthesized from multilingual prompts should preserve cross-lingual cultural consistency. We conduct a comprehensive analysis showing that current T2I models often produce culturally neutral or English-biased results under multilingual prompts. Analyses of two representative models indicate that the issue stems not from missing cultural knowledge but from insufficient activation of culture-related representations. We propose a probing method that localizes culture-sensitive signals to a small set of neurons in a few fixed layers. Guided by this finding, we introduce two complementary alignment strategies: (1) inference-time cultural activation that amplifies the identified neurons without backbone fine-tuned; and (2) layer-targeted cultural enhancement that updates only culturally relevant layers. Experiments on our CultureBench demonstrate consistent improvements over strong baselines in cultural consistency while preserving fidelity and diversity.

1. Introduction

Multilingual T2I models can produce culturally neutral or English-biased images, despite their semantic fidelity, exposing a gap in cross-lingual cultural grounding. The paper traces this to weak activation of cultural representations and proposes neuron-guided alignment strategies evaluated on CultureBench.

1. Introduction

  • Multilingual T2I models often produce culturally neutral or English-biased images, while language-based systems can preserve localized cultural context.This weakens cross-lingual cultural correspondence and leaves T2I behind LLMs and recommender systems in cultural grounding.Source: §1 · p. 1
  • The authors argue that cultural knowledge is present but noun-only prompts do not activate it strongly enough to produce culturally grounded interpretations.Adding culture-style modifiers can elicit country-specific characteristics.Source: §1 · pp. 1–2
  • A two-stage probing framework contrasts attention patterns and Top-K SAE activations to localize culture-sensitive representations to a few layers and neurons.The analysis examines two representative systems and reports the pattern across architecturally different diffusion methods.Source: §1 · p. 2
  • The paper proposes inference-time amplification of identified neurons and fine-tuning restricted to culturally relevant layers as complementary alignment strategies.The first strategy requires no backbone fine-tuning; the second updates only selected layers.Source: §1 · p. 2
  • CultureBench supports evaluation of cross-cultural consistency with multilingual prompts and images, and both strategies improve consistency while preserving fidelity and diversity.The benchmark covers 15 countries.Source: §1 · p. 2

2. Related Work

Prior work addresses cultural bias and neuron-level interpretability, but cross-lingual cultural grounding remains underdefined and lacks systematic diagnostic tools and lightweight interventions.

Cultural Text-to-Image Generation

  • Prior T2I research studies cultural bias, fairness, and stereotyping, but remains English-centric with limited coverage of low-resource languages and diverse cultural contexts.Existing work includes CCUB-based cultural coverage and visual-stereotype quantification.Source: §2 · p. 2
  • Definitions and systematic evaluation of cross-lingual cultural consistency remain limited, as do unified diagnostics, plug-and-play interventions, and corresponding benchmarks.Prior studies provide little insight into where culture-sensitive features reside or how to control them at layer or neuron level.Source: §2 · pp. 2–3

Neuron Interpretability

  • Neuron-level interpretability and multimodal causal probing enable concept localization and control, but existing work mainly targets objects or attributes rather than multilingual cultural representations.FEMN demonstrates neuron-level semantic control in a CLIP-style model.Source: §2 · p. 3

3. CultureBench Dataset

CultureBench collects and annotates culturally representative images across language-region groups, then separates training, evaluation, and probing data. Its CultureVQA task measures cultural attribution from visual cues.

Data Collection

  • CultureBench collects culturally representative images across 15 language and region groups through geo-constrained search and manual verification.Images are checked for clarity, authenticity, and representativeness.Source: §3 · p. 3

Data Annotations

  • Each sample receives a GPT5-Nano culture-style-modifier caption and a human-written noun-only caption, with expert review of the cultural taxonomy and sample subset.Review screens for appropriate, statistically grounded cues and treats mismatched cues as stereotypes.Source: §3 · p. 3

Data Distribution

  • The 7,932 samples are split 7:2:1 into training, test, and neuron-detection subsets, with test and probing data excluded from training, tuning, and model selection.The subsets support training, quantitative evaluation, and layer- and neuron-level probing, respectively.Source: §3 · p. 3

Cultural Evaluation

  • CultureVQA compares cultural attribution using visual cues alone, while Figure 4 contrasts culture-style-modifier-plus-noun prompts with noun-only prompts on the test subset.Qwen3-VL and Gemini-2.5-Flash select among 15 language-region labels or an “unrecognisable” option.Source: §3 · p. 3, Figure 4 · p. 3

4. Cultural Probing

Cultural modifiers improve CultureVQA performance over noun-only prompts, while attention and sparse-feature probing localize culture-sensitive layers and neurons. Masking the identified neurons sharply reduces cultural accuracy, unlike random masking.

4.1. Cultural Gap

  • Culture-style modifier + noun prompts achieve CultureVQA scores of 44.39 for AltDiffusion and 35.62 for PEA-Diffusion, substantially above noun-only prompts.The same trend across two structurally different diffusion models supports the hypothesis that cultural knowledge is present but insufficiently activated.Source: §4.1 · pp. 3–4

4.2. Culture Layer Detection

  • Modifier-to-noun attention is higher for culture-style prompts than noun-only prompts, and PEA-Diffusion’s ∆CA peaks at layer 16.The results indicate that cultural semantics concentrate in a key layer rather than being uniformly distributed.Source: §4.2 · pp. 4–5, Figure 6 · p. 4

4.3. Culture Neuron Detection

  • Weighted-frequency peaks identify a small set of culture-specific neurons; peak indices differ across six cultural backgrounds, indicating non-overlapping locations.Top-K is defined by peak neurons, with K adapting to the number of salient peaks.Source: Figure 7 · p. 5, §4.3 · p. 5

4.4. Cultural Validation

  • Masking identified Top-K neurons reduces mean CultureVQA from 35.62 to 7.65, while random masking yields 33.04, near baseline.The targeted drop supports the localization of neurons related to cultural semantics.Source: §4.4 · pp. 5–6

5. Methods

The methods target culturally sensitive representations in two ways: they amplify selected neurons at inference time or train an enhancer in a culturally relevant layer while keeping the backbone fixed.

5.1. Zero-Training Neuron Amplifier

  • The zero-training amplifier intervenes on a targeted culture-sensitive neuron subset in a key layer to strengthen cultural attributes in generated images.It encodes attention-correlation features with an SAE, modulates selected cultural neurons using λ, then decodes the representations back to attention space.Source: §5.1 · p. 6

5.2. Fine-Tuned Layer Enhancer

  • The fine-tuned enhancer inserts a small trainable module only at the culture-sensitive text-encoder layer, keeping all other parameters and the backbone fixed.It trains against CultureBench human-curated cultural reference images using pixel-level MSE, then applies the prompt-conditioned enhancer during inference.Source: §5.2 · p. 6

6. Experiments And Analysis

Across CultureBench comparisons, the proposed strategies improve cultural understanding while retaining semantic and visual quality. Qualitative, human-study, sensitivity, and ablation results further examine these gains and their conditions.

6.1. Implementation Details

  • Fine-tuned runs use 2,000 steps, AdamW at 5 × 10^-5, and batch size 1; the zero-training variant uses λ = 6 on one NVIDIA A6000 GPU.The evaluation uses CLIPScore and ImageReward for alignment and preference, LPIPS for similarity and fidelity, and CultureVQA for cultural recognition.Source: §6.1 · p. 6

6.2. Compare with SOTA Methods

  • CultureVQA is 36.63 for fine-tuned versus 23.05 for AltDiffusion and 21.65 for PEA-Diffusion; zero-training leads CLIPScore (0.291) and ImageReward (0.33).The authors report simultaneous gains in cultural understanding, text-image consistency, and perceptual quality without sacrificing visual fidelity.Source: Table 2 · p. 7, §6.2 · p. 7
  • Human-rated CSR reaches 77.6 versus the second-best 60.4, and qualitative comparisons show culturally grounded outputs with semantic accuracy where baselines often appear neutral.The study involved 50 cultural-studies experts and evaluated MCC, SCC, and CSR; Show-o2 also exhibited semantic drift in the qualitative examples.Source: §6.2 · pp. 7–8, Figure 8 · p. 7, Figure 9 · p. 8

6.3. Hyperparameter Sensitivity Analysis

  • CultureVQA peaks at 35.92 when λ = 7, then falls to 32.61 at λ = 8; the authors select λ = 7.Increasing λ progressively strengthens target-culture features, while excessively high values may overfit and slightly degrade metric performance.Source: Figure 10 · p. 8, Figure 11 · p. 8, §6.3 · p. 8

6.4. Ablation Study

  • CultureVQA gains are 7.01 on AltDiffusion and 12.26 on PEA-Diffusion for zero-training, rising to 9.61 and 14.98 after fine-tuning.Random activation or random fine-tuning yields minimal or negative improvements, and gains recur across both diffusion architectures.Source: Table 3 · p. 8, §6.4 · p. 8

7. Conclusion

The paper attributes multilingual T2I cultural gaps to weak activation of existing culture-sensitive representations, and reports that two targeted alignment strategies improve consistency on CultureBench while preserving fidelity and diversity.

7. Conclusion

  • The authors attribute culturally neutral or English-biased outputs to weak activation of culture-sensitive representations, not missing cultural knowledge.They localize culture signals to a small neuron set in a few layers and propose inference-time activation and layer-targeted enhancement; both improve CultureBench consistency while preserving fidelity and diversity.Source: §7 · pp. 8–9

A. Details of CultureBench Dataset

The appendix describes CultureBench’s composition and curation, then reports controlled evidence about cultural representations, CultureVQA reliability, cross-model neuron detection, and selection of the Top-K cutoff.

A. Details of CultureBench Dataset

  • CultureBench covers broad cultural spheres and language groups, with expert screening and annotation intended to capture macro-level identity and finer-grained regional diversity.Its collection process uses publicly available images and excludes samples judged stereotypical or biased by a cross-cultural expert team.Source: A. Details of CultureBench Dataset · p. 12

B. Further Evidence on the Culture Gap

  • With a fixed culture-neutral English prompt and sampling settings, activating different culture-neuron sets produces distinct cultural styles, while the baseline remains neutral.The tested sets correspond to China, Japan, Italy, Germany, and Ukraine, supporting the authors’ under-activation account rather than knowledge absence.Source: B. Further Evidence on the Culture Gap · p. 12, Figure 13 · p. 13

C.1. Reliability of CultureVQA

  • CultureVQA reaches 91.57% accuracy versus 94.18% for experts, a gap of 2.61 percentage points on the four-image cultural-choice task.Thirty domain experts and CultureVQA answered questions with four real-world images, only one of which matched the specified cultural element.Source: Table 4 · p. 13, §C.1 · p. 13

C.2. Cultural Probing Universal Type

  • AltDiffusion probing identifies layer 14 as culturally sensitive and finds culture-specific neuron peaks across six cultural contexts.This extends the probing analysis beyond the PEA-Diffusion results presented in the main text.Source: Figure 14 · p. 13, §C.2 · pp. 13–14, Figure 15 · p. 13

C.3. Cultural Validation in AltDiffusion

  • Masking AltDiffusion’s Top-K neurons lowers mean CultureVQA by 32.50 points, while random masking of the same number lowers it by 2.09 points.The contrast indicates that the identified neurons concentrate cultural information in this validation setting.Source: Table 5 · p. 14, §C.3 · p. 14

C.4. Selection of Threshold K

  • The threshold K is set at the elbow where a few high weighted-frequency-score peaks give way to a sharp drop, retaining salient neurons and discarding the weak-response tail.Figure 16 marks the selected Top-K neurons to the left of the threshold and scores below the cutoff to its right.Source: §C.4 · p. 14

C.5. Details of SAE

  • The Top-K SAE uses a hidden dimension of 4096, AdamW with learning rate 0.0004, a constant schedule without warmup, and MSE reconstruction loss.The reconstruction loss is used to encourage faithful feature reconstruction.Source: §C.5 · p. 14

C.6. Cross-Domain Generalization Results

On 100 out-of-domain captions with cue words absent from CultureBench, the method improved CultureVQA across both models and both alignment settings. The section also describes qualitative results and a three-task perceptual user study.

C.6. Cross-Domain Generalization Results

  • Zero-training gains were +7.00 and +5.00 CultureVQA points over PEA-Diffusion and AltDiffusion; fine-tuned gains were +20.00 and +17.00 points, respectively.Each comparison is against its named baseline model.Source: §C.6 · p. 14
  • For Chinese, Arabic, and French prompts, the approach maintained semantic and cultural coherence while capturing textual details and regional visual preferences.Figure 17 is described as showing more culturally appropriate generations.Source: §C.6 · p. 15, Figure 17 · p. 15

D. Detail of User Study

  • Fifty participants completed MCC, SCC, and CSR tasks, respectively selecting representative images, identifying culture from 16 options, and rating style-aware alignment from 1 to 5.Participants were aged 25–47, including 23 women.Source: D. Detail of User Study · p. 15

E. More Results

A same-prompt comparison with the model seed and other generation parameters fixed found that enhanced images more faithfully reflected the target cultural context.

E. More Results

  • With prompt, seed, and generation parameters fixed, enhanced images more faithfully reflect culture in scene elements, character depictions, and fine textures.The authors say these changes yield more accurate visual expressions of the target culture.Source: E. More Results · p. 15

F. More Discussion

The discussion defines cultural consistency as culturally grounded visual cues beyond literal semantic correctness and explains the paper’s benchmark, evaluation, and intervention choices. It also presents evidence for under-activated cultural knowledge and clarifies limits on interpretation and model transfer.

F. More Discussion

  • Cultural consistency means statistically grounded, contextually appropriate cues tied to the target language’s socio-cultural context, beyond literal semantic correctness.The operationalization uses a fixed language–region mapping, observable moderate visual grounding, and expert screening for stereotypes or inappropriate cues.Source: F. More Discussion · pp. 15–16
  • Figure 18 shows the MCC, SCC, and CSR user-study interface, while Figure 19 presents further AltDiffusion and PEA-Diffusion results under different methods.Source: Figure 18 · p. 16, Figure 19 · p. 16
  • CultureBench is a medium-scale diagnostic benchmark, with approximately 7.9k images across 15 language-culture regions, rather than a pre-training corpus or comprehensive cultural catalogue.The authors describe it as an extensible starting point.Source: F. More Discussion · p. 16
  • CultureVQA evaluates cultural correctness as semantic recognition across 15 regions, and its reliability is supported by a human–model consistency study.The authors contrast it with FID and CLIPScore, which focus on pixel-level similarity or general correspondence.Source: F. More Discussion · p. 16
  • Explicit culture-style modifiers elicit culturally grounded generations, while noun-only prompts yield neutral prototypes; masking identified Top-K neurons sharply reduces CultureVQA performance.Masking the same number of random neurons has minimal effect, and attention probing finds a stable culture-sensitive layer.Source: F. More Discussion · pp. 16–17
  • Localized neuron and layer interventions improve CultureVQA and human-perceived cultural fidelity while leaving CLIPScore, ImageReward, LPIPS, and visual diversity essentially unchanged.The authors do not claim definitive causal attribution for individual neurons; they frame the method as an interpretable control mechanism and a starting point for causal analysis.Source: F. More Discussion · p. 17
  • Changing a model’s text encoder requires re-running cultural sensitivity detection; MSE trains layer enhancers but is not the cultural-consistency evaluation metric.The discussion sets ε to 0 by default and adds β to the denominator to avoid zero when activations are absent or rare.Source: F. More Discussion · p. 17

G. Limitation

The benchmark covers 15 cultural regions but represents only a limited portion of global diversity. The paper also acknowledges potential misuse of image synthesis for deceptive content and disinformation.

G. Limitation

  • CultureBench’s 15 regions cover only a limited portion of global diversity, with low-resource and marginalized communities especially underrepresented in publicly available image resources.The authors plan to expand sources and add finer intra-cultural subdivisions.Source: G. Limitation · p. 17

H. Ethics Statement

  • The authors acknowledge that image synthesis could be misused to generate deceptive content and spread disinformation.They describe third-party oversight and detection and prevention mechanisms as part of their responsible-deployment approach.Source: H. Ethics Statement · p. 17

Related papers

Loading 2511.17282v1…