CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologies

• CultureBank combines structured cultural descriptors with grounded scenarios, personas, and questions for evaluating LLMs.Source: Figure 1 · p. 1
• The paper develops a pipeline to construct CultureBank in support of culturally aware language technologies.Source: Figure 1 · p. 1
• Fine-tuning a model on CultureBank improves performance on two cultural tasks.Source: Figure 1 · p. 1

• Cultural knowledge bases include ethnographic collections such as eHRAF, computationally constructed datasets, and resources extracted from C4 or Wikipedia.Source: §2 · p. 2
• The paper defines culture as knowledge shared by a relatively large group about shared beliefs, practices, and behaviors.Source: §2 · p. 2
• Online communities complement cleaner, norm-focused sources by providing descriptive cultural expressions for heterogeneous cultures.Source: §2 · pp. 2–3
• CultureLLM integrates cultural differences with augmented data, while this work proposes grounded evaluation and fine-tuning with descriptive cultural behaviors.Source: §2 · p. 3

• The taxonomy provides 11 fields, including cultural group, context, goal, actor, recipient, behaviors, topic, and agreement.Source: §3 · p. 3
• Its structured representation addresses the difficulty of searching and filtering noisy free-text cultural knowledge.Source: §3 · p. 3
• In-context examples permit group labels ranging from national groups to regional, ethnic, and social groups such as international students.Source: §3 · p. 3

4 Construction Pipeline

• The pipeline extracts descriptors, clusters and summarizes them, then post-processes results to calculate agreement and remove problematic content.Source: §4 · pp. 3–4, Figure 2 · p. 4
• A classifier selects culturally relevant comments before Llama-2-70B extracts taxonomy fields, with reported test accuracies of 79% and 82%, respectively.Source: §4 · p. 4

4.2 Descriptor clustering

• Hierarchical clustering groups descriptors encoded from concatenated fields, removes clusters smaller than five, and yields an average Silhouette score of 0.14.Source: §4.2 · p. 4
• A Mixtral summarizer distilled from 1K GPT-4-generated examples achieves a fidelity score of 89.7% and coherence score of 96.6%.Source: §4.2 · pp. 4–5

4.3 Post-processing

• Post-processing averages per-point agreement scores within clusters to provide agreement levels for cultural descriptors.Source: §4.3 · p. 5
• Moderation filters flagged content, manually reviews identified items, removes harmful content, and detects and removes personally identifiable information.Source: §4.3 · p. 5

5 CultureBank Dataset

• 12K cultural descriptors, 730 cultural groups, and 36 topics make up CultureBank-TikTok.The data came from 34K posts and 720K English comments collected from May 2019 to August 2023.Source: §5 · p. 5
• Four annotators assessed 200 samples for format, traceability, and meaningfulness; agreement was κ = 0.8.The authors assess traceability rather than factualness because the descriptors are self-reported and may be nuanced.Source: §5 · p. 5
• Examples include cross-cultural behaviors, language variation, diverse ethnic identities, recent practices, and disagreements about South African pricing and taxes.The examples include differing reactions to calculating prices and taxes separately in South Africa.Source: §5 · pp. 5–6

6 Evaluating LLMs’ Culture Awareness

• For each descriptor, a fine-tuned Mixtral generator creates a consulting scenario, client persona, and grounded question; 86% of questions were judged grounded in the descriptor.A self-refinement step improves generation using two quality-control metrics.Source: §6 · p. 6
• GPT-4 judges whether answers entail the original descriptor, while two experts compare answers and select the more culturally aware one.The evaluation presents a generated grounded question to the model before scoring its answer.Source: §6 · p. 6, Figure 3 · p. 6
• The evaluated systems include open-source Llama-2 and Mixtral, closed-source GPT models, and the authors’ fine-tuned models.All reported results are on the test set.Source: §6 · p. 7
• Mixtral-8x7B and GPT-4 achieved the highest overall entailment scores, 66.9 and 66.1, while scores fell for descriptors with fewer supports.The benchmark separates examples by cluster size; the low-support group contains 924 examples.Source: Table 4 · p. 6, §6 · p. 7
• Against base Mixtral, Mixtral-DPO won 36% of human comparisons and tied 49%; against GPT-4, Mixtral-SFT won 46% and tied 38%.The human-evaluation figure reports 50 examples per model pair and annotator agreement of κ = 0.87.Source: Figure 4 · p. 6, §6 · p. 7

7.1 Zero-shot transferability on downstream tasks

• The authors test whether CultureBank training can improve cultural awareness and transfer to other culture-related tasks.The downstream evaluation uses GlobalOpinionQA and CultureNLI.Source: §7 · p. 7, §7.1 · p. 7

7 Fine-tuning a More Culturally Aware Language Model

• Training first applies supervised fine-tuning to 9,402 training descriptors, then trains on a 2K subset of poor-performing cases using descriptor-augmented questions and answers.The second stage uses supervised fine-tuning or DPO.Source: §7 · p. 7
• On a test set mostly comprising out-of-domain descriptors, fine-tuned models perform better than their vanilla versions.The authors interpret this result as evidence that CultureBank can improve cultural awareness.Source: §7 · p. 7

7.1 Zero-shot transferability on downstream tasks

• CultureBank-fine-tuned models outperform vanilla counterparts on GlobalOpinionQA and CultureNLI in zero-shot evaluation.For Mixtral-SFT, the reported comparisons include 79.5 vs. 81.8 on GlobalOpinionQA and CultureNLI F1 of 59.9 vs. 61.5 (US) and 60.8 vs. 61.3 (India).Source: §7.1 · p. 7

8 Generalizing to Reddit

• The authors apply the pipeline to Reddit to test whether it transfers across online communities.They customize the pipeline for Reddit and report its running time and data volume in Table 8.Source: §8 · p. 7
• Because Reddit tags are less commonly used and comments are longer, the authors search cultural keywords and train a Reddit-specific relevance classifier on 1K annotated examples.They identify 2.6M cultural comments, then sample 528K for further processing; a Mixtral extractor is fine-tuned on 1K GPT-4-generated examples.Source: §8 · p. 8
• CultureBank-Reddit contains 11K cultural descriptors and 2K cultural groups, and human annotation indicates high-quality data.The authors present this as evidence that the pipeline generalizes to another platform.Source: §8 · p. 8

9.1 Cultural knowledge data

• Fine-tuning on CultureBank improves cultural awareness across various downstream tasks, supporting continued development of cultural knowledge databases.Source: §9.1 · p. 8
• Sources capture different cultural facets: Reddit emphasizes community and identity, whereas TikTok emphasizes daily life, so datasets should retain source labels.Source: §9.1 · p. 9
• CultureBank includes some cross-culture behaviors, while future resources should broaden them and capture perspectives, time, multilingualism, and multimodality.Source: §9.1 · p. 9
• Future data should track culture's temporal change, finer-grained groups, and adaptation to global influences.Source: §9.1 · p. 9

9.2 Cultural awareness evaluation

• Evaluation should involve global annotators and ground assessment in end applications, since human judgments vary and direct versus grounded evaluations differ.Source: §9.2 · p. 9

9.3 Training culturally-aware language technologies

• Culturally aware assistants need multi-turn grounded cultural data, training attentive to implicit dialogue nuances, and evaluation supporting model improvement and evolution.Source: §9.3 · p. 9

10 Conclusion

• CultureBank applies a generalizable pipeline to online communities, yielding 12K TikTok and 11K Reddit cultural descriptors.Source: §10 · p. 10
• CultureBank provides agreement levels for nuanced interpretation and contextualized scenarios for grounded evaluation; LLM evaluations show room for improvement.Source: §10 · p. 10
• Fine-tuning with CultureBank improves performance on downstream cultural tasks, and the paper offers insights for future culturally aware language technologies.Source: §10 · p. 10

Ethical Statement

• Despite filtering, CultureBank may still contain stereotypical, controversial, or negative content, prompting a call for better moderation.Source: Ethical Statement · p. 10
• CultureBank descriptors should not be interpreted as reflecting the personal views of the authors or online platforms.Source: Ethical Statement · p. 10
• The authors report anonymizing data where possible, complying with platform terms of service, and following relevant research ethics guidelines.Source: Ethical Statement · p. 10

A Limitations

• English-only processing and possible LLM extraction or summarization bias limit cultural coverage, while reaction-driven keyword sampling may overrepresent salient differences and CultureBank retains generic statements.These limitations may oversimplify cultural contexts, underrepresent mundane or shared cultural aspects, and obscure nuance in intercultural interactions.Source: A Limitations · p. 14

C.1 Descriptor extraction

• Descriptor extraction leaves values unrestricted except for binary agreement; optional negation conversion flips both behavior polarity and agreement before two-stage clustering.The pipeline removes clusters with fewer than five data points, retains comment years, and bins cluster size into support values.Source: §C.1 · p. 15, §C.2 · p. 15
• Cluster summaries use Mixtral 8x7B fine-tuned on 1k GPT-4-generated examples, and topic normalization consolidates manually reviewed themes into 36 categories.Human annotation evaluates fidelity and coherence; cultural-group synonyms are grouped by HAC and majority vote, while embedding-based topic clustering was too noisy.Source: §C.2 · p. 15

C.4 LLM Prompts for Different Steps

• Extraction prompts require culturally specific information directly supported by the input, while summarization prompts request concise cultural summaries and favor the dominant opinion in conflicting clusters.Source: §C.4 · pp. 16–18

C.5 Running time; D.1 Evaluated model details

• Supplementary tables report per-step runtime and remaining data volumes for TikTok and Reddit, and map paper model names to exact model versions and training details.Source: §C.5 · p. 18, Table 7 · p. 18, Table 8 · p. 19

C.6 Cultural Topic Distribution

• A table presents CultureBank’s cultural-topic distribution across TikTok and Reddit.Source: §C.6 · p. 19, Table 9 · p. 20

D.2 Direct Evaluation

• None of the evaluated models achieved perfect direct-evaluation scores; Llama-based models worsened on long-tailed behaviors, while GPT models outperformed others and ChatGPT exceeded GPT-4 across support levels.The authors also report that ambiguous behaviors can elicit conservative GPT-4 answers, and that correct classification does not ensure culturally nuanced advice in downstream scenarios.Source: §D.2 · p. 19
• Direct evaluation reports macro-averaged F1 for high-, mid-, and low-support descriptors and for the full test set.The supplied table caption does not provide the scores themselves.Source: Table 11 · p. 22

D.3.1 Data generation

• The authors sample 1K CultureBank descriptors, use GPT-4 to generate scenarios and questions, then fine-tune Mixtral 8x7B to generate scenario, persona, and indirect client questions at scale.The fine-tuned model generates evaluation questions for the full dataset.Source: §D.3.1 · p. 22, Figure 5 · p. 22
• A GPT-4-based reward model rates question relevance and subtlety, and the authors refine questions failing either threshold: 0.95 for relevance and 0.8 for subtlety.The improved model is then used to generate data for the entire set.Source: §D.3.1 · p. 22, Figure 5 · p. 22

D.3.2 Automatic evaluation

• For grounded evaluation, LLMs answer questions tied to practical scenarios, and GPT-4 assigns an entailment score using its predicted probability that the answer entails the cultural descriptor.Question quality is assessed for relevance and indirect reference to the knowledge.Source: §D.3.2 · pp. 23, 25

D.3.3 Human evaluation on model win rates

• Two annotators independently rated cultural awareness, readability, and relevance on 200 randomly selected outputs, reaching a Kappa score of 0.87 on the first 50 pairs.The remaining outputs were split between annotators, who were blind to the source model.Source: §D.3.3 · p. 26
• Fine-tuned models gave shorter, more culturally specific advice, while GPT-4 sometimes reused advice templates across inquiries with different cultural contexts.Standard RLHF-ed answers were often more comprehensive but less specific to the user’s inquiry.Source: §D.3.3 · pp. 26–27

E.1 Training

• Training used 9,402 cultural descriptors for eight epochs, then 2,000 poorly answered grounded questions augmented with gold-knowledge responses for eight more epochs of SFT or DPO.The second phase sampled questions with reward-model knowledge-entailment scores below 0.6.Source: §E.1 · p. 28

E.2.1 Global Opinions QA

• Global Opinions QA compares model and country-response answer distributions with JSD, averages similarity across questions and countries, and uses cross-country score variation to assess balance.The paper defines higher cultureSim as greater similarity and lower standard-deviation error as more balanced representation.Source: §E.2.1 · p. 30

E.3 CulturalNLI

• CulturalNLI prompts models to adopt cultural indicators and scores their predicted entailment, contradiction, and neutral distributions with F1 against entailment as the evaluation class.Source: §E.3 · p. 31

F Generalizing to Reddit

• For Reddit, keyword searches and a classifier adapted to longer comments selected 2.6 million cultural comments from 7 million keyword-filtered comments.The classifier was trained using GPT-4 annotations of 1,000 data points.Source: F Generalizing to Reddit · p. 31

G.1 Preliminary temporal analysis

• Reddit snapshots show cultural discussion changing over time, including increased study-abroad discussion, attention to LGBTQ+ rights, and declining DVD-related discussion as streaming and downloads spread.Source: §G.1 · p. 31
• Entertainment examples show rising hip-hop discussion from the late 2000s and superhero narratives incorporating more diverse characters and storylines.Source: §G.1 · p. 32