Source-linked AI summary
Probing Pre-Trained Language Models for Cross-Cultural Differences in Values
Arnav Arora, Lucie-Aimée Kaffee, Isabelle Augenstein
TL;DR
Cultural values embedded in pre-trained language models have not been systematically studied across cultures. The paper probes survey-derived questions in multilingual models and finds marked cross-cultural value differences that weakly correlate with established surveys, motivating caution in cross-cultural deployment.
Problem
Cultural values embedded in pre-trained language models and their variation across cultures remain understudied.
Method
The paper converts value-survey questions into cloze-style probes and evaluates them across 13 languages and three multilingual language models.
Results
Pre-trained language models capture marked differences in values between cultures, but these values are only weakly correlated with established surveys.
Takeaways & Limitations
Alignment with cultural value surveys matters when developing models for cross-cultural contexts, alongside more inclusive diversification of training data.
Takeaways & Limitations
The approach equates languages with countries and can misrepresent communities, minorities, and diaspora populations; the surveys also have known criticisms.
Abstract
from arXiv · showhide
Language embeds information about social, cultural, and political values people hold. Prior work has explored social and potentially harmful biases encoded in Pre-Trained Language models (PTLMs). However, there has been no systematic study investigating how values embedded in these models vary across cultures. In this paper, we introduce probes to study which values across cultures are embedded in these models, and whether they align with existing theories and cross-cultural value surveys. We find that PTLMs capture differences in values across cultures, but those only weakly align with established value surveys. We discuss implications of using mis-aligned models in cross-cultural settings, as well as ways of aligning PTLMs with value surveys.
1 Introduction
The paper addresses the understudied question of how cultural values and differences are embedded in pre-trained language models and whether they align with established surveys. It proposes probing survey-derived questions across languages and models to study these values.
- Cultural values and differences embedded in pre-trained language models remain understudied compared with social and potentially harmful biases.
- The study asks whether language models capture cultural differences in values across different models and languages.
- The authors reformulate questions from Hofstede’s theory and the World Values Survey into probes and compare model outputs with survey values.
- Experiments cover 13 languages and three multilingual language models, assessing value-alignment correlations with two large-scale surveys.
- The paper discusses implications of deploying models in multicultural contexts and ways to incorporate cultural values through more inclusive training data.
3 Value Probing
The paper grounds its comparison in established cross-cultural value surveys and narrows the analysis to survey dimensions and categories that can be represented with language-model probes.
- The study compares language-model value encodings with Hofstede’s cultural dimensions theory and the World Values Survey.
- Hofstede’s framework used here contains six dimensions: power distance, individualism, uncertainty avoidance, masculinity, long-term orientation, and indulgence.
- Hofstede’s survey contains 24 questions, with each dimension calculated from four questions using a defined formula.
- The World Values Survey collects more detailed cross-cultural value data in waves, with wave 7 conducted from 2017 to 2020.
- The experiments use 11 WVS categories and leave the Inglehart–Welzel cultural-map analysis for future work.
5 Probe Generation
The paper converts survey questions into cloze-style probes, translates them across selected languages, and masks response labels for multilingual model inference.
- Survey questions are manually reformulated into cloze-style probes whose masked labels represent opposing survey-response options.
- For Hofstede’s survey, the resulting probe set contains 24 questions with repeating labels.
- The WVS produces more probes because its questionnaire contains 238 questions and a wider variety of question types.
- English probes are translated into target languages with response labels inserted to preserve grammatical structure and support translation.
- Machine translation may perform poorly for low-resource languages and introduce additional biases, although the authors found these simple probes to translate well.
- The study investigates 13 languages mapped to one country each, while acknowledging that languages extend beyond national borders and communities.
6 Methodology
The methodology probes three multilingual masked-language models, converts mask-token predictions into survey-compatible scores, and evaluates rank correlations with survey values.
- The experiments use mBERT, XLM, and XLM-R, three widely used multilingual pre-trained language models.
- 6.2 Mask Probing: For each probe, an MLM head produces log probabilities for the masked token across the model vocabulary.
- 6.2 Mask Probing: The method subtracts the predicted logit for the highest-scoring response from the opposing response to normalize the survey endpoints into a question score.
- 6.2 Mask Probing: World Values Survey responses are normalized by question scale and averaged within categories to obtain country-level category scores.
- 6.3 Evaluation: Spearman’s ρ measures rank-based correspondence between model predictions and survey values, including per-question, category-level, value-level, and country analyses.
7 Results
The models produce culturally varied value predictions, but agreement across models and alignment with established surveys are limited. Cross-cultural differences are stronger for WVS probing than for Hofstede probing.
- Model predictions: XLM-R predictions vary substantially across cultures, with high average power distance and relatively low masculinity and indulgence scores.The model associates high power distance with Greece and South Korea, masculinity with Pakistan and Germany, and positive indulgence only with the Philippines and Malaysia.
- Cross-cultural differences: 42.31% of XLM-R’s WVS country pairs differ significantly, compared with 51.28% for mBERT and 46.15% for XLM.For Hofstede’s survey, the corresponding shares are 10.26% for XLM-R, none for mBERT, and 6.41% for XLM.
- Model agreement: Predictions are inconsistent across the three models, including low correlations between mBERT and XLM despite shared architecture and Wikipedia training.The results suggest that relatively minor changes in model training can affect the cultural values captured by the models.
- Alignment with surveys: For both Hofstede and WVS, model prediction scores show no statistically significant alignment with survey scores.The authors report weak correlations for some Hofstede values and attribute the low Hofstede significance rates partly to its six-dimensional test compared with more than 200 WVS questions.
8 Discussion
Multilingual PLMs encode sizable cross-cultural differences in values, but those differences only weakly align with established values surveys. This mismatch may reflect sensitivity to model design and training choices and raises concerns for cross-cultural deployment.
- Multilingual PLMs capture sizable differences in cultural values across cultures, even when trained on data from the same source.The findings suggest that model design and training choices affect the cultural biases encoded by these models.
- The values encoded by PLMs vary across cultures but are not aligned with values reported in existing large-scale surveys.The paper describes this as unexpected given prior evidence that PLMs pick up biases present in their training data.
- A lack of reflection of surveyed cultural values may indicate shortcomings in multilingual representation learning.The discussion identifies limited diversity in multilingual training data as one possible reason.
- Cultural diversity in training data is important for PLMs used across countries and for tasks such as question answering, dialogue, and information retrieval.The paper distinguishes cultural diversity from merely linguistic and typological diversity.
- Aligning models with human values requires auditing that considers both harmful cultural values and the downstream use case.The paper gives high masculinity, associated with misogynistic language and gender bias, as an example of a potentially harmful value.
9 Conclusion
The paper introduces a method for probing cultural values in multilingual PLMs and comparing them across models and with established surveys. It finds marked cross-cultural differences in PLM-encoded values, but only weak correlations with survey values.
- The study proposes a methodology for probing cultural values embedded in multilingual PLMs and assessing differences among models.It measures alignment both among the models and with existing values surveys.
- PLMs capture marked differences in values between cultures, but these values are only weakly correlated with established values surveys.The paper also discusses how training data and modelling choices may affect cultural bias.
- The paper discusses the importance of alignment for developing models in cross-cultural contexts.It offers suggestions for more inclusive diversification of training data to incorporate cultural values.
10 Ethical Considerations
The paper identifies ethical risks in simplifying cultural representation, equating languages with countries, and interpreting aggregate survey values. These choices can misrepresent minorities and subcultures and may negatively affect communities.
- Equating a language with a country can misrepresent communities and disregard minority and diaspora communities.The authors describe this as the closest approximation available for comparing their surveys and language models.
- Hofstede’s cultural dimensions theory has been criticised as too simplistic, which can encourage simplistic assumptions when probing language models.
- Including the World Values Survey helps address limitations associated with relying on a single values survey.
- Simplifying cultural representation through the approach and the original surveys may negatively affect communities.
- Such misrepresentation can disproportionately exacerbate the marginalisation of minority communities or subcultures.
- The probing methodology captures static, extrinsic token-level biases at inference time through output probabilities.Intrinsic bias measures exist, but they do not always correlate with extrinsic measures.
- Cross-cultural comparisons often use countries as surrogates for cultures, limiting how cultural diversity is represented.
- Surveys collapse individual values into question sets, while respondents may interpret questions differently and populations may be treated as homogeneous.Confounding factors also affect survey responses.
A Translation quality
The study evaluates translated probes for grammaticality and preserved meaning, while documenting the inference-only implementation and correlation analyses. Human evaluation covers a sample of probes in six languages.
- Human annotators rate translated probes for grammaticality and meaning on a 1-5 Likert scale.The sample contains 3 Hofstede probes and 23 World Values Survey probes, representing 10% of the total.
- The translation evaluation uses 26 probe questions assessed by native speakers of six of the 13 study languages.The six-language evaluation was conducted because of resource constraints.
- The experiments use mBERT, XLM, and XLM-R through PyTorch and Transformers, with probing performed only at inference time.The experiments ran on a single NVIDIA Titan RTX GPU for less than one hour.
- Table 6 reports averaged human-evaluation scores for grammaticality and preserved meaning of machine-translated probes.
- The analysis separately computes correlations for high-response and low-response survey labels to test the effect of logit subtraction.Results are reported separately for Hofstede and the World Values Survey.
- Table 7 reports per-dimension correlations between high-response mask-prediction scores and Hofstede’s values survey, marking p <= 0.05 with an asterisk.
D Example probes
The paper presents English question probes translated into multiple languages and reports correlations between masked-prediction scores and value surveys across categories and countries.
- English question probes are reformulated from original survey questions and translated into the different languages studied.
- Tables 8 and 9 report category-level correlations between masked-prediction scores and World Values Survey responses for high and low response labels.
E Hofstede Value Calculation
The Hofstede value calculation applies formulas from the original survey to probe results, using question-response means and constants while comparing cultural dimensions across countries.
- The calculations use Hofstede’s original survey formulas to derive value results for the probes.
- The formulas use indexed survey-question means, while the constant does not affect comparisons between countries.
- Hofstede’s dimensions include Power Distance, Individualism, Uncertainty Avoidance, Masculinity, Indulgence, and Long-Term Orientation.
- The paper compares probe correlations with Hofstede’s survey and the World Values Survey, marking statistically significant values at p <= 0.05 with an asterisk.
- The long-term-orientation description states that short-term-oriented cultures value reciprocating social obligations, tradition, face, steadiness, and stability more.