Investigating Multi-Cultural Value Selection Biases in LLMs

• Prior LLM evaluations commonly collapse cultural identity to one immutable label, despite evidence that identities are complex and people can switch cultural frames with context.The study asks how an LLM responds when a user signals ties to two countries simultaneously.Source: §1 · p. 1
• The framework fixes one country as a system-prompt persona and signals a second in the user turn, explicitly by nationality or implicitly through query language.It applies the same cues to WVS multiple-choice answers and open-ended sociopolitical generation.Source: §1 · p. 1, Figure 1 · p. 1
• The study does not define a correct composite-identity response; its measurements describe model changes across prompts, not values of genuinely composite-identity individuals.It compares cue shifts across countries and languages because unusually large shifts may indicate bias toward an identity or more distinct associated values.Source: §1 · pp. 1–2
• WVS language cues produce the largest distributional shifts but move answers away from the associated-country baseline, while explicit cues differentiate open-ended value changes more than language cues.The framework is applied to 1.6 million open-ended responses.Source: §1 · p. 2

• Existing LLM value studies use instruments such as Schwartz’s theory and the World Values Survey, but probe one culture at a time (Schwartz, 2012; Haerpfer et al., 2022; Ma et al., 2024; Shen et al., 2025b,a).Source: §2 · p. 2
• Personalization studies find that demographic cues, ideological labels, and personas affect opinions and judgments, while most evaluations encode identity as one attribute or a coherent persona.Intersectional work has measured stereotypes rather than conflicting cultural cues (Cheng et al., 2023).Source: §2 · p. 2
• Borah et al. (2026) pit cultural norms against personal preferences and report that country context moves model behavior more than age or gender.Source: §2 · p. 2
• Unlike Resck et al. (2026), which varies query language under a fixed persona, this study pits that persona against both explicit country and implicit language cues.The setup addresses users who may write in one language while identifying with a country where another is spoken.Source: §2 · p. 2

3 Methodology

• The study asks whose values models express when an explicit country cue conflicts with the system-prompt country, and which values they shift toward under a language cue.Source: §3 · p. 2

3.1 Values and Cultural Identity

• Values are measured as WVS answer distributions in multiple-choice probing and as concepts extracted from open-ended responses.Survey items permit exact comparisons across identities but omit values they do not ask about; free text captures values the model chooses to express.Source: §3.1 · p. 2

3.2 System Prompts and User Identity

• A static country identity is held in the system prompt while the user turn supplies a second identity explicitly as a birthplace or implicitly through query language.Explicit cues use natural first-person disclosures; implicit cues use the language associated with a country.Source: §3.2 · pp. 2–3

3.3 Models and Tasks

• The two tasks differ in answer freedom: WVS questions provide fixed options, while IssueBench templates elicit free-text responses on WVS topics.Country references are the model’s own responses under that country’s persona, and shifts are reported without treating them as errors.Source: §3.3 · p. 3
• The evaluation uses three open-weight instruction-tuned models from different families and scales, all covering the eight languages required for language cues.The models are Llama-3.1-8B-Instruct, Ministral-3-14B-Instruct-2512, and Gemma-3-27B-IT.Source: §3.3 · p. 3

4 Value Probing using WVS

• The analysis uses 144 high-opinion-variance WVS questions and compares ten countries with translations available for both explicit and language cues.Explicit-cue results for the other three countries are reported separately in the appendix.Source: §4 · p. 3
• For each question and base identity, the method measures movement as the change in Jensen–Shannon distance from the target country’s English-prompt baseline after adding a cue.A positive value indicates movement toward the target baseline and a negative value movement away; the two cue conditions differ in whether the user names a country or asks in its associated language.Source: §4 · pp. 3–4

4.1 Results

• 0.169 versus 0.051 mean absolute shift: language cues move WVS answer distributions away from target-country baselines for every model and country, about three times the explicit-mention effect.The negative language-cue result does not mean responses move toward another culture; composite responses are not closer to any of the 13 country baselines.Source: Table 1 · p. 4, §4.1 · p. 4
• Language Effect values range from −0.050 to −0.259, while Explicit Mention effects range from −0.082 to +0.074 and vary by model.Gemma shifts toward the target for most countries; Llama and Mistral shift away for nearly all, and language displacement is largest for Llama and smallest for Gemma.Source: §4.1 · p. 4

5.1 Experimental Setup

• The design combines a 13-country system persona with an explicit country named in the English user turn or an implicit cue from the query language.The open-ended study derives 144 sociopolitical topics from WVS questions, instantiates 720 prompts, and translates them into eight languages.Source: §5.1 · pp. 4–5, Table 2 · p. 5
• Non-English responses are backtranslated with Aya before annotation, and reclustering finds the multilingual response clusters remain the same overall as the English-only clusters.Against Google Translate on 100 examples per language, Aya scores 0.896 mean COMET and 0.916 BERTScore F1.Source: §5.1 · p. 5
• Value shifts compare expression percentages against a matched reference: English with the same persona for language cues, or other named countries for explicit cues.The taxonomy contains 512 clusters shared across models, countries, and languages; on a validation sample, judges found verbatim supporting quotes for 91.6% of extracted values.Source: §5.2 · p. 5, §5.1 · p. 5

5.2 Results

• Naming Japan raises harmony expression by 21.9 percentage points, while New Zealand cues introduce two Māori values that are nearly absent under other country names.Whanaungatanga and kaitiakitanga each appear in about 2% of responses when New Zealand is named, versus less than 0.2% otherwise.Source: §5.2 · p. 6
• Language cues produce less distinctive value shifts: Vietnamese prompts raise responsibility expression by 4.4 points, and language changes introduce no new values.After normalizing by the number of values per response, top-value shares increase by 0.35–2.78 points for countries and 0.46–1.85 points for languages.Source: §5.2 · p. 6
• Explicit country cues close 30–44% of the gap in open-ended responses for every model-country pair, but multiple-choice direction varies by model.Gemma moves toward the named country for 10 of 13 countries (mean 12.5%); Llama and Mistral move away for 13 and 12 countries, respectively.Source: §5.2 · pp. 6–7, Figure 2 · p. 7

6 Ablation: From Expressed Values to Choices

• The dilemma test used 2,999 two-action scenarios, presented in both action orders, and ranked 116 values appearing in at least 25 actions.A Bradley–Terry fit used values unique to chosen versus rejected actions to estimate preferences.Source: §6 · p. 7
• Across 533 country and cluster pairs, open-ended value changes and AIRiskDilemmas choice-rank changes were weakly correlated (Spearman ρ = 0.22, p < 10^-6).The correlation was positive for 9 of 13 countries and held only within a country.Source: §6 · p. 7
• Honesty rose most in choice rankings for the United Kingdom, Germany, New Zealand, Portugal, and Spain, while it fell most for Japan, China, and Vietnam.These country groups corresponded to open-ended increases in fairness, responsibility, community, or solidarity versus harmony or respect; Russia’s cue produced the largest table decline, 13.7 places for human trust preservation.Source: §6 · pp. 7–8, Table 3 · p. 8
• Stating a country closed 30 to 44% of the gap to that country in open-ended generation, but moved multiple-choice answers toward it only for Gemma.The authors note that both multiple-choice probing and dilemma choices select from fixed options, consistent with weak carry-over from expressed values to choices.Source: §6 · p. 8

7 Discussion and Conclusions

• Without ground-truth value distributions for genuinely composite individuals, the paper does not prescribe a normative target or a single weighting of country and language cues.The authors characterize their findings as empirical, not as a claim about the values or preferred behavior of composite-identity individuals.Source: §7 · p. 8
• Identity-conditioned value expression may support personalization, but the analysis does not cover the trade-off with safety responses.The discussion also notes that nationality personas can favor their own region and that models vary in how strongly country and language cues affect different value dimensions.Source: §7 · p. 8
• Country and language are not interchangeable cultural signals: language drives the largest WVS shifts, while country cues yield more differentiated open-ended values and culture-specific concepts.The effects depend on task and model, ruling out a single cue hierarchy and motivating evaluations that vary country and language independently.Source: §7 · p. 8

Limitations

• The experiments characterize model behavior under controlled cues, not the correct values of people with genuinely composite identities.The WVS analysis compares response distributions, while open-ended analysis compares the share of responses expressing each value; evaluation data from genuinely composite identities remains future work.Source: Limitations · pp. 8–9
• Using query language as an implicit identity cue tests language interactions under a system-prompted identity, not cultural identity itself.Language and cultural identity are related but distinct, and users of a language may not share the background most associated with it.Source: Limitations · p. 9
• Country as a culture proxy overlooks cultures that do not align with national borders and substantial variation within countries.The passage identifies variation across regions, generations, socioeconomic groups, and individual identities.Source: Limitations · p. 9
• The study tests only two simultaneous identities, leaving the generalizability of its findings to more complex intersections open.Real-world identities can intersect across language, nationality, religion, ethnicity, and lived experience.Source: Limitations · p. 9
• WVS translation availability constrained country selection, underrepresenting regions including much of sub-Saharan Africa and Southeast Asia.This limits the geographic scope of the conclusions.Source: Limitations · p. 9
• External models perform value extraction, clustering, naming, and translation; their errors can affect results, particularly for low-resource languages.Evaluation data for these components is scarcer in low-resource languages.Source: Limitations · p. 9

A Model details and Experiment Details

• Three instruction-tuned open-weight models—Llama, Gemma, and Mistral—produce 1,628,640 open-ended responses and 7,949,706 multiple-choice answers.The models span 8B, 14B, and 27B parameters; generation required approximately 10 days on 8 Nvidia A100s.Source: A Model details and Experiment Details · p. 12

B Full Input Prompt

• The response-generation prompt fixes one country in the system prompt and introduces a second through the user turn, explicitly or via query language.For WVS, a different system prompt lists age, gender, and cultural background without prescribing values.Source: Figure 3 · p. 13, B Full Input Prompt · p. 13

C Open-Ended Value Taxonomy

• A frozen taxonomy groups 13.6 million value mentions into 512 clusters, using embeddings and clustering on 100,008 unique cleaned strings.Cluster names summarize their members rather than imposing a predefined moral theory.Source: C Open-Ended Value Taxonomy · p. 13
• The automatically induced taxonomy retains near-duplicate labels, affecting label granularity more than cue contrasts because the same frozen mapping is used across conditions.Absence of a cluster means the extractor did not return a mapped value, not necessarily that the response lacks the concept.Source: C Open-Ended Value Taxonomy · p. 13

D Validating the Value Annotator

• 91.6% of extracted values were confirmed by a majority of three LLM judges, with precision ranging from 85.8% to 97.3% across model and setting combinations.The judges agreed on 85.1%–87.3% of pairwise decisions, with Fleiss’ κ of 0.53.Source: D Validating the Value Annotator · p. 13
• The annotator recovered 77.0% of clusters shared by at least two other annotators, while mean Jaccard overlap across annotator pairs ranged from 0.35 to 0.54.Rerunning the annotator reproduced its stored value set exactly for 75.6% of responses.Source: D Validating the Value Annotator · p. 14
• On sampled model responses, the annotator returned 8.0 values per response and covered 38.6% of judge-confirmed clusters, reflecting that its prompt does not request every value.With an exhaustive prompt, Gemma-3-27B covered 61.2% at 92.1% precision and Llama-3.1-8B covered 76.4% at 81.6% precision.Source: D Validating the Value Annotator · pp. 13–14
• On the Value Kaleidoscope test split, cluster-level precision was 32.6% and recall was 27.0% among situations with output, with at least one match in 66.8%.Across all 3,120 situations, recall was 17.8% and the match rate was 44.0%; generic frequent values contributed to low cluster-level precision and recall on this out-of-domain input.Source: D Validating the Value Annotator · pp. 14–15

E WVS Questions to Topics

• GPT-4.5 generated candidate topics from WVS questions, and three annotators edited and selected concise, clear, semantically faithful prompts by consensus.Candidates that were overly broad, ambiguous, or weakly grounded were revised or discarded.Source: E WVS Questions to Topics · p. 15

F Supporting Results for Open-Ended Generation

• The 30 most frequent value clusters accounted for 49% of labels, while only New Zealand’s two Māori values occurred in at least 1% of responses for one country and under 0.2% for others.Both country-specific increases remained significant after false-discovery-rate correction.Source: F Supporting Results for Open-Ended… · pp. 15–16, Table 8 · p. 17
• In an illustrative Japan-cue comparison, 20.2% of 77,760 paired responses expressed harmony only after the cue, and harmony was Japan’s largest increasing value.The two Gemma responses shared the system persona and prompt; the Japan-cued response expressed harmony, obligation, respect, and support.Source: F Supporting Results for Open-Ended… · p. 16
• For 11 of 13 named countries, the largest increase in response-level value frequency exceeded the largest increase for every language; after normalization, country and language increases overlapped.Normalized largest increases ranged from 0.35 to 2.78 points for countries and 0.46 to 1.85 points for languages.Source: F Supporting Results for Open-Ended… · p. 16
• Country-specific shifts compare naming one country with naming the other 12, and the findings concern values identified by an automated extractor in model responses, not people’s values.The pooled regressions omit model-by-factor interactions; the authors retain a linear probability model for percentage-point interpretability.Source: F Supporting Results for Open-Ended… · p. 17

G Explicit-Mention Effects for All Probed Countries

• The all-country supplement reports explicit-mention effects for all 13 probed countries, including three without a native-language counterpart in the WVS comparison.Those three countries are marked as lacking a WVS translation.Source: G Explicit-Mention Effects for All… · p. 17

H Convergence by Model

• Pooled closed-ended convergence averages across model disagreement: Llama moves away from the named country for all 13 cues, and Mistral does so for 12.In open-ended generation, every model converges for every cue.Source: H Convergence by Model · p. 17

I System-Prompt Robustness

• Under every system prompt, naming a country increased the percentage expressing its top value for 12 or 13 of the 13 countries, with 95% intervals above zero.The typical-citizen prompt was the main-experiment prompt; alternatives asked about user background, gave country-only information, or supplied timezone-only information.Source: I System-Prompt Robustness · pp. 17, 19
• Under the typical-citizen prompt, naming Japan increased harmony-expressing responses by 16.6 points, versus 5.2 to 7.9 points under the other three prompts.The median increase was 7.7 points under the typical-citizen prompt and 3.1 to 4.3 points under the alternatives.Source: I System-Prompt Robustness · pp. 19–20
• Explicit cues closed 17% to 31% of the named-country gap under the typical-citizen and user-background prompts, where persona profiles were clearly distinct.Under country-only and timezone-only prompts, persona profiles differed no more than two samples of the same profile, so there was no country profile to converge toward.Source: I System-Prompt Robustness · p. 20
• The robustness study used one model, 288 IssueBench prompts, and one cue sentence per prompt, and its absolute convergence was lower partly because the sample was smaller.The main experiment used 720 prompts and three cue sentences per cell.Source: I System-Prompt Robustness · pp. 19–20

J Where Language Cues Move Answer Distributions

• After language cues, shifted WVS distributions were 0.214 from their closest country baseline on average, about the 0.216 distance between two country baselines, and farther from every country than that in 39.3% of cells.The target country was closest in only 8.3% of cells, and the persona country in 10.7%.Source: J Where Language Cues Move Answer… · p. 20, Table 18 · p. 25
• The closed-ended country-cue result averages across models with different signs: only Gemma converges for most cues, while Llama moves away for all 13 and Mistral for 12.Language-cue divergence is consistent across models in the closed-ended setting.Source: Table 13 · p. 21, Table 14 · p. 21
• Language cues reduce country-specific answering rather than pull answers toward another culture: the target was closest in 8.3% of cells, while anglophone baselines were no nearer than the target on average.Anglophone baselines were closest in 24.2% of cells, and were 0.379 away on average, as was the target country.Source: J Where Language Cues Move Answer… · pp. 20–21

K Bradley–Terry Value Ranking

• The Bradley–Terry ranking treats each action choice as pairwise evidence between values unique to the competing actions, then estimates a strength for each value.Ranks are compared under each of 13 named-country cues against the mean rank under the other 12 countries.Source: K Bradley–Terry Value Ranking · pp. 21–22
• Japan’s harmony shift is the largest country-specific increase in the Figure 10 comparison, while language shifts use a radial scale about five times smaller.Figure 10 plots the three strongest increasing values for each country relative to the other 12 named countries; Figure 11 compares language cues with English.Source: Figure 10 · p. 23, Figure 11 · p. 24