PaperlayerView exact revision on arXiv

Source-linked AI summary

A Russian, a Brazilian, and a German Walk into a Bar, Ask the Same Question-- And Get Three Different Answers: Profiling LLM Perspective Diversity on Contested Topics

Wiem Ben Rim, Yao Lu, Pontus Stenetorp, Emine Yilmaz

arXiv:2610.34278v1cs.CL

TL;DR

Many real-world questions are subjective and culturally situated, while research on contested queries has largely focused on English. This paper builds an 18-language dataset and a framework for profiling model diversity, finding that stance varies by language and that models are sensitive to small framing changes.

  • Problem

    Research on LLM responses to contested queries has primarily focused on English and a narrow range of regional values, leaving broader cross-language perspective variation underexplored.

    Source: §1 · p. 1

  • Method

    The paper curates PERSPECTIVESQA, a manually verified dataset of contested questions across 18 languages, and proposes ROOTS to profile linguistic divergence and framing sensitivity.

    Source: §1 · p. 1

  • Results

    LLMs hedge in English but commit to positions in lower-resource languages; larger models are more cross-lingually consistent, while all models remain sensitive to minor input variations.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Capturing diverse perspectives is essential as users turn to LLMs as mediators of knowledge and opinion.

    Source: §1 · p. 1

    The authors acknowledge several limitations in their approach but do not characterize them in this passage.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large Language Models(LLMs) have shown strong performance on tasks with verifiable answers, but many real-world queries involve subjectivity, disagreement, and conflicting evidence. Most existing benchmarks for such queries remain English-centric, overlooking the vast diversity of perspectives held by users worldwide. To systematically study how LLM perspectives vary across languages, we curate PerspectivesQA, a manually verified multilingual dataset of globally contested topics covering 18 languages, and propose ROOTS, a framework that profiles linguistic, perspective, and framing diversity. We evaluate a diverse set of eight models covering both open-source and proprietary families at varying scales, establishing that LLMs hedge their answers in English but commit to positions in lower-resource languages. Larger models maintain a more consistent cross-lingual worldview, while smaller models are more easily swayed. All models also exhibit high sensitivity to minor surface-level input variations, producing notable stance even in larger models.

1 Introduction

Many subjective questions depend on users’ cultural, linguistic, and ideological contexts, while existing work on contested queries has largely focused on English and a narrow range of regional values. The paper introduces an 18-language dataset and ROOTS framework, then reports cross-language stance differences and sensitivity to input variation.

1 Introduction

  • Real-world queries often lack a single correct answer and depend on cultural, linguistic, and ideological context.The paper notes that such questions can invite models to present training-data biases as universal fact.Source: §1 · p. 1
  • Earlier work on contested and subjective queries primarily examined English-language contexts and a narrow range of regional values, leaving broader cross-cultural perspective variation underexplored.The paper asks whether models respond consistently across languages or collapse into a single dominant narrative.Source: §1 · p. 1
  • LLMs hedge in English but commit to positions in lower-resource languages, and all models show sensitivity to minor surface-level input variations.The paper frames these findings as a study of how model perspectives vary across languages and input framings.Source: §1 · pp. 1–2
  • PERSPECTIVESQA is a manually curated dataset of globally contested questions covering 18 languages.The paper also introduces ROOTS to profile linguistic diversity and framing sensitivity.Source: §1 · pp. 1–2

2 The PERSPECTIVESQA Dataset

PERSPECTIVESQA assembles human arguments for globally contested questions, then expands the questions across languages. It also uses existing datasets to test generalization and provide a factual negative control.

Dataset construction

  • The final dataset contains 20 topics and 1,280 arguments, with a median of 48 reference arguments per question and nearly even pro and con shares of 47% each.Each topic has one question.Source: §2 · p. 2

Topic Selection

  • Topics draw on the World Value Survey and Pew question database; Claude Sonnet generates related questions that are manually verified.The surveys supply topic ideas rather than generation-ready prompts.Source: §2 · p. 2

Argument Collection

  • KIALO arguments are retrieved by query similarity and manually filtered, while ChangeMyView comments are split into unique arguments and verified by two assessors.The resulting arguments form a reference space of positions for and against each question.Source: §2 · p. 2

Multilingual Expansion

  • Questions were translated into 18 languages using human translators for nine languages and machine translation with native or fluent-speaker verification for eight others.Annotators rated 84% of translations fully correct, and major errors occurred in 0.8%.Source: §2 · p. 2

Additional Datasets

  • DebateQA tests generalization beyond PERSPECTIVESQA, while factual SimpleQA serves as a negative control.SimpleQA covers topics including sports, politics, and technology.Source: §2 · p. 2

3 ROOTS: A Diversity Framework

ROOTS profiles model diversity through linguistic divergence and framing sensitivity, using experiments and metrics to compare model outputs with human viewpoints. The evaluation includes open-source and commercial models at different scales.

3 ROOTS: A Diversity Framework

  • ROOTS creates a diversity profile to compare model outputs with human viewpoints across linguistic and perspective divergence.The framework is organized around two axes, each with experiments and metrics.Source: §3 · p. 3

Diversity Profile

  • Linguistic Diversity measures output variation across languages for the same contested query, while Framing Sensitivity examines response distributions when the question is rephrased.The framework includes qualitative analysis and a cultural grounding study.Source: §3 · p. 3

Experimental Setup

  • The evaluation uses Qwen3 and Gemma open-source models at multiple sizes and commercial Claude Haiku, omitting reasoning outputs from model responses.Linguistic-divergence experiments use nine languages because generation, judging, and translation are computationally demanding.Source: §3 · p. 3

4 Diversity Profile Axes

The diversity profile measures cross-lingual response divergence and sensitivity to minor query rewordings. Results show stance variation across languages and framings, with model size associated with cross-lingual consistency and framing robustness.

4.1 Linguistic Divergence

  • The profile measures linguistic divergence geometrically with MMD and categorically with an LLM judge, using extracted atomic arguments and back-translation to estimate and correct language-specific embedding offsets.The judge assesses reference coverage and stance shifts relative to English; an additional measure compares English arguments with arguments extracted from back-translated responses.Source: §4.1 · pp. 3–4
  • SimpleQA provides a conservative noise floor for cross-lingual drift; after subtracting it, Amharic’s divergence is indistinguishable from noise, while Swahili retains a perspective shift above the floor.The two MMD approaches’ language means correlate strongly, and MMDp correlates with judge-assessed coverage and stance shifts.Source: §4.1 · p. 5
  • Smaller models shift stance more often from English, while larger models align more closely; divergence drops significantly at 4 billion parameters for Gemma and Qwen, but a low-resource-language gap remains.Increasing parameter count correlates with higher robustness to framing, although susceptibility also varies by model family.Source: §4.1 · p. 6, §4.2 · p. 7

Finding 4A: English both-sides controversies

  • English, Brazilian Portuguese, and Chinese take both_sides on 69–84% of PerspectivesQA questions, compared with 32% in Amharic and Kazakh; lower-resource languages also show larger stance shifts.The pattern is smaller on DebateQA, and Chinese, Arabic, and Portuguese also show stance shifts, though at lower rates than Amharic, Swahili, and Kazakh.Source: Finding 4A: English both-sides… · p. 6, Figure 3 · p. 6, Table 4 · p. 6
  • Observed cross-lingual shifts include hedging collapse, degraded comprehension, and localized worldview differences, so divergence can reflect more than a change in expressed perspective.The paper reports comprehension failures and degradation among smaller models, while larger models can still exhibit true perspective shifts.Source: Finding 4A: English both-sides… · pp. 6–7

4.2 Framing Sensitivity

  • Adversarial prompts polarize (shift stance to pro or con) an average of 27.4% of the time, versus 21.4% for neutral phrasings of the same question, an excess of +6.0 percentage points.Figure 4 shows shifts from baseline both_sides to pro or con under adversarial framing.Source: §4.2 · p. 7, Figure 4 · p. 8
  • Framing robustness varies by model: Gemma 4B changes stance 43.5% of the time, Qwen3 4B 40.8%, and Claude Haiku 21.6%.Increasing parameter count generally correlates with higher robustness.Source: §4.2 · p. 7

5 Related Work

Prior work on contested questions has largely centered on English and Western perspectives, while multilingual studies have addressed different tasks or used more constrained formats. This paper extends that work to open-ended argument generation across languages and treats cross-lingual behavior as descriptive, not as a measure of cultural alignment.

Controversial and multilingual QA

  • Earlier controversial-QA work focused mainly on English and Western-centric data, with some benchmarks detecting controversy or labeling human emotion rather than evaluating generated responses.The paper distinguishes its approach from DELPHI, which lacks textual responses, and from GlobalOpinionsQA, which uses multiple-choice questions.Source: §5 · p. 8
  • This work measures cross-lingual consistency in open-ended argument generation, complementing GlobalOpinionsQA’s country-tagged opinion distributions and multiple-choice evaluation.It also uses GlobalOpinionsQA in a cultural-grounding study.Source: §5 · p. 8
  • Prior research found that larger models can improve factual probing accuracy without improving cross-lingual consistency, while low-resource languages can degrade factual recall and quality.The paper treats this degradation as a confounder in its experiments.Source: §5 · p. 8

Cross-lingual evaluation and embedding

  • Unlike centroid subtraction, this paper estimates language-specific offsets from parallel back-translations and treats measured cross-lingual shifts as descriptive rather than evidence of cultural alignment.Language communities may correlate with, but do not determine, cultural context.Source: §5 · p. 8

6 Conclusion

The paper introduces PerspectivesQA and ROOTS to study linguistic divergence and framing sensitivity in contested-topic responses across 18 languages. It reports that models hedge in English but more often commit in lower-resource languages, while framing changes can shift stances across model sizes.

6 Conclusion

  • Models adopt nuanced stances in English but commit to positions more often in lower-resource languages, especially smaller models.The authors report that models of all sizes are sensitive to minor framing variations in both neutral and adversarial cases.Source: §6 · p. 8
  • The authors introduce PerspectivesQA, a manually verified dataset of contested questions in 18 languages, and ROOTS, which profiles linguistic divergence and framing sensitivity.They use back-translation in MMD calculations and a factual dataset as a noise control, reporting that measured shifts are not evaluation artifacts.Source: §6 · p. 8
  • The authors conclude that English-only evaluation underestimates the perspective variance users worldwide encounter in practice.They suggest current alignment and safety-training practices generalize unevenly across languages.Source: §6 · p. 8

Limitations

The authors identify limitations in how they assess stance, represent culturally salient topics, calibrate translation noise, and validate evaluator choices.

Limitations

  • The stance analysis relies on an LLM judge and back-translation; an English-aligned judge may misclassify low-resource responses as unilateral.The concern is that it may reward English-style “both-sides” positioning.Source: Limitations · p. 9
  • DebateQA’s Western-salient controversies measure responses to imported topics, while PerspectivesQA’s global target still makes localized and Western-focused responses difficult to distinguish.The authors call for pluralistic datasets sourced directly from diverse cultural contexts.Source: Limitations · p. 9
  • MMDbt calibration uses SimpleQA as a factual-noise baseline, assuming factual degradation is at least as large as degradation in complex subjective reasoning.Factual entities and logical counter-arguments may be dropped through different mechanisms under low-resource constraints.Source: Limitations · p. 9
  • Resource constraints prevented additional validation of LLM-as-a-judge models across all datasets.API costs blocked testing other judges, including GPT, Claude, and Gemini, on every dataset.Source: Limitations · p. 9

Ethics Statement

The statement addresses the contested nature of the dataset and cautions against treating language as a proxy for speakers’ values; it also describes dataset sourcing and argument review.

Ethics Statement

  • PerspectivesQA documents human-authored positions on contested topics, including arguments that may offend, without endorsing them.The authors state that excluding uncomfortable positions would undermine the benchmark’s intended measurement.Source: Ethics Statement · p. 9
  • Language is treated as a correlate, not a determinant, of cultural context; model stances—especially amid generation degradation—should not be attributed to language communities.The nine-language evaluation subset over-represents languages with large web presences, and high degeneration makes estimates for Amharic, Kazakh, and Swahili less reliable.Source: Ethics Statement · p. 9

A.1.1 Question Generation

  • Dataset questions were generated from WVS/Pew survey topics, with Claude Sonnet used to produce related questions.The supplied text names “Justifiability of Abortion” as an extracted topic example.Source: §A.1.1 · p. 12

A.1.2 Argument extraction

  • Two assessors verified arguments retrieved from Reddit and Kialo, removing disputed samples; participants were told their outputs would enter a publicly released dataset for language-model evaluation, with no personal information collected or published.PerspectivesQA contains 20 questions and 1,280 arguments; each question is available in 18 languages and manually verified.Source: §A.1.2 · p. 12

A.1.3 Annotators

The appendix reports annotation procedures and agreement, outlines embedding-based language-divergence measures, and notes a correlation between model size and robustness.

A.1.3 Annotators

  • Native speakers translated questions into nine languages; machine translations for eight more were verified by native or fluent annotators, with 84% and 76% rated correct for PerspectivesQA and DebateQA.Reported major-error rates were 0.8% and 3.5%, respectively; Gwet’s AC for major errors was 0.98 and 0.92.Source: §A.1.3 · p. 12

A.2 Language Divergence

  • Language divergence compares English and other-language argument embeddings with MMD, using an RBF similarity kernel and parallel-text offsets to adjust embeddings.The described alternatives also include corpus-centering embeddings by language.Source: §A.2 · p. 13

A.3.1 Number of Parameters effect on MMD

  • Increasing parameter count correlates with higher robustness across PerspectivesQA, DebateQA, and SimpleQA.The appendix refers to Figure 9 for model-size effects and Table 6 for correlations with other outcome metrics.Source: §A.3.1 · p. 13

A.3.2 Model Stance Composition

The appendix presents stance composition by model and dataset and describes degeneration and stance patterns across languages.

A.3.2 Model Stance Composition

  • Table 7 reports English-language pro, con, neutral, and both_sides stance percentages by model and dataset, with admissible-cell counts and modal stances marked.PerspectivesQA’s 20 questions mean each cell changes in five-point steps.Source: Table 7 · p. 14
  • Nearly 50% of Swahili and Amharic answers degenerate in PerspectivesQA, mostly through repetition collapse or irrelevant answers; a similar trend appears in DebateQA.Amharic, Kazakh, Swahili, and Hindi are identified as the most affected languages.Source: Figure 7 · p. 15
  • Lower-resource languages are more likely to commit to a stance, while higher-resource languages more often hedge in DebateQA.The figure arranges languages from lower to higher resource from left to right.Source: Figure 8 · p. 15

A.3.3 Cultural grounding

The cultural-grounding analysis compares population survey stances with model outputs; the framing design tests stance changes under prompt rewrites while keeping the question unchanged.

A.3.3 Cultural grounding

  • The analysis matches GlobalOpinionsQA survey topics with DebateQA and PerspectivesQA, then filters to countries speaking tested model languages (Durmus et al., 2024).Source: §A.3.3 · p. 15
  • For an American-surveillance privacy question answered in 44 countries, survey respondents agreed or disagreed, while models were generally neutral.Source: §A.3.3 · p. 15
  • Figure 9 plots parameter count against MMD for DebateQA, PerspectivesQA, and SimpleQA.Source: Figure 9 · p. 16

A.4.1 Framing Design

  • Each question receives three neutral domain-lens framings and two adversarial-premise framings; rewrites change only the prompt, not the question.The design includes 1,000 DebateQA questions and 20 PerspectivesQA questions.Source: §A.4.1 · p. 17

A.4.2 Judge validation

Judge validation found stronger agreement on language-level comparisons than on individual stance labels. The appendix also reports stance shifts across prompt framings and provides the judge prompts.

A.4.2 Judge validation

  • Both judges found the same polarization effect in DebateQA under neutral and adversarial prompting.Source: §A.4.2 · p. 17, Table 9 · p. 17
  • Per-language both_sides correlations were 0.82 for DebateQA and 0.78 for PerspectivesQA, despite 58% and 66% individual-label agreement.Source: §A.4.2 · p. 17, Table 10 · p. 18
  • Models were more likely to take sides under adversarial prompts and under neutral prompts that added context.Source: Figure 10 · p. 18

B LLM Judge Prompts

  • The judge-prompt appendix includes a system prompt and version-three stance prompts for language-divergence and framing-sensitivity experiments.The stance prompts were revised to approximate GPT-5 judgments and specify stance assignment.Source: Figure 11 · p. 19, Figure 12 · p. 19, Figure 13 · p. 19
Loading 2610.34278v1…