GYROval: A Robust Benchmark for Cultural Value Orientation in Large Language Models

1.1 Cultural value orientation as a measurement problem

• Model rank order is preserved across roles, topics, languages, and temperatures even when absolute scores shift by more than 20% of the scale, so scores require modality and domain context.The proposed stability metric compares models’ relative rankings across perturbation levels.Source: §1.1 · p. 2

1.2 Existing instruments and the reliability evidence they report

• Across fourteen instruments, only three report reliability statistics, and their theoretical frameworks are non-translatable, limiting cross-framework comparisons of instrument efficacy.CDEval supplies the item architecture, scoring metric, and domain framework adopted here.Source: §1.2 · pp. 2–3

1.3.1 Option-order bias: token or position, and the consequences for remedies; 1.3.2 Repeated sampling versus counterbalancing at a fixed budget

• Option bias is token-level and model-specific, making cyclic permutation preferable to fixed slots; at equal cost, counterbalanced GPT-4 achieved 62.5% versus 60.9% repeated-sample agreement.For binary items, cyclic permutation costs 2×; hosted APIs that return text responses rarely expose the token probabilities needed for contextual or batch calibration.Source: §1.3.1 · pp. 4–5, §1.3.2 · p. 5

1.3.3 Temperature-zero measurement

• Temperature zero does not guarantee deterministic API outputs, and deterministic serving does not guarantee construct validity; enforcing reproducibility can reduce throughput by roughly 33%.The cited empirical example reports that a temperature-0 model produced 80 unique completions across 1,000 runs.Source: §1.3.3 · p. 6

1.3.4 The defensible unit of analysis

• Analyses should use items as the unit of analysis; with panels of 11–20 models, nominal 5% tests can have inflated Type I error rates of 8%–12%.Treating repeated responses to one item as independent replicates violates independence assumptions; models are clusters when ranking them.Source: §1.3.4 · p. 6

1.3.5 Forced choice versus Likert

• Evidence on forced-choice formats is disputed, while this instrument’s second option is longer and more heavily hedged, creating a potential lexical cue.Prior findings variously suggest forced choice deflates or inflates alignment effects, or yields relative consistency with open-ended formats.Source: §1.3.5 · pp. 6–7

1.3.6 Whether the construct validity of human psychometric instruments transfers

• Human psychometric validity does not transfer straightforwardly: model forward- versus reverse-keyed correlations are +.61 to +.81 versus humans’ −.69 to −.82, and cross-cultural invariance remains disputed.Strict scalar measurement invariance is also absent from the human emancipative-values index across cultural zones and countries.Source: §1.3.6 · p. 7

1.4 Objectives and Contributions; 1.5 Evolution of the Benchmark: Dimensional Framework and Longitudinal Elicitation

• The benchmark comprises 224 items fully crossed across seven domains, four decision modalities, and eight value facets, evaluated on twenty models, with paired identical translations for eleven.It adapts CDEval’s binary semantic-vignette approach, shifts from Hofstede dimensions to the Inglehart–Welzel axes, and replaces one-shot prompts with contextualized vignettes. The prior study found language effects could exceed between-model differences and that bimodal responses could be obscured by reporting a mean pole share.Source: §1.5 · pp. 8–9, §1.4 · p. 8

1.5 Evolution of the Benchmark: Dimensional Framework and Longitudinal Elicitation

• Rank-order concordance significantly exceeds the empirical permutation null in all twelve factor–axis conditions across four non-dispositive perturbations.The perturbations are decision role, thematic domain, language of administration, and sampling temperature; associated absolute level shifts are also reported.Source: §1.5 · p. 9
• Role sensitivity varies across models by a factor of 6.5 on the emancipative axis and 5.3 on the sacred–secular axis.The study reports individual per-model sensitivity metrics.Source: §1.5 · p. 9
• Linear probing, difference-of-means directions, RSA with CKA, and causal activation patching confirm that benchmark-measured behavioral distinctions are reflected in model representations.Source: §1.5 · p. 9

2.1 Construct and factorial design of the instrument

• The 224-cell instrument scores each model by its share of responses on a counted pole across two axes, seven themes, four modalities, and eight nested value facets.The options are both legitimate, and the released dataset has 225 rows because one cell contains a duplicate item.Source: §2.1 · p. 9

2.2 Item generation, expert review and translation

• Three experts screened generated scenarios, but the released materials omit the specific inclusion criteria and inter-rater agreement metrics.The authors prioritize novel contemporary scenarios over reproducing legacy survey phrasing.Source: §2.2 · pp. 10–11

2.3 Model panel and administration conditions

• Twenty models took the English evaluation, while eleven also received the paired Russian items and a second decoding temperature; no human baseline or external criterion measures were collected.The study therefore targets model rank-order stability and concordance, not absolute accuracy, population calibration, or external criterion validity.Source: §2.3 · p. 11, §2.2 · p. 11

2.4 Unit of analysis

• The analysis aggregates responses into per-model, per-vignette proportions and treats vignettes—not repeated runs—as the units for concordance.Treating 6,750 runs as independent would inflate the sample size and underestimate standard errors; the 225-vignette design also limits power for small effects.Source: §2.4 · p. 12

2.5 Concordance coefficient and permutation null

• For each perturbation, factor levels act as raters and models as targets; tie-corrected Kendall’s W measures agreement between model rankings against a permutation null.The null uses 20,000 permutations of independent random rankings, with the expected value depending on the number of levels.Source: §2.5 · pp. 12–13

2.6 Bootstrap intervals and the three kinds of variation; 2.7 Treatment of refusals and specification sensitivity

• Vignette-resampling bootstrap intervals quantify item-sampling variance, while 48 analytical specifications test sensitivity to refusal handling, duplicates, scoring, and low-valid-run exclusions.Prompt-level variation is outside scope; the intervals assume the 225 vignettes represent a broader domain of relevant scenarios.Source: §2.6 · p. 13, §2.7 · pp. 13–14

2.8 Level shifts, role sensitivity, paired contrasts and stability ceilings

• The methods define role sensitivity as each model’s range across modalities, paired language and temperature shifts against a split-run noise floor, and empirical ceilings for rank stability.Item-factor ceilings preserve model-by-item interaction variance; paired-factor ceilings use identical vignettes and estimate run-to-run noise.Source: §2.8 · p. 14

2.9 Reasoning-mode condition; 2.10 Mechanistic sub-study

• A separate reasoning-mode condition measures choice agreement and trace-based metrics, while a mechanistic sub-study probes value directions, cross-model alignment, and causal patching in open-weight models.Neither the reasoning-mode condition nor the mechanistic analyses enter the main vignette-level concordance pipeline; the mechanistic dataset selects high-disagreement items and unanimous controls.Source: §2.9 · p. 14, §2.10 · p. 15

3.1 Response yield, run-level agreement and effective sample size

• Per-model self-consistency ranged from 91.7% to 100% at temperature 0, with a median near 99.5%; the effective sample size was 214–296 per model.The median effective sample size was 227, far below the 6,338–11,200 raw responses per model because runs cluster within vignettes.Source: §3.1 · p. 16
• Alice AI LLM and YandexGPT had identical per-vignette choice proportions on all 214 shared valid vignettes, leaving 19 distinct architectures across 20 designations.Ten design cells were unparseable for both models and excluded from joint statistical evaluations.Source: §3.1 · p. 16

3.2 Concordance under the four perturbations

• All experimental cells significantly exceeded their empirical nulls across decision role, theme, language, and temperature on both axes, indicating above-chance rank concordance.Item-factor stability ceilings are below 1, so exceeding chance is a baseline result rather than evidence of perfect stability.Source: §3.2 · p. 16, Figure 1 · p. 17
• Claude Sonnet 4.5 stayed in the top two across modalities on both axes, while mid-tier ranks shifted by 9–14 positions in some cases.T-lite ranged from rank 19 to 4 on the emancipative axis; gpt-oss-120b ranged from rank 4 to 18 on the sacred–secular axis.Source: §3.2 · p. 17

3.3 Level shifts by decision modality and theme

• The pooled counted-pole share ranged from 0.573 for Agent to 0.797 for Judgment of another’s behaviour, a modality spread of 0.224.Across themes, scores ranged from Family (0.756) and Wellness (0.753) to Scientific (0.549) and Work (0.578), a spread of 0.207.Source: §3.3 · p. 17, Figure 3 · p. 18, Figure 4 · p. 19

3.4 Role sensitivity per model

• Across models, role sensitivity ranged from 0.071 to 0.461 on the emancipative axis and from 0.094 to 0.500 on the sacred–secular axis.Mistral Medium 3.1, Claude Sonnet 4.5, and Gemini 2.5 Pro had ranges no greater than 0.12 on both axes; GigaChat 2 Max, A-vibe, and T-Lite had ranges of at least 0.35 on both.Source: §3.4 · p. 19, Table 1 · p. 20
• A-vibe’s sacred–secular counted-pole share varied by 0.500 across its extreme decision modalities, half the measurement scale.This illustrates the model-level role sensitivity summarized across modalities in Table 1.Source: §3.4 · p. 19, Table 1 · p. 20

3.5 Stability relative to empirical ceilings

• Seven of eight factor-by-axis cells fall below the empirical stability ceiling; decision role has the largest impact and temperature the smallest.The sole exception is thematic domain on the sacred–secular axis, which is consistent with complete rank stability.Source: §3.5 · pp. 20–21
• Restricting analysis to the eleven models tested in all four conditions lowers concordance values, but preserves the relative factor hierarchy.Decision role remains less stable than thematic domain across both axes; temperature preserves the highest proportion of stability.Source: §3.5 · p. 21

3.6 Language of administration

• Russian administration shifts scores toward the counted pole for ten of eleven models on the emancipative axis and seven of eleven on the sacred–secular axis.Mean absolute displacements are 0.029 and 0.036, respectively.Source: §3.6 · p. 21
• At the vignette level, two of 22 language contrasts are uncorrected-significant, and neither survives Bonferroni correction.At the individual-response level, twelve contrasts are uncorrected-significant and eleven survive correction; the analysis treats vignettes as the primary sampling unit.Source: §3.6 · p. 21
• English–Russian rank concordance is W = 0.909 overall, with W = 0.951 on the emancipative axis and W = 0.895 on the sacred–secular axis.Both axis-specific concordances fall below the 1st percentile of the permutation ceiling.Source: §3.6 · p. 21
• Language-induced score displacement does not systematically align with developer geographic origin.The four largest displacements span both Russian and non-Russian development organizations.Source: §3.6 · p. 21

3.7 Sampling temperature

• Temperature variation yields pooled W = 0.962 and no statistically significant differences for any of eleven tested models on either axis.It preserves 0.966 of theoretical stability on the emancipative axis and 0.989 on the sacred–secular axis.Source: §3.7 · p. 22

3.8 Reasoning mode

• Reasoning-mode selections align with base-mode selections on 90% to 97% of items across models, peaking at 97% for Alice AI LLM.A classifier using reasoning-trace embeddings predicts final selections with 0.854 accuracy, 0.812 balanced accuracy, and 0.885 ROC-AUC.Source: §3.8 · p. 22, Figure 5 · p. 22

3.9 Sensitivity to analysis specification, and two properties of the intervals

• Across 48 analysis-specification permutations, concordance conclusions remain stable; imputing non-A/B responses shifts Kendall’s W by at most 0.0051.The alternative pipeline imputes 1.02% of English response cells.Source: §3.9 · p. 22
• Bootstrap resampling attenuates concordance on average, so its upper interval bound should not be treated as an upper limit on W.Two of twelve percentile intervals exclude their empirical point estimates; pooled temperature instead has a bootstrap mean above its point estimate.Source: §3.9 · p. 23

3.10 Internal consistency

• Concordance measures rank-order preservation across conditions rather than internal test reliability within a single condition.Source: §3.10 · p. 23

3.11.1 Behavioural positions and composition of the activation set

• The four primary models agree on 61.8% of deterministic evaluations, while between-model disagreement occurs on 106 of 225 items.Of the 120 signal items selected for activation extraction, 25 have zero inter-model disagreement, leaving 95 variance-bearing items; control items were excluded from activation extraction.Source: §3.11.1 · p. 23

3.11.2 Linear probes

• Linear probes recover axis labels above majority-class baselines in all eight model–axis cells, with accuracies from 68.3% to 96.7%.The reported range is against baselines of 51.7% to 90.0%; for example, T-lite reaches 93.3% on the sacred–secular axis against a 53.3% baseline.Source: §3.11.2 · pp. 23–24, Figure 7 · p. 24

3.11.3 Difference-of-means directions

• Four of six within-group direction similarities are statistically indistinguishable from orthogonal random vectors; the strong exception is the Russian fine-tuned pair on the sacred–secular axis.That pair has z = 18.9, which the study cannot separate from base-model inheritance.Source: §3.11.3 · p. 24

3.11.4 Representational similarity and centred kernel alignment

• Representational similarity separates a-vibe pairs from non-a-vibe pairs: RSA is 0.024–0.037 versus 0.630–0.693, and CKA is 0.090–0.107 versus 0.701–0.730.The study cautions that dimensionality reduction and Procrustes alignment can induce artifacts in sample-limited cross-group comparisons; stronger within-family alignment is not supported.Source: §3.11.4 · p. 26, §3.11.3 · p. 26
• Within-model activation patching flips choices at rates from 50% to 100% across models and axes, with T-lite peaking in intermediate layers and the other three models at the embedding layer.The layer profile for T-lite differs from the three models whose flip rates remain uniform across depth.Source: §3.11.4 · p. 26, Figure 9 · p. 27
• Cross-model patching is directionally asymmetric on the sacred–secular axis: T-lite→Qwen3 flips 9/9 choices versus 1/9 for Qwen3→T-lite (p = 4.11 × 10^-4).The corresponding emancipative comparison, 7/8 versus 5/8 flips, is not significant (p = 0.569); item counts are small.Source: §3.11.4 · p. 26, Figure 10 · p. 28

4.1 Principal findings and their scope

• Rank-order concordance exceeds the empirical null across all four perturbations, both axes, and eight specifications, but does not establish magnitude precision or meaningful differences among mid-panel models.The result holds whether refusals are excluded or imputed.Source: §4.1 · p. 29

4.2 Rank stability and level invariance are distinct claims

• Absolute scores shift by more than 20% of the scale while rankings persist; scores therefore require the evaluation’s thematic and modality composition for interpretation.Rank stability is supported, but measurement invariance across conditions is explicitly rejected.Source: §4.2 · pp. 29–30

4.3 Role sensitivity as a property of the model

• Decision role has the strongest influence on expressed value orientation, with role sensitivity varying across models and rank mobility concentrated among mid-tier models.Three architectures have ranges ≤0.12 across both axes, three have ranges ≥0.35, and models show higher emancipative tendencies when evaluating third-party behavior than when acting for a user.Source: §4.3 · p. 30

4.4 Language, temperature and reasoning mode

• Russian administration shifts ten of eleven models toward the emancipative pole by a mean 0.029, while language, temperature, and reasoning mode generally preserve profiles and rankings.Only two of 22 language contrasts reach uncorrected significance, none survive Bonferroni correction, and reasoning-mode choice agreement is 90%–97%.Source: §4.4 · pp. 30–31

4.5 The two axes and the human reference

• Concordance is lower on the sacred–secular axis than the emancipative axis across both decision modality and thematic domain.Decision-modality concordance is 0.510 vs. 0.740; thematic-domain concordance is 0.497 vs. 0.651.Source: §4.5 · p. 31

4.6 What the mechanistic evidence establishes

• In five open-weight models, value labels are linearly decodable, interventions flip choices, and decodable vectors transfer within dimension groups, but the mechanistic evidence does not establish that such signals naturally govern unprompted behavior.The study also reports weak general cross-model directional alignment and does not verify feature specificity; its mechanistic results cannot be generalized directly to the full 20-model panel.Source: §4.6 · p. 31

4.7 Limitations

• The benchmark lacks human baselines and external criteria, uses one prompt template and scoring rule, and leaves full-design variance decomposition and facet-level or ipsative stability unmeasured.Baseline runs at temperature 0 may inherit deterministic-decoding estimation bias; the mechanistic study also uses only five open-weight models.Source: §4.7 · p. 32

4.8 Open questions and further research

• Future work is proposed on variance partitioning, axis-specific stability mechanisms, activation-vector nulls, feature specificity, tokenizer effects, dynamic cultural meaning, and non-Western epistemic frameworks.The conclusion limits interpretation to relative positioning at distribution extremes and role-dependent variance, not minor middle-panel rank differences or scores detached from item composition.Source: §4.8 · pp. 32–33, §4.9 · p. 33

Data availability

• The public repository provides all 224 instrument rows with theme, modality, facet, and axis metadata, both option texts, and Russian translations.It also includes English, Russian, and English temperature-run vignette files; model responses and run-level labels are excluded.Source: Data availability · p. 33
• The released audit script checks the factorial census, polarity partition, option-slot and hedging asymmetries, English–Russian pairing, and near-duplicates.The script runs against the public instrument without credentials.Source: Data availability · p. 33
• The authors state that every structural number in Sections 2.1 and 2.2 can be independently checked from the released materials.Source: Data availability · p. 33

Ethics statement

• No human subjects were recruited, no personal data were processed, and no institutional review was required or obtained.The absence of a human sample is also identified as a scientific limitation.Source: Ethics statement · p. 33
• The measured construct is a two-axis value orientation from a specific cross-national survey tradition, and model response shares are not country factor scores or statements about model-building or training populations.Source: Ethics statement · pp. 33–34
• The panel is weighted toward models from two countries and tested in English and Russian, so the benchmark should not be read as measuring cultural values in general.Refusals are reported but not modeled, and treating them as value signals would conflate value orientation with jurisdiction-specific regulation and safety policy.Source: Ethics statement · p. 34

A Comparative Table of Instruments

• Table A.1 is described as covering instruments for eliciting value or cultural orientation from language models.Source: Table A.1 · p. 38
• The table’s stated comparison includes the reliability evidence offered by each paper.Source: Table A.1 · p. 38
• A continuation of Table A.1 is also supplied.Source: Table A.1 · p. 39

• The stability-analysis fragments mention tie-corrected Kendall’s W and a 1,000-shuffle permutation null with variance decomposition.Source: appendix · p. 39
• The appendix lists 10 prompt-wording variants.Source: appendix · p. 39
• The human reference covers 66,018 respondents across 56 countries.Source: appendix · p. 39

• Table B.1 compares panel-order concordance across perturbation-factor levels under the main specification, with two interval limits that do not contain their point estimates.The specification drops refusals and a duplicate, and scores each level using the mean of per-vignette proportions.Source: Table B.1 · p. 42