PaperlayerView exact revision on arXiv

Source-linked AI summary

CuBEs: Culturally-Situated Behavioral Evaluations and the Limitations of Culture-Blind LLM Judges

Hoda Ayad, Tanu Mitra, Abhishek Mukherji

arXiv:2610.02622v1cs.AI

TL;DR

Situated behavioral evaluations often omit cultural context, despite variation in how behavior is perceived and judged. CuBEs incorporates localized contexts into behavioral evaluation and tests models across cultures. The authors find that cultural framing changes observed behavior and that culture-agnostic baselines miss these shifts.

  • Problem

    Situated behavioral evaluations often omit cultural context, although behavioral perception and standards of acceptability vary across social contexts.

    Source: §1 · p. 1

  • Method

    CuBEs extends an automated behavioral-evaluation pipeline to incorporate localized cultural contexts into behavior formulation, scenario generation, and evaluation.

    Source: §1 · pp. 1–2

  • Results

    Cultural context produces significant variation across six tested behaviors, and culture-agnostic judges miss shifts including religious and colonial dimensions of political bias.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Culture-blind baselines are inadequate predictors of localized behavior patterns, supporting evaluation that accounts for cultural context.

    Source: §6 · p. 8, §5 · p. 8

    The framework uses country as a proxy for culture, so regional baselines may flatten differences across class, gender, religion, and ethnicity.

    Source: §7 · p. 9

Abstract

from arXiv · show

Evaluating the occurrence and triggers of large language model (LLM) behaviors - such as sycophancy, self-preference, or over-confidence - is critical for predicting real-world model deployment risks. However, existing situated behavioral evaluations typically ignore cultural context, limiting their generalizability across an increasingly global user base. To address this gap, we propose CuBEs - Culturally-situated Behavior Evaluations that probe for response patterns across diverse user cultures. We first extend an automated testing pipeline to inject cultural context into behavioral test scenarios and subsequent evaluation. We assess the cultural adaptability of this pipeline by building a human-labeled dataset that captures nuanced dimensions of behavior understanding across 12 distinct cultures. Our dataset reveals significant cross-cultural variations that one-size-fits all judgments fail to capture. Through evaluating 13 open- and closed-source LLMs, we find that introducing cultural situatedness in the evaluation scenario creates significant variation in the presence of a behavior. For example, while our baseline experiments testing for political bias capture localized Western political dimensions like the American conservative-progressive divide, non-Western culturally situated evaluations surface entirely different axes of bias such as religious and colonial political issues. Our findings demonstrate that standard, culturally-agnostic evaluations fail to capture these shifts, highlighting the necessity of culturally situated behavioral testing for global deployments.

1 Introduction

Situated behavioral evaluations often omit cultural context, even though judgments of acceptable behavior vary socially. CuBEs adds localized cultural contexts to behavioral testing, and the authors report that culture-blind judges miss shifts in behavior, including political-bias dimensions.

1 Introduction

  • Cultural context matters because behavioral acceptability is socially constructed, limiting how well culturally agnostic evaluations apply to diverse interactions.The introduction frames behavioral perception as learned and subject to cultural variation.Source: §1 · p. 1
  • CuBEs evaluates LLM behaviors in simulated environments grounded in localized cultural contexts.The overview describes selecting behaviors and cultural or domain contexts, optimizing the pipeline, and then testing whether cultural situation changes behavior presence.Source: §1 · p. 1, Figure 1 · p. 2
  • The study incorporates cultural context into behavior formulation, scenario generation, and evaluation, then benchmarks six behaviors across 13 proprietary and open-weight models.The behaviors include self-preservation, delusional sycophancy, political bias, instructed long-horizon sabotage, self-preferential bias, and implicit stereotyping.Source: §1 · pp. 1–2
  • Cultural context produces significant variation across all six tested behaviors, while culture-blind judges miss shifts such as religious and colonial dimensions of political bias.The introduction argues that omitting cultural context creates a blind spot in behavioral assessment.Source: §1 · p. 2

2 Related Work

Prior behavioral evaluations address simulation quality, harmful behavior categories, and safe outcomes, but frameworks for real-world contexts do not incorporate culture. Cultural-alignment research includes discriminative and generative tests, while this paper takes a situated approach to behavioral interaction.

2.1 Behavioral Evaluation

  • Behavioral-evaluation research improves simulation quality, identifies harmful behaviors, and guides safe outcomes; automated systems use LLMs for experimental design and judging, a strategy this paper adopts.GDPVAL and Bloom contextualize evaluations for real-world use but do not include culture (Patwardhan et al., 2025).Source: §2.1 · pp. 2–3

2.2 Cultural Alignment

  • Cultural-alignment studies use discriminative or generative tests, whereas this paper observes naturalistic interactions to assess how cultural contexts affect harmful behaviors.Prior work finds that textbook cultural understanding can conflict with how a model operates in practice.Source: §2.2 · p. 3

3 Methods

The methods select behaviors and cultural contexts, optimize an agent-based evaluation pipeline, and apply it to cross-cultural testing. The setup spans six behaviors, 12 English-speaking countries, three domains, and model-selection and human-alignment experiments.

3 Methods

  • The methodology proceeds from selecting behavior and cultural-context taxonomies to optimizing an agentic evaluation framework and testing cross-cultural effects across target models.The three stages are described as taxonomy collection, framework adoption and optimization, and evaluation with the optimized framework.Source: §3 · p. 3

3.1.1 Behavioral Taxonomy

  • The six-behavior taxonomy covers self-preservation, delusional sycophancy, long-horizon sabotage, political bias, self-preferential bias, and implicit stereotyping.The behaviors are selected from the Bloom repository and additional literature.Source: §3.1.1 · p. 3

3.1.2 Construction of Culturally Situated Contexts

  • Culturally situated contexts span 12 major English-speaking countries and the Work, Health, and Family domains.The study focuses on contextual nuances such as cultural institutions, experiences, and perspectives.Source: §3.1.2 · p. 3

3.2.1 Experimental Setup

  • The team optimizes scenario generation, user simulation, and judging, including human-annotator alignment across cultures.Six models are compared for scenario generation using pairwise human preferences; simulator candidates are compared on behavior-presence scores. For judge alignment, 60 annotators across 12 cultures label 31 vignettes each, producing 1,860 labels used to compare four candidate judges with human-majority labels.Source: §3.2.1 · p. 4

3.3 Cross-Cultural Evaluation

  • The cross-cultural evaluation uses Gemini 3 Flash to judge behavior presence on a scale from 1 (no presence) to 10 (high presence).The setup applies the established pipeline to parameters listed in Table 1.Source: §3.3 · p. 4

4 Results

CuBE pipeline optimization selected models for scenario generation, user simulation, and judging, while human labels varied across cultures. In the full evaluation, cultural framing changed behavior-presence scores, and no-culture baselines sometimes substantially diverged from cultural means.

4.1 CuBE Pipeline Optimization

  • Claude Sonnet 4.5 led scenario generation, winning 80.0% of pairwise rankings (95% CI [65.2, 89.5]) among six tested models.Three human judges ranked scenarios for incorporating context and probing the target behavior.Source: §4.1 · p. 5
  • Gemini 3 Flash achieved a 4.09 average behavior-presence score versus Opus 4.5’s 4.32, about 95% of the standard at a fraction of the cost.It was selected as a cost-effective user simulation model for fiscal scalability.Source: §4.1 · p. 5
  • Human labels varied across cultures: only one of 11 alignment questions reached unanimity, while four elicited opposing labels across the 12 cultures.Eight questions elicited all three labels; stereotypes drew stricter judgments in Pakistan, the Philippines, and Nigeria, while delusional sycophancy drew lenient judgments across cultures.Source: §4.1 · p. 5, Figure 3 · p. 6
  • Gemini 3 Flash Preview had the highest accuracy (67.7%) and ordinal κw (0.631) among the three judge candidates considered after Opus 4.5’s costly overall-agreement reference result.Opus 4.5 had the highest overall agreement with human-majority labels (κw = 0.633) but was treated as an upper bound because of inference cost.Source: §4.1 · pp. 5–6

4.2 Cross-Cultural Evaluation

  • Across cultural framings, the most reactive models’ average behavior-presence scores shifted by more than four points: kimi-k2 (4.56), gemini-2.5-pro (4.22), grok-4.3 (4.11), and deepseek-r1 (4.06).Source: §4.2 · p. 6
  • Claude Sonnet 4.5 political-bias presence averaged 8.78 in the no-culture baseline versus 2.72 across 12 cultures, a 6.06-point gap.The baseline assessment shifted from “strongly present” to “absent” under cultural situatedness.Source: §4.2 · p. 6, Table 2 · p. 7
  • Anthropic models had low average presence for delusional sycophancy (1.01), instructed sabotage (1.09), and self-preservation (1.89); OpenAI’s gpt-5 subfamily ranked second except on self-preferential bias (6.11).The Anthropic family showed small cross-cultural ranges overall; the older gpt-4o had wider ranges than later generations on nearly every behavior.Source: §4.2 · pp. 6–7

5 Discussion

Culturally agnostic baselines can misrepresent localized model behavior, while automated judges still flatten cultural variation. The discussion calls for pluralistic evaluations and better alignment with situated human judgment.

Baseline evaluations miss nuances introduced by the CuBE approach

  • A 6.06-point political-bias gap for claude-sonnet-4.5 shifted its assessment from “strongly present” at baseline to “absent” under cultural situatedness.The discussion argues that baseline evaluations are inadequate predictors of localized behavior and calls for re-examining context-free evaluations.Source: §5 · p. 7

Variations across model families

  • Wide IQRs across models indicate that cross-cultural variation is not driven by a single outlier culture, while model families cluster in similar ranges.Figure 5 shows mean behavior presence across 13 cultural conditions, with behavior-specific facets.Source: Figure 5 · p. 8

Automated Evaluation Frameworks in Contextualization

  • Frontier judges achieve fair aggregate agreement with human majorities but tend to default to a culturally agnostic modal perspective, flattening label variance.Their responses under-shift relative to localized human annotators.Source: §5 · pp. 7–8
  • Future work should investigate targeted post-training, fine-tuning, or advanced prompting to better align automated evaluators with situated human judgment.The discussion presents this alignment as a next step for more reliable cross-cultural evaluations.Source: §5 · p. 8

Baseline evaluations miss nuances introduced by the CuBE approach

  • The authors call for moving behavioral evaluation from implicit, context-free scenarios and monolithic baselines toward pluralistic, culturally situated approaches.They argue that context-free evaluations cannot be relied on to predict model responses in the wild.Source: §5 · p. 8

6 Conclusion

The conclusion presents CuBEs as a way to examine how cultural contexts influence AI behavior across models and behaviors. It reports that cultural context reveals regional variation that standard baselines do not predict, supporting a shift toward culturally situated evaluations.

6 Conclusion

  • Across six behaviors, 13 language models, and 12 cultural contexts, the study finds that culturally agnostic baselines are inadequate predictors of localized behavior patterns.The conclusion also reports that frontier LLM judges flatten cross-cultural variance.Source: §6 · p. 8
  • The study introduces CuBEs, an approach designed to surface how cultural contexts influence AI behavior.Its evaluations use three domains and an automated simulation and evaluation pipeline.Source: §6 · p. 8
  • The authors conclude that culturally situated approaches should extend beyond cultural-alignment-specific evaluations to all settings.The conclusion frames this as a move away from monolithic evaluations.Source: §6 · pp. 8–9

7 Limitations

The limitations concern language, how culture is represented, and the range of behaviors studied. Together, they bound the study’s cultural scope and leave multilingual, within-region, and benign-behavior variation for future work.

Situatedness Through Language

  • CuBEs were not tested in a non-English environment, leaving multilingual evaluations as future work.The authors identify multilingual evaluation as a way to more closely approximate global interactions.Source: §7 · p. 9

Monolithic Cultural Representations

  • Country is used as a proxy for culture, but aggregating diverse voices into a regional baseline risks flattening within-country nuance.The authors describe the paradigm as mapping broader shifts, not defining a nation's internal cultural consensus.Source: §7 · p. 9

Behaviors Beyond Harm

  • The study focuses on behaviors with traditionally negative or risky implications, leaving culturally situated variation in benign behaviors unexamined.Examples named include politeness, deescalation, and empathy.Source: §7 · p. 9

8 Ethical Considerations

The appendix documents ethical safeguards, defines the evaluated behaviors, and describes scenario-generation procedures. The authors frame the cultural analysis as exploratory rather than as a definitive account of any culture.

8 Ethical Considerations

  • The findings are exploratory, not a definitive representation of any single culture, and CuBEs is intended as a diagnostic tool rather than a blueprint for localized echo chambers.The authors also note risks of flattening internal diversity and of superficial demographic optimization.Source: §8 · p. 9

Appendix

  • The appendix outlines behavior definitions, optimization experiments, experimental setup, ethical review, and AI-use disclosures.It organizes formal definitions in §A, optimization and annotation details in §B, experimental setup and ethics in §C, and AI-use disclosure in §D.Source: Appendix · p. 11

A.1 Behavior Definitions

  • The six behaviors span culturally salient social phenomena and less intuitively culture-variable interactions; the authors add implicit stereotyping to examine a harm axis untested on Claude models.Five definitions build on Bloom, while implicit stereotyping is introduced alongside political bias and delusional sycophancy as socially situated behavior.Source: §A.1 · pp. 11–12

B Optimization Experiments

  • The optimization appendix reports design considerations and human-annotation logistics, with full prompt texts made available in a linked repository.The prompts are long, so the authors direct readers to the repository for the relevant texts.Source: Appendix · p. 11, B Optimization Experiments · p. 12

B.1 Scenario Generation Details

  • Scenario generation adds cultural context to prompts and creates strategic multi-turn interactions; three in-house annotators judge scenario pairs for behavioral integrity and natural cultural integration.The process produced 48 unique scenarios paired into 120 comparisons; disagreements between two annotators were resolved by a reviewer.Source: §B.1 · p. 12

B.2 User Simulation Details

The user-simulation study selected Gemini 3 Flash Preview for scale-out despite Claude Opus 4.5's higher behavioral-presence score. The judge-alignment appendix describes culturally tuned questions and annotation across 12 cultures.

B.2 User Simulation Details

  • Seven candidate simulators engaged fixed target models in three-turn conversations across five behaviors, with Gemini 3 Flash judging average behavioral presence.Candidates were assessed on their ability to coax out target behaviors using controlled scenarios.Source: §B.2 · pp. 12–13
  • 4.32 was Claude Opus 4.5’s average behavioral presence score; Gemini 3 Flash Preview scored similarly at the lowest cost tier and was selected for scale-out.Figure 7 reports average behavior-presence scores by behavior across user-simulation models.Source: §B.2 · p. 13, Figure 7 · p. 13

B.3 Judge Alignment Details

  • The judge-alignment test constructed 11 questions targeting subjective nuances within each behavior.The questions probe culturally relevant judgments, including whether practical help validates a potentially delusional belief.Source: §B.3 · p. 13
  • Crowdsourced annotators affiliated with 12 cultures completed 31 culturally tuned tasks each, for 372 unique tasks total.At least five annotators were recruited for each culture, and annotations were gathered through an online platform.Source: §B.3 · pp. 13–14
  • Human labels captured judgments of behavior in scenario conversations, with within-culture inter-rater agreement reported separately.The annotation interface is shown in Figure 8; the passage points to Table 6 for agreement results.Source: §B.3 · p. 14, Table 6 · p. 14

C.1 Experimental Costs

The appendix identifies experimental cost reporting, model configurations, data-use limits, ethical review, and AI assistance. Only a representative sample of the proprietary dataset is publicly available.

C.1 Experimental Costs

  • The authors direct readers to Table 7 for total experiment costs by phase, without stating cost values in this passage.The table caption identifies it as the total cost of experiments.Source: §C.1 · p. 14, Table 7 · p. 14

C.2 Model Details

  • Experiments used OpenRouter models at temperature 1.0, with low reasoning effort for evaluator models and medium effort for target models.Table 8 lists model identifiers for each experimental stage and marks selected candidate-stage models in italics.Source: §C.2 · p. 14, Table 5 · p. 15, Table 8 · p. 16

C.3 Data and Framework Use

  • Only a partial sample of the underlying proprietary dataset is made publicly available; the code and representative data sample are intended for research, auditing, and cross-cultural evaluation.Source: §C.3 · p. 16

C.4 Data Privacy and Ethical Review

  • An independent institutional ethics committee reviewed and approved the data-collection and crowdsourcing protocols.Participants received legal consent materials and role-specific data-use disclosures before beginning work.Source: §C.4 · p. 16

D AI Use Disclosure

  • AI assistants supported manuscript formatting, helper-script and repository work, visualizations, scenario and vignette generation, and transcript evaluation; the authors state that the research conclusions remain their own.Source: D AI Use Disclosure · p. 16

E Full Baseline-vs-Cultural Results

Table 9 lists baseline-versus-cultural gaps for 78 model–behavior pairs, sorted by absolute difference; cell means and confidence intervals for Figure 4 are released with the code.

E Full Baseline-vs-Cultural Results

  • Table 9 covers baseline-versus-cultural gaps for 78 model–behavior pairs, sorted by absolute difference, while Figure 4 cell means and 95% confidence intervals are released with the code.The table caption does not provide the underlying cell values or identify which comparisons have the largest gaps.Source: Table 9 · p. 17
Loading 2610.02622v1…