PaperlayerView exact revision on arXiv

Source-linked AI summary

CRISP: Cultural Reward Modeling for Implicit Situated Propriety

Zekun Yuan, Yangfan Ye, Baohang Li, Shuaibo Zhao, Zekun Zhou, Ziming Li, Qichen Hong, Kun Chen, Xiaocheng Feng

arXiv:2609.34345v1cs.CL

TL;DR

Open-ended culturally situated behavior remains underexplored, even as LLMs increasingly need to respond appropriately across cultural contexts. The paper introduces CRISP-RM and Norm Grounding Supervision, reporting gains in reward-model selection and policy optimization.

  • Problem

    Research has focused largely on cultural knowledge or predefined-response tasks, while open-ended culturally situated behavior and reward modeling remain comparatively underexplored.

    Source: §1 · p. 1

  • Method

    The paper combines CRISP-RM, a culturally situated reward model, with a collaborative multi-agent framework for NormCompass data construction and optional Norm Grounding Supervision during policy optimization.

    Source: §1 · pp. 1–2

  • Results

    CRISP-RM achieves the strongest Best-of-N selection performance on both benchmarks and consistently improves culturally situated behavior across policy models; adding NGS can yield further gains.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CRISP-RM distinguishes responses that are similarly fluent and polite but differ in cultural appropriateness.

    Source: §9 · p. 9

    NormCompass norms and scenarios are contextual references, not universal prescriptions; its data may simplify heterogeneous practices or reflect biases in sources and language models.

    Source: §10 · pp. 9–10

Abstract

from arXiv · show

As large language models (LLMs) are increasingly deployed across countries and regions, the ability to recognize and respond appropriately to diverse cultural contexts becomes increasingly important. However, existing research has largely focused on cultural knowledge or tasks with predefined response spaces, while open-ended culturally situated behavior remains comparatively underexplored. In this work, we introduce CRISP-RM, a culturally situated reward model that assigns rewards according to cultural appropriateness in open-ended social scenarios. During policy optimization, we further introduce Norm Grounding Supervision (NGS), providing guidance that enhances the policy's sensitivity to relevant cultural norms. To construct culturally situated data, we employ a collaborative multi-agent framework that instantiates implicit cultural norms into diverse social scenarios and further curate NormCompass as a dedicated testbed. We conduct comprehensive experiments to evaluate the effectiveness of CRISP-RM in both reward modeling and policy optimization. Best-of-\(N\) experiments show that CRISP-RM consistently outperforms strong general reward models. During GRPO policy optimization, CRISP-RM generally improves culturally situated behavior, while incorporating NGS yields further gains. Further analyses demonstrate the advantages of CRISP-RM in distinguishing culturally appropriate behavior beyond superficial fluency and politeness, while NGS provides complementary gains during policy optimization by improving norm grounding.

1 INTRODUCTION

Open-ended culturally situated behavior requires judging responses in context, but reward modeling for that setting remains underexplored. CRISP-RM supplies cultural-appropriateness rewards, with Norm Grounding Supervision (NGS) adding explicit norm guidance during policy optimization.

1 INTRODUCTION

  • Prior work emphasizes cultural knowledge or predefined-response tasks; reward modeling for open-ended culturally situated behavior remains insufficiently explored.Situated competence requires recognizing relevant considerations in specific situations and responding appropriately.Source: §1 · p. 1
  • CRISP-RM is a culturally situated reward model that scores open-ended behavior, while NGS adds explicit norm supervision during policy optimization.The paper also introduces a multi-agent framework for constructing situated data and curates NormCompass as a testbed.Source: §1 · pp. 1–2
  • CRISP-RM achieves the strongest Best-of-N selection performance on both benchmarks, outperforming several substantially larger reward models.The introduction describes the model as providing culturally informed supervision for policy optimization.Source: §1 · p. 2
  • Across multiple policy models, CRISP-RM consistently improves culturally situated behavior during GRPO optimization, and adding NGS can yield further gains.The joint setup combines behavior-level cultural rewards with explicit supervision for norm grounding.Source: §1 · p. 2
  • CRISP-RM distinguishes culturally appropriate behavior from alternatives that are superficially fluent and polite, while NGS generally adds norm-grounding gains and helps preserve culturally grounded behavior.The analyses attribute NGS's complementary contribution to explicit supervision.Source: §1 · p. 2
  • The data framework instantiates implicit cultural norms in diverse social scenarios, and NormCompass evaluates culturally appropriate behavior in open-ended scenarios.The corpus spans 19 cultures and varied temporal, spatial, and interpersonal contexts.Source: §1 · p. 2

2 RELATED WORK

Related work covers general reward modeling and increasingly situated cultural evaluation, but open-ended culturally situated behavior remains a comparatively limited focus. This paper addresses that gap with a cultural reward model and policy optimization.

Reward Models for LLM Alignment

  • General reward models learn scalar preferences from pairwise comparisons, but strong general performance need not transfer across languages, domains, or culturally dependent preferences.Prior approaches include Think-as-Locals for cultural judgments and SCPO for balancing preferences across cultural subcommunities.Source: §2 · p. 2

Cultural Awareness in Large Language Models

  • The paper focuses on reward modeling for open-ended behavior where cultural norms are implicit in the scenario, then uses those rewards to optimize policy.The overview describes culturally situated data construction and combines CRISP-RM and Norm Grounding Supervisor signals for GRPO.Source: §2 · pp. 2–3, Figure 1 · p. 3
  • Cultural-capability research has progressed from knowledge, translation, values, and norms toward judgments or actions in concrete scenarios and open-ended responses.The cited literature includes Ramezani & Xu (2023), Shen et al. (2024), Chiu et al. (2025), Rao et al. (2025), Kim & Lee (2025), Wu et al. (2025), and Ye et al. (2026b).Source: §2 · pp. 2–3

3 CONSTRUCTION OF CULTURALLY SITUATED DATA

The data pipeline converts cultural norms into diverse implicit social scenarios, refines them through iterative quality control, and curates NormCompass for evaluating culturally situated decisions.

3.1 SCENARIO AND QUESTION GENERATION

  • Cultural norms are instantiated into concrete scenarios and questions using up to three chronotopes per norm, varying temporal, spatial, and social configurations.The question asks for a contextually appropriate choice, judgment, or action rather than recall or explanation of cultural knowledge.Source: §3.1 · p. 3

3.2 MULTI-AGENT REFINEMENT AND QUALITY CONTROL

  • A five-agent refinement loop diagnoses and revises candidates, then verifies them against appropriateness, contextual evidence, norm matching, reasoning support, and norm leakage.The Situated Reasoner works without access to the target norm; response shortcomings can prompt targeted regeneration, whereas norm leakage, missed norms, or inappropriate actions route to question revision.Source: §3.2 · p. 4

3.3 NORMCOMPASS

  • The refined dataset contains 4,192 questions across 19 cultures, and NormCompass contains 222 evaluation items for culturally situated decisions.The automatic evaluator showed strong consistency with human judgments, supporting its use for large-scale NormCompass evaluation.Source: §3.3 · pp. 4–5

4 CRISP REWARD MODEL

CRISP reward models are trained on within-scenario cultural preference pairs and evaluated through Best-of-N selection. They outperform general reward-model baselines on both benchmarks, including larger models.

Reward Model Training

  • Training responses are scored by an automatic evaluator, converted into within-scenario pairwise preferences, and used to train a scalar cultural-appropriateness reward model.Higher-scored responses are preferred, while ties are excluded; the model is trained with a Bradley–Terry objective.Source: §4 · p. 5

Best-of-N Evaluation Setup

  • Best-of-N selection scores multiple responses to the same scenario with a reward model and retains the top-ranked response for evaluation.CultureForest evaluation uses its Hard open-ended generation setting and includes cultural groups represented in the training data.Source: §4 · p. 5

Best-of-N Evaluation Results

  • CRISP reward models outperform random selection and all general reward-model baselines on both NormCompass and CultureForest.CRISP-RM-4B scores 3.89 and 3.92 on NormCompass at N = 8 and N = 16, respectively; both CRISP models retain their lead on CultureForest over substantially larger general reward models.Source: §4 · p. 5
  • Culture-specific preference supervision yields more consistent cultural-behavior reward signals than strong general reward modeling, despite substantially smaller model size.The reported Best-of-N advantage extends beyond the constructed testbed to CultureForest.Source: §4 · p. 5

5 REINFORCEMENT LEARNING WITH CULTURAL REWARDS

CRISP-RM is used as a reward signal for GRPO policy optimization, with evaluation on NormCompass, CultureForest, and CulShield Knowledge Coverage. The reported results show consistent gains over baselines and further gains from adding Norm Grounding Supervision (NGS).

5 REINFORCEMENT LEARNING WITH CULTURAL REWARDS

  • CRISP-RM scores are normalized using validation-set robust statistics and mapped to [0, 1] with a sigmoid before policy optimization.The stated aim is to reduce sensitivity to extreme scores; normalized scores compute group-relative advantages in the standard GRPO objective.Source: §5 · p. 6
  • A Norm Grounding Supervisor supplies an auxiliary grounding reward that is combined with the normalized cultural reward, with λ controlling its contribution.The supervisor formulates grounding as a binary classification problem using Grounded and Ungrounded states.Source: §5 · pp. 6–7
  • Policy optimization is evaluated on NormCompass and CultureForest, with CultureForest restricted to represented cultural groups and CulShield Knowledge Coverage serving as a multiple-choice complement.The comparisons include Skywork, CuSiR, and the effect of NGS.Source: §5 · p. 7
  • CRISP-RM yields consistent gains on NormCompass, transfers to CultureForest and CulShield, and NGS brings additional gains.The reported Qwen3-8B drop with NGS on CultureForest is mainly attributed to invalid output formatting.Source: Table 5 · p. 6, §5 · p. 7
  • On Qwen3-4B, CRISP maintains stronger performance through training, especially on CultureForest, where competing rewards degrade while CRISP remains stable or improves.Figure 2 tracks performance across training checkpoints on NormCompass and CultureForest.Source: §5 · p. 7, Figure 2 · p. 7

6 REWARD MODEL DISCRIMINATION ANALYSIS

A controlled triplet analysis tests whether CRISP-RM distinguishes culturally appropriate behavior from plausible but generic or norm-opposing alternatives. The analysis and case study indicate that its preferences extend beyond fluency and politeness.

6 REWARD MODEL DISCRIMINATION ANALYSIS

  • The triplet analysis contrasts an Appropriate response, a plausible Generic response that omits the decisive action, and an Opposite response that conflicts with the norm.The controlled responses are designed to remain comparable in fluency, style, and length.Source: §6 · p. 7
  • CRISP-RM maintains a finer Appropriate-versus-Opposite distinction than general reward models, which are less reliable on that contrast than Appropriate-versus-Generic.The comparisons use responses designed to be similar in fluency, style, and length, focusing on cultural appropriateness.Source: §6 · pp. 7–8
  • For a German workplace farewell, CRISP-RM favors opening a gift in front of the group, while general reward models favor the polite response that postpones opening it.The latter response is socially plausible, but its core action conflicts with the stated cultural norm.Source: §6 · p. 8, Figure 4 · p. 8

7 EFFECTIVENESS OF THE NORM GROUNDING

The analysis measures norm grounding across policy variants and examines answer quality by the original policy’s grounding level. CRISP-RM improves norm grounding, while NGS generally adds gains and helps mitigate a decline for initially grounded cases.

7 EFFECTIVENESS OF THE NORM GROUNDING

  • Norm Match is assigned to generated rationales on a three-level scale: 0 means the norm is not identified, 1 partial or implicit grounding, and 2 clear and accurate grounding.GPT-5.6 Sol evaluates the rationales.Source: §7 · p. 8
  • Scenarios are grouped by the original policy’s Norm Match score to compare answer quality after CRISP-RM optimization alone or with NGS.Figure 5 also compares Norm Match scores across policy variants before and after optimization.Source: Figure 5 · p. 8, §7 · p. 9
  • CRISP-RM improves Norm Match across policy models, and adding NGS further increases it; NGS generally improves answer quality and helps mitigate the decline at N = 2.For initially ungrounded cases, CRISP-RM substantially improves answer quality; performance decreases for N = 2.Source: Figure 5 · p. 8, §7 · p. 9

8 ROBUSTNESS ANALYSIS OF NORM GROUNDING SUPERVISION

The sensitivity analysis varies λ while holding the remaining training configuration fixed and finds stable performance overall, with λ = 0.1 and λ = 0.2 near the best results in most settings.

8 ROBUSTNESS ANALYSIS OF NORM GROUNDING SUPERVISION

  • λ = 0.1 and λ = 0.2 achieve results close to the best performance in most settings, while performance shows only limited fluctuations as λ varies.The analysis varies λ with the rest of the training configuration unchanged; Figure 6 presents results across policy models.Source: §8 · p. 9

9 CONCLUSION

The paper presents CRISP-RM and NormCompass for open-ended culturally situated behavior, and reports improved behavioral appropriateness with additional gains from Norm Grounding Supervision.

9 CONCLUSION

  • CRISP-RM improves culturally appropriate behavior and transfers across task formats, while providing a more stable policy-optimization reward signal.Norm Grounding Supervision further strengthens norm grounding and brings additional gains.Source: §9 · p. 9
  • CRISP-RM distinguishes responses that are similarly fluent and polite but differ in cultural appropriateness.Source: §9 · p. 9
  • The work combines a collaborative multi-agent data-construction framework, NormCompass, CRISP-RM, and Norm Grounding Supervision for open-ended cultural scenarios.Source: §9 · p. 9

10 ETHICS STATEMENT

The authors frame cultural norms as context-dependent and caution that the data may simplify cultural practices or reflect source and model biases.

10 ETHICS STATEMENT

  • Norms can vary across communities, contexts, and individuals, so NormCompass norms are contextual references rather than universal prescriptions.Source: §10 · p. 9
  • The construction process aims to preserve contextual specificity, but the resulting data may still simplify heterogeneous cultural practices.Source: §10 · p. 9
  • The data may reflect biases present in underlying sources and language models.Source: §10 · pp. 9–10

11 AI USE STATEMENT

The appendices detail AI-assisted research, construction and quality control of culturally situated scenarios, evaluation procedures, and reward-model training. The resulting dataset covers 19 cultural groups, and the reward models are trained on balanced preference pairs.

11 AI USE STATEMENT

  • The authors used DeepSeek and GPT models for data construction, selected analysis data, automated response evaluation, and manuscript language polishing; authors reviewed AI-assisted content and retained responsibility.Source: §11 · p. 10

A.1 CULTURAL NORM COLLECTION AND PROCESSING

  • CulturalBench and FORK norms were extracted with DeepSeek-V4-Flash, while NormAd's explicit norms were adopted directly, yielding 1,629 deduplicated norms.Source: §A.1 · p. 13

A.2 CHRONOTOPE CONSTRUCTION; A.3 SCENARIO AND QUESTION GENERATION

  • Chronotopes encode temporal, spatial, social, and activity context; up to three distinct configurations per norm guide generation of scenarios where the norm remains implicit.Source: §A.2 · p. 13, §A.3 · p. 13

A.4 MULTI-AGENT REFINEMENT

  • A multi-agent loop checks action appropriateness, contextual evidence, norm matching, reasoning, and norm leakage, revising or discarding candidates that fail verification.Candidates may undergo up to five rounds; retained samples must pass all criteria.Source: §A.4 · pp. 13–14

A.5 DATASET STATISTICS AND SPLITTING

  • The final dataset contains 4,430 questions across 19 cultural groups, with 222 questions in the evaluation test set.The test set is restricted to norms originating from NormAd; final experimental splits contain 1,528 norms and 4,192 scenarios.Source: §A.5 · pp. 14, 16

A.6 AUTOMATIC EVALUATION PROTOCOL; A.7 HUMAN EVALUATION

  • Automatic evaluation scores cultural appropriateness from 1 to 5, and two annotators independently applied the same scoring criteria to 100 responses.Source: §A.6 · p. 17, §A.7 · p. 18

B.1 REWARD MODEL TRAINING DATA CONSTRUCTION

  • Training candidates combine natural responses from ten models, controlled generation, and targeted augmentation of underrepresented score-2 and score-4 responses.Preference pairs compare different-scoring responses to the same question, with at most eight pairs retained per question.Source: §B.1 · pp. 18–20

B.2 REWARD MODEL ARCHITECTURE AND TRAINING DETAILS

  • Qwen3-0.6B and Qwen3-4B reward models use a fine-tuned Transformer backbone and scalar reward head, trained with the Bradley–Terry objective for two epochs.The final checkpoint is taken after the second epoch.Source: §B.2 · p. 20

B.3 BEST-OF-N EVALUATION DETAILS

The appendix reports generator-robust Best-of-N selection, details policy and supervisor training and evaluation, and describes controlled preference and norm-grounding analyses.

B.3 BEST-OF-N EVALUATION DETAILS

  • CRISP reward models maintain a Best-of-N advantage across candidate pools from different generators, rather than relying on one generator's response distribution.Each generator supplies 16 candidates, and selection is evaluated at N = 8 and N = 16.Source: §B.3 · p. 21, Table 22 · p. 23, Table 23 · p. 23

C.1 NORM GROUNDING SUPERVISOR TRAINING.

  • Norm Grounding Supervision achieves 88.31% macro-F1 on the independent test set.The supervisor is an open-book binary classifier trained to distinguish successful norm grounding from failures to recover the relevant norm.Source: §C.1 · pp. 21–22

C.2 TRAINING CONFIGURATION

  • GRPO optimization uses CRISP-RM rewards across Qwen3-4B, Qwen3-8B, and DeepSeek-R1-Distill-Qwen-7B, with eight sampled responses per prompt.The reward combines calibrated CRISP-RM scores with the predicted grounded-class probability from NGS.Source: §C.2 · pp. 22–23

C.3 BASELINES AND EVALUATION; C.4 VALID RESPONSE EVALUATION

  • Policy reward comparisons include Skywork-Reward and CuSiR, with core GRPO budgets aligned across reward models.Benchmark evaluation covers NormCompass, CultureForest, and CulShield; invalid outputs follow each benchmark's protocol and affect reported end-to-end performance.Source: §C.3 · pp. 23–24, §C.4 · p. 24

D CONTROLLED CULTURAL PREFERENCE ANALYSIS

  • The controlled preference analysis constructs Appropriate, Generic, and Opposite responses that preserve surface qualities while varying culturally decisive behavior.The 71 retained triplets were independently checked for the ordering sA > sG > sO.Source: D CONTROLLED CULTURAL PREFERENCE… · pp. 24–25

E NORM GROUNDING ANALYSIS

  • Norm grounding is assessed over 222 NormCompass scenarios by scoring whether the rationale identifies the relevant norm; scenarios are stratified by the original policy's grounding score.The same scenario groups are retained across Base, CRISP, and CRISP+NGS comparisons.Source: E NORM GROUNDING ANALYSIS · p. 25
  • Valid-response-only analyses preserve the main norm-grounding trends, indicating that CRISP-RM and NGS effects are not primarily driven by invalid-generation differences.Source: E NORM GROUNDING ANALYSIS · p. 26
Loading 2609.34345v1…