PaperlayerView exact revision on arXiv

Source-linked AI summary

CC-Mediation: Evaluating Large Language Models for Cross-Cultural Conflict Mediation

Suhyun Lee, Wenxuan Zhang, W. Quin Yow, Yang Deng

arXiv:2609.04855v1cs.CLcs.AI

TL;DR

Cross-cultural LLM mediation lacks benchmarks with measurable downstream effects and principled metrics for intercultural stance change. This paper introduces the DMIS-grounded CC-MEDIATION benchmark and two trajectory-level metrics, finding distinct limitations in when LLMs intervene and how they mediate.

  • Problem

    Existing resources lack mediation datasets with measurable downstream effects and principled metrics for intercultural stance change.

  • Method

    The paper constructs CC-MEDIATION, a DMIS-grounded corpus of 1,661 dialogues, and proposes Trajectory AUC plus signed Wasserstein-1 distance.

  • Results

    Current LLMs show pronounced limitations in both timing and content generation: below 33% turn accuracy and AUC at or below 1.3.

  • Takeaways & Limitations

    The metrics track substantive downstream stance shifts across the post-intervention trajectory rather than only surface-form changes.

  • Takeaways & Limitations

    The benchmark remains limited by synthetic LLM-generated dialogues whose external validity to human-authored interactions is an open question.

Abstract

from arXiv · show

Cross-cultural mediation by large language models (LLMs) requires deciding both when to intervene and how to respond in culturally grounded conflicts. Progress on this problem has been limited by the lack of (1) mediation datasets with measurable downstream effects and (2) principled metrics for evaluating intercultural stance change. To address these gaps, we introduce CC-Mediation, a cross-cultural mediation benchmark of $1{,}661$ ten-turn dialogues grounded in the Developmental Model of Intercultural Sensitivity (DMIS), containing culturally grounded conflicts, mediation interventions, and post-intervention trajectories. We further propose two DMIS-based evaluation metrics: Trajectory AUC, which measures the persistence of intercultural improvement over time, and a signed Wasserstein-1 distance, which measures the magnitude and direction of shifts in intercultural stance. Both metrics show strong agreement with human judgment of DMIS-grounded stance shift. Using CC-Mediation, we find that current LLMs have limitations on both axes: intervention timing (when) failure stems from a positional prior that ignores dialogue content, while mediation strategy (how) failure arises from a late-layer elicitation collapse rather than a knowledge deficit.

1 Introduction

Cross-cultural mediation requires theory-grounded interventions and evaluation of downstream intercultural stance change, gaps existing moderation resources do not address. CC-MEDIATION supplies a DMIS-grounded benchmark and reveals distinct LLM failures in intervention timing and mediation strategy.

  • Motivation: Existing datasets label toxicity or norm violations rather than culturally grounded mediation trajectories with measurable downstream stance effects.The gap concerns both benchmark coverage and principled measurement of intercultural stance change.
  • Framework: DMIS frames intercultural sensitivity as six ordinal stages from ethnocentric Denial, Defense, and Minimization to ethnorelative Acceptance, Adaptation, and Integration.The proposed framework uses this progression to ground interventions and measurements.
  • Findings: Current LLMs fail on both axes: TurnAcc remains below 33% for conflict-turn detection, while trajectory AUC is at or below 1.3 for mediation generation.The paper attributes timing failure to a positional prior and mediation failure to late-layer elicitation failure rather than knowledge deficit.
  • Findings: Fine-tuning on CC-MEDIATION lifts TurnAcc above 91% across all three open-source backbones.This result concerns conflict-turn detection under the benchmark.
  • Contributions: CC-MEDIATION contributes a cross-cultural mediation corpus and two DMIS-based metrics: Trajectory AUC for persistence and signed Wasserstein-1 distance for shift magnitude and direction.The corpus and metrics are designed to quantify stance changes induced by mediation interventions.

2 Related Work

Prior resources separately cover culturally sensitive dialogue or mediator responses, but do not jointly represent culturally grounded conflicts, interventions, and measurable downstream effects. CC-MEDIATION is positioned as the first dataset to combine these requirements with multi-turn stance-shift measurement.

  • Dataset gap: Existing cross-cultural dialogue and value-aware datasets describe culturally sensitive scenarios but contain no mediator responses.They therefore do not provide the intervention component required for cross-cultural mediation evaluation.
  • Dataset gap: Hate-speech intervention, prosocial response, and norm-remediation corpora provide mediator utterances for non-cultural conflicts without evaluating long-term recipient impact.Their limitation is the absence of measurable downstream effects in culturally grounded interactions.
  • CC-MEDIATION: CC-MEDIATION jointly models culturally grounded mediation dialogues and measurable intercultural stance shifts across cross-cultural speakers, cultural-value conflicts, and multi-turn dialogue.The paper presents this combination as absent from existing datasets.
  • Evaluation metrics: Prior mediator evaluation uses either utterance-level scoring or downstream measures such as stance shift, emotional trajectory, or task success.The related-work distinction separates surface or rubric-based evaluation from recipient-outcome evaluation.

3 Cross-Cultural Mediation

The paper defines mediation as a turn-level effort to move a speaker toward greater ethnorelativism during culturally grounded conflict. It constructs DMIS-labeled dialogues and evaluates interventions through trajectory-level metrics that capture both sustained progress and distributional movement.

  • Task formulation: Cross-cultural mediation targets a speaker’s shift toward greater ethnorelativism during culturally grounded conflict, with stage inferred from observable dialogue cues.The formulation treats intercultural stage as a momentary, turn-level expression rather than a fixed speaker identity.
  • DMIS framework: DMIS orders six stages from ethnocentric Denial, Defense, and Minimization to ethnorelative Acceptance, Adaptation, and Integration.Conflict speakers are restricted to the three ethnocentric stages in the described setup.
  • Evaluation metrics: An LLM judge predicts a probability distribution over six DMIS stages at each turn, from which the argmax stage supports mediation-effectiveness measurements.The stage distributions provide the input for both complementary metrics.
  • Trajectory AUC: Trajectory AUC integrates per-turn progress relative to the pre-intervention stage, with positive values indicating sustained advancement and negative values indicating regression.Unlike final-state comparison, it captures the full trajectory and distinguishes transient improvement from sustained progress.
  • Dataset construction: The construction pipeline generates culturally grounded dialogue, assigns turn-level DMIS labels, inserts mediation at the conflict turn, and filters examples using positive AUC and W1 utility gaps.The benchmark uses LLM-driven stages and human validation, with scenarios based on paired country profiles and opposed value dimensions.
  • Signed Wasserstein-1: Signed Wasserstein-1 distance measures distributional change on the ordinal DMIS scale, preserving stage geometry while indicating movement toward or away from ethnorelativism.It complements AUC by capturing sub-argmax distributional shifts and ranges from −5 to 5.
  • Dataset statistics: Retained mediations exceed unguided baselines by +0.701 in AUC and +0.507 in W1, while Acceptance comprises 56.6% of post-conflict turns.The training mediations pass the utility-based filtering criterion, whereas evaluation mediations are used unfiltered.

4 Experiments

Experiments evaluate when LLMs should intervene and how they should mediate cross-cultural conflicts. Results identify positional bias in timing, late-layer elicitation collapse in mediation, and substantial timing gains from supervised fine-tuning.

  • 4.1 Cross-culture Conflict Detection (RQ1): Turn-detection accuracy ranges from 11.3% to 35.8%, barely surpassing or falling below random selection, and chain-of-thought prompting does not resolve the failure.The experiments evaluate six instruction-tuned LLMs on ten-turn dialogues with intervention turns between t3 and t8.
  • 4.1 Cross-culture Conflict Detection (RQ1): Most models rely on an entrenched early-turn positional prior, with Cramér’s V of 0.17–0.36 for five models and accuracy at or below chance.Llama-3.1-8B is the exception, with V = 0.54.
  • 4.2 Cross-culture Conflict Mediation (RQ2): Baseline mediation AUC ranges from 0.68 to 1.44, below the unfiltered ground-truth reference of 1.64; explicit DMIS definitions raise AUC on every backbone by 0.03–0.20.Effects on W1 and Judge are model-dependent, and Gemma-2-9B shows negligible measurable rubric effects.
  • 4.2 Cross-culture Conflict Mediation (RQ2): Layer-wise probing finds near-ceiling prompt-end DMIS decodability under both conditions, but unguided utterance-end decodability collapses from 0.96 to 0.83.The result supports late-layer elicitation failure rather than missing DMIS knowledge; Gemma-2-9B reaches an utterance-end ND value of 0.80.
  • 4.3 Effect of Supervised Fine-tuning (RQ3): SFT raises turn accuracy to 98.1%, 96.9%, and 91.8% for Llama-3.1-8B, Gemma-2-9B, and Phi-3.5-mini, respectively.After controlling for timing, AUCexact improves on all three backbones: +0.517 → +0.893, +0.659 → +0.807, and +0.412 → +0.711.

5 Conclusion

CC-MEDIATION combines DMIS-grounded data and trajectory metrics to diagnose when and how LLM mediation fails, while highlighting limits in stance measurement, language coverage, synthetic validity, cultural framing, and misuse.

  • Contributions: CC-MEDIATION jointly anchors its corpus and AUC/W1 metrics in DMIS to track downstream intercultural stance shifts.The dataset and metrics move beyond surface-form checks toward trajectory-level evaluation.
  • Findings: Current LLM limitations span intervention timing and mediation content, attributed respectively to a positional prior and late-layer elicitation failure.The conclusion frames these as distinct mechanisms rather than a single knowledge deficit.
  • Limitations: The metrics capture expressed dialogue stance, but cannot fully distinguish genuine intercultural change from face-saving or conflict avoidance.They also do not directly measure mediator trust or broader relational dynamics.
  • Limitations: All dialogues are English-only, omitting multilingual identity, alignment, disengagement, and language-asymmetry signals present in many real conflicts.The paper identifies multilingual extension as an important direction for future work.
  • Limitations: Because CC-MEDIATION uses LLM-generated synthetic dialogues and supervision, its external validity to human-authored conflicts remains open.Real conflicts may contain emotional depth, non-verbal nuance, and adversarial dynamics that the pipeline cannot fully simulate.
  • Ethics: The paper mitigates cultural essentialism by modeling value dimensions and utterance-level DMIS stages rather than treating country pairs as fixed cultural portraits.The scenarios are intended as DMIS-stage triggers, not bilateral cultural descriptions.

A Reference Material

The reference material defines the DMIS stage axis and the stage-conditioned construction rules that support CC-MEDIATION’s conflict and mediation scenarios, alongside human checks of dialogue quality.

  • Construction rules: The CONFLICT_TABLE supplies stage-specific conflict descriptions and mediation moves for construction prompts.Its rows encode categorically different intervention strategies for the three ethnocentric stages.
  • Evaluation protocol: Human evaluation used randomized Prolific items, binary judgments for most studies, and pooled agreement with Wilson 95% confidence intervals.The protocol paid annotators £6 per hour.
  • DMIS taxonomy: CC-MEDIATION operationalizes six ordinal DMIS stages from ethnocentric Denial, Defense, and Minimization to ethnorelative Acceptance, Adaptation, and Integration.The benchmark restricts starting conflict positions to the three ethnocentric stages and targets movement toward the ethnorelative side.
  • Dialogue quality: For n = 159 scenarios, human ratings were 86% for Naturalness, 79% for Transition Smoothness, and 73% for Cultural Conflict Salience.All three criteria rejected the 50% chance null at p < 10^-9.

B.2 DMIS Labeler Accuracy

The DMIS labeler and mediation-admission criterion show substantial pooled agreement with human judgments, while AUC and signed W1 track both pairwise and continuous assessments of mediation success.

  • Labeler accuracy: 72.6% pooled human agreement supported the auto-assigned DMIS stages across 117 items.The Wilson 95% CI was [63.9, 79.9], and every stage rejected the 1/3 random baseline at p < 10^-4.
  • Acceptance validation: 73.3% pooled agreement supported auto-assigned Acceptance labels across 60 post-conflict turns.Acceptance accounted for 56.6% of post-conflict turns, while Adaptation and Integration appeared in 0.1% and 0.0%.
  • Utterance quality: 74.2% Controllability, 86.0% Specificity, and 80.6% Form ratings supported the quality of admitted mediation utterances on n = 93 items.The tests rejected a 50% chance baseline for all three criteria.
  • Pairwise validation: 72.0% pooled agreement showed that the joint AUC–W1 criterion selected the mediation humans judged more successful in pairwise comparisons.The Wilson 95% CI was [62.2, 80.1], with p < 10^-4 against the 50% chance null.
  • Pointwise validation: AUC correlated with human Acceptance ratings at Spearman ρ = +0.637, while signed W1 reached ρ = +0.562 across 30 samples.Leave-one-rater-out folds preserved the AUC correlation with ρ ∈ [+0.529, +0.686].
  • Construct validity: AUC also tracked a theory-independent mediation-effect judgment, though more weakly than the DMIS-defined construct.This reduces concern that the anchored correlation arose solely from shared Acceptance wording.

C.1 Cross-Model Robustness

Cross-model tests indicate that the reported mediation effects persist when mediator, simulator, and labeler models are replaced, while probing supports a late-layer stage-channel elicitation failure rather than an utterance-content gap.

  • Cross-model robustness: Across seven completed model combinations, post-intervention AUC remained strictly positive at +0.71 to +0.75 and W1 ranged from +0.96 to +1.13.Swapping the labeler lowered AUC by 0.10–0.15, while W1 increased; Claude substitutions preserved trajectory direction and magnitude.
  • Cross-model robustness: Claude-3.5-Haiku substitutions preserved mediation trajectories with Judge above 4.3 throughout.These substitutions test whether effects depend on the original mediator and continuation simulator family.
  • Content probe: The utterance-content probe responded to position, with Utterance-end cosine exceeding Prompt-end by 0.08–0.13.This confirms that the probe detects in-context utterance content.
  • Elicitation diagnosis: The rubric-versus-no-rubric utterance-content gap stayed within |∆utt elic| ≤ 0.02, opposite in sign and far smaller than the stage-channel ∆elic ≈ +0.15.The result supports a rubric-dependent stage anchor rather than a content anchor.

C.3 Effectiveness of FT Model

SFT removes the positional bias of base LLMs and stabilizes late-layer DMIS representations across the three trained backbones. Its timing gains generalize across value-dimension and template shifts, while country shifts expose a cultural-coverage boundary.

  • Timing calibration: Base models concentrate predictions on early turns, especially t3, across model scales and families.Claude-3.5-haiku assigns 82% to t3, while Gemma-2-9B assigns 76%.
  • Timing calibration: SFT realigns predicted intervention-turn distributions with ground truth across all three trained models.SFT probabilities over t3–t8 fall within 24–29%, 23–28%, and 18–31%, compared with the GT range of 25–28%.
  • Representation stability: SFT eliminates late-layer DMIS representation decay, with probe accuracy exceeding 0.95 from middle layers onward.The fragile Utterance-end·ND condition improves by approximately +0.04, +0.02, and +0.04 for Llama-3.1-8B, Gemma-2-9B, and Phi-3.5-mini.
  • Distribution shift: On held-out WVS-dimension and template shifts, larger backbones retain near in-distribution turn accuracy of 95.2–97.9% versus 98.1%.The country holdout alone lowers timing to 79–84% and Judge to 2.0–2.1, indicating dependence on cultural coverage.
  • Paired comparison: Paired Base–SFT comparisons on identical scenarios remove the apparent AUC degradation, with every AUC delta positive and no negative delta distinguishable from zero.Judge rises significantly on the two larger backbones, from 2.90 to 4.69 and from 3.06 to 4.21.

C.6 Evaluation-GT Performance by Filter Outcome

The post hoc Step-4 gate separates evaluation mediations by AUC/W1 improvement without serving as a general quality ceiling. The passed subset therefore provides an AUC/W1 reference rather than a universally superior evaluation subset.

  • Filter outcome: 41 of 158 evaluation ground-truth mediations pass the Step-4 gate, yielding a 25.9% pass rate.The gate is applied post hoc for analysis only; evaluation results remain based on the unfiltered set.
  • Filter outcome: +2.38 versus +1.38 AUC and +1.80 versus +1.45 W1 distinguish passed from failed mediations.The unfiltered AUC mean of 1.64 is pulled down by filter-failing cases and is not an attainable ceiling.
  • Filter outcome: Judge remains essentially flat across passed and failed subsets, at 4.93 versus 4.95.Because the gate conditions on AUC/W1 improvement, it is not a general quality selector.

C.7 Marker-Level Analysis of the Gemma-2-9B Rubric Anomaly

The Gemma-2-9B rubric anomaly is a surface-realization failure: the rubric restores internal stage information but does not reliably express it in generated mediation. The case studies illustrate how stage-appropriate reframing can shift dialogue trajectories without changing speakers’ substantive preferences.

  • C.7 Marker-Level Analysis of the Gemma-2-9B Rubric Anomaly: Gemma-2-9B’s rubric change is statistically negligible across AUC, W1, and Judge.AUC changes from 0.72 to 0.75, while W1 changes from 0.81 to 0.78 and Judge from 3.34 to 3.20; none is distinguishable from zero.
  • C.7 Marker-Level Analysis of the Gemma-2-9B Rubric Anomaly: Gemma-2-9B alone drops in stage-marker application from 82.4% to 61.6%, with 60.2% of marker-bearing utterances mismatching the target stage.Its utterance-end D probe reaches 1.00, locating the anomaly in surface realization rather than representation.
  • D Case Studies: The worked cases use ten-turn dialogues with original and mediated continuations, all drawn from the chosen path and scored 5.The mediator is Claude-3.5-haiku in all three examples.
  • D.1 Denial — sid 388 (tconf = 8): The Denial dialogue contrasts individualist self-fulfilment with collectivist family-pride through the speakers’ gift preferences.The unm ediated turns explicitly develop these competing cultural frames.
  • D.1 Denial — sid 388 (tconf = 8): In the Denial case, mediation redirects speakers from topic levelling to cultural symbolism and produces Acceptance while Omar retains the scarf preference.Without mediation, Omar reaches Defense and Kelly reaches Minimization.
  • D.1 Denial — sid 388 (tconf = 8): The mediated Denial continuation changes Omar’s stance toward the cultural dimension while preserving his substantive preference.He describes the scarf through heritage, family, and tradition rather than flattening cultural difference.

D.2 Defense — sid 62 (tconf = 6)

The Defense case shows mediation moving a polarized dispute onto the shared question of what makes a good neighbor. Both speakers then acknowledge cultural differences while retaining their original substantive views.

  • D.2 Defense — sid 62 (tconf = 6): Ryan and Mia’s conflict contrasts Mediterranean sociability with Singaporean preference for quiet, considerate neighbors.The scenario concerns whether heavy-drinking neighbors can support a good living environment.
  • D.2 Defense — sid 62 (tconf = 6): The unmediated dialogue sustains Mia’s Defense despite Ryan’s conciliatory attempts.Mia rejects heavy drinking as incompatible with respect, quiet, and a decent living environment.
  • D.2 Defense — sid 62 (tconf = 6): The mediator reframes the binary dispute around shared qualities of a good neighbor and a positive living environment.The intervention asks both speakers to identify common values concerning community and mutual respect.
  • D.2 Defense — sid 62 (tconf = 6): After mediation, both speakers reach Acceptance while maintaining their different views about drinking and neighborhood standards.Ryan emphasizes community and fun, whereas Mia prioritizes responsibility, safety, and peace.
  • D.2 Defense — sid 62 (tconf = 6): The broader case discussion contrasts Defense’s polarization with mediation that absorbs emotional charge without recapitulating the cultural contrast.The mediator names a shared concern before asking the reframing question.
  • D.3 Minimization — sid 359 (tconf = 4): The paper also distinguishes Minimization, in which a speaker treats family-pride as universal, from mediation that legitimizes cultural difference.That pattern is associated with sustained Acceptance rather than convergence on one preference.

E.1 Implementation details.

CC-Mediation-SFT uses a dual-head transformer to predict when to mediate and generate the mediation utterance in one forward pass. Its training and prompting procedures encode turn-level intervention decisions, DMIS conflict stages, and constrained dialogue generation.

  • Training and inference: Training inserts the SPEAK span at the gold intervention turn, whereas inference permits at most one span at the predicted turn.Decision labels mark the gold turn and assign zero elsewhere; language-model labels outside the mediation span are masked.
  • Model architecture: The dual-head architecture uses <TURN_END> states for intervention timing and [SPEAK] delimiters to generate one mediation utterance.The decision head selects the first turn whose SPEAK probability reaches 0.5; the LM head then decodes until [/SPEAK] or EOS.
  • Training objective: CC-Mediation-SFT jointly trains weighted binary cross-entropy for timing and masked next-token cross-entropy for mediation content.The timing loss corrects the roughly 9:1 SKIP:SPEAK imbalance, while the language-model loss is restricted to the single SPEAK span.
  • Dialogue construction: Dialogue prompts keep each agent’s cultural value stable, forbid capitulation, and require agents to yield from the exchange rather than their position.The generated conversations run for up to 10 message exchanges, with short turns and dialogue-only output.
  • Annotation and timing: The pipeline marks conflict transitions, labels post-conflict turns with six DMIS stages, and instructs intervention only when the ethnocentric pattern is enacted.The DMIS labeler produces a six-stage softmax distribution from first-token log probabilities, while the decision prompt distinguishes unrelated topics from transitions.

G Comparison with Existing Datasets

Existing corpora do not jointly satisfy the benchmark’s three requirements for cross-cultural mediation. In particular, they may omit explicit interspeaker conflict or fail to represent cultural backgrounds through interaction between speakers of different cultures.

  • Dataset coverage: No existing dataset jointly satisfies all three requirements for cross-cultural mediation identified by the paper.This coverage gap motivates the related-work claim and the need for a newly collected human-authored evaluation set.
  • Dataset coverage: The comparison distinguishes datasets lacking explicit interspeaker conflict from those lacking cross-cultural interaction between speakers.The table marks cultural values without explicit conflict and cultural backgrounds without interaction between speakers of different cultures as partial coverage.
Loading 2609.04855v1…