PaperlayerView exact revision on arXiv

Source-linked AI summary

CultureConverse: A Multilingual Multi-turn Simulation Harness for Culturally Grounded Assistance in East and Southeast Asia

Bryan Chen Zhengyu Tan, Weihua Zheng, Thong T. Doan, Bich Ngoc Doan, Jia Wang Peh, Xiaoyuan Yi, Jing Yao, Xing Xie, Nancy F. Chen, Zhengyuan Liu, JinYeong Bak, Wafi Shamdi, Soo Kai Chie, Liew Yu Siong, Aina Azyyati Binti Mohamad Rezal, Lew Yan Yan Vanessa, Huadan Wu, Dylan Raharja, Nadya Yuki Wangsajaya, Akane Fukushige, Kazushi Kato, Koji Inoue, Tatsuya Kawahara, Jaehyung Seo, Dongjun Kim, Seungyoon Lee, Zi Haur Pang, Rui Yang Tan, Charibeth Ko Cheng, Maria Regina Justina Estuar, Jann Railey Montalan, Pham Minh Duc, Roy Ka-Wei Lee

arXiv:2608.28405v1cs.CLcs.CY

TL;DR

Cultural evaluations often test single-turn factual recall rather than practical, multi-turn assistance that requires inferring implicit constraints. CultureConverse simulates culturally situated help-seeking dialogues, and its evaluation finds GPT-5 mini achieves the highest assistance quality.

  • Problem

    Existing cultural evaluations often use single-turn, knowledge-based MCQs that test factual recall rather than practical, multi-turn assistance.

    Source: §1 · p. 1

  • Method

    CultureConverse evaluates multi-turn dialogues between culturally situated simulated users and assistants, requiring assistants to infer constraints from user turns alone.

    Source: §1 · pp. 1–2

  • Results

    27,860 high-quality samples used for fine-tuning improve downstream performance on existing cultural and safety benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings suggest LLM-based harnesses can scale subjective, multi-turn evaluation without reducing cultural competence to trivia.

    Source: §6 · p. 8

    The benchmark is not an exhaustive account of cultural truth, and its 58-identity taxonomy necessarily compresses community diversity.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Current cultural evaluations for large language models (LLMs) often reduce culture to single-turn factual recall via MCQs, failing to capture a common use case: users seeking practical help over multiple turns in culturally grounded scenarios. We introduce CultureConverse, a scalable, multilingual simulation and evaluation harness for culturally grounded assistant dialogue that covers 10 East and Southeast Asian regions, 58 subgroup identities, and 7 domains. Each simulated and evaluated episode produces a scored interaction where the assistant assists the user and infers cultural constraints from partial information. The resulting CultureConverse-DS dataset contains 14,610 benchmark (evaluation) episodes and 274,295 oracle-guided (gold-mode) dialogues. In our benchmark evaluation of 18 models, GPT-5 mini achieves the highest assistance quality. Human annotation experiments suggest that our evaluation framework is a sufficient proxy for human judgment. Performance gains from fine-tuning on 27,860 high-quality CultureConverse-DS samples improve in-domain assistance and transfer out-of-domain to cultural MCQ and safety classification benchmarks. We release the harness, both splits, and judge prompts to support interactive evaluation of cultural competency.

1 Introduction

CultureConverse shifts cultural evaluation from static factual recall to multi-turn assistance, where models must infer hidden cultural constraints from users’ progressively disclosed turns. It introduces a multilingual simulation harness and dataset for evaluating and training culturally grounded assistance.

1 Introduction

  • Existing evaluations miss subgroup variation, multi-turn context maintenance, and pragmatic assistance; SEADialogues (Kautsar et al., 2025) and LiveCultureBench (Pham et al., 2026) differ from this help-seeking setting.Everyday queries can require inferring implicit cultural constraints across turns, and failures can produce fluent but socially inappropriate advice, disproportionately affecting users whose norms are underrepresented in training data.Source: §1 · p. 1
  • CultureConverse evaluates assistant-style help-seeking under hidden cultural constraints, requiring models to infer them from user turns rather than rely on cultural fact recall.Each episode is a multi-turn dialogue between a culturally situated simulated user and an assistant; region, subgroup identity, and active taboo tripwires are hidden from the assistant.Source: §1 · pp. 1–2
  • The harness covers ten East and Southeast Asian regions, 58 subgroup identities, and seven domains, and benchmarks 18 frontier LLMs across 14,610 simulated dialogues.The dialogues are grounded in curated Cultural and Taboo Knowledge Bases; automated metrics strongly align with human consensus.Source: §1 · p. 2
  • Fine-tuning on 27,860 highest-quality samples from CULTURECONVERSE-DS improves downstream performance on existing cultural and safety benchmarks.The dataset contains 274,295 guided training trajectories.Source: §1 · p. 2

2 Related Work

Prior work identifies language- and culture-specific gaps, studies cultural norms and interactive evaluation, and highlights challenges in judging cultural competence. CULTURECONVERSE frames practical assistance as culturally grounded, multi-turn interaction and validates its judge against human annotators.

2.1 Multilingual and Regional Evaluation

  • Multilingual and regional benchmarks expose language and local-commonsense gaps; CULTURECONVERSE shifts evaluation from static accuracy to stateful, culturally situated assistance.SEA-HELM and SEACrowd/SEA-LION broaden coverage of underrepresented Southeast Asian languages (Susanto et al., 2025b; Lovenia et al., 2024; Ng et al., 2025).Source: §2.1 · p. 2

2.2 Cultural Knowledge, Norms, and Values

2.3 Cultural Dialogue, Simulation, and Interactive Evaluations

  • CULTURECONVERSE models hidden user goals and progressive disclosure to evaluate situated, multicultural assistance across turns.Prior multi-turn evaluations find context loss and premature assumptions; interactive frameworks and recent cultural-simulation benchmarks provide related approaches, while synthetic personas can introduce demographic artefacts (Laban et al., 2025; He et al., 2024; Duan et al., 2024; Park et al., 2023; Zhou et al., 2024; Tan et al., 2025; Perez et al., 2024; Hu and Collier, 2024; Li et al., 2025a; Havaldar et al., 2025; Pujari and Goldwasser, 2025; Wu et al., 2025; Pham et al., 2026).Source: §2.3 · p. 2

2.4 LLM Judges for Cultural Evaluation

3 CULTURECONVERSE Construction

CULTURECONVERSE turns culturally grounded scenarios into scored multi-turn episodes where assistants infer and apply implicit cultural norms from partial information. Its construction pipeline separates scenario context, turn-by-turn disclosure, and evaluation, yielding a dataset spanning 10 regions, 58 identities, and 7 domains.

3.1 Overview

  • Each episode is a scored multi-turn dialogue that tests progressive inference of implicit cultural norms rather than static prompt–response performance.The five-stage seed-to-evaluation pipeline separates scenario definition, information timing, and visibility; assistants see public context and chat history, while users and judges have distinct private or oracle information. Scoring covers assistance quality (3H) and ecological validity (TDQ).Source: §3.1 · pp. 3–4

3.2 Persona and Scenario Universe

  • The scenario universe covers 7 domains and 42 subdomains, while 348 personas cross 58 region–identity pairs with three age cohorts and two gender labels.Identities are sampled uniformly to support statistical power for smaller subgroups in fairness analysis.Source: §3.2 · p. 4

3.3 Templates and Seeds

  • 420 portable, culture-agnostic templates combine with 348 personas to yield 146,160 seeds, allocated 50%/25%/25% across STANDARD, CONTEXTUAL, and NORMATIVE tiers.The tiers represent baseline cultural assistance, cross-cultural friction, and taboo or safety-trap detection, respectively.Source: §3.3 · p. 4

3.4 Blueprints

  • Blueprints expand seeds into scenario specifications grounded in retrieved cultural knowledge and taboo evidence, with public, private, and oracle context stored separately.This separation prevents hidden context from leaking into the assistant’s prompt; language directives and validation traces are also stored.Source: §3.4 · p. 4

3.5 Wikipedia and Taboo Knowledge Bases

  • Blueprint generation draws on a cultural knowledge base of 180,992 region-tagged Wikipedia passages and a taboo knowledge base of 962 curated entries.Retrieval uses region and persona-tag filters for cultural passages, and region and category filters for taboo entries; secondary-region filters support cross-cultural seeds.Source: §3.5 · p. 4

3.6 Shards

  • Shard plans contain 3–5 ordered phases that specify user goals, linguistic directives, and success or failure conditions, enabling progressive disclosure of constraints.Shard outputs are audited for structure, cultural plausibility, answerability, and challenge sufficiency; failures are regenerated with feedback within a bounded retry budget.Source: §3.6 · pp. 4–5

3.7 Episodes and Simulation

  • EVAL hides oracle context from assistants, whereas GOLD supplies oracle guidance for higher-fidelity supervised fine-tuning samples; GPT-5 mini serves as simulator and judge.Potential same-family evaluation circularity is checked against 42 region-matched human annotators and a pool of 11 candidate judges and 165 three-judge ensembles.Source: §3.7 · pp. 5–6

3.8 Dataset Statistics and Breakdown

  • 288,905 CULTURECONVERSE-DS episodes span 10 regions, 58 identities, and 7 domains.Table 2 reports breakdowns by split, language mode, region, and challenge configuration.Source: Table 2 · p. 6, §3.8 · p. 6

4 Experimental Setup

Section 4 defines the evaluation episode, the judge’s scoring criteria, and the paired setup used to compare assistant models.

4 Experimental Setup

  • Each episode combines a frozen shard, hidden oracle, safety and cultural tripwires, curated knowledge base, and dialogue; the assistant sees only dialogue while the user and judge receive different context.The user is conditioned on the shard, oracle, and applicable tripwires; the judge sees the transcript, shard, oracle, and relevant knowledge base and tripwires.Source: §4 · p. 6
  • The judge scores dialogue quality with TDQ and assistance with 3H; model comparisons also report the episode clean rate.TDQ averages naturalness, scenario plausibility, and cultural typicality; 3H averages helpfulness, honesty, and harmlessness against the oracle and tripwires, with clean episodes violating no tripwire.Source: §4 · p. 6

4.1 Implementation Details

  • Eighteen assistants are evaluated on 7,305 global-English and 7,305 native-language episodes each, using identical frozen shards for paired comparisons across 10 regions, 58 subgroup identities, and 7 domains.Prompts are fixed across models; GPT-5 mini serves as the user simulator and default judge, and failed or refused calls and malformed outputs are retried or regenerated.Source: §4.1 · p. 6

5 Experimental Results

Across 14,610 episodes per assistant, the experiments assess dialogue realism and cultural assistance, validate the judge against human ratings, and test fine-tuning transfer and performance disparities. GPT-5 mini leads on assistance quality, while results also show human alignment, out-of-domain gains, and uneven performance across identities and languages.

5.1 Evaluating Dialogue Quality (RQ1)

  • GPT-5 mini achieves the highest assistance quality (3H, 4.28), while GPT-5.4 has the highest clean rate (91.0%).Clean rate is the percentage of episodes in which the judge assessed that the assistant avoided all hidden safety tripwires and cultural taboos.Source: Table 4 · p. 7, §5.1 · p. 7
  • GPT-5 mini and GPT-5.4 score highest on simulated-interaction realism (TDQ), at 3.81 and 3.80, respectively.Cross-model variation is driven mainly by Plausibility and Typicality; weak assistants can distort interactions by failing to track users’ situations, languages, or constraints.Source: §5.1 · p. 7
  • Honesty (3.09–4.60) and Harmlessness (3.45–4.47) drive most assistance differences; TDQ and 3H correlate but remain distinct measures.Some models assist well despite weaker realism, while others converse realistically but provide weaker assistance, indicating that CultureConverse measures more than conversational fluency.Source: §5.1 · p. 7

5.2 Human-LLM Judge Alignment (RQ2)

  • Against human consensus, GPT-5 mini as judge achieves 90.1% ±1 agreement and 0.581 MAE, versus 87.5% agreement and 0.727 MAE for human–human ratings.Its slight negative bias (−0.126) suggests stricter-than-human scoring; the authors consider it adequate for aggregate comparisons, not a substitute for expert cultural-truth adjudication.Source: §5.2 · p. 7
  • GPT-5 mini remains the most human-aligned judge across comparisons with 11 models and 165 three-judge ensembles, supporting its use as the sole evaluator.It is competitive with the best ensembles.Source: §5.2 · p. 7

5.3 Downstream Generalisation (RQ3) and Performance Disparities

  • Fine-tuning on 27,860 perfect-score episodes improves overall assistance quality and transfers to 7 cultural MCQ benchmarks and 10 safety classification datasets.Llama-3.1-8B-IT gains +0.88 pp on MCQ and +2.49 pp on CLS; SEA-LION-v4-8B-IT gains +0.52 pp and +0.80 pp, respectively, despite no training on static MCQ or CLS formats.Source: Table 5 · p. 8, §5.3 · p. 8
  • GPT-5 mini shows a 0.165 3H spread across identities, with lower scores for some under-represented groups; language effects vary, and traps reduce 3H by 0.04–0.08 across all 18 models.Age and gender variance is negligible (gender has a 0.011 3H range); Gemini models gain +0.10 to +0.25 3H in native-language dialogues, while other models, including DeepSeek V3.1, degrade.Source: §5.3 · p. 8

6 Discussion

CULTURECONVERSE argues for evaluating pragmatic cultural assistance through extended interaction, while noting that simulator realism and authentic representation remain methodological challenges. Its subgroup sampling choices expose unresolved trade-offs around representation and cultural essentialism.

6 Discussion

  • CULTURECONVERSE targets pragmatic utility in extended, situated interactions, with human alignment and out-of-domain transfer supporting scalable evaluation beyond cultural trivia.The discussion contrasts this approach with MCQs and single-turn prompts that test factual recall.Source: §6 · p. 8
  • Stylistic directives promote conversational diversity while preserving reproducibility, making rigorous simulator controls important for approximating real users.The directives are provided in Appendix I.Source: §6 · p. 8

Subgroup Disparities and Cultural Essential-

  • Equal weighting across 58 subgroups upweights minoritised communities, but proportionality, sociolinguistic bottlenecks, and persona-label risks of cultural essentialism remain unresolved.The discussion warns that diverse community practices may collapse into monolithic stereotypes.Source: Subgroup Disparities and Cultural… · pp. 8–9

7 Conclusion

CultureConverse introduces a scalable harness and dataset for culturally grounded, multi-turn AI assistance. Experiments show the harness differentiates models in pragmatic help-seeking scenarios, aligns strongly with human consensus, and supports downstream generalization through fine-tuning.

7 Conclusion

  • CultureConverse introduced a scalable simulation and evaluation harness and CultureConverse-DS, a benchmark and training corpus for culturally grounded, multi-turn AI assistance.The conclusion characterizes CultureConverse-DS as large-scale.Source: §7 · p. 9
  • Experiments show CultureConverse differentiates models’ ability to apply cultural nuance in pragmatic, help-seeking scenarios, with strong alignment to human consensus.Source: §7 · p. 9
  • Fine-tuning on CultureConverse-DS yields measurable downstream generalization across interactive and static cultural benchmarks.Source: §7 · p. 9

Limitations

CultureConverse is a targeted evaluation artefact, not an exhaustive account of cultural truth or a perfect proxy for human prompting. Its taxonomy, generated interactions, and LLM-based evaluation have limitations that future work should address through human-informed review and calibration.

Limitations

  • Although its persona universe captures 58 intersectional identities, the taxonomy still compresses community diversity, and generation varies more for minoritised communities and low-resource languages.These groups face sociolinguistic bottlenecks in the underlying base models.Source: Limitations · p. 9
  • The benchmark’s LLM-based components remain susceptible to parametric biases and rhetorical preferences despite diversity controls and judge validation against human consensus.The taboo knowledge base is augmented with NormAd entries and validated against region-attributed public sources, but the authors recommend region-specific community review and a small human-authored gold dialogue subset to calibrate ecological validity.Source: Limitations · p. 9
  • Generated episodes are targeted evaluation artefacts, not a perfect proxy for human prompting behaviour, and the modest training gains show feasibility rather than large or consistent improvement.Future work should improve training and human-informed curation, audit scenario and dialogue distributions, quality, and ecological validity, and develop more robust, cost-efficient evaluation with broader human calibration.Source: Limitations · p. 9

Ethical Statement

The ethical statement warns against treating persona-based evaluations as deterministic accounts of real populations or using demographic results to rank groups. It calls for transparency and human oversight, especially for sensitive scenarios and real-world deployment.

Ethical Statement

  • Persona-based labels are not rules for real human behavior, episodes should not support broad sociological claims, and demographic slices expose deficits rather than rank groups.The benchmark’s taboo and safety components may include sensitive norms and potentially harmful scenarios.Source: Ethical Statement · p. 9
  • The authors preserve source metadata, recruit annotators from relevant regional and linguistic backgrounds when feasible, and compensate annotation-only contributors fairly.Compensation follows applicable institutional or local rates.Source: Ethical Statement · p. 9
  • Real-world deployment requires local expert review, continuous community feedback, and context-specific red-teaming.External data was used according to its licenses; AI assistants aided manuscript editing and structuring, with all content reviewed and verified by the authors, and CultureConverse-DS will be released under CC-BY-4.0 for research purposes.Source: Ethical Statement · p. 9

A Related Benchmark Comparison … Shard excerpts

CULTURECONVERSE-DS evaluates multi-turn, culturally grounded help-seeking dialogues under partial observability, addressing a gap in benchmarks focused on regional knowledge or single-turn etiquette. Its pipeline pairs diverse personas and flexible scenarios with retrieved cultural knowledge, then turns them into staged, scorable interactions.

A Related Benchmark Comparison

  • CULTURECONVERSE-DS evaluates multi-turn, culturally grounded assistant help-seeking dialogues with hidden states and released interaction trajectories, addressing gaps in regional-knowledge and single-turn etiquette benchmarks.The comparison positions it at the intersection of situated assistance, progressive disclosure, and downstream alignment data.Source: A Related Benchmark Comparison · p. 16

B Persona Universe and Identity Construction

  • The persona universe covers ten East and Southeast Asian regions and 58 subgroup identities, crossed with three age cohorts and two gender labels for 348 configurations.Identities combine socially relevant ethnicity, religion, language, and regional background; equal sampling supports subgroup-level comparisons rather than population-proportional representation.Source: B Persona Universe and Identity… · p. 16

C Scenario Universe: Domains and Subdomains

  • The scenario universe comprises 7 domains and 42 subdomains, using broad descriptions so cultural specificity emerges from pairing scenarios with personas and retrieved local knowledge.Subdomains guide both abstract template writing and later culture-specific blueprint grounding, enabling challenges across all ten regions.Source: C Scenario Universe: Domains and… · p. 16

D Knowledge Bases for Cultural Grounding; KB samples injected into context

E Generated Artefacts

  • A canonical item is traced through the generation pipeline, with a persistent item key allowing downstream scores to be traced to their underlying assumptions.Figures 13–16 illustrate the item’s progression from template through blueprint to shards.Source: E Generated Artefacts · p. 22

E.1 Template

  • Templates separate generic logistical or social challenges from cultural markers using variable slots, supporting synthesis across demographic permutations without manual authoring for each scenario.The communal-dining template is an example of this flexible scaffold.Source: §E.1 · pp. 22–23, Figure 13 · p. 22

E.2 Blueprint; Blueprint excerpt

  • Blueprints combine templates, personas, and retrieved evidence while separating the user’s public message from oracle-only taboos, safety tripwires, and success criteria.This separation is intended to mitigate prompt leakage and preserve realistic partial observability.Source: Blueprint excerpt · p. 23, Figure 14 · p. 23

E.3 Shard; Shard excerpts

  • Each blueprint becomes an ordered sequence of 3–5 shards specifying private user goals, cultural signifiers, and pass/fail transitions to structure progressive disclosure.The transition logic is designed to prevent premature task completion or simulator hallucination of progress.Source: Shard excerpts · p. 24, Figure 15 · p. 24

E.4 Transcript and Evaluation Record … K.1 Use of Gold Mode for Training Data

The appendices specify how CultureConverse records and scores episodes and generates culturally grounded, multi-turn challenges using blueprints, retrieved cultural knowledge, and persona- and language-sensitive simulations. They also describe prompt constraints, validation and variation controls, and oracle-guided Gold-mode generation for training data.

Transcript and evaluation excerpt

  • The final record combines the complete multi-turn dialogue, execution metadata, and multidimensional judge scores to expose successes and failures and filter trajectories for fine-tuning.Figure 16 illustrates the conversation-and-score record.Source: Transcript and evaluation excerpt · p. 24, Figure 16 · p. 24

F Model Assignments and Deployment Roles

  • GPT-5 mini serves as user simulator, gold-mode training assistant, and evaluator, while the blind benchmark varies the target assistant model.Table 9 outlines deployment configuration and provenance; Table 4 reports the comparative analysis.Source: F Model Assignments and Deployment Roles · p. 25
  • 43,830 judge calls scored one 7,305-episode language split across six metrics in approximately 52 minutes at concurrency 100, costing US$142–148.API costs vary with model/provider pricing and output length.Source: F Model Assignments and Deployment Roles · p. 25

Template Generator system prompt

  • Template prompts require neutral, universally applicable everyday scenarios with plug-and-play culture-specific slots, varied practical or lookup tasks, and only occasional realistic normative risks.The prompt discourages forced cultural nuance and asks templates to enable 3–5 inference steps without over-constraining instantiation.Source: Template Generator system prompt · p. 25

G.1 Stage 1: Template Generator

  • The Template Generator produces 10 culture-agnostic, parameterized scenario sketches per execution, guided by the target domain, subdomain, and requested output count.Templates use variable slots and aim for subtle cultural nuance and multi-hop reasoning while leaving room for creative instantiation.Source: Template Generator system prompt · p. 25

G.2 Stage 2: Blueprint Generator

  • The Blueprint Generator combines a persona, template, challenge configuration, split, and retrieved cultural and taboo knowledge into an interaction specification with separated public, private, and oracle-only contexts.Its prompt also addresses anti-leakage, safety-tripwire and taboo formatting, and downstream JSON structure.Source: §G.2 · p. 25

Blueprint Generator system prompt

  • Blueprint scenarios must require 3–5-hop cultural reasoning, fit the persona’s domain/subdomain, use at most one essential niche term initially, and keep private oracle context unseen by the evaluated assistant.They also require full native-language conversation scripts and caution against caricature and unsupported adherence assumptions.Source: Blueprint Generator system prompt · pp. 26–27
  • Global_English and Native_Language variants convey the same message content in different languages, while Western, Expat, and Eurasian personas use English.Source: Blueprint Generator system prompt · p. 27
  • Scenarios align challenge types with persona-shaped preferences, contextual friction, or identity-specific tripwires, and require subtle multi-clue reasoning across competing norms.The prompt calls for implicit cultural or ethical tripwires whose safe resolution combines clues such as kinship, calendar, status, religion, and legality.Source: Blueprint Generator system prompt · pp. 27–28
  • The generator separates assistant-visible public context, private simulator information, and oracle ground truth, with explicit, measurable safety expectations for harmful-action tripwires.Tripwires should describe harmful assistant actions, such as recommending bribery or unsafe medication advice, rather than aspirational requirements.Source: Blueprint Generator system prompt · p. 27
  • Initial user messages should preserve hidden cultural constraints for inference, use niche terms only when essential, and avoid enumerating multiple deliverables.The prompt also prohibits stating or indirectly leaking taboo constraints in the initial message.Source: Blueprint Generator system prompt · p. 28
  • Cultural knowledge should be shown through actions rather than announced, with correct romanization and practices matched to the persona’s identity and region.The guidance favors action-based signifiers, such as using a grandparent’s honorific, over explicit commentary about face or respect.Source: Blueprint Generator system prompt · p. 29
  • Blueprints distinguish implicit-constraint, contextual-friction, and normative-friction challenges, centering concrete practical tasks whose answers require inferring culture from natural cues.Normative challenges should include realistic taboo, safety, or legal risks; other challenges should use subtle cues such as kinship, time, place, or register.Source: Blueprint Generator system prompt · p. 29
  • Cross-cultural scenarios must be plausible and meaningfully driven by an information gap between regions, while taboo-grounded risks remain implicit to the user.The design calls for multi-hop reasoning across both cultures and requires taboo risks to derive from provided taboo entries that the user does not know about.Source: Blueprint Generator system prompt · pp. 29–30

G.2.1 Blueprint Generator runtime user prompt

  • The runtime prompt supplies seed, language mapping, optional subdomain context, template, output schema, opener directives, and optional taboo context in sequence.Its required JSON schema includes persona, challenge configuration, a multi-hop chain and complexity, scenario instantiation, public and private context, and evaluator-oracle context.Source: §G.2.1 · p. 30

Blueprint Generator runtime user prompt structure

  • Runtime prompt structure specifies the canonical native language and injects retrieved cultural context before the template to ground niche terms, real practices, and 3–5-hop inference.The native-language message field is left empty in this structure, with a separate instruction specifying the full native script variety.Source: Blueprint Generator runtime user… · p. 30

G.2.2 KB query prompts and context injection

  • Cultural knowledge retrieval uses natural-language queries with an instruction prefix and region filters, while taboo retrieval is queried separately for taboo-grounded normative scenarios.Retrieved context is formatted into the blueprint prompt as evidence for culturally specific slot filling and multi-hop inference.Source: §G.2.2 · p. 30

Cultural KB query prompt and retrieved-context injection

  • The query builder draws on cleaned template text, slot topics, identity, cultural dimensions, and region to retrieve articles about specific cultural terms and practices.Cross-cultural queries account for both regions; the retrieval instruction targets relevant locations, foods, rituals, and taboos.Source: Cultural KB query prompt and… · pp. 30–31
  • Cultural retrieval targets relevant terms, locations, practices, foods, rituals, and taboos in East and Southeast Asian cultures, with retrieved context and niche terms supplied for dialogue.The prompt format includes Wikipedia article titles and regions, plus niche cultural terms to use naturally.Source: Cultural KB query prompt and… · p. 31

Taboo KB query prompt and taboo-reference injection

  • Taboo retrieval seeks cultural taboo descriptions, forbidden actions, and social norms using scenario and persona details, then injects found entries as a taboo reference.Source: Taboo KB query prompt and… · p. 31

G.3 Stage 3: Shard Generator

  • The Shard Generator turns each validated blueprint into a 3–5-phase script with private user goals, cultural signifiers, bilingual directives, and explicit success or failure transitions.Phases structure how cultural details emerge over the interaction rather than presenting them all at once.Source: §G.3 · p. 31

Shard Generator system prompt

  • Shard instructions make cultural knowledge legible through actions and practices rather than explicit identity statements, while keeping taboo awareness out of user-facing fields.For taboo-grounded scenarios, the user unknowingly sets up the relevant situation; taboo awareness appears only in evaluator-private success and failure conditions.Source: Shard Generator system prompt · pp. 31–32
  • Users seek advice and provide context, while assistants provide recommendations or other help and are the actors evaluated.Shard goals should frame advice-seeking rather than document-production requests.Source: Shard Generator system prompt · pp. 31–32
  • Shard design is persona-specific and avoids stereotyping while incorporating relevant regional details and requiring both Global_English and full native-language-script directives.The native-language script is matched to persona identity, and native-language turns are not code-switched.Source: Shard Generator system prompt · pp. 31–32
  • Shard sequences gradually disclose signifiers, and assistant responses are judged against explicit success conditions and failure loops for missteps such as missed cues or unsafe advice.Each shard specifies 2–3 success conditions and 2–4 failure conditions.Source: Shard Generator system prompt · p. 32
  • Shard conditions are atomic, orthogonal, independently evaluable, persona-relevant, equally weighted, assistant-focused, and mostly achievable through direct advice.At least two of three conditions should be achievable without questions, and no more than one should require a question; goals and advice must remain plausible, culturally accurate, coherent across shards, and aligned with signifiers and taboos.Source: Shard Generator system prompt · pp. 32–33

G.3.1 Shard Generator runtime user prompt

  • The runtime user prompt supplies a validated blueprint and a minimal JSON schema defining the deployed structure and variable payloads.The required structure includes a seed, simulation script, interaction sequence, and secondary subdomains.Source: §G.3.1 · p. 33

Shard Generator runtime user prompt structure

  • The excerpted simulation-script schema includes a seed ID, shard ID, phase label, and user instruction, and lists success-exit and failure-loop conditions; secondary subdomains are optional.Source: Shard Generator runtime user prompt… · p. 33

G.4 Stage 4: User Simulator

  • The user simulator advances only after the assistant gives relevant, new, actionable advice, and terminates unresolved episodes after a maximum of five turns.If goals remain unresolved at that limit, the interaction is logged as a failure.Source: §G.4 · p. 33

User Simulator system prompt

  • The simulated user knows their own situation and seeks advice, while evaluating assistant responses for goal relevance, new information, and actionability before advancing topics.The simulator is instructed to answer assistant follow-up questions and continue the current topic when any of its evaluation checks fail.Source: User Simulator system prompt · p. 34

H Output Validation and Fault Tolerance

  • Pipeline checks validate blueprints, shards, simulated turns, and transcripts; structural faults trigger repair, retries, or exclusion, while genuine assistant failures remain scored.Episodes aborted by infrastructure or parsing errors are excluded from aggregate metrics, but refusals, harmful compliance, and hallucinations are evaluated.Source: H Output Validation and Fault Tolerance · pp. 34–35

I Enhancing Conversational Variety in the Simulator

  • Seeded diversity controls select two of 48 opener archetypes for the initial message and rotate among 40 behavioral directives on later turns.Later-turn directives are selected by hashing the episode seed and turn index; rerunning an identical seed yields highly consistent interaction, while varying seeds produces distinct conversational dynamics.Source: I Enhancing Conversational Variety in… · p. 35, Figure 17 · p. 35

J Gold-Mode (Oracle-Guided) Generation Prompt

  • Gold mode gives the assistant the user persona, shard goal, success criteria, safety tripwires, and cultural taboos, and prompts calibrated follow-up questions about unstated constraints.This oracle context is intended to make reference trajectories targeted, safe, and efficient without verbosity.Source: J Gold-Mode (Oracle-Guided) Generation… · p. 35

Gold assistant system prompt

  • The gold assistant must use oracle context for advice without revealing hidden persona or goal details, avoid invented facts, respect safety tripwires and taboos, and calibrate uncertainty.The prompt also calls for useful, conversational advice and compliance with the required response language and script.Source: Gold assistant system prompt · pp. 35–36

I Enhancing Conversational Variety in the Simulator

  • Forty response-style directives are pooled, with one deterministically selected and injected before each simulated user turn for t ≥1.Figures 18 and 19 detail the stylistic pools and selection logic, while Figure 17 shows prompt fragments injected during execution.Source: I Enhancing Conversational Variety in… · p. 35, Figure 18 · p. 36

K.1 Use of Gold Mode for Training Data

  • +0.13 3H assistance quality and +5.5 pp clean rate were achieved by Gold-mode generation versus blind generation, with negligible impact on scenario realism (TDQ).The comparison uses GPT-5 mini on the matched test split.Source: §K.1 · p. 37, Table 10 · p. 37

K.2 Ablations for GPT-5 mini as Judge · L Judge Prompts

GPT-5 mini is the most human-aligned individual judge, while three-judge ensembles offer little overall benefit; the judge prompts use episode-specific runtime context and metric-specific criteria.

K.2 Ablations for GPT-5 mini as Judge

  • Across regional and metric slices, GPT-5 mini generally stays within the human–human agreement bandwidth.The regional table reports signed bias as judge score minus human consensus.Source: §K.2 · p. 37, Table 12 · p. 37, Table 13 · p. 37
  • GPT-5 mini achieves the lowest individual-judge MAE (0.581) and highest within-one agreement (90.1%), while the best ensemble lowers MAE to 0.540 but reduces agreement to 86.4%.The best ensemble also fails to improve signed bias, supporting GPT-5 mini as the single canonical judge.Source: Table 11 · p. 37, §K.2 · p. 37
  • The judge-panel sweep compares 11 individual judges and 165 three-judge ensembles against a human-annotated subset.The authors weigh possible alignment gains from alternative frontier judges or ensembles against added computational overhead and pipeline complexity.Source: Table 11 · p. 37, §K.2 · p. 37

L Judge Prompts

  • Six core-metric prompt templates populate bracketed fields at runtime with the transcript, persona summary, oracle context, and generation metadata.The runtime variables are colour-highlighted in the prompts.Source: L Judge Prompts · p. 38

L.2 Helpfulness

  • The helpfulness prompt evaluates goal resolution, actionability, and persona-aware specificity against hidden success criteria and shard completion rates.These criteria are used to assess assistance in the episode.Source: Figure 21 · p. 38

L.4 Naturalness

  • The naturalness prompt evaluates conversational flow and suitability for use as training data.Source: Figure 23 · p. 39

M Human Annotation and Judge Alignment … O Performance Differences Across Personas, Regions, and Languages

The section reports GPT-5 mini as the strongest individual judge, illustrates both high- and low-scoring episodes, and describes analyses of performance differences across demographic and linguistic groups. The examples show that culturally specific advice can still fail when it overlooks religious preferences or recommends deception.

M Human Annotation and Judge Alignment

  • The validation subset contains 1,250 dialogues scored by 42 annotators on six 1–5 Likert metrics, with three independent scores per item–metric cell.Annotators were region-matched where feasible, and native or professional fluency was required for non-English dialogues.Source: M Human Annotation and Judge Alignment · p. 42
  • GPT-5 mini is the strongest individual judge; the best post-hoc ensemble offers negligible MAE improvement but lower within-one agreement.Human–judge MAE is measured against the human mean; signed bias indicates whether the judge is stricter or more lenient than the panel.Source: M Human Annotation and Judge Alignment · p. 42

N.1 Additional High-Scoring Examples

  • Figures 27–30 show shortened high-scoring GPT-5 mini evaluation episodes while preserving each scenario, language mode, and evaluation scores.The excerpts focus on household reciprocity and food hosting, wedding and partner-family introductions, native-language gift-giving, and household etiquette or allergy handling.Source: §N.1 · p. 42, Figure 27 · p. 44, Figure 28 · p. 45

N.2 Low-Scoring English Examples

  • Figures 31 and 32 present low-scoring English episodes with cultural and safety failures.Source: §N.2 · p. 42

Low-scoring English episode: Borneo neighbour hospitality and tuak

  • One low-scoring English episode recommends tuak as a neighbour gift without checking the neighbours’ religious preferences.Source: Figure 31 · p. 42

Low-scoring English episode: Batak churchwedding envelope

  • Avg. 3H = 2.67 in the Batak wedding-gift episode, where advice to secretly top up the envelope triggered a deception tripwire and Harmlessness scored 1.The response otherwise gave practical local gift ranges and discreet presentation advice.Source: Low-scoring English episode: Batak… · p. 43

O Performance Differences Across Personas, Regions, and Languages

  • Table 14 summarizes trap-episode performance degradation, Table 15 shows GPT-5 mini judge-score distributions across 14,610 episodes, and Table 16 gives the RQ3 fine-tuning configuration.Table 16 specifies training on 27,860 perfect-3H samples.Source: Table 14 · p. 47, Table 15 · p. 47, Table 16 · p. 47
  • The subgroup analysis reports means and maximum-to-minimum ranges across model, region, identity, age, gender, language, and challenge type.Composite 3H and TDQ scores are episode-level averages; native-language deltas are paired seed-level differences from English.Source: O Performance Differences Across… · p. 47

O.1 Bootstrap Confidence Intervals for Headline Results · O.2 Subgroup and Slice Breakdowns · P Out-of-Domain (OOD) Generalisability

The appendices report uncertainty and slice analyses for headline evaluation results, then describe a narrow LoRA fine-tuning setup used to test transfer to static out-of-domain tasks.

O.1 Bootstrap Confidence Intervals for Headline Results

  • Table 17 reports rounded 95% non-parametric bootstrap confidence intervals for the headline families and all six metrics underlying Table 4.The supplied passage does not provide the interval values.Source: §O.1 · p. 47

O.2 Subgroup and Slice Breakdowns

  • 0.04–0.08 points lower 3H scores on trap-labelled episodes across models (mean −0.06) indicate a small, consistent friction signal versus matched standard items.The appendix also reports the judge’s full score distribution across all metrics in Table 15.Source: §O.2 · p. 47
  • Table 15 reports the judge’s full score distribution across all metrics.The supplied passage does not give the distribution values.Source: §O.2 · p. 47

P Out-of-Domain (OOD) Generalisability

  • LoRA SFT runs use Llama-3.1-8B-IT (2024) and SEA-LION-v4-8B-IT (2026).Source: P Out-of-Domain (OOD) Generalisability · p. 47
  • The transfer test fine-tunes on 27,860 CultureConverse-DS trajectories scored exactly 5 for helpfulness, honesty, and harmlessness, without exposure to OOD answer formats.This setup tests transfer of interactive cultural-assistance behavior to static OOD tasks.Source: P Out-of-Domain (OOD) Generalisability · p. 47
  • Runs use LLaMA-Factory and Hugging Face PEFT LoRA on 4× NVIDIA H100 GPUs; one epoch takes about 4–5 hours, and Base–SFT comparisons use 500 episodes per language split.Each language split contains 7,305 episodes.Source: P Out-of-Domain (OOD) Generalisability · p. 47

P.1 OOD Evaluation Suite

The section introduces an independent out-of-domain suite to test whether interactive cultural fine-tuning transfers to static benchmarks without pipeline overlap. It covers normative and regional knowledge alongside localized safety and toxicity judgments.

P.1 OOD Evaluation Suite

  • The suite tests OOD generalization beyond conversational fluency using static multiple-choice knowledge benchmarks and classification benchmarks of localized safety and toxicity judgments.It targets normative and regional knowledge and safety judgments across East and Southeast Asian regions.Source: §P.1 · p. 47
  • 17 datasets comprise 29,828 evaluated examples: 7 MCQ datasets with 11,988 examples and 10 CLS datasets with 17,840 examples.The final filtered suite composition is summarized in Table 19.Source: §P.1 · p. 48, Table 19 · p. 54

Related papers

Loading 2608.28405v1…