Source-linked AI summary
Toward a Culturally Adapted Chinese Language Agent: A Wizard-of-Oz Study of Nonverbal Behavior in Chinese-German Intercultural Interaction
Siddhant Jain, Anna Lea Reinwarth, Dimitra Tsovaltzi, Rafael Math, Julia Renner
TL;DR
German learners may have difficulty adapting to Chinese social norms, while culturally adapted language-learning agents lack grounded behavioral data from authentic intercultural interaction. This paper proposes a Wizard-of-Oz study and multimodal system to collect and annotate responses to norm violations. Its planned corpus and framework are intended to support future automated cultural interpretation and behavior generation.
Problem
German learners may struggle to adapt to Chinese social norms, and culturally adapted agents currently lack grounded behavioral data from authentic intercultural interactions.
Source: §1 · p. 2, §1.1 · p. 2Method
The paper proposes an expert-informed Wizard-of-Oz study with four norm-conflict scenarios, a real-time multimodal system, and a four-layer annotation framework.
Source: §1.2 · p. 2, §1.3 · p. 2, §3.2 · p. 3Results
The proposed setup combines synchronized multimodal data collection with layered annotation designed to support future cultural interpretation and behavior-generation models.
Source: §1.3 · p. 2, §6 · p. 6Takeaways & Limitations
The layered schema separates observable behavior from interpretive labels, supporting corpus reuse across norm detection and behavior generation tasks.
Source: §6 · p. 6Empirical validation remains future work, and the online Chinese-German study may constrain gaze, interpersonal distance, and gesture range compared with face-to-face interaction.
Source: §6.4 · p. 6
Abstract
from arXiv · showhide
Successful intercultural communication requires more than grammatical competence. It demands sensitivity to culturally embedded social norms whose violation triggers subtle but meaningful nonverbal responses. For German learners of Mandarin Chinese, acquiring this sensitivity is critical yet poorly supported by existing language-learning agents. We present a Wizard-of-Oz (WoZ) study design and supporting real-time system for collecting multimodal behavioral data from native Chinese speakers reacting to social norm violations by German learners. The system features a photorealistic MetaHuman avatar driven by Live Link face capture and MediaPipe upper-body tracking, a wizard console for real-time behavior selection, and synchronized multimodal logging across agent and learner streams. A layered annotation framework, based on psychological theory and covering non-observable socioemotional reactions, norm interpretation, verbal, and observable behavior thereof, and future supervision targets enables the corpus to support training of future automated cultural interpretation and behavior generation models. Four ecologically valid interaction scenarios, developed with cultural and pedagogical experts, provide the methodological and technical foundation for a culturally adapted conversational agent for Chinese language learning.
1 Introduction
Chinese and German communication norms differ, and German learners may struggle to adapt their behavior to Chinese norms. This study aims to collect culturally grounded interaction data and build a culturally adapted Chinese-teaching agent and annotation framework.
1 Introduction
- The study targets differences in Chinese and German communication norms that may make adaptation difficult for German learners of Chinese.Norm violations can affect communication and relationship quality.Source: §1 · pp. 1–2
1.1 Motivation
- Culturally adapted agents are proposed as a low-risk setting for German Mandarin learners, but their development currently lacks grounded data from authentic intercultural interactions.The intended learning benefits include greater awareness of miscommunication tendencies and Chinese social norms.Source: §1.1 · p. 2
1.2 Research Objectives
- The research aims to collect culturally accurate verbal and nonverbal responses from native Chinese speakers during norm violations and use those data to develop a Chinese-teaching agent for German learners.The agent is supported by a WoZ method and a four-layer multimodal annotation framework [7].Source: §1.2 · p. 2
1.3 Contributions
- The proposed setup combines a photorealistic agent, Live Link face capture, MediaPipe upper-body tracking, real-time wizard control, and synchronized multimodal logging.Its four-layer annotation framework covers socioemotional reactions, norm interpretation, verbal and observable behavior, and future supervision targets.Source: §1.3 · p. 2
2 Theoretical Background
The theoretical background connects culturally adapted agents to nonverbal feedback, intercultural norms, psychodynamic constructs, and Wizard-of-Oz data collection. Together, these ideas motivate collecting authentic behavioral data for future agent development.
2.1 Hybrid Situation Aware Agents
- Hybrid situation-aware agents combine machine-learning perception with theory-based reasoning and adapt multimodal behavior to the ongoing interaction [2].The paper frames cultural and application-specific adaptation as important to agent design.Source: §2.1 · p. 2
2.2 Leveraging Mirroring and Marking for Nonverbal Situated Feedback in Cross-Cultural Communication
- Mirroring and marking could help make a participant’s affect available for reflection, but this behavior has not yet been implemented in intercultural settings.Mirroring replays an expression, while marking alters it to frame the emotion; the paper proposes culturally dependent alteration in agent reactions.Source: §2.2 · p. 2
2.3 Intercultural Differences
- Chinese and German traditions shape norms and expression, while cultural responsiveness involves self-awareness, relational awareness, and power-and-context awareness.Prior work reports that nonverbal features vary across cultures and influence engagement.Source: §2.3 · pp. 2–3
2.4 Psychodynamic Foundation
- OPD models recurring relational patterns through stable internal conflicts and a dominant lead affect, offering a theory-grounded way to represent motivation and affect.The paper connects these constructs to modeling conflict escalation and resolution in intercultural interactions.Source: §2.4 · p. 3
2.5 Wizard-of-Oz for Multimodal Data Collection
- WoZ lets a human operator simulate an unautomated system while participants interact naturally, supporting early collection of authentic learner responses [7].Wizard expertise and decision traces can support culturally grounded behavior and future model training [7].Source: §2.5 · p. 3
3 Study Design
The study is designed to elicit and characterize Chinese interlocutors’ responses to social norm violations by German Mandarin learners. It uses expert-developed scenarios, a fixed online session protocol, synchronized multimodal data, and planned analysis for future modeling.
3.1 Research Questions
- The study asks whether the WoZ setup elicits ecologically valid data, which verbal and nonverbal behaviors follow norm violations, and how response patterns vary by violation type.These questions target future automated behavior modeling.Source: §3.1 · p. 3
3.2 Interaction Scenarios
- Four expert-developed scenarios target realistic Chinese-German norm conflicts, with OPD-linked conflicts and annotations for socioemotional response, violation type, and markedness.They proceed from ambiguity to clear violation and end with an affirmative baseline for negative training examples.Source: §3.2 · p. 3
3.3 Participants
- Learners are native German speakers with B1–C1 Mandarin proficiency; two trained wizards must have been socialized in Chinese-speaking environments.The target is at least 20 learner sessions, with final sample size determined by power analysis; a questionnaire assesses prior cultural familiarity without priming.Source: §3.3 · p. 4
3.4 Experimental Protocol
- Sessions follow a fixed sequence of baseline, four scenarios, and intervening feedback, with brief affective and relational self-reports at baseline and after the highest-conflict scenario.Between scenarios, learners complete a debrief questionnaire without being told the cultural norm focus.Source: §3.4 · p. 4
3.5 Data Collection
- Each session logs synchronized agent behavior and wizard actions, learner video, audio and ASR transcript, and session metadata.The 52 ARKit facial blendshape parameters at 60 fps document display and are not used as training data.Source: §3.5 · p. 4
3.6 Planned Analysis
- Planned analyses will describe behaviors by scenario and violation type, assess cross-wizard consistency, and use the annotated corpus to train interpretation and behavior-selection components.Cross-wizard consistency is intended to distinguish individual wizard responses from broadly representative cultural patterns.Source: §3.6 · p. 5
4 System Architecture
The system links remote interaction, avatar presentation, wizard control, and multimodal data capture in a synchronized pipeline. Its real-time tracking and authoring components support wizard-selected culturally grounded behaviors and post-session annotation.
4 System Architecture
- The online architecture connects tracking, study computer, participant interface, and researcher console; the avatar presents wizard-selected behavior, while future modeling uses learner signals and wizard traces rather than rendering data.The learner participates remotely via Microsoft Teams.Source: §4 · p. 5
4.1 Tracking and Avatar Realization
- An iPhone 12 captures 52 facial blendshape parameters at 60 fps, while MediaPipe Pose tracks shoulder sway and forward lean for the avatar.The avatar is rendered in Unreal Engine 5 and streamed through OBS Studio to the learner; audio may be pre-recorded or live text-to-speech.Source: §4.1 · p. 5
4.2 Offline Authoring
- Experts prepare scenario scripts, persona configurations, animation templates grounded in documented Chinese nonverbal behavior, and a cultural knowledge base with OCC-based appraisal labels.Source: §4.2 · p. 5
4.3 WoZ Console and Logging
- The wizard console displays learner video and ASR, triggers behavior templates, and records timestamped actions; shared session timestamps align learner video, transcript, and wizard log for annotation.Communicative intention labels are assigned after the session.Source: §4.3 · p. 5, Figure 3 · p. 5
5 Annotation Framework
The framework separates observable interaction, interpretation and socioemotional reactions, wizard decisions, and session-level learning targets into independent annotation layers. Together, these layers support parallel annotation and future modeling.
5 Annotation Framework
- Independent layers enable parallel annotation; ASR transcripts are stored for post-session annotation, with agreement assessed using Cohen’s κ and intraclass correlation.Source: §5 · pp. 5–6, Figure 4 · p. 6
- Layer A records time-aligned, interpretation-free speech acts and facial, gaze, gesture, and posture labels for both learner and agent.Agent annotations also record the behavior-template identifier and wizard action label.Source: §5 · p. 5
- Layer B labels social obligations, norm adherence and markedness, cultural norm categories, and context-inferred discomfort, face threat, and relational tension.These socioemotional labels bridge observable behavior and the psychodynamic constructs in Layer D.Source: §5 · p. 5
- Layer C records wizard-selected templates and intentions, mirroring type and degree, and inferred conflict state to supervise future behavior-generation policies.Mirroring may be full, marked alteration, or absent; marking may be exaggerated, attenuated, or neutral.Source: §5 · pp. 5–6
- Layer D captures session-level learner repair, affect, interpersonal conflict, and escalation trajectory as supervision targets for future cultural interpretation and behavior generation.Source: §5 · p. 6
6 Discussion and Future Work
The discussion describes the corpus and WoZ design’s intended advantages, then outlines plans for model training, evaluation, and release. It also notes that empirical validation remains future work and identifies constraints of the wizard and online Chinese-German setting.
6 Discussion and Future Work
- The fixed scenario sequence is intended to control contextual variation while preserving behavioral naturalness; layered labels support reuse from norm detection to behavior generation.The framework incorporates OPD-based conflict constructs and mirroring-marking labels for theoretically grounded, explainable agent design.Source: §6 · p. 6
6.1 Training Plan
- The training plan proposes cultural interpretation, dialogue orchestration, and behavior-generation modules that progressively replace the wizard in the runtime pipeline.The modules use multimodal learner signals, wizard decision traces, and mirroring-marking labels as supervision.Source: §6.1 · p. 6
6.2 Evaluation Plan
- Evaluation is planned in two stages: corpus-based assessment of model predictions and action selection, followed by a user study of authenticity, ecological validity, and learner reflection.The user study will compare behavior with and without mirroring-marking, with expert qualitative validation.Source: §6.2 · p. 6
6.3 Reproducibility and Open Science
- The authors plan to release system tools and publish the annotation schema and behavior taxonomy; corpus access will depend on a data-sharing agreement, pseudonymization, ethical approval, and consent.Source: §6.3 · p. 6
6.4 Limitations
- Empirical validation remains future work; two-wizard consistency analysis is intended to mitigate selection subjectivity, while Chinese-German scenarios and online framing constrain scope and nonverbal naturalness.The online setup may particularly affect gaze, interpersonal distance, and gesture range compared with face-to-face interaction.Source: §6.4 · p. 6
7 Ethics
The study protocol has been submitted for ethical review, and participant consent, pseudonymization, secure storage, and post-study disclosure of the wizard role are specified.
7 Ethics
- The protocol has been submitted for institutional ethical review, and participants will give written informed consent for audio-video recording and data use.Source: §7 · pp. 6–7
- Data will be pseudonymized and stored on GDPR-compliant servers; the wizard role will be disclosed during the post-study debriefing, consistent with standard WoZ practice.Source: §7 · p. 7