PaperlayerView exact revision on arXiv

Source-linked AI summary

PragAlign: Evidence-Sensitive Reply Assistance Across Chinese and Japanese Appropriateness Judgments

Xin Zhong, Satori Hachisuka

arXiv:2608.26700v1cs.CL

TL;DR

Reply assistance must handle both linguistic fluency and culturally situated appropriateness without overburdening users with clarification. PragAlign separates evidence-sensitive context reading from selective clarification before fixed reply generation and evaluates it with matched Chinese and Japanese judgments. It performs significantly better than both baselines in Chinese, while Japanese results show a different, nonsignificant pattern and shared preferences in 5 of 10 scenarios.

  • Problem

    Reply assistance can produce socially misaligned wording, while cultural data cannot determine exchange-specific facts and asking about every missing detail creates interaction burden.

  • Method

    PragAlign uses a Context Reader and Gap Policy to structure evidence and ask at most one consequential clarification question before fixed reply generation, evaluated through language-matched judgments.

  • Results

    Chinese PragAlign ranks were significantly better than both baselines; Japanese had a nonsignificant omnibus effect, while matched groups selected the same top condition in 5 of 10 scenarios, including four shared PragAlign selections.

  • Takeaways & Limitations

    The findings identify shared and language-specific judgment patterns and provide an empirical foundation for culturally and linguistically informed reply assistance.

  • Takeaways & Limitations

    Synthetic data do not replace natural interactions or human labels, and unequal groups—nine Chinese versus three Japanese participants—reduce between-group precision.

Abstract

from arXiv · show

Reply assistance in multilingual settings requires linguistic competence and culturally situated judgments of appropriateness. We present PragAlign, which separates context reading from selective clarification, and evaluate it alongside Direct and Rule. Nine native Chinese speakers judged Chinese materials, while three native Japanese speakers judged matched Japanese versions. In the Chinese evaluation, PragAlign received significantly better ranks than both baselines. In the Japanese evaluation, Direct had the lowest mean rank, PragAlign had the highest top-rank rate, and the omnibus difference was not significant. The groups selected the same top condition in 5 of 10 scenarios, including four shared PragAlign selections. The results identify shared and language-specific judgment patterns and inform reply assistance designed to support linguistic and cultural understanding.

I. INTRODUCTION

PragAlign addresses socially situated reply appropriateness by structuring context and asking at most one consequential clarification question. The study compares it with Direct and Rule using language-matched Chinese and Japanese judgments.

  • Motivation: Fluent wording can still sound evasive, too direct, overly specific, or socially misaligned in professional and collaborative exchanges.Cross-cultural pragmatic failure occurs when intended meaning and social interpretation diverge.
  • Motivation: Culturally grounded data improve norm coverage but cannot establish exchange-specific facts such as audience privacy or deadline flexibility.Asking about every missing detail creates interaction burden.
  • Approach: PragAlign uses a Context Reader and Gap Policy to structure evidence and ask at most one question when its answer could materially change the reply.It is evaluated alongside Direct and Rule in matched own-language judgments.
  • Evaluation: Nine native Chinese speakers judged Chinese materials, while native Japanese speakers judged matched Japanese versions.The evaluation measures language-matched appropriateness judgments rather than nonnative reply-composition accuracy.
  • Research questions: The study asks how Direct, Rule, and PragAlign rank within each language version and where the two groups converge or diverge across matched scenarios.It also examines judgment criteria behind shared and different preferences.
  • Contributions: The paper contributes an evidence-sensitive clarification layer, controlled pragmatic scenarios, and a two-language ranking analysis of matched cases.The same ranking analysis is applied to both groups.

II. BACKGROUND AND RELATED WORK

PragAlign is grounded in pragmatic theories linking language choice to social relations and in controlled resources for cross-cultural, bilingual, and formality-sensitive analysis. It adapts clarification research by asking only when an unknown could change the reply.

  • Pragmatic foundations: Pragmatic competence selects language that fits a social situation involving face, distance, power, imposition, rights, obligations, and expectations.These accounts motivate variables including relationship, channel, audience, responsibility, factual commitment, and urgency.
  • Related resources: CulturePark, NormDial, and GYAFC exemplify controlled resources for cultural understanding, comparable bilingual dialogues, and formality-sensitive rewriting.PragAlign follows this controlled-data tradition with literature-grounded scenario dimensions and public-versus-withheld information.
  • Clarification research: Clarification research separates when to ask, what to ask, and how to use the answer.PragAlign asks only when an unknown field could change content, tone, responsibility, channel, or audience.

III. PROBLEM FORMULATION

PragAlign formalizes reply assistance as evidence-sensitive context reading followed by selective clarification and fixed generation. Unknown fields remain explicit so unsupported assumptions do not become confident reply content.

  • III. PROBLEM FORMULATION: The Context Reader maps public input x to an evidence-tagged frame z.Public input includes the user request, incoming message, and explicitly supplied scenario context.
  • III. PROBLEM FORMULATION: The Gap Policy selects PROCEED or ASK and produces one clarification question when needed.The question is empty when the action is PROCEED.
  • III. PROBLEM FORMULATION: A fixed generator produces the final reply directly from x and z, or from x, z, the question, and the user answer v.The notation distinguishes the generation inputs for PROCEED and ASK.
  • III. PROBLEM FORMULATION: Each context field is labeled OBSERVED, INFERRED, or UNKNOWN, and a consequential gap is an unknown whose resolution could change the reply.Preserving UNKNOWN prevents unmentioned relationships, audiences, attitudes, or cultural expectations from becoming assumptions.
  • IV. PRAGALIGN METHOD: PragAlign is a pre-generation decision layer rather than a replacement generator.Direct forwards public input, Rule adds a generic relationship/channel/tone instruction, and PragAlign supplies a case-specific frame with optional clarification.
  • IV. PRAGALIGN METHOD: The decision layer targets one unknown field across gap types such as facts, audience, permission, channel, urgency, recipient goal, and reply language.Its Context Reader emits a compact frame and its Gap Policy predicts PROCEED or ASK.

V. DATA CONSTRUCTION AND MODEL TRAINING

The study constructs controlled pragmatic vignettes and matched evidence-availability counterfactuals, then trains view-specific decision modules while keeping the final generator fixed.

  • Data construction: Each controlled vignette specifies a request, incoming message, role, relationship, channel, audience, factual constraints, and reply language.The variables follow politeness, rapport, and controlled social-norm and style resources.
  • Data construction: Each accepted root yields a matched PROCEED case exposing consequential information and an ASK case withholding one decisive variable.Other material is held constant so supervision depends on evidence availability rather than topic or length.
  • Data construction: Roots are filtered for grounding, public-versus-withheld separation, language consistency, and one answerable gap.The resulting synthetic cases provide auditable supervision, not an estimate of naturally occurring communication.
  • Model training: View-specific tasks supervise the Context Reader with evidence-tagged frames and the Gap Policy with an action, gap type, and one clarification question.PROCEED cases and multiple gap types discourage an always-ask policy.
  • Model training: The Context Reader and Gap Policy use train/validation/test splits of 1,520/305/311 and 1,627/343/368 examples, respectively.Both modules initialize from Qwen3-8B and use parameter-efficient adaptation.
  • Model training: The shared final generator is not fine-tuned.This keeps the evaluated generation component fixed while the decision modules are adapted.

C. Diagnostic Scope

The study uses matched Chinese and Japanese scenarios, identical ranking procedures, and controlled reply-generation paths, with diagnostics supplementing human evaluation.

  • Diagnostics: The primary evidence comes from human judgments of final replies, while held-out counterfactual diagnostics test structured outputs and PROCEED/ASK discrimination.The diagnostics address model behavior, whereas human evaluation remains the main evidence source.
  • Scenario design: 10 matched scenario specifications included five simple and five complex cases, with complexity defined by decision density rather than text length.Simple cases had two or three visible constraints; complex cases had at least five spanning relationship, audience, responsibility, factual commitment, channel, and urgency.
  • Evaluation procedure: 90 Chinese and 30 Japanese participant–case blocks were judged by native speakers who ranked three anonymized replies from 1 to 3.The task structure and analysis were identical across language versions, which were judged in the participants’ own languages.
  • Analysis: Friedman tests, bootstrap confidence intervals, rank rates, Kendall’s W, and corrected Wilcoxon tests quantify condition-level differences.Cross-language comparison uses matched cases rather than pooled participant–case blocks.
  • Experimental controls: Conditions shared scenario facts and a fixed generator; only the pre-generation path differed, with labels hidden and reply order randomized.Both groups used the same summary measures and omnibus test.

D. Overall Two-Language Pattern

Across languages, PragAlign performs best in the Chinese evaluation, while Japanese results show a different rank pattern and no significant omnibus effect. Matched scenarios converge on the same winner in half the cases.

  • Aggregate pattern: With equal language weights, PragAlign had mean rank 1.79, top-rank rate 45.6%, and worst-rank rate 24.4%; Direct and Rule had mean ranks 2.09 and 2.12.This is an equal-weight summary, not a pooled inferential test.
  • Chinese results: Chinese PragAlign had mean rank 1.61 (95% CI [1.47, 1.76]) and was ranked first in 51% of blocks.The overall Chinese condition difference was significant: χ2(2) = 22.87, p < .001, Kendall’s W = 0.13.
  • Chinese results: PragAlign beat Direct and Rule in Chinese, winning 70 of 90 paired comparisons against Direct and 55 against Rule.Holm-corrected tests favored PragAlign over Direct (p < .001) and Rule (p = .002).
  • Chinese results: The three Chinese conditions differed for both simple cases (p = .002) and complex cases (p = .003).The advantage therefore persisted across both scenario-complexity strata.
  • Participant and scenario patterns: PragAlign had the lowest mean rank for eight of nine Chinese participants and won seven of ten scenarios.Some exceptions favored brevity or clearer boundaries rather than longer replies.
  • Matched convergence: The groups selected the same winner in 5 of 10 matched scenarios, including four shared PragAlign wins.Agreement covered three simple and two complex cases.

F. Japanese-Language and Matched-Case Patterns

Matched Chinese and Japanese scenarios produced the same winning condition in half of cases, including four shared PragAlign selections, while divergence centered on brevity, boundaries, and explanation length.

  • Matched-case convergence: 5 of 10 matched scenarios had the same winner across language groups, with four shared winners being PragAlign.Agreement occurred in three simple and two complex cases, and shared PragAlign wins spanned both complexity levels.
  • Shared and divergent criteria: Agreement centered on factual handling and commitments, whereas divergence concerned brevity, boundaries, and explanation length.Because each group judged its own-language version, this is scenario-level convergence rather than inter-rater reliability.

G. Qualitative Explanations

The explanations associate favorable PragAlign judgments with politeness, clarity, actionable commitments, and social fit, while unfavorable judgments emphasize directness, excess specificity, length, and detail.

  • The condition-level table defines lower mean and worst ranks as better, and higher top-rank rates as better.
  • Japanese favorable judgments centered on politeness, while the few unfavorable mentions concerned clarity and directness.
  • Favorable Chinese judgments emphasized clarity, politeness, actionable commitments, and social fit, whereas unfavorable judgments clustered around directness, excess specificity, length, and detail.The coded explanations indicate that PragAlign’s benefit is not reducible to reply length, although over-elaboration remains a failure mode.

VII. DISCUSSION

The discussion interprets PragAlign as an evidence-sensitive decision layer whose Chinese advantages coexist with nonsignificant Japanese omnibus results and language-dependent preferences in directness and detail.

  • VII. DISCUSSION: Chinese rankings favored PragAlign over both baselines, whereas Japanese results showed a nonsignificant omnibus effect and different patterns for mean rank and top-rank rate.The equal-language aggregate is an equal-weight summary, not a pooled test.
  • VII. DISCUSSION: Agreement occurred in 5 of 10 matched cases, including four shared PragAlign selections, while divergence concerned directness, boundaries, and explanation length.
  • VII. DISCUSSION: PragAlign explicitly distinguishes observed, inferred, unknown, and consequential fields before deciding whether to proceed or ask one question.This distinction addresses a limitation of checklist-based Rule prompts, which cannot determine whether a field is consequential.
  • VII. DISCUSSION: The decision layer controls evidence and clarification, while the generator realizes that decision through language-specific brevity, politeness, and explicitness.Rankings measure relative appropriateness, while explanations diagnose grounded commitments and over-elaboration.

C. Implications for Additional Experiments

The paper motivates additional experiments using balanced groups and a shared language to separate language expression from cultural background and strengthen reply-composition comparisons.

  • C. Implications for Additional Experiments: A separate shared-language design is needed to evaluate non-native-language reply composition and better separate language expression from cultural background.
  • C. Implications for Additional Experiments: The study’s synthetic data, unequal groups, exchangeability assumption, and own-language evaluations constrain how precisely its Japanese and between-group findings can be interpreted.The Japanese per-participant and per-theme counts are descriptive rather than precise with three participants.
  • C. Implications for Additional Experiments: A preregistered replication should balance groups, use a shared language, independently annotate clarification quality, and test multiple generators.
  • C. Implications for Additional Experiments: The findings provide an empirical foundation for culturally and linguistically informed reply assistance while motivating a balanced, shared-language follow-up study.
Loading 2608.26700v1…