PaperlayerView exact revision on arXiv

Source-linked AI summary

Japanese Discourse Structure Analysis Based on Segmented Discourse Representation Theory Using LLMs

Satomu Tomisaki, Yosuke Mikami, Amane Watahiki, Hitomi Yanaka

arXiv:2610.36667v1cs.CL

TL;DR

Japanese SDRT parsing has been difficult to study because Japanese SDRT corpora were unavailable and prior work largely focused on English. The paper translates English SDRT corpora into Japanese and evaluates prompt-based and fine-tuned parsers. It reports comparable English and Japanese performance and finds that predictions rely heavily on contextual discourse relations.

  • Problem

    Japanese SDRT parsing remained unclear because resources and parsing research predominantly focused on English, and no Japanese SDRT corpus was available.

    Source: §1 · pp. 1–2

  • Method

    The authors automatically translate English SDRT corpora into Japanese and compare fine-tuned and prompt-based parsers across Japanese and English.

    Source: §1 · p. 2

  • Results

    Fine-tuned parsers outperform prompt-based parsers and perform comparably across English and Japanese.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The findings suggest cross-linguistic generalizability to Japanese, while the cue demo (でも; “but”) can facilitate prediction of particular discourse relations.

    Source: §6 · p. 9

    The Japanese corpora are machine translations of gaming-session chat logs, were not fully manually evaluated, and may differ from native Japanese, potentially confounding parser performance.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Discourse parsing based on Segmented Discourse Representation Theory (SDRT) is crucial for capturing the structural dynamics of dialogues. However, most prior research focuses on English, and its effectiveness for Japanese remains unclear owing to the lack of Japanese SDRT corpora. To address this issue, we construct Japanese SDRT corpora by translating existing English SDRT corpora using an LLM. Through a comparative evaluation of prompt-based and fine-tuned LLMs, we show that the models achieve comparable performance between English and Japanese, suggesting the cross-lingual applicability of SDRT-based discourse parsing. In order to investigate why the English and Japanese parsers achieve similar performance, we conduct ablation analyses on discourse markers and contextual inputs. Our results reveal that most of the Japanese discourse markers we examined have little impact, and the models' predictions depend heavily on contextual discourse relations rather than on the content of the context clauses. This structural reliance might explain the comparable performance despite the substantial typological differences between English and Japanese.

1 Introduction

The paper addresses the lack of Japanese SDRT corpora by translating English corpora and evaluating Japanese and English parsers. It finds comparable performance across languages and examines which cues the models use.

1 Introduction

  • Because Japanese SDRT corpora were unavailable, it was difficult to train Japanese parsers or assess how English methods generalize to Japanese.The paper notes that Japanese and English differ substantially in lexicon and syntax, and may use different relation cues.Source: §1 · p. 1
  • The study fills the Japanese SDRT resource gap by automatically translating existing English corpora.The translated corpora are used to study cross-lingual generalizability.Source: §1 · pp. 1–2
  • The authors compare fine-tuned and prompt-based parsers on the translated corpora and evaluate the best-performing Japanese and English models.Source: §1 · p. 2
  • Three ablations examine Japanese connectives and sentence-final particles, previous discourse relations, and context-clause content.Most examined markers contribute little, but demo (でも; “but”) substantially affects Con- trast relation identification; predictions primarily depend on prior discourse relations rather than context content.Source: §1 · p. 2
  • Japanese and English SDRT parsers perform comparably, while the models rely heavily on contextual discourse relations rather than context-clause content.The authors suggest that shared discourse relations in the translated corpora may explain the small performance gap.Source: §1 · p. 2

2 Background

The background introduces SDRT’s discourse-graph representation, summarizes prompt-based and fine-tuned parsing approaches, and describes the English corpora used in the study.

2.1 Discourse Analysis Framework

  • SDRT represents discourse as a directed acyclic graph whose nodes are elementary or complex units and whose edges encode discourse relations.EDUs are clauses, EEUs are salient nonlinguistic actions or events, and CDUs combine multiple elementary units.Source: §2.1 · p. 2
  • SDRT is grounded in formal semantics and, unlike RST as described here, supports analysis of multiparty conversations.Source: §2.1 · p. 2

2.2 SDRT parsing with LLMs

  • Prompt-based SDRT parsers have left room for improvement, while fine-tuned approaches have achieved higher performance; Llamipa is a simpler single-module alternative with competitive performance.The authors select Llamipa as their primary fine-tuned parser for its simplicity and efficiency.Source: §2.2 · pp. 2–3
  • Llamipa uses a target clause, preceding clauses within a fixed window, and the discourse relations among those context clauses to predict links and relations to the target.It processes clauses incrementally, using either gold or predicted relations to update the context relation set.Source: §2.2 · p. 3

2.3 SDRT Corpus

  • The study uses MSDC, an SDRT-annotated Minecraft dialogue corpus, and STAC, an SDRT-annotated corpus of Settlers of Catan chat and in-game events.MSDC includes player chat and descriptions of game actions; STAC includes chat and game information such as dice rolls and card plays.Source: §2.3 · p. 3

3 Japanese SDRT Corpus

The paper constructs Japanese SDRT corpora by translating MSDC and STAC, then assesses translation quality through manual and automatic evaluation. The corpora retain the source datasets’ statistics, while the manual evaluation covers a sample of Japanese MSDC test sessions.

3.1 Corpus Construction

  • The corpora were translated clause by clause with dialogue context to preserve discourse-unit order and support natural, context-aware translations.Because GPT-5.1 sometimes translated MSDC EEUs inconsistently, the authors used keyword-based rules for all EEUs; STAC followed prior methodology with simplifications.Source: §3.1 · p. 3

3.2 Dataset Statistics

  • MSDC’s training and test sets contain 288 and 97 dialogues, while STAC’s contain 1,017 and 113; their most frequent relations differ.MSDC’s most frequent relation is Result, whereas STAC’s is Question-answer pair; the English and Japanese versions have identical statistics.Source: §3.2 · pp. 3–4, §3.3 · p. 4

3.3 Quality Analysis

  • Manual quality checks covered 20 of the 97 Japanese MSDC test sessions, and annotator agreement was low (Krippendorff’s α = 0.132).One native Japanese-speaking author checked translation naturalness and label consistency; three native Japanese-speaking authors scored translation quality by speaker turn.Source: §3.3 · p. 4
  • Table 1 reports mean translation-evaluation scores and standard errors for each annotator.The paper notes that limited agreement is a known challenge in human evaluation of machine translation.Source: Table 1 · p. 4, §3.3 · p. 4
  • The Japanese MSDC translation received XCOMET-XL scores of 0.84 for training and 0.85 for test.The metric ranges from 0.0 to 1.0, with higher scores indicating better translation quality.Source: §3.3 · p. 4

4 Japanese SDRT Parsers

The authors evaluate fine-tuned and prompt-based Japanese SDRT parsers, then compare the best-performing parser across English and Japanese corpora. Llamipa’s English and Japanese performance is comparable, while on Japanese MSDC it has a higher weighted average F1-score than the prompt-based parsers.

4.1 Experimental Settings

  • The study fine-tunes Llama-3-8B models for Japanese and English, compares them with zero-, one-, and four-shot prompts, and evaluates weighted average F1.Prompt-based experiments use GPT-5.5 and Llama-3.1-8B-Instruct in both languages, plus a Japanese Swallow variant; the authors use newer LLMs for prompting because Llama-3’s context window is too short for these prompts.Source: §4.1 · p. 4

4.2 Results

  • On Japanese MSDC, Llamipa’s weighted average F1-score is around 0.79, while prompt-based parsers do not exceed 0.72.Llamipa is numerically highest overall; Table 2 reports relation-level F1-scores and support counts for the tested parsers.Source: §4.2 · p. 4, Table 2 · p. 5
  • Llamipa outperforms prompt-based parsers on most relations, but prompting performs better for Conditional and Explanation, which have few training instances.The authors suggest that sparsity may have hindered Llamipa’s fine-tuning on those two relations.Source: §4.2 · pp. 4–5
  • More few-shot examples improve GPT-5.5’s performance, but Llama-3.1 and Swallow-3.1 do not show the same trend.For Llama-3.1, weighted average F1 is 0.24 zero-shot, 0.32 one-shot, and 0.33 four-shot; for Swallow-3.1 it is 0.22, 0.30, and 0.09, respectively.Source: §4.2 · p. 5
  • The authors attribute the weaker gains from more exemplars for Llama-3.1 and Swallow-3.1 likely to input length and limited context capacity.The four-shot prompt exceeds the publicly released Swallow-3.1 checkpoint’s 8,192-token maximum sequence length, though inputs were not truncated.Source: §4.2 · p. 5
  • Across MSDC and STAC, Llamipa’s weighted average F1-scores are comparable between the original English and translated Japanese corpora.Across parsers, scores are around 0.79 for MSDC and 0.60 for STAC; the English Llama-3-based model’s STAC score is below Thompson et al. (2024a)’s reported 0.78, possibly reflecting setup differences.Source: §4.2 · p. 5

5 Ablation Tests

Ablations show that parser predictions rely strongly on contextual discourse relations, while the content of context clauses and most tested Japanese markers contribute little. The strong effect of demo on Contrast is a notable exception.

5.1 Settings

  • The ablations remove target expressions, contextual discourse relations, or context-clause content while using Llamipa on the MSDC test set with gold structure to keep contextual relations fixed.The study includes Llama-3-based parsers for English and Japanese and a Llama-3-Swallow-based parser for Japanese.Source: §5 · p. 6, §5.2 · p. 7, §5.1 · p. 7

5.2 Results

  • Removing demo lowers Contrast F1 from 0.73 to 0.11 for the Llama-3-based model and from 0.77 to 0.14 for the Llama-3-Swallow-based model.Only 2 of 46 and 2 of 48 original Contrast predictions, respectively, remained unchanged after ablation.Source: §5.2 · p. 7, Table 5 · p. 7
  • Removing sorekara changes some Narration predictions, but Narration F1 remains 0.89 →0.90 for both models; other tested connectives and sentence-final particles show no substantial performance changes.The unchanged overall sorekara scores may reflect that affected instances form only a small fraction of test-set Narration relations.Source: §5.2 · pp. 7–8
  • Removing contextual discourse relations lowers weighted average F1 by at least 0.29 for all parsers, with Narration dropping by at least 0.85.The decrease for Contrast in the Japanese models is relatively small, consistent with demo remaining available in the target clause.Source: Table 6 · p. 8, §5.2 · p. 8
  • Masking context-clause content causes only a slight decline in weighted average F1, indicating greater reliance on discourse relations than clause content.This pattern may explain comparable English and Japanese performance because the translated corpora share discourse structure.Source: Table 7 · p. 8, §5.2 · p. 8

6 Conclusion

The paper evaluates Japanese SDRT parsing using corpora translated from English and compares fine-tuned and prompt-based approaches across languages. Fine-tuned parsers outperform prompt-based parsers and perform comparably in English and Japanese; ablations point to contextual discourse relations as a key shared signal.

6 Conclusion

  • Fine-tuned parsers achieve higher SDRT parsing performance than prompt-based parsers and perform comparably across English and Japanese.The Japanese corpora were constructed by automatically translating the English MSDC and STAC corpora.Source: §6 · p. 8
  • Ablations indicate that models rely heavily on input discourse relations, which may explain the small performance difference between English and Japanese parsers.Among the examined Japanese expressions, demo particularly influences Contrast prediction, while the other expressions have less impact.Source: §6 · pp. 8–9
  • The findings suggest cross-linguistic generalizability to Japanese, while further work is proposed to test languages beyond English and Japanese and analyze how LLMs process discourse structure.The paper also proposes finer-grained analyses of which linguistic expressions provide useful cues.Source: §6 · p. 9

Limitations

The evaluation uses only MSDC and STAC gaming-session chat logs, and the Japanese corpora were not fully manually evaluated. Machine-translated Japanese may differ from native usage and introduce confounding factors.

Limitations

  • The study uses only gaming-session chat logs from MSDC and STAC, and the Japanese translations may be unnatural relative to native Japanese, potentially confounding parser performance.The authors identify expansion to other domains and genres and construction of a native Japanese SDRT corpus as future work.Source: Limitations · p. 9

Ethical Statement

The paper limits release of the translated corpus to research use and describes the discourse relations used in MSDC and STAC. Relation instances are indexed directed pairs, with the first argument as source and the second as target.

Ethical Statement

  • The translated corpus is restricted to academic and research purposes, with distribution governed by source-dataset licenses and copyright restrictions.The authors state that licensing is intended to respect the original datasets’ terms and their authors’ intellectual property.Source: Ethical Statement · p. 9

A Semantics of Discourse Relations used in MSDC and STAC

  • Discourse-relation instances are labeled directed pairs of indices, where x is the source and y the target in the discourse graph.The section provides definitions and illustrative examples for relations used in MSDC and STAC.Source: A Semantics of Discourse Relations… · p. 11
  • Other listed relations distinguish discourse functions such as clarification questions, questions seeking elaboration, and questions paired with answers.Clarification-Q asks about x when a speaker does not follow it; Q-Elaboration seeks more information about x; Question-answer pair links a question to its answer.Source: A Semantics of Discourse Relations… · pp. 11–12
  • Conditional links a hypothesis in x to its consequence in y, while Result describes x’s main eventuality as causing y’s eventuality.These are distinct relation definitions in the inventory.Source: A Semantics of Discourse Relations… · pp. 11–12

B Translation Examples and Error Analysis for MSDC

The MSDC translation examples expose errors in preserving interrogative intent, spatial meaning, and natural phrasing.

B Translation Examples and Error Analysis for MSDC

  • One translation changes a Builder’s intended question about whether any block color is allowed into an assertion.The proposed alternative is “どんな色でもいい？” (“Is any color fine?”), rather than “どんな色でも いいよ” (“Any color is fine.”).Source: B Translation Examples and Error… · p. 12
  • A translation of “on” gives the wrong placement instruction: the intended location is to the left of the topmost block.The passage identifies “一番上のオレンジのブロックに” (“on the top orange block”) as the correct translation.Source: B Translation Examples and Error… · p. 12
  • A final example redundantly repeats “上と下に” (“on top and bottom”) in the second clause, which should be removed for a more natural translation.Source: B Translation Examples and Error… · p. 12

C Detailed Experimental Results

The appendix identifies detailed parser-performance tables and describes simplifications made to STAC for parsing.

C Detailed Experimental Results

  • Tables 11 and 12 report precision, recall, and F1-score for constructed parsers on MSDC and STAC, while Table 13 covers prompt-based methods on MSDC in English and Japanese.Source: C Detailed Experimental Results · p. 12

D Simplification of STAC

  • STAC was simplified by flattening CDUs into constituent units, merging consecutive EEUs, and removing EEUs without relations to EDUs.These changes follow Thompson et al. (2024a) because Llamipa cannot directly handle CDUs.Source: D Simplification of STAC · pp. 12–13

E Evaluation on All Relations of MSDC

The all-relations MSDC evaluation restores relations excluded from the main evaluation because their distance exceeds the parser’s context window.

E Evaluation on All Relations of MSDC

F LLM-translation inconsistencies for EEUs in MSDC

The appendix describes inconsistencies in LLM translations of English EEUs and a rule-based approach for translating them consistently. It also provides the translation and parsing prompts and training hyperparameters, alongside captions for parser-performance tables.

F LLM-translation inconsistencies for EEUs in MSDC

  • Direct GPT-5.1 translation of EEUs can leave them untranslated, translate them into Japanese, or produce ungrammatical text, prompting a rule-based translation approach.The approach maps “place” to “設置,” “pick” to “採取,” and color words to Japanese equivalents.Source: F LLM-translation inconsistencies for… · p. 13

G The prompt used for the translation

  • Translation prompts for MSDC and STAC provide dialogue context and a target clause, with instructions for natural Japanese output and corpus-specific handling.The MSDC prompt says to keep in-game action commands in English; the STAC prompt specifies conversational player speech and concise system messages.Source: G The prompt used for the translation · pp. 13–14

H Prompts for Prompt-Based Discourse Parsing

  • The prompt-based parsing instructions represent dialogue structure as a directed acyclic graph of utterances or actions connected by 16 named relations.The requested output encodes each relation as [index1, index2, 'relation'].Source: H Prompts for Prompt-Based Discourse… · pp. 14–15

I Hyperparameters

  • The appendix includes a training-hyperparameter table and performance tables for constructed parsers, STAC, and two MSDC evaluation settings.The MSDC captions distinguish evaluation on a subset of gold relations from evaluation across all gold relations; the captions do not provide the tables’ cell values.Source: I Hyperparameters · p. 15, Table 10 · p. 15, Table 11 · p. 16
Loading 2610.36667v1…