PaperlayerView exact revision on arXiv

Source-linked AI summary

Molweni: A Challenge Multiparty Dialogues-based Machine Reading Comprehension Dataset with Discourse Structure

Jiaqi Li, Ming Liu, Min-Yen Kan, Zihao Zheng, Zekun Wang, Wenqiang Lei, Ting Liu, Bing Qin

arXiv:2004.05080v3cs.CL

TL;DR

Multiparty-dialog comprehension remains less represented than prose-based MRC, despite the complex discourse structure of multiparty conversations. Molweni builds a large Ubuntu-dialog MRC dataset with discourse annotations; BERT-wwm achieves 67.7% F1 on its questions. The authors conclude that discourse annotations can help taggers understand dialog and produce higher-quality questions.

  • Problem

    Multiparty-dialog MRC is underrepresented, and its complex discourse structure differs from the well-written prose used by most MRC datasets.

    Source: §1 · p. 1

  • Method

    Molweni samples and filters Ubuntu Chat Corpus dialogs, then annotates extractive answerable and unanswerable questions and discourse relations.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    67.7% F1: BERT’s whole word masking model scores on Molweni, while human performance is 80.2% F1.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report that discourse-structure tagging can help annotators understand dialog and raise the quality of proposed questions.

    Source: §5 · p. 9

    The concatenated-utterance input does not highlight speaker information and treats potentially incoherent adjacency as sequence rather than graph structure.

    Source: §4.1 · p. 8

Abstract

from arXiv · show

Research into the area of multiparty dialog has grown considerably over recent years. We present the Molweni dataset, a machine reading comprehension (MRC) dataset with discourse structure built over multiparty dialog. Molweni's source samples from the Ubuntu Chat Corpus, including 10,000 dialogs comprising 88,303 utterances. We annotate 30,066 questions on this corpus, including both answerable and unanswerable questions. Molweni also uniquely contributes discourse dependency annotations in a modified Segmented Discourse Representation Theory (SDRT; Asher et al., 2016) style for all of its multiparty dialogs, contributing large-scale (78,245 annotated discourse relations) data to bear on the task of multiparty dialog discourse parsing. Our experiments show that Molweni is a challenging dataset for current MRC models: BERT-wwm, a current, strong SQuAD 2.0 performer, achieves only 67.7% F1 on Molweni's questions, a 20+% significant drop as compared against its SQuAD 2.0 performance.

1 Introduction

Multiparty dialog makes comprehension difficult because adjacent utterances may be incoherent and discourse relations can link distant turns, while prior MRC work did not address these properties. Molweni combines large-scale multiparty-dialog MRC questions with discourse-relation annotations, and BERT-wwm’s results show the dataset remains challenging.

1 Introduction

  • Prior MRC datasets largely use well-written prose, while earlier dialog MRC resources provide limited multiparty-dialog coverage or scale.The introduction describes DREAM as including only a minute fraction of multiparty dialog and FriendsQA as small-scale.Source: §1 · p. 1
  • Multiparty dialog differs from prose because adjacent utterances may lack coherence, while discourse relations can connect distant utterances.The authors argue that interpreting these dialogs requires understanding their complex structure.Source: §1 · p. 2
  • Molweni provides 30,066 answerable and unanswerable questions over 10,000 Ubuntu Chat Corpus dialogs containing 88,303 utterances.Answerable questions are extractive, and plausible answers are annotated for unanswerable questions.Source: §1 · p. 2
  • 67.7% F1: BERT’s whole word masking model scores below human performance of 80.2% F1 on Molweni.The authors report that both BERT-wwm scores are more than 10% below human performance.Source: §1 · p. 2
  • Molweni adds annotations for 78,245 discourse relations across its dialogs, addressing the limited scale of prior multiparty-dialog discourse data.The authors identify STAC as the only prior dataset for multiparty-dialog discourse parsing and say its 1,000 dialogs disallow data-driven approaches.Source: §1 · p. 2

2 Related work

Prior multiparty-dialog discourse parsing relied on STAC, while existing dialog MRC datasets differ in scale and coverage. Molweni adds a larger multiparty-dialog corpus with discourse annotations and both answerable and unanswerable extractive questions.

Discourse parsing for multiparty dialog

  • Molweni contains 10,000 dialogs, 88,303 utterances, and 78,245 discourse relations, compared with STAC’s 1,091 dialogs, 10,677 utterances, and 11,348 relations.STAC derives from the online game The Settlers of Catan.Source: §2 · p. 3

Machine reading comprehension

  • Molweni is an extractive dialog-MRC dataset with answerable and unanswerable questions, including plausible answers for unanswerable items.The paper distinguishes it from Friends, DREAM, and FriendsQA by its discourse structure annotations and additional multiparty-dialog and unanswerable-question instances.Source: §2 · p. 3

3 The Molweni corpus

Molweni is built from selected Ubuntu Chat Corpus dialogs and combines multiparty-dialog MRC with annotated discourse structure. Its annotation covers answerable and unanswerable questions, discourse dependencies, and checks of annotation quality.

3 The Molweni corpus

  • Molweni contains 10,000 selected dialogs with 88,303 utterances; selection retained dialogs with 8–15 utterances and 2–9 speakers, excluding utterances over 20 words.The corpus was randomly sampled after filtering for these criteria.Source: §3 · p. 3

3.1 Annotation for machine reading comprehension

  • 30,066 questions include extractive answerable questions and unanswerable questions with plausible answers; Molweni is described as the only MRC dataset annotated with discourse structure.Answerable answers are continuous source-dialog spans; annotators also provide plausible answers for unanswerable questions.Source: §3.1 · p. 4

3.2 Annotation for discourse structure of multiparty dialogs

  • Multiparty dialogs are represented as directed acyclic graphs linking utterances; annotation predicts those links and classifies relation senses, using a 16-relation modified SDRT hierarchy whose four most frequent types cover over 80%.Source: §3.2 · p. 5
  • The discourse graph is assumed to be connected, with every utterance having at least one parent among earlier utterances.This assumption means no utterance is treated as isolated.Source: §3.2 · p. 5

3.3 Data Quality

  • Annotators’ final Kappa was 0.56 for both links and relations, close to 0.58 in the original STAC corpus; multi-label relation recognition can create ambiguity because multiple relations may hold between utterances.Annotation quality was checked manually by humans and programmatically.Source: §3.3 · pp. 6–7

4 Experiments

The experiments evaluate machine reading comprehension and discourse parsing on multiparty dialogs, using BERT models and a Deep Sequential model. BERT-wwm performs strongly on SQuAD 2.0 but substantially worse on Molweni, while Molweni training improves discourse-link prediction on STAC.

4.1 Machine reading comprehension for multiparty dialogs

  • The MRC baseline concatenates utterances, each with speaker and text, and evaluates BERT-base, BERT-large, and BERT-wwm using EM and F1.BERT-wwm uses whole-word masking during pretraining.Source: §4.1 · pp. 7–8
  • BERT-wwm achieves 89.1% F1 on SQuAD 2.0, with a 0.3% F1 gap to human performance, but reaches only 67.7% F1 on Molweni, where the gap is 12.5%.Source: §4.1 · p. 8
  • In a ten-utterance, three-speaker example, BERT-wwm gives the wrong answer to both questions, including confusing printer status with the status of people making printers.For the address question, the model predicts likwidoxigen instead of the gold answer nbx909.Source: §4.1 · p. 8, Figure 4 · p. 9
  • Concatenation does not highlight speaker information, although the authors identify speaker identity as essential for understanding dialogs.The authors also note that adjacent utterances may lack coherence and that multiparty discourse is better represented as a graph than a sequence.Source: §4.1 · p. 8

4.2 Discourse parsing for multiparty dialogs

  • The Deep Sequential model achieves higher link-prediction results than on STAC and comparable relation-classification results; combined STAC–Molweni training improves STAC link prediction.Table 8 reports discourse-parsing F1 scores; the combined setting trains on both datasets and tests on each separately.Source: §4.2 · p. 9, Table 8 · p. 9

5 Conclusion

Molweni is introduced as a multiparty-dialog MRC dataset with discourse dependency annotations and questions. The authors report that discourse annotation helps taggers understand dialogs and produce higher-quality questions.

5 Conclusion

  • Molweni combines multiparty-dialog MRC questions with discourse dependency annotations, which the authors report help annotators understand dialogs and produce higher-quality questions.The authors identify future work in multiparty discourse parsing and applying discourse structure to dialog reading comprehension.Source: §5 · p. 9

Related papers

Loading 2004.05080v3…