Source-linked AI summary
doc2dial: A Goal-Oriented Document-Grounded Dialogue Dataset
Song Feng, Hui Wan, Chulaka Gunasekara, Siva Sankalp Patel, Sachindra Joshi, Luis A. Lastras
TL;DR
Document-grounded dialogue must handle complex scenes and inter-relations between conversations and documents in information-seeking tasks. doc2dial constructs dialogue flows from document relations and uses crowd contributors to write utterances; its evaluations find user-grounding performance generally improves with more dialogue context, while agent-response generation remains challenging.
Problem
Document-grounded dialogue involves complex real-world scenes requiring better understanding of inter-relations between conversations and documents for information seeking.
Source: §1 · p. 1Method
The authors derive dialogue flows from document structure and semantic relations between discourse units, then present the flows to crowd contributors to create utterances.
Source: §1 · p. 1Results
User-grounding performance generally improves with added dialogue context, while agent-response generation remains challenging even when grounding spans are given.
Source: §3.1.1 · p. 6, §3.2.2 · p. 7Takeaways & Limitations
The authors present doc2dial as a resource intended to support dialogue and document modeling for goal-oriented dialogue tasks in practice.
Source: §5 · p. 8Contributors may adjust their writing toward less desirable dialogue scenes rather than reject those turns, despite receiving the same pay either way.
Source: §2.3 · p. 5
Abstract
from arXiv · showhide
We introduce doc2dial, a new dataset of goal-oriented dialogues that are grounded in the associated documents. Inspired by how the authors compose documents for guiding end users, we first construct dialogue flows based on the content elements that corresponds to higher-level relations across text sections as well as lower-level relations between discourse units within a section. Then we present these dialogue flows to crowd contributors to create conversational utterances. The dataset includes about 4800 annotated conversations with an average of 14 turns that are grounded in over 480 documents from four domains. Compared to the prior document-grounded dialogue datasets, this dataset covers a variety of dialogue scenes in information-seeking conversations. For evaluating the versatility of the dataset, we introduce multiple dialogue modeling tasks and present baseline approaches.
1 Introduction
doc2dial targets information-seeking dialogue that must account for complex relations between conversations and documents. It builds dialogue flows from document structure and discourse relations, then uses them to collect goal-oriented dialogues and evaluate modeling tasks.
1 Introduction
- The work addresses information-seeking dialogue's complex inter-relations between conversations and documents, beyond the scenes covered by earlier document-grounded tasks.The authors identify creating training instances that better approach real-world applications as their aim.Source: §1 · p. 1
- The dataset represents dialogue-based context spanning single or multiple turns and document-based context including headers, list items, and discourse units.Queries may be initiated by a user or an agent and carried across turns by both roles.Source: §1 · p. 1, Figure 1 · p. 2
- Dialogue flows combine higher-level relations between document components with lower-level semantic relations between discourse units to guide crowd contributors' utterances.Each turn specifies a dialogue act, role, and grounding content; the approach is intended to avoid noise from post-hoc human annotation.Source: §1 · pp. 1–2
- The dataset contains about 4,800 annotated conversations averaging 14 turns, grounded in over 480 documents from four domains.Its dialogues use a wider span of document context than prior work based on short text snippets.Source: §1 · p. 2
- The authors propose tasks for user utterance understanding, agent response generation, and relevant document identification, with baseline approaches and evaluation results.The stated goal is to encourage research on dialogue models that incorporate deeper contexts for goal-oriented information seeking.Source: §1 · p. 2
2 Doc2Dial
doc2dial pairs conversations between an assisting agent and an end user with their grounding documents, covering multi-part inquiries and repeated verification. Its construction pipeline turns document structure into dialogue flows and crowdsourced utterances, with annotated dialogue scenes and options for recomposing data.
2 Doc2Dial
- doc2dial links agent–user conversations to associated documents and includes multi-part inquiries and repeated verification of conditional contexts.The dialogues also include free-form utterances and varied document-based and dialogue-based contexts.Source: §2 · p. 2
2.1 Data Collection
- The collection pipeline processes documents, generates dynamic dialogue flows, then presents those flows to crowdsourced contributors for utterance writing.Source: §2.1 · p. 3, §2.1.1 · p. 3
2.1.1 Data Construction Approach
- Document spans are connected in a semantic graph; dialogue flows vary role, grounding span, and dialogue act while respecting graph relations and dialogue history.Candidate spans are updated to avoid reselecting spans whose status is established.Source: §2.1.1 · p. 3
2.1.2 Crowdsourcing Setup
- About 70 qualified contributors wrote both roles and completed whole dialogues, a setting found more coherent and less time-consuming than the alternative.Each contributor focused on one turn at a time while reviewing its scene and dialogue history.Source: §2.1.2 · p. 3
2.2 Document Data
- Documents from four government-service domains include structured content elements, but complex lists, tables, and multimodal webpage content remain unexplored.The extracted elements include tokens, spans, paragraphs, and titled sections.Source: §2.2 · pp. 3–4
2.3 Dialogue Data
- The dataset contains close to 4,800 conversations averaging 14 turns, each annotated with role, dialogue act, and grounding span; sampled flows show grounding coverage across document regions, denser near the beginning.Among 700 sampled flows, annotators rejected about 4% of turns; 70% of rejected turns involved difficulty interpreting a span as conditional context.Source: §2.3 · pp. 4–5, Table 3 · p. 5
2.4 Data Recomposition
- The pipeline supports reshaping dialogues by removing or recollecting rejected turns, inserting irrelevant sub-dialogues, or combining sub-dialogues grounded in different documents.Source: §2.4 · p. 5
3 Tasks and Baselines
The section evaluates doc2dial through user-utterance grounding, agent-response grounding and generation, and relevant-document identification. Across tasks, the reported results highlight the role of dialogue context and the challenges of grounding and generating agent responses.
3 Tasks and Baselines
- The three evaluation tasks cover user-utterance understanding, agent-response prediction, and relevant-document identification; dialogues are split 70/15/15, with half of dev and test grounded in unseen documents.The section also describes relevant-document identification using dialogue context and a pool of documents from all four domains.Source: §3 · p. 5, §3.3 · p. 7
3.1.1 User Utterance Grounding
- User-utterance understanding predicts a document grounding span from the user utterance, dialogue history, and document content, with turns that depend on history or the current utterance.The input can include the last two turns or all previous turns, in chronological or reverse order.Source: §3.1.1 · pp. 5–6
- User-utterance grounding generally improves as more information is added to dialogue context, indicating that these queries are highly conversational and contextual.With irrelevant turns included, identification of those turns reaches 92.1 with all previous turns in reverse order; overall scores are comparable with and without irrelevant turns.Source: §3.1.1 · p. 6, Table 4 · p. 6
3.2.1 Agent Response Grounding Prediction
- Agent-response grounding predicts the document span for the next agent response from dialogue and document context, distinguishing it from grounding what the user has already said.The authors formulate the prediction as span selection and use the same baseline approach and metrics as for user-utterance grounding.Source: §3.2.1 · pp. 6–7
- Agent-response grounding scores are generally much lower than user-utterance grounding scores, which the authors attribute to the task’s challenging nature.The authors identify document-guided dialogue management as a direction for improving this task.Source: §3.2.1 · p. 7, Table 5 · p. 7
3.2.2 Agent Response Generation
- For agent-response generation, BLEU scores are comparable with all prior turns or only the last two, while adding dialogue acts lowers performance.The authors suggest that human-written follow-up questions may differ from questions in pretrained models and conclude that generation remains challenging even when the grounding span is supplied.Source: §3.2.2 · p. 7, Table 6 · p. 7
3.3 Relevant Document Identification
- With more dialogue turns, the BERT-based approach performs better at identifying the relevant document, and the authors report stronger grounding to that document.The evaluation uses recall at k; Table 7 reports the relevant-document identification results.Source: §3.3 · p. 8, Table 7 · p. 8
4 Related Work
Related work addresses document-grounded conversational question answering, dialogue, and search. The paper distinguishes doc2dial by its document scope, finer grounding, and broader dialogue forms.
Related work
- Conversational QA datasets such as CoQA, QuAC, and DoQA address connected questions, coreference, and contextual reasoning over text, while doc2dial also includes agent questions for underspecified queries or verification.The cited works are CoQA (Reddy et al., 2019), QuAC (Choi et al., 2018), and DoQA (Campos et al., 2020); the comparison to agent questions is discussed alongside Saeidi et al. (2018).Source: §4 · p. 8
- Compared with MANtIS, which grounds conversations through document hyperlinks, doc2dial defines grounding at a finer level and uses entire documents with variable contexts.The comparison also notes MANtIS’s Stack Exchange source and varied user intents (Penha et al., 2019).Source: §4 · p. 8
- Compared with ShARC, doc2dial uses higher-level document structure, longer content spans, free-form utterances, and dialogues that need not correspond to one goal.ShARC uses short snippets and follow-up questions with yes/no answers (Saeidi et al., 2018).Source: §4 · p. 8
5 Conclusion
The conclusion presents doc2dial as a multi-domain dataset for goal-oriented, document-grounded dialogue, with broader scene variety and document coverage than prior work. It evaluates the dataset with three dialogue tasks and baseline approaches.
5 Conclusion
- Doc2dial covers dialogue scenes grounded in semi-structured and unstructured content across a wider span of associated documents than prior work.The paper evaluates user utterance understanding, agent response prediction, and relevant document identification with proposed baselines.Source: §5 · p. 8