Source-linked AI summary
Modeling Multi-turn Conversation with Deep Utterance Aggregation
Zhuosheng Zhang, Jiangtong Li, Pengfei Zhu, Hai Zhao, Gongshen Liu
TL;DR
Multi-turn response matching must model interactions among previous utterances rather than simply concatenating them. The paper proposes deep utterance aggregation with self-matching attention and attentive turns aggregation, achieving state-of-the-art results across three datasets, including a new e-commerce corpus.
Problem
Multi-turn conversation understanding remains challenging because prior work concatenates utterances while ignoring their interactions and differing relevance to the response.
Method
The model forms fine-grained context representations through turns-aware aggregation, self-matching attention that filters redundancy, and response matching over refined utterances.
Results
The model achieves state-of-the-art performance on three multi-turn conversation datasets, including the released E-commerce Dialogue Corpus, with a 4.8% R10@1 improvement over SMN on ECD.
Takeaways & Limitations
Deep utterance aggregation improves context representation for retrieval-based multi-turn response matching, including conversations involving real customer-service interactions.
Takeaways & Limitations
The ECD evaluation assumes one correct response per conversation, so semantically similar alternatives may be marked wrong under exact-match evaluation.
Abstract
from arXiv · showhide
Multi-turn conversation understanding is a major challenge for building intelligent dialogue systems. This work focuses on retrieval-based response matching for multi-turn conversation whose related work simply concatenates the conversation utterances, ignoring the interactions among previous utterances for context modeling. In this paper, we formulate previous utterances into context using a proposed deep utterance aggregation model to form a fine-grained context representation. In detail, a self-matching attention is first introduced to route the vital information in each utterance. Then the model matches a response with each refined utterance and the final matching score is obtained after attentive turns aggregation. Experimental results show our model outperforms the state-of-the-art methods on three multi-turn conversation benchmarks, including a newly introduced e-commerce dialogue corpus.
1 Introduction
Multi-turn retrieval dialogue systems must model which prior utterance information matters and how utterances interact, rather than simply concatenating a noisy context. The paper proposes deep utterance aggregation, releases a diverse e-commerce corpus, and reports state-of-the-art results on three benchmarks.
- Multi-turn natural language understanding remains challenging because systems must comprehend conversation context and produce informative, appropriate replies.
- Simple context concatenation introduces noise, while prior response matching overlooks differing utterance importance and internal utterance semantics.
- The model uses turns-aware aggregation to weigh previous conversations and attention-based recurrent networks to aggregate important words within each utterance.
- The released E-commerce Dialogue Corpus contains real human conversations spanning commodity consultation, logistics, recommendation, negotiation, and chitchat.
- The improved retrieval-based response matching model achieves state-of-the-art performance on three benchmark datasets, including the newly released e-commerce corpus.
2 Related Work
The paper distinguishes its approach from prior data-driven dialogue studies by modeling both the importance of previous conversations and salient word-level content within each utterance.
- Prior studies provide data-driven dialogue systems, including generation-based and retrieval-based approaches for conversational response handling.
- The model discriminates the importance of previous conversations instead of treating all turns equally in multi-turn response matching.
- It also accumulates substantial utterance parts according to word-level importance within each utterance.
3 Deep Utterance Aggregating Strategy
DUA builds fine-grained context representations by refining utterances, modeling their interactions with the last utterance, and aggregating attentive matching states for response retrieval.
- Utterance Representation: DUA encodes each utterance and response with recurrent representations before performing semantic matching between utterances and response candidates.Words are mapped to distributed representations and processed with GRUs to obtain utterance and response representations.
- Turns-aware Aggregation: Turns-aware aggregation fuses each utterance representation with the last utterance so preceding turns receive intention-relevant context.The fusion uses an aggregation operation implemented here as concatenation.
- Matching Attention Flow: Self-matching attention filters redundant sequence information by dynamically collecting salient content from each fused representation.The mechanism matches the fused representation against itself and produces an output sequence for subsequent processing.
- Matching Attention Flow: The model constructs word-level and utterance-level matching matrices, then applies CNN and max-pooling operations to extract salient local matching features.The resulting representations combine pooled information from both matching matrices.
- Attentive Turns Aggregation: Attentive turns aggregation feeds CNN outputs into a GRU and applies attention to accumulate matching information across turns before computing the final score.Context Fusion comprises turns-aware aggregation followed by attentive turns aggregation, representing two stages of cross-utterance interaction.
- Training: The model is trained by updating its parameters according to a cross-entropy loss.
4 Experiment
The experiments evaluate DUA on Ubuntu, Douban, and the newly released ECD corpus against single-turn and advanced multi-turn matching models. Results show gains from modeling utterance relationships, turns-aware aggregation, and attention-based selection, while error analyses identify challenges involving multiple intentions, topics, and suitable responses.
- Datasets and evaluation: DUA is evaluated on Ubuntu, Douban, and the newly released ECD corpus using MAP, MRR, P@1, and Rn@k metrics.The evaluation truncates conversations to at most 10 utterances and each utterance to 50 words, applying truncation and zero-padding when necessary.
- Baselines: The baselines include single-turn models that concatenate context utterances and advanced multi-turn models that model utterance relationships or match responses with individual utterances.The advanced multi-turn baselines include Multi-view, DL2R, and SMN.
- Overall results: DUA outperforms the other models on most metrics across the three corpora, while concatenation-based single matching models perform much worse.The comparison indicates that utterance relationships matter for multi-turn response matching.
- Overall results: 4.8% R10@1 improvement on ECD is reported over SMN, which lacks turns-aware aggregation and matching attention flow.The ECD advantage is reported as evidence that the approach handles real customer-service conversations rather than only chitchat.
- Conversation type analysis: Chitchat and logistics are easier conversation types, whereas recommendation, consultation, and negotiation are harder because they involve varied topics and intentions.These categories make ECD more challenging than previous chitchat- or question-answering-based corpora.
- Visualization: The attention visualization shows DUA distilling key words from an utterance and response to guide response matching for a complaint about a missing gift and slow delivery.The highlighted utterance and response words correspond to the user’s intention and a suitable response.
- Ablation study: Removing Matching Attention Flow reduces R10@1 by 6.9%, while removing Context Fusion reduces it by 4.8%.Removing both mechanisms produces the worst performance, supporting the importance of salient-word selection and utterance relationships.
- Error analysis: ECD conversations can contain multiple intentions in one message, causing the appropriate response to favor one aspect over another.The example combines packaging questions with delivery-company and delivery-time questions.
5 Conclusion
The paper proposes deep utterance aggregation for fine-grained context representation and reports new state-of-the-art results across three datasets.
- The model forms fine-grained context representations through deep utterance aggregation.
- Experiments on three datasets show new state-of-the-art results.
- The study releases the first e-commerce dialogue corpus for research communities.
- Future work will address contextual semantics and modeling conversations involving various intentions and topics.