Source-linked AI summary
Speaker-Aware BERT for Multi-Turn Response Selection in Retrieval-Based Chatbots
Jia-Chen Gu, Tianda Li, Quan Liu, Zhen-Hua Ling, Zhiming Su, Si Wei, Xiaodan Zhu
TL;DR
The paper addresses multi-turn response selection for retrieval-based chatbots, where speaker changes and long, entangled contexts challenge straightforward pre-trained-model use. It proposes SA-BERT with speaker-aware disentanglement and domain adaptation, achieving new state-of-the-art performance across five public datasets.
Problem
Multi-turn response selection must choose the best response from candidates given multi-utterance contexts, while shallow concatenation neglects speaker changes and is limited by BERT-Base’s 512-token sequence length.
Method
SA-BERT adds speaker embeddings and segmentation tokens, selects important utterances using speaker-aware disentanglement, and applies domain adaptation to incorporate in-domain knowledge.
Results
SA-BERT achieves new state-of-the-art performance across five public datasets, with large-margin improvements including 5.5% R10@1 on Ubuntu Dialogue Corpus V1 and 15.5% R100@1 on DSTC 8-Track 2-Subtask 2 Corpus.
Takeaways & Limitations
Speaker-aware modeling, disentanglement, and domain adaptation provide an effective approach for multi-turn response selection in retrieval-based chatbots.
Abstract
from arXiv · showhide
In this paper, we study the problem of employing pre-trained language models for multi-turn response selection in retrieval-based chatbots. A new model, named Speaker-Aware BERT (SA-BERT), is proposed in order to make the model aware of the speaker change information, which is an important and intrinsic property of multi-turn dialogues. Furthermore, a speaker-aware disentanglement strategy is proposed to tackle the entangled dialogues. This strategy selects a small number of most important utterances as the filtered context according to the speakers' information in them. Finally, domain adaptation is performed to incorporate the in-domain knowledge into pre-trained language models. Experiments on five public datasets show that our proposed model outperforms the present models on all metrics by large margins and achieves new state-of-the-art performances for multi-turn response selection.
1 INTRODUCTION
The paper adapts BERT to multi-turn response selection with speaker-aware representations, disentanglement, and domain adaptation. Across five datasets, SA-BERT achieves large-margin improvements and new state-of-the-art performance.
- Multi-turn response selection chooses the best-matched response from candidates given a conversation context of multiple utterances.
- Shallow concatenation neglects speaker changes, weakens chronological utterance relationships, and cannot handle contexts exceeding BERT-Base’s 512-token limit.
- SA-BERT adds speaker embeddings and special segmentation tokens to make BERT aware of speaker changes during conversations.
- Speaker-aware disentanglement selects a small number of important utterances using speaker information, while domain adaptation incorporates in-domain knowledge into the pre-trained model.
- 5.5% R10@1 on Ubuntu Dialogue Corpus V1, 5.9% R10@1 on Ubuntu Dialogue Corpus V2, and 15.5% R100@1 on DSTC 8-Track 2-Subtask 2 Corpus were reported among the large-margin gains.
- The paper presents three contributions: SA-BERT, adaptation analysis, and new state-of-the-art performance on five multi-turn response-selection datasets.
2 METHODOLOGY
SA-BERT adapts BERT for multi-turn response selection by encoding speaker changes, turn boundaries, and selected context utterances. It also uses domain adaptation to incorporate dataset-specific knowledge.
- Problem formulation: The task learns a matching model that scores context–response pairs, where labels indicate whether a candidate is an appropriate response.Training minimizes cross-entropy loss over dialogue examples.
- Speaker-aware input: SA-BERT adds speaker embeddings to token representations so the model tracks speaker identity as the conversation progresses.For two speakers, embeddings alternate when the speaker changes.
- Speaker-aware input: [EOU] and [EOT] tokens mark utterance and turn boundaries, modeling interactions between utterances in the context.The paper reports that these segmentation tokens improve performance consistently.
- Speaker-aware disentanglement: The speaker-aware disentanglement strategy filters context by selecting utterances sharing a spoken-from or spoken-to speaker with the response speaker.Selected utterances retain chronological order and receive the corresponding speaker embeddings.
- Domain adaptation: Domain adaptation uses each dataset’s training set and jointly optimizes next sentence prediction with masked language modeling.The approach incorporates in-domain knowledge without additional external knowledge.
- Output representation: The model represents each context–response pair with the [CLS] embedding and passes it to a sigmoid classifier that outputs a matching score.The score denotes the matching degree between the context and candidate response.
3 EXPERIMENTS
SA-BERT is evaluated on five public multi-turn response-selection datasets using established retrieval metrics. Results tables compare it with prior methods and include an ablation of speaker-aware disentanglement on DSTC 8-Track 2-Subtask 2.
- Datasets: SA-BERT is tested on five public datasets, with speaker-aware disentanglement applied specifically to DSTC 8-Track 2-Subtask 2.The other four datasets were disentangled in advance by their publishers.
- Evaluation metrics: Evaluation selects the k best responses from n candidates and measures recall Rn@k, with MAP, MRR, and P@1 also used for Douban.These metrics follow prior work’s settings.
- Results: 5.5% R10@1 on Ubuntu V1 and 5.9% R10@1 on Ubuntu V2 are reported improvements over prior state-of-the-art performance.The comparison is reported in the evaluation results for the Ubuntu corpora.
- Results: 3.2% MAP and 2.7% MRR on Douban, plus 8.3% R10@1 on E-commerce, are reported improvements over prior state-of-the-art performance.These results are included among the evaluation comparisons in the results tables.
- Results: 15.5% R100@1 on DSTC 8-Track 2-Subtask 2 is reported over prior state-of-the-art performance.Table 4 also evaluates speaker-aware disentanglement through ablation tests.
4 ANALYSIS
The analysis examines domain adaptation and speaker-aware components in SA-BERT. Adaptation helps across corpora, while speaker embeddings and disentanglement improve response-selection performance.
- 4.1 Adaptation Corpus: Adaptation improved performance regardless of which corpus was used, with the best results when adaptation and fine-tuning used the same corpus.The analysis attributes this pattern to the value of domain-specific knowledge and corpus-task similarity.
- 4.2 Speaker Embeddings: Speaker-embedding ablation caused a performance drop, verifying the effectiveness of speaker embeddings.
- 4.3 Speaker-Aware Disentanglement Strategy: 37.5% R100@1 followed speaker-aware disentanglement on IMN, compared with 32.2% R100@1 using its original last-70-utterance context.The strategy selected about 25 utterances instead of the original 70.
5 CONCLUSION
The paper studies pre-trained language models for multi-turn response selection in retrieval-based chatbots and proposes speaker-aware modeling and disentanglement. Experiments on five public datasets report new state-of-the-art performance, while future work will further adjust pre-trained models and design disentanglement strategies.
- The paper proposes a speaker-aware model and speaker-aware disentanglement strategy for multi-turn response selection.
- Experiments on five public datasets show that the proposed method achieves new state-of-the-art performance.
- Future work will further adjust pre-trained language models and design new disentanglement strategies.
A.1 Task Definition
Multi-turn response selection is illustrated as a task example in Table 7.
- Table 7 presents an example of the multi-turn response-selection task.
A.2 Related Work
Dialogue-system methods are categorized into generation-based and retrieval-based approaches. Retrieval-based systems match conversational contexts with response candidates, while prior work develops increasingly sophisticated interaction and attention mechanisms.
- Generation-based methods synthesize responses by maximizing generation probability given previous conversation context.
- Retrieval-based methods learn a matching model for conversational contexts and response candidates, selecting responses from a repository.
- Prior retrieval-based models include DAM, MRFN, IMN, IoI, and MSN, which use attention, multiple representations, or interaction mechanisms.
A.3 Input Representation
Figure 1 presents the visual architecture of SA-BERT’s input representation.
- Figure 1 illustrates the input representation architecture used by SA-BERT.
- The architecture is presented visually rather than described through an exhaustive BERT background.
- Readers are directed to Figure 1 for the visual structure of the input representation.
A.4 Adaptation Tasks
SA-BERT uses masked language modeling and next sentence prediction as adaptation tasks. These tasks support bidirectional representation learning and speaker-embedding pre-training.
- Adaptation tasks: The adaptation tasks employ both masked language modeling and next sentence prediction.
- MLM: Masked language modeling trains bidirectional representations by predicting tokens masked from the input.Tokens are replaced with [MASK], random words, or the original word at rates of 80%, 10%, and 10%.
- NSP: Next sentence prediction pre-trains speaker embeddings using true context responses as positives and randomly sampled responses as negatives.
A.5 Datasets
SA-BERT is evaluated on five public multi-turn response-selection datasets, with different dataset construction and evaluation procedures. The experiments also specify fixed training and testing configurations.
- Datasets: Five public datasets are used: Ubuntu V1, Ubuntu V2, Douban, E-commerce, and DSTC 8-Track 2-Subtask 2.The first four datasets were disentangled by their publishers, while the proposed disentanglement strategy was applied only to DSTC 8-Track 2-Subtask 2.
- Dataset characteristics: The Douban Conversation Corpus contains multiple correct candidates for a context and uses manually annotated labels.
- Experimental settings: Training used a 512-token maximum sequence length, batch size 25, and at most three epochs.The initial learning rate was 2e-5, and validation performance selected the stopping condition and best testing model.