PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-End Trainable Non-Collaborative Dialog System

Yu Li, Kun Qian, Weiyan Shi, Zhou Yu

arXiv:1911.10742v1cs.CLcs.AIcs.HC

TL;DR

Non-collaborative dialogue systems must address social and off-task content while pursuing goals that users do not share. This paper introduces a hierarchical intent scheme and MISSA, an end-to-end response-generation model with a response filter. Evaluations on anti-scam and persuasion tasks report that MISSA outperforms baseline methods, while the paper notes a limitation in tracking distant conversation history.

  • Problem

    Non-collaborative dialogue systems must handle on-task and off-task content because users and systems do not share a goal, and neglecting users’ questions may lead them to end the conversation.

    Source: Introduction · p. 1

  • Method

    The paper proposes a hierarchical on-task/off-task intent scheme and MISSA, which guides end-to-end response generation with intents and semantic slots and filters candidates against task or conversational constraints.

    Source: Introduction · p. 1

  • Results

    MISSA outperforms all baseline methods in fluency, coherency, and user engagement on both the anti-scam task and an existing persuasion task.

    Source: Conclusion and Future Work · p. 8

  • Takeaways & Limitations

    The paper presents MISSA as a dialog pipeline for non-collaborative settings that distinguishes on-task and off-task intents and handles diverse user utterances.

    Source: Conclusion and Future Work · p. 7

    MISSA can produce responses inconsistent with distant conversation history because GPT tracks only a limited history span.

    Source: Conclusion and Future Work · p. 8

Abstract

from arXiv · show

End-to-end task-oriented dialog models have achieved promising performance on collaborative tasks where users willingly coordinate with the system to complete a given task. While in non-collaborative settings, for example, negotiation and persuasion, users and systems do not share a common goal. As a result, compared to collaborate tasks, people use social content to build rapport and trust in these non-collaborative settings in order to advance their goals. To handle social content, we introduce a hierarchical intent annotation scheme, which can be generalized to different non-collaborative dialog tasks. Building upon TransferTransfo (Wolf et al. 2019), we propose an end-to-end neural network model to generate diverse coherent responses. Our model utilizes intent and semantic slots as the intermediate sentence representation to guide the generation process. In addition, we design a filter to select appropriate responses based on whether these intermediate representations fit the designed task and conversation constraints. Our non-collaborative dialog model guides users to complete the task while simultaneously keeps them engaged. We test our approach on our newly proposed ANTISCAM dataset and an existing PERSUASIONFORGOOD dataset. Both automatic and human evaluations suggest that our model outperforms multiple baselines in these two non-collaborative tasks.

Introduction

Non-collaborative dialog requires systems to handle both task-related and social content because users and systems do not share a goal. The paper proposes a hierarchical annotation scheme and MISSA, an end-to-end model for generating coherent, diverse responses and selecting task-appropriate outputs.

Introduction

  • Non-collaborative tasks involve users and systems with different goals, requiring social content to build rapport and trust alongside task-related dialogue.Neglecting an off-task question may prompt users to end a conversation, so systems need to address it while returning to the task goal.Source: Introduction · p. 1
  • The proposed hierarchical intent scheme separates on-task intents from off-task dialog acts, which can use general categories such as “open question.”Task-related intents represent on-task information; dialog acts convey syntax information for off-task content.Source: Introduction · p. 1
  • MISSA combines template coherence with language-generation diversity through intermediate intent and semantic-slot representations that guide conditional response generation.It follows TransferTransfo (Wolf et al. 2019), predicts user and system intents and slots, then generates multiple candidates with nucleus sampling and filters them using task or conversational constraints.Source: Introduction · pp. 1–2
  • MISSA outperforms multiple competitive baselines on ANTISCAM and PERSUASIONFORGOOD in both automatic and human evaluations.The paper also presents an anti-scam system and a persuasion system, and releases code and data.Source: Introduction · p. 2
  • ANTISCAM adds a dataset of conversations in which users defend themselves against attackers seeking personal information.The paper presents it as a resource for evaluating non-collaborative dialogue systems.Source: Introduction · p. 2

Related Work

Prior work covers several non-collaborative tasks and approaches to handling on-task and off-task content, but datasets not designed for this setting can make those contents difficult to disentangle. The paper positions its hierarchical intent scheme and end-to-end MISSA model as approaches for this broader task family.

Related Work

  • ANTISCAM is designed to interleave on-task and off-task content, addressing the difficulty of disentangling and measuring them in datasets not specifically designed for non-collaborative tasks.The paper proposes it as a benchmark for similar tasks.Source: Related Work · p. 2
  • The proposed hierarchical intent scheme separates on-task from off-task intents and decouples intents from semantic slots in multi-task training.The paper says the scheme can be adopted across non-collaborative tasks and can support quickly building end-to-end trainable dialog systems.Source: Related Work · p. 2
  • MISSA follows the TransferTransfo framework (Wolf et al. 2019), targeting non-collaborative tasks rather than the collaborative tasks addressed by Budzianowski and Vulić (2019).The paper describes traditional task-oriented systems as separate modules and notes the laborious dialog-state design and annotation of separately trained pipelines.Source: Related Work · p. 2
  • Hybrid systems generate task-oriented and non-task-oriented candidates separately, then select or connect outputs; MISSA is described as end-to-end trainable and easier to train and update.Cited examples include Yu, Black, and Rudnicky (2017), Papaioannou et al. (2017), Zhao, Romero, and Rudnicky (2018), and Baheti et al. (2018).Source: Related Work · pp. 2–3

Non-Collaborative Task Annotation Scheme

The annotation scheme separates task-specific on-task intents from broadly reusable off-task categories, while semantic slots capture utterance content. This design lets a new task define its own task actions and slots without redesigning the off-task categories.

Non-Collaborative Task Annotation Scheme

  • For a new non-collaborative task, the scheme requires designing on-task categories and semantic slots, while reusing universal off-task categories.The paper frames this separation as a way to decouple syntactic and semantic information and provide detailed supervision.Source: Non-Collaborative Task Annotation Scheme · p. 3
  • ANTISCAM defines elicitation, providing information, and refusal as task-specific on-task intents; PERSUASIONFORGOOD uses nine donation-related on-task intents.The PERSUASIONFORGOOD intents are based on its original dialog-act annotation scheme.Source: Non-Collaborative Task Annotation Scheme · p. 3
  • Both datasets share six general off-task dialog acts and six social intents, covering conversational forms such as questions, responses, greetings, apologies, and thanks.General acts capture syntactic meaning; social intents represent common social actions.Source: Non-Collaborative Task Annotation Scheme · p. 3
  • The anti-scam task uses 13 semantic slots to annotate sentence-level semantic content, including credit card numbers.Conversations are segmented into single sentences for annotation rather than annotated by turn.Source: Non-Collaborative Task Annotation Scheme · p. 3

Datasets

The experiments use two Amazon Mechanical Turk typing-conversation datasets with off-task dialogue interleaved. ANTISCAM supplies human-human anti-scam conversations; PERSUASIONFORGOOD supplies persuader–persuadee charity-donation conversations.

Datasets

  • Both datasets are Amazon Mechanical Turk typing conversations with off-task dialogue interleaved in the conversation.The evaluation uses ANTISCAM and PERSUASIONFORGOOD.Source: Datasets · p. 3

ANTISCAM Dataset

  • ANTISCAM contains 220 human-human anti-scam dialogs averaging 12.45 turns and 11.13 words per utterance; 172 of 220 users identified their partner as an attacker.The scenario involved attackers seeking users’ information through a popular Amazon customer service scam role-play.Source: ANTISCAM Dataset · p. 3
  • Two linguistically trained annotators labeled 3,044 sentences from 100 ANTISCAM dialogs, achieving an averaged weighted kappa of 0.874.Source: ANTISCAM Dataset · p. 4

PERSUASIONFORGOOD Dataset

  • PERSUASIONFORGOOD contains 1,017 dialogs, of which 300 have dialog-act annotations; workers were paired as persuader and persuadee for charity donations.The dataset averages 10.43 turns, and its original dialog acts were recategorized into on-task and pre-defined off-task intents.Source: PERSUASIONFORGOOD Dataset · p. 4

Model

MISSA adapts TransferTransfo (Wolf et al. 2019) for non-collaborative dialogue by adding intent and semantic-slot supervision. It generates responses conditioned on those representations, then filters candidates against task constraints.

Background

  • MISSA adapts TransferTransfo (Wolf et al. 2019) with intent and semantic-slot classifiers and rules for selecting an appropriate response.It retains the language-model and next-utterance prediction tasks while adding these components.Source: Background · p. 4

Intent and Semantic Slot Classifiers

  • The classifiers predict sentence-level human and system intents and slots from hidden-state representations, with losses combined across model tasks.The model concatenates <sep>-position embeddings with the last sentence's hidden state and can predict multiple intents and slots per utterance.Source: Intent and Semantic Slot Classifiers · p. 4

Response Generation

  • MISSA generates each response sentence conditioned on a system-intent token and can continue through multiple sentences until the end-of-sequence token.Training inserts response-intent tokens at sentence starts; Figure 1 also shows the appended response and distractor removed during testing.Source: Response Generation · pp. 4–5, Figure 1 · p. 5

Response Filtering

  • Nucleus sampling produces diverse candidate responses, and a task-specific filter selects one that satisfies conversation constraints.In the anti-scam setting, rules filter candidates such as those eliciting information already provided.Source: Response Filtering · p. 5

Experiments

The experiments evaluate MISSA on ANTISCAM and PERSUASIONFORGOOD, using automatic metrics and human evaluation on ANTISCAM. They compare MISSA with two baselines and examine component ablations.

Experiments

  • Experiments cover ANTISCAM and PERSUASIONFORGOOD, with an 80%/10%/10% training, validation, and test split.ANTISCAM targets occupying attackers and eliciting their information; PERSUASIONFORGOOD targets persuading people to donate to charity.Source: Experiments · p. 5

Baseline Models

  • The main baselines are vanilla TransferTransfo and a hybrid that routes on-task utterances to MISSA and off-task utterances to TransferTransfo.The ablations MISSA-sel and MISSA-con omit response filtering and the response-generation intent token, respectively.Source: Baseline Models · p. 5

Automatic Evaluation Metrics

  • Automatic evaluation includes perplexity and response-intent and response-slot prediction measures, including extended measures for acceptable intent transitions.The intent and slot predictors achieve 84% accuracy and 77% on ANTISCAM, respectively.Source: Automatic Evaluation Metrics · pp. 5–6

Human Evaluation Metrics

  • Human evaluation uses 15 volunteers, 225 dialogs, and 45 ratings per model to assess fluency, coherence, engagement, dialog length, and task success.Task success counts the average types of attacker information obtained, including name, address, and phone number.Source: Human Evaluation Metrics · p. 6

Results and Analysis

MISSA outperforms TransferTransfo and the hybrid model on almost all reported metrics across ANTISCAM and PERSUASIONFORGOOD. The analyses associate its results with on-task engagement, context handling, conditional generation, and response filtering.

Results and Analysis

  • MISSA outperforms TransferTransfo and the hybrid model on almost all metrics on both ANTISCAM and PERSUASIONFORGOOD.The main ANTISCAM results are in Table 3; PERSUASIONFORGOOD automatic results are in Table 5.Source: Results and Analysis · pp. 6–7, Table 5 · p. 7
  • 14.9 turns for MISSA versus 8.5 for TransferTransfo, and a task success score of 1.294 versus 1.025.The authors interpret the longer conversations as better attacker engagement and the higher score as more strategic information elicitation.Source: Results and Analysis · p. 6
  • In example dialogs, MISSA tracks attacker names and billing-address context that the hybrid model misses.The authors suspect the hybrid model's lack of a built-in semantic slot predictor contributes to these errors.Source: Results and Analysis · p. 7
  • MISSA scores 4.18 versus 3.78 in fluency and 3.75 versus 3.68 in coherence against MISSA-con.The authors attribute the gains to conditioning response generation on system intent.Source: Results and Analysis · p. 7
  • MISSA scores 3.69 versus 2.87 in engagement against MISSA-sel; the authors say filtering removes incoherent responses and encourages continued chatting.They conclude that both conditional generation and the response filter are essential to MISSA's performance.Source: Results and Analysis · p. 7
  • On PERSUASIONFORGOOD, MISSA and its variants outperform TransferTransfo and the hybrid model on all evaluation metrics, while MISSA-con leads on RIP and ERIP.The authors suspect the dataset has more possible responses per intent than ANTISCAM and suggest adjusting model structure to dataset characteristics.Source: Results and Analysis · p. 7, Table 5 · p. 7

Conclusion and Future Work

The paper presents a pipeline for non-collaborative dialog that combines hierarchical intent annotation with MISSA, an end-to-end response-generation model. It reports gains across two tasks while noting that distant conversation history can still produce inconsistent responses.

Conclusion and Future Work

  • The proposed pipeline combines hierarchical annotation with MISSA, an end-to-end neural response-generation model for non-collaborative dialog.The annotation scheme distinguishes on-task from off-task intents, and MISSA uses both as training supervision.Source: Conclusion and Future Work · pp. 7–8
  • MISSA outperforms all baseline methods in fluency, coherency, and user engagement on ANTISCAM and PERSUASIONFORGOOD.The paper describes ANTISCAM as a dataset focused on deterring phone scammers.Source: Conclusion and Future Work · p. 8
  • MISSA can produce responses inconsistent with distant history because GPT tracks only a limited history span.The authors identify longer-context tracking as future work.Source: Conclusion and Future Work · p. 8

Anti-Scam Collection Setting

ANTISCAM uses role-played conversations between attackers seeking personal information and users trying to identify attackers and protect or elicit information. The collected dialogs also contain substantial social and off-task content.

Anti-Scam Collection Setting

  • The study paired workers as attackers seeking personal information and everyday users aiming to protect their information and potentially elicit the attacker's information.Workers received personal data and could not see their partners' instructions.Source: Anti-Scam Collection Setting · p. 8
  • Users were instructed to identify possible attackers and, when suspicious, prolong the conversation and elicit information.Bonuses rewarded detecting attackers and eliciting real information such as names, addresses, and phone numbers.Source: Anti-Scam Collection Setting · p. 8
  • The 220 dialogs averaged 12.45 turns and 11.13 words per utterance; 172 users identified their partner as an attacker.Attackers posed as Amazon customer-service agents and were rewarded for eliciting address and payment-card details.Source: Anti-Scam Collection Setting · p. 8
  • Annotators labeled 3,044 sentences from 100 dialogs with averaged weighted kappa of 0.874.Both roles produced substantial social content: 292 attacker sentences and 252 user sentences.Source: Anti-Scam Collection Setting · p. 9
  • The annotation analysis found distinct role patterns: users had more refusals (74 vs 19), open questions (173 vs 54), and yes-no questions (165 vs 117).The authors link the question patterns to users being instructed to prolong conversation after detecting an attacker.Source: Anti-Scam Collection Setting · p. 9

Training details

The model is pretrained on PERSONA-CHAT and fine-tuned on ANTISCAM and PERSUASIONFORGOOD; the paper also points to an example human-human ANTISCAM dialog.

Training details

  • Training uses PERSONA-CHAT pretraining, then an 80/10/10 train-validation-test split for fine-tuning on ANTISCAM and PERSUASIONFORGOOD.The model uses Adam with learning rate 6.25e-5, L2 weight decay 0.01, and specified loss coefficients for language modeling, intent and slot classifiers, and next-utterance classification.Source: Training details · p. 9

Example Dialog

  • Table 7 provides an example human-human ANTISCAM dialog with slot values replaced by slot tokens.Source: Example Dialog · p. 10
Loading 1911.10742v1…