PaperlayerView exact revision on arXiv

Source-linked AI summary

Recipes for Safety in Open-domain Chatbots

Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, Emily Dinan

arXiv:2010.07079v3cs.CLcs.AI

TL;DR

The paper addresses how open-domain dialogue models inherit toxic and biased language from human-interaction data while needing to remain engaging. It compares safety strategies, introduces human-and-model adversarial training and evaluation plus Baked-in Safety, and reports safer models while maintaining engagingness relative to existing models. The authors also identify limits involving style-control results and the scope of language, groups, and evolving notions of offensiveness.

  • Problem

    Dialogue models trained on large human-interaction corpora can learn toxic or biased language, creating a need for safety methods that preserve engaging conversation.

  • Method

    The paper compares classifier layers, direct generation and decoding methods, topic avoidance, and bias mitigation, while introducing Bot-Adversarial Dialogue Safety and Baked-in Safety.

  • Results

    The new techniques outperform their respective generative or two-stage counterparts in safety while maintaining engagingness relative to the state of the art.

  • Takeaways & Limitations

    Safety can be incorporated through adversarial human-bot data and modified generative targets rather than relying only on an external deployment-time classifier.

  • Takeaways & Limitations

    The approaches are limited by language and locale coverage, consensus-based offensiveness judgments that may miss minority-group offense, and the need for continual updates as language evolves.

Abstract

from arXiv · show

Models trained on large unlabeled corpora of human interactions will learn patterns and mimic behaviors therein, which include offensive or otherwise toxic behavior and unwanted biases. We investigate a variety of methods to mitigate these issues in the context of open-domain generative dialogue models. We introduce a new human-and-model-in-the-loop framework for both training safer models and for evaluating them, as well as a novel method to distill safety considerations inside generative models without the use of an external classifier at deployment time. We conduct experiments comparing these methods and find our new techniques are (i) safer than existing models as measured by automatic and human evaluations while (ii) maintaining usability metrics such as engagingness relative to the state of the art. We then discuss the limitations of this work by analyzing failure cases of our models.

1 Introduction

Open-domain dialogue models can inherit toxic or biased language from human-human training data. This work compares safety strategies that aim to reduce offensive outputs without sacrificing engagingness.

  • Human-interaction training data can transmit toxic or biased language to dialogue models.
  • The paper evaluates safety methods along two axes: minimizing offensive language and maintaining engagingness.The authors note that evasive responses may be safe but far from engaging, while conversational engagement can increase exposure to adversarial or sensitive prompts.
  • The study compares unsafe-utterance detection, direct safe-response training and decoding, sensitive-topic avoidance, and gender-bias mitigation.
  • The paper introduces Bot-Adversarial Dialogue Safety and Baked-in Safety models as two new techniques.
  • Bot-Adversarial Dialogue Safety collects human-and-model adversarial conversations to train more robust safety classifiers.The collected dataset contains 5k conversations involving around 70k utterances, and the resulting two-stage model outperforms models using other existing safety classifiers.
  • Baked-in Safety modifies training targets so generative models incorporate safety without an external classifier at deployment time.

2 Base Models

The experiments start from a BlenderBot-style open-domain dialogue system and compare it with DialoGPT and GPT2 (Large). The base system uses a large Transformer and staged conversational training.

  • The main base model follows BlenderBot’s Seq2Seq Transformer architecture and ParlAI implementation.
  • The studied model has 2.7B parameters, 2 encoder layers, and 24 decoder layers.
  • Pre-training uses 1.5B English human-human conversation examples from a previously existing Reddit dataset.
  • Fine-tuning uses ConvAI2, Empathetic Dialogues, Wizard of Wikipedia, and Blended Skill Talk to provide personality, empathy, knowledge, and blended conversational skills.
  • Decoding uses beam search with beam size 10, context and label 3-gram blocking, and a minimum beam length of 20 BPE tokens.
  • DialoGPT and GPT2 (Large) serve as comparison models, although their engagingness is expected to be lower than the BST 2.7B baseline.

3 Safety Recipes

The safety recipes combine classifier-based safety layers, direct generation methods, topic avoidance, and gender-bias mitigation. A central contribution is Bot-Adversarial Dialogue Safety, which trains and evaluates robustness against users probing models for unsafe responses.

  • Safety strategies: The paper organizes safety methods into unsafe-utterance detection, safe-utterance generation, sensitive-topic avoidance, and gender-bias mitigation.
  • Two-stage safety layers: Two-stage models detect safety violations in both user utterances and the model’s own utterances before selecting a safe response or non sequitur.
  • Two-stage safety layers: Two-stage safety depends on classifier quality: false positives can harm conversation quality, while missed violations allow toxic language to enter the conversation.
  • Bot-Adversarial Dialogue Safety: Bot-Adversarial Dialogue Safety asks crowdworkers to converse naturally while probing or goading a bot into unsafe replies.
  • Bot-Adversarial Dialogue Safety: The task labels each bot response for safety, uses 14-turn conversations, and holds out data for classifier training and static evaluation.Labels are separately verified by three other annotators.
  • Bot-Adversarial Dialogue Safety: Compared with classifier-focused adversarial collection, BAD has the breaker elicit offensive messages from a dialogue model rather than fool a classifier.

3.2 Safe Utterance Generation

Safe utterance generation modifies training or decoding so generative models are less likely to produce offensive content. The paper contrasts filtering and control methods with its Baked-in Safety approach, which replaces unsafe targets with safe responses.

  • Motivation: A separate safety layer complicates open-source deployment, increases computational requirements, and can be removed to enable unsafe use.
  • Methods: The paper studies safe beam blocking or generation, safety and style control, and Baked-in Safety as a new approach.
  • Data preprocessing: Filtering removes training examples when the context or target triggers a safety classifier, either utterance-by-utterance or by author.In the author-based method, authors are removed when over 12% of their posts trigger the classifier.
  • Safe decoding: N-gram blocking avoids unsafe words during beam search but may be overly cautious and can miss unsafe responses formed without listed words.
  • Control: Control tokens specify safety or style during training, with the safety control fixed to Safe at test time.
  • Baked-in Safety: Baked-in Safety replaces targets flagged as unsafe with a safe response or non-sequitur, then trains the generative model on the modified targets.Training examples modified for safety receive separate weighting from unmodified examples.

3.3 Sensitive Topic Avoidance

The paper avoids sensitive conversational topics by classifying them from Reddit-derived data, while noting that the topic list depends on use case and changing events.

  • Sensitive topics can create problems when bots give controversial opinions, incorrect information, or unsound advice.
  • 131 conversations were collected for a validation set in which crowdworkers discussed assigned sensitive topics with partners.
  • The sensitive-topic list was chosen using topic sensitivity and training-data availability, and may need expansion for particular use cases.
  • A multi-class Transformer classifier predicts sensitive-topic labels from truncated Reddit threads, including a safe class for other topics.
  • Because subreddit-derived labels are noisy, the authors collect a separate Mechanical Turk validation set to measure classifier performance.
  • At deployment, flagged utterances can trigger a canned response instead of continuing normally.

3.4 Gender Bias Mitigation

The paper addresses gender bias through conditional generation, training models to control gendered-word usage and fixing the deployment control to minimize it.

  • Gender bias appears across conversational datasets and can overlap with toxic language, with offensive utterances about females more often containing gendered or swear words.
  • Models are trained with one of four gender-word bins appended to each context, then deployed with F0M0 to use as few gendered words as possible.
  • The approach is limited by construction to explicitly binarily gendered words.

4 Existing Work

Existing work spans definitions and detection of abusive content, bias mitigation, adversarial robustness, controlled generation, data curation, and evolving benchmarks, but comparison is complicated by differing annotation schemes.

  • 4.1 Scope of Abusive Content: Abusive content includes distinct facets such as hate speech, hostile messages, cyberbullying, profanity, and malicious intent.
  • 4.1 Scope of Abusive Content: Multiple annotation schemes make unified comparisons with prior work difficult.
  • 4.2 Existing Approaches to Mitigate Unsafe Behaviors: Existing work addresses bias through race or dialect priming, adversarial training, fairness constraints, dataset balancing, gender swaps, and related methods.
  • 4.2 Existing Approaches to Mitigate Unsafe Behaviors: Adversarial prompts and interactions can induce undesirable outputs, motivating robustness measures for user-facing systems.
  • 4.2 Existing Approaches to Mitigate Unsafe Behaviors: Prior dialogue research examines classifier-based filtering, avoidance, empathetic responses, counter-attacks, and strategies for reducing re-offense.
  • 4.2 Existing Approaches to Mitigate Unsafe Behaviors: Controlled generation conditions outputs on attributes such as politeness, while data curation improves targeted traits but may leave vulnerability to adversarial prompts.
  • 4.2 Existing Approaches to Mitigate Unsafe Behaviors: Dynamic benchmarks repeatedly update abusive-content evaluations because the target changes over time.
  • 4.2 Existing Approaches to Mitigate Unsafe Behaviors: This paper uses a universal user-agnostic model rather than learning user-level characteristics that may aid abuse detection or response selection.

5 Evaluation Methods

The evaluation measures both safety and conversational quality, combining automatic metrics with human judgments and using adversarial dialogues to test robustness.

  • The authors evaluate conversational ability and safety together because a model can trade off safety against engagingness.
  • Perplexity and F1 on human-human chat data serve as proxies for conversational quality and engagingness.
  • ACUTE evaluation compares models through randomized pairwise human judgments using the question of who workers prefer for a long conversation.
  • The proposed models are compared with the BST 2.7B baseline for simultaneous safety and engagingness rather than greater engagingness alone.
  • Automatic safety evaluation generates responses from Reddit and ConvAI2 contexts and scores them with word-list and Transformer classifiers.
  • Classifier-based safety evaluation may disagree with human judgments and is unsuitable for two-stage models using the evaluation classifier.
  • Human safety evaluation uses 180 held-out Bot-Adversarial Dialogue conversations sampled from multiple models.
  • Logistic regression models unsafe-rating outcomes using bot, worker, instruction, and conversation-position variables.

6 Results & Analysis

The evaluations show substantial safety vulnerabilities in baseline dialogue models, while the proposed adversarial-data and baked-in methods improve safety without sacrificing engagingness. Results also show that safety interventions differ in effectiveness across automatic, human-adversarial, sensitive-topic, and bias-related evaluations.

  • Base Models: 8.0% of GPT2 generations were flagged by a safety classifier on pushshift.io Reddit contexts, compared with 2.4% on ConvAI2.
  • Base Models: 8.1% to 1.8%: safety-classifier fires on ConvAI2 decreased when moving from pushshift.io Reddit 2.7B to BST 2.7B.The authors attribute this pattern to training on less toxic data.
  • Two-Stage Models: 55% to 87.2% OK: the BAD-trained safety classifier substantially improved BST 2.7B performance on the adversarial test set.This exceeded the 78.2% OK rate reported for existing work by Dinan et al. (2019b).
  • Bot-Adversarial Dialogue: 80.8 unsafe F1: the BAD-based classifier outperformed methods achieving 61.5, 61.0, and 60.7 on the bot-adversarial BAD test set.The multi-turn adversarial training version outperformed its truncated single-turn counterpart.
  • Safety Controls: 94.4% OK: the BAD two-stage model improved adversarial safety without significantly changing engagingness relative to base BST 2.7B.Safe beam blocking produced only small toxicity decreases, while style controls improved safety for positive styles and degraded it for negative styles.
  • Baked-in Safety: 10.0% to 0.9%: baked-in non-sequitur fine-tuning reduced BST 2.7B classifier fires on pushshift.io Reddit, while maintaining similar engagingness.The from-scratch model reached 0% classifier fires; the fine-tuned and from-scratch systems were not significantly different from the base 2.7B model in engagingness.

7 Conclusion and Discussion

The paper presents two promising safety techniques—baked-in safety for generative models and adversarial human-bot robustness for two-stage models—while acknowledging that safety remains incomplete and culturally limited.

  • The study identifies baked-in safety and adversarial human-bot robustness as promising research directions.Both techniques outperform their respective generative or two-stage model counterparts.
  • The authors publicly release an adversarial dialogue safety training and evaluation set, along with their models in ParlAI.
  • 96.6% of human ratings judged the safest model safe on the adversarial dialogue safety test set.The authors emphasize that this still leaves a 3.4% failure rate.
  • Perfectly safe dialogue models likely require deep language understanding and may remain an AI-complete problem.
  • The methods rely on English-speaking annotators in the United States, limiting their geographical and historical scope.
  • Consensus-based offensiveness judgments may miss offense to minority groups, while safety techniques and data require continual updating as language evolves.
  • The work focuses on machine learning methods, leaving broader cultural and contextual safety issues unaddressed.
  • Releasing open-source models may raise safety concerns, although the authors argue reproducible community research is needed to address them.

A Bot-Adversarial Dialogue Collection

The Bot-Adversarial Dialogue dataset is collected by having humans deliberately try to induce unsafe responses from conversational bots.

  • Bot-Adversarial Dialogues are collected by asking humans to adversarially talk to bots.

A.1 Further Collection Details

The collection uses a crowdsourced task in which humans interact with multiple chatbot models, producing adversarial conversations and measuring when bots respond offensively.

  • Figure 6 shows the crowdsourced task used to collect Bot-Adversarial Dialogues.
  • Models without safety classifiers were more vulnerable to adversarial prompting, although collection statistics are affected by crowdworker experience and sequential effects.
  • When humans used offensive messages, bots replied offensively about one-third of the time.
  • Safe human utterances were about 2.5× less effective than offensive messages for eliciting unsafe bot responses.
  • The dataset also reports the number of offensive turns per dialogue.
  • Human safety judgments use 180 dialogues drawn from six chatbot models, with workers annotating the offensiveness of model replies.

A.2 Offensive Language Types

The dataset categorizes offensive language into four types and evaluates annotation reliability, while reporting that implicit forms were commonly used to break bots.

  • A separate crowdsourced task labels offensive utterances using four primary categories.
  • Hate speech attacks or demeans a group based on attributes such as race, gender, ethnicity, religion, disability, age, or sexual orientation.
  • Personal attack includes rude remarks, insults, or threats targeting an individual.
  • Profanity includes sexual remarks, swearing, curse words, weak pejoratives, and obscenities.
  • Other offensiveness covers offensive text outside hate speech, personal attacks, and profanity.
  • Implicit hate speech and other offensive language were more commonly used by crowdworkers than personal attacks and profanity.
  • Krippendorff’s alpha was 0.41 for multilabel annotation and 0.53 for binary offensive-versus-safe annotation.
  • Table 21 reports bot utterance counts and offensive fractions by chatbot model, while Table 22 reports classifier results across truncation lengths and validation settings.

A.3 Training a Safety Classifier with BAD

The study trains multi-turn offensive-language classifiers on Bot-Adversarial Dialogue data while varying the dialogue context length. A classifier trained with four-message contexts achieves the strongest overall F1 across evaluated validation context lengths.

  • The classifiers are trained on Bot-Adversarial Dialogue data using different truncated dialogue context lengths.The truncation length counts the current utterance and preceding messages included for context.
  • Classifiers trained with different truncated dialogue lengths perform almost equally on WTC, S, BBF, and BAD.
  • Higher overall F1 is achieved by the classifier trained with k_tr = 4 across BAD validation sets truncated to k_v ∈ {2, 4, 6}.
Loading 2010.07079v3…