PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark

Hao Sun, Guangxuan Xu, Jiawen Deng, Jiale Cheng, Chujie Zheng, Hao Zhou, Nanyun Peng, Xiaoyan Zhu, Minlie Huang

arXiv:2110.08466v2cs.CL

TL;DR

Dialogue safety remains under-defined, and prior toxicity and bias research does not directly address the context and psychological-risk considerations of open-domain dialogue. The paper proposes a context-sensitive safety taxonomy and DiaSafety dataset, then trains a classifier and evaluates popular dialogue systems. Existing safety guards struggle on the dataset, and evaluated systems continue to show context-sensitive safety problems.

  • Problem

    Dialogue safety remains under-defined and datasets are scarce, while utterance-level toxicity and bias research does not capture the full contextual and psychological-risk considerations of dialogue.

    Source: §1 · p. 1

  • Method

    The authors propose a dialogue-safety taxonomy focused on context-sensitive unsafety, compile DiaSafety under it, and train a classifier for detection and model evaluation.

    Source: §1 · pp. 1–2

  • Results

    Existing safety guards struggle on DiaSafety, and popular dialogue systems show persistent context-sensitive safety problems, with all evaluated systems suffering more from these than utterance-level unsafety.

    Source: §1 · p. 2, §7 · pp. 8–9

  • Takeaways & Limitations

    The trained classifier provides a benchmark for evaluating context-sensitive safety in conversational models, including for model release testing.

    Source: §7 · p. 8

    The dataset is based on model-generated content and publicly available data.

    Source: Limitations and Ethics · p. 9

Abstract

from arXiv · show

Dialogue safety problems severely limit the real-world deployment of neural conversational models and have attracted great research interests recently. However, dialogue safety problems remain under-defined and the corresponding dataset is scarce. We propose a taxonomy for dialogue safety specifically designed to capture unsafe behaviors in human-bot dialogue settings, with focuses on context-sensitive unsafety, which is under-explored in prior works. To spur research in this direction, we compile DiaSafety, a dataset with rich context-sensitive unsafe examples. Experiments show that existing safety guarding tools fail severely on our dataset. As a remedy, we train a dialogue safety classifier to provide a strong baseline for context-sensitive dialogue unsafety detection. With our classifier, we perform safety evaluations on popular conversational models and show that existing dialogue systems still exhibit concerning context-sensitive safety problems.

1 Introduction

Generative conversational models face safety concerns that limit real-world deployment, while dialogue safety remains difficult to transfer from utterance-level toxicity and bias detection. The paper proposes a context-sensitive taxonomy and DiaSafety dataset, then uses a classifier to evaluate popular dialogue systems, finding persistent context-sensitive safety problems.

1 Introduction

  • Dialogue safety is hard to transfer from toxicity and bias research because it includes psychological-risk responses and requires context beyond the utterance.Prior datasets and studies focus overwhelmingly on utterance-level cases.Source: §1 · p. 1
  • The taxonomy focuses on context-sensitive unsafety: responses safe in isolation that become unsafe in context, distinct from cases unsafe at the utterance level.The paper names six categories: Offending User, Risk Ignorance, Unauthorized Expertise, Toxicity Agreement, Biased Opinion, and Sensitive Topic Continuation.Source: §1 · pp. 1–2
  • Existing safety guards struggle with context-sensitive cases; the authors train a classifier and find popular systems suffer more from these cases than from utterance-level unsafety.Blenderbot is comparatively safer than the other evaluated systems.Source: §1 · p. 2
  • The paper introduces DIASAFETY, a dataset of context-sensitive unsafe examples described as high quality and challenging for existing safety detectors.The dataset is compiled under the proposed dialogue-safety taxonomy.Source: §1 · p. 2
  • The paper benchmarks Blenderbot, DialoGPT, and Plato-2, highlighting safety problems especially involving context-sensitive unsafety.Source: §1 · p. 2

2 Related work

Prior work covers toxic language, social bias, and dialogue safety, but dialogue safety remains under-defined and existing datasets often focus on narrower aspects. This paper positions its taxonomy and dataset as extending context-aware work toward context-sensitive examples across a wider safety scope.

2 Related work

  • Earlier dialogue-safety research includes work on harassment, sensitive topics, context modeling, and generation detoxification.Dinan et al. (2019) found that context can yield more sophisticated attacks and improve BERT-based detector performance.Source: §2 · pp. 2–3
  • Existing work addresses toxicity, bias, and dialogue safety, but the field still lacks a settled definition of dialogue safety.Prior datasets often focus on a single safety aspect, while this work covers wider scopes and considerations.Source: §2 · p. 3
  • The authors distinguish their dataset as context-sensitive rather than merely context-aware, following calls to collect context-aware data.They compare DIASAFETY with other public datasets in Table 1.Source: §2 · p. 3

3 Safety Taxonomy

The taxonomy defines dialogue safety in terms of ethical and legal concerns and distinguishes utterance-level unsafety from context-sensitive unsafety. It further organizes context-sensitive cases by whether their impact is personal or non-personal.

3 Safety Taxonomy

  • Context-sensitive unsafety is a seemingly safe utterance that becomes unsafe given the dialogue context, unlike utterance-level unsafety that is unsafe regardless of context.The paper gives “I think so” in response to “black people are all violent” as an example.Source: §3 · p. 3

3.1 Personal unsafety

  • Personal unsafety covers direct harms to users through offense, ignored risks, or unauthorized expertise, with medical expertise the primary focus.The categories include potential psychological or physical harm and unreliable information that may induce risky behavior.Source: §3.1 · pp. 3–4

3.2 Non-personal Unsafety

  • Non-personal unsafety covers harms toward third parties or society through toxicity agreement, biased opinion, and sensitive-topic continuation.The paper focuses on gender, race, and religion for biased opinions, and treats expressing preferences while continuing sensitive topics as unsafe.Source: §3.2 · p. 4

4 Dataset Collection

DiaSafety contains 11K context-response pairs under the proposed unsafe categories, with all unsafe examples requiring dialogue context for correct labeling; Sensitive Topic Continuation is excluded. The collection combines multiple sources and annotation procedures to identify and verify context-sensitive cases.

4 Dataset Collection

  • DiaSafety contains 11K labeled context-response pairs, all context-sensitive, but excludes Sensitive Topic Continuation from data collection.The authors leave that category for future work because its definition varies across regions, cultures, and individuals.Source: §4 · p. 4, §4.1 · p. 4

4.1 Data Source

  • The dataset combines real-world Reddit conversations, public datasets, and model-generated responses selected using category-specific prompts.Collected data undergo format regularization and filtering for explicit utterance-level unsafety.Source: §4.1 · pp. 4–5

4.2 Human Annotation

  • Human annotation covers Risk Ignorance, Offending User, and Biased Opinion, while automatic labeling supports Unauthorized Expertise and Toxicity Agreement.Three workers label each human-annotation task, with majority vote as the golden label; utterance- and context-level rounds confirm context sensitivity.Source: §4.2 · p. 5

4.3 Annotation Quality Control

  • Annotators must meet experience and approval thresholds, and the dataset reports moderate overall inter-annotator agreement.Workers receive detailed guidelines and test questions; Table 3 reports dataset statistics, using κ for human-label agreement and accuracy for automatic-label quality.Source: §4.3 · pp. 5–6, Table 3 · p. 6

5 Context-sensitive Unsafety Detection

The experiments find that existing safety detectors struggle with context-sensitive cases, while context improves fine-grained classification. The authors conclude that separating utterance-level and context-sensitive checks is preferable for comprehensive safety evaluation.

5.1 Experimental Setup

  • The evaluation asks whether neural models can detect context-sensitive unsafety, how context affects detection, and whether existing guards can identify it.Experiments use RoBERTa with context and response separated by </s>, with train/dev/test splits of 80%/10%/10% for each category.Source: §5 · p. 6, §5.1 · p. 6

5.2 Fine-grain Classification

  • The fine-grained task classifies whether a response is unsafe and, if so, assigns its unsafe category.The reported procedure combines five one-vs-all classifiers, each predicting Safe, Unsafe, or N/A for its category; N/A handles other categories.Source: §5.2 · p. 6, §5.1 · p. 6
  • Adding context improves overall fine-grained F1 by 13.4%, while Unauthorized Expertise improves less noticeably.The authors attribute the smaller gain to cues such as medicine- and suggestion-related words that can make this category detectable at the utterance level.Source: §5.2 · p. 6, Table 4 · p. 6

5.3 Coarse-grain Classification

  • Detoxify and PerspectiveAPI achieve F1-scores close to random on context-sensitive cases, and context-aware models remain insufficiently sensitive to context.Finetuning Detoxify on DiaSafety improves its results, but the authors report that utterance-level unsafety pretraining degrades context-sensitive detection; they favor separate sequential checks.Source: §5.3 · p. 7, Table 5 · p. 7

6 Dialogue System Safety Evaluation

The paper evaluates conversational-model safety with a two-step check of utterance-level and context-sensitive unsafety. Across the evaluated models, safety problems remain, especially for context-sensitive cases.

6.1 Two-step Safety Detection Strategy

  • The two-step strategy checks a response with Detoxify for utterance-level unsafety, then uses the paper’s classifiers on responses that pass to check context-sensitive unsafety.The authors describe this as combining utterance-level resources with context-sensitive checking.Source: §6.1 · p. 7

6.2 Unsafety Metric

  • For each of five categories, the unsafety metric is the proportion of generated responses that are context-sensitive unsafe or utterance-level unsafe.Each model generates 10 responses per context; lower proportions indicate safer behavior.Source: §6.2 · p. 8

6.3 Evaluated Models

  • The evaluation covers three publicly available open-source models: DialoGPT, Blenderbot, and Plato-2.Blenderbot is described as trained with safety-oriented data cleaning and safe-response augmentation.Source: §6.3 · p. 8

6.4 Evaluation Results

  • Blenderbot has the best overall safety performance and the lowest unsafe proportion except for Toxicity Agreement; Plato-2 is weakest at utterance-level safety.The reported pattern is based on five context categories; “Overall” is their macro average.Source: §6.4 · p. 8, Figure 1 · p. 8
  • Larger models do not come with safer responses, and the evaluated sampling methods have little impact on conversational-model safety.The authors speculate that larger models may be more confident in unauthorized suggestions and implicit offensiveness, while smaller models may be more cautious.Source: §6.4 · p. 8

7 Conclusion and Future Work

The paper presents DIASAFETY, a context-sensitive dialogue-safety dataset grounded in a proposed taxonomy, and uses it to evaluate conversational models. The authors conclude that context-sensitive unsafety warrants further attention and propose expanding the dataset and exploring category-specific detection.

7 Conclusion and Future Work

  • The paper presents a dialogue-safety taxonomy and the corresponding context-sensitive dataset DIASAFETY.The paper reports that the dataset is high quality and can deceive existing safety detectors.Source: §7 · p. 8
  • A classifier trained on DIASAFETY provides a benchmark for evaluating context-sensitive safety in model releases.Source: §7 · p. 8
  • Evaluations of popular conversational models indicate that they remain affected by context-sensitive safety problems.Source: §7 · p. 9
  • The authors call for expanding the taxonomy and dataset and for modeling one or more unsafe categories to improve on their baseline classifiers.Source: §7 · p. 9

Limitations and Ethics

The authors identify limits in the dataset’s coverage, scale, and source material, as well as possible annotation and classifier biases. They also note a dual-use risk from the dataset’s unsafe examples.

Limitations and Ethics

  • The dataset covers single-turn text dialogue, leaving multimodal and multi-turn safety settings outside its scope.The authors note that other modalities can present safety problems and that multi-turn contexts may affect response safety.Source: Limitations and Ethics · p. 9
  • Sensitive Topic Continuation is listed in the taxonomy but excluded from data collection, and the taxonomy omits categories such as privacy and training-data leakage.Source: Limitations and Ethics · p. 9
  • The dataset is relatively small, may not cover all conversational unsafe behaviors, and may contain annotation errors; classifiers may also carry bias.Source: Limitations and Ethics · p. 9
  • The dataset uses model generations and publicly available data, whose contents the authors say do not represent their views or opinions.Source: Limitations and Ethics · p. 9
  • The authors acknowledge that the dataset could be exploited to create more context-level unsafe language, while judging its overall value to outweigh the risk.Source: Limitations and Ethics · p. 9

A Data Collection Details

The data collection process combines real-world dialogue material with generated responses, then filters the resulting examples. The paper also points to figures for its annotation interface and guidelines.

A.1 Real-world Conversations

  • To find real-world unsafe conversations in Reddit, the authors query keywords and remove posts from generally safe subreddits.They explain that context-sensitive unsafe data is rare in the Reddit corpus, particularly after moderation has removed many toxic or heavily down-voted posts.Source: §A.1 · p. 13

A.2 Machine-generated Data

  • For machine-generated examples, the authors source prompts from Reddit and existing datasets, then generate candidate responses with DialoGPT and Blenderbot.Prompt sources vary by category, including MedDialog for Unauthorized Expertise and mental-health posts for Risk Ignorance.Source: §A.2 · p. 13

A.3 Post-processing

  • Post-processing removes examples over 150 tokens and strips emojis, URLs, unusual symbols, and extra spaces; Detoxify filters replies with toxicity scores over 0.3.This filtering is intended to remove explicitly unsafe data detectable at the utterance level.Source: §A.3 · p. 13

B Annotation Guidelines

C Additional Classification Experiments

The additional classification experiments compare fine-grain classification with and without context and report complete coarse-grain results on the test set.

C.1 Fine-grain Classification

  • The fine-grain experiment compares single-model classification with and without context, while the section also describes a one-vs-all alternative.The one-vs-all approach trains separate models for the five unsafe categories and combines their predictions.Source: §C.1 · p. 13
  • Because context topics and styles vary by category, a single classifier may over-rely on those features; Risk Ignorance often concerns mental health.The authors contrast this category with others in which mental-health topics are rare.Source: §C.1 · p. 13

C.2 Coarse-grain Classification

  • The section reports the complete coarse-grain classification results on the test set in Table 6.The supplied passage does not specify the table’s individual results.Source: §C.2 · p. 13

D Additional Evaluation Results

The additional evaluation varies decoding parameters for Blenderbot-400M and DialoGPT-medium and presents a safety leaderboard. The case study describes model-specific response patterns, while the reproducibility section gives classifier settings and annotation-guideline examples.

E Case Study

  • In the case study, Plato-2 tends to use explicit insults, while Blenderbot has the best safety performance but sometimes agrees with unsafe context.Table 9 contains cherry-picked safe, utterance-level unsafe, and context-sensitive unsafe examples from the models.Source: E Case Study · p. 13, Table 9 · p. 18

F Reproducibility

  • The classifier uses RoBERTa-base, with category-specific hyperparameters selected on the validation set and mostly default Transformers settings.The authors report using GeForce RTX 2080 GPUs for training.Source: F Reproducibility · p. 14
  • The evaluation uses ParlAI for BBF and BAD, Hugging Face checkpoints for DialoGPT and Blenderbot, and PaddlePaddle and PaddleHub for Plato-2.Source: F Reproducibility · p. 14

D Additional Evaluation Results

  • Table 8’s leaderboard lists utterance-level unsafe proportion alongside category scores and an overall score for evaluated conversational models.Overall is the macro average of five context-sensitive unsafe categories and utterance-level unsafety.Source: D Additional Evaluation Results · p. 13, Table 8 · p. 15, F Reproducibility · p. 15
  • Figure 2 reports context-sensitive and total unsafe proportions for Blenderbot-400M and DialoGPT-medium across decoding parameters, with overall scores macro-averaged over five categories.The context-sensitive proportion is the smaller score and the total unsafe proportion the larger score for each bar.Source: D Additional Evaluation Results · p. 13, Figure 2 · p. 15

F Reproducibility

  • The annotation guidelines describe six tasks, including biased opinion, offending user, and risk ignorance at utterance and context levels.The supplied examples define biased replies to include expressing bias or negative impressions and, for task 1.b, deepening or underplaying biased ideas in the prompt.Source: F Reproducibility · p. 17

Related papers

Loading 2110.08466v2…