Source-linked AI summary
ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation
Zi Lin, Zihan Wang, Yongqi Tong, Yangkun Wang, Yuxin Guo, Yujia Wang, Jingbo Shang
TL;DR
Toxicity detection has been studied mainly on social-media data, leaving real-world user-AI interactions and their implicit toxic content insufficiently explored. The paper introduces TOXICCHAT, a benchmark from real user-AI conversations, and finds that existing models generalize poorly to this domain, while benchmark fine-tuning improves detection. The authors position the dataset as a resource for more nuanced toxicity detection, while noting that proprietary chatbot data and exhaustive model optimization were outside scope.
Problem
Existing toxicity benchmarks mainly use social-media content, leaving the distinct and sometimes implicit toxicity of real-world user-AI interactions insufficiently explored.
Method
The paper constructs TOXICCHAT from real user-AI interactions collected through a Vicuna-based open-source chatbot and evaluates existing toxicity models on it.
Results
Existing toxicity models generalize poorly to TOXICCHAT, while fine-tuning on the benchmark notably improves detection of toxic interactions.
Takeaways & Limitations
TOXICCHAT provides a resource for developing more robust and nuanced toxicity detection models for real-world user-AI conversations.
Takeaways & Limitations
The study lacks access to proprietary data from higher-traffic chatbots and does not test all possible methods or pursue state-of-the-art benchmark performance.
Abstract
from arXiv · showhide
Despite remarkable advances that large language models have achieved in chatbots, maintaining a non-toxic user-AI interactive environment has become increasingly critical nowadays. However, previous efforts in toxicity detection have been mostly based on benchmarks derived from social media content, leaving the unique challenges inherent to real-world user-AI interactions insufficiently explored. In this work, we introduce ToxicChat, a novel benchmark based on real user queries from an open-source chatbot. This benchmark contains the rich, nuanced phenomena that can be tricky for current toxicity detection models to identify, revealing a significant domain difference compared to social media content. Our systematic evaluation of models trained on existing toxicity datasets has shown their shortcomings when applied to this unique domain of ToxicChat. Our work illuminates the potentially overlooked challenges of toxicity detection in real-world user-AI conversations. In the future, ToxicChat can be a valuable resource to drive further advancements toward building a safe and healthy environment for user-AI interactions.
1 Introduction
TOXICCHAT addresses the underexplored challenges of detecting toxicity in real-world user-AI conversations, where seemingly friendly questions or instructions can contain implicit toxicity. The benchmark and evaluations expose domain differences that limit existing toxicity detectors.
- Existing toxicity research mainly uses social-media or LLM-generated data, while real-world user-AI conversations remain insufficiently studied.
- User-AI interactions often involve questions or instructions whose seemingly friendly wording can conceal implicit toxic content.
- TOXICCHAT is a benchmark built from real user-AI chat data to study toxicity detection and user behavior in chatbot interactions.
- The dataset adds a special label for jailbreaking, where users try to induce toxic chatbot output through seemingly harmless prompts.
- Existing toxicity models generalize poorly to TOXICCHAT, particularly on jailbreaking cases, because their training data differ from real-world chatbot prompts.
- Fine-tuning on TOXICCHAT performs best on a real-world validation set even when prior toxicity datasets are ten times larger.
2 TOXICCHAT Construction
TOXICCHAT combines sampled Vicuna user-AI interactions with human and AI-assisted annotation, including a dedicated jailbreaking label. The resulting dataset contains 10,166 examples with a 7.10% toxicity rate and a 1.75% jailbreaking rate.
- Data collection: Data were sampled from a Vicuna-based online demo between March 30 and April 12, 2023, then cleaned, de-identified, and limited to first-round interactions.
- Annotation: Four fluent-English researchers assigned binary toxicity labels by strict majority vote, with each example reviewed by two annotators and conflicts discussed.
- Annotation: Jailbreaking queries were labeled separately because users deliberately tried to induce toxic chatbot content through seemingly harmless text.
- Human-AI collaborative annotation: Moderation APIs were inadequate for clean separation, so the framework filtered high-confidence non-toxic examples with Perspective API and retained human annotation for others.
- Dataset composition: 10,166 data points comprise the released TOXICCHAT benchmark after two collection stages, filtering, manual annotation, and removal of unsuitable data.
- Dataset statistics: 7.10% overall toxicity and 1.75% jailbreaking rates characterize the completed dataset.
3 Baseline Evaluations
The benchmark evaluates toxicity detectors on TOXICCHAT using precision, recall, F1, and jailbreaking recall, finding that prior tools and models perform poorly.
- TOXICCHAT evaluates toxicity detection with precision, recall, F1, and jailbreaking recall, where higher scores indicate better detection.Jailbreaking recall measures the percentage of jailbreaking text identified as toxic.
- Prior toxicity detection APIs and models fail to deliver good-quality results on TOXICCHAT.The evaluated systems include OpenAI Moderation, Perspective API, HateBERT, and ToxDectRoberta.
- The poor baseline performance indicates a large toxicity-domain discrepancy between real-world user-AI conversations and social media platforms.
4 Ablation Study
The ablation study shows that toxicity detectors trained on TOXICCHAT transfer poorly from other datasets, especially for jailbreaking, while chatbot responses provide an additional detection feature.
- 4.1 Domain Difference: Models trained on TOXICCHAT consistently outperform models trained on six prior toxicity datasets on the real-world validation set.This holds even when the TOXICCHAT training data is not particularly larger than the comparison datasets.
- 4.1 Domain Difference: A strong toxicity detector may not be equally effective at detecting jailbreaking cases.
- 4.1 Domain Difference: All out-of-domain datasets except Toxigen show decreased jailbreaking recall, suggesting jailbreaking detection transfers even less readily than toxicity detection.
- 4.2 Response Features: Chatbot responses can be used either alone or alongside user inputs as features for toxicity prediction.Responses may contain additional signals, such as refusal language or toxic generated content.
5 Conclusion
TOXICCHAT is a real-world user-AI toxicity benchmark that exposes domain-specific cases and supports improved detection through benchmark fine-tuning.
- TOXICCHAT is a real-world user-AI toxicity detection benchmark containing 10k user-AI conversations.
- The benchmark reveals special toxicity cases in user-AI interactions that differ from mainly social-media-based toxicity datasets.
- Fine-tuning on TOXICCHAT notably improves a baseline model’s ability to detect toxic interactions.
Limitations
The study identifies limitations concerning dataset coverage, annotation scale and cost, and the breadth of toxicity-evaluation methods tested.
- Dataset scope: The benchmark uses user queries from Vicuna rather than interactions from higher-traffic proprietary chatbots such as ChatGPT, Bard, or New Bing.The authors state that proprietary data could better reflect user demographics and languages but was inaccessible.
- Annotation: The dataset contains a few thousand human-model collaboratively annotated examples rather than tens or hundreds of thousands annotated entirely by humans.The authors note that larger-scale human annotation could improve quality but would require costly, trusted workers because of data sensitivity and quality requirements.
- Evaluation scope: The benchmark does not test all possible toxicity-evaluation methods or pursue best possible performance through extensive tuning or larger language models.The stated goal was to expose domain differences and raise community awareness rather than achieve state-of-the-art benchmark performance.
Ethical Considerations
The dataset-release discussion addresses provenance, privacy, access, language coverage, and risks arising from releasing toxic examples. It balances these concerns against the dataset’s research timeliness and uses safeguards including authorization, masking, author annotation, and an access form.
- Data provenance: The dataset is a random sample of first-turn user queries from the fully anonymous Vicuna online demo.The demo permits possible reuse and distribution of user dialogue data under a CC-BY license.
- Data provenance: The authors obtained authorization from the Vicuna team to conduct the study.
- Privacy and annotation: The authors annotated the data themselves, manually removed identifiable personal information, and obtained verbal agreements before annotation.These measures address possible PII and the sensitivity of toxic data without using crowdsourcing.
- Scope: The dataset is mainly English and may not represent non-English user-AI conversations.The release discussion specifies that more than 99% of the dataset consists of English conversations.
- Release risks: Releasing the dataset creates risks because toxic text could train toxic models or help attackers find queries that bypass models trained on it.The authors nevertheless argue that the dataset’s timeliness and research value outweigh these concerns, with a request-access form as an additional safeguard.
A Related Work
Prior toxicity-detection research has largely used social-media or generated text, while user-AI conversations differ in interaction style and may contain more implicit toxicity. The study compares TOXICCHAT with benchmarks and models spanning several domains.
- Existing datasets: Previous toxicity datasets mainly draw from social media platforms such as Twitter and Reddit, alongside website-based corpora.
- Comparison resources: The comparison uses six toxicity datasets: HSTA, MovieReview, Jigsaw, Toxigen, RealToxicPrompts, and ConvAbuse.
- Existing datasets: Toxigen consists of instances generated with GPT-3, whereas Real Toxicity Prompts contains 100K English web-text prompts with multiple toxicity-related features.
- Comparison resources: Baseline systems include OpenAI Moderation, Perspective API, ToxDectRoberta, and HateBERT, trained on mixtures of public toxicity data, user comments, tweets, or Reddit posts.
B Data Pre-processing
The preprocessing retains informative English prompts while removing noisy or privacy-sensitive content. It also masks personal information while preserving prompt syntax.
- Filtering: Prompts with non-ASCII characters and fewer than three words are removed as uninformative or excessively short.The filtering is motivated by the need for dialogue context in toxicity detection.
- Language filtering: Only English prompts are retained using the fastText language-identification model.The English focus follows the nature of Vicuna and the annotation resources available.
- Privacy processing: Personal information such as emails, phone numbers, and addresses is replaced with generic placeholders.The masking aims to reduce privacy risks while maintaining the prompts’ syntactic structure.
C Jailbreaking Example
Table 4 reports examples of jailbreaking cases.
- Table 4 reports some jailbreaking examples.
D Study on Moderation APIs
The study compares OpenAI Moderation and Perspective API on 720 trial data, examining confidence, toxicity separation, and human-annotation needs. Perspective API is selected as the collaborative model under a 1% absolute safe-rate setting.
- 720 trial data are evaluated with OpenAI Moderation and Perspective API, with Figure 2 showing their toxicity distributions.The figure reports the percentage of total data represented by each bar.
- Neither moderation API provides an absolute threshold that cleanly separates toxic from non-toxic examples.Model confidence is nevertheless described as relatively reliable for performance.
- At a 1% tolerable-error rate per bin, OpenAI Moderation can leave 40% of the data for automated handling, requiring human annotation for 60%.The pending human-annotation percentages vary with the chosen absolute safe rate.
- Perspective API is chosen as the collaborative model at a 1% absolute safe rate, using a threshold equal to 1e −1.43.The authors justify this choice by comparing the safe rate with inter-annotator agreement and the annotation burden.