Source-linked AI summary
Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contexts
Ashutosh Baheti, Maarten Sap, Alan Ritter, Mark Riedl
TL;DR
Context-dependent offensiveness makes it important to assess not only what dialogue models say, but whether they align with offensive statements. The authors create and analyze an annotated Reddit corpus, finding that humans and dialogue models agree with offensive comments more than safe ones. They also explore controllable generation to reduce contextually offensive responses.
Problem
Offensiveness can depend on conversational context, and existing systems may stop responding when offensive inputs are detected even though classifiers can produce false positives.
Source: §1 · pp. 1–2Method
Results
42% of human responses agreed with offensive comments versus 13% with safe comments, and neural dialogue models were more likely to agree with offensive comments.
Source: §1 · p. 2Takeaways & Limitations
For some applications, avoiding agreement with offensive statements may be preferable to shutting down output, while non-toxic counter-speech is a future possibility.
Source: §1 · p. 2The crowd annotation task had relatively low agreement levels.
Source: §3.2 · p. 3
Abstract
from arXiv · showhide
Dialogue models trained on human conversations inadvertently learn to generate toxic responses. In addition to producing explicitly offensive utterances, these models can also implicitly insult a group or individual by aligning themselves with an offensive statement. To better understand the dynamics of contextually offensive language, we investigate the stance of dialogue model responses in offensive Reddit conversations. Specifically, we create ToxiChat, a crowd-annotated dataset of 2,000 Reddit threads and model responses labeled with offensive language and stance. Our analysis reveals that 42% of human responses agree with toxic comments, whereas only 13% agree with safe comments. This undesirable behavior is learned by neural dialogue models, such as DialoGPT, which we show are two times more likely to agree with offensive comments. To enable automatic detection of offensive language, we fine-tuned transformer-based classifiers on ToxiChat that achieve 0.71 F1 for offensive labels and 0.53 Macro-F1 for stance labels. Finally, we quantify the effectiveness of controllable text generation (CTG) methods to mitigate the tendency of neural dialogue models to agree with offensive comments. Compared to the baseline, our best CTG model achieves a 19% reduction in agreement with offensive comments and produces 29% fewer offensive replies. Our work highlights the need for further efforts to characterize and analyze inappropriate behavior in dialogue models, in order to help make them safer. Our code and corpus are available at https://github.com/abaheti95/ToxiChat .
1 Introduction
Offensive language is context-dependent, and agreeing with an offensive statement can itself make a response offensive; blanket shutdown is problematic when classifiers produce false positives. The paper measures this behavior in human and model responses and explores controllable generation as a mitigation.
1 Introduction
- Context-dependent toxicity can make apparently innocuous text offensive in conversation, and classifier false positives make stopping output problematic.For some applications, avoiding agreement may be preferable; the authors also point to non-toxic counter-speech as a future possibility.Source: §1 · pp. 1–2
- 42% of human responses in offensive contexts agreed, compared with 13% of responses to safe comments.The introduction also reports that users were three times more likely to agree with offensive comments in an analysis of five million Reddit threads.Source: §1 · p. 2
- Neural dialogue models, including DialoGPT, GPT-3, and Blender, were more likely to agree with offensive comments.The paper presents this as a behavior learned by models from human conversations.Source: §1 · p. 2
- Initial controllable text generation experiments suggest domain-adaptive pretraining reduces contextually offensive responses but does not eliminate the problem.The authors identify controllable stance as needing further research.Source: §1 · p. 2
- The paper releases ToxiChat, a 2,000-conversation corpus annotated for targeted offensiveness and stance, and uses it to study and control model responses.It also presents the corpus as supporting classifier training and evaluation.Source: §1 · p. 2
2 Creating the TOXICHAT Corpus
ToxiChat annotations capture both offensiveness and stance in dialogue context, treating alignment with an offensive statement as a way a user or chatbot can become offensive. Model responses also receive grammatical and contextual plausibility annotations.
2 Creating the TOXICHAT Corpus
- The corpus annotation interface is designed to capture whether a response is offensive and whether it agrees with earlier comments.The authors frame both concepts as necessary to address problematic conversational responses.Source: §2 · p. 2
- The annotation scheme labels each utterance Offensive or Safe and its stance toward every preceding utterance as Agree, Disagree, or Neutral.Stance is judged in the context of the entire thread and sociocultural setting; aligning with an offensive statement is treated as potentially offensive.Source: §2 · p. 2
- Dialogue-model responses are additionally annotated for grammatical and contextual plausibility given the conversation.The paper points to an appendix screenshot of the annotation interface.Source: §2 · p. 2
3 Data Collection
The authors construct the dataset from Reddit threads, augment them with dialogue-model responses, and collect crowd annotations. Their sampling deliberately includes both random threads and threads predicted to contain offensive comments.
3 Data Collection
- The offensive-versus-safe annotation is deliberately limited to offensiveness, although safe comments may still be inappropriate, such as when they contain misleading information.Threads were drawn from all subreddits and from subreddits identified as toxic in prior studies or community reports.Source: §3 · pp. 2–3
- The 2,000-thread sample combines 1,000 randomly selected threads with 1,000 additional threads whose last comment was classifier-predicted offensive at probability ≥0.7.The selection classifier had approximately 85.4 Offend-label F1 on the SBIC development set.Source: §3 · p. 3
3.1 Generating Dialogue Model Responses
- The sampled threads are extended with DialoGPT, GPT-3, and Blender responses; generation settings differ across models.DialoGPT and GPT-3 use nucleus sampling with p = 0.9, while Blender uses beam search with beam size 10 and minimum sequence length 20.Source: §3.1 · p. 3
3.2 TOXICHAT Corpus Statistics
- Five crowd-workers annotate each thread, with Krippendorff’s α of 0.42 for offensiveness and 0.22 for stance.Pairwise agreement is 82.8% for offensive labels and 85.1% for stance labels; the authors attribute relatively low agreement to the task’s inherent complexity.Source: §3.2 · p. 3
- Aggregating worker labels yields F1 scores of 0.91 for offensive language and 0.94 for stance against an in-house annotator’s labels on 40 threads.A gold label is assigned when at least two of the five workers agree.Source: §3.2 · p. 4
4 Stance Dynamics in TOXICHAT
Responses in ToxiChat show agreement with offensive comments across human and model replies, alongside both direct and contextual offensiveness. The analysis also compares target groups and profanity across human, DialoGPT, and GPT-3 responses.
Directly vs Contextually Offensive Replies
- Most offensive responses were directly offensive, though contextual offensiveness was also nontrivial; GPT-3 had 389 offensive responses out of 2,000 versus DialoGPT’s 179.Overall, dialogue-model responses were less offensive than human replies.Source: §4 · p. 4, Figure 2 · p. 4
Agreement with Offensive vs Safe comments
- 41.62% of human responses agreed with offensive comments versus 12.89% with safe comments; both models were almost twice as likely to agree with offensive comments.The authors hypothesize that Reddit users may be more likely to engage with offensive comments when they agree, potentially creating an echo chamber.Source: §4 · p. 4, Figure 3 · p. 4
Target-Group Distribution
- Human responses targeted demographic groups and specific individuals, while model responses were more offensive toward individuals and women.Models more often directed personal attacks at individuals than offense toward a demographic group.Source: §4 · p. 4, Figure 4 · p. 5
Profanity in Model Responses
- Profanity appeared in 3.35% of DialoGPT’s offensive responses, 39.59% of GPT-3’s, and 66.47% of Reddit users’ offensive responses.The authors used Toxicity Triggers, a lexicon of 378 words, phrases, and regular expressions; filtering offensive training instances reduces profanity but does not eradicate offensive behavior.Source: §4 · pp. 4–5
5 Offensive Language and Stance Classification
The section defines offensive-language and stance classification tasks for dialogue threads, compares architectures and loss functions, and evaluates their performance. Full-thread DialoGPT outperforms models without global context, while stance classification remains difficult.
5 Offensive Language and Stance Classification
- The tasks predict an offensive-or-safe label for each utterance and a neutral, agree, or disagree stance for each utterance pair.Stance is predicted for every pair (ui, uj) where i < j.Source: §5 · p. 5
5.1 Model Architectures
- NBOW and BERT classify sentence or pair representations, while DGPT encodes the full thread and uses utterance representations for stance prediction.The DGPT stance classifier combines the two utterance representations by concatenation, subtraction, and element-wise multiplication.Source: §5.1 · p. 5
5.2 Loss Functions
- Because stance labels are imbalanced, the authors use weighted cross-entropy and also experiment with class-balanced focal loss.Agree and Disagree occur at about 1:10 and 1:40 relative to Neutral; weighted cross-entropy uses weights (1, 100, 100).Source: §5.2 · p. 5
5.3 Evaluation
- The evaluation uses a 70-15-15 train, development, and test split, reporting offensive F1 and stance per-class and macro-F1.Metrics are reported for all utterances, first utterances, and replies; stance scores also cover adjacent utterance pairs.Source: §5.3 · p. 6
5.4 Results and Analysis
- Full-thread DGPT outperforms BERT and NBOW models that lack global context.Stance models have low F1 for both Agree and Disagree labels.Source: §5.4 · p. 6, Table 1 · p. 6
- DGPT+ achieves 0.714 F1 on all utterances; its F1 is 0.673 on GPT-3 responses and 0.489 on DGPT responses.For stance, class-balanced focal loss achieves the highest overall Macro-F1, while weighted cross-entropy achieves the highest Agree-label F1 but performs poorly on Disagree.Source: §5.4 · p. 6, Table 2 · p. 6
6 Mitigating Offensive Behavior
The section tests whether controllable generation can reduce offensive replies and agreement with offensive comments. Neutral-stance DAPT reduces agreement in both automatic and human evaluations, but the mitigation remains imperfect.
6 Mitigating Offensive Behavior
- CTG controls target offensiveness, stance toward the immediately preceding comment, or both; models are tested on 500 threads ending in an offensive utterance.The experiments compare against DGPT-medium, GPT-3, and Blender using automatic and human evaluation.Source: §6 · p. 7, §6.1 · p. 7
6.1 Modeling, Training and Testing Details
- Training data comes from five million threads labeled by high-precision classifiers, with disagreeing responses discarded and the retained data split 95:5 into training and development sets.The sample contains about 100,000 offensive responses and 75,000 agreeing responses; DialoGPT-medium is fine-tuned for three epochs.Source: §6 · p. 7, §6.1 · p. 7
6.2 Automatic Evaluation
- In automatic evaluation, neutral-stance DAPT reduced agreement while generating fewer offensive responses; ATCON both also improved %Off and %Agree over DGPT, but by smaller margins.Pretrained DGPT and GPT-3 generated approximately 30% and 41% offensive responses, respectively, in offensive contexts.Source: §6.2 · pp. 7–8, Table 3 · p. 8
6.3 Human evaluation
- Human evaluation found DAPT produced the fewest agreeing and most neutral responses, but was slightly more offensive than Blender, which was least offensive and most agreeing.The authors conclude that DAPT is the most effective tested technique for reducing offensive behavior, though it remains far from perfect.Source: §6.3 · p. 8, Table 4 · p. 8
7 Related Work
Prior toxicity research often treats posts in isolation or studies adversarial and single-turn settings. This work instead examines stance in multi-turn offensive Reddit conversations and mitigates contextual offensiveness through neutral and safe responses.
Identifying Toxicity
- Many prior toxicity-detection methods analyze isolated posts or comments and are ill-equipped for conversationally contextual offensiveness.The cited work includes studies of contextual offensive language and context-sensitive detection in adversarial or single-turn settings.Source: §7 · pp. 8–9
- The study focuses on stance dynamics in dialogue-model responses to offensive Reddit conversations spanning more than one turn.Earlier contextual studies used adversarial human-bot conversations or labeled single-turn datasets.Source: §7 · p. 9
Inappropriate Language Mitigation
- Unlike approaches that discourage inappropriate generation or avoid sensitive topics, this work fine-tunes models to produce neutral and safe responses in offensive contexts.Related mitigation work includes bias-focused training, toxicity-classifier reinforcement learning, and changing the conversation topic.Source: §7 · p. 9
8 Conclusion
The paper introduces TOXICHAT to study stance and targeted offensiveness in offensive conversations, then uses it to analyze human and model responses. Its analyses find greater agreement in offensive contexts, while safe and neutral DAPT mitigates that tendency to some extent.
8 Conclusion
- TOXICHAT contains 2,000 Reddit conversations augmented with two dialogue models’ responses and crowd-annotated for targeted offensiveness and stance.Classifiers trained on the corpus can automatically evaluate conversations containing contextually offensive language.Source: §8 · p. 9
- The analyses find that Reddit users agree more with offensive contexts, and dialogue models learn to agree more frequently in those contexts.The authors suggest echo chambers could explain the user pattern.Source: §8 · p. 9
- Fine-tuning on cleaner data containing safe and neutral responses can mitigate this agreement tendency to some extent.The authors identify further research on offensive-context detection and nonprovocative counter-speech as important for strengthening dialogue safety.Source: §8 · p. 9
9 Societal and Ethical Considerations
The paper frames its work as addressing neural-model safety and conversational civility, and reports ethical review and worker compensation details. Its preprocessing also limits comment lengths and restricts posts to textual data.
9 Societal and Ethical Considerations
- The study was conducted with Georgia Tech IRB approval, and the paper reports payments for crowd and in-house annotators.Mechanical Turk workers received $0.8 per HIT, with quality bonuses and an estimated hourly pay of $12.26; in-house annotators received $13 per hour.Source: §9 · p. 9
- The authors caution that classifiers trained on this dataset are fallible and should be used carefully.The paper frames the work around neural-model safety and making conversations more civil.Source: §9 · p. 9
A Data Preprocessing
- Preprocessing replaced thread URLs with a special token, limited posts to 70 words and comments to 50, and retained only posts containing text.Source: A Data Preprocessing · p. 14
B Offensive SubReddit Data Collection
The corpus is collected from selected offensive subreddits, compared with SemEval-2017 Task 8, and used in experiments with specified model-training settings.
B Offensive SubReddit Data Collection
- Threads with a last offensive comment were sampled from 10 subreddits selected for offensive-thread proportion and availability.The sampling used a BERT offensive-comment classifier trained on SBIC, with P(offensive) ≥ 0.7.Source: B Offensive SubReddit Data Collection · p. 14
D Model Implementation Details
- Experiments use Adam with a 2 × 10−5 learning rate; NBOW trains for 30 epochs, while BERT and DGPT train for 12.DGPT with class-balanced focal loss for stance used a 5 × 10−5 learning rate and 16 epochs; checkpoints were selected using task-specific development-set F1.Source: D Model Implementation Details · p. 14
C Comparison with SemEval-2017
- Compared with SemEval-2017 Task 8, TOXICHAT has more and longer sentences, longer threads, more stance labels, and annotations for all utterance pairs.SemEval-2017 annotates stance only with respect to the first comment in each thread.Source: Table 5 · p. 14, C Comparison with SemEval-2017 · p. 14
E Classifier Analysis on Reddit
Across six months of analyzed Reddit data, classifier predictions show more agreement with offensive than safe contexts. The analysis also reports differing rates of ambiguous stance predictions across those contexts.
E Classifier Analysis on Reddit
- About 9% of safe-context stance predictions are ambiguous, compared with about 27% of offensive-context predictions.The reported distributions are based on classifier predictions for 5M Reddit threads downloaded for controlled text generation experiments.Source: E Classifier Analysis on Reddit · p. 14
- Reddit users agree 3× more with offensive contexts than safe contexts across the six months analyzed.Figure 6 plots stance-label distributions for replies to offensive versus safe comments, using high-precision predictions for Agree, Disagree, and Neutral and marking low-precision predictions Ambiguous.Source: E Classifier Analysis on Reddit · p. 14, Figure 6 · p. 15