Source-linked AI summary
Conversations Gone Awry: Detecting Early Signs of Conversational Failure
Justine Zhang, Jonathan P. Chang, Cristian Danescu-Niculescu-Mizil, Lucas Dixon, Yiqing Hua, Nithum Thain, Dario Taraborelli
TL;DR
Online systems need to identify when civil conversations are at risk of becoming antisocial before attacks occur. The paper analyzes politeness and rhetorical cues in initial exchanges using a controlled Wikipedia dataset, finding that these cues provide early signal of later derailment and that automated detection is feasible. Its correlational, precision-focused, Wikipedia-centered design limits causal interpretation, recall, and generalization to other venues.
Problem
The paper asks whether warning signs can predict a civil conversation’s derailment into antisocial behavior early enough for the conversation to remain salvageable.
Method
The authors analyze politeness strategies and unsupervised rhetorical prompts in initial exchanges within a controlled, crowdsourced Wikipedia dataset.
Results
Pragmatic cues from the first exchange provide signal about subsequent derailment, and the framework recovers some human predictive intuition; human accuracy is 72%.
Takeaways & Limitations
The findings support the feasibility of automatically detecting early warning signs of future misbehavior in collaborative interactions.
Takeaways & Limitations
The correlational, precision-focused procedure may miss subtle attacks and does not establish causal mechanisms; the authors also propose testing other public discussion venues.
Abstract
from arXiv · showhide
One of the main challenges online social systems face is the prevalence of antisocial behavior, such as harassment and personal attacks. In this work, we introduce the task of predicting from the very start of a conversation whether it will get out of hand. As opposed to detecting undesirable behavior after the fact, this task aims to enable early, actionable prediction at a time when the conversation might still be salvaged. To this end, we develop a framework for capturing pragmatic devices---such as politeness strategies and rhetorical prompts---used to start a conversation, and analyze their relation to its future trajectory. Applying this framework in a controlled setting, we demonstrate the feasibility of detecting early warning signs of antisocial behavior in online discussions.
1 Introduction
The paper introduces early prediction of whether initially civil online conversations will derail into personal attacks, rather than detecting antisocial behavior after it occurs. It studies pragmatic and rhetorical cues in opening exchanges within a controlled Wikipedia setting.
- The task is to detect warning signs that a civil conversation may derail while it remains potentially salvageable.
- 72% human accuracy on the balanced task demonstrates feasibility while showing that prediction is far from trivial.
- They develop a framework covering politeness strategies and additional rhetorical prompts used to initiate conversations.
- The authors study conversational derailment from within, where previously civil participants begin attacking each other.
- The controlled dataset contains initially civil conversations that either remain healthy or later derail into personal attacks, with topic effects addressed through causal-inference techniques.
- Pragmatic cues from the first comment-reply pair provide signal about later derailment: hedged remarks are associated with more sustained civility than forceful questions or direct address.
- The work frames automated warning-sign detection as a basis for further study of factors that derail or sustain healthy collaborative conversations.
2 Further Related Work
Related work has examined antisocial behavior and conversation trajectories, but this paper shifts attention toward linguistic signals of future conversational outcomes.
- Prior studies examine disruptive interactions including aggression, harassment, and bullying, along with effects on engagement, discussion quality, teamwork, and persuasion.
- Earlier conversation-trajectory models largely focus on structural aspects, whereas this work seeks linguistic signals of future outcomes.
3 Finding Conversations Gone Awry
The authors construct a controlled Wikipedia dataset by reconstructing conversations, machine-filtering candidates, crowdsourcing labels, and pairing comparable discussions. The resulting collection contains 1,270 paired conversations for studying derailment from initially civil openings.
- Wikipedia revision histories are translated into structured conversations, yielding roughly 50 million conversations across 16 million talk pages.
- A toxicity classifier preselects candidate conversations, after which crowdsourcing vets labels and constructs the controlled dataset.
- Candidates are restricted to conversations that begin civilly, continue beyond the first exchange, and focus on attacks initiated by a participant.
- The toxicity score t is used to identify conversations civil throughout with t < 0.4 and conversations that become toxic after the initial civil exchange.
- Crowdsourcing Job 1 selects conversations containing contextual personal attacks, retaining the 14% unanimously judged by three annotators to end in such an attack.
- Job 2 removes candidates that were not civil before the attack and verifies that on-track conversations remain civil throughout.
- Awry-turning conversations are paired with on-track conversations from the same talk page, using closest timing when multiple pairings exist.
- The final dataset contains 1,270 paired conversations spanning 582 talk pages, with an average conversation length of 4.6 comments.
4 Capturing Pragmatic Devices
The framework captures politeness strategies and domain-specific rhetorical prompt types from the initial exchange, using predefined linguistic markers and unsupervised representations of recurring phrasings and replies.
- Politeness strategies: The framework examines politeness strategies, including positive and negative politeness, as pragmatic markers of conversational trajectory.Positive strategies include gratitude, greetings, and “please”; negative strategies include indirectness and uncertainty, such as hedging.
- Conversation prompts: It also captures domain-specific conversation prompts, such as invitations to collaborate or statements of dispute, discovered without supervision.The goal is to identify rhetorical patterns that initiate conversations and may signal different levels of tension.
- Prompt-type inference: Prompt types are derived from low-rank reply and prompt representations of recurring dependency-parse phrasings, then clustered by similarity in the reply space.New comments are represented by averaging constituent-phrasing vectors and assigning the result to a cluster.
- Prompt-type inference: The method extracts six prompt types from an unlabeled Wikipedia talk-page corpus and assigns comments that are distant from all centroids to an ignored null type.The prompt types reflect recurring conversation-starter signatures across topics and contexts.
- Prompt-type inference: Using more prompt types produced finer-grained categories and qualitatively similar results, but sparse assignments reduced statistical power on the small dataset.This motivated choosing six prompt types for the reported analyses.
5 Analysis
The analysis links pragmatic cues in the first exchange to whether conversations later turn awry, with direct language associated with future attacks and politeness or coordination cues associated with staying on track.
- Initial-comment patterns: Direct questions and sentence-initial you start awry-turning conversations significantly more often than on-track conversations, with both comparisons at p < 0.001.The analysis quantifies each marker using its log-odds ratio in the initial exchange of awry-turning versus on-track conversations.
- Initial-comment patterns: On-track conversations more often begin with gratitude, greetings, coordination prompts, hedges, and opinion prompts than awry-turning conversations.These markers represent positive or negative politeness and active efforts to soften impositions or foster constructive teamwork.
- Reply patterns: First replies in awry-turning conversations contain more second-person pronouns, whereas on-track replies contain more sentence-initial I/We, with both differences at p < 0.001.The contrasting pronoun patterns are interpreted as pushing back against versus working with the initiator.
- Interlocutor behaviors: In attacker-initiated conversations, both participants exhibit direct markers associated with future attacks, and non-attacking repliers particularly often use second-person pronouns.The reported attacker-initiated subset contains 608 conversations.
- Interpretation: The discovered markers are tied to future trajectory, and their inference methodology is unsupervised and applicable beyond Wikipedia.Figure 2 distinguishes first- and second-comment effects, significance, effect sizes, and attacker versus non-attacker behavior.
6 Predicting Future Attacks
The paper evaluates whether pragmatic features from a conversation’s first comment-reply pair can predict future personal attacks in a balanced, page-held-out task. Pragmatic features achieve above-chance accuracy and retain predictive power for attacks occurring after the fourth comment.
- Task and evaluation: The prediction task uses only the first exchange and evaluates logistic regression with leave-one-page-out cross-validation to prevent page-specific information leakage.Each test fold holds out all conversation pairs from one talk page.
- Results: Pragmatic prompt features achieve 59.2% accuracy, politeness-strategy features achieve 60.5%, and their combination reaches 61.6% accuracy.The task is balanced and asks which of two conversations will eventually lead to a personal attack.
- Results: Pragmatic features retain 67.4% accuracy when the first personal attack occurs after the fourth comment, while sentiment and bag-of-words baselines fall to random chance.This evaluates prediction beyond the immediate next couple of comments on 282 conversation pairs.
- Conclusion: The results demonstrate feasibility of reconstructing some human intuition about whether an ostensibly civil conversation will eventually turn awry.The conclusion concerns early prediction from linguistic properties of the initial exchange.
7 Conclusions and Future Work
The paper frames conversational derailment as a prediction task, using pragmatic and rhetorical devices in initial exchanges to study future attacks. It demonstrates feasibility while identifying causal, coverage, interaction-span, and cross-domain limitations.
- Conclusions: The framework analyzes how politeness strategies and domain-specific rhetorical prompts in initial exchanges relate to whether civil discussions later derail into personal attacks.
- Conclusions: 72% human accuracy and computational recovery of some human intuition demonstrate the feasibility of predicting future attacks from a controlled conversational setting.
- Limitations: Correlational analyses do not identify causal mechanisms of derailment; randomized experiments could address this limitation.
- Limitations: Precision-focused data collection may miss subtle attacks overlooked by the toxicity classifier, while additional antisocial-behavior indicators could broaden the studied attack types.
- Future Work: The analysis focuses on the first exchange, leaving more comprehensive interaction modeling and sequential prediction of escalation for future work.
- Future Work: Misclassified interactions can remain civil through level-headed interlocutors and explicit acts of reparation, showing that initial apparent risk does not determine the eventual trajectory.
A Details on annotation procedure
Constructing labels for personal attacks required a concrete conceptualization of a subjective phenomenon and an iterative process combining qualitative examples with crowdsourced filtering.
- Annotation Procedure: The labeling challenge arose because personal attacks are complex, subjective, and open to multiple definitions.
- Annotation Procedure: The researchers developed a well-defined conceptualization and an accurate discovery procedure to analyze conversational derailment at Wikipedia scale and diversity.
- Annotation Procedure: An initial qualitative investigation produced a seed set of conversational failures that informed the subsequent crowdsourced filtering procedure.
Initial qualitative investigation
The initial investigation used automatically selected candidate conversations, manual inspection, and discussions with Wikimedia anti-harassment experts to define the target phenomenon and assess human predictability.
- Initial Candidate Selection: A random Wikipedia talk-page sample yielded conversations judged initially civil by a trained classifier but containing a later toxic comment.
- Initial Candidate Selection: Manual inspection identified divergences between classifier outputs and human judgments, including sarcasm and self-deprecation that used aggressive language collegially.
- Target Phenomenon: The researchers centered the study on personal attacks from within, where one participant in an ostensibly civil exchange later attacks the other.
- Paired Conversations: Each seed conversation was paired with an on-track counterpart from the same talk page using the candidate selection procedure.
- Human Performance: 72% majority-vote accuracy was achieved by volunteer human annotators predicting which conversation in 100 pairs would lead to a personal attack from the initial exchange.
- Crowdsourcing Design: The crowdsourcing procedure was iteratively formulated from the seed set to scale manual filtering across the dataset.
Crowdsourced filtering
Crowdsourced filtering used two jobs to identify attacks and enforce civil starts or sustained civility, with quality controls and consensus-based selection producing a paired dataset.
- Filtering Jobs: The two-job procedure first selected conversations ending in personal attacks, then enforced that awry-turning conversations started civil and on-track conversations remained civil.
- Quality Control: 82 seed conversations served as balanced test questions, and workers missing at least 20% were automatically blocked.
- Annotation Criteria: Annotators specified attack targets to distinguish personal attacks toward interlocutors from attacks toward groups or Wikipedia generally.
- Annotation Criteria: 804 candidate on-track conversations were added to balance the many attacks in the candidate set and calibrate annotator judgments.
- Consensus: Unanimous agreement among three annotators produced higher-quality labels than majority voting by omitting ambiguous cases, sacrificing recall for precision.
- Annotation Criteria: Annotators were instructed not to confuse civil disagreement with toxicity, which was defined as rude, insulting, or disrespectful behavior likely to disrupt participation.
- Annotation Criteria: Because full conversations overwhelmed annotators, the procedure adjusted the amount of displayed text; labels required at least three annotators to identify toxicity across snippets.
- Overall Flow: 3,218 candidates were reduced to 435 unanimously labeled attack-ending conversations, 30 pairs were removed in Job 2, and the final dataset contained 1,270 paired conversations.
B Further examples of prompt types
Table 4 presents representative comments for each automatically extracted prompt type alongside typical replies produced by the methodology.
- Table 4 pairs representative comments with automatically extracted prompt types and typical replies prompted by each type.The examples are produced using the unsupervised methodology described in Section 4.
- Bolded text marks common prompt and reply phrasings identified by the framework in the respective examples.
- Comments and replies shown in the same row do not necessarily correspond to one another.