Source-linked AI summary
Defining and Detecting Toxicity on Social Media: Context and Knowledge are Key
Amit Sheth, Valerie L. Shalin, Ugur Kursuncu
TL;DR
Toxicity detection is difficult because interpretation depends on context beyond explicit content. The paper frames toxicity through psychological and social dimensions and incorporates explicit knowledge into statistical learning; it reports that about 8% of conversations contain profanity, though user intent may not be toxic.
Problem
Method
The paper frames toxicity through psychological and social dimensions and incorporates explicit knowledge into a statistical learning approach.
Source: §1 · p. 2, §3.3 · p. 12, §5 · p. 16Results
About 8% of conversations contain profanity and curse words, but users’ intentions may not necessarily be toxic.
Source: §3.1 · p. 9Takeaways & Limitations
The framework emphasizes analyzing content, individuals, and communities, with knowledge representation supporting toxicity detection.
Source: §5 · p. 16
Abstract
from arXiv · showhide
Online platforms have become an increasingly prominent means of communication. Despite the obvious benefits to the expanded distribution of content, the last decade has resulted in disturbing toxic communication, such as cyberbullying and harassment. Nevertheless, detecting online toxicity is challenging due to its multi-dimensional, context sensitive nature. As exposure to online toxicity can have serious social consequences, reliable models and algorithms are required for detecting and analyzing such communication across the vast and growing space of social media. In this paper, we draw on psychological and social theory to define toxicity. Then, we provide an approach that identifies multiple dimensions of toxicity and incorporates explicit knowledge in a statistical learning algorithm to resolve ambiguity across such dimensions.
1. Introduction
Online platforms distribute information widely, but social media has also amplified harmful content, and toxicity detection is difficult because it depends on context beyond explicit content. The paper sets out to identify the psychological and social dimensions of toxicity, assess computational limitations, and outline a knowledge-driven technical approach.
1. Introduction
- Social media has expanded the reach of harmful content, while effective toxicity detection remains challenging because it requires context beyond explicit content.The paper also notes that human moderation has produced relatively unsatisfactory outcomes.Source: §1 · p. 2
- The paper identifies psychological and social dimensions of toxicity as a central goal.Source: §1 · p. 2
- The paper also aims to identify limitations of contemporary computational approaches and outline a knowledge-driven technical approach.Source: §1 · p. 2
2. A PsychoSocial problem Meets Computation
Toxicity extends beyond offensive words: its interpretation depends on cultural, psychological, social, and interactional context. The section shows how content, decision criteria, source, target, and group relationships each shape what can be identified as toxic.
2. A PsychoSocial problem Meets Computation
- The paper extends toxicity beyond threats, obscenity, insults, and identity-based hate to include harassment and socially disruptive persuasion, arguing culture informs interpretation.Source: §2 · p. 3
2.1. Content Analysis
- Slang, irony, humor, and multimodal content make isolated lexical judgments unreliable; words and images can acquire meaning from cultural and interactional context.An apparently offensive term may be benign in religious text, while a cheerful caption can accompany an embarrassing image.Source: §2.1 · pp. 3–4
2.2. Culturally Bound Decision Criteria
- Toxicity thresholds vary with annotators’ cultural interpretations, while class imbalance can make lower thresholds yield many more false positives than true positives.Datasets also reflect population-specific cultural practices, limiting generalization across groups.Source: §2.2 · pp. 4–5
2.3. Identifying the toxic source
- Bullying is defined by a source’s intent to hurt or harm, but identifying a toxic source requires evidence aggregated across the candidate’s content.Source: §2.3 · p. 5
2.4. Identifying the Target
- A target’s experienced harm differs from the source’s intent, and harassment is bullying involving a protected class.The relevant harm can include offense, reputational loss, manipulation, embarrassment, or other effects from the recipient’s perspective.Source: §2.4 · p. 6
2.5. Participants’ Relationship and Group Membership
- Friendship, power differences, group membership, and exchange history shape interpretation; a potentially toxic message should be assessed in broader historical and network context.Repeated messages and group dynamics can alter how an episode appears, while expanded context can introduce conflicting indicators and stale content.Source: §2.5 · pp. 6–7
3. Technical Challenges to Automated Detection of Toxic Language
Toxicity detection requires more than classifying isolated posts: interpretation depends on social context, domain expertise, curated data, and explicit knowledge. The section illustrates these needs through cursing, extremism, harassment, and corpus design.
3. Technical Challenges to Automated Detection of Toxic Language
- Toxicity detection is an interdisciplinary problem, while web-trained models can inherit bias and cause harm, raising usability and safety concerns.The section argues that classification should draw on theory, empirical models, and knowledge rather than content processing alone.Source: §3 · pp. 7–8
3.1. Need for Empirical Models
- Domain expertise is scarce, and available empirical models are impoverished, motivating structured or semi-structured models that machines can use.The authors use cursing, extremism, and harassment to illustrate the need for better models of content.Source: §3.1 · p. 8
- About 8% of studied Twitter conversations contained profanity, but intention, emotion, and social context affect whether cursing is toxic.The study found cursing could accompany positive or negative emotions and occurred more when people woke up or in relaxed virtual environments.Source: §3.1 · p. 9
- The Islamist-extremism model distinguishes religion, ideology, and hate because “jihad” can mean spiritual struggle or intent to harm.The authors argue that religious knowledge can help reduce ambiguity, false alarms, and unfairness.Source: §3.1 · p. 9
- A five-class harassment approach covers sexual, racial, appearance-related, intellectual, and political content, including ambiguous words and idioms.The authors argue that multiple subclasses matter because perpetrators can target victims in more than one way.Source: §3.1 · p. 10
3.2. Need for Curated Corpora
- Keyword-based corpus collection can mislabel playful sarcasm or religious references, while extremism models depend on both professionally judged positives and suitable distractors.A balanced corpus can still misrepresent natural class imbalance, so even high precision may yield many false alarms.Source: §3.2 · p. 10
- The ALONE corpus captures exchanges between identified high-school participants, their interaction history, multimodal content, and network structure.Its public-account-only access boundary accompanies an approximation of realistic benign-to-toxic class balance.Source: §3.2 · p. 11
3.3. Need for Computationally Accessible External Knowledge
- External knowledge can contextualize multiple dimensions of toxicity, including personal semantics, source intention, interaction history, and recipient emotion.The authors argue that omitting harm-related knowledge contributes to annotator disagreement and poor, biased classification.Source: §3.3 · pp. 11–13
4. A Knowledge-enhanced Socio-technical Approach to Toxicity Detection
The proposed approach models toxicity as a personalized, evolving social exchange and infuses explicit knowledge into learning to address ambiguity across perspectives and levels of representation.
4. A Knowledge-enhanced Socio-technical Approach to Toxicity Detection
- Toxicity detection involves identifying toxic sources and vulnerable victims, whose differing belief systems shape how content is interpreted.The framework uses Knowledge-infused Learning to model meanings from different perspectives and categorizes infusion as shallow, semi-deep, or deep.Source: §4 · pp. 13–14
- The framework calls for dynamic hybrid models across modalities and content, individual, and community levels, with experimentally controlled data to support validation.It also allows individual-level details to change through network interaction.Source: §4 · p. 14
- Explicit structural relations in knowledge graphs provide context that statistical learning algorithms can incorporate to enhance latent contextual space, adjusting emphasis on sparse-but-essential and irrelevant-but-frequent terms and concepts, boosting recall without reducing precision.Source: §4 · p. 15
- Deep infusion merges knowledge embeddings with latent data representations during learning, while prior work suggests infusion can mitigate unfair outcomes by reducing false positives.The authors describe deep infusion as an open research area.Source: §4 · p. 15
5. Conclusion
The paper concludes that toxicity detection should move beyond conventional content analysis by modeling multiple dimensions and levels and incorporating explicit knowledge into statistical learning.
5. Conclusion
- The proposed theory-founded framework analyzes content, individuals, and communities while using explicit knowledge in statistical learning to address ambiguity.The conclusion presents knowledge representation and infusion as an emergent solution for toxicity detection and related problems.Source: §5 · p. 16