Source-linked AI summary
A Just and Comprehensive Strategy for Using NLP to Address Online Abuse
David Jurgens, Eshwar Chandrasekharan, Libby Hemphill
TL;DR
Online abuse is widespread, but NLP has focused on a narrow subset of abusive behavior. This position paper proposes broader abuse recognition, proactive prevention, and justice-oriented goals, while noting that community norms complicate moderation.
Problem
NLP has largely focused on a narrow range of abuse despite the existence of subtle, dangerous, and community-dependent forms that affect victims.
Method
The paper develops a position and research agenda spanning expanded abuse definitions, proactive interventions, new evaluation tasks, and justice frameworks.
Results
The paper concludes that NLP should address extreme and subtle abuse, intervene before harm occurs, and situate its work within a broader framework of justice.
Takeaways & Limitations
Healthy online communities require NLP to pursue equitable participation and community-aware responses rather than only identifying and removing abusive speech.
Takeaways & Limitations
Community norms can conflict with broader social acceptability, and norms within some communities may perpetuate inequalities.
Abstract
from arXiv · showhide
Online abusive behavior affects millions and the NLP community has attempted to mitigate this problem by developing technologies to detect abuse. However, current methods have largely focused on a narrow definition of abuse to detriment of victims who seek both validation and solutions. In this position paper, we argue that the community needs to make three substantive changes: (1) expanding our scope of problems to tackle both more subtle and more serious forms of abuse, (2) developing proactive technologies that counter or inhibit abuse before it harms, and (3) reframing our effort within a framework of justice to promote healthy communities.
1 Introduction
Online abuse is widespread and spans a broad spectrum, including subtle tactics and behaviors with severe consequences. The paper argues that NLP should broaden its scope, develop proactive interventions, and pursue justice-oriented goals for healthier communities.
- 40% of Internet users report experiencing online abuse, with underrepresented groups targeted even more often.
- Abuse ranges beyond explicit hate speech to tactics such as rape threats, gaslighting, First Amendment panic, and veiled insults.
- The paper calls for broader abuse definitions, proactive technologies that prevent harm, and a community-wide realignment toward justice.
2 Rethinking What Constitutes Abuse
NLP abuse research should recognize a wider range of dangerous, subtle, and community-dependent behaviors while accounting for classification consequences and deployment needs. The paper proposes new tasks and evaluation practices aligned with stakeholders, data constraints, and real-world use.
- 2 Rethinking What Constitutes Abuse: NLP should recognize infrequent physically dangerous abuse, common subtle abuse, and community norms rather than focusing on a narrow abuse scope.
- 2.1 Physically Threatening Online Abuse: Physically threatening behaviors may use innocuous or coded language, requiring contextual recognition of behaviors such as doxxing, swatting, and incitement.
- 2.2 Subtle Abuse: Subtle abuse includes condescension, minimization, benevolent stereotyping, and microaggressions, which can be emotionally harmful despite being implicit.
- 2.3 Community Norms Need to be Respected: Community norms vary, and systems that ignore context can remove content unnecessarily and erode trust, while some local norms may perpetuate broader inequalities.
- 2.4 Challenges for Creating New NLP Shared Tasks on Abusive Behavior: New shared tasks should define target-community guidelines, address scarce and potentially harmful data, identify end-users, and evaluate deployment outcomes beyond precision and recall.
3 Proactive Approaches for Abuse
Because existing abuse technologies usually intervene only after harm occurs, the paper advocates proactive computational strategies. These strategies include bystander interventions, pre-abuse de-escalation, and author nudges toward less offensive wording.
- Existing computational approaches are primarily reactive, whereas proactive technologies aim to prevent harm before abuse occurs.
- Computational bystander interventions can steer conversations away from abuse, and prior work found an empathy-based bot response produced long-term behavior change.
- Intervening before abuse may shift interactions more constructively, whereas post-escalation de-escalation may only prevent further abuse.
- Explainable detection, paraphrase, and style transfer can help authors identify offensive text and revise unintended offenses before posting.
4 Justice Frameworks for NLP
The paper frames healthy online communities around social, restorative, and procedural justice rather than merely eliminating unacceptable speech. These frameworks address participation, responses to wrongdoing, and fair procedures.
- Justice requires equitable participation, not merely the absence of certain speech acts.
- Social justice: Social justice uses capabilities to articulate the actions and opportunities an online community should support.Examples include expressing emotion and forming affiliations.
- Restorative justice: Restorative justice emphasizes repairing wrongdoing through consequences decided jointly by victims and transgressors.Responses may include banning, apology, or reconciliation, while considering both parties’ emotions.
- Restorative justice: NLP can help identify violated community norms through classification and explainable machine-learning techniques.
- Procedural justice: Procedural justice requires transparent and fair abuse-detection systems because compliance depends on perceived institutional and algorithmic legitimacy.
5 Conclusion
The paper proposes expanding abuse detection, developing proactive interventions, and placing NLP efforts within a justice framework to support productive online communities.
- NLP should address subtle and extreme abuse, prevent harm proactively, and pursue capabilities, restorative, and procedural justice.