PaperlayerView exact revision on arXiv

Source-linked AI summary

A Framework of Severity for Harmful Content Online

Morgan Klaus Scheuerman, Jialun Aaron Jiang, Casey Fiesler, Jed R. Brubaker

arXiv:2108.04401v2cs.HC

TL;DR

Online harms are diverse and vary in severity, but existing frameworks offer limited support for comparing and prioritizing them. The paper develops a grounded-theory framework from interviews and card sorting with 52 participants, identifying four Types of Harm and eight Dimensions of Severity. The framework supports analysis and prioritization of overlapping harms, although it does not quantify content-level severity and is geographically limited.

  • Problem

    Existing online-harm frameworks do not adequately explain relative severity or support prioritization across diverse, overlapping harms.

  • Method

    The authors used grounded-theory analysis of interviews and card-sorting activities with 52 expert and general-population participants.

  • Results

    The framework identifies four Types of Harm and eight Dimensions of Severity for understanding overlapping, contextual online harms.

  • Takeaways & Limitations

    Researchers, practitioners, and policymakers can use the framework to analyze specific harms and prioritize policies addressing multiple harms.

  • Takeaways & Limitations

    The study does not quantify content-level severity differences and its sample is concentrated in North America and South Asia.

Abstract

from arXiv · show

The proliferation of harmful content on online social media platforms has necessitated empirical understandings of experiences of harm online and the development of practices for harm mitigation. Both understandings of harm and approaches to mitigating that harm, often through content moderation, have implicitly embedded frameworks of prioritization - what forms of harm should be researched, how policy on harmful content should be implemented, and how harmful content should be moderated. To aid efforts of better understanding the variety of online harms, how they relate to one another, and how to prioritize harms relevant to research, policy, and practice, we present a theoretical framework of severity for harmful online content. By employing a grounded theory approach, we developed a framework of severity based on interviews and card-sorting activities conducted with 52 participants over the course of ten months. Through our analysis, we identified four Types of Harm (physical, emotional, relational, and financial) and eight Dimensions along which the severity of harm can be understood (perspectives, intent, agency, experience, scale, urgency, vulnerability, sphere). We describe how our framework can be applied to both research and policy settings towards deeper understandings of specific forms of harm (e.g., harassment) and prioritization frameworks when implementing policies encompassing many forms of harm.

1 Introduction

Online harms vary in kind, affected people, and severity, making simple labels such as harassment insufficient for research and moderation. The study develops an empirically grounded framework to compare these harms and support prioritization.

  • 1 Introduction: Gamergate involved multiple kinds of harm at different severity levels, affecting people differently through threats, disrupted events, and misconduct accusations.The example contrasts prolonged rape and death threats with accusations of journalistic misconduct.
  • 1 Introduction: Online harms span hate speech, violent imagery, misinformation, harassment, bullying, and other contextual experiences that remain unevenly addressed in social computing.
  • 1 Introduction: Recognizing that behavior is harmful is insufficient; researchers and moderators also need to understand how severe harms are relative to one another.
  • 1 Introduction: The study uses grounded theory, interviews, and card sorting with experts and general users to identify what makes online harms more or less severe.The data were collected over ten months from 52 participants, including 40 experts and 12 general-population users.
  • 1 Introduction: The resulting framework identifies four Types of Harm and eight Dimensions of Severity for analyzing complex, contextual, and overlapping harms.The Types are physical, emotional, relational, and financial; the Dimensions include perspectives, intent, agency, experience, scale, urgency, vulnerability, and sphere.

2 Related Work

Prior social-computing research documents diverse online harms and moderation practices but lacks a framework for assessing their relative severity. The paper positions severity-based prioritization as a way to improve harm mitigation under finite resources.

  • 2 Related Work: Social-computing research has focused especially on interpersonal harms such as bullying, hate speech, and harassment, while also examining technical and content-based harms.
  • 2 Related Work: Existing harm frameworks classify relationships among online harms but do not explain how to assess their severity or operationalize moderation priorities.
  • 2 Related Work: Content moderation includes volunteer, paid commercial, and automated practices, each addressing unwanted content at different organizational scales.
  • 2 Related Work: Moderating harmful content at scale is difficult because platforms must triage among different harms while balancing labor, engineering capacity, and disagreement about appropriate mitigation.
  • 2 Related Work: A severity framework could guide prioritization by helping moderators handle the worst content first and focus automated-moderation efforts.

3 Assessing the Severity of Harm in Other Domains

Other domains use severity frameworks to prioritize responses, allocate scarce resources, and assess impact. Their practical failures also show that formal prioritization systems can break down in implementation.

  • 3 Assessing the Severity of Harm in Other Domains: Prioritization frameworks in other domains inform decisions about resource allocation, impact assessment, and which harms or cases receive attention.
  • 3 Assessing the Severity of Harm in Other Domains: Law uses punitive severity assessments in which sentencing reflects the harm caused and the intentionality of the offense.
  • 3 Assessing the Severity of Harm in Other Domains: Emergency dispatch uses time-sensitive triage that prioritizes danger to human beings as calls move through an operational response pipeline.
  • 3 Assessing the Severity of Harm in Other Domains: Mental-health assessment emphasizes persistence, symptom duration, debilitation, and physical threat to determine severity.
  • 3 Assessing the Severity of Harm in Other Domains: Prioritization systems often represent ideals that degrade in practice, weakening public trust and service efficacy when implementation fails.

4 Methods

The study combines interviews and card sorting with experts and general-population users, analyzing data iteratively through inductive and deductive comparison. Its design simplifies content categories and uses textual examples to address feasibility and ethical constraints.

  • 4 Methods: The study recruited 52 participants across expert and general-population groups to examine how people assess relationships among online harms.
  • 4 Methods: Researchers used grounded-theory-inspired interviews and card sorting, moving from open-ended data collection to iterative analysis and deductive validation.
  • 4 Methods: Data collection and analysis proceeded in inductive and deductive phases across expert subgroups and general-population participants, with comparisons across groups.
  • 4.1 Participant Recruitment: The sample included experts from content moderation and related fields alongside general-population social-media users recruited across North America, Western Europe, and South Asia.
  • 4.3 Content Categories for Card Sorting: Card sorting used simplified high-level content categories rather than a complete inventory of potentially harmful content.The simplification supported manageable discussions but excluded some categories.
  • 4.3 Content Categories for Card Sorting: Researchers used textual examples instead of actual harmful content because showing graphic or illegal material posed ethical and practical problems.

5 A Theoretical Framework of Severity

The framework separates four Types of Harm from eight contextual Dimensions of Severity that shape how severe a particular harm is judged.

  • The framework describes the thinking behind severity rankings rather than ranking specific card-sorting instances.
  • Four Types of Harm—physical, emotional, relational, and financial—organize the harms participants considered.
  • Eight Dimensions of Severity explain contextual factors that increase or decrease harm severity and shape comparisons between harms.

5.1 Types of Harm

Participants identified four intertwined Types of Harm—physical, emotional, relational, and financial—that can compound one another rather than forming isolated categories. Physical harm was often viewed as having the highest capacity for severity, while emotional, relational, and financial harms could also be severe and mutually reinforcing.

  • Physical Harm: Physical harm includes bodily injury, self-injury, sexual abuse, and death, and participants generally viewed it as having the highest capacity for severe harm.Participants nevertheless described physical harm as intersecting with emotional harm, especially in cases of physical abuse.
  • Emotional Harm: Emotional harm ranges from annoyance to stress or trauma, and its severity can be underestimated, particularly when emotional and physical harms intersect.Participants considered both spam and traumatic exposure to harmful imagery as emotional harms with different severity levels.
  • Relational Harm: Relational harm damages reputation or interpersonal, professional, and community relationships, often producing emotional and financial consequences.Non-consensual sexual imagery was described as potentially affecting emotions, income, and how others perceive the target.
  • Financial Harm: Financial harm includes material loss, lost digital assets, scams, account theft, bribery, and blackmail, and can accompany or result from other harms.Participants described sextortion as an example that can create financial burdens while encompassing multiple Types of Harm.
  • Types of Harm: Four Types of Harm emerged: physical, emotional, relational, and financial, with participants often describing them as intertwined or compounding.The framework classifies harms arising from online content, behaviors, and interactions.

5.2 Dimensions of Severity

The framework treats severity as contextual: eight Dimensions shape how participants evaluate harm across different situations. Perspectives, intent, and the roles of actors, viewers, and targets illustrate how severity judgments vary with impact, purpose, and position.

  • Dimensions of Severity: Eight Dimensions of Severity shape whether harm is judged more or less severe: Perspectives, Intent, Agency, Experience, Scale, Urgency, Vulnerability, and Sphere.These Dimensions are contextual factors that can increase or decrease perceived severity.
  • Vulnerability: Participants considered harm against children and animals more severe than comparable harm against adults, illustrating how vulnerability can compound other severity Dimensions.The framework allows each Dimension to vary in degree rather than treating it as uniformly present or absent.
  • Perspectives: Participants evaluated severity from actor, viewer, and target perspectives, with targets most associated with severe real-world outcomes and viewers often affected emotionally.The roles could shift, and viewers or targets could become actors in later interactions.
  • Perspectives: Live-streamed violence or sexual abuse was described as especially traumatic for viewers because the medium intensified exposure to graphic harm.Participants contrasted live video with other forms of harmful content when discussing severity for viewers.
  • Intent: Intent and impact were weighed differently across cases: some participants prioritized perceived purpose, while many prioritized impact for emotional or physical harms.Participants’ judgments about child nudity differed depending on whether benign intent or potential impact on the child received greater weight.

5.2.3 Agency (of the Target and/or Viewer)

Agency shaped severity judgments, with voluntary participation often lowering perceived severity and lack of choice increasing it. Participants’ disagreements also revealed that personal experience and cultural attitudes influenced how they assessed harmful content while they still attempted objective judgments.

  • Agency: Participants often judged harm as less severe when the person involved had chosen the activity and more severe when they lacked meaningful choice.Agency was discussed by 34 participants and shaped rankings of content such as drug sales, mass-murder coordination, bullying, suicide promotion, and sexual exploitation.
  • Agency: Agency judgments were contested, with some participants viewing non-consensual imagery or self-harm content as less severe because they attributed responsibility to the person involved.These minority views differed from the majority position on sexual abuse and non-consensual harm.
  • Experience: Participants’ personal experiences shaped severity assessments, sometimes making personally experienced or personally relevant harms seem more severe.Twenty-two participants explicitly discussed life experiences as influencing their judgments, and individual rankings sometimes diverged sharply from others.
  • Experience: Although participants recognized harm as subjective, they still tried to adopt an objective perspective, especially when assessing harms outside their own experience.This effort was particularly salient among experts discussing severity in industry contexts.

5.2.5 Scale (of the Harm)

Participants treated scale as a key severity factor: harms affecting more people, involving coordinated attacks, or threatening society broadly were generally judged more severe, while prevalence alone did not increase severity.

  • 5.2.5 Scale (of the Harm): Harms affecting society broadly, such as coordinated political attacks, were viewed as more severe than unpleasant interpersonal interactions.
  • 5.2.5 Scale (of the Harm): Hate speech was sometimes judged more severe than bullying or harassment because it can target larger vulnerable groups and incite broader violence.
  • 5.2.5 Scale (of the Harm): A coordinated flood of harassment toward one person was considered more severe than an isolated piece of content.
  • 5.2.5 Scale (of the Harm): Larger-scale harms, including coordinated attacks directly harming people, were generally ranked among the most severe categories.Scale referred to both the number of people affected and the number of actors attacking an individual or group.
  • 5.2.5 Scale (of the Harm): Content prevalence did not determine severity: low-prevalence child exploitation and terrorist coordination remained more severe than prevalent spam.
  • 5.2.6 Urgency (to Address the Harm): Urgency increased perceived severity when rapid action was needed to mitigate real-world harm, especially for child pornography.

5.2.7 Vulnerability (of the Target and/or Viewer)

Participants evaluated severity partly through target vulnerability and content medium, generally ranking harms to vulnerable groups and highly immersive or live media as more severe.

  • 5.2.7 Vulnerability (of the Target and/or Viewer): Direct harm to children was consistently ranked the most severe category because children were viewed as unable to protect themselves.
  • 5.2.7 Vulnerability (of the Target and/or Viewer): The more vulnerable participants considered a person or group, the more severe they judged harm against them, including harm targeting marginalized groups.
  • 5.2.8 Medium (of the Harmful Content): Visual content was often judged worse than text because images and videos were harder to avoid and could leave a lasting impression.
  • 5.2.8 Medium (of the Harmful Content): Live video was viewed as particularly severe because abuse was understood to be occurring immediately and required immediate action.
  • 5.2.8 Medium (of the Harmful Content): Perceived severity also depended on audiovisual fidelity, including whether video was in color, contained sound, or reached larger audiences.
  • 5.2.8 Medium (of the Harmful Content): The analysis did not include purely auditory content because participants did not discuss it, leaving audio relationships for future work.

5.2.9 Sphere (the Harm Occurs In)

Sphere shaped how participants understood harms occurring publicly or privately, but the study found no consistent severity ranking for the dimension.

  • 5.2.9 Sphere (the Harm Occurs In): Private harms such as sextortion could leave targets isolated and unsure how to seek help or respond.
  • 5.2.9 Sphere (the Harm Occurs In): Public harassment could restrict participation in online public spaces by producing a chilling effect.
  • 5.2.9 Sphere (the Harm Occurs In): Sphere was relevant to evaluating harms, but participants produced no consistent ranking of whether public or private settings were more severe.

6 Discussion

The framework rejects simplistic views of online harm by modeling severity through overlapping Types of Harm and contextual Dimensions. It is intended to support research, policy, and moderation prioritization without prescribing a single ranking of harms.

  • Framework implications: The framework treats physical, emotional, relational, and financial harms as potentially overlapping and compounding rather than mutually exclusive.For example, a harm may begin as financial and compound with physical harm through consequences such as food insecurity.
  • Framework implications: Perspectives, agency, urgency, vulnerability, experience, and other Dimensions shaped perceived severity, sometimes compounding rather than acting independently.Low agency generally increased perceived severity, while highly personal content was perceived as more severe; public versus private sphere produced divergent views.
  • Framework implications: Legality was not considered a relevant indicator of severity, including by participants with legal expertise.Participants instead associated severe harms with combinations of Dimensions linked to negative outcomes.
  • Applications: The framework can help platforms prioritize severe content, distribute moderation labor, and organize interventions across different harm types.It can also support researchers in mapping which Types and Dimensions existing studies address and identifying gaps.
  • Applications: Different participant groups may assess some harms differently, as shown by disagreement over self-injury and eating-disorder promotion involving physical-harm urgency and agency.Such disagreements may inform which participants are suited to assess particular content categories and Dimensions.
  • Applications: At platform scale, the framework addresses the unavoidable need to prioritize among many harms when moderation and engineering resources are finite.The discussion frames prioritization as necessary while acknowledging that platforms must weigh multiple forms of harm.

7 Limitations and Future Work

The study’s framework is limited by its regional sample and by its decision not to quantify content-level severity differences. Future work should broaden cultural perspectives and measure rankings across harm categories or specific content.

  • Scope limitations: The sample was concentrated in North America and South Asia, so additional regions and cultures could yield new Types or Dimensions.The authors call for perspectives from experts and general-population participants across more cultures.
  • Scope limitations: The study did not provide quantitative or statistical rankings of which content categories are more or less severe.It also did not measure how much worse particular harms might be or quantify expert-versus-general-population differences.
  • Future work: Future research should measure severity rankings for categorical and content-specific harms to strengthen understanding of online harm.Such work could also examine how rankings vary culturally and between participant groups.
  • Future work: The framework does not prescribe a single prioritization of specific harms, but offers a first step for assessing prioritization approaches in research and moderation.The authors encourage future work to develop and evaluate methods for prioritizing harms by severity.

8 Conclusion

The paper presents a high-level, empirically grounded framework of online-harm severity informed by experts and general-population users. It is intended for contextual use across research, policy, and moderation rather than as a complete account of every dimension in every project.

  • Conclusion: Interviews with 52 participants, including harm and moderation experts and general-population social-media users, informed the theoretical framework.The framework addresses relationships among varied online harms from both expert and public perspectives.
  • Conclusion: The framework offers researchers and practitioners an empirically grounded tool for assessing singular or multiple online harms in research, policy, and moderation.Users can adopt, critique, adapt, and apply it across general-population, expert, survivor, and target communities.
  • Conclusion: Applications should be contextualized because a specific project may not capture every dimension of severity.The authors present the framework as adaptable rather than universally exhaustive.
Loading 2108.04401v2…