Source-linked AI summary
Quarantined! Examining the Effects of a Community-Wide Moderation Intervention on Reddit
Eshwar Chandrasekharan, Shagun Jhaver, Amy Bruckman, Eric Gilbert
TL;DR
Platforms face an unresolved question about whether and how to intervene when communities repeatedly violate rules, especially when outright bans conflict with preserving discussion. This paper studies Reddit’s quarantine intervention through causal analyses of TRP and TD, finding sharply reduced newcomer influx but little change in offensive discourse. It presents quarantining as a possible compromise between restricting harmful communities and preserving access.
Problem
The paper addresses limited evidence about community-level moderation and whether quarantining can restrict controversial communities without producing a Streisand effect.
Method
The authors analyze two Reddit quarantine case studies using causal inference, timelines from TRP and TD, and 100 control subreddits corresponding to each community.
Results
79.5% and 58% drops in new-user influx occurred for TRP and TD, respectively, while existing users’ misogynistic and racist language showed scant change.
Takeaways & Limitations
Quarantining can sharply limit newcomer influx while leaving offensive communities accessible, making it a potential compromise between banning and preserving expression.
Takeaways & Limitations
The study reports results for only two communities because identifying and analyzing the 200 control subreddits made scaling difficult.
Abstract
from arXiv · showhide
Should social media platforms override a community's self-policing when it repeatedly break rules? What actions can they consider? In light of this debate, platforms have begun experimenting with softer alternatives to outright bans. We examine one such intervention called quarantining, that impedes direct access to and promotion of controversial communities. Specifically, we present two case studies of what happened when Reddit quarantined the influential communities r/TheRedPill (TRP) and r/The_Donald (TD). Using over 85M Reddit posts, we apply causal inference methods to examine the quarantine's effects on TRP and TD. We find that the quarantine made it more difficult to recruit new members: new user influx to TRP and TD decreased by 79.5% and 58%, respectively. Despite quarantining, existing users' misogyny and racism levels remained unaffected. We conclude by reflecting on the effectiveness of this design friction in limiting the influence of toxic communities and discuss broader implications for content moderation.
1 INTRODUCTION
The paper examines Reddit’s quarantine as a softer alternative to banning controversial communities, using TRP and TD as case studies. Quarantining added access barriers while leaving the communities available, sharply reducing newcomer influx without changing offensive discourse.
- Motivation: Reddit introduced quarantining as a softer alternative to outright bans amid debate over how platforms should moderate controversial content.The intervention is examined in the context of communities containing offensive or extremist material.
- Intervention: Quarantining isolated communities from ordinary discovery while preserving access through deliberate user action.Users encountered warning screens, and quarantined posts were excluded from search, recommendations, non-subscription feeds, and r/all.
- Case studies: TRP and TD were influential quarantined communities associated with misogynistic, racist, far-right, or otherwise offensive content.TRP had over 250,000 subscribers and TD over 750,000 at quarantine.
- Research questions: The study used causal inference to examine posting activity, newcomer influx, and misogynistic or racist language in TRP and TD.The research pipeline used timelines from the two treated communities and 100 control subreddits corresponding to each.
- Findings: 79.5% and 58% drops in newcomer influx occurred for TRP and TD, respectively, while existing users’ offensive discourse showed scant change.TRP activity decreased substantially, whereas TD activity increased slightly but not significantly relative to controls.
- Implications: The findings support design friction as a potentially effective way to limit toxic communities’ growth while preserving their continued existence.The authors frame quarantining as a compromise between censoring offensive speech and preserving freedom of expression.
2 BACKGROUND AND RELATED WORK
The background situates quarantining within research on online communities, moderation, graduated sanctions, and design friction. It emphasizes the gap in evidence about community-level interventions and the distinct role of barriers that interrupt discovery and access.
- 2.1 The Manosphere: The manosphere comprises online communities organized around misogynistic, anti-feminist ideas and beliefs that men are under attack.The literature links parts of the manosphere to harassment, violence, and the Alt-Right.
- 2.1 The Manosphere: Red Pill communities reject modern feminism and progressivism and promote a masculinity centered on sexual conquest and social dominance.Their rhetoric draws on evolutionary psychology and includes a misogynist, racist, and heterosexist lexicon.
- 2.1 The Manosphere: Social media affordances enable MRA and Red Pill groups to form geographically dispersed echo chambers around emotionally charged claims of victimhood.The cited work also argues that platform algorithms can prioritize dominant demographic interests.
- 2.2 The Donald Subreddit: The_Donald evolved from a Trump-focused political forum into a prominent hub for far-right talking points, coded racism, and trolling.It influenced news content on other social media sites and was later quarantined by Reddit in 2019.
- 2.3 Community-level interventions: Prior moderation research has focused mainly on user-level interventions, leaving comparatively little evidence about moderating entire communities.Platforms can ban communities or quarantine them as alternative community-level interventions.
- 2.3 Community-level interventions: Graduated sanctions extend from individual online enforcement to community-wide moderation, offering an alternative to imposing total bans immediately.The paper applies Ostrom’s account of graduated punishment to community-level interventions.
- 2.4 Design Friction: Design friction adds deliberate difficulty to technological interactions so users pause rather than act automatically.On Reddit, quarantine removes communities from r/all and search and adds a click-through acknowledgement.
3 DATA
The study focuses on two large quarantined Reddit communities, using six-month post timelines and matched control subreddits to examine quarantine effects. Its scope is limited to these two communities, while the analysis incorporates extensive Reddit data and causal comparisons.
- Study scope: The analysis focuses on TRP and TD, selected because they were the largest and most discussed quarantined subreddits.The authors state that their findings speak only to quarantining these large Reddit communities.
- Study scope: The study’s generalizability is constrained because only two communities were analyzed and Reddit provided no official list of quarantined communities.The authors note that identifying and analyzing additional communities would require substantial manual and computational effort.
- Data collection: The dataset contains all submissions and comments posted on TRP and TD during 2018–2019, examining 180 days before and after each quarantine.TRP was quarantined on September 28, 2018, and TD on June 26, 2019.
- Controls: The study compares TRP and TD with control subreddits selected using users’ co-posting behavior to account for Reddit-wide trends.Tables 2 and 3 list control communities associated with each treated subreddit.
- Research design: The authors identify the quarantine’s causal effects by examining changes in posting activity, new-user influx, and misogyny or racism after intervention.The analysis asks whether quarantine caused significant within-community changes across these outcomes.
4 METHODS
The methods operationalize community participation, recruitment, and toxic speech over time, then validate the language-based measures against manually labeled posts. New users are defined longitudinally, while misogyny and racism are detected with specialized lexicons and precision checks.
- Temporal measures: Subreddit timelines are divided into 10-day windows indexed by w_i to measure changes around each quarantine.The quarantine date serves as the reference point for the time-window analysis.
- Participation measures: Posting activity is measured as the total number of posts assigned to each time window.Each submission or comment is assigned according to its timestamp before aggregation.
- New-user influx: New users are users who had never posted on the subreddit before the relevant time window, identified after a 30-day buffer.The method updates a cumulative seenUsers set as later windows are processed.
- Toxic speech measures: The misogyny measure uses a 1,300-term lexicon, while the racism measure uses a lexicon developed from posts in a banned racist subreddit.These lexicons operationalize toxic-language prevalence in TRP and TD.
- Instrument validation: Manual validation found 87% precision for misogyny detection in TRP and 82% precision for racism detection in TD.The validation sampled 100 lexicon-matching posts from each community.
5 CAUSAL INFERENCE STRATEGY
The study uses Interrupted Time Series analysis and bootstrapping tests to estimate quarantine effects while accounting for temporal and Reddit-wide trends. It compares TRP and TD with highly likely-to-be-quarantined control subreddits.
- Outcomes: The analysis examines changes in posting activity, new-user influx, and misogyny or racism levels within TRP and TD.Change in level is the primary metric because the analysis focuses on short-term quarantine effects.
- Study Design: TRP and TD were not randomly selected for quarantine, so the study uses control subreddits selected by users’ co-posting behavior as a proxy for quarantine likelihood.The design approximates a randomized comparison rather than relying on random assignment.
- Bootstrapping Tests: Bootstrapping compares TRP and TD’s ITS coefficients with coefficient distributions from 100 control subreddits to assess Reddit-wide trends and chance variation.The procedure repeatedly samples control coefficients and evaluates how extreme the treated subreddit’s coefficient is relative to that distribution.
- Interrupted Time Series: Interrupted Time Series analysis compares pre- and post-quarantine changes while modeling ongoing temporal trends.The pre-treatment series predicts how behavior would have continued without quarantine, helping distinguish treatment effects from general trends.
- Measurement: Metrics are computed in 10-day windows, with posting volume modeled using Poisson regression and hate-speech usage using OLS models.The dependent variables are tracked over subreddit timelines before and after the quarantine dates.
6 RESULTS
Quarantining sharply reduced new-user influx in both communities and reduced TRP posting activity, but it did not significantly change misogynistic or racist language. TD posting activity slightly increased, without a significant difference from controls.
- Changes in Posting Activity Level: 52.37%: TRP posting activity decreased significantly, and bootstrapping supported quarantine rather than a general Reddit-wide decline as the cause.The ITS coefficient was β_TRP = -0.7418, with p-val < 0.0001.
- Changes in Posting Activity Level: 3.74%: TD posting activity increased significantly in ITS analysis, but the change was not extreme compared with control subreddits.The control comparison gave p-val > 0.05, and manual inspection linked the slight increase to discussions about the quarantine.
- Sharp Decreases in the Influx of New Users: 79.51% and 58.24%: new-user influx decreased in TRP and TD, respectively, with strong evidence that quarantine caused both declines.ITS coefficients were β_TRP = -1.58 and β_TD = -0.8732, with p-val < 0.0001 for both.
- Control Comparisons: The treated-community effects were more extreme than corresponding control-subreddit effects for TRP and TD’s new-user influx.Bootstrapping tests found one-sided p-values < 0.0001 for both communities’ new-user declines.
- No Significant Effects on Hate Speech Usage: Quarantining did not significantly change misogynistic language in TRP or racist language in TD.The observed effects were β_TRP = −0.0002 and β_TD = −0.0000, with p-val > 0.5; temporary immediate shifts did not persist.
7 DISCUSSION
The discussion frames the findings as evidence about quarantining’s effectiveness as a moderation strategy and its broader theoretical and design implications. It also identifies limitations and future research directions.
- Discussion: The paper interprets its findings in relation to quarantining’s effectiveness for curbing hateful online groups.The discussion also addresses theoretical and design implications, limitations, and directions for future work.
7.1 The Effectiveness of TRP’s and TD’s Quarantine
Quarantining disrupted the recruitment of new users but did not reliably reduce activity or toxic-language use among existing members. The intervention therefore limits community growth more consistently than entrenched behavior.
- Posting activity in TRP dropped drastically, but remained misogynistic. Posting activity in TD slightly increased, and remained racist.: Quarantining did not consistently deter community activity: TRP activity fell, whereas TD activity slightly increased without a significant control comparison.Existing members’ misogynistic and racist language levels also showed no significant change.
- Influx of new users to TRP and TD declined: 79.5% and 58%: new-user influx to TRP and TD dropped drastically after quarantine.The decline occurred despite widespread discussion of the quarantine, providing no evidence of a Streisand effect.
- Influx of new users to TRP and TD declined: Design friction can substantially disrupt the growth of toxic communities by warning prospective users and adding access barriers.Quarantined communities remain accessible, but visitors must deliberately choose to view them and they lose non-subscription-based promotion.
- Effectiveness: The intervention’s strongest supported effect was restricting integration of new individuals into TRP and TD, rather than changing entrenched users’ behavior.The authors connect this result to platforms’ difficulty monitoring thousands of active communities.
- Additional Analyses: Manual review found no official moderator announcements directing users to migrate elsewhere during the six-month post-quarantine analysis period.This additional analysis examined whether migration could explain reduced activity.
7.2 Design and Theoretical Implications
Quarantining illustrates how design friction can function as a community-level moderation strategy while preserving continued access. The intervention reduced newcomer influx, but its effects on existing discourse were limited and could not be attributed to either design change separately.
- Design friction as moderation: Quarantining relegated TRP and TD from aggregated content and required visitors to deliberately choose whether to view them.The intervention combined removal from non-subscription feeds, recommendations, and search with a warning splash page.
- Design friction as moderation: Design friction offers a softer alternative to banning amid tensions between free expression and platform responsibility.The authors frame quarantining as a potential compromise for regulating content near the boundaries of acceptability.
- Empirical implications: Existing users' offensive behavior was not clearly changed, and the study cannot disentangle feed removal from the warning screen's effects.The intervention's components were implemented together, preventing separate attribution of their effects.
- Evaluation framework: The causal framework is designed to evaluate moderation interventions across communities and is intended to be extendable to future cases.The authors state that the methods are independent of the target community and that the code is publicly available after peer review.
- Empirical implications: New-user influx declined, suggesting that narrowly applied quarantining can reduce recruitment into large misogynistic and racist communities.The empirical conclusion is limited to the specific communities studied.
- Theoretical implications: Community-level interventions fit a hierarchy of graduated sanctions alongside user-level removals, bans, and suspensions.The paper raises when platforms should escalate from user-level to community-level action.
7.3 Limitations and Future Work
The study's conclusions are bounded by its focus on two communities, within-community outcomes, measurement choices, and unresolved risks around migration, insularity, and moderation governance.
- Scope of analysis: The analysis focuses on effects within TRP and TD rather than members' migration to other communities or websites.A more rigorous analysis of migration to other websites is explicitly beyond the paper's scope.
- Scope of analysis: The study examines only two quarantined communities, so more cases are needed to assess whether quarantining generally works.The authors describe the study as a first step toward answering the broader question.
- Long-term outcomes: Most quarantined subreddits are subsequently banned, while the long-term discourse of communities released from quarantine remains uncertain.Future work should examine whether improved discourse persists or returns to incendiary behavior.
- Measurement: Misogyny and racism estimates are limited by the lexicons, which may omit newly devised terms used by the communities.The robustness checks support causal interpretation but do not remove this measurement limitation.
- Measurement: The 30-day buffer may classify recently inactive users as new, although the authors expect most identified users are genuinely new.The median gap between posts is eight days, supporting the buffer's intended use.
- Constructive discussion: Quarantining may reduce toxic-community recruitment while leaving a need to support constructive discussions about masculinity, race, and conservative principles.The authors distinguish limiting toxic communities from suppressing legitimate discussion of these topics.
- Unintended consequences: Quarantine-induced insularity could intensify shared views, foster intolerance, and encourage communities to favor exit over constructive criticism.The paper identifies these as risks requiring empirical evaluation rather than established effects of the intervention.
- Governance: Platforms face difficult and contested decisions about where to draw the line between offensive and non-offensive speech.The authors suggest better data-analysis and visualization tools could inform those decisions.
8 CONCLUSION
The paper evaluates quarantining in two influential subreddits and finds that it sharply reduced newcomer influx without changing offensive discourse. It presents quarantining as a low-cost compromise between censoring offensive speech and preserving freedom of expression.
- Conclusion: The study examines quarantining as a community-wide intervention in the influential TRP and TD subreddits.Its focus is end-user behavior in these two communities.
- Conclusion: Quarantining dramatically diminished newcomer influx but did not modulate offensive discourse.The conclusion characterizes the intervention as effective on recruitment but not on the offensive behavior of existing communities.