PaperlayerView exact revision on arXiv

Source-linked AI summary

The Risks, Benefits, and Consequences of Prepublication Moderation: Evidence from 17 Wikipedia Language Editions

Chau Tran, Kaylea Champion, Benjamin Mako Hill, Rachel Greenstadt

arXiv:2202.05548v2cs.HC

TL;DR

Online communities must weigh immediate publication against screening contributions before they become public, but evidence about prepublication moderation’s effects has been limited. The paper evaluates FlaggedRevs across 17 Wikipedia communities using panel data and finds it kept much low-quality content from public view, with limited evidence of substantial broader effects.

  • Problem

    Online communities must balance rapid publication against preventing harmful content from becoming public, while the effects of prepublication moderation on participation and contribution quality remain uncertain.

    Source: §1 · p. 1

  • Method

    The authors analyze FlaggedRevs deployment in 17 Wikipedia communities using community-level panel data and interrupted time-series and user-level mixed models.

    Source: §1 · p. 2, §4.3 · p. 9

  • Results

    FlaggedRevs kept a large portion of vandalism and other low-quality contributions by untrusted users from public view, without strong evidence of meaningful long-term contribution-quality change.

    Source: §8 · p. 20

  • Takeaways & Limitations

    The findings suggest prepublication moderation can eliminate inappropriate content without substantial decreases in newcomer retention in the studied communities.

    Source: §8 · p. 21

    The authors cannot know whether or how findings from these Wikipedia communities generalize to other settings.

    Source: §7 · p. 20

Abstract

from arXiv · show

Many online communities rely on postpublication moderation where contributors, even those that are perceived as being risky, are allowed to publish material immediately and where moderation takes place after the fact. An alternative arrangement involves moderating content before publication. A range of communities have argued against prepublication moderation by suggesting that it makes contributing less enjoyable for new members and that it will distract established community members with extra moderation work. We present an empirical analysis of the effects of a prepublication moderation system called FlaggedRevs that was deployed by several Wikipedia language editions. We used panel data from 17 large Wikipedia editions to test a series of hypotheses related to the effect of the system on activity levels and contribution quality. We found that the system was very effective at keeping low-quality contributions from ever becoming visible. Although there is some evidence that the system discouraged participation among users without accounts, our analysis suggests that the system's effects on contribution volume and quality were moderate at most. Our findings imply that concerns regarding the major negative effects of prepublication moderation systems on contribution quality and project productivity may be overstated.

1 INTRODUCTION

Online communities must balance rapid publication against keeping harmful content from public view, while prepublication review may discourage contributors or burden established members. This study evaluates FlaggedRevs across 17 Wikipedia communities and finds it kept low-quality contributions from becoming visible, with limited evidence of broader harms or quality gains.

1 INTRODUCTION

  • Communities face a choice between reviewing contributions before publication, which can delay good-faith contributors, and moderating them after publication.The trade-off is between immediacy and preventing inappropriate content from being displayed publicly.Source: §1 · p. 1
  • Prepublication moderation may deter less-trusted users or reduce participation, while increasing review could also inspire more or better contributions.The introduction asks whether these systems affect contribution volume, continued participation, or contribution quality.Source: §1 · p. 2
  • FlaggedRevs was very effective at blocking low-quality contributions from becoming visible, with little evidence of broad participation costs or measurable quality improvement.The evaluation covered deployment in 17 Wikipedia communities.Source: §1 · p. 2
  • The study uses community-level panel data and user-level models to evaluate FlaggedRevs across 17 Wikipedia communities.The analysis tests hypotheses about activity and contribution quality.Source: §1 · p. 2

2 BACKGROUND

The background frames prepublication review as a trade-off: low entry barriers and healthy communities can support participation, while moderation can prevent harmful content but add delays and workload. The section then states hypotheses about effects on visible low-quality contributions, quality, activity, and newcomer return.

2.1 Peer Production and Barriers to Entry

  • Peer production theories emphasize low transaction costs, while prior research links higher entry barriers and quality-control policies with reduced participation and community growth.Low barriers are also associated with critical mass, open boundaries, and a wider range of perspectives.Source: §2.1 · pp. 2–3

2.2 Moderation and Community Health

  • Moderation can protect community health from antisocial content, but volunteer moderators may face emotional labor and burnout risks.Prior work also reports that a moderated community elicited greater intent to participate than an unmoderated one.Source: §2.2 · p. 3

2.3 Moderation Design Decisions

  • Postpublication moderation makes contributions immediately visible but can leave vandalism briefly exposed and harmful behavior unnoticed.The literature describes moderation along human/computer, transparent/secret, ex ante/ex post, and centralized/distributed dimensions.Source: §2.3 · pp. 3–4
  • Prepublication review withholds contributions until approval, potentially preventing harmful content but requiring moderator time and delaying contributors’ gratification.The literature describes it as suitable for some high-risk settings; pre- and postpublication moderation can coexist.Source: §2.3 · p. 4

2.4 Testing the Effects of Prepublication Moderation on Contribution Actions

  • Prepublication moderation is expected to reduce visible low-quality contributions from affected users and the community overall, with no effect expected for unaffected users.The hypotheses also predict higher contribution quality for affected users and the community, but not unaffected users.Source: §2.4 · p. 5
  • The hypotheses predict fewer contribution actions from affected users, unaffected users, and the community overall under prepublication moderation.The proposed explanations include reduced self-efficacy for affected users and added review demands on established contributors.Source: §2.4 · p. 5

3 EMPIRICAL SETTING

Wikipedia’s open editing model exposes its language communities to vandalism and other low-quality contributions, while smaller communities may have fewer resources to address them. FlaggedRevs holds certain revisions from public view until review, and its deployment across Wikipedia editions created an opportunity to study a previously underexamined system.

3 EMPIRICAL SETTING

  • Wikipedia permits editing with few barriers, but malicious contributions can threaten the quality and reputation of its language communities.Smaller communities may have fewer volunteer resources for fighting vandalism, and uncaught harmful material can remain visible for extended periods.Source: §3 · pp. 5–6
  • FlaggedRevs displays the most recent flagged revision rather than the latest revision, holding less-trusted users’ edits for review before public display.Trusted users’ contributions are made public immediately; trusted volunteers can approve, reject, or edit provisional revisions.Source: §3 · p. 6
  • In typical deployments, users without accounts and newer or less-active account holders are treated as untrusted until they meet community-specific thresholds.The system is configured wiki by wiki; new deployments were paused in April 2017 amid technical problems and staffing costs.Source: §3 · pp. 6–7
  • Despite deployment in multiple large Wikipedia communities, the system’s effectiveness and effects on contributions were little understood before this study.An earlier Wikimedia Foundation report had noted that FlaggedRevs’ impact on new-user contributions was not yet fully understood.Source: §3 · p. 7

4 METHODS

The study assembles wiki-level and user-level data from FlaggedRevs deployments and defines measures of visibility, contribution quality, article activity, and newcomer return. It estimates intervention-associated changes using interrupted time series and a user-level mixed model.

4.1 Data

  • Wiki-month data aggregated article-page revisions by editor group, excluding nonarticle pages and wikis averaging fewer than 30 monthly contributions per group.The analysis used the wiki-month as its unit of analysis.Source: §4.1 · pp. 7–8
  • Revision-history and FlaggedRevs review data from 24 enabled wikis were assembled, but missing publication-timing or publication-method data narrowed the study to 17 language communities.The included communities were observed in the 12 months before and after FlaggedRevs was enabled; outcomes were standardized within each wiki, and the user-level dataset contained 1,972,861 observations from users whose first contributions fell within 90 days of deployment.Source: §4.1 · pp. 7–8

4.2 Outcome Measures

  • Visible rejected contribution actions measured low-quality contributions kept visible, while total reverts and revert rate measured contribution quality.Visible rejected actions were standardized counts of visible reverted contributions per wiki-month; revert rate was the proportion of contributions eventually reverted.Source: §4.2 · p. 8
  • Productivity counted article-page contributions, while newcomer return meant making another article-page edit session within 60 days of the first.An edit session was a sequence of edits with less than an hour between consecutive saves; nonarticle activity was excluded from productivity.Source: §4.2 · p. 9
  • Outcomes were stratified into IP editors, first-time registered editors, returning registered editors, and all editors to distinguish affected-user measures from community-level effects.IP and first-time registered editors were treated as affected by FlaggedRevs; returning registered editors and all editors supported the other hypothesis groups.Source: §4.2 · p. 9

4.3 Analytical Plan

  • Monthly panel interrupted time series estimated average changes in outcome level and slope after implementation, accounting for each wiki’s baseline trend.The study used two years of monthly observations, spanning 12 months before and after deployment.Source: §4.3 · pp. 9–10
  • The analysis checked interrupted time series assumptions and model residuals, and applied a Bonferroni correction for multiple tests.The interrupted time series required at least eight observations in each pre- and post-intervention period; the corrected threshold was α=0.00625.Source: §4.3 · pp. 9, 11
  • A user-level logistic binomial mixed model with wiki random intercepts and slopes tested how newcomer return related to deployment, registration status, and whether the first edit was published.The interaction terms assessed whether the association varied with account status and public visibility of the newcomer’s contribution.Source: §4.3 · pp. 10–11

5 RESULTS

FlaggedRevs clearly reduced visible reverted contributions among affected users and across the community, while returning registered users appeared largely unaffected. The results show little evidence of major contribution-quality changes, alongside a participation decline among IP editors and a small reduction in newcomer returns.

5 RESULTS

  • FlaggedRevs clearly reduced visible reverted contributions among affected users and across the community, while returning registered users appeared largely unaffected.The intervention's primary goal was to keep low-quality contributions from becoming visible.Source: Figure 1 · p. 12, Table 1 · p. 12, §5 · p. 13
  • IP editors’ reverted contributions spiked when FlaggedRevs retroactively sent past edits for review, then quickly dropped; broader effects were not strong.The number of reverted contributions among first-time registered editors followed its preintervention trend.Source: Figure 2 · p. 13, §5 · pp. 13, 15, Table 2 · p. 14
  • FlaggedRevs showed neither a significant immediate change nor a change in trajectory for reverted rates, providing little evidence of major contribution-quality effects.The authors could not conclude that FlaggedRevs altered the quantity or quality of newcomers’ contributions.Source: Figure 3 · p. 14, Table 3 · p. 15, §5 · p. 15
  • The deployment was associated with an edit-volume decline among IP editors, but first-time registered editors showed no similar effect and other groups’ trends remained fairly flat.The authors characterize the participation effect as concentrated among the group with the lowest commitment and most targeted by the safeguard.Source: §5 · p. 15, Figure 4 · p. 16, Table 4 · p. 16
  • A 67% newcomer return rate was reduced by 2% after FlaggedRevs for newcomers whose first session was unpublished and who registered for an account.Return means a second edit session within 620 days of the first session.Source: §5 · pp. 15, 17, Figure 5 · p. 17, Table 5 · p. 17
  • Most nonsignificant estimates were near zero, though 0.1–0.5-standard-deviation standard errors leave meaningful uncertainty; simulations suggest likely effects are below half a standard deviation.The authors report substantial variation in effects across wikis.Source: §5 · pp. 17–18

6 DISCUSSION

The discussion weighs FlaggedRevs’ quality-control benefit against participation and adoption costs. It favors locally tailored decisions and comparing prepublication review with realistic alternatives.

6 DISCUSSION

  • German Wikipedia’s success was associated with tens of thousands of active reviewers and nearly all queued contributions being handled within 24 hours.That success alone did not convince more wikis to adopt the system.Source: §6 · p. 18
  • Poor maintenance, granular configuration, and review workload make FlaggedRevs difficult to adopt; missing notifications and an unintuitive interface also hinder review.Users cited the interface as one reason wikis struggled to recruit enough reviewers.Source: §6 · pp. 18–19
  • FlaggedRevs kept low-quality contributions out of public view without deterring the majority of new and existing users.The authors conclude that systems like FlaggedRevs deserve more attention.Source: §6 · p. 19
  • The authors recommend community-reliant deployment, with moderators deciding which contributors or content require vetting and who can review it.Source: §6 · p. 19
  • Deployment decisions should compare prepublication review with realistic alternatives, including blocking user groups and automated postpublication review.The authors suggest mandatory prepublication review might improve on automatic deletion in some cases.Source: §6 · p. 19
  • A streamlined review system could provide a path for evaluating contributions from populations otherwise deemed too high-risk to participate.The authors suggest this could support diverse voices and open discussion rather than outright exclusion.Source: §6 · p. 19

7 LIMITATIONS

The analysis is limited by variation among communities, imperfect identification of who was affected by FlaggedRevs, and uncertainty about generalizing beyond Wikipedia.

7 LIMITATIONS

  • Review conditions varied sharply: median delays for edits by users without accounts were two hours in German Wikipedia and more than 13 days for similar edits in Russian Wikipedia.The authors say such heterogeneity may contribute substantially to variation in estimated relationships.Source: §7 · pp. 19–20
  • The H4 and H1–3b measures combine affected and unaffected users because configuration data and IP-to-user matching were unavailable.The authors say H4 estimates are likely conservative and should be interpreted with caution.Source: §7 · p. 20
  • The authors cannot know whether or how results from a single platform generalize to other settings, despite studying 17 varied Wikipedia communities.Source: §7 · p. 20

8 CONCLUSION

Across 17 Wikipedia communities, FlaggedRevs kept much vandalism and other low-quality contributions by untrusted users from public view. The study found no strong evidence of meaningful long-term contribution-quality change, while the negative effect on newcomer retention was small.

8 CONCLUSION

  • FlaggedRevs kept a large portion of vandalism and other low-quality contributions by untrusted users from public view.The study describes this as an unambiguous success in achieving the system’s intended purpose.Source: §8 · p. 20
  • The analysis found no strong evidence of meaningful long-term change in contribution quality.Communities seeking relief from poor-quality contributions may not see the long-term improvement they expect from prepublication moderation.Source: §8 · p. 20
  • FlaggedRevs had a negative, statistically significant but small effect on newcomer retention, suggesting the impact may not be dramatic.The conclusion characterizes the deployments as occurring without substantial decreases in newcomer retention.Source: §8 · pp. 20–21
  • The authors caution that null results can be difficult to interpret, while suggesting FlaggedRevs offers lessons for platforms seeking to combat vandalism without sacrificing openness.The stated scope of openness includes outsiders, newcomers, and anonymity seekers.Source: §8 · p. 21
Loading 2202.05548v2…