Source-linked AI summary
"There Has To Be a Lot That We're Missing": Moderating AI-Generated Content on Reddit
Travis Lloyd, Joseph Reagle, Mor Naaman
TL;DR
This paper asks how generative AI may change online communities, focusing on moderators in Reddit communities that restrict its use. Interviews show concerns about content, social dynamics, and governance, alongside difficult enforcement; the study’s scope is limited to communities taking steps to govern AIGC.
Problem
The study examines how AIGC affects online communities, where its presence may alter social dynamics and moderators’ responses to its use remain insufficiently understood.
Source: §1 · pp. 1–2Method
The authors conducted fifteen semi-structured interviews with volunteer Reddit moderators, focusing on communities that had restricted AIGC.
Source: §1 · p. 2Results
Moderators reported concerns about content quality, social dynamics, and governance, while describing AIGC rules as difficult to enforce with time-consuming, imperfect detection heuristics.
Source: §1 · p. 2Takeaways & Limitations
Participants reported that clarifying community norms made their rules currently sufficient despite enforcement challenges.
Source: §7 · p. 19The sample covers moderators of communities taking steps to govern AIGC and does not necessarily explain how all Reddit communities consider it.
Source: §6 · p. 19
Abstract
from arXiv · showhide
Generative AI is altering how we work, learn, communicate, and participate in online communities. How might online communities be changed by generative AI? To start addressing this question, we focused on online community moderators' experiences with AI-generated content (AIGC). We performed fifteen in-depth, semi-structured interviews with moderators of Reddit communities that restrict the use of AIGC. Our study finds that rules about AIGC are motivated by concerns about content quality, social dynamics, and governance challenges. Moderators fear that, without such rules, AIGC threatens to reduce their communities' utility and social value. We find that, despite the absence of robust tools for detecting AIGC, moderators were able to somewhat limit the disruption it caused by working with their communities to clarify norms. However, moderators found enforcing AIGC restrictions challenging, as they rely on time-intensive and inaccurate detection heuristics. Our results highlight the importance of supporting community autonomy and self-determination in the face of this sudden technological change, and suggest potential design solutions that may help.
1 Introduction
The study examines how moderators of Reddit communities that restrict AI-generated content understand and respond to it. Fifteen interviews reveal concerns about community quality, social dynamics, and governance, alongside difficult enforcement and proposed platform support.
1 Introduction
- The study addresses how AIGC may alter community social dynamics, including through subtle inaccuracies or perceptions of inauthenticity.The introduction situates these concerns alongside existing challenges such as harassment and coordinated disinformation.Source: §1 · p. 1
- Fifteen semi-structured interviews with volunteer Reddit moderators focused on communities that had restricted AIGC use, not all Reddit communities.The interviews took place within a year of ChatGPT’s launch, capturing early concerns as communities adapted.Source: §1 · p. 2
- Moderators restrict AIGC over concerns that it could harm content quality, social dynamics, and governance, despite acknowledging potential benefits.Participants described it as poorly written, inaccurate, and off-topic, and worried it could reduce human connection or make problematic behavior harder to manage.Source: §1 · p. 2
- Moderators find AIGC rules difficult to enforce: detection relies on time-consuming, imperfect heuristics, and available tools cannot reliably automate decisions.Accusing members of using AIGC is sensitive; some moderators said clarifying norms made AIGC no more threatening than other problematic content, even as it increased their workload.Source: §1 · p. 2
- The authors discuss platform designs to support authentic communication, evolving community norms, and communities’ ability to opt out of generative AI features.They also discuss potential bias in moderators’ detection methods and design solutions that synthesize relevant information sources.Source: §1 · p. 3
2 Related Work
Prior work suggests that community values shape how AIGC is perceived, while deception, declining participation, harassment, and moderation demands pose related concerns. The literature specifically leaves moderators’ AIGC enforcement practices understudied.
2.1 Online Community Types and Values
- Community values and purposes vary, so the perceived threat of AIGC is context-dependent; Reddit’s community-level rules provide a setting for studying those differences.Existing analysis associates AI rules more with Content Generation communities and negatively with Social Support communities.Source: §2.1 · pp. 3–4
2.2 Generative AI and Threats to Healthy Online Communities
- AIGC may amplify deception and interpersonal distrust, while research on participation reports varied effects across platforms and communities.Studies cited in the passage report declines in Stack Exchange activity after ChatGPT’s launch, a null effect in similar Reddit communities, and lower question-and-answer volume and quality after a Stack Exchange ban.Source: §2.2 · p. 4
2.3 The Challenges of Volunteer Community Moderators
- Research on moderation highlights strained volunteer labor and community alignment, but the specific challenges of moderating AIGC remain understudied.Existing work raises concerns about biased AI-use determinations and the limits of detection tools, including their performance and potential bias against non-native English speakers.Source: §2.3 · p. 5
3 Methods: Interview Study
The authors used qualitative interviews to examine moderators’ experiences with AIGC in Reddit communities that restricted its use. Recruitment targeted communities with relevant rules, and interviews were conducted through multiple channels before thematic coding.
3 Methods: Interview Study
- The study used semi-structured interviews to explore how moderators understand and respond to AIGC, recruiting through public stances, outreach, and snowball methods.The qualitative approach aimed to understand the meaning individuals or groups ascribe to a social problem.Source: §3 · p. 5
3.1 Recruitment
- The researchers searched subreddit rules for AIGC-related language after crawling 337,399 subreddits, using the results to guide recruitment rather than estimate prevalence precisely.The crawl included 87,596 subreddits with non-empty rules and English as a primary language; the authors note the text-search method may be imprecise.Source: §3.1 · pp. 5–6
- AI rules appeared in 4% of all subreddits examined and 20% of the top 1% by subscriber count, prompting a recruitment focus on larger communities.Fifteen participants moderated more than 100 subreddits collectively, ranging from fewer than ten members to more than 32 million.Source: §3.1 · p. 6
3.2 Interviews
- Participants could choose video, audio, synchronous Reddit messages, or email interviews, and the researchers used these channels to broaden participation.Written interviews could yield shorter exchanges, but the authors report that all produced meaningful conversations; combined, the media offered a more nuanced picture.Source: §3.2 · pp. 6–7
- The first two authors iteratively coded transcripts, refining an inductive code hierarchy and grouping approximately 400 tagged phrases into three themes.They independently applied the codes to three transcripts and resolved a small number of disagreements by merging similar codes.Source: §3.2 · p. 7
4 Findings
The interviews surfaced concerns about AIGC and how communities respond to it, including varied boundaries around what counts as AI-generated content. Participants’ views ranged from subjective judgments about AI assistance to categorical rejection.
4 Findings
- Three interview themes address moderators’ concerns about AIGC, communities’ responses, and the challenges of enforcing AIGC rules.Source: §4 · p. 7
- Participants identified images, wholly or partly AI-generated text, and posts by automated accounts as relevant AIGC categories, and communities sometimes restricted them differently.Source: §4 · p. 7
- Some moderators judged AI use by intent or degree of reliance, accepting assistance that preserved a poster’s own contribution, such as translation or wording support.Source: §4 · p. 8
- Other moderators described categorical bans, with one community treating AI use as antithetical to its purpose.The passage notes that simpler categorical stances avoid disagreements that can arise from more subjective definitions.Source: §4 · p. 8
4.1 Findings: Moderator Concerns About AIGC (RQ1)
Moderators of communities that restrict AIGC described overlapping concerns about content quality, social dynamics, and governance, while acknowledging possible benefits. Their concerns ranged from community-specific harms to its use in focused attacks.
4.1 Findings: Moderator Concerns About AIGC (RQ1)
- AIGC concerns spanned content quality, social dynamics, and governance, with their severity varying across moderators and weaponized uses especially concerning.The study recruited moderators from communities already restricting AIGC; participants generally opposed its use, though some reported that it had become less pressing over time.Source: §4.1 · pp. 8–9
4.1.1 Content Quality Concerns.
- Moderators criticized AIGC as poorly written, inaccurate, and off-topic, fearing that fake citations could make it seem researched and weaken community trust.Some also said low-effort content could discourage original contributions, while communities focused on photography prioritized excluding AI-generated images.Source: §4.1.1 · pp. 9–10
4.1.2 Social Dynamics Concerns.
- Ten participants worried AIGC would reduce human connection or undermine communities built around human-authored dialogue, shared practice, and learning.Some also objected to models trained on creative work without artists’ knowledge, credit, or compensation.Source: §4.1.2 · pp. 10–11
- Participants feared AI accusations could fuel incivility and arguments, while some objected to AIGC as incompatible with their communities’ values.In a political subreddit, the possibility of AIGC was linked to accusations that comments were AI-written; a Zelda moderator described attacks both on AI users and on those accused of gatekeeping art.Source: §4.1.2 · pp. 10–11
4.1.3 Governance Concerns.
- Eight participants said AIGC could amplify spam, harassment, trolling, and influence operations, including automated accounts used to accumulate karma or promote products deceptively.Some moderators also described targeted trolling as easier to produce with AI, though one felt existing moderation could handle it.Source: §4.1.3 · pp. 11–12
- Some moderators described AIGC as uncommon but highly disruptive, and reported its use to expedite objectionable content their communities had already banned.Source: §4.1.3 · pp. 11–12
4.1.4 Potential Benefits of AIGC.
- Moderators saw possible benefits in helping people contribute or improve writing, including for non-native English speakers, and in reviving inactive conversations, but all felt harms currently outweighed benefits.One moderator framed automated posts as potentially useful in some contexts, not as something their own community had done or necessarily valued.Source: §4.1.4 · p. 12
4.2 Findings: Online Communities’ Responses to AIGC (RQ2)
Communities addressed AIGC by developing norms with members and codifying them into rules. Enforcement remained difficult because detection took time and could be wrong, even as moderators shared the work and used platform tools.
4.2.1 Clarifying Community Norms.
- Moderators and community members often established AIGC stances collaboratively and iteratively, sometimes revisiting bans or allowing integration before members objected again.Participants sought alignment with community interests; one community used discussion and member votes to enact a ban after abuse and declining quality.Source: §4.2.1 · pp. 12–13
- Almost all participants codified norms in public rules, which seven said reduced AIGC, though moderators doubted rules would deter bots or bad actors.Source: §4.2.1 · p. 13
4.2.2 Enforcing Community Rules.
- All participants’ communities tried to enforce AIGC rules, but moderators said assessing verbose content was slow and false accusations required caution.The effort to produce AIGC could be faster than moderators’ review, and one moderator described watching a user over time rather than risk banning someone who had not used GPT.Source: §4.2.2 · p. 13
- Moderators shared enforcement with community members and used automated tools, but user reports risked incivility and tools could miss AIGC, prompting some to seek admin help.The passage notes Reddit admins had more sophisticated tools than moderators, while the available moderator tools depended on potentially flawed detection heuristics.Source: §4.2.2 · pp. 13–14
4.3 Findings: Methods for Detecting AIGC (RQ2)
Moderators use content, account, community-norm, and domain-knowledge signals to identify suspected AIGC, but these heuristics take time and may miss content. They supplement them with automated tools despite concerns about those tools’ reliability.
4.3 Findings: Methods for Detecting AIGC (RQ2)
- Moderators worry that their current detection techniques may become less effective as generative AI improves, and that they are already missing AIGC.They reported using custom heuristics and technical tools, while one moderator anticipated increasingly sophisticated content.Source: §4.3 · p. 14
4.3.1 Detection Heuristics.
- Twelve participants used content patterns, such as repetition or unnaturalness, as signals that a post might be AIGC.The paper groups signals into recognizable language patterns, account details and behavior, deviations from community style, and inaccurate information.Source: §4.3.1 · p. 14
- Six participants considered account details and posting history, including abrupt changes in a user’s writing style, when assessing suspected AIGC.Four also noted that account indicators previously used to identify traditional bots could apply to AIGC.Source: §4.3.1 · p. 14
- Four participants used familiarity with subreddit communication norms or domain knowledge to flag atypical, false, or incoherent posts.For example, a museum moderator compared posts with the subreddit’s canon of material.Source: §4.3.1 · pp. 14–15
4.3.2 Technical Tools.
- Moderators said their heuristics worked but were time-consuming, prompting them to use technical tools to speed detection.Participants acknowledged that the tools were unreliable; some programmed heuristics into Automoderator, while six manually copied posts into third-party detectors.Source: §4.3.1 · p. 15, §4.3.2 · p. 15
- Moderators rarely trusted third-party detectors outright, and two participants found them especially ineffective on short text.One moderator described short comments and poems being flagged as AI; another said they used several layers and tried to talk with users.Source: §4.3.2 · p. 15
5 Discussion
The discussion connects moderators’ concerns about AIGC to community values and the challenge of establishing and enforcing norms. It argues for community self-determination while noting that attitudes may change and that detection heuristics can be burdensome and biased.
5 Discussion
- The study’s discussion emphasizes that community social value matters in the age of generative AI and that communities should be able to take their own stances on AIGC.The authors connect context-sensitive AIGC decisions to community self-determination.Source: §5 · p. 15
5.1 Community Values Shape Attitudes Towards AIGC
- The authors expect AIGC objections to vary by community purpose and size, with concerns spanning social value, accuracy, content quality, governance, and ideology.They expect some objections to lessen over time, while others may persist.Source: §5.1 · p. 16, §5.1.1 · p. 16
5.1.1 Community Types and AIGC Attitudes.
- Across the community types encountered, participants rejected AIGC when they saw their communities’ purpose as sharing human-created posts.The authors note that their sample focused on communities with concerns about AIGC.Source: §5.1.1 · p. 16
5.1.2 Attitudes May Shift Over Time.
- Because community norms and generative AI may change, the authors argue platforms should support both acceptance and collective refusal, including community opt-outs from AI features.They suggest communicating opt-outs through the interface and considering community values when adding generative AI features.Source: §5.1.2 · pp. 16–17
5.2 The Challenges of Establishing and Enforcing Norms
- Communities are clarifying and codifying AIGC norms, which can guide well-intentioned members even when moderators struggle to identify AIGC.The authors relate this challenge to the broader work of establishing and enforcing community norms.Source: §5.2 · p. 17, §5.2.1 · p. 17
5.2.1 Clarifying Norms in a Moment of Flux.
- Making rules more explicit may reduce misunderstandings, but UI reminders may be overlooked and member interviews add time burdens that could deter participation.Suggested approaches include showing rules in authoring interfaces or onboarding new members.Source: §5.2.1 · pp. 17–18
5.2.2 Equitably Enforcing AIGC Bans.
- AIGC detection heuristics can be time-intensive and biased, potentially burdening minorities and people seeking legitimate participation.The authors say most participants were confident in their heuristics and unaware of potential bias; they call for moderator education and further evaluation.Source: §5.2.2 · p. 18
- Communities may develop effortful ways to signal authenticity, but whether these practices can offset AIGC’s effects on interpersonal communication remains uncertain.The authors suggest platforms could provide affordances to support such practices.Source: §5.2.2 · p. 18
6 Limitations and Future Work
The study’s sampling focused on Reddit communities already taking steps to govern AIGC, limiting how broadly its findings can be applied. The authors call for research spanning different community stances, sizes, and platforms.
6 Limitations and Future Work
- The interviews covered moderators of communities already taking steps to govern AIGC, so the findings do not necessarily explain how all Reddit communities consider it.The authors say understanding AIGC’s impact on online communities requires future work from different angles.Source: §6 · p. 19
- The study excluded communities that embraced AIGC or lacked AI-specific rules and focused recruitment on larger subreddits.The authors recommend studying a wider range of community stances and sizes.Source: §6 · p. 19
- Research beyond Reddit and across different methods could provide a fuller account of community responses and the relationship between community values and AIGC stances.The authors identify surveys, quantitative methods, and studies of other platforms as future directions.Source: §6 · p. 19
7 Conclusion
The study finds that moderators see AIGC as a concern for content quality, social dynamics, and governance, and that communities use rules to clarify norms despite difficult enforcement. It calls for further attention to how generative AI affects online communities and how moderators can be supported.
7 Conclusion
- Moderators were concerned that AIGC could affect their communities’ content quality, social dynamics, and governance, even as they recognized potential contribution benefits.Source: §7 · p. 19
- Communities enacted bans enforced through time-intensive, imperfect heuristics, while participants reported that rules clarified norms and were currently sufficient.Source: §7 · p. 19
- The authors call for research and design work on how generative AI affects online community dynamics, practices, and health, and how moderators can be supported through technological change.Source: §7 · p. 19