PaperlayerView exact revision on arXiv

Source-linked AI summary

Informational Help-Seeking on Reddit Did Not Decline After ChatGPT

Hazem Ibrahim, Yasir Zaki

arXiv:2609.12447v1cs.SIcs.CY

TL;DR

Prior studies disagree on ChatGPT’s effect on online help-seeking, and no study had compared affected communities with contemporaneous communities AI cannot easily substitute for. The study compares monthly posts in informational Reddit communities with size-comparable hobby controls over the same months, repeats the analysis at 66 pre-launch dates, and tests sampled posts and comments with AI-text detectors. AI-written posts rose 2–3 percentage points more in informational than hobby communities, depending on which of four LLMs calibrated the detector.

  • Problem

    Prior studies disagree on ChatGPT’s effect on online help-seeking, and no study had compared affected communities with contemporaneous communities AI cannot easily substitute for.

    Source: §1 · p. 1

  • Method

    The study compares monthly posts in informational Reddit communities with size-comparable hobby controls over the same months, repeats the analysis at 66 pre-launch dates, and tests sampled posts and comments with AI-text detectors.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    AI-written posts rose 2–3 percentage points more in informational than hobby communities, depending on which of four LLMs calibrated the detector.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Platform-scale studies should use same-platform contemporaneous controls and pre-event placebo regressions because Reddit community types drifted apart before ChatGPT.

    Source: §8 · p. 10

    The design cannot resolve the 2.6% decline reported in Xue et al.’s conference version.

    Source: §9 · p. 10

Abstract

from arXiv · show

Did people stop asking other people for advice online once generative AI could answer their questions? Prior work on ChatGPT's effect on online help-seeking disagrees in both size and sign, in part because no study has compared affected communities against similar communities that AI cannot easily substitute for, over the same months. In this paper, we track monthly post counts in 26 Reddit informational communities against 90 size-comparable hobby communities over the same six calendar months before and after the launch of ChatGPT. We also repeat the entire analysis at 66 earlier dates, before ChatGPT existed, to see what our method reports when no ChatGPT-effect exists. We find that informational help-seeking did not decline. Our results rule out any decline in posting larger than 3.4%, far smaller than the 8% to 25% declines documented in prior work. Steady post counts could still be misleading if AI-written posts had replaced human ones. We test this possibility by scoring 274,411 posts and 223,775 comments with AI-text detectors, compared in a way that cancels out detector false-positives on human-written text. AI-written posts rose only 2-3 percentage points more in informational communities than in hobby communities, short of the 5.1 points that would be needed to hide even the smallest decline previously reported for Reddit. In addition, the comments people receive show no such rise at all. Why, then, do published studies disagree? Reddit community types were already drifting apart before ChatGPT existed, at rates comparable to every published estimate, and without same-time controls, that drift can look like an effect of generative AI. Our own largest estimate, an 18% fall in posts to low-stakes curiosity communities, matches its pre-existing trend. Humans still ask humans for help, and, as far as detection can tell, humans still answer them.

1 Introduction

Prior estimates disagree on whether ChatGPT reduced online help-seeking, and none compared affected communities with same-platform controls over the same months. This study finds no decline in informational help-seeking and shows that pre-existing community trends can produce effects as large as those previously reported.

1 Introduction

  • Informational posts rose 5.1%, and the design rules out declines larger than 3.4%.The comparison uses contemporaneous hobby-community controls and is repeated at 66 pre-ChatGPT placebo dates.Source: §1 · p. 1
  • At placebo dates, effects reached published magnitudes, showing that pre-existing community drift can mimic reported launch effects.Without same-time controls, that drift could produce estimates in either direction.Source: §1 · p. 1
  • AI-written posts rose 2–3 percentage points more in informational than hobby communities, too little to hide Reddit’s smallest previously reported decline.The estimate varies with which of four LLMs is used to calibrate the detector.Source: §1 · p. 2
  • Neither detector found extra AI content accumulating in informational communities’ answers.The comment analysis used Fast-DetectGPT and Binoculars because Fast-DetectGPT alone could not reliably separate machine-generated comments from human ones.Source: §1 · p. 2
  • Curiosity-community posts fell 18.3%, but roughly 18% annual pre-existing shrinkage against controls accounts for the estimate.The reported launch-window estimate has p = 0.002.Source: §1 · p. 2

2 Related Work

Prior work examines online help-seeking, contribution volume, and possible mechanisms for changes after ChatGPT, but its comparison strategies vary. This section distinguishes informational from emotional support and explains why detector-based comparisons and contemporaneous controls matter.

Help-seeking in online communities

  • Informational support is valued for facts and procedures, whereas emotional support is valued for who provides it (Cutrona and Suhr 1992).People rate the same supportive message as less helpful when they know it came from AI (Yin, Jia, and Wakslak 2024).Source: §2 · p. 2

Estimates of ChatGPT’s effect on help-seeking

  • Prior comparisons use other websites, one-year self-controls, or a single-community time series, leaving platform-specific changes or long-running trends uncontrolled.No prior study compares against contemporaneous communities that AI cannot easily substitute for.Source: §2 · p. 2, Table 1 · p. 2
  • Related work also studies contribution volume; Google AI Overviews raised comments in indexed Reddit communities by 12% without statistically detectable movement in posts.The paper’s 2024–25 window also reflects the fast but uneven adoption of generative AI.Source: §2 · p. 2

Mechanism

  • The social-fabric explanation predicts socially valuable communities are protected, while the privacy explanation predicts personal communities decline first; pre-existing drift makes this contrast unclear.The authors also omit account-level histories because following users in sensitive communities risks reidentification.Source: §2 · pp. 2–3

Machine-generated-text detection

  • Statistical detectors score text by its surprise to a language model, but document-level errors and bias against non-native English writers are documented limitations.Comparing average scores against controls cancels fixed detector bias, but not changes in who writes over time.Source: §2 · p. 3

3 Data and Sample

The study builds a Reddit community sample for monthly post-count comparisons and a separate text sample for detector analysis. Groups are assigned by the kind of help or reply they provide, and filters ensure consistent pre-launch histories and adequate activity.

3.1 Monthly post counts

  • Monthly community post counts come from the Arctic Shift Reddit archive; the count analysis uses no post text or author information.The archive provides per-community monthly counts through a public endpoint.Source: §3.1 · p. 3
  • The study assigns communities to informational, human-anchored, curiosity, or hobby-control groups based on what a satisfying reply would provide.The 90 hobby controls cover the treated groups’ size range and have less plausible exposure to substitution.Source: §3.1 · p. 3
  • After four pre-ChatGPT history and activity filters, the sample contains 151 communities: 26 informational, 29 human-anchored, 6 curiosity, and 90 controls.Filters require founding by December 2017, minimum post volumes, and no missing months in the 59-month history.Source: §3.1 · p. 3

3.2 Text samples for the detector analysis

  • Detector samples cover pre-ChatGPT and later windows, draw up to 100 posts and comments per community-month, and exclude documents under 200 characters.No author field is requested, and text, titles, and identifiers are discarded after scoring.Source: §3.2 · p. 3

4 Design

The design compares monthly community post counts with controls using matched calendar windows and pre-release placebo dates, then applies a parallel comparison to detector scores. It also tests the design’s trend assumption and quantifies its detection threshold.

4.1 The two-way fixed-effects specification

  • Two-way fixed effects compare each group’s post-launch monthly-count change with controls, but pre-existing divergence makes the informational-versus-support contrast uninterpretable.The model includes community and month fixed effects; the authors report the contrast without interpreting it because drift corrections remain biased or too noisy.Source: §4.1 · pp. 3–4

4.2 The six-month comparison window

  • Same-calendar six-month windows remove seasonal mismatch; nearby windows limit slow community drift, while later comparisons overlap other Reddit-wide events.The primary comparison uses December 2021–May 2022 versus December 2022–May 2023, dropping the intervening months.Source: §4.2 · p. 4

4.3 What the design can detect

  • With 26 treated communities, the design detects declines only around 14% or larger at 80% power and a 5% significance level.Seasonal matching removes about 4% bias; adding months would not improve sensitivity because slow community-level drift dominates noise.Source: §4.3 · p. 4

4.4 Placebo launch dates before ChatGPT

  • Across 66 pre-release placebo dates, nonzero estimates reflect pre-existing group trends because no ChatGPT launch occurred.Each placebo uses six months before a pretend launch and the same six calendar months one year later, dropping intervening months.Source: §4.4 · p. 4

4.5 The detector difference-in-differences

  • Detector-score difference-in-differences cancels fixed scoring offsets; matched LLM-generated documents calibrate score shifts into machine-written shares.Four LLMs generate matching posts or replies, and their score separation from human baselines converts score shifts into estimated share changes.Source: §4.5 · pp. 4–5

5 Results

Informational help-seeking did not decline detectably after ChatGPT’s launch: the estimate rules out declines larger than 3.4%. Placebo dates show that community groups already had divergent trends, including a long-running decline in curiosity communities.

5.1 Informational help-seeking did not decline

  • The informational-community estimate is +0.050 log points (+5.1%) in monthly posts, with a 95% confidence interval of [−0.034, +0.135] that excludes declines larger than 3.4%.The estimate suggests informational help-seeking did not decline after ChatGPT’s launch.Source: Table 2 · p. 5, §5.1 · p. 5
  • The interval rules out published declines of 8.3%, 14%, 18.2%, and 25%, but not the earlier 2.6% estimate from Xue et al. (2023).Table 3 compares published declines against the study’s 95% confidence interval.Source: Table 3 · p. 5, §5.1 · p. 5

5.2 Every estimate matches its pre-existing trend

  • Launch-period movements match placebo-date movements, while placebo trends separate human-anchored communities (+0.104 log points/year) from curiosity communities (−0.197), about 0.30 log points in a year.Four in five placebo windows return a negative change; the window immediately before launch gives −0.142.Source: §5.2 · pp. 5–6, Table 4 · p. 6
  • Because neighboring placebo windows overlap, their estimates are correlated and their spread cannot serve as a standard error; the mean remains the quantity used.The overlap does not bias the placebo mean.Source: §5.2 · p. 5

5.3 The curiosity-community decline matches its pre-treatment trend

  • Curiosity-community posts fell 18.3% after launch, but the estimate differs from the pre-existing placebo trend by only 0.006 log points, versus 0.090 needed to detect a change.These communities had been losing posts at about 18% a year for at least five years before ChatGPT existed.Source: §5.3 · p. 6

5.4 The estimate survives sample and window changes

  • The estimate keeps its positive sign when communities are held out or dropped one at a time; using a 59-month pre-window gives +0.136 log points.The held-out 60% estimate is +0.084, the exploratory 40% estimate is −0.003, and leave-one-community-out estimates range from +0.035 to +0.071.Source: §5.4 · p. 6

6 The 2024–25 Window

The longer 2024–25 window estimates a rise in informational-community posts, but concurrent changes and limited placebo evidence prevent reading it as a clean ChatGPT effect. The window still constrains how large a hidden decline could be given the observed counts.

6 The 2024–25 Window

  • The launch estimate for δQ falls among all 66 negative placebo estimates, while δI’s placebo estimates center on zero and contain its launch estimate.The section cautions that the difference ∆ inherits drift from its component estimates, making its launch estimate uninterpretable.Source: Figure 2 · p. 7
  • +0.218 log points (a 24% rise) in informational-community monthly posts against controls; the estimate is not a clean ChatGPT effect.The post-period includes Google’s Reddit licensing deal and AI Overviews, and only one placebo window is available, so the drift adjustment has no interval.Source: §6 · p. 7
  • Across robustness subsamples and the alternative pre-period, every estimate except the near-zero exploratory subsample has the primary estimate’s sign, and none approaches −0.086 log points, the threshold for an 8.3% decline.The longer-period complete-data sample contains 149 communities; December 2023 is excluded so the post-period follows the years of early adoption.Source: Table 5 · p. 7, §6 · p. 7
  • About one-third more posts from machine writing and search-driven arrivals combined would be needed to conceal an 8.3% decline beneath the observed rise.Concealing a 25% decline would require about two-thirds more posts; detector estimates put differential machine-like posting at roughly 2–3 percentage points.Source: §6 · p. 7

7 Testing Whether Machine-Generated Content Masks a Decline

The analysis tests whether AI-written text could mask declining human participation by comparing detector scores across informational and control communities. Estimated post inflation stays below the threshold needed to hide the smallest reported Reddit decline, while comments show no differential rise; detector coverage and the relative design constrain interpretation.

7 Testing Whether Machine-Generated Content Masks a Decline

  • The design compares detector-score changes across groups, using a 5.1% extra machine-post share as the threshold for potentially masking the 8.3% decline.It scores 223,775 comments as well as posts, and uses continuous scores so differencing cancels fixed detector bias rather than classifying individual documents.Source: §7 · p. 7, §7.2 · p. 8, §7.3 · p. 9

7.1 Which documents can be scored

  • The 200-character scoring floor selects a larger and differently composed share of posts later, but reweighting by length leaves the estimate essentially unchanged.The scoreable share rises from 0.54 to 0.75 in group I and 0.21 to 0.42 in controls; ten control communities lack scoreable posts in one period and drop out.Source: §7.1 · p. 8
  • Fast-DetectGPT separates all four LLMs from human posts but misses most machine-generated comments; Binoculars still cannot detect DeepSeek-V3 or GLM-5.2 comments.Posts have AUROCs of 0.875–0.966; Fast-DetectGPT comment AUROCs for three other LLMs range from 0.552 to 0.664.Source: §7.1 · p. 8, Figure 3 · p. 8

7.2 Posts show differential inflation of 2–3%

  • +2.1% to +3.4% is the estimated differential increase in machine-written posts across calibrations, below the 5.1% threshold for no longer ruling out an 8.3% decline.The estimates use 274,411 scored posts; the GLM-5.2 interval crosses the threshold, although its point estimate remains below it.Source: §7.2 · pp. 8–9, Table 6 · p. 9

7.3 Reddit comments show no differential rise in machine-likeness

  • Comments show no differential accumulation of machine-generated text: Binoculars estimates are −0.5% and +0.8% under its two usable calibrations.Fast-DetectGPT estimates a 5.7% relative decline using its only reliable comment calibration, GPT-4o; Binoculars gives tight null estimates.Source: §7.3 · p. 9, Figure 4 · p. 9

8 Discussion

The Reddit findings are consistent with several possible reasons people keep asking publicly, but the study cannot distinguish among them. Its conclusions are limited to Reddit, while its control-and-placebo checks are proposed for other archival-platform studies.

8 Discussion

  • The design cannot distinguish whether substitutable questions had already left, public answers retain value, or AI helps users prepare better questions.Each mechanism is consistent with the finding that people did not stop asking.Source: §8 · p. 9
  • The results cover English-language Reddit communities above a volume floor, not smaller or non-English communities.Substitution may differ in private groups, identity-bound networks, or expert platforms such as Stack Overflow.Source: §8 · p. 10
  • Future archival-platform studies should pair contemporaneous, less-exposed controls with placebo regressions at earlier dates and test whether generated text inflates counts.On Reddit, community types drifted apart by up to 0.3 log points per year even without an event.Source: §8 · p. 10

9 Limitations

The study’s conclusions are bounded by an uninterpretable direct comparison, the first six months’ observation window, and limitations in what detector scores can establish. Its realized interval rules out larger declines but cannot resolve the smallest previously reported estimate.

9 Limitations

  • The direct comparison between informational and human-anchored communities is uninterpretable because placebo estimates span 0.41 log points.The drift reflects group H communities generally; the main estimate instead compares each group with contemporaneous controls that pass placebo checks.Source: §9 · p. 10
  • The realized confidence interval rules out declines beyond 3.4%, but cannot resolve the 2.6% estimate in Xue et al.’s conference version.The design was initially powered to detect declines of about 14%.Source: §9 · p. 10
  • The primary window covers only the first six months after launch, ending before June 2023 API-pricing changes and subreddit blackouts.The longer window extends to late 2025 but overlaps other events, and the study does not divide its +24% among possible sources.Source: §9 · p. 10
  • Detector estimates are consistency checks, not estimates of machine-generated content prevalence, and miss text too short to score or too difficult for the calibrated detectors to detect.The comment analysis also excludes body-conditioned replies from DeepSeek-V3 and GLM-5.2 because their separation falls below the conversion threshold.Source: §9 · p. 10

Ethics Statement

The study separates aggregate post counting from sampled-text detection and avoids constructing individual-level records. Its privacy procedures also limit who encounters sensitive-community content and what leaves the scoring environment.

Ethics Statement

  • The counting analysis uses aggregate monthly public-post counts, while detector analysis scores sampled public post and comment text.The author field is never requested, and deleted or removed content is dropped.Source: Ethics Statement · p. 12
  • No research-team member reads or quotes documents, and sensitive-community participation is never read by a team member or excerpted.Scoring outputs retain only group, community, month, length, and score.Source: Ethics Statement · p. 12
  • The study reports aggregates across hundreds of communities and makes no claim about individuals or members of any single community.Detector scores are used only as group-level means, never to label a document, author, or community as machine-generated.Source: Ethics Statement · p. 12
  • Calibration transmitted public titles from 300 pre-period posts, and truncated comment bodies, to four commercial model APIs without author information.The text leaves the compute environment only in this scoped calibration case.Source: Ethics Statement · p. 12

A Placebo Construction Robustness

The robustness analysis uses 66 seasonally matched pre-launch windows, where the true effect is zero, to examine how the placebo estimates behave. Overlap makes the full set correlated, while checks on window composition and less-overlapping subsets preserve the reported picture.

A Placebo Construction Robustness

  • The 66 runs use seasonally matched six-versus-six-month windows ending before ChatGPT’s release.Each eligible calendar anchor contributes one run.Source: A Placebo Construction Robustness · p. 12
  • Table 7’s 66 placebo estimates show pre-existing trends in log points of monthly posts relative to controls; δI is the only quantity with a mean close to zero.Source: Table 7 · p. 12
  • Adjacent placebo windows share ten of twelve months, so their standard deviation describes a correlated family rather than a sampling distribution.Restricting to five December anchors, or splitting those into two non-overlapping subsets, leaves the same picture; the overlap does not bias the mean.Source: A Placebo Construction Robustness · p. 12
  • The full sample of 26/29/6/90 communities appears in 53 of 66 windows, and no window loses more than three communities.Missing communities reflect archive gaps rather than dormancy; r/CasualUK is the recurring exception.Source: A Placebo Construction Robustness · p. 12

B Estimate and Window Diagnostics

Placebo diagnostics show that group contrasts can reflect pre-existing community drift rather than a ChatGPT effect. Seasonal matching removes a roughly 4% I–H contrast bias, while longer separations reveal persistent community-level variation.

B Estimate and Window Diagnostics

  • At three of five near-independent placebo dates, Δ was significantly negative; δI passed every placebo check, with mean +0.011.The nearest negative contrast was −0.142, 94% of the target effect size; δI’s placebo SD was 0.052 versus a clustered SE of 0.048.Source: B Estimate and Window Diagnostics · p. 12
  • All four alternative estimators left Δ’s placebo bias at or above two-thirds of the target effect size.The passage attributes this to real community drift, not a functional-form artifact; group-specific detrending was rejected for extrapolation risk.Source: B Estimate and Window Diagnostics · p. 12
  • An unmatched window biases the I–H contrast by roughly 4%, while a seasonally matched window removes the seasonal term.December-to-May averages were −0.008 log points for I, −0.047 for H, and −0.006 for C relative to annual means.Source: B Estimate and Window Diagnostics · p. 12
  • Cross-community dispersion rose from 0.310 at a 3-month separation to 0.349 at 24 months, identifying persistent community-level drift as the dominant error component.The analysis holds the six-month window shape fixed as the post window moves farther from the pre window.Source: B Estimate and Window Diagnostics · pp. 12–13

C The Community Sample

This block documents the Reddit community sample, detector-score composition and calibration, long-window estimates, and reproducibility checks. The results show that detector-based post estimates depend on length composition and calibration assumptions, while community drift persists at longer horizons.

C The Community Sample

  • The sample was assigned before outcomes and filtered by four pre-ChatGPT rules; the groups comprise 26 informational, 29 human-anchored support, 6 curiosity, and 90 control communities.Two informational communities dropped from the constant long-window sample, leaving 24 I / 29 H / 6 Q / 90 C.Source: C The Community Sample · p. 13

D Shares of Documents Long Enough to Score

E Detector Calibration Details

  • Generated-post AUROCs ranged from 0.875 for GLM-5.2 to 0.966 for GPT-4o; generated-comment AUROCs ranged from 0.552 to 0.794, respectively.For Binoculars, the first two comment-model separations supported conversion, while the last two did not.Source: E Detector Calibration Details · p. 14
  • All groups’ mean post scores fell pre-to-post, and differencing removes this common level drift rather than interpreting score levels.Post-score shifts were I −0.102, H −0.079, Q −0.031, and C −0.155.Source: E Detector Calibration Details · p. 14
  • δI was +0.055 [+0.013, +0.099] after within-community length adjustment, versus +0.053 unadjusted; restricting to 500-character documents gave δI = +0.087 [+0.030, +0.148].The 500-character estimate converts to 3.5% to 5.6% and reaches the 5.1% threshold under the weakest calibration.Source: E Detector Calibration Details · p. 14
  • Under all four measured calibrations, the point estimate stayed below the threshold at which an 8.3% decline is no longer ruled out; only GPT-4o’s interval also ruled it out.An LLM separation below 1.04 on this text would be invisible to the test.Source: Table 10 · p. 14

F The Long Window in Full

  • In the long-window table, δI had p = 0.040, but its sole placebo window provided no drift-adjusted interval; δQ’s placebo estimate was −0.561.Window A absorbs excluded months into the pre-period, which includes twelve months of ChatGPT and the API blackout and attenuates estimates toward zero.Source: Table 11 · p. 14, F The Long Window in Full · p. 14

G Reproducibility

  • Scripts regenerate the paper’s reported numbers and figures from archived count and score files, with self-checks asserting published cells against recomputation.The archive query log was retained, and six randomly selected cached counts were re-verified against the live endpoint.Source: G Reproducibility · p. 15
Loading 2609.12447v1…