PaperlayerView exact revision on arXiv

Source-linked AI summary

Expectation, Backlash, Recovery, and Excitement: How Model Releases Shape Reddit Perceptions of Conversational AI Systems

Vahid Rahimzadeh, Yury Zhauniarovich, Savvas Zannettou

arXiv:2608.24654v1cs.CLcs.CY

TL;DR

CAIS perceptions are often studied as static snapshots despite continual changes in models, features, safety, and access. This paper analyzes Reddit discussions with sentiment classification and thematic concept analysis around releases, finding dynamic, provider-dependent reactions and release-specific patterns of backlash, recovery, praise, and concern.

  • Problem

    Existing CAIS perception research provides aggregate snapshots, but offers limited evidence on responses to different interventions across systems and providers.

  • Method

    The study analyzes Reddit posts with LLM-based model-mention and sentiment pipelines, interpretable thematic concepts, and symmetric provider-specific windows around model releases.

  • Results

    CAIS perceptions are dynamic and intervention-sensitive: Anthropic has the clearest positive release profile, while OpenAI, Grok-3, and DeepSeek-R1 show distinct release-specific patterns.

  • Takeaways & Limitations

    Model releases are consequential user-facing events whose reception depends on the model and the broader conditions surrounding deployment, making static perception snapshots insufficient.

  • Takeaways & Limitations

    The findings reflect text-based Reddit discussions and may not generalize to the broader population of conversational AI users.

Abstract

from arXiv · show

Conversational AI systems (CAISes) continuously change through model releases, feature updates, safety interventions, and access-policy shifts, yet user perceptions are often studied as static snapshots. We conduct a long-term, large-scale analysis of Reddit discussions to examine how users perceive CAIS model release interventions across providers. By combining sentiment classification and thematic concept analysis, we show that CAIS perceptions are dynamic and intervention-sensitive. Anthropic exhibits the clearest positive release profile through Claude Code and product-model fit, OpenAI shows backlash-and-recovery dynamics around GPT-5 and GPT-5.1, Grok-3 is shaped by provider identity and political discourse, and DeepSeek-R1 combines engineering praise with concerns about censorship, access, and reliability. These findings show that model releases are not merely technical updates, but user-facing interventions that reshape sentiment, expectations, and public discussion.

1 Introduction

CAIS perceptions are dynamic and intervention-sensitive, motivating a release-centered study of sentiment and thematic changes across providers. The analysis combines large-scale Reddit data, sentiment classification, and thematic concepts to examine how model releases reshape discussion.

  • Motivation: Prior studies provide aggregate snapshots, but CAIS interventions continually change models, features, safety, and access policies.These changes make static perception measures increasingly incomplete.
  • Motivation: Replacing GPT-4o with GPT-5 was experienced as a disruptive loss of functionality and relationship, illustrating that model changes can become socially meaningful events.The case motivates studying whether reactions extend beyond exceptional backlash events.
  • Research questions: The study asks how Reddit sentiment and discussion themes change before and after CAIS model releases using symmetric release-centered windows.The design compares sentiment and thematic concept prevalence around each release.
  • Approach: The dataset covers 20 subreddits and 668K posts from October 2022 to December 2025, analyzed with human-validated LLM pipelines and qualitative-research-inspired concepts.Sentiment identifies where perceptions shift, while thematic concepts interpret what those shifts concern.
  • Findings: Anthropic has the clearest positive release profile, while OpenAI shows more modest improvements and other providers often remain closer to zero net change.At the provider level, OpenAI and Google discussions become more critical over time, whereas Anthropic shows a distinctive positive trend.
  • Findings: Release-specific discussions connect Claude Code with product-model fit, GPT-5 and GPT-5.1 with backlash and recovery, Grok-3 with provider identity and politics, and DeepSeek-R1 with engineering praise alongside concerns.DeepSeek-R1 concerns include censorship, access, and reliability.

2 Related Work

Prior research examines broad CAIS perceptions and specific concern taxonomies, but rarely tracks perception changes across recurring release events. This paper extends event-focused qualitative work with systematic, multi-provider analysis at scale.

  • Prior work: Existing studies show that users understand AI systems as tools, collaborators, threats, or social actors, affecting trust, privacy concerns, reliance, and perceived value.This work spans explanations, certification, metaphors, mental models, and everyday use.
  • Research gap: Research on user concerns develops measures or taxonomies for particular patterns but does not systematically compare release events.
  • Positioning: Unlike aggregate snapshots, this study treats model releases as recurring socio-technical events and tracks perception concepts across releases and providers.It builds on the #Keep4o study while examining perception shifts systematically at scale.

3 Dataset

The dataset is built from publicly available Reddit submissions collected across provider- and product-focused communities. After subreddit filtering and removal of deleted or removed content, the corpus contains 668,063 text posts.

  • Collection: The study collects Reddit submissions made between October 2022 and December 2025 from communities selected using the LLM Arena leaderboard.Providers and products with public user-facing interfaces were retained after manual checking.
  • Corpus construction: Filtering 849,603 submissions to 20 relevant subreddits and excluding 181K removed or deleted posts yields 668,063 posts.
  • Dataset composition: Table 1 reports the number of posts per provider in the resulting dataset.

4 Methodology

The methodology identifies and normalizes CAIS mentions, measures post-level sentiment and thematic concepts, and compares changes around provider-specific model releases. Validation and robustness checks support the extraction, classification, and release-window analyses.

  • Pipeline: The pipeline identifies normalized model mentions, measures sentiment and thematic concepts, and analyzes changes within provider-specific release windows.
  • Mention identification: A four-level taxonomy represents each model as < provider, family, generation, tier > and contains 189 entries across 9 providers.The taxonomy supports LLM-based mention identification despite varied user naming.
  • Thematic concepts: LLooM extracts excerpts, clusters them into interpretable concepts with inclusion criteria, and scores posts against the resulting concept set.Exact-name duplicates are merged, reducing 4,979 raw concepts to 236 canonical concepts.
  • Thematic concepts: 20.6% of posts contain no concepts, 23.5% contain exactly one, and 48.9% contain between two and five concepts.
  • Sentiment: The sentiment annotator classifies single-mention posts as positive, neutral, or negative and achieves 0.82 weighted F1 against majority human labels.Validation used 150 human-annotated posts and yielded a three-way nominal Krippendorff’s α of 0.71.
  • Release analysis: Release analyses use single-model text posts in symmetric provider-specific windows and compute post-minus-pre sentiment and concept-prevalence deltas.Concept changes remain robust when windows shrink to 0.6× the provider-specific size, with Pearson r ≥.73.

5 Results

Reddit perceptions of conversational AI systems shift over time and around model releases, with provider-level trends and release-specific reactions varying substantially. Sentiment and thematic analyses show favorable Anthropic releases, OpenAI backlash and recovery, and distinct expectation, access, reliability, and political dynamics across models.

  • Overall sentiment trends: 38% of Reddit posts were negative, compared with 20% positive and 42% neutral, making critical discussion nearly twice as common as favorable discussion.
  • Overall sentiment trends: OpenAI’s negative sentiment rose from 27% to 46% while neutral sentiment fell from roughly 45% to 33% between late 2022 and late 2025.Negative sentiment increased at +0.116 pp/week, while neutral sentiment declined at −0.117 pp/week and positive sentiment changed little (−0.008 pp/week).
  • Overall sentiment trends: Anthropic was the only provider with significantly increasing positive sentiment and declining negative sentiment, shifting from roughly 20% positive and 49% negative to 35% and 27%.
  • Release-specific sentiment shifts: Anthropic releases averaged +3.0/−4.4 percentage points in positive/negative sentiment, compared with OpenAI’s +0.9/−0.5 pp and near-zero changes for Google, Meta, and xAI.These provider-level averages are descriptive summaries across releases.
  • Release-specific sentiment shifts: Release-specific reactions were heterogeneous: DeepSeek R1 had a −8.1/+16.0 pp positive/negative shift, GPT-5 −3.5/+10.3 pp, and Claude 4 +5.3/−10.5 pp.
  • Thematic changes in concepts: Grok 3’s reception combined anthropomorphic engagement with political discourse, while DeepSeek R1’s discussion explosion centered on access failures, reliability, and availability.

6 Conclusion

The study finds that model releases are consequential user-facing events, reshaping perceptions of capability, reliability, trust, access, continuity, and emotional attachment. Reactions vary across providers and release contexts, making static perception snapshots insufficient for evolving systems.

  • Model releases affect how users experience conversational systems beyond technical upgrades, including capability, reliability, trust, access, continuity, and emotional attachment.
  • User reactions vary substantially across providers and release contexts.
  • The findings imply that providers should align capability announcements with launch access, preserve established models during transitions where possible, and prepare infrastructure for release-driven demand.

Limitations

The study’s evidence is bounded by its text-only Reddit dataset, potentially unrepresentative users and authorship, scalable concept construction, release-window choices, and imperfect LLM-based classification.

  • The analysis covers only text-based Reddit posts and excludes image-only or video-only discussions.
  • Reddit users may be more technically engaged and early-adopting than broader conversational-AI users, so findings reflect these communities rather than general public opinion.
  • Some Reddit posts may be AI-generated or AI-assisted, and the study cannot reliably identify them.
  • Batching the LLooM analysis may have introduced redundancy into the final concept list.
  • Symmetric release windows may smooth short-lived reactions, although major and sustained changes should remain visible.
  • LLM-based mention, sentiment, and concept assignments are imperfect and may miss or misclassify evidence.

Ethical Considerations

The study analyzes publicly shared Reddit data under aggregate, privacy-preserving practices. It avoids deanonymization and cross-site tracking, uses open-source LLMs, and paraphrases reported quotes.

  • The analysis uses data shared publicly on Reddit and follows aggregate-analysis, non-deanonymization, and non-tracking practices.
  • The researchers use open-source LLMs so user data are not disclosed to external LLM providers.
  • Reported user quotes are paraphrased to prevent linking them to specific Reddit users.

A.1 Reddit Dataset

The appendix documents the Reddit dataset’s sources, subreddit selection, provider coverage, co-mention statistics, concept assignments, and implementation pipeline. These materials characterize longitudinal discussion volume and comparative provider attention.

  • Dataset construction: The dataset was constructed from external Reddit sources and an LLM Arena text-to-text leaderboard, with provider-specific subreddit selection.
  • Concept characterization: Concept assignments are right-skewed across posts, while weekly top-10 concept prevalence is tracked against total discussion volume.
  • Provider statistics: Pairwise provider comparison uses post-level co-mention overlap, with Jaccard similarity defined as |A∩B|/|A∪B|.
  • Provider statistics: Monthly attention share and cross-provider co-mention rates characterize comparative discussion across the top five providers.
  • Concept characterization: The appendix reports concept inclusion criteria and notes an Expectation Gap trend beginning around the GPT-4o release.
  • Implementation: The pipeline includes model-mention extraction, taxonomy construction, concept induction, sentiment annotation, validation, and release-window delta computation.
  • Provider statistics: Monthly solo-mention rates are smoothed with a three-month rolling mean, where declines indicate increasing comparative discussion.
  • Implementation: Model names were normalized into provider, family, generation, and tier fields through iterative researcher review, producing a 189-entry taxonomy.

B.1.3 Model Mention Extraction Validation

The study validates its model-mention extraction pipeline against manually annotated Reddit posts and defines the release-centered analysis used to compare sentiment around interventions.

  • Model Mention Extraction Validation: 397 randomly sampled posts produced a gold set of 601 raw model mentions for validating the extraction pipeline.The sample was drawn from the 668K-post corpus and annotated independently of extractor outputs.
  • Model Mention Extraction Validation: 83% of extracted raw model mentions were exact matches, while taxonomy mapping achieved weighted F1 scores of 1.00 for provider, 0.956 for family, 0.967 for generation, and 0.976 for tier.The results indicate accurate taxonomy mapping among extracted mentions, despite possible missed mentions.
  • Sentiment Classification: Sentiment is assigned to single-mention posts using positive, neutral, and negative labels, with validation based on 150 posts annotated by three annotators.Three-way nominal Krippendorff’s α was 0.706 among annotators.
  • Intervention Design: Release analyses compare provider-complete posts within provider-specific pre- and post-release windows, excluding posts that mention models from other providers.Window sizes use the minimum observed inter-release gap, except OpenAI uses the second-smallest gap to avoid conflating distinct release lines.
  • Intervention Design: The intervention analysis computes changes in positive, neutral, and negative sentiment shares across release windows and examines thematic concept prevalence before and after selected releases.Thematic interpretation uses 20 randomly sampled posts per concept for releases with distinct sentiment profiles.

C.1.3 Statistical Significance of Sentiment Shifts

Release-level sentiment shifts are tested by comparing pre- and post-release sentiment distributions, with exact alternatives for sparse cells. Across 35 releases, 17 show statistically significant shifts, but provider-level averages remain descriptive rather than inferential.

  • Testing Procedure: Each release is tested with a chi-squared omnibus test on pre/post windows crossed with positive, neutral, and negative sentiment.Positive and negative shares additionally receive two-proportion z-tests, while neutral share is determined by the other two.
  • Testing Procedure: Monte-Carlo exact p-values replace asymptotic χ2 approximations when the smallest expected cell count is below five.The exact procedure uses 200,000 fixed-margin replicates.
  • Results: Five of the six releases discussed in Section 5 show highly significant shifts at p < 0.001; GPT-4o is the exception with p = 0.153.This supports describing GPT-4o reception as mixed rather than directional.
  • Interpretation: Provider-level release-effect averages are descriptive because OpenAI’s mean effects are not statistically supported by Wilcoxon signed-rank tests.The reported p-values are .46 for positive share and .25 for negative share.

C.2.1 Analysis

Claude 4.1 was Anthropic’s only adverse release, linked to a performance-regression incident and a rate-limit policy whose complaints shifted from anticipation to workflow disruption. Gemini 2.5 showed a small aggregate positive change alongside simultaneous feature enthusiasm and dissatisfaction with the existing 2.5 Pro experience.

  • Claude 4.1: Claude 4.1 was Anthropic’s only adverse release, following Claude 4’s most favorable release profile.
  • Claude 4.1: Performance Decline Perception rose +6.8 pp, with 80% negative and 14% positive sentiment, amid concerns about an Opus 4.1 performance bug and credibility.
  • Claude 4.1: Rate-limit complaints shifted from pre-release Pricing & Limits and Reliability Availability Frustration to post-release workflow disruption and expectation mismatch.Pre-release coverage fell from 12.2% to 6.1% for Pricing & Limits and from 8.9% to 1.8% for Reliability Availability Frustration; post-release Inconsistent Behavior Frustration rose +6.1 pp.
  • Claude 4.1: Collaborative Productivity rose +5.2 pp and was the only positive-dominant rising concept for Claude 4.1.
  • Gemini 2.5: Gemini’s aggregate shift was small: ∆+3.7 pp positive and +0.1 pp negative.Its 104-day window also included later product events and policy changes across a broad product surface, making a single-cause interpretation unsafe.
  • Gemini 2.5: Gemini 2.5 combined negative perceptions of performance decline, limitations, and inconsistent behavior with enthusiasm for newer generation and screen-sharing features.The evidence is consistent with expanded positive feature coverage alongside perceived degradation of the existing 2.5 Pro experience.
  • Gemini 2.5: Falling Gemini concepts suggest an older product surface receding from attention rather than current complaints being resolved.

C.2.3 Robustness to Release-Window Size

The release-window findings remain broadly stable when the analysis uses shorter provider-specific windows, although the shortest windows become less reliable when pre-release samples are small.

  • Window sensitivity: Shorter-window recomputations correlated each release’s concept-level ∆pp values with the reported results at 0.8× and 0.6× windows.At 0.8×, every release had Pearson r ≥.89; at 0.6×, every release had r ≥.73.
  • Window sensitivity: At the approximately nine-day 0.2× window, GPT-4o and GPT-5 retained substantial correlations of r = .73 and r = .78.
  • Limitations: Weaker shortest-window correlations coincided with sharply reduced pre-release samples, including 24 DeepSeek R1 posts and 85 Grok 3 posts.With such small samples, a single post can move concept coverage by several percentage points.
  • Statistical analysis: Concept coverage changes compare proportions between independent pre- and post-release samples because assignment is binary at the post level.
  • Statistical analysis: The analysis uses two-proportion z-tests or Fisher’s exact tests, Benjamini–Hochberg FDR correction, and Newcombe 95% confidence intervals.
  • Statistical analysis: 58 of 60 reported concept changes were significant at FDR < 0.05; both exceptions were DeepSeek R1 concepts affected by its small pre-release window.
Loading 2608.24654v1…