PaperlayerView exact revision on arXiv

Source-linked AI summary

Association between Player Perception and AI Disclosure on Steam: An Exploratory Study

Fabian Wiesinger, Johanna Pirker, Denis Helic

arXiv:2610.30178v1cs.IRcs.HC

TL;DR

Empirical evidence about whether Steam’s AI-disclosure notices are associated with AI-related criticism in reviews has been limited. This study analyzes Steam reviews across disclosure groups using linguistic comparisons and statistical checks. AI-disclosed games are associated with higher AI-related criticism in negative reviews, though the reported keyword-rate difference is descriptive rather than statistically significant under game-level bootstrap sampling.

  • Problem

    The study addresses limited empirical evidence on whether Steam AI-disclosure notices are associated with explicit AI-related criticism in reviews.

    Source: §I · p. 1

  • Method

    The authors compare Steam reviews for AI-disclosed and other games using keyword, word-frequency, and characteristic-word analyses alongside statistical and robustness tests.

    Source: §I · p. 1, §II · p. 1

  • Results

    AI-disclosed games are associated with higher AI-related criticism in negative reviews; AI-related keywords occur in 22.26% versus 17.20%, with a +5.07 percentage-point difference whose bootstrap interval spans zero.

    Source: §I · p. 1, §IV · p. 3

  • Takeaways & Limitations

    The findings provide an exploratory, descriptive basis for hypotheses in future causal studies of player perception and AI disclosure on Steam.

    Source: §I · p. 1

    Steam reviews are selective and do not equally reflect all players’ opinions, and the dataset is limited to the Steam platform.

    Source: §V · p. 4

Abstract

from arXiv · show

The use of generative AI in game development is growing rapidly and is the subject of intense debate in which player reactions play a central role. This paper contributes to this discussion by analyzing Steam reviews to examine whether games with AI disclosure are associated with different review patterns. Using a dataset across multiple disclosure groups, we apply linguistic analysis including keyword-based, word frequency, and characteristic word analysis to investigate the AI-related discussion on Steam. Our results show that AI-disclosed games are associated with higher levels of AI-related criticism in negative reviews. Our findings provide a first exploratory step towards understanding how players perceive AI disclosure.

I. INTRODUCTION

Steam’s AI-disclosure policy has raised concern that labels may accompany stronger negative player reactions, while empirical evidence about review discourse has been limited. This study compares reviews of AI-disclosed and other games, finding an association between disclosure and greater AI-related criticism in negative reviews.

I. INTRODUCTION

  • Steam’s January 2024 policy requires developers to disclose generative AI use and describe it, distinguishing pre-generated from live-generated content.Public discussion raised concerns that labels could be associated with stronger negative reactions; broader work suggests people may evaluate algorithmically supported results more critically after disclosure.Source: §I · p. 1
  • Empirical evidence on players’ reactions to AI-generated content on gaming platforms was limited, including whether Steam disclosure notices are associated with explicit AI-related criticism.The paper asks whether games with AI disclosure receive more AI-related criticism than comparable games without disclosure.Source: §I · p. 1
  • The study combines linguistic analyses with statistical and robustness tests to describe associations between review language and AI disclosure, not causal effects.The authors characterize it as an initial step in studying this relation on Steam.Source: §I · p. 1
  • AI-disclosed games are associated with higher levels of AI-related criticism in negative reviews than comparable games without disclosure, with the pattern robust across review thresholds and resampling strategies.The authors frame the findings as a basis for hypotheses in future causal studies and a practical contribution for platform operators and developers considering greater transparency.Source: §I · p. 1

II. DATASET

The dataset combines reviews from games with Steam AI labels and a comparison group selected using category and popularity criteria. The final analysis focuses on games with at least 30 negative reviews.

II. DATASET

  • The study identifies AI-labeled games from Steam store pages and platform metadata, then retrieves their reviews using web scraping and public rating endpoints.The paper calls games with AI disclosure labels “AI-labeled games.”Source: §II · p. 1
  • Figure 1 displays the dataset’s game-category distribution, distinguishing AI-labeled games in blue from games without AI labels in orange.The caption describes the visual encoding but does not provide category counts.Source: Figure 1 · p. 2
  • Comparison games without AI labels were selected from the ten most common AI-labeled game categories and restricted to 30–5,000 reviews.The category and popularity criteria aim to reduce distributional differences, but the authors note that this design cannot eliminate all potential confounding factors.Source: §II · p. 2
  • The initial dataset contains 50,153 reviews across 602 games; the final analysis includes games with at least 30 negative reviews and 6,601 negative reviews.The final set also includes 6,153 positive reviews; the AI-labeled group contributes 5,665 negative and 5,456 positive reviews, while the group without AI labels contributes 936 negative and 697 positive reviews.Source: §II · p. 2

III. METHODS

The methods compare review language across sentiment and disclosure groups using keyword counts, word frequencies, characteristic-word tests, and LIWC-22 scores. The review-category figure distinguishes positive and negative reviews by hatch style.

III. METHODS

  • AI-related criticism is identified in negative reviews by counting terms spanning technical AI references, ethics and copyright, quality criticism, and AI-specific references.The keyword set was built iteratively from recurring terms in public debates about generative AI in creative work.Source: §III · p. 2
  • Figure 2 shows the distribution of reviews over Steam categories, using different hatch styles for positive and negative reviews.The caption specifies the visual encoding but gives no category-level review counts.Source: Figure 2 · p. 2
  • Word-frequency analysis compares cleaned positive and negative review text, while characteristic-word analysis tests differences in word use with chi-square tests, odds ratios, and Benjamini-Hochberg correction.Characteristic-word tests include words appearing in at least 20 reviews; cleaning removes standard stop words and high-frequency gaming terms before tokenization.Source: §III · p. 2
  • LIWC-22 scores are compared between games with and without AI disclosure across categories including tone, affect, and word counts.The analysis computes group means and standard deviations for selected variables.Source: §III · p. 2

IV. RESULTS

Review-language analyses show differences between sentiment and disclosure groups, including an AI-related keyword-rate contrast whose game-level bootstrap interval includes zero. In AI-labeled games, some AI-associated terms lean negative, while broader tone and affect patterns also differ.

IV. RESULTS

  • 22.26% of negative reviews for AI-labeled games and 17.20% for games without AI labels contain AI-related keywords; the +5.07 percentage-point difference has a bootstrap interval spanning zero.The authors interpret this as a positive descriptive association, not a statistically significant group difference under game-level bootstrap sampling; at least one AI-related keyword appears in 24.31% versus 14.35% of negative reviews under a broader proportion measure.Source: §IV · p. 3
  • 51.19% of the 7,697 words meeting the document-frequency threshold differ significantly in use between positive and negative reviews after false-discovery-rate correction.The significant-word proportions are 51.52% for AI-labeled games and 37.73% for games without AI labels.Source: §IV · p. 3
  • Figure 3 encodes negative and positive words in red and teal, with word size representing frequency normalized within each review category.The caption describes the word cloud’s encoding rather than reporting a separate quantitative comparison.Source: Figure 3 · p. 3
  • In AI-labeled games, “generative,” “soulless,” “slop,” “lazy,” and “stolen” are used more often in negative than positive reviews; the strongest comparison-group examples are general quality terms.The LIWC-22 analysis reports lower Tone and higher Affect, tone neg, emo pos, and emo neg for AI-labeled games, with no substantial difference in Authentic, WPS, or BigWords; keyword-rate results are described as stable across thresholds.Source: §IV · p. 3

V. DISCUSSION

Negative reviews in the AI-labeled group explicitly mention AI, alongside quality-related and ethical criticism. These patterns are descriptive, and the study does not establish causal links.

V. DISCUSSION

  • Terms such as “AI,” “AI-generated,” and “generative” appear directly in negative reviews, especially in the AI-labeled group.The authors describe AI as part of the language users use to rate and discuss games.Source: §V · pp. 3–4
  • Negative reviews also use quality-related and normative terms such as “slop,” “trash,” “garbage,” and “lazy,” and sometimes ethical terms related to copyright, plagiarism, or theft.The authors caution that these terms do not exclusively refer to AI; the interpretation is descriptive.Source: §V · p. 4
  • Keyword analysis may miss linguistic context or produce false positives, and Steam reviews do not represent all players equally.The study is limited to Steam and does not control for confounders such as game quality or users’ prior exposure to AI labels, so it identifies correlations rather than causal links.Source: §V · p. 4

VI. CONCLUSION

Across comparison strategies, AI-labeled games show the strongest indication of AI-related criticism, while games without AI labels score lowest. The study describes review-discourse associations on Steam without causal interpretation.

VI. CONCLUSION

  • Across comparison strategies, the AI-labeled group shows the strongest indication of AI-related criticism, while the Without AI label group has the lowest score.Frequency analyses, AI-driven keyword filtering, contingency tables, LIWC analysis, and bootstrap uncertainty tests support the conclusion that disclosed and non-disclosed games have measurably different review-discourse patterns.Source: §VI · p. 4
  • The study provides a descriptive and comparative analysis of how AI disclosure is reflected in player language on Steam, without causal interpretation.The authors present the results as a relationship between disclosure and review language, not evidence of causation.Source: §VI · p. 4
  • Future work could use topic modeling, transformer-based models, longitudinal studies, and cross-platform analysis to extend the investigation.The proposed directions address thematic structure, stance classification, changes over time, and whether the observed associations are specific to Steam.Source: §VI · p. 4
Loading 2610.30178v1…