PaperlayerView exact revision on arXiv

Source-linked AI summary

From Web(logs) to Web(AI): Questions, Platforms, and Methods across Twenty Editions of ICWSM

Koustuv Saha, Eshwar Chandrasekharan

arXiv:2610.00205v1cs.SIcs.CLcs.CYcs.HC

TL;DR

Research on online social life spans changing platforms, interactions, and methods, raising questions about what a two-decade record of ICWSM can tell us as AI reshapes communication. Analyzing 2,139 contributions, the paper finds stable community-topic share alongside rising governance cues, shifting platform mentions, and increasing harm framing, and offers a reporting checklist for research across changing systems.

  • Problem

    Because platforms and interactions change, findings from one platform, population, or period may reflect the systems through which behavior was observed.

    Source: §1 · p. 1

  • Method

    The paper analyzes 2,139 ICWSM contributions using NMF topic estimates, explicit textual cues for problem framings, and a methodological review.

    Source: §1 · p. 2, Platforms, Topics, and Problem Framings · p. 3

  • Results

    Community topic share changes from 10.5% to 10.0%, while governance cues within it rise from 4.0% to 34.3%; platform mentions shift from blogs toward Twitter and Reddit.

    Source: §1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    A reporting checklist makes methodological assumptions visible so readers can assess whether findings may hold as platforms or data-generating processes change.

    Source: §6 · p. 8

    Title and abstract mentions are a coarse proxy for the platform actually analyzed, and resampling intervals exclude model and interpretation uncertainty.

    Source: §7 · p. 8

Abstract

from arXiv · show

Over twenty editions, the ICWSM community has examined social life online as platforms, interactions, and research methods have changed. What can this body of research tell us at this critical juncture, as AI increasingly reshapes how people communicate online? We analyzed 2,139 indexed contributions from 2007 to 2026, distinguishing topics identified through nonnegative matrix factorization from problem framings captured through explicit textual cues. We find that platform mentions shift from blogs toward Twitter and, more recently, Reddit. Online community research maintains a similar topic share (10.5% to 10.0%), but governance cues within it increase from 4.0% to 34.3%. Harm-related cues also increase after restricting abstracts to a fixed length. Our review also traces advances in sampling, measurement, and causal and experimental methods. We discuss how AI-mediated interactions complicate these questions and provide a reporting checklist to support research across changing platforms.

1 Introduction

This paper asks how ICWSM’s topics, platform contexts, problem framings, and methods changed across twenty editions, and what that history means for research on AI-mediated interaction. It combines a longitudinal analysis with a review of methodological developments.

Motivation and scope

  • 2,139 contributions from 2007–2026 provide a record of changing platforms, online interactions, and research methods.The paper frames the twentieth edition as an opportunity to examine what changed, what remained, and what the developments mean for AI-mediated interactions.Source: §1 · p. 1

Research questions

  • Topics alone cannot show how research questions change within a subject, while methods shape whose behavior is observed and which comparisons support explanation or causal claims.For example, online community research can examine participation, support, harm, or rule enforcement.Source: §1 · p. 1

Approach

  • The analysis combines NMF topic estimates and textual cues for problem framings with sensitivity analyses and a targeted review of methodological contributions.The review covers sampling and representation, construct measurement, causal and experimental design, governance and disagreement, and research infrastructure.Source: §1 · pp. 1–2

Findings

  • Community research retains a similar topic share, from 10.5% to 10.0%, while governance cues within it rise from 4.0% to 34.3%.Harm-related framing also increases after abstracts are restricted to a fixed length; platform mentions shift from blogs toward Twitter and, more recently, Reddit.Source: §1 · p. 2

Contributions

  • The paper provides a longitudinal corpus and account, distinguishes topics from problem framings, and offers a reporting checklist for changing platforms and AI-mediated interactions.It argues that accumulated knowledge remains valuable while identifying assumptions researchers must reconsider as AI changes how online interactions are produced, encountered, and interpreted.Source: §1 · p. 2

2 Related Work

Prior retrospectives show how research communities’ interests, methods, and assumptions can be studied, while work on digital traces and measurement highlights limits in what publication and text analyses establish.

Computational histories of research fields

  • Retrospectives have tracked ideas in computational linguistics [Hall, Jurafsky, and Manning 2008], co-word patterns at CHI [Liu et al. 2014], and technologies, methods, and values at CSCW [Wallace, Oji, and Anslow 2017].These examples show that a field’s history depends on the evidence a retrospective follows.Source: §2 · p. 2

Reflection within social computing

  • The FAccT retrospective paired publication analysis with a questionnaire [Laufer et al. 2022], while an ICWSM study found population representation differs from topic or platform diversity [Septiandri, Constantinides, and Quercia 2024].That ICWSM analysis reported 37% of papers focused exclusively on Western populations, compared with 76% at CHI and 84% at FAccT.Source: §2 · p. 2

Computational social science and its evidence

  • Digital-trace research raises concerns about platform selection, access, and treating observed users as a population [Tufekci 2014].Wallach (2018) distinguishes social-scientific explanation from applying computational methods to social data.Source: §2 · p. 2

Measurement, text, and causal inference

  • Topic-model fit does not establish semantic validity, and text-based measures require substantive interpretation and validation.The reviewed work also distinguishes intended constructs from operational measures, notes dictionary-survey divergence, and identifies challenges for causal research with text [Keith, Jensen, and O’Connor 2020].Source: §2 · p. 2

3 Corpus and Methods

The study analyzes indexed ICWSM contributions using title and abstract text to estimate topic distributions and framing-cue prevalence, then checks measurement sensitivity and reviews selected methodological studies.

Scope and Retrieval

  • The corpus contains 2,139 contributions with titles and abstracts, combining 2,067 records from 2008–2026 with 72 linked abstracts from the inaugural 2007 program.It excludes workshop issues, front matter, and workshop contributions; its scope is bounded by the retrieved sources.Source: Scope and Retrieval · p. 2

Platforms, Topics, and Problem Framings

  • Platform mentions come from title and abstract dictionaries, while 15-component NMF estimates topics from TF–IDF unigrams and bigrams; the model is treated as exploratory.The community component remains relatively stable across model runs, with its most-recent-period share ranging from 9.9% to 10.7%.Source: Platforms, Topics, and Problem Framings · pp. 2–3
  • Eight overlapping problem framings are measured through explicit title and abstract cues, so reported rates indicate cue prevalence rather than each paper’s primary concern.The iterative codebook check examined 32 cue-positive abstracts but did not measure recall or independently validate the categories; within-topic rates weight papers by topic membership.Source: Platforms, Topics, and Problem Framings · p. 3

Sensitivity and Resampling

  • Sensitivity checks address abstract length, historical vocabulary, shared topic-framing terms, and paper composition while holding the topic model and codebook fixed for resampling.The analysis also removes seven governance-matching features without refitting the topic model; resampling intervals do not include coverage, coding, or model-selection uncertainty.Source: Sensitivity and Resampling · p. 3

Methodological Comparison and Validation Scope

  • The methodological review selects 75 contributions for relevance and coverage across editions, forming an illustrative bibliography rather than estimating conferencewide adoption or influence.The review groups studies around sampling and representation, construct measurement, causal and experimental design, governance and disagreement, and research infrastructure.Source: Methodological Comparison and… · p. 3

4 Topics, Problem Framings, and Platforms

Across the corpus, topic shares and problem framings do not always move together: community research remains near one tenth of topic weight while governance cues within it rise sharply. Platform mentions and other topic shares also shift over time.

Changing Topics and Persistent Subjects

  • Networks/diffusion/recommendation, search/question answering/tagging, and sentiment/opinion mining lose relative topic share, while politics/polarization and health/social support gain share.From the first to last five-edition periods, the respective changes are 16.6% to 7.8%, 11.0% to 3.2%, 10.5% to 4.1%, 3.8% to 11.8%, and 2.8% to 6.3%; falling share can coexist with continued publication on a subject.Source: Changing Topics and Persistent Subjects · p. 3
  • The language-modeling component reaches 20.5% in 2026, while explicit LLM/generative-AI terms appear in 32.8% of contributions that year.The explicit-term counts are 23/161 in 2024, 41/170 in 2025, and 61/186 in 2026; these measure proceedings visibility, not AI-generated content on social media.Source: Changing Topics and Persistent Subjects · p. 3

Different Problems within Community Research

  • Governance cues within community-topic weight rise from 4.0% to 34.3%, while community topic share changes from 10.5% to 10.0%.Removing governance-matching vocabulary leaves a rise from 3.6% to 32.5%; stricter text and platform-composition checks also retain a large contrast.Source: Different Problems within Community… · pp. 3–4, Figure 1 · p. 4

Harm Framing and Measurement Sensitivity

  • Harm/information-integrity cues increase from 6.2% to 38.1% across contributions, and the increase persists when abstracts are restricted to a fixed length.Using only the first 100 abstract words, harm-cue prevalence rises from 5.7% to 31.0%; expanded historical vocabulary changes the last-period estimate to 38.3%, while 42.5% of early contributions match none of the eight categories.Source: Harm Framing and Measurement Sensitivity · p. 4

Platform Context

  • Blog mentions fall from 35.2% to 0.9%, Twitter-associated mentions peak at 45.7% in 2012–2016, and Reddit mentions rise to 15.4% in 2022–2026.In 2026, Twitter/X and Reddit appear in 15.6% and 16.1% of contributions, respectively; these mentions describe publication contexts, not equivalent data access. Mastodon and Bluesky have small, uneven counts, and annual profiles do not support sharp phase boundaries.Source: Platform Context · p. 4

5 Methodological Developments and Recurring Questions

The review traces recurring methodological questions across five areas as platforms and available data change, illustrating developments and continuing challenges without estimating how common the approaches are across ICWSM.

Sampling and Representation

  • Sampling choices determine whose behavior is visible: stream fidelity depends on scale and unit, while off-platform traces and community selection leave representativeness questions.Demographic reweighting need not improve population prediction, and multilingual work does not automatically make research representative.Source: Sampling and Representation · pp. 4–5

Construct Measurement

  • Trace measures need construct-specific validation: studies compare traces with surveys or self-reports, while later work shows proxies and lexicons can diverge across contexts.Lexicons offer transparency, low cost, and comparability, but their validity depends on context and population.Source: Construct Measurement · pp. 5–6

Causal and Experimental Design

  • Causal designs make exposures and comparison groups explicit, but matching can leave unmeasured confounding and temporal or network comparisons require additional assumptions.The review also highlights evaluating text-adjustment estimators against known-effect tasks.Source: Causal and Experimental Design · p. 6

Governance and Disagreement

  • Moderation labels sit within community rules and norms, while LLM annotation raises questions about whose judgments define agreement.Evaluations should document whose labels define the reference and where those decisions are used.Source: Governance and Disagreement · pp. 6–7

Research Infrastructure

  • Datasets and research tools expand what can be studied and repeated, while access rules and provenance concerns shape which interactions can be observed.For AI-mediated settings, provenance includes model versions and whether content was composed, edited, selected, or generated.Source: Research Infrastructure · p. 7

6 Discussion

The discussion finds visible shifts in platform contexts alongside less visible changes within stable topics, and argues that methodological assumptions and AI-mediated interactions complicate interpretation across changing systems.

Discussion

  • Platform contexts shift from blogs toward Twitter and, more recently, Reddit, while changes within topics can be less visible.Publication trends describe changing research contexts, not necessarily changes in online behavior.Source: §6 · p. 7
  • Community research keeps a similar topic share while governance becomes a more common framing, showing that stable topics can contain changing questions and stakeholders.Understanding a field therefore requires examining both what it studies and the questions asked about those subjects.Source: §6 · p. 7
  • The phase analysis does not support a clean division into distinct periods, and publication records cannot explain why research interests changed.The authors suggest archives, calls for papers, access histories, and community interviews as avenues for understanding those shifts.Source: §6 · p. 7
  • Methodological conclusions depend on assumptions about sampling, trace validity, comparisons, whose judgments define harm, and data access.Methods can address particular sources of uncertainty without resolving every limitation, including differences in people’s offline experiences.Source: §6 · pp. 7–8
  • The reporting checklist makes methodological assumptions visible by asking researchers to document populations, measures, comparisons, sensitivity, access, and system versions.It is a reporting aid, not evidence that the assumptions are valid.Source: §6 · p. 8
  • AI can modify, augment, or generate messages, so final text alone may not reveal the influence of people, models, interfaces, and platform rules.Studies should document what was generated, displayed, selected, edited, and received, as well as model and interface versions and interaction context.Source: §6 · p. 8

7 Limitations and Future Directions

The authors identify limits in construct measurement, corpus composition, the methodological review, and inference from publication trends, and describe validation and comparison steps that could strengthen the analysis.

7 Limitations and Future Directions

  • Topic labels and framing cues only approximate the intended constructs, and sensitivity checks cannot establish precision or recall without independent judgments.The specified coding and intrusion-test protocol is not incorporated in the present estimates.Source: §7 · p. 8
  • The corpus combines publication formats whose representation may change over time, so full-paper analyses should compare verified formats within the same editions.Reconciling every issue and track would strengthen the coverage audit.Source: §7 · p. 8
  • The exploratory bibliography cannot estimate how prevalent or influential particular methodological designs are across the conference.A structured full-text review sampling beyond cue-positive abstracts and documenting disconfirming cases could strengthen the synthesis.Source: §7 · p. 8
  • Publication year imperfectly proxies study conception or data collection, platform mentions are coarse proxies for analyzed platforms, and fixed-measurement intervals omit model and interpretation uncertainty.Publication trends also do not explain why interests changed; AI research trends need direct evidence about people’s and communities’ use of the systems studied.Source: §7 · p. 8

8 Conclusion

Across twenty editions, the paper finds platform turnover alongside recurring research questions, with community research retaining a similar topic share while governance framing rises. It connects these patterns to recurring methodological challenges and the need to record provenance in AI-mediated interaction.

8 Conclusion

  • Across 2,139 contributions, platform contexts change while research questions about representation, measurement, causal comparison, governance, and access recur.The conclusion summarizes the contrast as: “Platforms change; the questions recur.”Source: §8 · p. 9
  • Community research stays near one tenth of topic weight, while governance cues within it rise from 4.0% to 34.3%.The conclusion says the contrast remains after removing shared governance vocabulary, shortening abstracts, and tightening the governance cue.Source: §8 · p. 9
  • AI systems produce, rank, and moderate interaction, making content and exposure provenance part of the evidence; the reporting checklist provides a common record for future comparisons.The corpus and analysis materials are slated for release upon publication.Source: §8 · p. 9

Ethical Considerations

The study documents its corpus and platform-mention measures while noting ethical risks in interpreting publication counts and framing cues. Platform rates count mentions rather than data use.

Ethical Considerations

  • Publication counts and framing cues could be misused as indicators of research quality or community concern, so the study defines measures and reports sensitivity analyses.The authors will release derived measurements, code, dictionaries, and documentation, but not source papers or bulk abstracts.Source: Ethical Considerations · p. 9

A Corpus and Platform Dictionaries

Platform families

  • The platform dictionary covers twenty named families plus a separate weblog vocabulary, with seven X-only cases added to Twitter.Mention rates can overlap and do not establish data use.Source: Platform families · p. 12

B Topic Components and Complete Distributions

The paper presents an exploratory 15-component NMF representation and defines framing cues as lexical approximations of research problems. Its topic components span social, informational, and technical domains.

B Topic Components and Complete Distributions

  • The 15-component NMF representation is exploratory; Table A1 lists each component's twenty highest-weight terms, while Figure A2 orders topic and framing changes separately.Component weights and cue prevalences have different denominators.Source: B Topic Components and Complete… · p. 12, Table A1 · p. 12
  • The components cover sentiment and opinion mining, communities and participation, events and collective attention, identity and self-presentation, and search, Q&A, and tagging.Their labels are interpretive summaries of the terms shown for each component.Source: B Topic Components and Complete… · p. 13
  • Other components represent health and social support, hateful and abusive speech, visual and multimodal media, and location and mobility.Source: B Topic Components and Complete… · p. 13

C Framing Codebook

  • Framing cues approximate intended research problems through lexical rules; a cue may be incidental, its absence does not prove a concern is absent, and categories can overlap.Example cues are illustrative; the analysis uses regular expressions for exact matching.Source: C Framing Codebook · p. 13
  • The resources-and-access framing requires nearby evidence of data, tools, or research infrastructure; generic API or publicly available mentions alone do not qualify.Providing a tool does not by itself make a paper a dataset or demonstration paper.Source: C Framing Codebook · p. 14

D Sensitivity Results

Sensitivity analyses examine whether platform composition and lexical overlap account for the community-governance contrast. The adjusted contrast remains substantial, including among papers mentioning neither Reddit nor Twitter.

D Sensitivity Results

  • The sensitivity checks cover harm, community-governance, and platform-composition estimates across the four main comparison periods.Density is mean paper-level cue matches per 100 title/abstract words, not paper prevalence; the historical harm union adds trolling, flaming, and vandalism, while spam is already in the baseline lexicon.Source: D Sensitivity Results · p. 14, Table A3 · p. 14

Platform composition

  • Platform standardization yields governance-cue rates of 4.0%, 1.6%, 11.6%, and 29.7%, a 25.7-point endpoint difference versus 30.3 points crude.Among papers mentioning neither Reddit nor Twitter, rates rise from 4.3% to 30.1%, a 25.8-point difference; platform turnover accounts for about one sixth of the contrast.Source: Platform composition · p. 14

Direct term overlap

  • For the community component, moderation is the only governance-expression match among its top twenty terms; overlap checks mask all matching unigrams and bigrams in the fitted vocabulary.The term matches are literal, not measures of semantic independence.Source: Direct term overlap · p. 14, Table A4 · p. 14

E Exploratory Temporal Segmentation

Exploratory temporal segmentation finds recurring candidate boundaries but does not establish definitive historical phases. The accompanying validation protocol specifies independent checks for framing cues and topic interpretability.

E Exploratory Temporal Segmentation

  • Segmentation equally weights concatenated square-root topic and normalized overlapping framing shares, with each edition weighted equally.This diagnostic differs from pooling papers within the fixed periods used in the main analysis.Source: E Exploratory Temporal Segmentation · p. 15
  • The baseline five-segment partition recurs in 29/200 resamples (14.5%), while a 2025 start appears in 47/200 resamples (23.5%) under a two- or three-edition minimum.These are conditional exact-year frequencies, not confidence levels for historical events; the baseline minimum makes 2024 the latest possible final start.Source: E Exploratory Temporal Segmentation · p. 15, Table A5 · p. 15
  • The analysis does not name definitive five-phase history or use segmentation for headline comparisons because segment count lacks external validation and boundaries can shift with model choices.Broad emphases can overlap even though segmentation assigns each year to one interval.Source: E Exploratory Temporal Segmentation · pp. 15, 17

F Independent Validation Protocol

  • The framing-validation protocol samples 20 papers per edition and asks two coders to judge substantive objectives, distinguishing central or evaluated concerns from incidental background.It allows multiple categories, flags uncertain cases, and withholds automatic predictions; pooled period estimates should use inverse-inclusion weights Nt/20.Source: F Independent Validation Protocol · p. 17
  • Topic interpretability checks use word-intrusion tasks with chance rates of 1/6 and 1/4, but these assess interpretability rather than independently validating substantive labels.The protocol specifies 75 component-level items and 80 document-level items.Source: F Independent Validation Protocol · p. 17
Loading 2610.00205v1…