PaperlayerView exact revision on arXiv

Source-linked AI summary

The Community Archive: An Ethically Sourced X (Twitter) Dataset

Brandon Duderstadt, Omar Shehata, Ivan Vendrov, Francisco Carvalho

arXiv:2610.30392v1cs.SI

TL;DR

Large-scale social media research lacks accessible datasets that combine post content and metadata with user consent. The Community Archive assembles an opt-in, consent-based collection of X posts and demonstrates its use in homophily analysis, finding temporal but not aggregate content-based homophily.

  • Problem

    Legal and ethical barriers restrict access to large-scale social media datasets, many of which are collected without explicit user consent.

    Source: §1 · p. 1

  • Method

    The paper builds an opt-in archive of X posts and uses it for a sample analysis of content-based homophily.

    Source: §1 · p. 1, §5 · p. 7

  • Results

    The analysis finds no evidence of aggregate content-based homophily under a core-periphery stochastic block model, but finds that core users tend to adopt lexical innovations before peripheral users.

    Source: §5 · p. 7

  • Takeaways & Limitations

    The findings underscore the value of temporally resolved, content-rich data for studying cultural transmission online.

    Source: §5 · p. 7

    Opt-in sampling is nonrepresentative of the broader Twitter population and may introduce structural and topical biases that affect network-topology or content-diffusion analyses.

    Source: §6 · p. 8

Abstract

from arXiv · show

We present The Community Archive, a novel dataset of over 5 million posts from the social media site X (previously known as Twitter). In contrast with other large scale social media datasets, which are generally assembled without user consent via web scrapers, The Community Archive uses an opt-in model to obtain author consent for every post in the dataset. This enables us to release both the metadata and the content of users' posts at a previously unprecedented scale. Our dataset supports work in a wide variety of domains, including social network analysis, narrative tracking, and diachronic language modeling. We demonstrate the utility of our dataset by conducting a sample analysis of content-based homophily in our network.

1 Introduction

Large-scale social media research lacks accessible datasets amid legal and ethical barriers, while existing datasets are often scraped without explicit consent. The Community Archive addresses this with an opt-in collection of over 5 million X posts, author consent, and released post content and metadata.

1 Introduction

  • Large-scale social media datasets face legal and ethical barriers to distribution, and many are assembled without explicit user consent.These barriers limit widespread dataset adoption.Source: §1 · p. 1
  • The Community Archive contains over 5 million X posts collected through a fully opt-in process.Source: §1 · p. 1
  • Explicit author consent for every post allows the release of both post content and metadata at unprecedented scale.Source: §1 · p. 1
  • The authors present the archive as a resource for social network analysis, narrative tracking, and diachronic language modeling, and demonstrate it with content-based homophily analysis.They describe content-based homophily as the tendency of users discussing similar content to be connected.Source: §1 · p. 1

2 Related Work

Existing X datasets vary in their coverage of post text, metadata, scale, and consent. The paper identifies a need for a broad dataset combining post content and metadata with ethical sourcing.

2 Related Work

  • Existing datasets often lack either source-post text or metadata such as likes and follower-following relationships; those with both were collected without user consent.The comparison is summarized in Table 1.Source: Table 1 · p. 2
  • Specialized text corpora include PHEME’s 4,842 rumor-related tweets and the Broad Twitter Corpus’s 10,000 geographically sampled tweets, while TweetEval and MTEB omit post metadata.Their limited size and scope restrict utility for broad social network analysis.Source: §2 · p. 2
  • The Internet Archive Twitter Stream and Twibot-22 provide broader coverage, but were collected without explicit user consent.Source: §2 · p. 2
  • The paper identifies a need for a broad, ethically sourced X dataset containing both post content and metadata.Such a resource would support large-scale research into linguistic and network properties while avoiding challenges posed by unauthorized collection.Source: §2 · p. 2

3 The Community Archive

The Community Archive combines opt-in contributions from hundreds of accounts with post content, metadata, and social graph data. Its asynchronous archive collection creates temporal coverage constraints, while its statistics show concentrated posting and topic distributions.

3.1 Collection Methodology

  • The archive contains over 5 million posts from 244 opt-in accounts, including post content, account and post metadata, and social graph data.All collected data is released under an MIT License.Source: §3.1 · p. 2
  • Users export their X archives, upload the resulting zip files, and have them compiled into a unified database.Figure 1 depicts the database schema, including full post data and follower/following relationships.Source: §3.1 · p. 2, Figure 1 · p. 3
  • Because archives are exported asynchronously, a topic’s apparent decline may reflect sampling bias rather than a change in popularity.For example, an author may post about a topic after submitting their archive.Source: §3.1 · p. 3
  • The temporal subset includes posts before June 1, 2024 from authors whose archives were created after that date, retaining over 86% of the dataset.This filter ensures missing posts in the period reflect what an author did not post, rather than an archive submitted too early.Source: §3.1 · p. 3

3.2 Dataset Statistics

  • 5,374,594 posts come from 244 unique accounts, with 71.1% posted between January 1, 2020 and June 1, 2024.The authors conjecture that the post-2024 availability decline reflects decreasing archive uploads.Source: §3.2 · p. 3
  • The top 20% of authors by post count contribute 64.21% of posts, indicating a highly skewed author-level distribution.Figure 2 visualizes this distribution.Source: §3.2 · p. 3, Figure 2 · p. 4
  • Topics were computed using Nomic Atlas and post-processed by merging duplicates, yielding 598 topics and one additional Miscellaneous topic; computed topics are released for each tweet alongside the dataset.Source: §3.2 · pp. 3–4

4 Example Application: Content Based Homophily

The paper illustrates how The Community Archive can support content-based homophily analyses, comparing topic similarity across users and the timing of lexical innovation adoption. The aggregate test rejects content-based homophily under its assumed model, while the temporal analysis finds that the core adopted lexical innovations earlier than the periphery.

4 Example Application: Content Based Homophily

  • The analysis is presented as an illustration of the dataset's utility, not as an extensive study.It examines both time-aggregated topic distributions and adoption rates of lexical innovations.Source: §4 · p. 4

4.1 Aggregate Content-Based Homophily

  • The content affinity graph has the same number of edges as the follower-following network but shows less pronounced core-periphery structure.The graph retains the top 9,939 user-pair edges by Bhattacharyya similarity between their topic distributions.Source: §4.1 · p. 5, Figure 5 · p. 6
  • The aggregate test compares the content affinity graph with a bootstrap null distribution generated from a core-periphery model estimated on the follower-following network.The analysis samples 10,000 networks from the estimated model and compares their log-likelihoods with the content graph's.Source: §4.1 · p. 5
  • A p-value of 0 leads to rejection of aggregate content-based homophily for this network under the assumed model.The content affinity graph's log-likelihood falls far outside the left tail of the bootstrap null distribution.Source: §4.1 · pp. 5–6, Figure 6 · p. 7
  • The authors conjecture that a short lag between core-originated content and periphery discussion could explain the aggregate test's rejection.They expect little difference in core and periphery content distributions when aggregation spans a sufficiently long period.Source: §4.1 · p. 6

4.2 Temporal Content-Based Homophily

  • Using the temporal subset, the analysis tracks lexical innovations—text tokens absent before 2018 but frequent after 2018—and selects the top 100 by unique users; KDE timestamps appear left-shifted for core versus periphery.Source: §4.2 · p. 6
  • A p-value of 6e−13 supports the conclusion that the core adopted lexical innovations significantly earlier than the periphery.The observed adoption-time distributions are left-shifted for the core group.Source: §4.2 · pp. 6–7, Figure 7 · p. 7

5 Conclusion

The Community Archive is an opt-in dataset of over 5 million X posts, and a sample analysis finds temporal but not aggregate content-based homophily. The authors present it as a foundation for responsible, scalable research.

5 Conclusion

  • The Community Archive contains over 5 million X posts collected through an opt-in process.Source: §5 · p. 7
  • The authors demonstrate the dataset’s utility through a sample analysis of content-based homophily.Source: §5 · p. 7
  • The analysis finds no aggregate content-based homophily under a core-periphery stochastic block model, but core users tend to adopt lexical innovations before peripheral users.Source: §5 · p. 7
  • The authors present the archive as a foundation for responsible, scalable, and transparent research.Source: §5 · p. 7

6 Limitations

The archive’s opt-in collection introduces sampling bias, and asynchronous uploads constrain temporal coverage. Its analyses are simple, while its scale remains small beside non-consensual datasets.

6 Limitations

  • Opt-in contributors may not represent the broader Twitter population, potentially biasing network topology and content-diffusion analyses.Contributors may be more technically inclined, privacy aware, or embedded in particular ideological or professional communities.Source: §6 · p. 8
  • Asynchronous archive uploads leave activity unavailable after each user’s export, with coverage declining sharply thereafter.The temporal subset reduces available data and does not fully eliminate time-based distortions.Source: §6 · p. 8
  • The presented analyses of content-based homophily and lexical innovation adoption are simple.The core-periphery hypothesis test also lacks a correction for parameter-estimation variance.Source: §6 · p. 8
  • Although the largest opt-in corpus of its kind, the archive is small beside datasets with hundreds of millions or billions of posts.Source: §6 · p. 8

7 Broader Impacts

The Community Archive offers consented posts with content and metadata for research, while its limited scale and opt-in sampling bias call for uncritical use to be avoided.

7 Broader Impacts

  • The archive provides a large-scale corpus of consented posts with both content and metadata.Source: §7 · p. 8
  • It supports responsible research in discourse modeling, network dynamics, and language change by lowering legal and ethical barriers to analysis.Source: §7 · p. 8
  • Limited scale and opt-in sampling bias may skew insights if the archive is used uncritically.Source: §7 · p. 8

A Full Bag of Lexical Innovations

The section lists lexical innovations, including terms related to online culture, technology, and social discourse.

A Full Bag of Lexical Innovations

C Privacy Policy

The Community Archive privacy policy explains that uploaded archives become part of a public database and API, and describes data risks and contributor controls. Contributors can exclude likes or limit the published date range, while more granular permissions are planned.

Public Database and API

  • Uploaded archives enter a public database and API under an MIT License, making tweets and likes visible to anyone for access and analysis.The policy names digital anthropology research and language-model fine-tuning as possible uses.Source: C Privacy Policy · p. 9

Data Accessibility

  • The upload interface prominently displays the database’s open nature and its intended future use by anyone.Source: Figure 8 · p. 10

Privacy Options

  • A delete-archive option enables users to remove their data from the archive.Source: Figure 9 · p. 10

Potential Risks

  • Malicious actors could use the more accessible data to infer contributors’ psychology or attempt targeted phishing.Source: C Privacy Policy · p. 10

Privacy Options

  • Contributors can exclude likes or publish only a selected date range; more granular permissions, such as access for mutuals or specific individuals, are planned.Source: C Privacy Policy · pp. 10–11

Related papers

Loading 2610.30392v1…