PaperlayerView exact revision on arXiv

Source-linked AI summary

The Illinois Social Attitudes Aggregate Corpus (ISAAC): An Open Tool and Reproducible Pipeline for Analyzing Social Group Discourse at Scale

Babak Hemmatian, Sarah Hadjarab, Jessica Chen, Benedek Kurdi

arXiv:2609.27059v1cs.CLcs.SI

TL;DR

Existing social-group discourse corpora lack generalization-focused labels. ISAAC brings together Reddit posts across six social-group distinctions, using multi-stage filtering; its residual irrelevance met a single-digit criterion, and its semantic classifiers performed at or near state-of-the-art levels.

  • Problem

    No existing corpus of social-group-related discourse offers generalization-focused labels.

    Source: The Illinois Social Attitudes… · p. 12

  • Method

    ISAAC assembles a 527-million-plus-post Reddit corpus spanning 17 years and six social-group distinctions, using aggressive multi-stage filtering to address topical noise.

    Source: Discussion · p. 48

  • Results

    Residual irrelevance met the single-digit development criterion for every social-group distinction and both post types, while semantic classifiers performed at or near state-of-the-art levels.

    Source: Discussion · p. 48

  • Takeaways & Limitations

    ISAAC provides a structure for studying naturalistic social-group discourse at scale.

    Source: Conclusion · p. 53

    ISAAC reflects a digital ecosystem rather than representative general-population baselines.

    Source: Limitations and Possibilities for… · p. 52

Abstract

from arXiv · show

We introduce the Illinois Social Attitudes Aggregate Corpus (ISAAC), an open, modular, and accessible corpus of 527 million+ English-language Reddit posts selected for relevance to six key social group distinctions based on race, sexuality, age, ability, body weight, and skin tone, covering the 17-year period from 2007 to 2023. A multi-step, human-audited filtering pipeline was used to keep irrelevant content in the curated dataset below 10%, both overall and for each social group distinction. Each post was then algorithmically annotated with the user's estimated home region, along with a suite of validated off-the-shelf and custom semantic labels including moralization, sentiment, emotion, and linguistic generalization. We confirm the validity of the resulting corpus through convergent evidence linking ISAAC to macro-level societal trends, such as online search behavior, temporal spikes during major societal events (both nationally and regionally), and long-term shifts in public attitudes. By offering a unified, public infrastructure, ISAAC eliminates research fragmentation and enables seamless replication while supporting diverse empirical workflows at scale. Specifically, ISAAC allows investigators to perform cross-category comparisons, conduct high-precision tracking of long-term temporal shifts in social group discourse, and map spatial variation onto localized public opinion and policy outcomes. ISAAC's fully public, modular pipeline facilitates easy extension of the corpus to new platforms, languages, and social categories. To accommodate various research needs, ISAAC is accessible both without coding through a point-and-click website and labeler web-apps, and programmatically via an SQL playground, a Python package, and HuggingFace.

The Illinois Social Attitudes Aggregate Corpus (ISAAC)

This section presents ISAAC, a large, continuously labeled Reddit corpus spanning six social group distinctions from 2007 to 2023, with human-audited filtering and multiple access pathways. It outlines the corpus’s annotation schema, including estimated-location outputs and labels for moralization, sentiment, and emotion.

The Illinois Social Attitudes Aggregate Corpus (ISAAC)

  • ISAAC contains 527,060,919 Reddit submissions and comments posted from 2007 to 2023, selected for relevance to six distinctions involving sexuality, race, age, ability, weight, and skin tone.The distinctions are straight versus gay, White versus Black, young versus old, abled versus disabled, thin versus fat, and light-skinned versus dark-skinned.Source: The Illinois Social Attitudes… · p. 10
  • Human audits found less than 10% irrelevant content across all six social group distinctions and the full year range, supporting the corpus’s validity, reliability, and statistical power.The authors describe this as an end-to-end benchmark for naturalistic social-group discourse.Source: The Illinois Social Attitudes… · pp. 10–11
  • A unified pipeline pairs the six categories with a broad label set, including estimated user home locations and semantic measures such as moralization, sentiment, emotion, and linguistic generalization.The corpus presents the broadest collection of labels among comparable resources, according to the paper.Source: The Illinois Social Attitudes… · pp. 10–11
  • ISAAC supports cross-category comparisons, long-term and event-linked analyses, and links between discourse and localized outcomes, while offering no-code and programmatic access through an open, modular pipeline.The authors note limitations: the corpus targets aggregate language rather than interpersonal dynamics, labels observable language rather than internal attitudes, and draws on Reddit alone; its modular design allows future investigators to address these limitations.Source: The Illinois Social Attitudes… · p. 12

Validation Strategy

  • ISAAC’s validation combines component and end-to-end audits with external comparisons to cultural flashpoints, state ballot outcomes, internet searches, and national surveys, supporting its ability to capture attitude shifts across time and geography.The pipeline also evaluates ensemble-label agreement and in-house model performance on held-out data, with validation data released alongside the corpus.Source: Validation Strategy · p. 13

Overview of Corpus Construction

  • Corpus construction used a four-step filtering pipeline to extract relevant Reddit discourse and remove irrelevant content, then paired entries with Reddit metadata, anonymized user metadata and location estimates, and algorithm-generated semantic labels.The construction overview describes three phases and points to Figure 1; released entries include variables from all three families.Source: Overview of Corpus Construction · p. 14
  • Location fields include a persistent random identifier, estimated home location with model confidence, and hierarchical U.S.-state and non-U.S.-region labels, including unknown.Confidence values range from 0 to 1.Source: Overview of Corpus Construction · p. 15
  • The schema includes a binary moralization label.Source: Overview of Corpus Construction · p. 15
  • Sentiment fields include positive, neutral, and negative counts, sentence counts, VADER compound polarity, and TextBlob polarity and subjectivity.VADER compound polarity ranges from -1 (negative) to 1 (positive); TextBlob polarity ranges from -1 to 1, and subjectivity from 0 to 1.Source: Overview of Corpus Construction · p. 15
  • Three emotion models provide continuous scores from 0 to 1 for anger, disgust, fear, joy, sadness, surprise, and neutral content.Source: Overview of Corpus Construction · p. 15

Ethical Approval

The corpus draws on Reddit archives and applies four filtering steps to extract discourse relevant to six social-group distinctions.

Ethical Approval

  • The research design was found not to meet the regulatory definition of human-participant research under 45 CFR 46.102, while development followed current social-media research recommendations.The paper directs readers to Appendix A for ethical details.Source: Ethical Approval · p. 16

Data Source

  • ISAAC uses discontinued Pushshift-formatted Reddit archives and retains content without bot filtering or a minimum post length.This preserves naturalistic discourse for researchers to apply their own inclusion criteria.Source: Data Source · p. 16

Computational Infrastructure

  • The released corpus was built on high-performance computing clusters, and the ISAAC website demonstrates the end-to-end pipeline on an illustrative subset.The construction setup included up to 200 CPU cores and 20 GPUs.Source: Computational Infrastructure · pp. 16–17

Filtering

  • Four filtering steps process the raw archive to minimize irrelevant posts in the final corpus.The pipeline first matches keywords, then filters language, classifies relevance, and applies complex patterns.Source: Filtering · p. 17

Step 1: Keyword Matching

  • Keyword matching captures both poles of each distinction across Reddit communities, including general-interest subreddits that subreddit-based filtering would miss.Lists were compiled iteratively from thesauri and supplemented with slurs and group-specific slang.Source: Step 1: Keyword Matching · p. 17

Step 2: Language Filtering

  • fastText retains posts identified as English, addressing non-English content and keywords whose meanings vary across languages.The model was selected for the speed needed to scan billions of posts.Source: Step 2: Language Filtering · p. 18

Step 3: Relevance Classification

  • Separate RoBERTa classifiers use full-post context to distinguish social-group references from irrelevant keyword uses.Each classifier was trained on about 1,500 keyword-filtered posts, and two trained annotators rated sampled posts for relevance.Source: Step 3: Relevance Classification · p. 18

Step 4: Complex Pattern Matching

  • Complex regular-expression patterns remove structural false positives, such as proper names and pop-culture references that classifiers mislabel.The patterns were iteratively developed by inspecting surviving posts and applied with a fast matching engine.Source: Step 4: Complex Pattern Matching · p. 19

Filtering Quality Assessment

The section describes human audits of filtering and the metadata, location estimates, anonymization, and semantic labels included in ISAAC.

Filtering Quality Assessment

  • 10% was the target residual-irrelevance threshold across both post types and all social-group distinctions.Development ended after stratified human ratings met that threshold; audits also checked whether quality transferred from comments to submissions.Source: Filtering Quality Assessment · pp. 19–20

Reddit Metadata

  • Each entry retains post and parent identifiers, timestamp, subreddit, score, post type, and matched keywords; identifiers support thread reconstruction and linking a user's posts.Relinking identifiers to identifiable Reddit accounts is prohibited to protect privacy.Source: Reddit Metadata · p. 20

User Location Estimation

  • A hierarchical model estimates users' home locations from word use, subreddit participation, and timestamps, retaining coarse labels when finer estimates are weak and marking unsupported cases unknown.It first separates U.S. from non-U.S. users, then assigns U.S. users to state-level jurisdictions and others to world regions; location labels were validated in five ways.Source: User Location Estimation · p. 20

Anonymization

  • Usernames are replaced with consistent random identifiers across posts and all six distinctions, while the username-to-identifier mapping is not distributed.The consistent identifiers preserve user-level dependencies for longitudinal and spatial analyses.Source: Anonymization · p. 21

Moralization

  • Moralization is represented by a binary classifier trained on Reddit's MFRC, with annotation disagreements resolved by majority vote and residual ties assigned the moralized label.A binary label was chosen instead of separate moral-foundation labels; the trained classifier is publicly released and evaluated on held-out MFRC data.Source: Moralization · pp. 21–22

Sentiment

  • Three sentiment models with distinct designs contribute separate outputs, allowing researchers to build ensemble estimates or examine model disagreement.The models are VADER, TextBlob, and Stanza; their outputs are included as separate dataset columns.Source: Sentiment · p. 22

Emotion

  • Three emotion classifiers provide independent estimates for six basic emotions and neutral content, with model-specific ratings retained for ensemble and disagreement analyses.The classifiers' score scales differ: two produce probabilities summing to one across seven categories, while roberta-base-go_emotions produces independent per-category probabilities.Source: Emotion · p. 23

Generalization

ISAAC provides clause-level generalization labels and multiple access routes, alongside a modular, auditable pipeline intended for adaptation. Its evaluation strategy combines model metrics with corpus-level checks against external trends and events.

Generalization

  • Clause-level labels capture genericity, eventivity, and—among eventive clauses—boundedness and habituality, while separately counting questions and imperatives.The pipeline decomposes posts into clauses, labels each clause, and preserves clauses, labels, and per-feature counts and proportions so researchers can trace anecdotal focus and broad generalizations within a post.Source: Generalization · p. 24

Corpus Access

  • ISAAC offers five access pathways: a coding-free interface and four routes for programmatic or large-scale workflows.The routes include an SQL playground, a Python package, HuggingFace interfaces, and scripted downloads.Source: Corpus Access · p. 25

Data Use Agreement

  • Use of ISAAC’s data, tools, and scripts is limited to non-commercial academic research and governed by a Data Use Agreement.The agreement prohibits re-identification, generative-AI training, harmful or discriminatory use, and surveillance, and restricts redistribution and copies.Source: Data Use Agreement · pp. 26–27

Adapting and Extending ISAAC

  • The public, modular pipeline can be adapted to new topics, languages, models, annotations, and non-Reddit sources by changing the corresponding components.Released resources and configuration defaults make the construction steps transparent and auditable; the corpus also provides durable access to a slice of Reddit history after broad public distribution of the underlying raw archives discontinued.Source: Adapting and Extending ISAAC · pp. 27–28

Analytic Strategy

  • Validation checks compare corpus-volume fluctuations with Google Trends, post volumes around major events with pre-event baselines, and semantic-label patterns with external measures.Location labels are also checked against localized discourse spikes around nine state-level marriage-equality ballot measures; semantic-label validation includes cross-label interactions and comparison of a longitudinal sentiment composite with Gallup opinion tracking.Source: Analytic Strategy · p. 29

Filtering Pipeline Performance

Audits found residual irrelevance below the corpus-development target in sampled posts, and filtering required different numbers of stages across social-group distinctions. The corpus contains more than 527 million entries, while location-model accuracy varies by geographic resolution and degrades sharply for masked state-level text.

Filtering Pipeline Performance

  • Age, sexuality, and body weight reached the 10% target after one classification and pattern-matching sequence, while ability, race, and skin tone required multiple filtering stages.The latter pipeline added reinforced classification and secondary pattern-matching passes; the number of steps varied by distinction.Source: Filtering Pipeline Performance · pp. 29–30
  • Residual irrelevance was 3.3% (23/700) under the lenient rule, where one rater’s approval sufficed, and 6.1% (43/700) under the stringent rule, requiring agreement by both independent raters.Source: Filtering Pipeline Performance · p. 30

Key Descriptive Statistics

Location Model Evaluation

  • A manual audit of extracted location disclosures found accuracy of 95.7% for U.S. versus non-U.S., 90.2% for U.S. state, and 90.0% for non-U.S. region labels.State-level confidence scores are described as unreliable; at the finest tier, conservative thresholds had already removed ambiguous cases.Source: Location Model Evaluation · pp. 33, 35
  • Masking place names left U.S.-versus-non-U.S. macro F1 at .900 and world-region macro F1 at .789, but reduced state-level macro F1 from .698 to .371.Even when the masked model missed the exact state, its predictions tended to be nearby or in the correct U.S. census region, supporting state–year discourse-marker research questions.Source: Location Model Evaluation · p. 34
  • State shares among roughly 5% of users with specific U.S. state labels correlated with 2024 census shares at r = .93, or .88 when weighted by post frequency.The paper notes systematic deviations from census distributions.Source: Location Model Evaluation · pp. 34–35
  • The models achieved macro F1/raw accuracy of .913/.915 for U.S. versus non-U.S. users, .825/.881 for four global regions, and .698/.754 for 51 U.S. state classes.Source: Location Model Evaluation · pp. 33–34

Semantic Label Validation and Performance

This section evaluates ISAAC’s semantic labels and location labels using held-out performance, model agreement, and correspondence with external search patterns and major societal events. It includes state-level analysis of sexuality-related posting around marriage-equality votes.

Moralization

  • For moralized posts, precision was .733, recall .788, and F1 .760; overall accuracy and cross-class macro F1 were both .755.Evaluation used a held-out 10% MFRC slice (k = 2,682; 49.1% moralized).Source: Moralization · p. 36

Sentiment

  • Stanza classified 64.0% of posts as negative, compared with VADER’s 35.1% and TextBlob’s 20.3%; the lexicon-based models agreed more strongly with each other than with Stanza.Adjusting for each model’s baseline scoring tendencies and re-evaluating category assignments yielded only a minor performance improvement.Source: Sentiment · p. 36

Emotion

  • 77.7% of corpus posts received a two-of-three consensus emotion label, yielding roughly 410 million categorized records.Three-model intensity ICC ranged from .53 to .68 across all seven emotions; unanimous agreement occurred in 24.6% of cases.Source: Emotion · p. 37

Generalization

  • The 18-way situation entity classifier achieved accuracy .737 and macro F1 .514; collapsed ISAAC features yielded F1 .852 for genericity, .879 for eventivity, and .804 for boundedness/habituality.The corresponding accuracies were .860, .894, and .850, respectively.Source: Generalization · pp. 37–38

Volume-Based Validation

  • Monthly ISAAC text volumes showed strong baseline alignment with search interest, reaching ρ .96 for skin tone, .88 for ability, .82 for race, .79 for weight, and .33 for age.First-difference correlations remained positive across all social group distinctions, from ρ .03 to .35.Source: Volume-Based Validation · p. 40
  • Sexuality-related volume rose 46% above baseline during the June 2015 Obergefell ruling; race-related posts rose 219% in June 2020, the distinction’s largest month in 17 years.Race volume increased 65% in May 2020 and the June peak was at the 99.2nd percentile (+153% share-adjusted).Source: Volume-Based Validation · pp. 41–42
  • Skin-tone posts surged 173% in June 2020; ability-related discourse rose 32% above baseline during the January 2017 ACA repeal debate (+12% share-adjusted).The skin-tone peak was at the 99th percentile; the ability increase was at the 99th percentile raw and 97th percentile share-adjusted.Source: Volume-Based Validation · p. 42
  • Weight discourse climbed to +157% above baseline by summer 2022 and peaked at +194% raw and +224% share-adjusted in March 2023; age volume rose 12% at COVID-19’s onset but fell 4% share-adjusted.Weight discourse remained +138% elevated through December 2023; age’s share-adjusted result was at the 23rd percentile.Source: Volume-Based Validation · p. 42
  • The location check compared sexuality-related posting volume in each vote-holding state with the corresponding non-voting-state index at the same dates.Source: Volume-Based Validation · p. 43
  • Sexuality-related discourse rose to more than eight times the state’s trailing baseline in California and North Carolina, regardless of whether the vote aimed to restrict or extend marriage equality.Source: Volume-Based Validation · pp. 43–44
  • Voting states exceeded the non-voting median in all nine cases by factors of 1.07 to 3.85 (p = .004; geometric mean elevation 1.78, 95% CI [1.39, 2.34]).The exact two-sided sign test found the 9-out-of-9 outperformance statistically significant.Source: Volume-Based Validation · pp. 43–44, Figure 8 · p. 44

Semantic Validation

Semantic labels show distinct emotional and moralization profiles across social-group distinctions, including contrasts between race and ability discourse. The sexuality sentiment composite aligns with long-term Gallup marriage-equality opinion trends, although year-to-year changes do not systematically co-move.

Inter-Variable Alignment

  • Race and ability discourse were both highly moralized and negative, but differed sharply in their underlying emotional composition.Race was anger-dominant, whereas ability showed relatively high sadness and fear.Source: Semantic Validation · p. 45
  • Race was anger-dominant (26.6%), while ability had elevated sadness (14.2%) and fear (3.9%); sexuality was 58.4% neutral, and body weight was least moralized (49.3%) and most joyful (28.7%).The authors suggest supportive, self-improvement communities may dominate body-weight corpus volume.Source: Semantic Validation · p. 45
  • Moralized posts showed a pronounced anger increase across all six distinctions, largest for skin tone (+22 percentage points) and race (+19 percentage points).In weight-related discourse, moralization was associated with a 15-percentage-point decline in joy.Source: Semantic Validation · pp. 45–46
  • Marginalized poles had more negative sentiment for race, ability, sexuality, and body weight, but the direction reversed for old-age and dark-skin-tone poles.The mechanisms behind the age and skin-tone reversals remain unclear.Source: Semantic Validation · p. 46

Validation Against National Public Opinion

  • Sexuality-discourse sentiment was highly and significantly correlated with Gallup’s national marriage-equality support (r = .71, 95% CI [.35, .89], ρ = .69, p = .001), tracking macro shifts in public attitudes.Sentiment was most negative in 2008, corresponding with Gallup’s baseline support low of 40%, and most positive in 2023, as national support plateaued at 71%.Source: Semantic Validation · p. 47

Discussion

The discussion considers how ISAAC’s large-scale Reddit discourse data, validated measures, and model-uncertainty information support comparative, temporal, spatial, and cross-method research, while noting limits on what its platform-specific discourse markers can measure. It also addresses the modular pipeline’s capacity for expansion to additional groups, platforms, languages, and labels.

Discussion

  • Human coders found residual irrelevance below the single-digit development criterion across every group distinction and for both Reddit submissions and comments.The location model separated U.S. users, non-U.S. global regions, and extractable U.S. states with high sensitivity and specificity; semantic classifiers performed at or near state-of-the-art levels.Source: Discussion · p. 48
  • ISAAC’s volume aligned with independent search interest and major national and local events, while a sexuality-post sentiment composite tracked longitudinal survey data.The paper argues for validity through convergence across these benchmarks rather than reliance on any single comparison.Source: Discussion · p. 49

ISAAC’s Scope of Intended Use

  • ISAAC offers naturalistic text with temporal, geographic, and semantic indicators that can complement existing resources and support cross-method triangulation.Its fixed public framework also lets later studies build on, extend, or contest prior findings rather than restarting with study-specific data collection.Source: ISAAC’s Scope of Intended Use · pp. 49–50
  • Applying the same pipeline across categories enables direct comparisons, including uniformly high baseline moralization but divergent affective profiles for race and ability discourse.The text attributes cross-category variation to discursive patterns rather than pipeline artifacts.Source: ISAAC’s Scope of Intended Use · p. 50
  • The 17-year corpus supports temporal modeling of short- and long-term discourse dynamics, with search and public-tracking alignment supporting higher-resolution analyses.The passage identifies interrupted time-series and difference-in-differences designs for acute shocks and lead-lag analyses for gradual shifts in public opinion.Source: ISAAC’s Scope of Intended Use · p. 50
  • Location labels enable mapping discourse to regional benchmarks; localized surges around state marriage-equality ballots demonstrate spatial resolution for linking text to regional variables.The passage also describes linking text-derived markers to regional attitudinal data and external regional indicators.Source: ISAAC’s Scope of Intended Use · p. 51
  • Semantic labels support large-scale tests of social-science theories, including analysis of moralized framing, affective outrage, and generic or essentialist claims.The disaggregated multi-label architecture also supports training and evaluating classifiers on complex, unstructured social discourse without collapsing distinct semantic dimensions.Source: ISAAC’s Scope of Intended Use · pp. 51–52

Limitations and Possibilities for Expansion

  • ISAAC indexes English-language Reddit discourse shaped by the platform’s predominantly young, male, U.S.-based user base and does not provide representative population baselines.It supplies disaggregated discourse markers, not direct latent-attitude labels; inferring attitudes requires separate construct validation, including consideration of platform changes that can alter text expressions.Source: Limitations and Possibilities for… · p. 52
  • Independent model confidence scores and recalibration mappings let researchers customize precision–coverage trade-offs for their design requirements.Source: Limitations and Possibilities for… · p. 52
  • The repository publishes sentiment and emotion models’ outputs alongside alignment statistics, making cross-model variability transparent and analytically tractable.Source: Limitations and Possibilities for… · pp. 52–53
  • The modular pipeline supports expansion through new social-group dictionaries, alternative-platform data loaders, multilingual filters, and additional semantic labels.Each pathway is operationalized in the open-source repository.Source: Limitations and Possibilities for… · p. 53
  • The modular design lets independent laboratories build technical contributions on common, versioned infrastructure.Source: Limitations and Possibilities for… · p. 53

Conclusion

ISAAC integrates a standardized, decade-spanning corpus of social-group discourse with temporal, geographic, and semantic indicators across six categories. Its varied access routes and modular architecture support different research workflows and future expansion, while some supporting materials remain withheld for privacy.

Conclusion

  • ISAAC provides a standardized infrastructure for studying naturalistic social-group discourse at scale, with temporal, geographic, and semantic indicators across six categories.The authors describe its open, modular architecture as supporting cumulative quantitative social science.Source: Conclusion · p. 53
  • Multiple access pathways accommodate varied statistical and engineering requirements, while the modular architecture supports expansion to overcome current limitations.The released corpus, stratified samples, and validation data serve as the reproducibility anchor for the reported results.Source: Conclusion · p. 53, Authors’ Contributions · p. 57

Data Availability

  • Location-model training data are withheld because their self-disclosure-derived labels could enable cross-referencing to identifiable Reddit accounts.Access to the corpus requires acceptance of the project Data Use Agreement.Source: Data Availability · p. 56

Appendix A

The appendices document ethical safeguards and annotation procedures, then report validation evidence for the relevance pipeline. Human agreement and held-out classifier scores are strong overall, though complex race and skin-tone false positives required targeted retraining and a conservative threshold.

Appendix A

  • The project treats public Reddit discourse about social groups as ethically sensitive and uses anonymization and Data Use Agreement restrictions to mitigate re-identification risks.Source identifiers remain in the corpus for thread reconstruction and provenance, so the authors do not claim re-identification is technically prevented.Source: Ethics and Data Handling Protocols · pp. 72–73

Appendix B

  • The relevance annotation guide distinguishes irrelevant, relevant, and unclear references to a social distinction and its privileged or underprivileged group.The guide supports training algorithms for both relevance and attitudes toward the groups.Source: Relevance Annotation Instructions · p. 75

C.1 Annotator Agreement on Classifier Training Samples

  • Human annotations of the relevance-training samples showed substantial or almost-perfect agreement across all six categories.Weight had κ = .690 and 89.6% raw agreement; its lower κ reflected base-rate differences between raters rather than disagreement on clear cases.Source: §C.1 · p. 76

C.2 Relevance Classifier Performance

  • Held-out F1 scores ranged from .781 for skin tone to .987 for body weight across the six relevance classifiers.Complex false positives persisted for race and skin tone, prompting targeted retraining and a conservative threshold to meet the development target.Source: §C.2 · pp. 76–77

C.3 Residual Irrelevance Across Development Stages

  • Figure C1 tracks residual irrelevance across filtering stages in stratified development samples, with separate audits for comments and submissions.Ability received an additional pattern pass, and the comment-tuned pipeline was checked for transfer to submissions.Source: §C.3 · p. 78

C.4 Corpus-Level Validation Procedures

This section describes corpus-level validation procedures for local-event and longitudinal sentiment analyses, reports high moralization in ISAAC’s social-group discourse, and outlines its scalable user-location labeling approach. It also presents hand-reviewed evidence supporting location-annotation accuracy at national, state, and regional levels, and notes the size of the annotated user set and privacy-related withholding of training data.

C.4.1. Localized Event Analysis

  • For marriage-equality ballot measures, sexuality-related posting volume in voting states was compared with each state’s trailing baseline and a simultaneous non-voting-jurisdiction ratio.The baseline spans months −14 to −3, excluding the two months before each vote; identical windows were applied to placebo states.Source: §C.4.1 · p. 79

C.4.2. Sentiment Composite

  • The longitudinal sentiment index samples 500 sexuality-related comments from each monthly archive from 2007 to 2023 and averages annual standardized outputs from VADER, TextBlob, and Stanza.The comparison against national public opinion uses k = 101,888.Source: §C.4.2 · p. 80

C.5 Affective–Moral Profiles Across Distinctions

  • 49% to 74% of ISAAC discourse is marked as moralized across social-group distinctions, with the contrast attributed to data selection and measurement.ISAAC filters for socially scrutinized topics and uses a binary moralization measure; the cited everyday-speech and personal-post estimates are 2% to 5%.Source: §C.5 · p. 80

D.1. Overview

  • ISAAC retains self-statement location labeling while using coarser state-or-region labels, per-feature logistic-regression mixtures, confidence thresholds with an unknown fallback, and persistent caching.The modular approach is designed to make full-archive inference and future corpus extension feasible with ordinary research resources.Source: §D.1 · p. 83

D.2.1 User Location Annotation

  • The location training set adapts self-location candidate sampling and gazetteer resolution to 2007–2023, with the 2019–2023 candidates as a novel addition.Sampling excludes travel or migration terms; resolution uses GeoNames place and administrative-region entries with a 15,000-person minimum, then geocodes candidates.Source: §D.2.1 · pp. 83–84
  • U.S. classification was correct in 88 of 92 cases (95.7%), state classification within the U.S. in 46 of 51 (90.2%), and region classification outside the U.S. in 36 of 40 (90.0%).Users whose conversation context made their location unverifiable at the indicated level were excluded from these validation counts.Source: §D.2.1 · p. 85
  • The final annotated set contained 119,019 unique users: 68,336 U.S. and 50,683 non-U.S., a ratio comparable to Harrigian, 2018.The U.S. and non-U.S. geocoded sets were stored separately for more fine-grained model training.Source: §D.2.1 · p. 85
  • For user-privacy reasons, the training dataset is not distributed publicly; researchers interested in reuse should contact the corresponding author.Source: §D.2.1 · p. 85

D.2.2 User Feature Views

This section presents scalable user-level geographic feature views and a hierarchical location-labeling model for the Reddit archive, including its prediction branches, confidence thresholds, and fallback rules. It also describes model evaluation under masking and validation through state-level event analyses.

D.2.2 User Feature Views

  • The three views—words, subreddits, and activity hours—use each labeled user’s entire Reddit history rather than only posts curated into ISAAC.The views follow feature classes used by Harrigian (2018), while the pipeline’s substantive change is downstream in how they are combined.Source: §D.2.2 · p. 85
  • The views supply different geographic signals: words can be sparse and noisy, subreddit participation is thinner but less ambiguous, and activity hours are dense but weak time-zone proxies.Activity hours are represented as a 24-bin histogram of post timestamps in UTC.Source: §D.2.2 · p. 86

D.2.3 Engineering for Scale

  • A streaming pipeline used deterministic username-hash buckets and per-bucket partial-count merges to process user features without cross-bucket coordination.Partial counts were periodically flushed to disk as the pipeline streamed through posts.Source: §D.2.3 · pp. 87–88
  • The adjusted approach validated Harrigian’s feature families on a corpus two orders of magnitude larger using workstation-grade resources.The 2007–2023 raw Reddit dump contains 2.29 TB of compressed data.Source: §D.2.3 · pp. 87–88

D.3.1 Model Design

  • The location model combines separate logistic regressions for word and structured features in a hierarchical cascade for U.S. status, states, or non-U.S. regions.The structured features are subreddit and activity-hour views; the cascade applies separate classifiers to U.S.-versus-non-U.S., state, and non-U.S. region prediction.Source: §D.3.1 · pp. 88–89
  • U.S.-versus-non-U.S. prediction achieved F1 above 0.90 with and without location mentions, region F1 was around 0.80, and state prediction degraded substantially under masking.The state-level degradation motivates confidence-aware fallback in the model design.Source: §D.3.1 · p. 90
  • The model accepts U.S.-versus-non-U.S. predictions above 0.60 and assigns UNK otherwise; U.S. state predictions need a 0.05 margin or fall back to the top-level label.The 0.60 cutoff represents a clear 60–40 preference.Source: §D.3.1 · p. 90
  • After a non-U.S. determination, the regional prediction must exceed its competitor by at least 0.10, a 40% premium over the 25% chance level for four classes.The margin is intended to exclude near-ties.Source: §D.3.1 · p. 90
  • The 0.05 state-level margin is 2.5 times the 2% chance level, and the thresholds seek roughly comparable evidence above chance across tasks.The passage attributes the different cutoffs to the branching and difficulty of each prediction task.Source: §D.3.1 · pp. 90–91
  • The resulting labels are validated through state-level event-locked analyses described in the Results section.The cited analysis is titled “Localized Event Spikes.”Source: §D.3.1 · p. 91

D.3.2 Training

D.3.2 trains location classifiers across word and structural feature views and selects mixture weights using masked validation. D.3.3 applies the model to monthly ISAAC files using bounded history windows and caching.

D.3.2 Training

  • Preprocessing converts word histories to TF–IDF, subreddit counts to log1p-transformed L1-normalized frequencies, and activity hours to L1-normalized 24-bin histograms.Source: §D.3.2 · pp. 91–92
  • For each task and feature set, logistic regressions are trained with balanced classes, and regular and masked held-out metrics are logged.The task set is top, state, and region; models use C = 3.0 and up to 200 iterations.Source: §D.3.2 · p. 92
  • Mixture weights sweep from 0.00 to 0.50 in 0.05 steps, selecting the masked-validation top-1 accuracy maximum as a proxy for users without self-statements.Source: §D.3.2 · pp. 92–93
  • Accuracy-optimal and calibration-optimal mixture weights differ at every hierarchy level, and no weight setting calibrates scores at the most granular level.Source: §D.3.2 · p. 93

D.3.3 Inference at Scale (label_location)

  • The deployed labeler combines the feature models’ leading candidates; on held-out users, this changes top-1 accuracy by at most 0.4 percentage points and mean confidence by .007.Source: §D.3.3 · p. 93
  • Inference uses complete Reddit-month windows centered on each curated month, with default windows of 30 months per side for 2007–2019 and 20 for 2020–2023.Per-user sample caps are 50 posts for 2007–2019 and 25 for 2020–2023, matching training-time sample distributions.Source: §D.3.3 · p. 94
  • Persistent caches reuse user labels and feature counts across overlapping windows, while post-ID checks prevent curated posts from being counted twice.Source: §D.3.3 · pp. 94–95
  • Users with fewer than 10 usable posts receive UNK, and full labeling requires substantial resources, including up to 25 GB peak memory per month.The authors suggest running a full ISAAC labeling run only on a CPU-heavy cluster with good memory resources; caches also require significant storage.Source: §D.3.3 · pp. 94–95

D.3.4 Label Distribution

The appendix evaluates ISAAC’s estimated location labels, finding close alignment with state population shares but uneven geographic representation. Confidence is useful for ranking users at coarser geographic levels, while state-level confidence is unreliable and requires special handling.

D.3.4 Label Distribution

  • Pearson’s r = .93 for ISAAC accounts’ state shares versus Census population estimates (.88 when weighted by post frequency), with state-level over- and under-representation remaining systematic.Small, distinctive states tended to be over-represented, while populous states sharing a metro area with a neighbor tended to be under-represented.Source: §D.3.4 · p. 95

Figure D1

  • Figure D1 shows location-label distributions across hierarchy levels and group distinctions; its state panel ranks states, while shaded bands show variation across distinctions.The figure also reports world-region distributions among accounts assigned at the region level.Source: §D.3.4 · p. 95, Figure D1 · p. 97

D.3.5 Performance

  • Held-out classifier performance under standard and masked conditions is reported in Table 4, with state-level values shown in Figure D2.The supplied passages identify the evaluation displays but do not report their performance values.Source: §D.3.5 · p. 98, Figure D2 · p. 98

D.3.6 Calibration of the Reported Confidence Values

  • Confidence scores strongly rank accuracy for U.S.-versus-non-U.S. and world-region classifications (r = .98 and r = .92, respectively), but are not reliably predictive at the U.S. state level, where accuracy clustered between .73 and .83.Across confidence scores, accuracy rose from .78 to .97 for U.S.-versus-non-U.S. classification and from .53 to .92 for world regions.Source: §D.3.6 · p. 99

D.3.7 Confidence Recalibration

  • Isotonic recalibration reduced emitted-label ECE from .09 to .011 for U.S./non-U.S. and from .11 to .007 for region, without changing labels or their ranking.No state mapping is released because state confidence is dissociated from prediction quality; state assignments should be treated as categorical labels with a uniform error rate, not confidence-weighted.Source: §D.3.7 · pp. 99–100
  • Recalibration holds in aggregate for users whose locations were recoverable from explicit self-disclosure; per-subgroup calibration was not assessed.State labels were correct about 77% of the time for users with explicit location mentions and about 52% for those without them.Source: §D.3.7 · p. 100

Appendix E

Appendix E documents ISAAC’s pipeline as reusable resources for reproducing or adapting the corpus. It describes the command-line workflow, scale-oriented computation support, and implementation choices behind key filtering and labeling steps.

Appendix E

  • Appendix E recasts the corpus-construction method as reusable resources for recreating or extending ISAAC to new groups, constructs, languages, platforms, or time periods.The appendix assumes basic command-line familiarity.Source: Reusing and Adapting the ISAAC Pipeline · p. 102

E.1 Resources and the Command-Line Interface

  • Table E1 maps main-text steps to resource scripts; keyword filtering must run first because it is the only resource that reads raw data.Other resources follow the canonical table order unless customized.Source: §E.1 · p. 102, Table E1 · p. 102

E.2 Support for Corpus-Scale Computation

  • The pipeline supports automatic resumption, workstation and SLURM parallelization, and GPU acceleration for compute-intensive labelers; integrity checks reconcile outputs against inputs.The checks can catch truncated or corrupted files.Source: §E.2 · p. 105

E.3 Algorithmic Details Behind the Main-Text Descriptions

  • Aho–Corasick matching scans multiple keywords in input-linear time, making exact matching tractable against trillions of raw tokens.The implementation is parallelized across workstation workers.Source: §E.3 · p. 106
  • Race and skin-tone relevance classifiers use asymmetric penalties and a 0.6 relevance threshold, favoring precision over recall; other groups use the most-likely-class rule.Language filtering uses fastText for accuracy and throughput, while complex patterns use CPU-parallelized Intel Hyperscan.Source: §E.3 · p. 106

E.4 Adaptation Recipes

The appendix gives recipes for adapting ISAAC and inventories its released assets and access routes. These include direct files, browser-based SQL, a Python package, and a HuggingFace mirror, with some data and model access constrained by agreements or availability.

E.4 Adaptation Recipes

  • Researchers can adapt the pipeline to new data sources, languages, social groups, constructs, or location models; extending beyond 2023 depends on access to newer platform data.Location training resources reproduce feature preparation, hierarchical classifiers, and feature-class weighting, but ISAAC’s location training data are not released.Source: §E.4 · p. 107

E.5 Released Assets

  • Released assets include relevance classifiers and samples, moralization and generalization models, location checkpoints, recalibration maps, keyword lists, configurations, and demo applications.Location checkpoints are available by email under the ISAAC Model Use Agreement, without their training data.Source: §E.5 · p. 107, §E.4 · p. 107

Appendix F

  • The corpus is organized by social group and month from January 2007 through December 2023, in Parquet or CSV, with Data Use Agreement acceptance required for access.The appendix documents technical routes for scripted, composable, or pipeline-integrated use.Source: Access Routes for Technical Users · p. 109

F.1 Direct Downloads

  • Direct downloads provide manifest-catalogued files that can be retrieved through Globus and used in systems such as DuckDB or Spark without conversion.The manifest lists each file’s size and row count.Source: §F.1 · p. 109

F.2 SQL Query Playground

  • The browser-based SQL playground queries published Parquet files locally through DuckDB-Wasm, sending neither queries nor corpus data through project servers.It suits aggregation and moderate result sets; display is capped at 10,000 rows and CSV exports at 100,000.Source: §F.2 · pp. 109–110

F.3 Python Package (isaac-data)

  • The isaac-data Python package catalogs and loads selected groups, months, and columns into pandas, and can download whole files with resumable, cached transfers.Its stratified sampling is reproducible and spread equally across selected months.Source: §F.3 · p. 110

F.4 HuggingFace Datasets and Models Hub Interfaces

  • The gated HuggingFace dataset mirror supports streaming a social group or materializing it locally for Python machine-learning workflows.Access requires accepting the Data Use Agreement and agreeing to cite the project.Source: §F.4 · pp. 110–111

F.5 Implementation Notes

  • The website’s Python and Shell backend and NodeJS frontend are publicly available under the MIT License for inspection and further development.The backend is optimized for efficient sampling and compression at corpus scale.Source: §F.5 · p. 111
Loading 2609.27059v1…