Can Large Language Models Transform Computational Social Science?

Motivation

• Existing supervised CSS methods require extensive human-annotated training data, whereas unsupervised methods can produce uninterpretable outputs, constraining the theories and subjects researchers can study.The work investigates whether zero-shot LLM labels and summary codes can remove these constraints.Source: §1 · p. 2

Research questions

• The research framework examines zero-shot viability, model selection, domain utility, and functionality for augmenting human annotation and supporting classification or summary-explanatory tasks.These questions address whether LLMs can match human reliability, how model properties affect performance, whether utility varies across fields, and which task types they support.Source: §1 · pp. 2–3

Evaluation design

• The study evaluates 13 language models on 24 representative CSS tasks spanning classification, parsing, and generation through zero-shot prompting.The benchmark covers models with differing architectures, sizes, pretraining, and fine-tuning paradigms, across linguistic, psychological, cultural, and explanatory use cases.Source: Figure 1 · p. 2, §1 · p. 3

Findings

• LLMs generally do not match carefully fine-tuned classifiers, but achieve fair human agreement across diverse social-science classification tasks and can support iterative joint labeling.Their benefits compound as models scale, suggesting faster and improved text analysis without fully replacing human annotation.Source: §1 · p. 3
• Leading LLMs generate explanations matching or exceeding dataset references in relevance, coherence, faithfulness, and fluency, with humans preferring model outputs 50% of the time.This supports extending human-AI collaboration from labeling to jointly coding new constructs, analyses, and summaries.Source: §1 · p. 3

2. An Overview of CSS

• CSS combines computational analysis of behavioral and linguistic data with objective hypothesis formation and testing, distinguishing it from the more subjective focus of Digital Humanities.The overview surveys applications across history, literature, linguistics, political science, psychology, and sociology.Source: §2 · pp. 3–4

2.1 History

• Historical NLP uses event extraction to represent event locations, times, causes, and participants, alongside analysis of discourse and semantic change.Event extraction comprises event detection and event argument extraction, benchmarked later in §3.3.1 and §3.3.2.Source: §2.1 · p. 4

2.2 Literature

• Literary NLP addresses narratives by parsing agents, relationships, and events, while also supporting social-role, emotion, figurative-language, and humor analysis.Named entity recognition and toponym resolution can identify settings, whereas topic modeling for themes is outside LLM scope.Source: §2.2 · p. 4

2.3 Linguistics

• Computational sociolinguistics uses language variation to study social identity and dialect structure across groups, regions, classes, ages, and genders.These analyses support classification of dialects, measurement across population strata, and study of language-change diffusion.Source: §2.3 · pp. 4–5

2.4 Political Science

• Political-science NLP studies ideology, stance, framing, persuasion, and discourse structure to analyze agendas, public opinion, voting, policy, movements, and international relations.The paper benchmarks ideology, stance, entity framing, persuasion strategy, and discourse-act classification.Source: §2.4 · p. 5

2.5 Psychology

• Psychological applications connect personality-related factors with political, social, linguistic, humorous, persuasive, and interpersonal behavior, including empathy, emotional summarization, and positive reframing.The evaluated tasks include humor, persuasion, politeness, empathy detection, emotional aspect-based summarization, and cognitive-behavioral positive reframing.Source: §2.5 · pp. 5–6

2.6 Sociology

• Sociological NLP examines collective life by tracing content diffusion across actors, covering micro-scale power and roles and macro-scale norms, contagion, collective behavior, and movements.The paper applies LLMs to power relations, unhealthy conversations, emotion, hateful-group speech, and figurative language.Source: §2.6 · p. 6

Selection rationale

• The task selection targets field-specific CSS challenges, including expert taxonomies, large label spaces, temporal grounding, and domain-specific parsing schemes.It is designed as a representative, though not exhaustive, survey of CSS needs and to address Research Questions 1–4.Source: §3 · p. 6

Task organization

• The benchmark comprises 20 classification tasks—10 utterance-level, 6 conversation-level, and 4 document-level—plus 5 generation tasks for explaining constructs and restructuring text.The generation tasks apply psychological theories to restructure text.Source: §3 · p. 6

Distinctive scope

• Unlike BIG-Bench, the benchmark covers dialect, discourse relations, character tropes, event detection, ideology, misinformation, persuasion, politeness, power relations, semantic change, and stance, including extensive conversation-level analysis.The authors describe this as the first extensive effort to study LLM performance on tasks critical to social scientists.Source: §3 · p. 6

3.1 Utterance-Level Classification

• The evaluation covers dialect, hate speech, figurative language, emotion, humor, misinformation, ideology, persuasion, semantic change, and stance classification at the utterance level.Utterances support analysis of linguistic phenomena, human states, communicative intent, and stable traits such as stance and ideology.Source: §3.1 · p. 7

3.1.1 Dialect Features.

• Dialect classification uses expert-labeled Indian English utterances mapped to 22 grammatical features, including inversion, copula, tense, and aspect patterns.Indian English is presented as a widely spoken low-resource variety, and the dataset is one of the few hand-labeled by a domain expert.Source: §3.1.1 · p. 7

3.1.2 Emotions.

• Emotion detection uses weakly labeled Twitter data organized into Plutchik’s eight categories: anger, anticipation, disgust, fear, joy, sadness, surprise, and trust.The task is motivated by applications in psychology, political science, emotional contagion, and economic-market research.Source: §3.1.2 · p. 7

3.1.3 Figurative Language.

• Figurative-language classification uses FLUTE’s 9k premise–hypothesis pairs to identify sarcasm, simile, metaphor, or idiom.The benchmark is described as comprehensive and draws examples from multiple prior datasets.Source: §3.1.3 · p. 8

3.1.4 Hate Speech.

• Hate-speech detection uses Latent Hatred’s fine-grained taxonomy, assigning coded or indirect utterances to six categories including violence, inferiority, irony, stereotypes, threats, and white grievance.The task emphasizes inferring a subtle social taxonomy from U.S. hate-group speech.Source: §3.1.4 · p. 8

3.1.5 Humor; 3.1.6 Ideology; 3.1.7 Misinformation; 3.1.8 Persuasion; 3.1.9 Stance

• The remaining social classification tasks include binary humor detection over approximately 16K jokes, perceived ideology classification, binary misinformation detection, six-class persuasion strategy classification, and three-way stance detection.The datasets include jokes from multiple sources, ideological book sentences, fact-checked COVID-19, climate-change, and cancer headlines, Reddit pizza requests, and tweets about Donald Trump.Source: §3.1.5 · p. 8, §3.1.6 · pp. 8–9, §3.1.7 · p. 9

3.1.8 Persuasion

• Persuasion is evaluated on RAOP requests labeled with Evidence, Impact, Politeness, Reciprocity, Scarcity, and Emotion strategies.The task classifies utterance-level requests in which Reddit users attempt to obtain free food.Source: §3.1.8 · p. 9

3.1.10 Semantic Change.

• Semantic-change detection uses TempoWiC to classify whether two sentences containing the same word use the same or different sense.Pairwise classification can support clustering surface-form usages into sense groups when classification is perfect.Source: §3.1.10 · p. 10

3.2.1 Discourse Acts.

• The discourse-act task classifies Reddit comments in context using a focused taxonomy that retains seven evaluated categories after omitting sparse and catch-all labels.The taxonomy covers questions, answers, elaborations, announcements, appreciation, agreements, disagreements, negative reactions, and humor; Announcements, Negative reactions, and Other are omitted from evaluation because of limited examples.Source: §3.2.1 · pp. 10–11

3.2.2 Empathy.

• The empathy task classifies a listener’s reply to a seeker’s post as Strong Exploration, Weak Exploration, or No Exploration.The evaluation uses the TalkLife dataset and focuses on Exploration within EPITOME’s multi-stage scheme because open questions have been effective for peer support.Source: §3.2.2 · p. 11

3.2.3 Persuasion.

• The persuasion task predicts whether a reply thread in r/ChangeMyView successfully changes the persuadee’s mind, using the awarded delta as a binary label.The Winning Arguments Corpus contains 3,051 conversations in which a persuader attempts to convince a persuadee.Source: §3.2.3 · p. 11

3.2.4 Power and Status.

• The power-and-status task uses an editor’s complete Wikipedia Talk Pages comment history to classify whether the editor is an administrator.Power is represented as a binary label, with linguistic accommodation and features such as politeness and hedging motivating the analysis.Source: §3.2.4 · p. 12

3.2.5 Toxicity Prediction.

• The toxicity task predicts from the first two messages whether a Wikipedia conversation will later contain toxic language.Using the Conversations Gone Awry corpus, the task targets early cues that may precede overt toxicity.Source: §3.2.5 · pp. 12–13

3.2.6 Politeness.

• The politeness task classifies Wikipedia requests as Polite, Neutral, or Impolite according to Mechanical Turk annotators’ interpretations of workplace norms.The Stanford Politeness Corpus contains requests between Wikipedia contributors, and the task tests recognition of conversational social norms.Source: §3.2.6 · p. 13

3.3 Document-Level Classification

• Document-level CSS analyzes media documents as unified narratives to code ideological content, portrayed events, and the agents, roles, or tropes they exhibit.Documents comprise collections of utterances forming a single narrative and can encode sequences of ideas, temporal events, and interpersonal relationships.Source: §3.3 · p. 13

3.3.1 Event Detection.

• Event detection identifies whether sentences describe new events, using Hippocorpus’s 6,854 crowdworker-collected stories as a benchmark.The evaluated task is binary sentence-level event classification; in the example, lines A and D denote new events.Source: §3.3.1 · pp. 13–14

3.3.2 Event Argument Extraction.

• Event argument extraction fills predefined event templates by identifying related concepts and participant roles, supporting analysis of sociopolitical, economic, narrative, legal, health, and policy domains.WikiEvents benchmarks document-level extraction from news articles with 67 event types in a three-level KAIROS ontology, richer than ACE’s 33 event types.Source: §3.3.2 · p. 14

3.3.3 Ideology.

• Ideology detection classifies entire news articles from the Article Bias Corpus into exactly one of three labels: Left, Right, or Centrist.The task requires relating stances distributed across an article to its political leaning, even when views are diffuse rather than concentrated in slogans or phrases.Source: §3.3.3 · pp. 14–15

3.3.4 Roles and Tropes.

• Character trope classification infers which of 72 movie tropes best fits a character using only the character’s quotes in the CMU Movie Corpus.The broader role and persona tasks support research on personality, group dynamics, narrative agents, stereotypical bias, and representational harms.Source: §3.3.4 · p. 15

3.4 Generation Tasks

• Generation tasks assess LLMs’ ability to summarize relevant aspects, explain hidden social meaning, and stylistically restructure utterances.These tasks address RQ4 Functionality by testing generative explanations and summaries alongside classification.Source: §3.4 · p. 15

3.4.1 Emotion-Specific Summarization.

• Aspect-based summarization identifies experiences triggering a target emotion, using COVIDET’s 1,883 Reddit posts and Plutchik emotions (Zhan et al. 2022).The task targets domain-specific summaries for social scientists studying population-level emotional responses, rather than generic summarization (Ahuja et al. 2022).Source: §3.4.1 · pp. 15–16

3.4.2 Figurative Language

• Figurative-language explanation generates a sentence explaining whether a figurative sentence entails or contradicts its literal counterpart in FLUTE.FLUTE contains 9k literal–figurative sentence pairs with entailed or contradictory meanings.Source: §3.4.2 · p. 16

3.4.3 Implied Misinformation Explanation.

• Implied misinformation explanation converts a news headline into its underlying writer’s intent in plain English.The task uses the Misinfo Reaction Frames corpus (Gabriel et al. 2022) and supports understanding beyond black-box classification.Source: §3.4.3 · p. 16

3.4.4 Social Bias Inference.

• Social-bias inference generates free-text explanations of why hate speech targets a specific demographic subgroup, mirroring SBIC’s conditional-generation setup (Sap et al. 2020a).SBIC contains 34K annotated inferences linking hate speech to explanations of subgroup targeting.Source: §3.4.4 · p. 16

3.4.5 Positive Reframing.

• Positive reframing translates distorted negative utterances into complementary positive viewpoints using cognitive behavioral therapy strategies without contradicting the speaker’s meaning (Rothbaum et al., 2000).The task is motivated by scaling mental-health counseling, volunteer-listener training, and CBT technique instruction.Source: §3.4.5 · p. 16

4.1 Model Selection and Baselines

• The study evaluates FLAN-T5 across six sizes, OpenAI GPT-3 variants, gpt-3.5-turbo, and GPT-4, alongside supervised RoBERTa-large classification and T5-base generation baselines.FLAN-T5 ranges from 80M to 11B parameters, with UL2 at 20B; GPT-4 is described as a 1.7-trillion-parameter multimodal model. Baseline hyperparameters are selected by validation F1 or BLEU and averaged across three random seeds.Source: §4.1 · p. 17

4.2 Prompt Engineering

• Prompt engineering uses task-specific natural-language instructions, five semantically equivalent prompts, and guidelines that constrain outputs and reduce uncertainty, refusal, and overgeneralization.An initial expert-written prompt is supplemented with four gpt-3.5-turbo paraphrases, and results are averaged across all five prompts; decoding uses temperature zero and valid-class constraints where available.Source: §4.2 · pp. 17–18

4.3 Test Set Construction

• Each task uses a class-stratified sample of at most 500 test instances, with dataset sizes and class counts reported in Table 2 and released materials supporting future comparison.When no test-set designation exists, sampling is performed from the entire dataset.Source: §4.3 · p. 19, Table 2 · p. 19

4.4 Evaluation Metrics

• Classification and structured parsing are evaluated with macro F1 averaged over five prompt perturbations, whereas generation requires human assessment because overlap and embedding metrics poorly track observed quality.String matching evaluates alphabetical candidate labels; BLEU, BERTScore, and BLEURT showed inadequate or inconsistent alignment with manual judgments for generation.Source: §4.4 · p. 19
• Human evaluation combines blind expert Likert scoring and forced-choice ranking, with unanimous voting used to determine reported metrics and model-versus-human differences.Experts assess faithfulness, coherence, relevance, and fluency on 1–5 scales; ranking evaluations compare generated explanations with gold human annotations, while author annotation was used after pilot crowdworker ratings proved unstable.Source: §4.4 · p. 20

5. Classification Results

• Zero-shot results rarely exceed supervised RoBERTa baselines, but larger instruction-tuned open-source LLMs are preferable for classification tasks.Table 3 reports macro F-1 averaged across five prompt permutations for zero-shot models and three seeds for supervised baselines.Source: §5 · p. 20, Table 3 · p. 21

5.1.1 Zero-Shot Viability.

• Best models reach 77.4 F1 on misinformation detection, with 8/17 tasks achieving moderate-to-good human agreement and some exceeding reported inter-human agreement.FLAN-UL2 reaches κ = 0.55 on misinformation detection versus κ = 0.51 inter-human agreement; other strong tasks include Stance, Emotion, Figurative Language, and utterance-level Ideology Classification.Source: §5.1.1 · p. 21, Table 4 · p. 22

5.1.2 Zero-Shot Error Analysis.

• Error analysis shows models often share systematic mistakes, including generic-label overprediction, taxonomy misalignment, and difficulty interpreting questions, identity terms, and subtle social-science categories.In Figurative Language, models commonly classify similes as metaphors; in Emotion, at least 18/50 disagreements were judged gold mislabels; and in Implicit Hate, 86% of stereotypical false-positives contained identity terms.Source: §5.1.2 · pp. 23–24

5.1.3 Few-Shot Viability.

• Few-shot prompting does not uniformly improve classification, producing small gains mainly on subjective Empathy and Persuasion tasks and requiring further engineering for substantial improvements.Empathy improves for 2 of 6 model sizes and Persuasion for 5 of 6 model sizes.Source: §5.1.3 · pp. 24–25, Table 5 · p. 25

5.2 Model-Selection (RQ2)

• Model choice matters: gpt-3.5-turbo is best for structured event extraction, whereas FLAN-UL2 is best for most classification tasks; scale helps FLAN more consistently than OpenAI GPT models.Each order-of-magnitude increase in FLAN size yields a 5.0-point median average-task improvement, while RLHF improves text-davinci-003 and gpt-3.5-turbo over text-davinci-002 by 3.5 points.Source: §5.2 · pp. 25–27, Figure 3 · p. 26

5.3 Domain-Utility (RQ3)

• Performance overlaps across academic fields, suggesting no strong disciplinary bias, but task difficulty increases for complex document inputs and subtle expert taxonomies.Misinformation detection reaches F1=77.4 while implicit hate reaches F1=32.3; document-level tasks are among the lowest-performing tasks.Source: §5.3 · p. 27

Evaluation Setup

• FLAN models were excluded after manual inspection found that they failed to follow instructions, so evaluation focused on the OpenAI suite.The generation experiments use the human evaluation setup described in §4.4.Source: §6 · p. 27

Human Scoring Evaluation

• Larger models produce useful, relevant, coherent, and fluent explanations, whereas smaller models often fail task specifications, including faithfulness scores below 2 out of 5 in COVIDET, FLUTE, and SBIC.For Positive Reframing and Figurative Language explanation, text-davinci-003 matches gold-standard quality and outperforms the fine-tuned baseline.Source: §6 · p. 28

Human Ranking Evaluation

• Model generations outrank gold human references at least 38% of the time, with the best models approaching parity and RLHF models showing significant benefits.These generation tasks systematically benefit from human feedback, unlike the classification results.Source: §6 · p. 28, Table 7 · p. 30

Human–AI Collaboration

• Because no model substantially outperforms human annotation, researchers could correct generations with minor edits rather than code explanations from scratch.The results suggest a collaboration model in which 2 to 3 of every five generations require no additional annotator effort, though its effect on cognitive load remains for future work.Source: §6 · p. 28

Human Scoring Evaluation

• Leading generative models match or exceed fine-tuned baselines and human references on faithfulness, relevance, coherence, and fluency across zero-shot generation tasks.Table 6 reports average expert ratings on 1–5 Likert scales for davinci-003 and GPT 3.5 against baselines and human references.Source: §6 · pp. 27–28, Table 6 · p. 29

Evaluation Challenges

• Reference-based metrics including BLEU, BERTScore, and BLEURT produced uninterpretable scores, motivating costly and less reproducible reference-free expert evaluation.As models approach or exceed gold-reference quality, reference-based scoring may no longer measure true utility, underscoring the need for new CSS evaluation procedures.Source: §6 · pp. 29–30

7. Discussion

• LLMs can augment, but not entirely replace, the traditional CSS research pipeline, according to a comprehensive evaluation across representative CSS tasks and supervised baselines.The evaluation is intended to guide when LLMs are suitable and which models fit different research needs.Source: §7 · p. 30
• CSS researchers should integrate LLMs-in-the-loop for large-scale labeling, prioritize open-source models for classification, and use larger instruction-tuned models for high-quality generations.The recommended generation priorities are faithfulness, relevance, coherence, and fluency.Source: §7 · p. 30
• The discussion recommends investigating new CSS paradigms built on LLMs’ multipurpose capabilities over the long term.These opportunities extend beyond annotation and include new research paradigms.Source: §7 · p. 30
• Human-AI collaboration can scale annotation while reducing variance in gold labels, with human annotation validating model-provided annotations.Figure 5 illustrates this workflow using a misinformation-detection example.Source: Figure 5 · p. 31, §7 · p. 31

7.1 How Annotation Fits Into CSS

• With only 25 gold annotations, DSL achieved confidence-interval coverage above 86% across all 17 classification tasks, including those with the lowest LLM pseudo-label accuracy.Using FLAN-UL2 pseudo-labels, DSL also computed unbiased regressions with only a handful of gold-labeled instances.Source: §7.1 · p. 31
• Averaging even highly accurate LLM annotations can bias downstream estimates and produce invalid confidence intervals.Social scientists generally use labels and codes for downstream inferential statistics, so valid estimators and confidence intervals are essential.Source: §7.1 · p. 31

7.2 LLMs Can Augment Annotation

• LLM annotators achieved fair to strong human agreement on 12 of 17 classification tasks, but unusably low performance on some tasks prevents replacing human annotators.High internal agreement can persist despite inaccurate predictions, so unchecked overconfidence may corrupt CSS conclusions and policy decisions.Source: §7.2 · p. 31
• Combining LLM pseudo-labels with human annotation can produce unbiased estimators with 7–13% monetary savings when LLMs match human agreement levels.The authors recommend reinvesting anticipated efficiency savings to train expert annotators rather than replacing them with crowdworkers.Source: §7.2 · p. 32
• Prompted LLMs provide low-cost, better-than-chance labels that human validation can iteratively refine through weak supervision, targeted cleaning, and active learning.This flywheel focuses human expertise where it is most needed and supports dataset construction from curated LLM generations.Source: §7.2 · p. 32

7.2 LLMs Can Augment Annotation · 7.3 When To Use What LLM

• Generation LLMs were rated superior to human gold annotations in over 38% of evaluations across all five tasks, with proprietary models approaching human parity.The authors identify gpt-3.5-turbo as the leading generation model by price and performance, while cautioning that generated explanations are not causal and may hallucinate.Source: §7.2 · p. 32, §7.3 · p. 33

7.3 When To Use What LLM

• For CSS classification, open-source FLAN models match state-of-the-art closed-source LLMs and offer fine-tuning and programmatic constraints when researchers have suitable GPUs.Their low marginal cost can benefit continuous monitoring and enormous-scale analysis, whereas OpenAI instruction-tuned models cannot currently all be further fine-tuned.Source: §7.3 · p. 32
• For hardware-constrained groups, gpt-3.5-turbo is generally more cost-efficient than hosting FLAN-T5-XXL, which costs $170 per day and carries greater operational overhead.That hosting cost is described as equivalent to processing approximately 50 million words with gpt-3.5-turbo.Source: §7.3 · pp. 32–33
• All LLMs struggle with conversations and full documents, while weak cross-document reasoning limits applications such as topic modeling and near-term impact in several discourse-focused fields.These limitations motivate research on conversations, long documents, and cross-document reasoning, alongside continued caution about generation errors and hallucinations.Source: §7.3 · p. 33

• LLMs can blend supervised and unsupervised CSS by generating hypotheses from summarized data and flexibly labeling human samples for low-cost hypothesis testing.Their ability to retrieve, label, and condense information at scale supports new research paradigms.Source: §7.4 · p. 33

Simulation

• LLMs offer more expressive CSS simulations by replicating human biases without explicit conditioning, extending beyond rule-based utility-function models.Recent applications simulate social computing systems, communities, public opinion, and subjective experience description (Bail 2023; Jones and Steinhardt 2022; Koralus and Wang-Ma´scianica 2023; Park et al. 2022; Park et al. 2023; Argyle et al. 2022; Chu et al. 2023).Source: §7.4 · pp. 33–34
• Simulated samples have limited predictive and explanatory power because social systems evolve unpredictably, while LLM behavior remains opaque compared with explicit utility-based assumptions.These limitations parallel those of utility-based simulations but differ in the transparency of their assumptions.Source: §7.4 · p. 34
• Because current LLMs show more homogeneous opinions than humans, combining them with true human samples is essential to avoid algorithmic monoculture and fragile findings.The resulting findings may cover only the limited perspectives represented (Argyle et al. 2022; Santurkar et al. 2023a; Kleinberg and Raghavan 2021; Bommasani et al. 2022).Source: §7.4 · p. 34

7.5 The Need for A New Evaluation Paradigm

• Open-ended CSS coding requires metrics that capture the semantic validity of free-form explanations rather than accuracy or word overlap.Predictive metrics can conflict with explainability and human evaluations, while crowdwork provided unreliable quality metrics for FLUTE.Source: §7.5 · p. 34
• Human evaluation can also be unreliable for nuanced CSS generation, reinforcing the need for new evaluation paradigms.The paper observes this directly for FLUTE, where crowdwork did not provide dependable quality metrics.Source: §7.5 · p. 34

7.6 CSS Challenges for LLMs

• LLMs struggle with expert taxonomies containing technical terms and specialized definitions that may be absent from pretraining data.Examples include copula omission, persuasive scarcity, and white grievance; understanding novel social-science constructs requires further technical work.Source: §7.6 · p. 34
• CSS label spaces can exceed 72 classes, stressing LLMs through limited memory, finite processing windows, and quadratic space complexity.This challenge is compounded relative to widely used NLP classification tasks.Source: §7.6 · p. 34
• Hierarchical and richly constrained annotation schemes make zero-shot prompting prone to inconsistent machine-readable outputs.Prompting best practices improve consistency, but do not fully solve complex structural parsing across CSS tasks.Source: §7.6 · p. 35
• Changing language, norms, beliefs, and political structures require LLMs to maintain strong temporal grounding across distribution shifts.Rapid analysis of breaking events is especially difficult because continual knowledge updates may be prohibitively expensive as models scale.Source: §7.6 · p. 35

7.7 Issues in Bias and Fairness

• LLMs can amplify social, stereotype, and viewpoint biases, with classification performance varying systematically across demographic groups.These biases may emerge in open-ended explanation and paraphrasing tasks, while available datasets prevented systematic analysis of performance discrepancies.Source: §7.7 · p. 35
• Proprietary LLMs obscure training-corpus selection and bias sources, limiting targeted mitigation and potentially undermining reliable downstream scientific analyses.Authoritative writing styles may evade human review, and black-box APIs may prevent fine-tuning-based mitigation.Source: §7.7 · p. 35
• Closed-source APIs introduce privacy and legal risks because proprietary models may reproduce copyrighted materials and sensitive personally identifiable information.Researchers may become unknowingly accountable for knowledge containing false or missing attributions or private information.Source: §7.7 · p. 35

7.8 Limitations

• The benchmark omits sensitive domains requiring expert or community-specific knowledge, and LLMs should not provide medical or legal advice or interfere with democratic processes.The available data also overrepresent standard-dialect text and WEIRD populations, motivating separate cross-cultural CSS research.Source: §7.8 · p. 36
• Data leakage from pretraining may artificially inflate test performance, especially for proprietary models with undisclosed training sets.Suggested mitigations include prompts that force forgetting the test set and custom perturbed test sets.Source: §7.8 · p. 36
• The study evaluates predictive power rather than causal explanations, although LLM-derived phenomena can serve as structured features for later causal analysis.The authors frame predictions as exposing and elaborating social phenomena latent in text, not as causal theories.Source: §7.8 · p. 36
• Because LLMs lack a grounded causal model of the world, they are not independently reliable for mining causal relationships or contrastive explanations.Future work should investigate explanations of why P instead of Q and other causal forms.Source: §7.8 · p. 36

Related work

• The passages list prior work on computational text analysis, topic modeling, sociolinguistics, discourse, and online social behavior.Examples include computational text analysis for social science, topic modeling, semantic change, politeness, framing, polarization, and trolling.Source: 2008. Unsupervised learning of narrative · pp. 40–41
• The bibliography also includes work on large language models, language-model evaluation, and automated annotation for social science.Cited topics include compute-optimal language models, multitask language understanding, large-language-model annotations, and ChatGPT-based text annotation.Source: 2008. Unsupervised learning of narrative · pp. 41–43
• Prior work includes unsupervised learning for dynamic fictional relationships in narrative texts.The cited study examines “Feuding families and former Friends.”Source: 2008. Unsupervised learning of narrative · p. 43
• The surrounding literature spans computational social science, language change, metaphor, ideology, stance, and event extraction.These topics appear across the cited bibliography entries.Source: 2008. Unsupervised learning of narrative · pp. 43–45

Language-model methods and evaluation

• The bibliography covers language-model capabilities, prompting, transfer learning, evaluation metrics, bias, and human-feedback training.The cited works address zero-shot generalization, unified text-to-text transfer, NLG metrics, benchmark capabilities, societal bias, and instruction following.Source: 2008. Unsupervised learning of narrative · pp. 47–50

Narrative and discourse

• The references include work on computational narrative understanding, narrative flow, literary events, and interpersonal relations.Examples include narrative theory, narrative clause types, narrative flow, literary event detection, and interpersonal relations in narrative summaries.Source: 2008. Unsupervised learning of narrative · pp. 47, 49, 50

Computational social science applications

• The related work also spans computational studies of emotion, empathy, ideology, power, persuasion, political stance, and social interaction.Referenced applications include empathic conversations, affect recognition, political ideology prediction, power detection, persuasive requests, and social-bias analysis.Source: 2008. Unsupervised learning of narrative · pp. 48–50

Related work

• No substantive claim about unsupervised narrative learning is stated in the supplied passages.The passages consist of partial bibliographic citations rather than methodological descriptions or empirical findings.Source: 2008. Unsupervised learning of narrative · p. 51

Scope of supplied evidence

• The passages do not provide evidence about how true and false news spreads.They consist of citation fragments and bibliographic metadata rather than substantive findings.Source: 2018. The spread of true and false news · pp. 51, 53