Source-linked AI summary
Can Large Language Models Transform Computational Social Science?
Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, Diyi Yang
TL;DR
CSS researchers need computational methods that can analyze textual social phenomena without extensive labeled training data, motivating tests of zero-shot LLM reliability. The paper develops prompting guidance and evaluates 13 models on 25 English CSS benchmarks. LLMs generally complement rather than replace human annotation: classification trails fine-tuned systems but achieves fair human agreement, while generated explanations can exceed crowdworker references.
Problem
The paper asks whether zero-shot LLMs can reliably classify and explain social phenomena for CSS without extensive human-annotated training data.
Method
The study provides prompting best practices and evaluates open- and closed-source LLMs across a representative suite of CSS classification and generation tasks.
Results
LLMs generally do not outperform carefully fine-tuned classifiers but achieve fair human agreement, while some generated explanations exceed dataset references across several quality dimensions.
Takeaways & Limitations
LLMs can augment CSS through human-AI annotation teams and joint coding of new constructs, analyses, and summaries.
Takeaways & Limitations
The task suite is not exhaustive, relies heavily on available standard-dialect data, and excludes sensitive domains requiring expert or community-specific knowledge.
Abstract
from arXiv · showhide
Large Language Models (LLMs) are capable of successfully performing many language processing tasks zero-shot (without training data). If zero-shot LLMs can also reliably classify and explain social phenomena like persuasiveness and political ideology, then LLMs could augment the Computational Social Science (CSS) pipeline in important ways. This work provides a road map for using LLMs as CSS tools. Towards this end, we contribute a set of prompting best practices and an extensive evaluation pipeline to measure the zero-shot performance of 13 language models on 25 representative English CSS benchmarks. On taxonomic labeling tasks (classification), LLMs fail to outperform the best fine-tuned models but still achieve fair levels of agreement with humans. On free-form coding tasks (generation), LLMs produce explanations that often exceed the quality of crowdworkers' gold references. We conclude that the performance of today's LLMs can augment the CSS research pipeline in two ways: (1) serving as zero-shot data annotators on human annotation teams, and (2) bootstrapping challenging creative generation tasks (e.g., explaining the underlying attributes of a text). In summary, LLMs are posed to meaningfully participate in social science analysis in partnership with humans.
1. Introduction
This work asks whether zero-shot LLMs can reliably support CSS classification and generation, reducing reliance on extensive human-annotated training data. It evaluates diverse CSS tasks and finds that LLMs complement rather than replace human annotation.
- The study evaluates open- and closed-source LLMs across 24 representative CSS tasks, segmented by discourse type.
- Zero-shot viability is the central question: whether LLMs can provide reliable labels and summary codes for CSS without supervision.
- LLMs generally do not match carefully fine-tuned classifiers and are often insufficient to replace human annotation, though they achieve fair human agreement.
- These classification results span conversation-, utterance-, and document-level tasks across multiple social-science fields.
- Some LLMs generate explanations that match or exceed dataset references in relevance, coherence, faithfulness, and fluency.
- Human-AI collaboration may extend from labeling to joint coding of new constructs, analyses, and summaries.
2. An Overview of CSS
CSS develops computational methods for analyzing behavioral and linguistic data across social-science and humanistic domains. Its applications range from event extraction and literary analysis to sociolinguistic, political, psychological, and sociological inquiry.
- CSS is defined as developing and applying computational methods to scientifically analyze behavioral and linguistic data.
- The surveyed applications span psychology, linguistics, political science, economics, sociology, literature, anthropology, and history.
- History: Event extraction converts unstructured historical text into structured data capturing event location, time, cause, and participants.
- Literature: Literary analysis uses computational tools to study themes, settings, and narratives.
- Linguistics: Sociolinguistic analysis measures stylistic and structural language features associated with identity, group membership, region, class, age, and gender.
- Computational approaches support research on political agendas and ideologies, psychological traits, and social structures and processes.
3. Representative CSS Task Selection
The benchmark is designed as a representative, functionally organized survey of CSS needs, covering classification at multiple text levels and generation tasks. Its tasks address linguistic, psychological, cultural, political, conversational, and historical phenomena.
- The task selection is representative rather than exhaustive and is intended to answer four research questions about LLM viability, model selection, domain utility, and functionality.
- The benchmark contains 20 classification tasks and 5 generation tasks organized by utterance-, conversation-, and document-level analysis.
- Compared with BIG-Bench, this benchmark covers additional CSS domains including dialect, discourse, events, ideology, misinformation, persuasion, power, semantic change, and stance.
- Utterance-Level Classification: Utterance-level tasks cover dialect, hate speech, figurative language, emotion, humor, misinformation, ideology, persuasion, semantic change, and stance.
- Utterance-Level Classification: The benchmark includes Indian English detection over 22 grammatical features and weakly labeled emotion detection using Plutchik’s 8 categories.
- Utterance-Level Classification: Fine-grained hate-speech classification distinguishes six categories, including incitement, inferiority language, irony, stereotypes and misinformation, threats, and white grievance.
- Conversation-Level Classification: Conversation-level evaluation includes empathy exploration, power relations, and other interactional phenomena.
- Document-Level Classification: Document-level evaluation includes binary sentence-level event detection, treating events as units for organizing historical documents.
4. Evaluation Methods
The evaluation combines prompt engineering, supervised baselines, stratified benchmark sampling, automatic metrics, and human assessments to measure LLM performance on CSS tasks. Prompt perturbations and constrained decoding target reproducibility and valid outputs, while generation quality is assessed through expert scoring and ranking because automatic overlap metrics can be inadequate.
- Evaluation design: The study evaluates zero-shot LLMs through prompt engineering and compares model architectures, sizes, and pre-training or fine-tuning paradigms for CSS research.FLAN-T5 is selected as an open-source model with strong zero-shot and few-shot performance.
- Evaluation design: Supervised RoBERTa-large and T5-base models provide comparison baselines for classification and generation tasks, respectively.The baselines are intended as utility comparison points rather than fair methodological comparisons.
- Prompt engineering: Each task receives an author-written prompt plus four semantically equivalent gpt-3.5-turbo perturbations, with results averaged to reduce instruction-based variance.Prompts use natural-language instructions and multiple-choice formatting where applicable.
- Prompt engineering: The prompting guidelines address overgeneralization, uncertainty, offensive language, and consistent machine-readable outputs in CSS codebooks.The guidelines are reasonable approximations to prompts a non-AI expert could design, not claims of task-optimal engineering.
- Automatic evaluation: Evaluation samples at most 500 class-stratified test instances per task, releases datasets and outputs, and uses macro F1 averaged across five prompt perturbations for classification and structured parsing.String matching is straightforward because labels are mapped to alphabetical candidate options and decoding is biased toward valid options.
- Human evaluation: Generation quality is assessed with expert Likert scoring and faithfulness-based human ranking because BLEU, BERTScore, and BLEURT may poorly reflect semantic quality.Experts rate faithfulness, coherence, relevance, and fluency on 1-5 scales, while pilot variance led the authors to serve as blind annotators for one evaluation.
5. Classification Results
Zero-shot LLMs generally do not surpass carefully fine-tuned classifiers, but they achieve useful human agreement on many classification tasks. Performance varies by task and model, with scaling and training choices affecting results while some subjective or complex tasks remain poor.
- Overall viability: 47% of classification tasks achieve moderate-to-good agreement with human gold labels, spanning κ = 0.40 to 0.65.These tasks include misinformation, stance, emotion, figurative language, and utterance-level ideology classification.
- Overall viability: Zero-shot results rarely exceed carefully tuned supervised RoBERTa baselines, although some match classifiers used in published studies.The authors describe zero-shot models as a potentially data-efficient alternative, with human validation and prompt tuning still recommended.
- Task limitations: 29.5% of tasks have poor agreement, with performance near random guessing on Semantic Change, Empathy, conversation-level Persuasion, and Toxicity.The authors attribute low performance to structural complexity or subjective, non-standard taxonomies whose meanings differ from pretraining.
- Error analysis: OpenAI models were oversensitive to the stereotypical implicit-hate class, with 71% and 65% false-positive rates for davinci-003 and ChatGPT.86% of false positives contained identity terms, suggesting models often treated identity mentions as sufficient evidence of stereotyping.
- Model selection: Each order-of-magnitude increase in FLAN parameter count yielded a 5.0-point median average-task improvement, whereas OpenAI gains depended more on training changes than scale alone.davinci-001 to davinci-002 produced an 11-point absolute classification improvement, exceeding gains from scale alone.
6. Generation Results
Prompted LLMs generate informative explanations for social-science constructs, with leading models matching or exceeding human references on several quality dimensions. Human rankings show helpful outputs can approach parity with gold explanations, while remaining limitations motivate human-AI collaboration and improved evaluation.
- Human Scoring Evaluation: Leading models match or exceed human gold references and fine-tuned baselines on faithfulness, relevance, coherence, and fluency.Expert scoring uses average ratings on 1-5 Likert scales.
- Human Scoring Evaluation: Scale matters: smaller models often fail to produce faithful explanations, whereas larger LLMs generate useful, relevant, coherent, and fluent outputs.text-ada-001 receives average faithfulness scores below 2 out of 5 in several tasks.
- Human Ranking Evaluation: Model generations outrank dataset gold references in at least 38% of pairwise rankings, with the best models approaching parity with human preferences.The authors report a near 50-50 preference split for the best models.
- Human-AI Collaboration: Despite strong generation quality, no model substantially outperforms human annotation, supporting collaboration rather than full replacement.Researchers could apply minor edits to model-generated explanations instead of coding them from scratch.
- Evaluation Challenges: Automatic reference-based metrics can be uninterpretable for high-variation generation, making manual validation necessary but costly and difficult to reproduce.The paper motivates reference-free expert evaluation while identifying cost and reproducibility concerns.
7. Discussion
The paper concludes that LLMs can augment, but not replace, the CSS research pipeline. It recommends integrating LLMs into annotation and using larger instruction-tuned models for high-quality generation.
- The evaluation supports augmenting, rather than entirely replacing, the traditional CSS research pipeline.
- CSS researchers should integrate LLMs-in-the-loop to transform large-scale data labeling.
- Larger instruction-tuned models are recommended for generations prioritizing faithfulness, relevance, coherence, and fluency.
- Figure 5 illustrates misinformation annotation in which an LLM scales annotation and reduces variance while humans validate model-provided labels.
7.1 How Annotation Fits Into CSS
CSS labels and generative codes support downstream statistical inference about broader social phenomena. Their validity therefore depends on estimators that are asymptotically unbiased and support valid confidence intervals.
- Social scientists use classification labels and generative codes to explain wider phenomena through downstream inferential statistics.
- Valid downstream inference requires estimators of the underlying construct to be asymptotically unbiased and support valid confidence intervals.
7.2 LLMs Can Augment Annotation
LLMs can reduce human annotation labor when used as imperfect collaborators rather than replacements. Human validation remains necessary, while generation outputs can support new coding and dataset-construction workflows.
- 12 out of 17 classification tasks achieved fair to strong agreement between an LLM annotator and humans.
- LLMs should be integrated with human annotation because unchecked hallucinations and inaccuracies can distort CSS conclusions and policy actions.
- LLMs can provide imperfect, low-cost labels that human validation refines through weak supervision, targeted cleaning, and active learning.
- In all 5 evaluated generation tasks, LLMs were rated superior to human gold annotations more than 38% of the time.
- Human expertise can curate generated outputs for domain-specific accuracy and quality, extending collaboration beyond labeling.
7.3 When To Use What LLM
Model choice depends on the task and available infrastructure: open-source models are recommended for classification when hardware is available, while proprietary models lead generation tasks. LLMs remain limited on conversational, full-document, and cross-document reasoning tasks.
- For CSS classification, open-source FLAN models are reported as capable as state-of-the-art closed-source OpenAI LLMs.
- Running FLAN-T5-XXL costs 170 dollars per day, equivalent to processing approximately 50 million words with gpt-3.5-turbo.
- For generation, the best proprietary models consistently outperform fine-tuned baselines and approach parity with crowdworker-evaluated human gold annotations.
- For generation, gpt-3.5-turbo is identified as the leading LLM by both price and performance for interpretations and explanations.
- All evaluated LLMs struggle most with conversational and full-document data, while limited cross-document reasoning constrains applications such as topic modeling.
7.4 Blending CSS Paradigms
LLMs blur the traditional boundary between supervised and unsupervised CSS methods by supporting hypothesis development, large-scale analysis, and simulation. However, simulated samples have limited predictive and explanatory power, requiring combination with true human samples.
- LLMs can blend supervised and unsupervised CSS paradigms by supporting hypothesis development and testing with one multi-purpose tool.They can retrieve, label, and condense relevant information at scale.
- Simulation: LLMs can serve as simulated sample populations for social computing systems, communities, public opinion, and subjective experience descriptions.These simulations extend beyond traditional utility-based models.
- Simulation: Simulated samples have limited predictive and explanatory power because social systems evolve unpredictably.Unlike utility-based simulations, LLM behavior is also based on an opaque model of human behavior.
- Simulation: Current LLMs exhibit more homogeneous opinions than humans, risking an algorithmic monoculture and fragile findings.Combining LLMs with true human samples is essential to avoid relying on limited represented perspectives.
7.5 The Need for A New Evaluation Paradigm
Blended CSS methods require evaluation approaches suited to open-ended coding and explanation, which do not fit fixed taxonomies or regular output templates.
- Open-ended coding and CSS explanations require new evaluation methods because they follow neither predefined taxonomies nor regular output templates.Accuracy metrics suit fixed-taxonomy classification, while word-overlap metrics suit generation with well-defined references.
7.6 CSS Challenges for LLMs
LLMs face recurring CSS challenges from specialized taxonomies, large and structured label spaces, and changing linguistic and social distributions. These demands strain zero-shot prompting and temporal grounding.
- Specialized expert taxonomies contain technical terms and nonstandard definitions that may exceed LLM representations.Examples include copula omission, persuasive scarcity, and white grievance.
- CSS label spaces may contain upwards of 72 classes, challenging LLMs with limited memory, finite processing windows, and quadratic space complexity.Large target label spaces compound the difficulty of classification.
- Hierarchical and richly constrained annotation schemes create dependencies among labels that zero-shot prompting may render inconsistently.Complex structural parsing tasks are especially difficult for prompted models.
- Changing language, norms, beliefs, and political structures require extremely high temporal grounding, especially for breaking-event analysis.Rapid synchronous analysis may be prohibitively expensive.
7.7 Issues in Bias and Fairness
Applying LLMs in CSS raises risks from amplified social and viewpoint biases, opaque proprietary training data, privacy concerns, and intellectual-property disputes.
- LLMs can amplify social stereotypes and viewpoint biases, including in open-ended explanation and paraphrasing tasks.These risks are especially relevant in subjective domains.
- Proprietary LLMs obscure training-corpus selection and formatting, limiting targeted mitigation of bias and social harm.Their use also introduces distinct privacy implications for research.
- Closed-source APIs may expose researchers to privacy issues, legal disputes, copyrighted material, and replicated personally identifiable information.Researchers may also become accountable for false or missing attributions.
7.8 Limitations
The study’s limitations include restricted task coverage, potential pretraining data leakage, and limited support for causal explanations. Its data resources also overrepresent standard-dialect and WEIRD populations.
- Task selection: The task selection excludes sensitive domains such as mental health and cultural studies, where expert or community-specific knowledge and ethical concerns add challenges.The study also cautions against using LLMs for legal or medical advice, diagnosis, or interference with democratic processes.
- Task selection: Available data are dominated by standard-dialect text from Western, Educated, Industrial, Rich, and Democratic populations.The authors identify cross-cultural CSS as a priority for future study.
- Data leakage: Test-set exposure during pretraining may artificially inflate LLM performance, especially for closed-source models.This is identified as a data-leakage concern in evaluation.
- Causality and explanations: The analysis evaluates predictive power rather than causal explanations, although predicted phenomena may serve as structured features for later causal analysis.The authors distinguish exposing textual phenomena from explaining their causes.
- Causality and explanations: Because LLMs lack a causal model of the world, they are not independently reliable tools for mining causal relationships in text.Contrastive and causal explanations are left for future work.
2008. Unsupervised learning of narrative
The supplied passages are bibliographic entries covering computational methods for topics, conversations, humor, cascades, trolling, and related social analysis.
- Topic modeling: Topic-modeling references address human interpretation, evaluation methods, sociological applications, and reading large text collections.The entries include work on interpreting topic models, evaluating them, and applying them to newspaper coverage.
- Related applications: Other entries cover humor recognition, recurring cascades, trolling behavior, empathy-oriented systems, and data-efficient annotation.The references include deep-learning humor detection, cascade recurrence, trolling causes, empathy design, and limited-budget data collection.
- Conversation analysis: Conversation-analysis references include toolkits, coarse discourse sequences, conversational failure, social roles, and power differences in interaction.These works span computational tool support and modeling of interactional phenomena.
2018. The spread of true and false news
The supplied passages are bibliographic entries spanning event extraction, persuasion, social roles, humor, language models, and computational text analysis.
- Language technologies: Several references concern language understanding and generation, including event extraction, semantic parsing, text generation evaluation, and language-model calibration.The entries include work on event extraction, conversational semantic parsing, BERTScore, and improving few-shot performance.
- Social applications: Social-science applications include persuasive strategies, social roles in health communities, teamwork roles, and causal-versus-predictive explanation.These references apply computational methods to persuasion, online health roles, teamwork, and explanatory modeling.
- Language and society: The references also address humor detection, online discussion, bias, idiolects, lexical change, and AI ethics.The listed studies examine humor, discussion structure, gender bias, individual language styles, vocabulary change, and ethical issues in ChatGPT.