Source-linked AI summary
AI for social science and social science of AI: A Survey
Ruoxi Xu, Yingfei Sun, Mengjie Ren, Shiguang Guo, Ruotong Pan, Hongyu Lin, Le Sun, Xianpei Han
TL;DR
As LLMs gain human-like capabilities, researchers are exploring their use in social science and treating AI agents as subjects of social inquiry. This survey organizes that work into two directions and reviews their applications and tools. It concludes that LLMs can support research efficiency, while their reliability and ability to represent humans remain unresolved concerns.
Problem
LLMs’ human-like capabilities have drawn social science interest, but their roles as research tools and as objects of social study need to be understood distinctly.
Source: §1 · p. 3Method
Results
The survey concludes that LLMs can serve at multiple research stages and enhance efficiency, but the reliability and authenticity of generated text need verification.
Source: §6 · p. 24Takeaways & Limitations
Distinguishing the two directions helps researchers understand the application scope of LLMs in social science and their study as social entities.
Source: §1 · pp. 3–4, §3 · p. 5Whether and which LLMs can truly represent humans remains an open question, and the approach depends on algorithmic bias shaped by training data.
Source: §3.2.2 · p. 9
Abstract
from arXiv · showhide
Recent advancements in artificial intelligence, particularly with the emergence of large language models (LLMs), have sparked a rethinking of artificial general intelligence possibilities. The increasing human-like capabilities of AI are also attracting attention in social science research, leading to various studies exploring the combination of these two fields. In this survey, we systematically categorize previous explorations in the combination of AI and social science into two directions that share common technical approaches but differ in their research objectives. The first direction is focused on AI for social science, where AI is utilized as a powerful tool to enhance various stages of social science research. While the second direction is the social science of AI, which examines AI agents as social entities with their human-like cognitive and linguistic capabilities. By conducting a thorough review, particularly on the substantial progress facilitated by recent advancements in large language models, this paper introduces a fresh perspective to reassess the relationship between AI and social science, provides a cohesive framework that allows researchers to understand the distinctions and connections between AI for social science and social science of AI, and also summarized state-of-art experiment simulation platforms to facilitate research in these two directions. We believe that as AI technology continues to advance and intelligent agents find increasing applications in our daily lives, the significance of the combination of AI and social science will become even more prominent.
1. Introduction
The introduction frames AI and social science as two connected but distinct lines of inquiry: AI supporting social science research, and social science examining AI agents. This survey reviews both, clarifies their scope, and collects simulation resources for researchers.
1. Introduction
- The survey distinguishes AI used across social science research from social science that studies AI agents as social entities.The two directions share some technical approaches but differ in their research subjects and objectives.Source: Figure 1 · p. 2, §1 · p. 3
- For AI for social science, the survey organizes applications by research stage; for social science of AI, it groups behavioral studies by discipline.The review covers hypothesis generation and verification, then examines AI behavior through psychology, sociology, economics, politics, and linguistics.Source: §1 · p. 4
- The survey proposes a framework for understanding the connections and distinctions between the two directions and the scope of related work.It reviews research developments, limitations, and potential future directions, with emphasis on progress associated with large language models.Source: §1 · p. 4
- The authors collect and compare open-source, large-language-model-based simulation tools intended to facilitate research in both directions.These platforms use LLM agents to implement interventions and simulate social situations, interactions, and behaviors.Source: §1 · p. 4
2. Background
The background describes rapid LLM advances and growing interest in their uses and social implications. It positions the survey as an overview of these developments and the limitations and potential of the field.
2. Background
- LLMs have attracted attention through strong natural-language understanding and text-generation capacities for complex tasks.The passage describes LLMs as transformer language models trained on massive text data, typically with hundreds of billions or more parameters.Source: §2 · p. 4
- LLMs have been explored for social science tasks including literature review, hypothesis development, data analysis, and writing assistance.Research also examines LLM collaboration, emergent social behavior among generative agents, and agents as substitutes for human participants.Source: §2 · p. 4
- LLMs offer possibilities for improving social science research processes and automated science, alongside potential risks and ethical issues.The passage calls for careful consideration of whether and how researchers apply them.Source: Figure 3 · p. 5
- The survey aims to provide an overview of LLM developments at the AI-social-science intersection and address the field’s limitations and potential.Source: §2 · p. 5
3. AI for social science
This section surveys the use of large language models across social science research, including data analysis, creative research work, human simulation, and tasks from hypothesis generation to quantitative verification. It also addresses their exploratory applications and limitations, including reliability, bias, domain knowledge, transparency, complex tasks, and practical drawbacks.
3.1. Hypothesis Generation
- LLM-assisted hypothesis generation shows promise, but remains exploratory and requires researchers to guard against fabricated information, prompt sensitivity, and limited context.Interactive prompts can support query formulation, literature extraction, and hypothesis generation; future directions include domain knowledge, improved prompts, and larger context windows.Source: §3.1 · pp. 6–7
3.2. Hypothesis Verification
- For hypothesis verification, the survey focuses on quantitative methods because LLMs are currently limited in their applicability to qualitative research.Verification uses data collection and analysis to support or refute hypotheses; traditional approaches include experimental, survey, nonreactive, field, and historical-comparative research.Source: §3.2 · p. 7
3.2.1. Experiment Research
- In experiments, LLMs can assist researchers and act as human-behavior proxies, but their believability and transparency remain unresolved.They can help create scenarios and synthesize information, while simulations offer efficiency, scalability, and a way to explore experiments that may be unethical with human subjects; current agent designs commonly use profile, memory, planning, and action modules.Source: §3.2.1 · pp. 7–8
3.2.2. Survey Research
- Survey measurement may shift toward open-ended questionnaires and interactive interviews, while LLMs can help analyze qualitative responses but are not yet primary tools for quantitative analysis.Open-ended psychological questionnaires achieved accuracy that exceeded or rivaled rating-scale reliability; risks in LLM-assisted measurement include bias and data leakage.Source: §3.2.2 · p. 9
- LLMs can emulate survey subgroups, but training-data bias and uncertain representativeness argue against replacing traditional sampling.Synthetic responses may match baseline averages yet diverge on variance, subgroups, and statistical inferences; the survey also identifies LLMs as potential interactive interviewers and text-analysis tools, not primary tools for numerical analysis.Source: §3.2.2 · pp. 8–9
3.2.3. Nonreactive Research
- In nonreactive content analysis, LLMs support text classification and generation, with performance varying across tasks and complexity.ChatGPT reached 96.9% accuracy on cancer misinformation detection and 72% on fact-checking; across evaluated NLP tasks, LLMs exceeded 70% accuracy on misinformation, stance, and emotion classification, but performed poorly on structurally complex or subjective tasks.Source: §3.2.3 · pp. 10–11
- LLMs also generate explanations and future scenarios, but prediction applications remain constrained by scenario complexity and limited data.GPT-3-generated explanations for online hate speech still lagged human explanations in clarity and informativeness, while its Ukraine-conflict predictions relied on prewar data and could not capture complex behavior patterns.Source: §3.2.3 · p. 11
- For existing-statistics analysis, GPT-3 accuracy was below 50% on financial reports and BloombergGPT achieved 43.41% exact match accuracy; ChatGPT has also been explored for stock-return prediction.The cited stock-prediction study prompted ChatGPT to evaluate the effect of prior-day company headlines on prices using sentiment analysis; the supplied passage does not report its results.Source: §3.2.3 · p. 11
- ChatGPT sentiment scores correlated positively with subsequent daily stock returns, but its stock-trend predictions had limited success and underperformed traditional methods using price features alone.Xie, Han, Lai, Peng and Huang (2023a) nevertheless noted potential in combining social-media sentiment with historical stock-price information.Source: §3.2.3 · p. 12
- LLMs can transform nonreactive research through annotation assistance, zero- or few-shot text analysis, and bootstrapping creative generation tasks.They can match fine-tuned models without extensive training, apply across disciplines, and lower technical barriers, but often struggle with conversational or full-document data and expert taxonomies.Source: §3.2.3 · p. 12
3.3. Revisiting Applications Across Disciplines
- Across disciplines, social-science LLM tasks mainly involve text classification, structured parsing, and natural-language generation, while complex aggregation, multi-document summarization, and topic modeling may exceed current capabilities.LLMs can address many tasks with prompt design as the main variation, but face potential bias, high computational cost, and task-specific fine-tuning difficulties.Source: §3.3 · p. 13
3.4. Discussions
- LLM applications in social science span data analysis, creative research assistance, and human simulation, but only the first has been extensively examined; the latter two remain exploratory.Future directions include automated research pipelines, expert models, benchmarks for human-like attributes, tool integration, multimodal models, and ethical frameworks.Source: §3.4 · p. 13
4. Social science of AI
This section examines AI agents as social-science subjects, focusing on their behavioral patterns and social behavior across psychology, sociology, economics, politics, and linguistic questions. It also addresses limitations in existing research and calls for stronger theory, group-level studies, standardized methods, and evaluations tailored to AI agents.
4.1. Psychology of AI
- Language models show personality tendencies, but these are unstable and can be changed through fine-tuning or enhanced memory.Questionnaire-based studies report varied personality profiles, including darker-than-human results for tested models; emotional and value tendencies may reflect pre-training and fine-tuning.Source: §4.1 · pp. 14–16
- Advanced models can match or exceed human performance on some cognitive tasks, yet their reasoning differs from human cognition and remains poorly explained.Findings vary by task: GPT-3 performed comparably on many decision and reasoning tasks but was sensitive to vignette presentation, while humans were more robust on a problem-solving benchmark. The review flags possible training-data leakage and a lack of systematic analysis of model size, objectives, and data.Source: §4.1 · pp. 15–16
4.2. Sociology of AI
- Research finds social biases in language models, while studies of interactions among AI agents remain sparse despite examples of human-like community behavior.Reported biases include gendered portrayals and bias against disabled people; InstructGPT fine-tuning reduced toxic outputs. A sandbox of 25 agents showed spontaneous party planning, but the review says AI sociology has yet to reach its potential.Source: §4.2 · pp. 16–17
4.3. Economics of AI
- Language models show economic expertise in some non-computational areas, but perform notably worse than humans in computational accounting tasks.Reported capabilities span economics, finance, marketing, and auditing, but performance varies by task; current research largely tests GPT-3.5-turbo and may not generalize to unfamiliar, ecologically valid settings.Source: §4.3 · pp. 17–18
- With well-defined prompts, agents can show human-like economic preferences and cooperation, but they struggle to adapt strategies to others’ behavior.Studies report similarities in bargaining and survey responses alongside limits in repeated games, including failure to adjust behavior through conditional reciprocity; the review notes that macroeconomics of AI remains unstudied.Source: §4.3 · p. 18
4.4. Politics of AI
- ChatGPT shows systematic left-leaning political biases in studies, while political leanings can reflect training data and shift with contextual or implicit prompts.The review characterizes research on AI politics as nascent and calls for investigation of how varied training data shape political orientations.Source: §4.4 · p. 19
- Language models may perpetuate or amplify online political perspectives, raising concerns about user influence and political or electoral harms and the need for robust bias measurement.Their capacity to understand and handle political issues also warrants further study.Source: §4.4 · p. 20
4.5. Linguistics of AI
- LLMs show ambiguity in language understanding, GPT-generated languages follow Zipf’s law like humans (Diamond, 2023), and models associate unfamiliar words by form and reinterpret noise-corrupted unreasonable sentences as humans do (Cai et al., 2023).Source: §4.5 · p. 20
- Altered GPT-2 parameters produced GPT-D language traits associated with Alzheimer’s disease (Li, Knopman, Xu, Cohen and Pakhomov, 2022), while future-tense references influence LLMs’ human-like time and reward preferences.The review suggests integrating linguistic and NLP perspectives to better explain model mechanisms, including vocabulary probabilities and tokenization’s role in understanding affixes.Source: §4.5 · p. 20
4.6. Discussions and Future Works
- Research on LLM social behavior largely examines individual agents, with little study of agent groups or systems, partly because such systems are not yet present.Source: §4.6 · p. 20
- Results can vary with model version, settings, and prompt order; psychological experiments also face training-text exposure and design effects, while human-derived ToM tests may misjudge LLM abilities.The review notes reproducibility concerns for changing ChatGPT versions (Tu, Li, Yu, Wang, Hou and Li, 2023), temperature-linked personality shifts, prompt-order sensitivity (Bartolo, Moore, Riedel and Stenetorp, 2022; Zhao, Wallace, Feng, Klein and Singh, 2021), and differing GPT-3 mental-ability results. Ullman (2023) found failures on false-belief tasks with disturbances.Source: §4.6 · pp. 20–21
- Future work should develop systematic theory, examine group dynamics, standardize experiments, tailor AI-agent evaluations, and combine social-science and AI theories.The review likens human and LLM cognition to black boxes, probed through external performance and behavior measures, and suggests that NLP methods and neuroscience may offer complementary ways to investigate them.Source: §4.6 · p. 21
5. Public Tools and Resources
The section reviews open-source LLM simulation tools and compares their functions to help researchers select platforms for different needs.
5.1. Public Simulation Tools
- SkyAGI emphasizes realistic NPC responses, while AgentVerse supports customizable multi-agent environments and plugins.SkyAGI is framed as a role-playing game experience; AgentVerse lets researchers define settings and prompts for environments such as chat rooms.Source: §5.1 · p. 21
- Table 4 compares open-source simulation toolkits; its single-agent columns are omitted because the tools share population modeling, prompt-based memory, and predefined models.The caption says all compared toolkits share these three single-agent features.Source: Table 4 · p. 22
- LangChain supplies agent-building components but requires custom interaction code; GenerativeAgents offers modifiable simulations, and Agents configures state transitions through SOP.LangChain includes a time-weighted vector-store retriever for agent memory; GenerativeAgents also provides a map for visualization and interaction.Source: §5.1 · p. 22
- AgentLab creates heterogeneous agents from profiles, supports learning through model weights or inputs, and automates their interactions after conditions are set.Researchers can customize the agents’ social backgrounds.Source: §5.1 · p. 22
5.2. Core Functions of Simulation Tools
- The proposed functional hierarchy groups simulation requirements into single-agent, multi-agent, and environment levels, including interaction rules and physical and social context.The framework discusses serial and parallel interactions, effects of the physical environment, and backgrounds such as norms and institutions.Source: §5.2 · pp. 22–23
- Current open-source toolkits offer simplistic agent learning, restrict model choice, and inadequately support agent-environment interaction.The authors identify these as limitations to address for more comprehensive agent simulation.Source: §5.2 · p. 23
5.3. Discussions and Future Works
- LLM agents remain short of human behavioral diversity and struggle with complex interactions, while modeling varied social, physical, and temporal contexts challenges researchers.The authors associate some limits with missing multimodal information and a possible lack of theory of mind.Source: §5.3 · pp. 23–24
- Future platform work may incorporate cognitive theories, use multimodal LLM capabilities, and establish qualitative evaluation frameworks.The proposed directions target agent interpretability or correctness, information acquisition and expression, and platform assessment.Source: §5.3 · p. 24
6. Conclusion
The conclusion frames AI for social science and social science of AI as complementary directions in LLM research. It also identifies human–AI interaction as a further research direction and says AI will be integrated into research without replacing sociologists.
6. Conclusion
- The two directions are complementary: social science of AI can guide AI development, while AI for social science can make research on AI more efficient.The authors maintain that researchers must verify LLM-generated text and consider validation costs and bias risks when using models as research tools.Source: §6 · p. 24
- Further work should study how AI influences human behavior in computer–human interaction and develop methods for investigating societal issues arising from those interactions.The conclusion distinguishes this direction from research focused on human populations or AI-agent populations.Source: §6 · p. 24