PaperlayerView exact revision on arXiv

Source-linked AI summary

S$^3$: Social-network Simulation System with Large Language Model-Empowered Agents

Chen Gao, Xiaochong Lan, Zhihong Lu, Jinzhu Mao, Jinghua Piao, Huandong Wang, Depeng Jin, Yong Li

arXiv:2307.14984v3cs.SI

TL;DR

Social-network simulation needs realistic models of individual behavior and population dynamics, but conventional approaches offer limited human-like behavioral modeling. S3 uses LLM-empowered agents with prompt engineering and tuning to simulate demographics, emotions, attitudes, and interactions in real-data environments, achieving considerable accuracy across multiple metrics. The system is an initial effort toward LLM-based social-science simulation, with applications examined alongside stated limitations and future improvements.

  • Problem

    Social-network simulation must represent complex user content, emotions, attitudes, and interactions while supporting individual- and population-level analysis.

  • Method

    S3 constructs a real-world social-network environment and uses LLM agents, prompt engineering, and prompt tuning to simulate demographics, emotions, attitudes, content generation, and interactions.

  • Results

    The system achieves considerable accuracy across multiple metrics in systematic individual- and population-level evaluations using real social-network data.

  • Takeaways & Limitations

    S3 provides an initial LLM-empowered paradigm for social-network simulation and supports scientific investigation and real-world applications within the stated scope.

  • Takeaways & Limitations

    The S3 system is an initial endeavor for applying LLM capabilities to social-science simulation, with limitations and future improvements requiring further analysis.

Abstract

from arXiv · show

Social network simulation plays a crucial role in addressing various challenges within social science. It offers extensive applications such as state prediction, phenomena explanation, and policy-making support, among others. In this work, we harness the formidable human-like capabilities exhibited by large language models (LLMs) in sensing, reasoning, and behaving, and utilize these qualities to construct the S$^3$ system (short for $\textbf{S}$ocial network $\textbf{S}$imulation $\textbf{S}$ystem). Adhering to the widely employed agent-based simulation paradigm, we employ prompt engineering and prompt tuning techniques to ensure that the agent's behavior closely emulates that of a genuine human within the social network. Specifically, we simulate three pivotal aspects: emotion, attitude, and interaction behaviors. By endowing the agent in the system with the ability to perceive the informational environment and emulate human actions, we observe the emergence of population-level phenomena, including the propagation of information, attitudes, and emotions. We conduct an evaluation encompassing two levels of simulation, employing real-world social network data. Encouragingly, the results demonstrate promising accuracy. This work represents an initial step in the realm of social network simulation empowered by LLM-based agents. We anticipate that our endeavors will serve as a source of inspiration for the development of simulation systems within, but not limited to, social science.

1 Introduction

Social simulation provides a way to study evolving individual and population states, while online social networks make simulation increasingly important. S3 uses LLM-empowered agents to model demographics, emotions, attitudes, and interactions, with evaluations targeting individual and population behavior.

  • Social simulation models individual interactions and population states to support prediction and intervention-oriented experiments.It includes both microlevel and macrolevel simulation approaches.
  • The growth of online social networks has made their interactive activities a central focus for social science simulation.
  • S3 constructs a real-data social-network environment with LLM agents that infer demographics and simulate attitudes, emotions, and behaviors.Agents observe followed users’ content and may forward content, create content, or remain inactive.
  • The system evaluates gender-discrimination and nuclear-energy scenarios using individual- and population-level accuracy metrics.The scenarios examine information, sentiment, policy attitudes, and interactions between opposing groups.
  • S3 is presented as an agent-based simulation paradigm that learns from real social-network data and achieves considerable accuracy across multiple metrics.

2 Related Works

Related work traces social simulation from equation-based approaches toward agent-based models and increasingly capable machine-learning agents. LLM-based studies suggest promise for reproducing human-like behavior, but prior examples lacked real-data evaluation or focused on small virtual environments.

  • 2.1 Social Simulation: Social simulation studies social activities through dynamic individual interactions and changing population states.
  • 2.1 Social Simulation: Earlier simulation methods used discrete events or system dynamics, while agent-based simulation explicitly models individuals and their state updates.
  • 2.1 Social Simulation: Machine-learning agents extend agent-based simulation by perceiving surroundings dynamically and producing behavior resembling human actions.
  • 2.2 Large Language Model-based Simulation: LLMs align with agent-based simulation because prompts can endow agents with identities, information, preferences, and behavioral capacity.
  • 2.2 Large Language Model-based Simulation: Prior LLM-agent studies reproduced selected experiments or created virtual environments, but one cited virtual-town simulation lacked real-data evaluation.

3.1 System Overview

S3 augments a social-network framework with LLM agents to pursue quantitative accuracy at both individual and population levels. Its simulations cover behaviors, attitudes, emotions, and population-level propagation phenomena.

  • S3 uses LLM-augmented agents within a social-network framework to target quantitative accuracy at individual and population levels.
  • Individual-level simulation aims to reproduce users’ behaviors, attitudes, and emotions from user characteristics and information.
  • The system overview is accompanied by datasets used for social-network simulation and individual prediction-task performance.

3.2 Social Network Environment

The study focuses on controversial gender-discrimination and nuclear-energy topics selected for their extensive social-media data. Because direct user information is limited, demographic attributes are inferred from textual data.

  • The simulations focus on gender discrimination and nuclear energy because their controversial nature provides extensive data.
  • The nuclear-energy scenario examines public attitudes toward nuclear power versus fossil fuels.
  • The gender-discrimination scenario examines emotional experiences at individual and population levels.
  • S3 infers missing user demographics from posts and personal descriptions to construct more extensive user personas.

3.3 Individual-level Simulation

S3 simulates individual users’ emotions, attitudes, content generation, and interaction decisions within a real-world social-network environment. LLM-based modeling produces promising accuracy across these behavioral facets and supports later population-level simulation.

  • Individual-level Simulation: Individual simulation lets users perceive information, update emotions and attitudes, then forward posts, create content, or remain inactive.The system models emotion, attitude, and interaction behavior as sequential responses to the user’s informational environment.
  • Emotion Simulation: Users’ emotions evolve across calm, moderate, and intense levels through a Markov process using profiles, histories, and current emotional state.The process predicts the emotion level at the subsequent time step after users become aware of an event.
  • Emotion Simulation: 71.8% accuracy was achieved for three-class emotion prediction on real-world data.The evaluation concerns predicting users’ emotions at the next time step.
  • Attitude Simulation: Attitudes begin from user profiles and histories, then change through a binary negative-positive Markov process driven by unfolding events.The model uses the LLM to determine both initial attitudes and attitude changes.
  • Content-generation and Interactive Behavior Simulation: LLMs generate user content from profiles and current attitudes or emotions, while interaction agents decide from sensed information environments learned from observed data.These mechanisms model both content-generation behavior and decisions to forward, post, or do nothing.
  • Content-generation Behavior Simulation: Generated content achieved Perplexity 19.289 and average cosine similarity 0.723 for Gender Discrimination, versus 16.145 and 0.741 for Nuclear Energy.The metrics compare generated text with actual user-generated text.
  • Interactive Behavior Simulation: Interaction decisions reached 66.2% Accuracy, 0.662 AUC, and 0.667 F1-Score in Gender Discrimination, and 69.5% Accuracy, 0.681 AUC, and 0.758 F1-Score in Nuclear Energy.The results indicate commendable efficacy across both scenarios.

3.4 Population-level Simulation

S3 simulates information, emotion, and attitude propagation in real-world social-network events using LLM-based agents. Across two events, it reproduces observed information-spread, emotional-peak, and attitude-change patterns.

  • Propagation scope: S3 models information, emotion, and attitude propagation as population-level phenomena in social networks.Information concerns event-news transmission, emotion concerns socially contagious feelings, and attitude concerns exchanged viewpoints.
  • Information Propagation: The simulator accurately forecasts the propagation patterns of the Eight-child Mother and Japan Nuclear Wastewater Release events.It also captures the gradually marginal rise rate over time.
  • Emotion Propagation: S3 extracts emotions from real-world interactions and simulates emotional propagation among LLM-based agents.The evaluation uses emotional density extracted from textual interactions among agents.
  • Emotion Propagation: Two emotional peaks in the Eight-child Mother event are reproduced from real-world initialization.The authors associate a secondary peak with slower news spread across a larger community.
  • Attitude Propagation: During the Japan Nuclear Wastewater Release event, S3 reproduces the sharp decline and gradual recovery of positive attitudes toward nuclear energy.The pattern occurs through repeated interactions among agents.

3.5 Comparative Evaluation

S3 is evaluated against established simulation baselines across information, opinion, and emotion propagation tasks. It performs competitively on information and opinion propagation and outperforms all listed baselines on emotion propagation in a zero-shot setting.

  • Evaluation setup: The evaluation compares S3 with baselines across information, opinion, and emotion propagation tasks.Information propagation uses Linear Threshold and Independent Cascade models, while opinion and emotion propagation use five additional baselines.
  • Information Propagation Comparison: S3 is compared with the Linear Threshold and Independent Cascade models using MSED and correlation coefficient metrics.The MSED metric is defined over actual and predicted state values at each time t.
  • Information Propagation Comparison: S3 achieves the lowest MSED and highest correlation on the Gender Discrimination dataset for information propagation.Overall, the method is reported as competitive with traditional models.
  • Opinion and Emotion Propagation Comparison: S3 operates zero-shot, whereas the opinion and emotion baselines use 50% training, 10% validation, and 40% evaluation data.The comparison includes Voter, DeGroot, FNN, SINN, and NDCN.
  • Opinion and Emotion Propagation Comparison: S3 performs competitively with training-dependent baselines for opinion propagation and outperforms all baselines for emotion propagation.This result is reported in the comparative evaluation summarized by Table 5.

4 Architecture and Methodology

S3 constructs a real-world social-network environment and models users through demographic, emotional, attitudinal, and interaction-related attributes. LLM-based modules support demographic inference and simulation of message propagation across a focused network subgraph.

  • Environment Construction: The message-propagation framework models users and their interactions within a directed social-network environment.The network contains mutual and one-way following relationships, represented through node indegree and outdegree.
  • User Characterization: The system represents users with relatively stable demographics and more dynamic attitudes and emotional states.Demographics include gender, occupation, and age, while attitudes and emotions can change over shorter periods.
  • Message Propagation: Message selection considers temporal influence and content relevance to the user’s characteristics.Older messages receive lower time scores, while relevance reflects differences such as age-related interest in event types.
  • Environment Construction: The initialized network uses keyword-relevant posts, their authors, connected users, and directed follow relationships within a selected subgraph.Message dissemination during simulation occurs only between extracted source nodes and corresponding target nodes.
  • User Demographics: Demographic prediction combines tuned LLMs for gender and age with pretrained-LLM inference for occupation.Gender uses ChatGLM with P-Tuning-v2; age uses prefix tuning with blog data; occupation uses posts and profile descriptions.
  • User Demographics: The age predictor obtains MSE 128, MAE around 7.53, and a 21.5% unified percentage error, while occupation inference identifies 1,016 occupations.The gender predictor also generates valid predictions despite missing explicit gender information in some personal descriptions.

4.3 Emotion and Attitude Simulation

S3 uses LLMs to update users’ emotional and attitudinal states and to generate interaction behavior and content. These states and user profiles guide decisions about reposting, new posts, and responses to received messages.

  • Emotion Simulation: Emotion simulation uses a Markov-chain process driven by user demographics, current emotion, and the received post.Emotions are represented as calm, moderate, or intense states.
  • Emotion Simulation: A decaying coefficient models the gradual reduction of emotional states over time.The coefficient controls the emotion-decay rate, while prompts guide the LLM’s decision about state changes after messages arrive.
  • Content Generation: Content generation is conditioned on user profile information and the current emotional or attitudinal state toward an event.The generated content represents the user’s internal state together with external event-related influences.
  • Content Generation: The LLM determines how event-related content is shaped from the supplied profile and state information, reducing manual intervention.The design aims to emulate user-generated content through prompted behavioral decisions.
  • Content Generation: The approach aligns generated posts with users’ emotional or attitudinal dynamics and is reported to emulate content creation with high fidelity.This connects internal user states to the observable text produced in the social network.
  • Interaction Behavior: Interaction behavior determines whether a user reposts received content or creates a new post about the same event.These actions propagate messages to the user’s followers, supporting information spread through the network.

5 Discussions and Open Problems

The discussion frames S3 as an initial LLM-enabled social-science simulation system with applications in prediction, explanation, pattern discovery, and policy making. It also identifies limitations and directions for broader, more scalable simulation.

  • Scope: S3 is presented as an initial effort to use LLM capabilities for social-science simulation, with further application and limitation analysis planned.The paper characterizes the system as an early endeavor rather than a completed solution.
  • Applications: Agent-based simulation can support prediction, reasoning, explanation, pattern discovery, theory construction, and policy-making applications.The discussion describes these as potential uses of configurable LLM-based agents and social environments.
  • Improvement on Individual-level Simulation: Individual-level simulation remains limited by incomplete behavioral knowledge and insufficient modeling of historical, social, and personal context.The proposed improvement is to strengthen agents’ perception and interpretation of contextual cues.
  • Improvement on Population-level Simulation: Combining agent-based simulation with system dynamics could represent both individual interactions and population-level systemic behavior.The stated goal is a more comprehensive representation of population dynamics, including effects of individual decisions on the overall system.
  • Improvement on Population-level Simulation: Broader validation should incorporate additional societal, economic, and cultural phenomena, including opinion dynamics, cultural diffusion, inequality, and disease spread.The discussion presents this expansion as a way to validate effectiveness and gain further insights.
  • Future Improvements: Future system improvements include additional information channels, more efficient large-scale computation, and interfaces for testing policy interventions.Examples include recommender-system inputs, parallel or distributed computing, and controlled policy experimentation.

6 Conclusion

The conclusion presents S3 as an LLM-enabled framework for social-network emulation focused on emotion, attitude, and interactive behavior. It positions the work as an initial contribution with potential relevance beyond social science.

  • Conclusion: S3 uses LLM-based agents to emulate social networks through simulated emotion, attitude, and interactive behaviors.The framework leverages LLM capabilities in perception, cognition, and behavior.
  • Conclusion: The paper identifies S3 as a pioneering integration of LLM-empowered agents for social-network simulation.It suggests that the methodology may stimulate simulation-system development across diverse domains.
  • Conclusion: The authors state that the methodology can provide researchers and policymakers with insights into complex social dynamics for informed decision-making.This stated scope extends the system’s intended use from social-network emulation to addressing societal challenges.
Loading 2307.14984v3…