Source-linked AI summary
Simulating Opinion Dynamics with Networks of LLM-based Agents
Yun-Shiuan Chuang, Agam Goyal, Nikunj Harlalka, Siddharth Suresh, Robert Hawkins, Sijia Yang, Dhavan Shah, Junjie Hu, Timothy T. Rogers
TL;DR
The paper asks whether LLM agents can provide a more realistic way to simulate opinion dynamics than simplified ABMs. It models interacting persona-based agents communicating in natural language and finds that they generally converge toward factual consensus, while induced confirmation bias produces opinion fragmentation. These findings show both the promise and the limitations of LLM agents for modeling human belief dynamics.
Problem
Opinion-dynamics research needs models that capture social influence and human conversational complexity beyond traditional ABMs’ simplified representations.
Method
The paper simulates multi-agent conversations with persona-based LLM agents and manipulates confirmation bias, memory updates, and topic framings to study opinion evolution.
Results
LLM agents tend to converge toward ground-truth consensus, whereas stronger prompt-induced confirmation bias produces greater final opinion diversity.
Takeaways & Limitations
LLM agents can support opinion-dynamics simulations, but their factual bias limits role-playing of fact-resistant beliefs; confirmation-bias prompting partially addresses this limitation.
Takeaways & Limitations
The study exclusively uses RLHF-trained models, so different language models may exhibit different opinion-dynamics patterns and broader generalizability remains unassessed.
Abstract
from arXiv · showhide
Accurately simulating human opinion dynamics is crucial for understanding a variety of societal phenomena, including polarization and the spread of misinformation. However, the agent-based models (ABMs) commonly used for such simulations often over-simplify human behavior. We propose a new approach to simulating opinion dynamics based on populations of Large Language Models (LLMs). Our findings reveal a strong inherent bias in LLM agents towards producing accurate information, leading simulated agents to consensus in line with scientific reality. This bias limits their utility for understanding resistance to consensus views on issues like climate change. After inducing confirmation bias through prompt engineering, however, we observed opinion fragmentation in line with existing agent-based modeling and opinion dynamics research. These insights highlight the promise and limitations of LLM agents in this domain and suggest a path forward: refining LLMs with real-world discourse to better simulate the evolution of human beliefs.
1 Introduction
The paper motivates LLM-based opinion-dynamics simulations as a more linguistically and behaviorally expressive alternative to traditional ABMs. It investigates whether interacting LLM agents can provide a more realistic framework while revealing important limitations in their human-like opinion modeling.
- Motivation: Opinion-dynamics models matter for forecasting polarization and designing interventions against harmful social effects.Applications include public-health campaigns, conflict mediation, and misinformation mitigation.
- Limitations of classic ABMs: Traditional ABMs map beliefs and messages to numerical values, overlooking linguistic nuance in real-life conversation.They may represent opinions and communicated signals as scalar values and rely on hand-crafted mathematical updates.
- Limitations of classic ABMs: Rule-based ABMs also struggle to simulate complex human interactions and variability in demographics, worldviews, ideology, and personality.This motivates models that better capture individual differences.
- LLM-based approach: The paper explores LLM agents as an alternative by simulating natural-language conversations among persona-based agents whose beliefs are updated by a transformer-based LLM.The framework manipulates confirmation bias and memory-update functions to study opinion evolution.
- Central finding: LLM agents tend to converge toward denying inaccurate information regardless of their role-played personas, limiting their authenticity for fact-resistant viewpoints.The paper also demonstrates that prompt-induced confirmation bias can create more stubborn agents.
2 Methods
The framework simulates dyadic opinion dynamics among persona-based LLM agents whose verbal interactions update memories and produce classified numerical opinions. Experiments vary memory, confirmation bias, framing, initial distributions, and interaction restrictions, then evaluate final opinion distributions across 15 grounded topics.
- Simulating Opinion Dynamics: Each timestep samples a speaker-listener pair: the speaker writes a message, the listener reacts, and the reaction is classified as a numerical opinion.The opinion space contains five values, from strongly negative to strongly positive.
- Simulating Opinion Dynamics: The simulation records each agent’s opinion trajectory and defines the final opinion distribution as empirical frequencies over the discrete opinion space.An agent’s opinion remains unchanged unless that agent is selected for an interaction.
- Agent’s Persona and Memory: The study compares cumulative memory, which appends experiences, with reflective memory, which continuously integrates experiences into a compact summary.Both strategies are empirically evaluated for their effects on opinion dynamics.
- Agent’s Persona and Memory: Agents begin with distinct personas, initial opinions, and optional cognitive-bias or closed-world instructions represented in text memory.Personas encode demographic and personal attributes, while memory influences future message generation and message assessment.
- Experimental Conditions: Experiments manipulate confirmation bias, framing, and initial opinion distributions across science, history, and commonsense topics with known ground truths.The study uses 15 topics, each paired with true and false framings, and evaluates bias and diversity in final distributions.
- Open-world vs. Closed-world Settings: Closed-world simulations restrict agents to social information, whereas open-world simulations permit external facts; results focus on the closed-world setting because open-world hallucination occurred about 15% of the time.The closed-world setting produced no hallucinations in the reported measurement.
3 Experimental Settings
The experiments simulate opinion dynamics with LLM agents across grounded topics and controlled framings, measuring final bias and diversity. They vary memory, confirmation bias, initial distributions, and interaction controls to assess opinion evolution and inherent model effects.
- Model and implementation: The simulations use ChatGPT agents with temperature 0.7 and FLAN-T5-XXL as an opinion classifier, with GPT-4, Vicuna, and larger networks tested for sensitivity.Personas, optional closed-world restrictions, and cognitive biases are placed in system messages, while historical events enter user messages.
- Topics and framings: The study covers 15 topics spanning scientific theories, historical events, and commonsense knowledge, each presented in true and false framings.The prompts prepend “Theory XYZ” to control wording effects across topics.
- Evaluation metrics: Bias B is the mean final opinion, while diversity D is the standard deviation of the final opinion distribution.Low B and low D indicate central consensus, whereas low B and medium D indicate a diversified distribution.
- Controls and manipulations: Additional experiments skew initial opinions and compare interactive agents with no-interaction controls, with or without role-playing.These controls separate effects associated with personas and social interactions from topic-level LLM biases.
4 Results
LLM agents generally converge toward ground-truth opinions, revealing an inherent accuracy bias that persists across conditions. Prompted confirmation bias instead increases final diversity, while framing produces stronger denial of false claims than endorsement of true ones.
- Inherent bias: Across 15 topics, repeated interaction shifts role-playing agents toward ground-truth consensus, and similar movement occurs when agents are repeatedly queried without communication.Under false framing, B = −1.33 without cognitive bias; under true framing, B = 0.52.
- Confirmation bias: Stronger prompted confirmation bias produces greater final diversity D under both cumulative and reflective memory.This reproduces the classic ABM finding that confirmation bias generates opinion fragmentation.
- Initial opinion distribution: Regardless of the initial opinion distribution, agents shift their expressed opinions toward ground truth, although some extreme false starts remain partly resistant.When all agents initially denied global warming under true framing, B shifted from −2 to −1.2.
- Framing asymmetry: False framing elicits stronger denial than true framing elicits agreement: cumulative-memory B = −1.33 versus 0.52, and reflective-memory B = −1.37 versus 0.60.The authors exclude systematic classifier bias and speculate that training may more strongly encourage refutation of false information.
- Sensitivity analyses: The main trends remain consistent across GPT-4, Vicuna, 20-agent networks, and three temperature-sampling runs on global warming.These sensitivity analyses test model choice, network scale, and sampling randomness.
5 Related Work
Opinion-dynamics research traditionally relies on mathematical ABMs, while newer LLM-based agents simulate social behavior through natural-language interaction. The paper positions LLMs as a more linguistically expressive alternative while retaining links to established fragmentation findings.
- Traditional ABMs: ABMs model agents’ opinion updates with mathematical rules and can produce fragmented opinion clusters when confirmation bias discounts contrasting information.Their numeric representations simplify the complexities of human communication.
- LLM-based agents: LLM-based approaches use natural language to represent more sophisticated social interactions than numeric ABM formulations.Prior work reports human-like responses, planning, memory, event organization, and social-media generation.
- Positioning: The paper states that it is the first to propose LLMs as an alternative to ABMs for opinion-dynamics simulation.This work extends a growing literature on LLM agents simulating social behavior at group level.
6 Conclusion
The study evaluates LLM agents communicating in natural language as a basis for simulating group opinion dynamics. It finds both promise and a factual-alignment limitation, and suggests that real human discourse may be needed to broaden simulated viewpoints.
- Conclusion: LLMs can interpret and produce natural language, role-play differing personas, and simulate human-like linguistic communication in opinion-dynamics settings.The study therefore considers them comparable in purpose to classical ABM simulations while using richer communication.
- Conclusion: LLM agents tend to align with factual information regardless of persona, restricting role-play for individuals with fact-resistant beliefs.Prompting alone may be insufficient to reproduce the full diversity of human viewpoints.
- Future direction: Fine-tuning LLM agents with actual human discourse is proposed as a future direction for more accurate models of human belief dynamics.The proposal is framed as a potential approach rather than an established result.
Limitations
The study is limited by its model selection, one-dimensional opinion representation, topic selection, demographic analysis, and network structure. These constraints narrow how broadly its findings describe human opinion dynamics.
- Model Dependency and Generalizability: Using only RLHF-trained models may contribute to the agents’ tendency to converge toward truth, and other models may produce different dynamics.The authors call for studies spanning a broader range of language models.
- Reduction of Opinion to One-Dimensional Scalar: Reducing opinions to a one-dimensional scalar simplifies the complex process of opinion formation.The authors propose finer-grained or qualitative analyses of belief changes, persuasive messages, and persona-dependent persuasion.
- Limitation in Topic Selection: The study focuses on topics with clear, established ground truths, excluding more open-ended and subjective societal questions.Future work should include topics lacking definitive truth to capture a broader range of opinions and debates.
- The Role of Other Demographic Information: The study gives limited attention to how demographic information shapes information exchange and belief updating.The authors identify political leaning and other demographic features as areas for finer-grained or qualitative analysis.
- Limitation in Network Structure: Allowing every LLM agent to interact with every other agent does not closely represent real-world social interaction patterns.The authors note that people with similar backgrounds tend to interact and recommend more realistic network structures.
Ethics Statement
The authors frame induced bias as a tool for studying undesirable group-level phenomena rather than as an endorsement of biased behavior. This supports examining misinformation, polarization, and echo-chamber formation within the simulation’s scope.
- Ethics Statement: Inducing confirmation bias can produce opinion fragmentation and reduce convergence on factual consensus.The authors position this behavior as relevant to studying misinformation spread, polarization, and echo-chamber formation.
A Results of Global Warming Topic with Reflective Memory
This section documents the reflective-memory setup, including its agent personas, discrete opinion scale, interaction prompts, and opinion classification. It identifies the figure as showing opinion trajectories and the final distribution, while the classifier’s output is reported as more reliable than self-reports.
- Results: Reflective memory is used for the global-warming opinion-dynamics result shown in Figure 6, contrasting with cumulative memory in Figure 4.The passage identifies Figure 6 as the reflective-memory result.
- Results: Figure 6 displays LLM-agent opinion trajectories together with the final opinion distribution F_T.The figure caption identifies the trajectories and final distribution as its contents.
- Personas: Agents receive diverse demographic personas incorporating names, political leanings, ages, genders, ethnicities, education, and occupations.The listed personas instantiate these attributes alongside initial beliefs.
- Opinion Representation: The simulation represents opinions using five discrete values from -2, strongly negative, to 2, strongly positive.The scale also includes -1, 0, and 1 for slightly negative, neutral, and slightly positive opinions.
- Interaction and Memory: At each time step, one speaker emits a message and one listener updates beliefs in a dyadic interaction.The framework distinguishes interaction types including writing and reviewing tweets, with memory prompts updated case by case.
- Opinion Classification: The opinion-value output is more reliable than agents’ self-reported belief ratings in sampled responses and Flan-T5-XXL ratings.The comparison concerns the extracted opinion value versus self-reported ratings.
J Results on GPT-4 and Vicuna
The sensitivity analysis extends the model set to GPT-4 and Vicuna, but each model evaluation uses a restricted experimental scope. GPT-4 is evaluated only on science-based topics, while Vicuna is evaluated only with reflective memory.
- GPT-4: GPT-4 evaluation is restricted to science-based topics because of budget constraints, costing 499.52 USD.The passage reports the model version and scope of this sensitivity analysis.
- Vicuna: Vicuna evaluation is restricted to reflective memory because of GPU-memory constraints, keeping the memory size approximately constant.The passage reports Vicuna-33B-v1.3 as the evaluated model.
- GPT-4 Results: Table 3 summarizes GPT-4 bias and diversity across five science-based topics under both memory strategies, framing conditions, and induced confirmation-bias levels.The table reports averages across five topics with standard errors and notes that increasing confirmation bias correlates with increasing diversity.
K Results on a Larger Network
The larger-network sensitivity analysis doubled the simulation to 20 agents and 200 time steps, producing the same opinion-dynamics patterns as the smaller network. Across conditions, stronger induced confirmation bias was associated with greater opinion diversity, while true framing produced more positive bias.
- N = 20 agents and T = 200 time steps reproduced the same patterns as the smaller 10-agent, 100-step network.The larger-network test focused on Global Warming and used both cumulative and reflective memory strategies.
- Increasing confirmation-bias strength correlated with increasing diversity D in the final opinion distribution.This pattern was reported across framing conditions and memory strategies.
- Under true framing, bias B tended to be more positive, indicating greater agreement, than under false framing.The comparison was reported for both the larger-network table and aggregated results.
L Measurement of Hallucination Rate
The study measured hallucinations by having independent annotators assess sampled tweets and responses in open- and closed-world settings. Hallucinated experiences occurred in the open-world setting but not the closed-world setting, so the reported results focus on the latter.
- 6 of 40 sampled tweets and responses contained hallucinated experiences, yielding an estimated 15% hallucination rate in the open-world setting.Three annotators independently assessed 20 tweets and 20 responses, with 100% agreement.
- The closed-world setting had a 0% hallucination rate, whereas hallucinations occurred in the open-world setting.The paper therefore focuses its reported results on the closed-world setting.
- One open-world response was influenced by a nonexistent article that had not appeared in the tweets read by the agent.The example illustrates how an invented external source could affect an agent’s opinion.
- The study’s opinion-rating validation found 84% agreement between FLAN-T5’s ratings and the majority vote of human annotators.Three authors independently annotated 100 responses, and inter-rater Krippendorff’s alpha was 0.81.