PaperlayerView exact revision on arXiv

Source-linked AI summary

Stop Drawing Scientific Claims from LLM Social Simulations Without Robustness Audits

Jinyi Ye, Lei Cao, Ding Chen, Emilio Ferrara

arXiv:2605.18890v1physics.soc-phcs.AIcs.CYcs.MA

TL;DR

LLM social simulations offer a powerful way to study collective behavior, but their architectural flexibility creates a validation gap: small implementation changes can alter macro-level outcomes. This paper tests that problem in two case studies and introduces TRAILS, finding that sensitivity varies across design choices and model families and arguing that audit strength should match claim strength.

  • Problem

    LLM social simulations lack shared standards for determining how much robustness evidence is required before their outputs support scientific or policy claims.

  • Method

    The paper uses controlled repeated Prisoner’s Dilemma and social-media echo-chamber case studies, then organizes robustness audits through TRAILS across agent, interaction, and system levels.

  • Results

    Sensitivity is uneven across architectural choices and model families, with persona-format perturbations producing cooperation-rate shifts of up to 76 percentage points.

  • Takeaways & Limitations

    Scientific claims from LLM social simulations should be calibrated to robustness audits that test the architectural dimensions relevant to the claim.

  • Takeaways & Limitations

    Comprehensive audits across every TRAILS dimension, multiple models, and seeds can be infeasible, so the framework prioritizes dimensions most likely to confound a claim.

Abstract

from arXiv · show

The scientific claims drawn from LLM social simulations should be no stronger than the robustness audits that support them. Generative agents bring new expressive power to agent-based modeling, enabling simulations of collective social processes like cooperation, polarization, and norm formation. Yet they also introduce complexity through additional architectural choices, such as agent specification, memory representation, interaction protocols, and environment design. Small perturbations that appear minor to researchers can cascade into macro-level outcomes through repeated interaction, creating a "butterfly effect." Consequently, scientific claims drawn from LLM social simulations may reflect implementation artifacts rather than the social mechanisms being modeled. We support this position with two case studies: a repeated Prisoner's Dilemma and a social media echo chamber simulation. Across multiple models, minor perturbations in persona format and game-instruction framing shift cooperation rates by up to 76 percentage points, while network homophily and hub assignment produce significant and consistent shifts in polarization metrics. We also find that sensitivity is unevenly distributed across both architectural choices and model families: the same perturbation that produces the 76 pp shift in one frontier model only shifts another by 1 pp. Robustness is therefore a property that should be measured per claim and per model, not assumed. To address this validation gap, we introduce TRAILS (Taxonomy for Robustness Audits In LLM Simulations), a robustness-audit taxonomy spanning three levels of simulation design: agent (micro-level), interaction (meso-level), and system (macro-level). We call for robustness to become a first-order validation requirement before LLM social simulations are used to explain mechanisms, evaluate interventions, or inform decisions.

1 Introduction

LLMs expand the expressive range of social simulations, but added architectural choices make seemingly minor implementation differences capable of changing collective outcomes. The paper therefore argues for calibrating scientific claims to robustness audits and introduces TRAILS as a framework for doing so.

  • LLMs enable agent-based models to generate social behavior directly from natural language, expanding the design space for studying collective phenomena.
  • 76 percentage points separate cooperation rates when surface formatting changes but persona content remains constant in a repeated Prisoner’s Dilemma.The result comes from two persona prompts in a 10-round game using gpt-5.2 and 30 seeds per condition.
  • Realism-centered validation is insufficient because simulation conclusions should also remain stable across reasonable alternative implementations.
  • Small changes in prompts, memory, networks, or interaction protocols can compound through repeated interactions into substantially different macro-level outcomes.
  • Two controlled case studies across four LLMs demonstrate that small architectural perturbations can shift macro outcomes unevenly across design dimensions and model families.
  • TRAILS organizes robustness audits across agent, interaction, and system levels and offers heuristics for prioritizing audits by claim type, simulation complexity, and domain stakes.
  • Robustness audits are intended to distinguish stable social dynamics from artifacts of simulation design as LLM simulations become scientific tools and policy testbeds.

2 Opportunities and Validation Gap in LLM-based Social Simulations

LLM-based social simulations improve behavioral expressiveness but introduce additional uncertainty and implementation complexity. The resulting validation gap requires attention not only to realism, but also to whether collective conclusions remain stable under reasonable design alternatives.

  • The paper defines its scope as LLM-mediated human-like agents interacting in shared environments to produce aggregate collective phenomena.
  • The scope distinguishes goal- or incentive-structured scenarios from open-ended scenarios without a single shared task or payoff structure.
  • LLM agents add natural-language reasoning, memory, communication, and adaptation to social simulation, addressing limitations in behavioral realism.
  • The same generative flexibility introduces uncertainty from black-box behavior, cultural biases, and additional simulation design choices.
  • Small implementation choices can propagate through repeated interactions and produce large differences in macro-level outcomes, creating risks for simulation stability and controllability.

3 Position: Calibrate Robustness Audits to the Strength of Claims

The paper argues that robustness evidence should scale with the strength of the scientific claim, because LLM simulations can amplify small design and representation changes into unstable collective outcomes. TRAILS therefore frames auditing around claim type, simulation complexity, and domain stakes.

  • Minor prompt, formatting, output-constraint, and option-ordering changes can alter individual LLM decisions, creating micro-level robustness risks.
  • A small change in one agent’s response can become a different action, message, memory update, or social signal for other agents.
  • Design-level perturbations change the substantive simulation setup, whereas representation-level perturbations change how a fixed setup is presented to the LLM.
  • Exploratory probes require modest robustness evidence, while mechanism claims and policy claims require progressively stronger audits.
  • Audit requirements also depend on simulation complexity, including whether collective behavior is goal-structured or theory-guided and open-ended.
  • High-stakes domains such as public health, policy, platform governance, and financial markets require stronger robustness evidence because results may inform real-world conditions or interventions.
  • The field lacks shared standards for the robustness required before simulated outcomes can support exploratory, mechanism, or policy claims.
  • Similar-looking echo-chamber patterns may conceal unstable agent movement and interaction behavior across runs.

4 Are LLM Social Simulations Robust? Case Studies of the Butterfly Effect

Two controlled case studies show that seemingly minor architectural choices can substantially alter collective outcomes in both repeated cooperation and echo-chamber simulations. Sensitivity varies by perturbation and model family, so robustness cannot be assumed.

  • Study design: The study tests architecture sensitivity across a repeated Prisoner’s Dilemma and an open-ended social-media echo-chamber simulation.The two settings span explicit incentive structure and network-mediated collective behavior.
  • Goal-structured simulation: The Prisoner’s Dilemma varies persona format, game-instruction framing, and memory representation while holding the substantive game fixed.The perturbations target design choices that are inconsistently reported and rarely ablated in LLM-cooperation research.
  • Goal-structured simulation: 76 percentage points separated DESCRIPTIVE and PLAIN personas in two-agent cooperation, while TABULAR personas earned 1.46 more points per round than DESCRIPTIVE personas against AlwaysCooperate.The persona formats preserve the same underlying strategic meaning but produce different cooperation and payoff outcomes.
  • Goal-structured simulation: Memory representation produced mostly small effects, with most payoff shifts below 0.2 and cooperation-rate shifts below 0.1.These changes did not shift the simulation equilibrium as persona format or game framing did.
  • Open-ended simulation: Modest increases in initial network assortativity raised final stance assortativity from 0.142 to 0.247 and 0.287, shifting interaction toward echo-chamber-like structure.Weighted same-group edge ratios also crossed from 0.462 to values above 0.5 in the higher-homophily conditions.
  • Open-ended simulation: Hub assignment changed the strength but not the presence of echo-chamber effects, with weighted same-group edge ratios of 0.759 for pro-regulation hubs and 0.690 for anti-regulation hubs.Final stance assortativity did not differ significantly across hub conditions.
  • Open-ended simulation: Increasing recommendation feed size from 5 to 10 posts increased stance assortativity from 0.247 to 0.277 and weighted same-group edge ratio from 0.542 to 0.568.Activation probability and memory window produced no significant changes in either metric.
  • Cross-model findings: Sensitivity is uneven across architectural choices and models: persona formatting produced gaps of approximately 77, 36, and 1 percentage points in claude-haiku-4-5, gemini-2.5-flash, and deepseek-v3, respectively.The same perturbation therefore ranges from substantial to negligible depending on model family.

5 Toward Robustness Audits for LLM Social Simulations

The paper proposes TRAILS as a practical framework for auditing whether collective outcomes remain robust to simulation-design and representation changes. It organizes audits across simulation levels and prioritizes them according to the claim, complexity, and stakes.

  • TRAILS framework: TRAILS is a robustness-audit framework for testing whether simulated collective outcomes remain stable under changes in simulation systems.It is introduced to identify actionable perturbations in LLM social simulations.
  • TRAILS framework: TRAILS-D audits design-level perturbations, while TRAILS-R audits representation-level perturbations that preserve the substantive simulation condition.Measurement and evaluation sensitivity are treated as part of the downstream evaluation pipeline rather than simulation perturbations.
  • TRAILS-D: TRAILS-D organizes design perturbations across eight dimensions following the three-level structure of LLM social simulation.The framework links these perturbations to the coupling between LLM agents and simulation design.
  • TRAILS-R: TRAILS-R covers formatting, instruction order, labels, context compression, and interaction sequencing as representation-level perturbations.These changes can alter how agents interpret an otherwise fixed simulation condition.
  • Prioritizing audits: The proposed audit heuristics align perturbations with the claim’s mechanism, simulation complexity, and domain stakes.The paper does not require every study to audit every TRAILS dimension.

6 Alternative Views and Discussion

The paper addresses objections about sensitivity, evidentiary scope, premature standardization, and audit cost by proposing calibrated, transparent robustness practices for LLM social simulations.

  • Alternative views: LLM simulations need explicit sensitivity protocols because their textual perturbation surface is distinct, even though other empirical methods also exhibit sensitivity.The paper compares LLM simulations with established practices such as cross-validation, ablations, stress tests, and multiverse analysis.
  • Alternative views: Audit strength should scale with claim type, simulation complexity, and domain stakes rather than restricting LLM simulations to hypothesis generation.Policy-testbed uses require the strongest audits, while claims surviving TRAILS-D and TRAILS-R deserve more weight than claims that do not.
  • Alternative views: TRAILS is intended as a minimal, revisable vocabulary for reporting tested and unaudited perturbations, not as a fixed methodological checklist.This approach addresses concerns that premature standardization could freeze an evolving design space.
  • Alternative views: Audit costs can be managed by matching audit strength to claim strength and prioritizing dimensions most likely to confound the claim.Transparent reporting allows evidence to accumulate across studies without requiring every study to test every perturbation.
  • Recommendations: Robustness audits are needed so simulated findings reflect stable social dynamics rather than artifacts of simulation design.The paper frames robustness as necessary for simulations used as scientific tools and policy testbeds.
  • Recommendations: Future studies should report their evidentiary role, tested perturbations, stable and sensitive findings, and unaudited dimensions.The paper also calls for reusable perturbation libraries, open datasets, benchmarks, and empirical work on when butterfly effects occur.

A TRAILS (Taxonomy for Robustness Audits In LLM Simulations)

This section situates LLM social simulation within agent-based modeling and defines its scope, scenarios, and representative testbeds, while introducing design dimensions relevant to TRAILS audits.

  • Background: LLM agents extend agent-based modeling by generating reasoning, memory, communication, and adaptation through natural language.The paper presents this flexibility as addressing limited behavioral realism while leaving validation and sensitivity challenges unresolved.
  • Scope: The paper defines social simulation through micro-level agents, meso-level interactions, and macro-level collective outcomes.The scope includes human-like agents interacting in shared environments to produce phenomena such as polarization, echo chambers, and norm emergence.
  • Scope: The scope excludes single-agent simulations, systems representing non-human or aggregate actors, and task-oriented multi-agent systems.These exclusions keep the focus on modeling collective human behavior.
  • Scenarios: Goal-structured scenarios generate collective outcomes through incentives, constraints, rules, or payoffs, whereas open-ended scenarios generate patterns through local interaction.The paper connects these scenarios to cooperation, negotiation, polarization, and echo chambers.
  • Case studies: The repeated Prisoner’s Dilemma provides a tightly controlled testbed in which action space, payoffs, and horizon are specified.This leaves architectural and prompt-level choices as the main sources of variation.
  • Case studies: The echo-chamber study varies homophily, hub assignment, activation probability, memory window, and recommendation-feed size.These perturbations target network structure, selective exposure, memory or context, and engagement dynamics.

C.1 Shared Experimental Protocol

The experiments hold core simulation settings fixed while varying architectural choices across two case studies and multiple frontier models. Each condition uses repeated independent runs and controlled comparisons.

  • Model coverage: Four frontier models are evaluated, with gpt-5.2 used for reported analyses and three additional models used for cross-model robustness checks.The additional models are claude-haiku-4-5, gemini-2.5-flash, and deepseek-v3.
  • Statistical protocol: Each condition uses N = 30 independent simulation runs with distinct random seeds, treating the simulation run as the unit of analysis.The analysis reports mean differences, confidence intervals, and Cohen’s d, with Holm correction for multiple pairwise comparisons.
  • Controls: Decoding and parsing settings remain fixed within each comparison so observed differences can be attributed to the intended perturbation.Unless otherwise specified, calls use temperature = 0.3 and top-p = 0.95.
  • Prisoner’s Dilemma design: The Prisoner’s Dilemma varies persona format, game-instruction framing, and memory format in single-agent and two-agent interaction modes.The game remains a 10-round repeated Prisoner’s Dilemma with a fixed payoff matrix.
  • Prisoner’s Dilemma design: The Prisoner’s Dilemma episodes run for T = 10 rounds, with Cooperate and Defect actions and a fixed payoff matrix.The payoffs are (C, C) = (3, 3), (C, D) = (0, 5), (D, C) = (5, 0), and (D, D) = (1, 1).
  • Prisoner’s Dilemma design: The Prisoner’s Dilemma uses fixed action parsing, randomized prompt order in two-agent runs, and DEFECT as the fallback after an invalid response.These defaults are applied consistently across persona, instruction, and memory perturbations.

C.2.2 Evaluation Metrics

The evaluation metrics capture individual performance and cooperation in the Prisoner’s Dilemma, while the experimental factors isolate alternative prompt and memory representations. The echo-chamber design evaluates behavior on a fixed network with frozen stances.

  • Prisoner’s Dilemma metrics: Average payoff measures overall performance under the game’s incentive structure, while cooperation rate measures how often an agent chooses cooperation.Both metrics are used to evaluate the repeated Prisoner’s Dilemma.
  • Prompt perturbations: The Prisoner’s Dilemma tests persona presentation as PLAIN, DESCRIPTIVE, or TABULAR while holding the underlying strategic meaning fixed.The manipulation changes only how the persona is presented.
  • Prompt perturbations: Game instructions use CANONICAL, MORALIZED, or RISK framing while keeping the payoff matrix and persona fixed.The framings describe the same underlying game with different language.
  • Memory perturbations: Memory representation crosses table versus narrative history with whether summary statistics are included.Summaries can include cooperation rates, cumulative payoffs, and joint outcome counts.
  • Echo-chamber design: The echo-chamber simulation uses 100 agents with frozen AI-regulation stances and varies their exposure and interaction conditions on a fixed network.Agents see recent posts from direct neighbors and choose POST, REPOST, REPLY, or DO NOTHING.

C.3.2 Evaluation Metrics

The echo-chamber evaluation measures whether interactions concentrate among similar stances and how network design choices alter exposure and connectivity. The study preserves key structural properties while perturbing targeted architectural features.

  • Evaluation metrics: Stance assortativity measures whether agents preferentially interact with others holding similar AI-regulation positions.The coefficient compares observed within-group interaction with expected within-group interaction under random mixing.
  • Evaluation metrics: Same-group edge ratio measures the fraction of interaction edges connecting agents with the same side label.Its weighted version accounts for repeated interactions between the same agents.
  • Perturbations: The study tests input-network homophily, hub assignment, activation probability, memory window, and recommendation feed size as perturbations.These choices target commonly under-audited mechanisms in echo-chamber simulations.
  • Network construction: Input-network homophily is varied while preserving a fixed degree sequence, changing which stance groups are connected without changing each agent’s degree.Three target bands are used: lower, medium, and higher homophily.
  • Network construction: The network uses a heavy-tailed degree sequence with a small number of high-degree agents and many lower-degree agents.Hub assignment then changes which agents occupy those high-degree positions while retaining the same degree sequence.
  • Data collection: Simulation logs record agent decisions, network positions, interactions, prompts, outputs, and run-level summaries for computing behavioral and network outcomes.The Prisoner’s Dilemma logs support cooperation and payoff analyses, while echo-chamber logs support interaction and polarization metrics.

D Extended Experimental Results

The extended results examine how alternative instruction and memory presentations affect Prisoner’s Dilemma outcomes and how echo-chamber parameters affect network metrics. Recommendation feed size is the only tested echo-chamber parameter here with consistent significant effects.

  • Game-instruction framing: Instruction framing changes payoff and cooperation even when the payoff matrix and persona remain fixed.The extended comparisons include CANONICAL, MORALIZED, and RISK framings.
  • Memory representation: Memory conditions compare table and narrative histories with and without summary statistics.The results are shown for both single-agent and two-agent settings.
  • Echo-chamber parameters: Increasing recommendation feed size from 5 to 10 posts significantly increases stance assortativity from M = 0.247 to 0.277 and weighted same-group edge ratio from M = 0.542 to 0.568.The reported p-values are 0.014 and 0.003, respectively.
  • Echo-chamber parameters: Activation probability and memory window produce no statistically significant differences in either final stance assortativity or weighted same-group edge ratio.Recommendation feed size is the only one of these three parameters that consistently strengthens echo-chamber-like interaction.

D.3 Prisoner’s Dilemma: Experiment Results With Other LLMs

The Prisoner’s Dilemma robustness checks replicate prompt-perturbation experiments across four frontier models. The reported figures compare how persona format, game-instruction framing, and memory representation relate to payoff and cooperation outcomes.

  • Cross-model design: The experiments replicate the Prisoner’s Dilemma analyses on gpt-5.2, claude-haiku-4-5, gemini-2.5-flash, and deepseek-v3.The additional models serve as a cross-model robustness check rather than a benchmark of model quality.
  • Persona format: Persona-format effects are evaluated in both single-agent and two-agent settings using average payoff and cooperation rate.The figures compare persona formats across models, with the two-agent analysis using the same persona format for both agents.
  • Persona format: Pairwise persona-format comparisons report statistically significant differences in average payoff and cooperation rate across models.These comparisons are shown separately for the single-agent and two-agent Prisoner’s Dilemma settings.
  • Game-instruction framing: Game-instruction framing is compared across instruction variants, opponent policies, and models using average payoff and cooperation rate.The two-agent framing analysis gives both agents the same game framing and also reports pairwise differences across framings.
  • Memory representation: Memory-representation effects are compared across memory conditions in both single-agent and two-agent settings using average payoff and cooperation rate.Pairwise heatmaps report statistically significant differences across memory conditions.

D.4 Echo Chamber: Experiment Results With Other LLMs

The echo-chamber experiments replicate network-structure and exposure-design checks across multiple language models, while the Prisoner’s Dilemma experiments vary persona format, game framing, and memory representation. These experiments operationalize robustness audits through controlled changes to agent prompts, interaction settings, and information presentation.

  • Echo Chamber: The echo-chamber experiments replicate sensitivity checks with claude-haiku-4-5, gemini-2.5-flash, and deepseek-v3 alongside gpt-5.2.The cross-model check tests whether sensitivity to network structure and exposure design persists beyond one model family.
  • Echo Chamber: The echo-chamber analyses compare final stance assortativity and weighted same-group edge ratio under varied initial network homophily, hub assignment, activation probability, memory window, and recommendation-feed size.Separate cross-model figures report boxplot comparisons for each architectural or network-design perturbation.
  • Prisoner’s Dilemma: The single-agent Prisoner’s Dilemma setting uses a model against an undisclosed fixed external opponent policy over repeated rounds.The prompt specifies simultaneous Cooperate-or-Defect choices, payoff outcomes, history-based decisions, and JSON-formatted action responses.
  • Prisoner’s Dilemma: The two-agent Prisoner’s Dilemma setting uses two model-driven players receiving the same prompt structure with role-specific histories and memory blocks.Each agent is assigned a role label and chooses simultaneously against another model-driven player under the same repeated-game framing.
  • Prisoner’s Dilemma: The prompt perturbations change strategic-persona format, game-instruction framing, and memory presentation while preserving specified task elements.Persona variants include descriptive, bullet-list, and tabular formats; game framing includes canonical, moralized, and risk-framed variants; memory varies table versus narrative history and added statistics.
  • Prisoner’s Dilemma: The memory perturbation compares raw histories shown as tables or narrative text, with some variants adding cooperation, payoff, and joint-outcome statistics.The four listed conditions cross presentation format with whether summary statistics supplement the raw history.
Loading 2605.18890v1…