PaperlayerView exact revision on arXiv

Source-linked AI summary

Validating LLM-generated political networks with survey-reported partisan exposure

Marco Tonin, Michele Tizzoni, Bruno Lepri, Luca Maria Aiello

arXiv:2610.23199v1cs.SIcs.AIphysics.soc-ph

TL;DR

LLM network evaluations often emphasize aggregate properties, although plausible structures may not reproduce individuals’ political environments. This study compares networks generated from ANES-based personas with survey-reported partisan exposure. It finds model- and group-specific exposure errors, including differences between networks with similar political assortativity.

  • Problem

    Evaluations of LLM-generated social networks often focus on coarse aggregate properties, despite limited evidence about whether they reproduce individuals’ political environments.

    Source: §1 · p. 2

  • Method

    The study generates networks from ANES survey-based personas and compares their ego-level partisan exposure with respondents’ reported exposure, alongside structural and assortativity analyses.

    Source: §1 · p. 3, §2 · p. 3

  • Results

    Some models generate stronger partisan separation than the survey reports, others mainly underestimate Republican exposure, and similar political assortativity can accompany different individual-level environments.

    Source: §1 · p. 3, §3 · p. 12

  • Takeaways & Limitations

    Plausible aggregate network properties can hide differences in partisan environments around individuals, supporting evaluation at both aggregate and individual levels.

    Source: §3 · p. 13

    The benchmark is respondents’ self-reported exposure rather than direct observations of complete social networks, and the study uses a reduced ANES sample of 100 respondents.

    Source: §3 · p. 13

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used to simulate human behavior in social science research, including decisions, interactions, and social relationships. However, in network formation tasks, existing evaluations often focus on aggregate network properties, while networks with plausible structural properties may still fail to reproduce individuals' social and political environments. Here, we generate LLM-based political networks using personas sampled from the nationally representative ANES 2020--2022 Social Media Study and use respondents' survey-reported partisan exposure as an empirical benchmark for the political environments generated for the corresponding personas. We first characterize the structural properties and political assortativity of the generated networks, and then compare ego-level partisan exposure with the survey. We find that the structure and political assortativity of the generated networks vary across experimental configurations. More importantly, when comparing generated partisan exposure with the survey, some models generate stronger separation between Democratic and Republican egos than reported in the survey, with higher same-party and lower cross-party exposure, while others mainly underestimate Republican exposure across ego groups. Moreover, networks with similar political assortativity can still generate different partisan environments around the same personas, including among networks with plausible structural properties. Finally, generated networks show group-specific errors, under- or overestimating exposure to either Democrats or Republicans for specific social groups. These findings show that evaluating LLM-based social simulations only through aggregate outcomes can hide errors in the local environments through which those outcomes emerge, highlighting the importance of validation at both aggregate and individual levels.

1 Introduction

LLM-based social simulations can reproduce plausible aggregate outcomes while still misrepresenting individuals’ and subgroups’ social environments. This study benchmarks networks generated from survey-based personas against respondents’ reported partisan exposure, finding model-specific errors and differences in local political environments even between networks with similar assortativity.

1 Introduction

  • Persona-based LLM simulations may flatten multidimensional differences into simplified profiles, potentially reproducing stereotypes or group-specific biases.These concerns extend to simulated collective dynamics and social networks shaped by homophily and assortative mixing.Source: §1 · p. 2
  • Existing evaluations emphasize coarse network properties, while realistic aggregate structure can coexist with inaccurate political contacts for individuals or subgroups.The paper identifies scarce fine-grained empirical data as a reason for this limited analytic resolution.Source: §1 · p. 2
  • The study generates networks from ANES 2020–2022 Social Media Study personas and uses respondents’ reported partisan exposure as an empirical benchmark.It evaluates structural plausibility, ego-level exposure, and whether similar political assortativity yields similar environments around the same personas.Source: §1 · p. 3
  • Some models generate stronger partisan separation than the survey reports, while others mainly underestimate Republican exposure across ego groups.The study also finds differing partisan environments among networks with similar assortativity and group-specific exposure errors.Source: §1 · p. 3
  • Survey-reported exposure provides an empirical benchmark for whether generated networks reproduce respondents’ political environments beyond network-level measures alone.The paper argues that aggregate network properties may not capture the political environments generated around individuals.Source: §1 · pp. 2–3

2 Results

Across models and prompting configurations, generated network structure and political assortativity vary, and survey comparisons reveal model- and group-specific differences in partisan exposure. Similar political assortativity does not ensure similar partisan environments around the same personas, even among structurally plausible networks.

2.1 LLM-generated network structure varies across configurations

  • Pairwise c1 networks vary most in structure, while larger list-based candidate sets generally reduce degree and clustering and increase path length and modularity.Structural properties move closer to empirical ranges on average, especially for c25 and c50; political-vote assortativity is stronger than demographic assortativity and varies across models and prompts.Source: §2.1 · pp. 5–6, Figure 2 · p. 5
  • 332 of 1,000 reciprocal networks (33.2%) meet all five empirical structural criteria; plausibility rises from 2% for c1 to 80% for c50.The average-degree criterion is the most restrictive; none of the Mistral 7B networks meet all five criteria.Source: §2.1 · p. 6

2.2 Survey-based evaluation reveals model-specific errors in partisan exposure

  • Some models produce stronger Democratic–Republican separation than the survey reports, while others mainly underestimate Republican exposure across ego groups.The two Llama models and Qwen2.5 72B show stronger separation; Mistral 7B and Qwen2.5 7B mainly underestimate Republican exposure.Source: §2.2 · pp. 7–8
  • Differences between models exceed average differences across configurations, and the main exposure-error directions remain relatively stable across list sizes, attribute orderings, and temperatures.The pairwise c1 configuration is an exception for several models, producing exposure values closer to survey reports; reciprocal ties also yield clearer partisan separation than nominations.Source: §2.2 · p. 8

2.3 Political assortativity does not fully capture ego-level partisan exposure

  • Networks with similar political assortativity can still differ in exposure around the same personas, including reciprocal networks meeting all five plausibility criteria.Some plausible Qwen2.5 7B reciprocal-network relations differ by around one point on the 1–5 scale; Mistral 7B is absent because none of its reciprocal networks meet all five criteria.Source: §2.3 · pp. 9–10, Figure 4 · p. 9

2.4 Group-specific errors in LLM-generated partisan exposure

  • Exposure errors vary by demographic and political group: Democratic exposure is both under- and overestimated, while Republican exposure is mainly underestimated.The selected groups and error magnitudes differ across models; Republican-voter exposure is an exception, with the two Llama models and Qwen2.5 72B overestimating it.Source: §2.4 · pp. 10–11, Figure 5 · p. 11

3 Discussion

Generated political networks can appear structurally plausible while misrepresenting survey-reported partisan exposure around individuals and social groups. Model size and prompting choices do not ensure more realistic environments, supporting evaluation at both aggregate and individual levels.

3 Discussion

  • Some models generated stronger partisan separation than the ANES survey, while others mainly underestimated Republican exposure across ego groups.Plausible structural properties did not ensure that generated environments matched survey-reported exposure.Source: §3 · p. 11
  • Larger models did not necessarily produce environments closer to the survey, and similar political assortativity could accompany different ego-level exposure patterns.Assortativity summarizes the overall tendency to connect politically similar people, not exposure across different ego and alter groups.Source: §3 · p. 12
  • The authors situate their findings alongside prior micro–macro mismatches in collective dynamics, individual behavior, and synthetic survey responses [40],,.Their results show that aggregate network plausibility can also coexist with errors in ego-level partisan environments.Source: §3 · p. 12
  • Exposure errors varied across demographic and political groups, with some models overestimating same-party and underestimating cross-party exposure and others underestimating Republican exposure.Such errors may alter political contexts around personas and affect simulated influence, persuasion, communication, or polarization.Source: §3 · pp. 12–13
  • Pairwise and list-based prompts produced different network structures, making network realism dependent on how tie formation is operationalized.The authors caution against treating realistic network generation as an inherent model property.Source: §3 · p. 13
  • The benchmark uses self-reported exposure and 100 respondents, and the one-step friendship task with reciprocal ties omits dynamic relationship formation.The personas also include a limited set of attributes, and the open-weight instruction-tuned models studied may not generalize to other model types.Source: §3 · p. 13
  • Plausible aggregate network properties and similar assortativity can hide differences in partisan environments around individuals and social groups.The authors argue for evaluating LLM social simulations at both aggregate and individual levels, especially when micro-level decisions contribute to collective outcomes.Source: §3 · p. 13

4 Materials and Methods

The study uses a weighted, reduced ANES sample to generate persona-based networks across models and prompting configurations. It evaluates network structure and assortativity, compares ego-level partisan exposure with survey responses, and tests whether exposure errors vary across network pairs and social groups.

4.1 Data sources and sampling

  • The 100-persona sample is drawn from eligible ANES respondents using weighted stratified sampling, with its composition remaining close to the weighted survey population.The survey measures friends’ and family members’ Democratic and Republican affiliations on a five-point scale.Source: §4.1 · p. 14

4.2 Models

  • Five open-weight instruction-tuned models generate ties at temperatures 0.1 and 0.7, using local GPU resources.Two larger models are quantized for local deployment.Source: §4.2 · p. 14

4.3 Persona construction and tie formation task

  • Personas include demographic and political attributes; models assess friendship either one alter at a time or in lists of 5, 10, 25, or 50 candidates.The study builds directed nomination networks from ego choices and undirected reciprocal networks only when both personas nominate each other; exposure analyses use nominations, while structural evaluations use reciprocal ties.Source: §4.3 · pp. 14–15

4.4 Network measures and structural plausibility

  • Structural plausibility is assessed on reciprocal networks using five measures and weighted reference ranges from 57 empirical networks across 13 datasets.A network is classified as plausible only if all five measures fall within their reference ranges; density is excluded because it depends directly on network size.Source: §4.4 · pp. 15–16

4.5 Validation of the partisan exposure using the ANES survey

  • Generated partisan exposure is compared with ANES responses by categorizing each ego’s share of outgoing nominations to Democratic or Republican alters into five equally spaced bins.The denominator includes all alters with valid political affiliation, including those classified as neither party; egos without valid outgoing ties receive zero exposure.Source: §4.5 · p. 16

4.6 Political assortativity and ego-level partisan exposure

  • To test whether similar vote assortativity implies similar local environments, the study compares same-persona exposure across network pairs whose vote assortativity differs by less than 0.02.The analysis is repeated at a 0.05 threshold; reciprocal-network comparisons are restricted to structurally plausible networks, and pairwise results are descriptive because networks may occur in multiple pairs.Source: §4.6 · p. 17

4.7 Group-specific errors in partisan exposure

  • Group-specific mean signed deviations are computed across demographic and political subgroups, excluding groups represented by fewer than five sampled personas.Deviations are calculated for each generated network and group, then averaged across runs; positive values indicate overestimation and negative values underestimation of exposure to the corresponding party.Source: §4.7 · pp. 17–18

Supplementary Information for: Validating

The supplement identifies the paper and its authors’ affiliations.

Supplementary Information

  • The paper is titled “LLM-generated political networks with survey-reported partisan exposure” and lists Marco Tonin, Michele Tizzoni, and Bruno Lepri as authors.The listed affiliations include the University of Trento, Fondazione Bruno Kessler, and IT University of Copenhagen.Source: Supplementary Information for:… · p. 25

S1 Sample

The reduced ANES sample is reported to reproduce the weighted population composition across the main demographic and political characteristics.

S1 Sample

  • The N = 100 sample’s demographic and political composition is compared with the weighted eligible ANES distribution, and the reported composition remains close to that population.Table S1’s last column gives absolute differences in percentage points.Source: S1 Sample · p. 26, Table S1 · p. 26

S2 Network metrics

The supplement provides additional network metrics and empirical reference information across reciprocal and nomination networks and experimental configurations.

S2 Network metrics

  • Fig. S1 covers reciprocal networks at T = 0.7, while Fig. S2 covers nomination networks at T = 0.1 and T = 0.7 across prompting configurations.The tables provide full structural and assortativity measures; Tables S6 and S7 document benchmark networks and structural-plausibility reference ranges.Source: S2 Network metrics · p. 27, Figure S1 · p. 27, Figure S2 · p. 28

S3 Partisan exposure results

The supplementary results compare generated partisan exposure with the ANES survey for nomination and reciprocal networks. They report signed deviations and exposure distributions across models and experimental configurations.

S3 Partisan exposure results

  • Fig. S3 compares exposure deviations across models and configurations and shows cumulative exposure distributions for Democratic and Republican egos in reciprocal networks.Positive MSD indicates overestimation relative to the ANES survey; negative MSD indicates underestimation.Source: Figure S3 · p. 39, S3 Partisan exposure results · p. 39
  • Tables S8 and S9 report complete MSD values for nomination and reciprocal networks, respectively, across experimental configurations.Each table reports unweighted means across 10 runs with standard errors; positive values indicate overestimation and negative values underestimation.Source: Table S8 · p. 40, Table S9 · p. 49
  • Table S8 covers prompting methods, temperatures, and attribute orderings, including whether vote appeared first or last in the persona description.Its values compare generated exposure with the corresponding ANES personas.Source: Table S8 · p. 42

S4 Political assortativity and partisan exposure

The matching analysis examines whether networks with similar political assortativity produce similar partisan exposure for the same personas, using two assortativity thresholds and two network representations.

S4 Political assortativity and partisan exposure

  • The matching analysis includes nomination networks and reciprocal networks meeting all five structural plausibility criteria.The less restrictive 0.05 threshold is compared with the main 0.02 threshold.Source: S4 Political assortativity and… · p. 58, Figure S4 · p. 58
  • Fig. S4 compares within-person exposure differences for network pairs matched at assortativity thresholds of 0.02 and 0.05.Exposure is compared for the same personas on a 1–5 scale; boxes show interquartile ranges, medians, and 5th–95th percentile whiskers.Source: Figure S4 · p. 58, S4 Political assortativity and… · p. 58
  • Table S10 reports eligible matched-pair counts for both thresholds and network representations.Nomination counts use all generated networks; reciprocal counts use only networks meeting all five structural plausibility criteria, and matched networks differ in experimental configuration.Source: Table S10 · p. 59

S5 Group-specific exposure results

The supplementary group-specific analysis reports signed deviations between generated and survey-reported partisan exposure for demographic and political groups, complementing the main-text results across temperatures.

S5 Group-specific exposure results

  • Fig. S5 shows group-specific exposure deviations at T = 0.7, complementing Fig. 5's corresponding results at T = 0.1.Negative MSD indicates underestimation relative to the survey, while positive MSD indicates overestimation.Source: S5 Group-specific exposure results · p. 60, Figure S5 · p. 60
  • Fig. S5 highlights social groups with the largest deviations and distinguishes prompting methods by point style, with larger markers indicating model averages.The two panels show exposure to Democrats and Republicans.Source: Figure S5 · p. 60
  • Table S11 reports group-specific MSDs for exposure to Democrats and Republicans in nomination networks for demographic and political groups represented by at least five personas.Values are model averages across prompting methods and attribute orderings, reported separately by temperature with standard errors across configurations.Source: S5 Group-specific exposure results · p. 60, Table S11 · p. 61

Related papers

Loading 2610.23199v1…