PaperlayerView exact revision on arXiv

Source-linked AI summary

ElectionSim: Massive Population Election Simulation Powered by Large Language Model Driven Agents

Xinnong Zhang, Jiayu Lin, Libo Sun, Weihong Qi, Yihang Yang, Yue Chen, Hanjia Lyu, Xinyi Mou, Siming Chen, Jiebo Luo, Xuanjing Huang, Shiping Tang, Zhongyu Wei

arXiv:2410.20746v3cs.CLcs.CYcs.HC

TL;DR

Election simulation needs to represent complex individual backgrounds at massive scale while supporting realistic population distributions and systematic evaluation. ElectionSim addresses this with a million-level social-media-derived voter pool, demographic sampling, and the PPE benchmark. It reports Micro-F1 0.812 for voter-level vote-related tasks and matches actual outcomes in 47/51 states and 12/15 battleground states.

  • Problem

    Traditional ABM methods struggle to incorporate complex individual backgrounds and provide interactive prediction interfaces, while existing LLM simulations struggle with massive-scale diversity and quantity.

    Source: §1 · p. 2

  • Method

    ElectionSim combines a million-level Twitter-derived voter pool, demographic sampling for real-world distribution alignment, and the PPE poll-based presidential election benchmark.

    Source: §1 · p. 2, §2.1.1 · p. 4

  • Results

    0.812 Micro-F1 is achieved on voter-level vote-related tasks, with predictions matching actual outcomes in 47/51 states and 12/15 battleground states.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework supports high-accuracy U.S. presidential election simulations at both voter and state levels, with further analyses reporting robustness and effectiveness.

    Source: §1 · p. 2, §8 · p. 22

    The lower Macro-F1 than Micro-F1 indicates room to improve simulation of minority opinions and suggests some model bias.

    Source: §4.1 · p. 13

Abstract

from arXiv · show

The massive population election simulation aims to model the preferences of specific groups in particular election scenarios. It has garnered significant attention for its potential to forecast real-world social trends. Traditional agent-based modeling (ABM) methods are constrained by their ability to incorporate complex individual background information and provide interactive prediction results. In this paper, we introduce ElectionSim, an innovative election simulation framework based on large language models, designed to support accurate voter simulations and customized distributions, together with an interactive platform to dialogue with simulated voters. We present a million-level voter pool sampled from social media platforms to support accurate individual simulation. We also introduce PPE, a poll-based presidential election benchmark to assess the performance of our framework under the U.S. presidential election scenario. Through extensive experiments and analyses, we demonstrate the effectiveness and robustness of our framework in U.S. presidential election simulations.

1 Introduction

ElectionSim addresses limitations of traditional and existing LLM-based election simulations by combining a million-level voter pool, customized demographic sampling, systematic evaluation, and interactive simulated-voter dialogue. In U.S. presidential election experiments, it reports strong voter-level and state-level agreement with actual outcomes.

Motivation

  • Traditional ABM methods struggle to incorporate complex individual backgrounds and provide interactive interfaces for explaining predictions, limiting their accuracy and persuasiveness.ABM typically uses heuristic-like rules or mathematical functions to scale individual actions into collective forecasts.Source: §1 · p. 2
  • Existing LLM-based simulations have difficulty supporting election simulations with both the diversity and quantity required at massive population scale.Prior work has focused on individual behavior generation and multiagent collaboration rather than massive population election simulation.Source: §1 · p. 2

Challenges

  • The framework targets accurate individual simulation because aggregated macro outcomes depend on precise individual behavior and sufficiently personalized contextual information.The paper identifies missing contextual data as a barrier to modeling diverse voter behavior, motivations, and decisions.Source: §1 · p. 2
  • The paper frames demographic alignment as essential because random sampling can miss real-world distributions and introduce source-driven biases.Its evaluation design is intended to assess election simulations across multiple aspects rather than prediction accuracy alone.Source: §1 · p. 2

Framework

  • ElectionSim collects 171,210,066 tweets to build a million-level voter pool and uses demographic sampling to align simulated voters with real-world distributions.The framework is designed to improve individual simulation and population-level representativeness.Source: §1 · p. 2

Results

  • 47/51 states and 12/15 battleground states match the actual 2020 presidential election results, while voter-level vote-related tasks reach Micro-F1 0.812.These results are reported under the ElectionSim framework and PPE benchmark.Source: §1 · p. 2

Contributions

  • ElectionSim introduces a flexible customized distribution strategy, a poll-based presidential election benchmark, and an interactive system for selecting and dialoguing with simulated voters.The interactive system supports selection by attributes or responses to specific questions.Source: §1 · p. 3

2 ElectionSim: Massive Population Election Simulation

ElectionSim builds a massive voter-simulation pipeline from social-media data, demographic annotation, real-world population distributions, and iterative proportional fitting. The framework supports customized sampling by aligning simulated voters with demographic and ideological distributions.

2 ElectionSim: Massive Population Election Simulation

  • ElectionSim processes social-platform data into a massive voter pool and enables sampling from customized target distributions.The pipeline aggregates user data, applies demographic tagging, and uses interactive proportional fitting to approximate joint distributions.Source: §2 · pp. 3–4

2.1.1 Social Media Data Collection

  • 171,210,066 tweets from 9,596,198 users across 50 languages form the Twitter-based source dataset for the voter pool.The collected tweets span January 1, 2020 to December 29, 2020, with an average of 21.69 words per tweet.Source: §2.1.1 · p. 4
  • The preprocessing pipeline aggregates posts by user, filters non-English content, retains users with over 30 posts, samples 30 posts, and removes highly repetitive content.User cleaning uses average pairwise Jaccard overlap scores from five sampled tweets.Source: §2.1.1 · p. 4

2.1.2 Demographic Feature Annotation

  • Demographic annotation combines a poll-informed taxonomy, API-generated labels, manual verification, majority voting, and five Longformer-based classifiers.Three APIs label 10,000 training users, while five annotators validate a 200-user test set; final models are selected by test-set F1.Source: §2.1.2 · pp. 5–6

2.2.2 Methods for Distribution Sampling

  • The population distribution combines Census and ANES data to represent state-level gender, age, race, ideology, partisanship, and related attributes.Census data provide direct demographic distributions, while ANES distributions are treated as representative for ideology, partisanship, and community type.Source: §2.2.2 · pp. 7–8

2.2.3 Iterative Proportional Fitting

  • IPF estimates each state’s joint distribution over gender, race, age group, ideology, and partisanship before simulation.Although IPF usually does not converge, 888 of 918 marginals have gaps below 5%, and the estimated joint distribution is used for simulation.Source: §2.2.3 · p. 8

3 PPE: Poll-based Presidential Election Benchmark

PPE defines a poll-based benchmark that aggregates simulated individual responses into state-level opinion tendencies. It adapts ANES questionnaires into a simpler agent-oriented survey and evaluates several prompt-based simulation baselines.

3.1 Task Definition

  • PPE simulates each person’s poll responses and aggregates them according to state-level demographic distributions to obtain macro-level opinion tendencies.The benchmark is presented as an application of massive social simulation.Source: §3.1 · p. 8

3.2 Poll Collection

  • The benchmark uses the ANES 2020 Time Series Study, whose pre-election and post-election interviews support analysis of U.S. presidential-election opinions and behavior.The source includes a two-wave panel design.Source: §3.2 · p. 9
  • The original ANES survey contains screener, pre-election, and post-election components with modular sections and logically dependent follow-up questions.A total of 8,280 respondents completed pre-election interviews, and 7,453 also completed post-election reinterviews.Source: §3.2 · p. 9

3.3 Poll-based Questionnaire Design

  • The redesigned simulation questionnaire covers 24 socially significant topics in 49 questions, mostly with three to five answer options.The design selects topics, removes complex question types, and preserves broad issue coverage.Source: §3.3 · pp. 9–11
  • Questionnaire simplification removes multiple-answer, fill-in-the-blank, and conditional questions, while merging unclear responses into a single “DK/RF” category.The changes make the remaining questions applicable across respondents and reduce ambiguous answer categories.Source: §3.3 · p. 11
  • The redesign converts intensity questions and seven-point Likert scales into basic position questions using three-point scales.This reduces the number of choices agents must evaluate and removes distinctions such as “Agree strongly” versus “Agree somewhat.”Source: §3.3 · p. 11

3.4 Prompt-based Evaluation Baselines

  • The evaluation compares prompting with user profiles, demographic distributions, and profiles augmented with users’ social-media experiences.The third baseline adds posts to distinguish users with similar demographic features.Source: §3.4 · p. 12

4 Evaluation

Evaluation covers voter-wise and state-wise simulations under the PPE benchmark, using individual-response and state-election metrics. The framework reaches strong voter-level performance and accurately predicts 2020 presidential election outcomes across many states, while revealing minority-opinion bias and sensitivity to temporal constraints.

4.1 Voter-wise Simulation

  • Voter-wise simulation predicts responses from demographic tags, evaluating overall fit with Micro-F1 and minority-opinion expressiveness with Macro-F1.The experiment samples 1,000 individuals from 8,280 ANES 2020 respondents and evaluates questions across multiple topics.Source: §4.1 · p. 13
  • Voting-related questions perform better than the full dataset, with the optimal model exceeding 80% Micro-F1 on the voting subset.Table 9 reports performance for both the overall test set and the six-question voting-related subset.Source: §4.1 · p. 13, Table 9 · p. 14
  • Macro-F1 is generally much lower than Micro-F1, indicating weaker simulation of minority opinions and a degree of model bias.The evaluation excludes respondents who refused to answer, reflecting the study’s focus on expressed respondent tendencies.Source: §4.1 · p. 13
  • Open-source 70b models are comparable to commercial models, while 7b models underperform; GPT-4o-mini balances cost, performance, and efficiency.Unless otherwise specified, commercial-model simulations use GPT-4o-mini.Source: §4.1 · p. 14

4.2 State-wise Simulation

  • State-wise simulation aggregates sampled individual predictions using state demographic information to estimate overall opinion tendencies and election outcomes.Sampling is performed state by state using demographic distributions, with sample sizes based on 1/10,000 of each state’s 2020 Census population.Source: §4.2 · p. 14

4 Evaluation

  • Using Qwen2.5-72b with a 1/1000 population sample of around 300,000 agents, the approach predicts outcomes in 47 states and 12 battleground states, with reduced vote-share RMSE.Source: Table 10 · p. 15

4.2 State-wise Simulation

  • Baseline2 and Baseline3 accurately predict battleground-state results, and the framework outperforms naive random sampling in public-opinion modeling.Baseline3 adds temporally filtered social-media posts to demographic guidance, excluding posts from November 2020 onward to avoid election knowledge leakage.Source: §4.2 · pp. 14–15
  • Baseline3 performs similarly to Baseline2 for open-source models, whereas GPT-4o-mini performs worse under temporal constraints than with some election knowledge leakage.State-level evaluation uses CER for result consistency and CVS as RMSE between simulated and actual vote shares.Source: §4.2 · pp. 14–15

5 Further Analysis

Further analyses select direct-answer prompts, examine the roles of temporal and ideological information, characterize response distributions, and compare state-level predictions with ABM and actual results.

5.1 Prompt Strategy Study

  • Direct answers with dictionary-format personal information perform best on GPT-4o-mini and Claude-3.5-sonnet, so the main experiment adopts this configuration.Biography prompts provide insignificant improvements on most metrics and may introduce hallucinated profile information.Source: §5.1 · p. 16, Table 11 · p. 16

5.2 Ablation Study on Voter-wise Simulation

  • Removing temporal information has the least effect on individual-modeling precision, but temporal context remains important for time-dependent group modeling and opinion forecasting.The authors associate temporal removal with possible pseudo-predictions based on world knowledge, while time-based historical sampling is not reflected at the individual level.Source: §5.2 · p. 16
  • Ideology affects overall answer accuracy, whereas party affiliation produces larger changes on voting-related subsets, distinguishing their influences on voter behavior.The ablation table summarizes this distinction between ideological and party-based effects.Source: §5.2 · pp. 16–17, Table 12 · p. 17

5.3 Response Behavior Analysis

  • Model-simulated responses have higher HHI than ANES responses, while Llama-3-70b-Instruct has better JS Divergency despite weaker state-wise simulation than GPT-4o-mini and Qwen2.5-72b-Instruct.The analysis uses JS Divergency for distribution similarity and HHI for response concentration.Source: §5.3 · p. 18
  • Negative responses remain below 40% for every question, indicating that the adjusted samples remain representative despite excluding negative responses.The negative-response ratio is reported against the original data for each question.Source: Figure 10 · p. 17, §5.3 · p. 18

5.4 Case Study

  • GPT-4o-mini matches actual election outcomes in all six reported states, but consistently overestimates the Biden-Harris vote share.The comparison includes ABM predictions and actual results for Michigan, Ohio, Pennsylvania, Indiana, West Virginia, and Missouri.Source: §5.4 · pp. 18–19, Table 13 · p. 18

6 Visualization

The visualization work organizes macro trends, micro-level voter information, and interactive filtering into linked views that support progressive exploration from populations to individual agents.

6.1 Design Rationale

  • The interface requirements are to display macro trends, expose micro characteristics, and provide interactive filtering of groups and their cross-dimensional relationships.These requirements address distributions, detailed voter choices, and correlations among selected groups.Source: §6.1 · p. 19

6.2 Visualization Design

  • Figure 11 links map, condition, individual, distribution, and high-dimensional views through a shared dataset and interactive population selection.Selecting conditions updates the displayed population across the views.Source: §6.2 · pp. 19–20
  • Map and condition filters narrow populations geographically or by question responses, while the map view displays support rates, state distributions, and selected-question options.Multiple conditions can be combined with geographic filters, and map selections update other components.Source: §6.2 · p. 20
  • Individual Information shows 100 randomly selected voters, their demographic and sample details, and a chat interface for questioning the selected voter agent.The 100-voter sample refreshes whenever map or condition filters change the visualization population.Source: §6.2 · p. 20
  • Distribution Overview contrasts state-level relative option proportions with participation and ideological trends across nine question categories.Color intensity encodes within-state proportions, while bubble size and vertical position represent DK/RF responses and radicalism or conservatism.Source: §6.2 · p. 20
  • High-Dimensional View uses Sankey diagrams and scatterplot matrices to display option combinations across selected questions, using sampled populations for stability in large states.Users choose dimensions, and the system computes counts for each combination.Source: §6.2 · p. 20

6.3 Usage Strategies

  • The workflow moves from national distributions to filtered groups, category-specific trends, and conversational inspection of individual agents.Figure 12 presents this progression from aggregate exploration to individual dialogue.Source: §6.3 · p. 21, Figure 12 · p. 21

7 Related Works

Related work spans conventional and agent-based election prediction, alongside LLM simulations focused on individual behavior, multi-agent coordination, and domain-specific workflows.

7.1 Political Election Research

  • Election research has progressed from polls, expert judgment, and statistical models toward ABM and LLM-based approaches for modeling voter behavior and political phenomena.ABM combines micro-individual characteristics with macro-socioeconomic factors, while preliminary LLM studies address electoral prediction, policy evaluation, and public sentiment.Source: §7.1 · p. 21

7.2 Muti-Agent Simulation by Large Language Models

  • LLM agent simulations distinguish individual role-playing agents from multi-agent systems, with the latter emphasizing interaction mechanisms and task-level outcomes.Multi-agent work further separates general-purpose scenarios from domain-specific workflows in fields such as journalism, economics, and social media.Source: §7.2 · p. 21

8 Conclusions

ElectionSim combines a million-level voter pool and customized distribution sampling for U.S. presidential election simulation, achieving high-accuracy voter- and state-level results.

8 Conclusions

  • ElectionSim achieves high-accuracy simulation results at both voter and state levels, with prompt, ablation, and case-study analyses supporting its reported robustness and effectiveness.The framework is evaluated under the poll-based presidential election benchmark PPE.Source: §8 · p. 22

A Simulation Result of the 2024 Presidential Election

The simulation forecasts a Democratic advantage in the 2024 election while documenting the prompts, filtering, and repeatability procedures used to construct and operate simulated voters.

A Simulation Result of the 2024 Presidential Election

  • Harris is projected to win 8 of 15 battleground states, while Trump is projected to win 7, giving Democrats a reported advantage.The forecast uses 2020 ANES and 2022 Census distributions because 2024 demographic data were unavailable.Source: A Simulation Result of the 2024… · p. 27

B.1 Prompt for Demographic Annotation

  • Demographic annotation classifies users by age, gender, race, party affiliation, and ideology, returning predefined category labels in JSON.The prompt asks annotators to assess a person holistically from their historical speeches.Source: §B.1 · p. 29

B.2 Prompt for Voter-wise Simulation

  • Voter-wise simulation prompts agents to answer 2020 ANES questions as real people using their state and personal information, with responses formatted as JSON.The state-wise prompt additionally supplies historical comments and candidate information for the 2020 presidential election.Source: §B.2 · p. 29, §B.3 · p. 30

B.4 Prompt for Generating Personal Biography

  • The framework generates second-person biographies from personal information to provide agents with descriptions of their past experiences.The requested output includes both an answer and a reason in JSON format.Source: §B.4 · p. 30

C.1 Language Filtering

  • Language filtering uses langid to detect post languages, although English posts may include users from other English-speaking regions such as the UK.Source: §C.1 · p. 31

C.2 Repeatability Calculation

  • The repeatability procedure samples five historical posts, computes pairwise Jaccard similarity over unique-word sets, and averages the similarities.Users with repeatability above 0.28 are filtered out to remove spam accounts and advertisements.Source: §C.2 · p. 31

C.3.1 Implementation Setting

Commercial APIs receive a fixed demographic-annotation prompt at temperature 0, and inconsistent outputs undergo manual verification to establish ground truth.

C.3.1 Implementation Setting

  • Commercial APIs use prompt template B.1, clean outputs against predefined answers, and run at temperature 0.Source: §C.3.1 · p. 31

C.3.2 Manual Verification

  • Five annotators manually verify inconsistent samples using Twitter homepage links, while unresolved attributes are marked null.At least two annotators handle each sample, and consistency scores are reported in Figure 14.Source: §C.3.2 · p. 31

C.4 Consistency Calculation

API consistency is evaluated against manually established ground truth, while the supplementary material presents example agent interactions.

C.4 Consistency Calculation

  • API consistency compares each commercial API’s annotations with ground truth and measures the proportion of matching samples.The score is intended to assess annotation reliability.Source: §C.4 · pp. 31–32
  • Annotator agreement averages 67.23% in the reported consistency figure.Source: Figure 14 · p. 32

D Supplementary Case Study

D.1 Case 1

The case study illustrates an agent whose vote follows perceived policy and value alignment and can change after a major candidate shift or scandal.

D.1 Case 1

  • Agent A initially supports Biden for his vision, governing experience, issue commitments, and perceived ability to unite the country.Source: §D.1 · p. 33
  • After Biden withdraws, Agent A switches support to Harris because of her leadership, progressive policies, and commitment to social justice.Source: §D.1 · p. 33
  • Agent A says a major policy shift or scandal involving Harris could change the vote, indicating that the simulated preference is conditional on value alignment.Source: §D.1 · p. 33

D.2 Case 2

Case 2 presents a conservative voter who consistently supports Donald Trump because his broader platform and values align with her beliefs, even when issue priorities differ. The accompanying questionnaire covers policy preferences, social issues, immigration, government, and democratic norms.

D.2 Case 2

  • The simulated voter identifies Trump as more conservative because of positions on taxes, regulation, national defense, and the Second Amendment.Biden is characterized as supporting more progressive positions, including social-program spending and stronger environmental regulations.Source: §D.2 · p. 34
  • Trump’s overall platform and conservative values outweigh the possibility that Harris would prioritize the voter’s preferred issues.The voter also cites trust in Trump’s leadership and vision for the country’s future.Source: §D.2 · p. 34

Q15 Environment

  • Additional questions assess views on education, defense, health care, social welfare, and aid to disadvantaged groups.The items include preferences about public services, defense spending, government responsibility, and assistance to Black Americans.Source: E Questionnaire · pp. 35–36

E Questionnaire

  • The questionnaire measures preferences on government spending, environment, elections, the economy, health care, immigration, and candidate-party performance.Items ask respondents to select whether spending should increase, decrease, or remain unchanged and which party would handle issues better.Source: E Questionnaire · p. 36

Q20 Aid to Blacks

  • The questionnaire also covers LGBTQ+ rights, abortion, criminal justice, international affairs, climate policy, parental leave, immigration, unrest, government control, election integrity, and democratic norms.Democratic-norm questions address institutional checks, consequences for misconduct, shared facts, press freedom, and presidential power.Source: Q20 Aid to Blacks · p. 39, Q34 Immigration · p. 40

Q34 Immigration

  • Immigration and governance items include citizenship for children of unauthorized immigrants, treatment of long-term undocumented residents, border-wall support, protest activity, party control, government responsiveness, and felon voting rights.The questionnaire further asks about news organizations’ freedom to criticize political leaders and the importance of institutional checks.Source: Q34 Immigration · pp. 40–41

Related papers

Loading 2410.20746v3…