Synthesizing Public Opinions with LLMs: Role Creation, Impacts, and the Future to eDemorcacy

1 Introduction

• Declining response rates and non-response bias make representative survey data difficult to obtain, especially for demographic and political subgroups.Even large samples may leave too few observations for valid subgroup inferences.Source: §1 · p. 1
• MRP estimates subgroup opinions by borrowing information from larger groups, but its effectiveness depends on assumptions that may not always hold.The paper situates LLM-generated synthetic data among alternative approaches to these challenges.Source: §1 · p. 1
• LLMs can use demographic or political context to generate survey responses intended to reflect specified traits, but their accuracy as representations of real opinions remains a concern.Researchers are developing methods to evaluate synthetic data quality.Source: §1 · p. 2
• The paper introduces role creation based on knowledge injection and asks whether role knowledge improves human-response mimicry and whether the approach generalizes across models.It also asks whether survey opinion data can be simulated using LLMs.Source: §1 · p. 2

2.1 The Promises of Synthetic Public Opinions with LLMs

• LLMs have been used to simulate political preferences and survey responses, and can also support question pretesting, longitudinal studies, and open-ended data annotation.Prior work reports simulations of voter choices and General Social Survey responses, alongside other research uses.Source: §2.1 · pp. 2–3

2.2 Approaches to Customize LLMs for Public Opinion Research

• LLM customization spans pre-training and post-training, with domain-specific pre-training offering task specialization at substantial training cost and time.Continuous pre-training adapts a general-purpose model using domain-specific data.Source: §2.2 · p. 3, §2.2.1 · p. 3

2.2.2 Post-training

• Post-training includes fine-tuning and prompt engineering; fine-tuning changes model weights, while prompt engineering uses examples or instructions and may yield limited improvement.Fine-tuning requires time, cost, and AI expertise.Source: §2.2.2 · pp. 3–4
• Knowledge injection can add domain information through methods such as fine-tuning or RAG, but general RAG approaches are not designed specifically for role-play and opinion polling.The paper proposes RAG enhanced with role profiles to simulate specific population groups more accurately than simple RAG.Source: §2.2.2 · p. 4

3 Simulating Voter Preferences

• Existing survey-simulation research reports human-like response patterns, while prompt-based approaches may not capture individual differences and personal traits that shape responses.The paper proposes dynamically creating preferences to produce more nuanced simulated survey feedback.Source: §3 · pp. 4–5

3.1 LLMs as Simulated Opinion Sources

• LLM conditioning uses demographic traits or political orientations in the input context to guide the distribution of subsequent generated tokens.The model predicts text based on preceding tokens and can generate contextually appropriate text from learned statistical patterns.Source: §3.1 · p. 5
• The framework creates roles from attributes and political leanings, stores them in a vector database, and retrieves relevant profiles through RAG for question-specific opinion generation.The retrieved profile guides the LLM to role-play when answering the survey question.Source: §3.1 · p. 5

4 Experiments and Evaluation

• The study compares responses to dataset questions with human answers, using data not used for training and no few-shot examples for either the framework or comparison models.Source: §4 · p. 8

4.1 Data

• The study uses 30 issue-related questions from the 2021 CES, focusing on issue attitudes as frequent and influential polling targets.Source: §4.1 · p. 8

4.2 Experimental Setup

• The experiments ran on Ubuntu 24.04 with an RTX 4090, using fresh interaction states and temperature 0.Responses were generated with llama.cpp and Vulkan API support.Source: §4.2 · p. 8

4.3 Result Adherence

• The evaluation examines both respondent-level and topic-level accuracy to assess how closely LLM responses align with human responses.Source: §4.3 · p. 8

4.3.1 Accuracy for Individual Responses

• Individual-response alignment is measured as a percentage, comparing few-shot prompted pretrained models with role-generated opinions using the Figure 4 template.Source: §4.3.1 · p. 8

5 Discussion

• Survey opinions could be simulated, but respondents’ demographics and profiles had to be supplied through few-shot prompts or role creation.Source: §5 · p. 9
• The framework dynamically creates roles with personality attributes and voter preferences, then uses RAG to select roles for answering survey questions.Source: §5 · p. 9
• All three tested models improved over few-shot-prompted base models, with larger models showing larger gains, possibly reflecting stronger understanding of nuanced instructions.The authors cite Table 3 for adherence trends across targeted questions.Source: §5 · pp. 8–9, Table 3 · p. 9
• Performance gains were inconsistent across models, and the framework can struggle when a model does not understand complex instructions or nuances.Source: §5 · p. 10
• The authors leave fine-tuning with trigger-based generation and the role of non-English languages in opinion simulation as open research questions.Source: §5 · p. 10

• LLM-based opinion synthesis could make polling cheaper and faster, support survey-question pretesting, help campaigns test messages, and simulate views of marginalized or silenced populations.The passage also describes its potential where economic conditions or government restrictions make public-opinion surveys difficult.Source: §6 · p. 10

7.1 Technology Limitations

• Training-data memorization, generalization distortions, and differences across models can undermine synthetic samples’ reliability.The paper notes that researchers may need to understand models’ relative strengths or select high-quality models for ensembles to mitigate inconsistencies.Source: §7.1 · p. 10
• Prompt wording and structure can substantially influence LLM outputs, though role creation makes this sensitivity less significant and merits further study.The authors identify investigating how role creation enhances RAG and in-context learning to mitigate prompt variation as a future research direction.Source: §7.1 · p. 10

7.2 Localization and Less Represented Languages

• Limited training data, imbalances, biases, and cultural nuances challenge polling in less commonly spoken languages and non-Western cultures.The authors identify practical use in these settings as an area for future research.Source: §7.2 · pp. 10–11

7.3 Data Quality

• Polling accuracy depends on the quality and timeliness of data used for knowledge injection and fine-tuning as public opinions and policy preferences shift.Source: §7.3 · p. 11

7.4 Continuous Updates and Learning

• RAG databases and role profiles need regular updates to reflect opinion shifts, demographic changes, and political events, creating ongoing labor and financial costs.Reducing these costs without compromising polling accuracy remains a research challenge.Source: §7.4 · p. 11

7.5 Compliance

• Wider LLM polling adoption could raise compliance concerns, including potential misuse to spread false information and challenges to public trust.Source: §7.5 · p. 11

8 Conclusion

• Role creation with RAG significantly enhances simulated-opinion accuracy and improves answer adherence to an existing human dataset.The authors describe potential impacts on the cost and time of opinion collection and democratic processes, particularly in under-resourced or restrictive environments.Source: §8 · p. 11