Source-linked AI summary
Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations
Yong Cao, Haijiang Liu, Arnav Arora, Isabelle Augenstein, Paul Röttger, Daniel Hershcovich
TL;DR
Large-scale surveys are valuable but costly, motivating accurate simulation of group-level response distributions. The paper specializes LLMs with first-token probability fine-tuning and finds improved performance across seen and unseen countries, questions, and an unseen survey, while systematic limitations remain.
Problem
Large-scale surveys are costly, and prior LLM simulation work has mainly used prompting rather than specializing models for response-distribution simulation.
Method
The paper fine-tunes LLMs by aligning first-token option probabilities with human response distributions for survey questions.
Results
Fine-tuning substantially improves simulation across seen and unseen WVS countries and questions and generalizes to the unseen Pew survey.
Takeaways & Limitations
Specialization benefits survey-response simulation, but the paper cautions against using current LLMs for this task because systematic inaccuracies remain.
Takeaways & Limitations
The trained models are highly specialized for predicting answer distributions for specified populations and survey questions, so general-purpose alignment cannot be claimed.
Abstract
from arXiv · showhide
Large-scale surveys are essential tools for informing social science research and policy, but running surveys is costly and time-intensive. If we could accurately simulate group-level survey results, this would therefore be very valuable to social science research. Prior work has explored the use of large language models (LLMs) for simulating human behaviors, mostly through prompting. In this paper, we are the first to specialize LLMs for the task of simulating survey response distributions. As a testbed, we use country-level results from two global cultural surveys. We devise a fine-tuning method based on first-token probabilities to minimize divergence between predicted and actual response distributions for a given question. Then, we show that this method substantially outperforms other methods and zero-shot classifiers, even on unseen questions, countries, and a completely unseen survey. While even our best models struggle with the task, especially on unseen questions, our results demonstrate the benefits of specialization for simulation, which may accelerate progress towards sufficiently accurate simulation in the future.
1 Introduction
The paper frames survey-response distribution simulation as a way to reduce the cost of large-scale surveys and specializes LLMs for this task. First-token fine-tuning improves performance across seen and unseen settings, but systematic inaccuracies remain.
- Large-scale surveys inform policy, industry, and social science, but they are costly and time-intensive.
- LLM survey simulations could accelerate research and inform policy, but out-of-the-box models often produce erroneous, stereotypical, or overconfident answers in culturally diverse contexts.
- The paper specializes LLMs using first-token probabilities to minimize divergence between predicted and actual country-level response distributions.
- Fine-tuning substantially improves prediction accuracy across seen and unseen WVS countries and questions and generalizes to the unseen Pew Global Attitudes Survey.
- Even the best-fine-tuned models remain imperfect, especially on unseen questions, and all tested models produce less cross-country diversity than human survey data.
- The paper introduces distribution prediction as a simulation task, releases three adapted datasets, and reports that specialization performs best among tested methods.
2 Related Work
Prior work uses LLMs to simulate human responses and explores prompting-based improvements, while this paper treats distribution simulation as a calibration problem beyond majority-answer accuracy.
- Previous studies found reasonably accurate LLM group-level simulations in behavioral, economic, and US political-survey settings.
- Follow-up research identified biases and conceptual challenges in LLM-based simulations.
- Other work investigated prompting strategies and frameworks for improving simulation accuracy.
- This paper frames distribution simulation as human calibration for multiple-choice surveys, rather than evaluation against only the majority answer.
3 Cultural Survey Simulation Dataset
The dataset is built primarily from the 2017–2022 WVS, formatted around survey questions, response options, countries, and population-level distributions. Splits test generalization across questions, countries, and an unseen Pew survey.
- Data Source: The main dataset uses the 2017–2022 WVS, which covers 66 countries and more than 80,000 respondents.
- Data Source: The analysis retains 65 countries with more than 1,000 respondents to support robust cross-cultural representation.
- Prompt Settings: The formatted data preserves original questions and options while excluding infrequent validity-check responses such as “not applicable” and “refuse to answer”.
- Dataset Split: The dataset uses the first 259 WVS questions and divides questions and countries into topic- and population-based groups for challenging generalization tests.
- Dataset Split: Training, validation, and test splits evaluate unseen value questions, unseen regional countries, and representative medium-GDP countries.
- Dataset Split: A Pew Global Attitudes Survey subset evaluates generalization to a completely unseen survey with different cultural questions.
4 Methodology
The methodology defines simulation as matching group-level human response distributions rather than predicting single answers. It fine-tunes LLM first-token probabilities against observed distributions using KL divergence and parameter-efficient adaptation.
- Framework: The framework targets generalization across populations and survey questions through first-token alignment fine-tuning.
- Probability Distribution Simulation: Unlike direct multiple-choice prompting, the task predicts the full response-option distribution for a question and evaluates alignment with human distributions.
- Probability Distribution Simulation: For question Q with options O, models are evaluated by matching P(O|Q) to the response distribution of a specified population rather than selecting the majority option.
- First-Token Probability Alignment: The model converts first-token logits for each answer option into probabilities using softmax normalization.
- First-Token Probability Alignment: KL divergence loss aligns the LLM’s first-token probability distribution with the human response distribution.
- First-Token Probability Alignment: Low-Rank Adaptation is used to make the fine-tuning process more parameter-efficient.
5 Experimental Setup
The experiments compare seven models across three model families, evaluating fine-tuning against zero-shot prompting and control conditions using distribution-alignment metrics.
- 5.1 Models: Seven models span three families: Vicuna1.5, Llama3, and Deepseek-Distilled-Qwen.The tested parameter sizes range from 7B to 32B.
- 5.2 Baselines: Fine-tuning is compared with zero-shot prompting, while [ctrl] randomly replaces countries to test sensitivity to country context.Additional baselines, including K-Nearest Neighbors, are reported in the appendix.
- 5.3 Metrics: Evaluation reports 1-JSD and Earth Mover Distance for alignment between predicted and country-level reference response distributions.Higher 1-JSD and lower EMD indicate greater similarity, and both metrics range from 0 to 1.
6 Results
Fine-tuning substantially improves survey-response distribution simulation across unseen countries, questions, models, languages, and an unseen survey, but unseen questions remain difficult and diversity does not consistently predict accuracy.
- RQ1: Generalization Performance: Fine-tuned models consistently outperform zero-shot prompting across model sizes and types, including a 34.3% 1−JSD increase and 0.069 EMD decrease for Llama3-8B-Instruct Avg.Fine-tuning improves alignment with real-world response distributions.
- RQ1: Generalization Performance: Unseen questions are harder than unseen countries, with Llama3-Instruct reaching 0.781 versus 0.886 1−JSD, respectively.The models are more robust to new country distributions than to questions absent from training.
- RQ1: Generalization Performance: Fine-tuning improves performance across three model families and model sizes, narrowing or reversing initial zero-shot differences between models.For example, Llama3-Instruct gains 34.3% in 1−JSD versus 6.1% for Llama3-Base.
- RQ1: Generalization Performance: Fine-tuning increases option-prediction accuracy across models and test subsets, with particularly notable gains on unseen countries.Accuracy is computed by selecting the highest-probability option and comparing it with the survey majority choice.
- RQ2: Variation Sensitivity: Fine-tuned models are more sensitive to country context than zero-shot models, with a 16.7% versus 3.7% average 1−JSD drop under random country replacement.This pattern suggests specialization in country-specific cultural distributions beyond modeling prior response frequencies.
- RQ2: Variation Sensitivity: Fine-tuning slightly increases Base-model country diversity but reduces Instruct-model diversity, and post-fine-tuning diversity does not consistently correlate with distribution accuracy.The analysis compares 1−JSD across countries for selected questions.
- RQ2: Variation Sensitivity: African-country accuracy is relatively high for seen questions but drops substantially for unseen questions, especially in Egypt and Tunisia; Instruct models generally outperform Base models.Ethiopia and Nigeria approach 80% accuracy on seen questions.
- Additional Analyses: Chinese results are marginally lower than English results on 1−JSD and EMD, but the difference is not significant.Distilled-Qwen also does not outperform Llama3 on this task despite stronger performance on most other benchmarks.
7 Conclusion
Fine-tuning substantially improves survey-response simulation across diverse countries and questions, including generalization to an entirely new survey. However, even the best models show systematic inaccuracies, especially for unseen questions and culturally diverse contexts.
- Fine-tuning substantially improves response simulation for seen and unseen countries and questions, and generalizes to an entirely new survey dataset.
- Models remain systematically inaccurate, particularly when simulating responses to unseen questions.
- Even the best-performing models require caution in culturally diverse contexts.
Limitations
The study’s limitations concern the narrow specialization of its models, restricted language and country coverage, and evaluation on models no larger than 32B parameters.
- Scope: The trained models are specialized for predicting answer distributions for specified survey questions and human populations, not general-purpose applications.
- Scope: Whether the fine-tuning approach produces less biased or more aligned general-purpose models remains for future work.
- Language and Countries Coverage: The study uses only English prompts and countries as proxies for cultures, limiting applicability to non-English models and fine-grained cultural contexts.
- Model Choice: The evaluation excludes models with more than 32B parameters because of computational resource constraints.
Ethics Statement
The study uses open-access, properly licensed datasets, models, and prompt settings, while acknowledging ethical risks from deploying LLMs and interpreting or misusing generated content.
- All datasets, models, and prompt settings are sourced from open-access repositories and properly licensed to their original creators.
- Although the framework involves no inherently risky operations, deploying LLMs carries potential ethical implications.
- Users are encouraged to consider safety and ethical factors because generated content may be misused or misinterpreted.
D More Option Prediction Analysis
Additional analyses show stronger option prediction on seen than unseen questions, improved country-distribution alignment after fine-tuning, and consistently better performance than the tested baselines.
- Option Prediction Accuracy: Llama3 performs better on seen C3-Q1 than unseen C3-Q3 questions, with Instruct, fine-tuned, and country-specific differences across evaluations.
- Country Distributions: Fine-tuned models improve 1−JSD scores across countries, especially where zero-shot performance was poor, and align Base and Instruct performance.
- Baseline Comparison: Fine-tuned models consistently outperform KNN, Avg_Culture, JSON-ZS, and other tested baseline methods on 1-JSD scores.
- Country Distributions: On unseen cultural questions, the Instruct model shows a more distinct improvement than the Base model in the distribution of 1−JSD global scores.
F More Model Evaluation
Additional evaluations show that Qwen-7B is harder to train than the compared Llama and Distill-Qwen models, while training level appears more influential than language differences in 1-JSD performance. English fine-tuning also correlates more strongly with Chinese fine-tuning than with English zero-shot performance.
- Original Qwen Models: Qwen-7B shows more difficulty converging than Llama-8B-Instruct and Distill-Qwen-32B during training.The authors suggest extensive safety or policy alignment may introduce additional optimization constraints.
- Language and Training Effects: Figure 9 compares 1-JSD scores across languages using 100 randomly selected samples.The figure’s red dashed line marks the average value for the displayed score distributions.
- Language and Training Effects: Training level has a larger apparent influence than language differences on model performance.The English FT–Chinese FT correlation is 0.579, compared with 0.349 between English ZS and English FT.
- Language and Training Effects: The English FT–Chinese FT correlation is 0.579, exceeding the 0.349 correlation between English ZS and English FT.Both correlations are positive, but the stronger fine-tuned cross-language relationship supports the reported training-level effect.