Source-linked AI summary
Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinions
Joseph Suh, Erfan Jahanparast, Suhong Moon, Minwoo Kang, Serina Chang
TL;DR
Existing prompting approaches have struggled to predict human survey-response distributions, motivating a method that directly fine-tunes LLMs on structured survey data. The paper introduces SubPOP and trains models to match subpopulation response distributions, achieving improved agreement with human responses and strong generalization to unseen settings.
Problem
Prompt-based steering has limited success reproducing human response distributions, while existing survey datasets provide limited thematic and institutional coverage.
Method
The paper fine-tunes LLMs on SubPOP by pairing survey questions and subpopulation descriptions with observed human response distributions.
Results
32-46% reduction in Wasserstein distance versus top-performing baselines shows improved distributional agreement across subpopulations.
Takeaways & Limitations
Fine-tuned LLMs generalize to unseen subpopulations, survey waves, question topics, and survey families, supporting opinion prediction for untested settings.
Takeaways & Limitations
The paper leaves open when generalization may fail and whether opinions along unexplored intersecting demographic dimensions are faithfully captured.
Abstract
from arXiv · showhide
Large language models (LLMs) present novel opportunities in public opinion research by predicting survey responses in advance during the early stages of survey design. Prior methods steer LLMs via descriptions of subpopulations as LLMs' input prompt, yet such prompt engineering approaches have struggled to faithfully predict the distribution of survey responses from human subjects. In this work, we propose directly fine-tuning LLMs to predict response distributions by leveraging unique structural characteristics of survey data. To enable fine-tuning, we curate SubPOP, a significantly scaled dataset of 3,362 questions and 70K subpopulation-response pairs from well-established public opinion surveys. We show that fine-tuning on SubPOP greatly improves the match between LLM predictions and human responses across various subpopulations, reducing the LLM-human gap by up to 46% compared to baselines, and achieves strong generalization to unseen surveys and subpopulations. Our findings highlight the potential of survey-based fine-tuning to improve opinion prediction for diverse, real-world subpopulations and therefore enable more efficient survey designs. Our code is available at https://github.com/JosephJeesungSuh/subpop.
1 Introduction
The paper addresses limited distributional fidelity in prompt-based LLM opinion prediction by fine-tuning on structured survey data. SubPOP enables scaled training and evaluation, improving prediction across diverse and unseen subpopulations and surveys.
- Surveys are valuable for public-opinion research but costly, time-consuming, and vulnerable to non-response and sampling biases.
- The method is intended to support pilot testing, oversampling decisions, and analysis-pipeline testing before full surveys, rather than replace human participants.
- Prompt-based steering has struggled to reproduce human response distributions, especially for underrepresented groups.
- SubPOP contains 70K subpopulation-response distribution pairs, 6.5× larger than previous datasets, enabling fine-tuning at scale.
- Fine-tuned models generalize to unseen subpopulations, survey waves, and survey families, including evaluations across ATP and GSS.
- 32-46% reduction in Wasserstein distance versus top-performing baselines demonstrates improved agreement between LLM and human response distributions.
2 Related Work
Prior opinion-prediction work uses survey datasets and prompting or community text, while related alignment methods often target different objectives. This paper focuses on distributional prediction for fine-grained US subpopulations with cross-survey generalization.
- OpinionQA contains roughly 500 questions from 14 ATP waves but has limited thematic diversity and relies on a single survey family.
- Prompting methods steer LLMs with demographic profiles, few-shot survey examples, interviews, or personal narratives.
- Community-text fine-tuning can use large, continuously updated corpora but struggles with disproportionate online representation and incomplete subpopulation coverage.
- Other survey-based approaches use individual-response collaborative filtering, meta-learning for new groups, or WVS-based country and culture alignment.
- This work studies finer-grained US subpopulations and whether training on one US-representative survey generalizes to another.
- Pluralistic-alignment research addresses related but different problems in fine-tuning LLMs.
3 Methods
The method fine-tunes LLMs to map a question and target subpopulation to a human response distribution, using forward KL divergence. SubPOP expands thematic and institutional coverage, while Wasserstein distance evaluates distributional agreement.
- Fine-tuning setup: Survey data is represented as a question, target subpopulation, and observed human opinion distribution.
- Fine-tuning setup: The model outputs probabilities for each multiple-choice option, producing a complete response distribution for the specified subpopulation.
- Fine-tuning setup: Forward KL divergence trains predicted option probabilities to match the empirical human distribution for each question–subpopulation pair.
- Fine-tuning setup: Forward KL penalizes underestimating responses that are frequent in the human data, encouraging coverage of the real distribution.
- SubPOP dataset: SubPOP training uses 3,229 ATP questions, while evaluation adds 133 GSS questions to test new topics and a different survey institution.
- Evaluation metric: Wasserstein distance measures distributional mismatch after mapping ordinal answer categories to numeric values.
- Evaluation metric: Unlike one-hot accuracy, Wasserstein distance retains ordinal relationships, assigning larger penalties to predictions farther from the human distribution.
4 Experiments
Experiments compare fine-tuning on SubPOP with prompting and other baselines, evaluating distributional accuracy, subgroup generalization, steerability, and dataset scaling. Fine-tuning consistently improves Wasserstein distance, generalizes beyond training groups and survey families, and continues benefiting from more data.
- Overall results: Fine-tuning on SubPOP reduces WD by 32–46% on OpinionQA and 39–42% on SubPOP-Eval versus the best baselines.SubPOP-Eval uses GSS surveys, whereas SubPOP-Train uses ATP data, creating a distribution shift across survey families.
- Overall results: Zero-shot and few-shot prompting provide only modest or minimal WD improvements, while fine-tuning substantially improves distributional matching.Few-shot prompting supplies five question-response distribution examples, but gains remain minimal.
- Generalization across target subpopulations: All 22 subpopulations improve by 38%–54%, with a 46.7% average relative improvement and 4.4% standard deviation.The authors attribute balanced gains partly to uniformly distributed training data across subpopulations.
- Generalization across target subpopulations: Unseen age groups achieve a 44.7% average relative improvement, comparable to seen groups; other unseen groups reach 43.1% with 6.7% standard deviation.SubPOP-Train contains no opinion distributions for the cited age groups, including ages 18–29 and 65+.
- Steerability: Fine-tuning reproduces human-like intergroup disagreement patterns for unseen education groups, unlike zero-shot prompting with the base model.Only two education groups were included in fine-tuning, while four other groups were unseen.
- Effect of scaling the dataset: Using 25% of SubPOP-Train yields 72%–78% of the total improvement, but performance continues increasing through 100% across three models.Similar trend-line slopes suggest comparable data efficiency across model architectures.
- Effect of scaling the dataset: A dataset 25 times larger is estimated to reduce Mistral-7B’s WD to 0.07, closer to the empirical lower bound of 0.031.The estimate is based on fitted trend lines rather than a directly observed result.
5 Conclusion
Fine-tuning LLMs on SubPOP markedly improves prediction of human response distributions, including for unseen subpopulations, survey waves, topics, and survey families. Performance also scales favorably with dataset size, while important questions remain about generalization limits and broader opinion dimensions.
- SubPOP is 6.5× larger than previous collections and supports fine-tuning and evaluation for survey response distribution prediction.
- Fine-tuning captures group-specific opinion variability and generalizes to unseen subpopulations, survey waves, question topics, and survey families.
- Model performance continues to scale favorably as the fine-tuning dataset grows.
- Generalization remains an open issue, including when models may fail and whether they faithfully capture intersecting identities or changing opinions over time.
A.1 American Trends Panel Datasets
The American Trends Panel provides individual-level survey data with demographic, response, and weighting information, while OpinionQA selects contentious questions with strong intergroup disagreement. The appendix describes 22 analysis subpopulations and the topic coverage of included waves.
- ATP releases anonymized individual records containing identification numbers, demographic details, survey responses, and survey weights.
- OpinionQA contains 500 contentious questions from 14 ATP waves selected for high intergroup disagreement.
- Ordinality mappings structure response options and enable calculation of one-dimensional Wasserstein distance.
- The analysis uses 22 subpopulations, with respondent counts reported for ATP Wave 82.
- Table 4 documents ATP wave topics included in SubPOP-Train and OpinionQA, including COVID-19 questions embedded in waves 68–79 surveys.
A.3 SubPOP-Train
SubPOP-Train expands ATP coverage across 53 waves, filters unsuitable questions, minimally refines wording with GPT-4o, and adds semantically broader question coverage than OpinionQA. The dataset also includes GSS questions for out-of-distribution evaluation.
- The curation gathers 53 ATP waves from Waves 61–132 after removing waves with missing demographic or ideology information or unavailable data.
- Questions with more than 10 response options, redacted data, prior-question dependencies, or direct inferability are excluded.
- GPT-4o minimally refines remaining question wording to make questions suitable for language-model prompting without modifying response options.
- Question embeddings show SubPOP is 6.5× larger than OpinionQA and covers semantic areas not represented in OpinionQA.
- The evaluation subsamples 133 questions from GSS 2022 to assess out-of-distribution generalization.
A.5 Inspection of Identical Questions
The study identifies semantically identical questions between SubPOP-Train and OpinionQA using embedding similarity, manually inspecting highly similar pairs and selecting a conservative threshold to maximize recall.
- Question-pair cosine similarities reveal a high-similarity tail, which is inspected in the 0.8–1.0 range.
- Figure 7 compares two-dimensional t-SNE embeddings for questions from OpinionQA, SubPOP-Train, and SubPOP-Eval.
- A cosine similarity threshold of 0.87 detects all manually identified semantically identical question pairs.
- The conservative 0.87 threshold prioritizes recall over precision to ensure overlapping questions are detected.
B Experiment Details
The experiments fine-tune models with LoRA using survey response distributions and compare forward KL-divergence with Wasserstein distance as training objectives. KL and WD produce similar WD outcomes, so the experiments primarily use KL because it is broadly applicable and does not require ordinal information.
- Training setup: LoRA fine-tuning uses selected learning rates and batch sizes for Llama-2-7B, Mistral-7B-v0.1, Llama-2-13B, and Llama-3-70B.The full-data configurations use batch size 256, with learning rates ranging from 2e-5 to 2e-4; Llama-3-70B uses a heuristic setting without hyperparameter search.
- Training setup: The training setup applies LoRA with rank 8, alpha 32, dropout 0.05, and updates to query and value matrices.LoRA parameters are initialized with σ=0.02, and optimization uses AdamW with weight decay 0.
- Training objective: The experiments compare forward KL-divergence and Wasserstein distance as training objectives for matching response distributions.Wasserstein distance is simplified using cumulative distribution functions because survey responses are one-dimensional and ordinal.
- Training objective: The two objectives yield similar results in WD, the primary measure of opinion-distribution matching.The comparison uses train and validation loss curves for Llama-2-7B fine-tuned on 90% of OpinionQA, with 10% held out for validation.
- Training objective: KL-divergence is used primarily because it has broader applicability and does not require ordinal information.The authors attribute the objective similarity to KL penalizing low-probability assignments without substantially changing overall distribution geometry.
C.1 Effect of Response Distribution Modeling
The study evaluates explicit probability modeling against one-hot and augmented-data approaches, alongside prompting baselines and fine-tuned models. Explicit modeling better captures opinion diversity, while fine-tuned models improve distribution matching and reproduce human intergroup disagreement patterns, including for unseen subpopulations.
- Response distribution modeling: Explicit probability modeling substantially outperforms one-hot prediction, which fails to capture opinion diversity within subpopulations.One-hot prediction models only the most probable response rather than the full response distribution.
- Response distribution modeling: Explicit probability modeling also outperforms augment by N while using fewer data samples and avoiding replication overhead.The performance gap exceeds the quantization error from discretization, approximately 0.01 or 0.005 in the reported cases.
- Unseen subpopulations: Fine-tuned models improve performance across unseen subpopulations and reproduce human intergroup disagreement patterns more closely than zero-shot prompting.The analysis examines demographic and political-ideology traits, including heatmaps for unseen groups and mixed seen/unseen ideology groups.
- Baselines: The baselines include QA zero-shot prompting, few-shot prompting, and Modular Pluralism, alongside human lower and uniform upper bounds.QA prompting uses group information with survey questions and derives response distributions from first-token log-probabilities; few-shot prompting adds similar training-question distributions.
E Wave, Group-level Opinion Matching
Wave-level and group-level analyses evaluate Wasserstein distance across OpinionQA and SubPOP-Eval. Fine-tuned models show consistent distribution-matching improvements across diverse subpopulations, survey waves, and waves whose topics were absent from training.
- Evaluation scope: Fine-tuned models consistently improve distribution matching across diverse subpopulations and survey waves.The evaluation reports group-level and wave-level averaged Wasserstein distance for OpinionQA and SubPOP-Eval.
- Intergroup disagreement: The heatmaps encode target human groups on the y-axis and source human or simulated groups on the x-axis for intergroup disagreement.The figures cover age and political ideology, including both seen and unseen ideology subpopulations.
- Wave-level matching: OpinionQA wave-level evaluation compares zero-shot QA prompting with SubPOP fine-tuning, including waves whose topics are not covered in training.The table reports per-wave Wasserstein distance for each base model before and after fine-tuning.
- Group-level matching: Per-group evaluation reports Wasserstein distance before and after SubPOP fine-tuning for OpinionQA and SubPOP-Eval.The comparisons are organized by base model, with the zero-shot QA condition serving as the base reference.
- Prompting examples: The prompting examples condition survey questions on demographic or ideological group information before obtaining response distributions.The QA format uses first-token log-probabilities, while few-shot prompts add response distributions from similar questions.