PaperlayerView exact revision on arXiv

Source-linked AI summary

Questioning the Survey Responses of Large Language Models

Ricardo Dominguez-Olmedo, Moritz Hardt, Celestine Mendler-Dünner

arXiv:2306.07951v4cs.CL

TL;DR

It is unclear whether prompting LLMs with established human surveys produces interpretable responses that resemble human population data. The authors evaluate 43 language models on 25 questions from the 2019 ACS using multiple-choice probability prompting and record probability distributions over answer choices. Models exhibit ordering biases and, after adjustment, consistently high-entropy responses that do not resemble Census populations across model sizes or fine-tuning conditions.

  • Problem

    It is unclear whether prompting LLMs with established human surveys produces interpretable responses that resemble human population data.

    Source: §1 · p. 1

  • Method

    The authors evaluate 43 language models on 25 questions from the 2019 ACS using multiple-choice probability prompting and record probability distributions over answer choices.

    Source: §1.1 · p. 2, §2 · p. 4

  • Results

    Models exhibit ordering biases and, after adjustment, consistently high-entropy responses that do not resemble Census populations across model sizes or fine-tuning conditions.

    Source: §1.1 · p. 2, §6 · p. 12

  • Takeaways & Limitations

    Survey-derived alignment often reflects how close a subgroup’s aggregate answers are to uniform rather than an inherent property of the model or its training data.

    Source: §1.1 · p. 3, §6 · p. 12

    The findings concern a popular multiple-choice survey-prompting methodology and caution against expecting robust human-population insights from it.

    Source: §3.1 · p. 7, §6 · p. 12

Abstract

from arXiv · show

Surveys have recently gained popularity as a tool to study large language models. By comparing survey responses of models to those of human reference populations, researchers aim to infer the demographics, political opinions, or values best represented by current language models. In this work, we critically examine this methodology on the basis of the well-established American Community Survey by the U.S. Census Bureau. Evaluating 43 different language models using de-facto standard prompting methodologies, we establish two dominant patterns. First, models' responses are governed by ordering and labeling biases, for example, towards survey responses labeled with the letter "A". Second, when adjusting for these systematic biases through randomized answer ordering, models across the board trend towards uniformly random survey responses, irrespective of model size or pre-training data. As a result, in contrast to conjectures from prior work, survey-derived alignment measures often permit a simple explanation: models consistently appear to better represent subgroups whose aggregate statistics are closest to uniform for any survey under consideration.

1 Introduction

The paper questions whether LLM survey responses can validly represent human populations, despite surveys’ appeal for studying model biases and demographics. Using the ACS, it finds systematic response biases and adjusted distributions that remain unlike human population data.

1 Introduction

  • Survey-based LLM research aims to compare aggregate model responses with human reference populations and potentially simulate otherwise hard-to-reach groups.Prior work applies surveys to study biases and to emulate human populations, with possible uses in social science and costly data collection.Source: §1 · p. 1

1.1 Our work

  • The study uses 25 questions from the 2019 ACS to evaluate 43 language models against U.S. Census responses.It asks what model survey responses reveal about LLMs and their training data, and whether those responses resemble U.S. population data.Source: §1.1 · p. 2
  • Models’ apparent response entropy varies with model size because systematic ordering and labeling biases confound their answer distributions.After randomized choice ordering, responses become similar across models and tend toward balanced answer distributions.Source: §1.1 · p. 2
  • After adjustment, models are no closer to Census or state populations than to a uniform baseline and retain higher entropy than human populations.This pattern persists after instruction-tuning, with a possible reduction in Census divergence only for models larger than 70 billion parameters.Source: §1.1 · pp. 2–3
  • Across ACS and other surveys, apparent alignment with demographic subgroups is explained by subgroup response entropy rather than model data or training procedure.The authors present this as context for prior survey-based studies of LLM biases.Source: §1.1 · p. 3
  • The paper cautions that LLM survey responses should not be treated as faithful representations of human populations or as robust alignment evidence.Human-survey design quality does not automatically transfer to logits produced by LLMs.Source: §1.1 · p. 3

1.2 Related work

  • Survey-response evaluation differs from question answering because it analyzes probability distributions over choices rather than only the most likely completion.This methodology has been used for demographic, transnational opinion, and moral-belief surveys.Source: §1.2 · p. 3

2 Surveying language models

The paper applies a standard next-token-probability method to multiple-choice ACS questions, comparing model distributions with Census and uniform reference distributions. It evaluates entropy, KL divergence, prompt variants, randomized answer orderings, and a broad set of models and questions.

Prompting

  • For each multiple-choice prompt, the method extracts next-token probabilities for answer labels and renormalizes them into a distribution over survey answers.The prompt lists the question and labeled choices before an answer field.Source: §2 · p. 4
  • The standard prompting setup is supplemented by prompt ablations, whose takeaways remain robust, because the goal is to examine popular scientific practices rather than engineer better prompts.The prompt style follows established question-answering and social-science practices.Source: §2 · p. 4

Survey questions and models

  • The study surveys 43 models ranging from 110M to 175B parameters on 25 representative multiple-choice questions from the 2019 ACS.Questions cover demographics, education, healthcare, disability, family, veteran status, employment, and income.Source: §2 · p. 4

Reference data & evaluation

  • Model response distributions are compared with 2019 ACS microdata from around 3.2 million U.S. individuals and with a uniform answer distribution.The evaluation uses normalized entropy for variation and KL divergence for distributional dissimilarity.Source: §2 · p. 5
  • KL divergence from the uniform distribution corresponds to an entropy difference, yielding KL(Cq ∥Uq) = kq(1 −H(Cq)) for normalized entropy.This links the uniform reference directly to the entropy-based evaluation.Source: §2 · p. 5

Randomized choice ordering

  • Randomized choice ordering permutes answers across alphabetically ordered labels to separate preference for an answer from preference for a label.The expected distribution under uniform ordering is treated as the adjusted survey response; up to 5000 permutations are used for questions with more than six answers, and up to 50 for OpenAI models.Source: §2 · p. 5

3 Systematic biases in models’ survey responses

Naively prompted models show response entropies driven more by model size and systematic ordering or labeling biases than by the ACS question. After randomized answer ordering, these biases are adjusted, revealing response patterns that are largely uniform, while instruction-tuned models show greater entropy variation.

3 Systematic biases in models’ survey responses

  • Models’ response entropy increases log-linearly with model size and varies less across ACS questions than the U.S. census entropy.This pattern holds across the ACS questions, with model-to-model variation exceeding question-to-question variation.Source: §3 · p. 6, Figure 2 · p. 6
  • Models’ answer distributions appear largely independent of the survey question, suggesting that systematic biases drive observed differences across models.The observed response distributions differ more across models than across questions.Source: §3 · p. 6

3.1 Testing for systematic biases: A-bias

  • All models exhibit substantial A-bias, with models of a few billion parameters or fewer showing particularly strong bias and tending toward mono answers.Instruction- or RLHF-tuned models have A-bias similar to base models.Source: §3.1 · p. 7, Figure 3 · p. 7
  • Models rely on spurious answer ordering and labeling signals rather than semantic meaning, and A-bias persists even when the position of “A” is randomized.These findings contrast with human primacy bias and align with evidence that multiple-choice survey responses may not consistently reflect free-form outputs.Source: §3.1 · p. 7
  • Systematic response biases make survey-derived inferences about the opinions or populations that LLMs represent sensitive to answer ordering.Reversing the SEX answer order could make GPT-2 appear to represent substantially different female-to-male population proportions.Source: §3.1 · p. 7
  • Randomizing answer order and averaging responses across orderings removes the model-size entropy trend, leaving base-model response entropy near 1 across questions.The adjustment is designed to average model responses over all possible answer orderings.Source: Figure 4 · p. 8

4 Inspecting adjusted responses

After randomized choice ordering, model responses become approximately uniform across questions and models, while instruction tuning does not make them reliably closer to census populations.

4 Inspecting adjusted responses

  • Models’ adjusted responses have normalized entropy approximately 1, regardless of model size or survey question.This removes the log-linear size trend seen before adjustment and supports systematic ordering biases as the main source of earlier variation.Source: §4 · p. 8

4.2 Comparing model responses to the U.S. census

  • Figure 5 compares KL divergence from the overall census, state populations, and a balanced uniform baseline, with smaller values indicating greater similarity.The analysis uses average KL divergence because the ACS questions are predominantly nominal rather than ordinal.Source: Figure 5 · p. 9, §4.2 · p. 9

4.1 Effect of instruction tuning

  • Instruction-tuned models show more entropy variation across questions, but their responses generally remain higher-entropy than census responses.Despite deviating more from uniform, they do not consequently become closer to U.S. census responses.Source: §4.1 · p. 9

4.2 Comparing model responses to the U.S. census

  • Models are by far more similar to the uniform baseline than to any human reference population.This comparison covers the overall U.S. census and individual U.S. states.Source: §4.2 · p. 9, Figure 5 · p. 9
  • Only 6 of 25 questions for GPT-4 are closer to the census than to the uniform baseline, and all models remain significantly closer to uniform overall.Instruction tuning appears to reduce divergence only for larger models, without changing the overall pattern.Source: §4.2 · p. 9

5 Implications for survey-based alignment metrics

Survey-derived alignment tracks the entropy of reference subgroups more consistently than model-specific properties, and this pattern extends from the ACS to additional opinion surveys.

5 Implications for survey-based alignment metrics

  • Models appear more aligned with high-entropy subpopulations, while adjustment removes model-order differences attributable to systematic biases.The entropy trend persists before adjustment, but relative ordering among models changes substantially after adjustment.Source: §5 · p. 10
  • Across models and surveys, alignment measures are more informative about reference-population entropy than about the language models being evaluated.Pre-training data, instruction tuning, and reinforcement learning with human feedback appear to have little impact on which population is best represented.Source: §5 · pp. 10–11

5.1 Beyond the ACS

  • On ATP, GAS, and WVS surveys covering thousands of questions and roughly 60 subgroups or national populations, alignment again shows a linear relation with response entropy after adjustment.The study adopts alignment measures from Santurkar et al. [2023] and Durmus et al. [2023].Source: §5.1 · p. 11, Figure 7 · p. 11
  • Prior findings that models align with lower-income, moderate, and Protestant or Catholic groups [Santurkar et al., 2023] correspond to those groups having responses closest to uniform in the ATP surveys.Groups reported as poorly reflected, including people aged 65+ and widowed individuals, have responses furthest from uniform; apparent RLHF shifts may also reflect systematic biases.Source: §5.1 · p. 11

6 Conclusion

The study finds that multiple-choice survey responses from LLMs are dominated by ordering biases and remain unlike human population data after adjustment. Consequently, survey-derived alignment often reflects subgroup response entropy rather than inherent model or training-data properties.

6 Conclusion

  • Across 43 models, adjusted responses retain consistently high entropy instead of the natural question-level entropy variation found in U.S. census data.This remains true regardless of model size or fine-tuning with human preferences.Source: §6 · p. 12
  • Survey-derived alignment appears to favor subgroups with balanced answers, making it a proxy for subgroup entropy rather than an inherent model or training-data property.This pattern holds across all models and surveys considered.Source: §6 · p. 12
  • The paper cautions that human-survey quality does not seamlessly transfer to LLM logits and calls for more research on methods that measure inherent model biases.The authors frame their contribution as a critique of multiple-choice prompting methodology, not a rejection of surveys as a benchmarking resource.Source: §6 · p. 12

A Experimental details

The experiments use 2019 ACS microdata and a curated multiple-choice questionnaire, with several transformations needed to represent non-multiple-choice or complex ACS variables.

A Experimental details

  • The study uses 2019 American Community Survey Public Use Microdata Sample files downloaded through the Folktables package.The files come directly from the U.S. Census Bureau.Source: A Experimental details · p. 17
  • The experiments use publicly available model weights and require approximately 1500 NVIDIA A100 GPU hours plus about $200 for OpenAI model queries.The code and visualization notebooks are released for replication.Source: A Experimental details · p. 17

A.1 Survey questionnaire used

  • The questionnaire contains 25 ACS variables, with HISP reduced to Yes/No and RAC1P represented by nine single-choice race categories.These changes replace the ACS’s original free-response, multiple-selection, or more extensive coding structures.Source: §A.1 · p. 17
  • Employment aggregates ESR and COW are formulated as questions, while AGE, WKWN, WKHP, and PINCP integer responses are converted to multiple choice by binning.The questionnaire therefore operationalizes several ACS variables rather than reproducing every original response format.Source: §A.1 · p. 17

B Detailed experimental results

Before correcting for answer-order effects, model response entropy increases with model size and is similar across survey questions, often approaching uniform-distribution entropy in larger models.

B Detailed experimental results

B.1 Model responses across questions before adjusting for A-bias

  • Entropy increases log-linearly with model size, often matching uniform-distribution entropy for larger models across survey questions.This pattern is independent of the questions’ response distributions in the U.S. Census.Source: §B.1 · p. 17, Figure 8 · p. 18

C Ordering bias: further experiments

The experiments separate positioning from labeling bias and show that letter labels can shift the dominant bias, while removing strong labels exposes greater position bias.

C Ordering bias: further experiments

  • Randomized experiments collect 3,000 responses under joint choice-order and letter-label randomization to estimate position and labeling distributions.Under no bias, both expected distributions would be uniform; chi-square tests assess deviations from uniformity with high power.Source: C Ordering bias: further experiments · p. 18, §C.1 · pp. 18–19

C.1 Disentangling ordering bias into positioning bias and labeling bias

  • Models show significant positioning and labeling bias for most survey questions, with labeling more prevalent and more persistent across model sizes.Position bias decreases more strongly with model size, whereas labeling bias remains prevalent; A-bias exceeds first-choice bias especially in smaller models.Source: §C.1 · p. 19, Figure 9 · p. 19, Figure 10 · p. 19

C.2 I-bias

  • Replacing B with the more frequent sentence-initial letter I causes small models to exhibit I-bias rather than A-bias.The result supports the hypothesis that written-English letter frequency contributes to labeling bias.Source: §C.2 · p. 19, Figure 11 · p. 20

C.3 Using letters with similar frequency in written English

  • Using labels with similar written-English frequencies somewhat reduces significant labeling bias but substantially increases position bias across model sizes.Without a strong label signal such as A or I, models exhibit higher choice-ordering bias.Source: §C.3 · pp. 19–20, Figure 12 · p. 20

D Prompt ablations

Prompt ablations vary system instructions, question framing, answer formats, and sequential context to test whether the reported survey effects depend on prompting choices.

D Prompt ablations

  • Prompt ablations produce results consistent with the main-text findings across the tested prompt variations.These ablations are conducted for models with publicly available weights.Source: D Prompt ablations · p. 20

D.1 System rompt used for GPT-3.5 and GPT-4

  • OpenAI models require a single-letter system prompt because the API exposes only the top five logits, which may omit answer-choice labels.Source: §D.1 · p. 20

D.2 Individual survey questions

  • Individual-question ablations vary direct instructions and answer-format examples for selecting one listed option.Source: §D.2 · p. 21
  • Additional-context ablations identify the questions as American Community Survey items or ask respondents to answer as U.S. households.Source: §D.2 · p. 21
  • Another ablation reformulates questions in the second person rather than the third person.For example, it changes “What is this person’s sex?” to “What is your sex?”.Source: §D.2 · p. 21
  • Chat-style and interview-style prompts reframe individual survey questions through Human/Assistant or Interviewer/Me exchanges.Source: §D.2 · p. 21

D.3 Sequential generation

  • Sequential-generation ablations retain previous questions and answers in context, using either question-answer or interview-style formats.These prompts test alternatives to summarizing prior responses with bullet points.Source: §D.3 · pp. 21–22

E Results for ATP, GAS, WVS, and ANES surveys

Across additional surveys, model responses remain closer to uniformly random baselines than to human demographic or national-population subgroups, while sequentially generated datasets remain highly distinguishable from census data.

E Results for ATP, GAS, WVS, and ANES surveys

E.1 ATP surveys

  • All models are more aligned with a uniformly random baseline than with any ATP demographic subgroup.The analysis uses the alignment metric and demographic groups considered by Santurkar et al. [2023].Source: §E.1 · p. 22, Figure 13 · p. 23

E.2 GAS and WVS surveys

  • Models’ GAS and WVS responses are more similar to a uniformly random baseline than to any demographic subgroup or country population.The comparison uses the countries and similarity metric considered by Durmus et al. [2023].Source: §E.2 · p. 22, Figure 14 · p. 23

E.3 Relative alignment for ATP and GAS/WVS surveys

  • Relative alignment with ATP and GAS/WVS subpopulations is highly correlated with the entropy of those subpopulations’ responses.Source: §E.3 · p. 24

E.4 ANES survey

F Sequential sampling of responses

F.2 The discriminator test

  • Discriminator classifiers achieve over 90% accuracy distinguishing sequentially generated model data from census data, implying empirical total-variation distance above 0.9.The result indicates little similarity between the generated and surveyed population datasets.Source: §F.2 · p. 25

F.3 Contrast with silicon samples

The study contrasts its survey-based model responses with silicon sampling and finds that models’ generated data markedly differ from human survey data. This distinction limits conclusions about synthetic respondents drawn from feature-imputation performance.

F.3 Contrast with silicon samples

  • Silicon sampling constructs synthetic individuals by predicting each answer from that respondent’s answers to all other survey questions.Source: §F.3 · p. 25
  • The reproduction uses the 2016 ANES questionnaire and interview-style prompt to test whether the findings transfer beyond the ACS setting.Source: §F.3 · p. 26
  • Classifiers discriminate model-generated data from ANES data with accuracy > 99%, showing marked differences between model and human responses.Source: §F.3 · p. 26
  • Reasonable feature-imputation performance does not imply that LLMs can generate synthetic respondents resembling human survey populations.Source: §F.3 · p. 26
  • The authors therefore urge caution when using LLMs to emulate human populations without human data, while related work reports calibration problems for similar imputation tasks.Source: §F.3 · p. 26
  • The experiment instead prompts models to sequentially complete blank questionnaires using only their own preceding responses, unlike Argyle et al.’s use of actual human responses.Source: §F.3 · pp. 25–26

Related papers

Loading 2306.07951v4…