PaperlayerView exact revision on arXiv

Source-linked AI summary

Silicon Samples Match the Answers but Not the Respondents

Yusuke Kumagae, Masahiro Kaneko

arXiv:2610.30900v1cs.CLcs.CY

TL;DR

Existing silicon-sampling evaluations measure answer accuracy, per-question distributions, or cross-question correlations, rather than whether generated respondents match the human respondent distribution. This paper introduces RCG to measure respondent coverage and finds that evaluated LLMs leave human respondents uncovered, with higher RCG than a Gaussian copula across tested settings.

  • Problem

    Existing silicon-sampling evaluations compare answers or question-level distributions and correlations, leaving respondent-level distributions unmeasured.

    Source: §1 · p. 1

  • Method

    The respondent coverage gap (RCG) measures each human profile’s distance to its nearest generated profile relative to the corresponding human-to-human distance.

    Source: §1 · p. 2

  • Results

    Across tested prompting and inference settings, every evaluated LLM has a higher RCG than the Gaussian copula, and none achieves human-level coverage on either survey.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Synthetic populations intended to stand in for human populations should be evaluated by whether their respondents, not only their answers, match.

    Source: §6 · p. 8

    The LLM findings cover only US respondents from WVS wave 7 and the 2024 ANES wave; other populations and settings require further experiments.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Silicon sampling generates survey respondents by having a large language model (LLM) answer a survey as a person with given demographic attributes, and the resulting synthetic populations are increasingly used in place of human ones. Existing evaluations focus on individual-answer accuracy, per-question answer distributions, and correlations across questions. None of these compares the respondents themselves, so a generator whose respondents have combinations of answers found in no human respondent can still score well if it reproduces every answer distribution. We propose the respondent coverage gap (RCG), which represents each respondent as a point defined by their complete answer profile and measures how far each human respondent is from the nearest generated one, relative to the distance between human samples. On two US social surveys, all six LLMs we evaluate leave human respondents uncovered, and under our main settings, they fall behind a baseline that samples each answer independently from the human marginals. Across the prompting and inference settings we test, every LLM has a higher RCG than a Gaussian copula fitted to the human data.

1 Introduction

Silicon sampling uses LLMs to create survey respondents from demographic personas, but existing evaluations compare answers rather than complete respondent profiles. The paper introduces the respondent coverage gap (RCG) and finds that evaluated LLMs leave human respondents uncovered and trail simple baselines under the main settings.

1 Introduction

  • Existing metrics assess individual answers, per-question distributions, or cross-question correlations, but none compares how respondents are distributed as complete answer profiles.Matching every question’s answer distribution can still produce generated profiles absent from the human population.Source: §1 · pp. 1–2
  • RCG measures each human profile’s distance to its nearest generated profile, normalized by the corresponding distance among human samples.A ratio of one means coverage comparable to another human sample; larger values mean the least-covered human respondents are farther from generated respondents.Source: §1 · p. 2
  • All six evaluated LLMs leave human respondents uncovered and, under the main settings, fall behind a baseline sampling answers independently from human marginals.Across tested settings, no LLM has a lower RCG than Copula, which additionally models pairwise dependence.Source: §1 · p. 2

2 Related Work

Prior silicon-sampling evaluations score individual answers, question-level distributions, or associations between question pairs. This paper instead evaluates generated and human populations as distributions of complete respondent profiles.

2.1 Silicon Sampling and Its Evaluation

  • Silicon-sampling evaluations include individual-answer prediction and marginal fidelity, but both score responses question by question rather than comparing respondents as a population.Prior work notes that accurate predictions can still misrepresent how attitudes covary across a sample.Source: §2.1 · pp. 2–3

2.2 Beyond Marginal Evaluation

  • Correlation-based methods compare human and generated inter-item association matrices, capturing dependence between questions without comparing individual respondents.Williams et al. (2026) defer individual-level structure to future work.Source: §2.2 · p. 3

2.3 Comparing Generated and Human Respondents as Distributions

  • Generated-sample metrics such as precision/recall and MAUVE compare samples using neighborhood membership or quantization, while this paper retains nearest-neighbor distances and focuses on undercovered human respondents.The paper distinguishes its focus from prior nearest-neighbor comparisons used for authenticity testing or privacy risk.Source: §2.3 · p. 3

3 Evaluation Framework

The framework treats each respondent’s complete answer vector as one sample and compares generated profiles with human profiles sharing the same attributes. Its RCG and OSR ratios quantify undercoverage and off-support generation relative to human-to-human distances.

3 Evaluation Framework

  • Each respondent is represented by a vector of answers across the survey’s questions, and generated and human profiles are compared as samples.The framework shifts evaluation from answers along the question axis to distributions at the respondent level.Source: §3 · p. 3

3.1 Conditional Generation

  • LLMs generate respondents conditioned on demographic attributes and answer questions sequentially, with earlier answers included in later-question prompts.Only a few earlier answers are sampled because using the full history makes prompt length grow as O(M^2); prior work often queries questions independently, randomizing answer pairings across respondents.Source: §3.1 · p. 3, §3.2 · p. 3

3.2 Respondent Coverage Gap

  • RCG compares the 95th-percentile human-to-generated nearest-neighbor distance with the same percentile among humans; OSR reverses the numerator direction while retaining the human reference.The 95th percentile reduces sensitivity to outliers, and equal human and generated sample sizes are used because nearest-neighbor distances shrink as sample size grows.Source: §3.2 · p. 4
  • RCG above one indicates undercovered humans, while OSR above one indicates generated profiles unusually far from the human sample; similar RCG values can reflect either off-support drift or contraction.The toy off-support and contraction examples have similar RCG values (2.17 and 2.06), but OSR values of 2.24 and 0.55, respectively.Source: §3.2 · p. 4, Figure 2 · p. 4
  • Unlike threshold-based coverage measures, the proposed ratios retain how far profiles lie from the human sample and do not require a neighborhood-size parameter k.The 95th-percentile aggregation choice remains, and its effect is examined in Appendix H.Source: §3.2 · p. 4

4 Evaluation Setup

The evaluation uses two US surveys, conditions generation on respondent attributes in cells, and tests varied prompts and models. It compares profiles using Gower distance and includes baselines fitted to human responses.

4.1 Datasets

  • The study uses WVS wave 7 and the 2024 ANES wave, sampling 100 questions from each survey and representing each respondent with 100 answers.The samples include 2,594 WVS respondents and 5,453 ANES respondents; WVS questions concern social, religious, and political values, while ANES questions concern vote choice and partisanship.Source: §4.1 · pp. 4–5

4.2 Conditioning Cells

  • Each conditioning cell fixes values of demographic attributes, retains cells with at least 100 human respondents and complete attributes, and pairs those humans with 100 generated respondents.The evaluation includes 730 WVS cells and 603 ANES cells across L = 0–8 conditioning attributes.Source: §4.2 · p. 5

4.3 Prompts

  • Prompts combine task instructions, a persona, earlier-answer examples, and the current question; instruction and persona formats, question order, and some answer-choice orders are randomized.The prompt uses up to three prior question–answer pairs as examples.Source: §4.3 · p. 5

4.4 Models

  • The evaluation tests two proprietary and four open-weight models, using vendor-default sampling settings and generally untuned default decoding for open-weight models.The proprietary models are gpt-5.4-nano and gemini-3.1-flash-lite; two higher-tier models are included in an ablation.Source: §4.4 · p. 5

4.5 Distance

  • Gower distance measures profile dissimilarity by averaging per-question distances and excluding questions missing from either profile.The metric is selected because the survey questions have mixed types.Source: §4.5 · p. 5

4.6 Baselines

  • The baselines include Marginals, which independently samples each answer from human marginals, and Copula, which couples estimated marginals through a shrinkage-estimated correlation matrix.Copula and the two-class latent-class baseline are fitted to the evaluation respondents, so they serve as references rather than direct competitors to the LLMs.Source: §4.6 · pp. 5–6

5 Results

Across two surveys, LLMs leave human respondents uncovered, and marginal calibration improves but does not restore human-level coverage. Existing aggregate metrics can miss these respondent-level failures, which persist across sample sizes and tested prompting and inference settings.

5.1 Main Results

  • All LLM RCG values exceed 1.0 and all three baselines on both datasets; human-reference RCG is 1.000–1.001.For every LLM, the 95% confidence interval for the RCG difference from Marginals excludes zero.Source: §5.1 · p. 6
  • Calibrated RCG ranges from 1.126 to 1.150 on WVS and 1.178 to 1.232 on ANES, below Marginals but above the human reference, Copula, and LCA.Calibration reduces RCG for all six LLMs on both datasets, but does not restore human-level coverage.Source: §5.1 · p. 6

5.2 Failure Modes

  • OSR separates LLM failure modes: three models contract, while three show off-support behavior on ANES; RCG alone does not reveal the distinction.The lower group has OSR below one on both datasets; the upper group has OSR above one on ANES.Source: §5.2 · p. 6

5.3 Prior Metrics Can Mask Coverage Failures

  • Calibrated LLMs have the lowest JSD, yet their RCG remains above Copula and LCA; Marginals nearly matches the human correlation ceiling while missing respondent coverage.On WVS, JSD values cluster at 0.08–0.11 while RCG ranges from 1.04 for Copula to 1.16 for Marginals; Marginals has ρ = 0.994 versus a 0.995 human split-half ceiling.Source: Figure 3 · p. 7, §5.3 · p. 7
  • The correlation-structure score is unchanged when answers are permuted separately within each question, even though this destroys within-respondent alignment.The authors note this is not a defect for measuring aggregate cross-cell structure, but the score cannot substitute for respondent-level evaluation.Source: §5.3 · p. 7

5.4 Number of Generated Respondents

  • At |Gc| = 10,000, LLM RCG remains above one, while Marginals reaches 1.003; extrapolation estimates 5 × 10^5 to 7 × 10^5 open-weight samples per cell for RCG = 1.The extrapolation is a scale estimate because the decrease per decade is already smaller in the second decade than the first.Source: Table 2 · p. 8, §5.4 · p. 8, §5.5 · p. 8

5.5 Robustness to Prompting, Inference Settings, and Model Tier

  • Across tested settings, full response history lowers RCG in all twelve model–dataset pairs, while temperature changes RCG by at most 0.033; no tested setting beats Copula.Higher reasoning effort and higher-tier models also reduce RCG in several comparisons, with some point estimates below Marginals but still above Copula and LCA.Source: §5.5 · p. 8

6 Conclusions

The paper proposes respondent-level evaluation of silicon sampling and finds that evaluated LLMs do not yet match human respondents. Existing per-question metrics can miss this gap.

6 Conclusions

  • The paper proposes the respondent coverage gap to evaluate silicon sampling at the respondent level.Source: §6 · p. 8
  • No evaluated LLM achieves human-level coverage on either survey; under the main settings, all trail the independent-answer Marginals baseline.Across tested temperatures, response histories, reasoning efforts, model tiers, and generated sample sizes, every LLM has a higher RCG than the Gaussian copula baseline at the same sample size.Source: §6 · p. 8
  • Per-question metrics miss respondent-level failures, so synthetic populations should be shown to match respondents, not only their answers, before standing in for human ones.Source: §6 · p. 8

Limitations

The LLM analysis covers US respondents in two surveys, and does not examine several alternative prompting or training approaches. The authors identify these as boundaries for interpreting and extending the findings.

Limitations

  • Findings that all six LLMs have RCG above one, with contraction and off-support patterns, are shown only for US WVS wave 7 and 2024 ANES respondents.Other domains, non-US populations, and multilingual settings require further experiments.Source: Limitations · p. 9
  • The evaluation uses off-the-shelf prompting and does not examine backstories, criteria-based prompting, Verbalized Sampling, or String Seed of Thought.Source: Limitations · p. 9
  • Training-based methods and representation-level interventions are also outside the evaluation; RCG can test whether their per-question improvements extend to individual respondents.Source: Limitations · p. 9

Ethical Considerations

This section considers how silicon sampling could flatten or caricature identity groups, and how RCG and OSR can flag off-support generation and contraction within evaluated questions and cells. It reports RCG across conditioning depths and generated sample sizes, describes complementary metrics, and gives a proof about correlation-score invariance under within-cell answer permutation.

Ethical Considerations

  • Silicon sampling could misportray or flatten identity groups or caricature them, potentially reinforcing stereotypes when synthetic populations replace human respondents.The authors used WVS wave 7 and ANES wave 2024 under terms requiring research-only use, no redistribution of data files, and no investigation of specific respondents; AI assistants helped polish English and experimental code, which authors reviewed and verified.Source: Ethical Considerations · p. 9

C Prompts; E LLM Settings and Computational Cost

  • Prompts combine an instruction, demographic persona, few-shot examples, and a question block, with prompt variants and attribute order randomized by respondent.The six LLMs use vendor defaults for sampling parameters and the lowest available reasoning effort; proprietary-model queries cost approximately $11,160, and open-weight runs used approximately 1,960 H200 GPU-hours.Source: C Prompts · p. 13, E LLM Settings and Computational Cost · p. 16

D Building Cells

  • The evaluation retains cells only if they contain at least 100 human respondents and no missing-code values, then pairs human answers with 100 generated answers and computes metrics per cell.Each cell also includes US residence as an attribute.Source: D Building Cells · pp. 15–16

F Gower Distance

  • Gower distance averages question-level distances over jointly answered items, treating rating scales as numeric, naturally ordered multiple-choice items as ordinal, and other multiple-choice items as nominal.Don’t-know codes, survey missing codes, and out-of-range answers are treated as missing; respondents with fewer than 10 valid answers are excluded.Source: F Gower Distance · p. 16

G Ablation Settings

  • Ablations vary API and open-weight model temperatures, reasoning effort for gpt-5.4-nano and gpt-oss-20b, and model tier by adding gpt-5.4 and gemini-pro.Unless varied, settings match the main experiment; the ablations use up to five cells per conditioning-attribute level, totaling 41 WVS and 37 ANES cells.Source: G Ablation Settings · p. 17

H Sensitivity to Percentile Choices

  • All six LLMs have cell-averaged RCG above one at every tested percentile, and every LLM exceeds Marginals in both datasets for q = 0.90, 0.95, 0.99, and 1.00.At q ≤ 0.75, only gemini-flash-lite, gemma-4-12B, and Llama-3.3-70B have lower RCG than Marginals; small q can overlook poorly covered human respondents.Source: H Sensitivity to Percentile Choices · p. 17, Table 5 · p. 18

I Baseline Fitting

  • Under Hold-out, every LLM still has higher RCG than every baseline except on ANES, where Llama-3.3-70B and Marginals have nearly equal RCGs, 1.273 and 1.272.On the 196 ANES cells, Llama-3.3-70B is 0.010 above Marginals under In-cell, and 0.028 above it across all 603 cells.Source: Table 7 · p. 18, I Baseline Fitting · pp. 18–19

K RCG by L

  • On WVS, all six LLMs have higher RCG than Marginals at every tested number of conditioning attributes, and all series remain above 1.Cells at different conditioning-attribute levels represent different subpopulations.Source: K RCG by L · p. 19
  • The gap to Marginals persists at every tested conditioning depth on WVS (L = 0–8) and ANES (L = 1–6), while shrinking cell counts prevent attributing rising RCG to added attributes.The reported cell count decreases from 2,594 at L = 0 to 114 at L = 8.Source: Figure 7 · p. 19, K RCG by L · p. 19

M Proof of Proposition 1

  • Within-cell permutation preserves each question’s marginal and mean, leaving the correlation-structure score unchanged; applying the same permutation to human respondents yields a score of one against the original data.Source: M Proof of Proposition 1 · pp. 20–21

N Number of Generated Respondents

O Full-History Ablation

Full response history improves RCG across model–dataset pairs but does not bring any mean below 1 or Copula. The calibration settings remap generated answers toward human answer distributions, using different transport choices for ordered and nominal questions.

O Full-History Ablation

  • Full response history lowers mean RCG in all 12 model–dataset pairs, but every mean remains above 1 and Copula.Only gemma-4-12B on WVS and Llama-3.3-70B on ANES have RCG significantly below Marginals.Source: O Full-History Ablation · p. 21

P Calibration Settings

  • For each question, calibration uses a transition matrix to remap generated answers independently across questions and respondents, leaving missing answers unchanged.The matrix is constrained to reproduce the human answer distribution for that question.Source: P Calibration Settings · p. 21
  • Ordered questions use quantile mapping to minimize |a-b|, while nominal questions use minimum-rewrite remapping to preserve matching-choice mass and reallocate excess.The paper treats these as optimal transport plans with costs |a-b| and 1[a != b], respectively.Source: P Calibration Settings · p. 21

Q Missing-Value Ablation

Matching generated respondents to human missingness patterns lowers RCG, but the six LLMs remain above Marginals. The analysis identifies missing-answer asymmetry as a possible source of RCG inflation.

Q Missing-Value Ablation

  • Human-to-human distances use fewer questions than human-to-generated distances because humans leave more answers missing, potentially inflating RCG.Fewer compared questions make distances more variable and nearest-neighbor distances smaller.Source: Q Missing-Value Ablation · pp. 21–22
  • Each generated respondent is masked using unanswered questions from a sampled human outside the evaluation sample; cells with exactly 100 human respondents use one-to-one mask assignments.The procedure tests sensitivity to mismatched missingness patterns.Source: Q Missing-Value Ablation · p. 22
  • Masking generated answers with human missingness patterns lowers RCG, but all six LLMs remain above Marginals.RCG falls by 0.03–0.05 on ANES and at most 0.014 on WVS, where human respondents leave fewer answers missing (1.1% versus 6.0%).Source: Q Missing-Value Ablation · p. 22

R Temperature Ablation

Temperature changes RCG little relative to OSR: RCG stays above Marginals at every tested temperature, while OSR can cross one for some models.

R Temperature Ablation

  • RCG varies by at most 0.033 for any model and stays above Marginals at every temperature.The largest range, 0.033, is for Qwen3.5-9B on WVS.Source: R Temperature Ablation · p. 22
  • On WVS, Qwen3.5-9B OSR rises from 0.812 at temperature 0.1 to 1.087 at 1.5, crossing one as temperature increases.OSR also crosses one on WVS for gpt-5.4-nano and gpt-oss-20b.Source: R Temperature Ablation · p. 22

S Reasoning Effort Ablation

Increasing reasoning effort generally lowers both RCG and OSR, with the largest RCG drops occurring when reasoning is first turned on. One model–dataset pair reaches RCG below Marginals at high effort.

S Reasoning Effort Ablation

  • Turning reasoning on for gpt-5.4-nano lowers RCG by 0.045 on WVS and 0.110 on ANES; raising effort further changes RCG by at most 0.031.At high effort, OSR drops by 0.073 to 0.169 from default across all four model–dataset pairs.Source: S Reasoning Effort Ablation · p. 22
  • RCG stays above Marginals except for gpt-5.4-nano on ANES, which equals Marginals at medium effort (1.266) and falls below it at high effort (1.253).Unparsable answers are treated as missing; removing five questions changes RCG and OSR by at most 0.008.Source: S Reasoning Effort Ablation · p. 22
  • The reported 95% confidence intervals for RCG differences from Marginals cover variation across attribute sets while keeping human and generated responses fixed.The intervals use 10,000 bootstrap resamples of conditioning-attribute sets.Source: Table 13 · p. 23

T Model Tier Ablation

Higher-tier models mostly lower RCG but shift toward contraction. The tier comparison shows lower RCG in three of four model–dataset pairs, with one increase for gemini-pro on WVS.

T Model Tier Ablation

Related papers

Loading 2610.30900v1…