PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMs as survey respondents: Comparison to simple semantic baselines

Esko Lehtonen, Tommi Buder-Gröndahl

arXiv:2610.29645v1cs.CL

TL;DR

It remains unclear why LLMs mimic human responses, including whether they use inferences based on internalized knowledge. The study compares three LLMs and two embedding-similarity baselines predicting held-out responses across two surveys and varying shares of known answers. LLMs generally outperformed the baselines, except at 20% known responses in LEV, and their advantage and response-rate sensitivity varied across surveys.

  • Problem

    It remains unclear why LLMs mimic human responses, including whether they use inferences based on internalized knowledge.

    Source: §1 · p. 1

  • Method

    The study compares three LLMs and two embedding-similarity baselines predicting held-out responses across two surveys and varying shares of known answers.

    Source: §1 · p. 1, §3.2 · p. 3

  • Results

    LLMs generally outperformed the baselines, except at 20% known responses in LEV, and their advantage and response-rate sensitivity varied across surveys.

    Source: §1 · p. 1, §5 · pp. 3–4

  • Takeaways & Limitations

    LLM augmentation capacity varies across surveys and response rates, and simpler semantic-similarity metrics can match LLM performance in some conditions.

    Source: §6 · p. 4

    The study covers three English-dominant models and two English-language surveys in specific domains, so broader conclusions require evaluations across more domains, models, languages, and demographic groups.

    Source: Limitations · p. 5

Abstract

from arXiv · show

Large language models (LLMs) have been explored as surrogates for human survey respondents. A more promising application is data augmentation, where LLMs predict new responses from partial human data. However, the semantic similarity between questionnaire items is a known source of correlation between answers. It is thus unclear whether LLMs actually infer meaningful relationships between question-response pairs, or use surface-level comparison between questions as a heuristic. To assess this, we evaluated how three LLMs can predict responses to held-out items in two surveys measuring attitudes toward automated or light-electric vehicles, based on varying proportions of known responses. Performance was compared to two baselines, where new responses were derived by maximizing embedding-similarity to a prior response. While LLMs mostly predicted human responses better than baselines, their performance consistently declined with a lower share of known responses, even aligning with the baseline in one survey. Our results show that LLMs' capacities for augmenting replies varied across surveys, and can revert to the same levels as simpler embedding similarity baselines when less data is available.

1 Introduction

The paper asks whether LLMs predict survey responses through meaningful question–response inference or rely on question semantics as a surface-level heuristic. It compares three LLMs with embedding-similarity baselines across two surveys and different shares of known answers.

1 Introduction

  • LLM surrogates have been proposed for generating new responses or augmenting existing survey data with related questions.Their potential appeal includes reducing the costs of conducting surveys.Source: §1 · p. 1
  • The study asks whether LLMs infer relationships between question–response pairs or use surface-level question comparison.The concern arises because survey responses may be predictable from semantic similarity between questions.Source: §1 · p. 1
  • The comparison uses three LLMs and two sentence-embedding baselines across AV and LEV surveys, varying known human responses from approximately 20% to 80%.The surveys measure attitudes toward automated vehicles and light-electric vehicles.Source: §1 · p. 1
  • LLMs generally aligned with human responses better than the baselines, but their performance was lower for LEV and matched the baselines there at a 20% known-response rate.The results motivate examining whether LLM performance reflects access to contextual and memorized information or simpler semantic similarity.Source: §1 · p. 1

2 Background

Prior work explores LLMs as survey respondents and reports both alignment improvements and persistent concerns. The paper situates its comparison within evidence that question semantics can predict survey responses.

2.1 LLMs as survey respondents

  • Prior LLM survey studies report response variation, ordering biases, human–LLM discrepancies, and alignment changes from fine-tuning or prompting.Reported prompting effects are mixed: some studies find improvements, while Morocho et al. (2026) report limited and partly negative effects.Source: §2.1 · pp. 1–2
  • Response-conditioned LLMs are presented as a way to augment existing datasets by inferring novel answers, rather than replacing human participants.LLMs have also been explored for missing-value imputation.Source: §2.1 · p. 2

2.2 Semantic predictability of responses

  • The semantic theory of survey response holds that question similarity can predict human answers; prior work reports substantial prediction of response variance and patterns.Arnulf et al. (2014) report explaining 60%–86% of variance, and Arnulf et al. (2024) contend similarity could predict response patterns with 80%–90% accuracy.Source: §2.2 · p. 2
  • A prior automated-vehicle study used Sentence-BERT to select the most embedding-similar prior response and captured 40% of between-variable covariance.The method provides an embedding-based baseline for the present comparison.Source: §2.2 · p. 2

3 Methods

The study tests response-conditioned imputation on two human survey datasets, using three known-response shares. It compares LLM-generated answers with embedding-based baselines and assesses alignment, covariance recovery, and factor structure.

3.1 Data

  • AV uses 22 questions from 500 respondents, while LEV uses 21 questions and eight additional items from 426 respondents after exclusions.AV measures six factors; LEV measures four factors and is not publicly available online.Source: §3.1 · p. 2
  • For each survey, the researchers masked answers at 20%, 50%, or 80% known, randomized question order, and used identical inputs across models.Answers were masked from the end of each participant’s response sequence.Source: §3.1 · p. 2

3.2 Response generation

  • Three LLMs generated answers to masked questions from known question–answer pairs, while Semagent baselines greedily selected the prior response with greatest embedding similarity.The baselines used SBERT or Gemini embeddings and were initialized with responses from known pairs.Source: §3.2 · pp. 2–3

3.3 Analysis

  • Human–model alignment was measured with weighted Cohen’s κ, while covariance recovery used matrix-distance and shape measures and factor recovery used confirmatory factor analysis.Confidence intervals and κ contrasts were bootstrapped, with respondent-level clustering and Bonferroni correction for multiple comparisons.Source: §3.3 · p. 3

4 Results

LLMs generally aligned with human responses better than the embedding baselines, though the advantage depended on the survey and known-response share. More known answers improved alignment and covariance preservation, while factor recovery varied by response share.

4.1 Alignment with human data

  • LLMs always outperformed the best Semagent in AV; in LEV, Gemini outperformed baselines at 50% and 80% known responses, and GPT at 50%.In AV, Gemini produced more aligned responses than SBERT at 50% and 80%; otherwise that baseline difference was not statistically significant.Source: §4.1 · p. 3

4.2 Known response rates

  • LLMs were more aligned at 50% than 20% known responses across both surveys; improvement from 50% to 80% was significant only for Gemini in LEV.Semagent alignment improved from 20% to 50% only in AV.Source: §4.2 · p. 3

4.3 Covariance and factor structure

  • Increasing known responses improved covariance preservation; at 20%, LLMs had an advantage over Semagents in L3Pilot, while otherwise they showed no strong improvement.The reported improvement concerns both covariance-matrix shape and magnitude.Source: §4.3 · p. 3
  • LLM-imputed datasets recovered the human factor structure with 50%–80% known answers; at 20%, only Mistral converged, and Semagents never converged.Generated data also showed systematic item-level biases.Source: §4.3 · p. 3

5 Discussion

LLMs generally outperformed semantic baselines, but the size of the advantage varied by survey and known-response rate. The LEV results at 20% known responses were a notable exception, and the authors suggest survey access and training-data overlap may influence the contrast.

5 Discussion

  • Except for LEV at 20% known responses, all LLMs outperformed both semantic baselines across experiment variants.The LLM advantage was greater in AV; in LEV, performance was closer to the baselines.Source: §5 · pp. 3–4
  • With 50%–80% known response rates, LLM and human data aligned in factor structure.Source: §5 · p. 4
  • LEV at 20% known responses was an outlier: all models performed poorly, and the usual distinction between LLMs and semantic baselines disappeared.The authors suggest AV's open access and potentially greater overlap with LLM training data may have influenced the survey contrast; LEV is not open access.Source: §5 · p. 4
  • The authors suggest LLM performance may depend on access to both contextual and memorized information.For LEV at 20%, they propose that LLMs may have lacked memorized question-answer pairs and reverted to generic semantic information, as the baselines do.Source: §5 · p. 4

6 Conclusions

The authors conclude that LLM response-augmentation performance varies across surveys and known-response rates, and does not always exceed simpler semantic baselines. They argue that embedding-similarity baselines should be included in future evaluations of LLM survey surrogates.

6 Conclusions

  • LLM response-augmentation capacity varies across surveys and known-response rates.Embedding-similarity comparisons show that LLMs do not always outperform simpler semantic metrics.Source: §6 · p. 4
  • The authors recommend including embedding-similarity baselines in future evaluations because survey responses are substantially predictable from question semantics.They note that this comparison has been rare in prior work on LLM surrogates.Source: §6 · p. 4

Limitations

The evaluation covers three English-dominant LLMs, two semantic baselines, and surveys with English questions about specific domains.

Limitations

  • Broader conclusions require evaluations across more survey domains, model architectures, fine-tuning and prompting regimes, languages, and sociodemographic groups.This short paper tested three English-dominant LLMs and two semantic baselines on English questions in highly specific domains.Source: Limitations · p. 5

Ethical considerations

The authors report that using LLMs or semantic baselines for data augmentation may introduce systematic item-level biases.

Ethical considerations

  • LLM or semantic-baseline data augmentation may introduce systematic item-level biases that should be considered.The public survey data contained no personal data, and respondents were adults who consented to research use of their responses.Source: Ethical considerations · p. 5

Code and data availability

The AV dataset is openly available, while LEV dataset inquiries go to the first author; analysis code is planned for public release if the paper is accepted.

Information about use of AI assistants

The appendix reports AI assistance in code modification and debugging, examines prompt robustness, and documents model settings, the task prompt, and pairwise testing procedures.

Information about use of AI assistants

  • The first author used Gemini and Copilot CLI to modify and debug software code, and reviewed and edited the content.The author takes full responsibility for the published article.Source: Information about use of AI assistants · p. 5

A System prompt and its robustness

  • With 100 participants and 50% known answers, consistent prompt variants differed only slightly, while the inconsistent prompt produced negative κ values indicating disagreement with human answers.The results indicate robustness to small differences among consistent prompts and an effect of prompt choice on responses.Source: A System prompt and its robustness · p. 8

Appendix Table 2: LLM hyperparameters

B Example task prompt

  • The example task prompt gives historical question-answer pairs, asks for answers to masked questions, and constrains outputs to a five-point agreement scale in an ID-response format.Source: B Example task prompt · p. 10

C Pairwise tests

  • Pairwise κ differences used respondent-clustered bootstrap confidence intervals, with paired resampling for within-dataset comparisons and independent resampling between datasets.Intervals were bias-corrected and accelerated, with Bonferroni correction for multiple comparisons.Source: C Pairwise tests · p. 11

D Testing Semagent with different neighbourhood sizes

The appendix tests Semagent neighborhood sizes and reports their effects on agreement and covariance preservation, alongside factor-structure and item-level checks of imputed data.

D Testing Semagent with different neighbourhood sizes

Appendix Table 4: Semagent covariance preservation with different k values.

E Factor structure and item-level biases

Related papers

Loading 2610.29645v1…