PaperlayerView exact revision on arXiv

Source-linked AI summary

How Well Do LLMs Simulate Survey Responses Following a Breast Cancer Screening Intervention?

Kenneth Koh, Ryan Jak Yang Lim, Alessandro Sparacio, Peh Joo Ho, Mile Sikic, Borame L Dickens, Mikael Hartman, Jingmei Li

arXiv:2609.07141v1cs.SI

TL;DR

Singapore breast cancer screening uptake remains low, and existing approaches have limited ability to predict behavior under interventions. This study evaluates profile-grounded LLM agents for reproducing post-intervention survey distributions. Individual Profile agents reduced NWD relative to zero-shot prompting, with performance comparable to Aggregate Profile agents.

  • Problem

    With Singapore screening uptake at 35.2%, existing descriptive methods identify groups less likely to screen but have limited ability to predict behavior under interventions.

    Source: Introduction · p. 3

  • Method

    The study evaluates whether aggregate- and individual-profile LLM agents can predict population-level post-intervention responses, comparing them with alternative baselines.

    Source: Comparators for prediction · p. 4, §4.3 · p. 14

  • Results

    58.5% (Gemma 4 E4B) and 39.2% (Qwen3.5 9B) lower NWD than zero-shot prompting for Individual Profile agents, with performance comparable to Aggregate Profile agents.

    Source: Results · p. 8

  • Takeaways & Limitations

    LLM agents may help explore responses to public-health interventions but are not substitutes for human participants; local knowledge may help address uneven performance.

    Source: §4.3 · p. 14

    LLM-generated response distributions may have lower variance than human data, so simulated treatment effects require validation before applying standard human-data inferential approaches.

    Source: §4.2 · p. 14

Abstract

from arXiv · show

Collecting survey data is laborious and limited by privacy constraints. Large language models (LLMs) have shown promise as predictive social simulations. It is unclear whether they can replicate population-level response distributions before and after a healthcare intervention. Using information derived from 4125 women aged 35-59 years, we evaluate whether agents informed solely by pre-intervention profile information can reproduce post-intervention response distributions. Groups of LLM agents (n=50) were created with Gemma 4 E4B and Qwen3.5 9B; conditions ranged from zero-shot prompting to agent profiles enriched with aggregate or individual-level demographic characteristics and pre-intervention questionnaire responses. We compared predicted and observed response distributions with Total Variation Distance (TVD) and Normalized Wasserstein Distance (NWD). Across both LLMs, profile-based agents improved distributional accuracy relative to zero-shot and random baselines. Nevertheless, direct sampling of 50 real participants remained more accurate. Prediction errors were also higher among participants aged 55-59 years and those living in private property. Errors also varied by question theme and LLM model, with the highest errors observed for cancer fatalism and post intervention attitudes toward genetics. Sensitivity analyses showed that performance was influenced by prompt template changes and temperature hyperparameter. Our results show the potential of LLM-based agents to model behavioral responses to interventions in silico. However, profiles containing additional information beyond demographics did not consistently outperform simpler ones. Certain cultural constructs and population groups also remain inadequately represented by the LLM models evaluated. Future work may include building behaviorally grounded and locally validated virtual populations.

Background … Generation of agents using profiles of individuals

The study evaluates whether LLM agents built from pre-intervention profiles can reproduce survey responses in a Singapore breast cancer screening study. It contrasts profiles generated from aggregate demographic distributions with profiles sampled from individuals.

Keywords

  • The stated keywords include LLMs, breast cancer screening, simulation, surveys, policy intervention, synthetic data, and Singapore.The keyword list also includes AI.Source: Keywords · p. 3

Introduction

  • Despite national programmes and subsidies, breast cancer screening uptake among women in Singapore remains 35.2%, and descriptive methods have limited ability to predict behaviour under interventions.Regression analysis, surveys, and focus groups can identify sub-populations less likely to screen.Source: Introduction · p. 3
  • Prior work reported up to 86% survey-response accuracy from interview-informed agents and a correlation of r=0.85 between GPT-4 treatment-effect predictions and actual outcomes.The literature also raises concerns about flattened within-group heterogeneity and challenges involving diversity, bias, sycophancy, alienness, and generalization.Source: Introduction · p. 3
  • The study assesses whether LLM simulations can reproduce responses in a Singapore personalized risk-based breast cancer screening study, across question themes, demographic subgroups, and input-parameter settings.The motivation is that prior LLM survey-simulation research focused mainly on North American contexts, while evidence from other settings suggests difficulty capturing population heterogeneity.Source: Introduction · p. 3

2.1 Simulation setup

  • The simulation framework uses profiles built from demographics and pre-intervention responses for 4,125 women aged 35–59 years.The cohort began with 4,129 women; four with null responses were excluded, while 17 who developed breast cancer within six months were retained.Source: §2.1 · p. 4

Generation of agents using aggregated information

  • Aggregate-profile agents independently draw each demographic variable from its aggregate distribution, preserving marginal distributions but not correlations.Iterative Proportional Fitting was not used because no starting seed matrix preserving associations was available.Source: Generation of agents using aggregated… · p. 4

Generation of agents using profiles of individuals

  • Individual-profile agents use demographic profiles sampled without replacement from the cohort, preserving correlations between demographic variables.The approach tests whether preserving actual combinations of traits benefits simulations compared with marginal distributions alone.Source: Generation of agents using profiles of… · p. 4

Comparators for prediction · Information given to agent profiles

The study compared LLM agent profiles with empirical and simple-response baselines, then simulated pre- and post-intervention questionnaires using staged BCRR information. Its design used soft probability aggregation and distributional metrics, with assumptions about profile construction and sampling variability.

Comparators for prediction

  • Agents were compared with uniform-random, zero-shot, age-only, minimal-profile, majority-response, n=50 sampling-variation, and 50/50 split-sample empirical-marginal baselines.The minimal profile contained age, ethnicity, and housing type; the split-sample baseline estimated question-specific distributions from one cohort half and evaluated them on the other.Source: Comparators for prediction · p. 4

Information given to agent profiles

  • Pre-intervention agents received only information available before baseline completion; post-intervention agents also received BCRR and, for aggregate or individual profiles, baseline responses.Baseline responses preceded intervention exposure and contained no post-intervention outcomes; questions came from the Breast Cancer Education Survey and Risk Report Feedback Survey in BREATHE.Source: Information given to agent profiles · pp. 4–5

2.2 Simulation of questionnaire responses

  • Agents answered 16 pre-intervention questions, received a BCRR, and then answered 20 post-intervention questions assessing report understanding, emotional reactions, and screening intentions.Individual-profile agents received their actual study risk report, while age-only, minimal, and aggregate profiles received reports based on age-stratified risk distributions; BCRR memory was retrieved before each post-intervention answer.Source: §2.2 · p. 5

2.3 Prompt template and parameters

  • Questions were presented sequentially in a stateless format, so agents answered from their current state without retaining context from earlier questions.This reduced computational load and the need to randomize question order, while avoiding simultaneous question presentation that could cause omissions.Source: §2.3 · p. 5
  • Soft probability aggregation asked each LLM for numeric probabilities across answer options, normalized the scores, and aggregated distributions rather than selecting only each agent’s most likely answer.The approach was intended to address limited diversity in LLM simulations; hard voting instead aggregates each agent’s argmax response.Source: §2.3 · pp. 5–6

2.4 Simulation assumptions

  • Aggregate-profile demographics were sampled independently, agents did not interact, and aggregate agents received age-stratified BCRRs, whereas individual-profile agents received reports based on age and actual risk.The study also assumed survey responses were valid proxies for underlying screening attitudes.Source: §2.4 · p. 6

2.5 Statistical analysis

  • Sampling variability for n=50 was estimated with 200 bootstrap resamples, comparing each sample’s response distribution with the population ground truth.This provided a reference for expected population-distribution error from sampling 50 individuals.Source: §2.5 · p. 6
  • TVD, JSD, and NWD measured disagreement on normalized 0–1 scales, with lower values indicating less distributional error; NWD accounted for ordinal response distance.NWD was computed only for ordinal questions and was the preferred metric when available.Source: §2.5 · pp. 6–7

Sensitivity and subgroup analyses

Profile-based agents achieved lower post-intervention NWD than random and zero-shot baselines, while errors varied across demographic subgroups, question themes, and models. Sensitivity analyses used matched runs and descriptive comparisons, but no sensitivity-analysis outcomes are reported in the supplied passages.

Sensitivity and subgroup analyses

  • Sensitivity comparisons held the simulated population constant across matched runs, varied one parameter, and reported mean metric changes descriptively across five runs per LLM.Subgroup NWD differences were assessed with a Linear Mixed Model and Holm-Bonferroni correction across five demographic categories.Source: Sensitivity and subgroup analyses · p. 7

Results

  • Before intervention, Qwen’s lowest TVD was 0.122 ± 0.0017 for Age-Profile agents, while Gemma’s was 0.157 ± 0.0035 for Individual Profile agents.Sampling 50 observed responses yielded TVD 0.052 ± 0.012; more detailed profiles were not consistently best across both models.Source: Results · p. 8
  • All profile-grounding methods increased reasoning cosine similarity over zero-shot, with aggregate profiles producing the largest increases for Gemma (β=0.077) and Qwen (β=0.040).Source: Results · p. 8
  • Individual Profile agents had the lowest post-intervention NWDs: 0.092 ± 0.005 for Gemma 4 E4B and 0.110 ± 0.006 for Qwen3.5 9B, outperforming random and zero-shot baselines.Aggregate Profile agents performed comparably, with NWDs of 0.093 ± 0.006 and 0.118 ± 0.003, respectively; repeated samples of 50 observed responses had NWD 0.037 ± 0.004.Source: Results · p. 8
  • Grounded agents’ average Likert responses shifted toward human responses but remained closer to the scale midpoint, including for agreement and strong disagreement items.Human agreement-item scores ranged 2.0–2.3 versus agent scores of 2.3–2.8; for strong-disagreement items, human scores were 3.7–3.9 versus agent scores of 3.1–3.5.Source: Results · p. 9
  • Adjusted subgroup analyses found higher errors for women aged 55-59 (β=0.02), never-employed women (β=0.06), women in private or other non-public housing (β=0.02), and widowed women (β=0.06).These estimates were exploratory; the widowed and never-employed subgroups had small sample sizes.Source: Results · p. 9
  • Pre-intervention cancer fatalism had the highest mean TVD for Gemma (0.367) and Qwen (0.187), while prevention and screening knowledge (0.128) and screening barriers (0.122) were lowest for Gemma, and risk perception (0.089) for Qwen.Source: Results · p. 9
  • Post-intervention errors peaked for positive emotional reactions (TVD 0.232) and genetics attitudes (0.231) in Gemma, and genetics attitudes (0.357) and report understanding (0.323) in Qwen.Source: Results · pp. 9–10

Soft probability aggregation versus single choice prompting · Sensitivity to prompt template · Sensitivity to temperature hyperparameter

Soft probability aggregation improved agreement with observed survey distributions, while changes to prompt template and temperature affected NWD and agent rankings. Profile-grounded agents remained more accurate than zero-shot agents under both sensitivity changes.

Soft probability aggregation versus single choice prompting

Sensitivity to prompt template

Sensitivity to temperature hyperparameter

  • On Qwen3.5 9B, changing temperature from 0.6 to 0.0 yielded mean Δ NWD from -0.004 to 0.014, a change of -3.3% to 10.5%.Source: Sensitivity to temperature… · p. 10
  • At temperatures 0.6 and 0.0, profile-grounded agents had mean NWD 0.1194 - 0.1359 versus 0.2215 for zero-shot agents; the best configuration shifted from Minimal to Aggregate Profile Agents.Source: Sensitivity to temperature… · p. 10

LLM differences · Relationship between errors and number of simulated agents

Gemma 4 E4B had lower NWD than Qwen3.5 9B for profile agents, while softprobability aggregation errors plateaued at approximately 50–100 agents.

LLM differences

  • Gemma 4 E4B had lower NWD than Qwen3.5 9B for Aggregate and Individual Profile agents, with longer inference times (~11000s vs ~6000s per run).The longer inference time was partially attributed to Gemma's longer responses to the same prompt.Source: LLM differences · p. 10

Relationship between errors and number of simulated agents

  • With softprobability aggregation, discrepancies between simulated and observed response distributions decreased rapidly as agent population size increased, plateauing at approximately 50–100 agents.The plateau was reported in Multimedia Appendix 2, Figure S7.Source: Relationship between errors and number… · p. 10

FUNDING STATEMENT

The study received institutional and programme funding, with additional individual support for MH and KK; funders had no role in the research or manuscript.

FUNDING STATEMENT

  • The study was funded by JurongHealth Fund, A*STAR, and the PRECISE CIP Fund.The JurongHealth Fund reference number was JHF-20-RE-003.Source: FUNDING STATEMENT · p. 15
  • MH received support from six named cancer research, university, and institutional funds.These were the JurongHealth Fund, PRECISE CIP Fund, SSHSPH-Res-Prog-BCPP, NUHSRO/2020/121/BCSPP/LOA, NCIS Centre Grant Programme (CGAug16M005), and Asian Breast Cancer Research Fund.Source: FUNDING STATEMENT · p. 15
  • Funders were not involved in study design, data collection, analysis, interpretation, or manuscript writing; KK received the A*STAR Research Internship Award.Source: FUNDING STATEMENT · p. 15

CONFLICTS OF INTEREST … Mean NWD (SD)

The simulation code and analysis materials are publicly available, with sample simulated inputs provided instead of restricted research data. Pre-intervention accuracy is evaluated with TVD and post-intervention accuracy with NWD; lower values indicate less error.

CODE AVAILABILITY

  • The simulation code and analyses are available in a public repository.The repository includes analysis scripts, prompt templates, experiment configurations, random seeds, software dependencies, and experiment-creation procedures.Source: CODE AVAILABILITY · p. 15
  • Sample simulated inputs are provided in place of the actual research data because data sharing is restricted.Source: CODE AVAILABILITY · p. 15

Mean NWD (SD)

  • Figure 1 presents the simulation design.Source: Figure 1 · p. 22
  • Figure 2 illustrates soft probability aggregation versus hard voting.Source: Figure 2 · p. 23
  • Pre-intervention accuracy is measured with TVD and post-intervention accuracy with NWD; lower values indicate lower error.Figure 3 presents survey distributional accuracy for both time points.Source: Mean NWD (SD) · p. 24, Figure 3 · p. 24
  • Figure 4 shows prediction errors by question theme before and after intervention for the Gemma 4 E4B Aggregate Profile Agent.Heatmap cells show means for TVD, JSD, and NWD with standard deviations across five simulation runs; color indicates relative error, increasing from green to red.Source: Figure 4 · p. 25

Multimedia Appendix 1. · Multimedia Appendix 2.

Multimedia Appendix 1 identifies the simulation inputs, while Multimedia Appendix 2 points to detailed experimental results and distributional similarity metrics for evaluating survey accuracy against human responses.

Multimedia Appendix 1.

  • Multimedia Appendix 1 identifies the simulation inputs: agent profile fields, questionnaires, and prompt templates.Source: Multimedia Appendix 1 · p. 26

Multimedia Appendix 2.

  • Multimedia Appendix 2 points to detailed tables and figures for experimental results and distributional similarity metrics for evaluating survey accuracy against human response distributions.Source: Multimedia Appendix 2 · p. 26

Multimedia Appendix 4.

This appendix documents agent profile inputs, prompts, missingness in post-intervention questionnaire items, and supplementary accuracy, subgroup, aggregation, sensitivity, and response-pattern analyses. In the listed comparisons, soft probability aggregation shows lower JSD and TVD than hard selection.

Simulation Inputs

  • The input tables distinguish age-only, minimal, aggregate, and individual agent profiles, with separate pre- and post-intervention fields for the aggregate and individual profiles.Source: Multimedia Appendix 4 · pp. 28–31

Prompt Templates

  • The response prompt requires a JSON distribution covering every option, weights totaling 100, a highest-weight response, and a reasoning field.Agents are instructed to answer as a woman in Singapore using the described circumstances and intuitive reasoning.Source: Table S6 · p. 34, Multimedia Appendix 4 · p. 35
  • A separate memory prompt asks agents to rate the usefulness of experienced information from 0 to 100 and briefly describe how a person would recall it.The output is a JSON dictionary containing each item's importance score and memory summary.Source: Multimedia Appendix 4 · p. 35

Post-Intervention Questionnaire

  • 394 (8.7%) missingness was recorded for six post-intervention items spanning general well-being, report understanding, and positive emotional reactions.The items covered health, quality of life, understanding of risk classification and recommendations, confidence in the classification, and feeling relieved or happy.Source: Multimedia Appendix 4 · p. 37

Pre-Intervention Survey Distributional Accuracy

  • NWD was not provided for the pre-intervention survey because only 3 of 16 questions were ordinal and the rest were nominal.This limitation is noted for both the Gemma and Qwen pre-intervention accuracy tables.Source: Table S1 · p. 40, Table S2 · p. 41

Qwen3.5 9B Subgroup Difference

  • For Qwen3.5 9B subgroup analysis, widowed participants had NWD 0.208 and an estimated subgroup difference of 0.0996 (95% CI 0.08–0.12; p<0.001).The analysis generated 200 agents per LLM across two runs; individual-profile agents enabled comparison of subgroup-specific NWDs with corresponding subgroup ground-truth distributions.Source: Multimedia Appendix 4 · p. 46, Table S7 · p. 46

Probability Aggregation vs Single Choice Selection

  • For JSD, soft aggregation lowered error versus hard selection for Gemma zero-shot (0.461±0.02 vs 0.765±0.004) and Qwen zero-shot (0.387±0.004 vs 0.489±0.008).For Gemma, age-only agents scored 0.267±0.01 vs 0.335±0.02 and minimal-profile agents 0.270±0.01 vs 0.346±0.009; for Qwen, they scored 0.298±0.015 vs 0.388±0.02 and 0.290±0.0077 vs 0.389±0.013, respectively.Source: Multimedia Appendix 4 · p. 48, Table S9 · p. 48
  • For TVD, soft aggregation lowered error versus hard selection for Gemma zero-shot (0.487±0.02 vs 0.758±0.006) and Qwen zero-shot (0.396±0.0046 vs 0.486±0.011).For Gemma, age-only agents scored 0.243±0.01 vs 0.300±0.021 and minimal-profile agents 0.249±0.01 vs 0.317±0.012; for Qwen, they scored 0.285±0.016 vs 0.376±0.027 and 0.277±0.0075 vs 0.379±0.017, respectively.Source: Multimedia Appendix 4 · p. 48, Table S9 · p. 48

Effect of Number of Agents on Distributional Error

  • The agent-count analysis sampled cohorts of 2–200 agents with replacement across 200 iterations, but did not constrain samples to match the true population distribution.The authors describe the plot as consistent with informal observations, not a replacement for actual experimentation.Source: Multimedia Appendix 4 · p. 60, Figure S7 · p. 60

Extra: Radar Plots for pre and post intervention prediction errors by Question Theme

This extra section presents TVD-only radar plots of pre- and post-intervention prediction errors by question theme for Qwen3.5-9B and Gemma 4 E4B, using a visualization style similar to the earlier heatmaps.

Multimedia Appendix 3. · A. Distributional Similarity Metrics

The appendix defines TVD and JSD for comparing response distributions, and normalized Wasserstein Distance for ordinal responses. It describes their bounds, scale properties, and the assumption used to normalize NWD.

A. Distributional Similarity Metrics

B. Linear Mixed Model Equations Linear Mixed Model Equation (For Subgroups):

This section specifies mixed-effects models for subgroup NWD and for grounding-method differences in reasoning length and cosine similarity, using question-level random intercepts and method-specific comparisons with zero-shot agents where applicable.

Linear Mixed Model Equation (For Subgroups)

  • For subgroup NWD, β0 represents reference-subgroup NWD and βd the mean difference for subgroup d, with question- and run-level random intercepts and residual error.The question and run intercepts are normally distributed around zero; each demographic variable was fitted with a separate model.Source: B. Linear Mixed Model Equations Linear… · p. 63

Linear Mixed Model Equation (For Reasoning Length)

  • For reasoning length, the model compares average word counts for grounding methods with zero-shot agents, using a question-specific random intercept and residual error.The intercept represents the zero-shot agents’ average number of words, and the method coefficient estimates the difference from that reference.Source: B. Linear Mixed Model Equations Linear… · p. 63

Linear Mixed Model Equation (For Cosine Similarity)

  • For cosine similarity, the model compares grounding methods with zero-shot agents, using a question-specific random intercept and residual error.The intercept represents zero-shot agents’ cosine similarity, and the method coefficient estimates the difference from that reference.Source: B. Linear Mixed Model Equations Linear… · p. 63
Loading 2609.07141v1…