PaperlayerView exact revision on arXiv

Source-linked AI summary

Synthetic Data in Marketing Research: How to Evaluate and When to Trust

Oded Netzer, Rajan Sambandam

arXiv:2609.13995v2cs.AI

TL;DR

The paper argues that the debate over whether LLMs can replace human respondents misses the more useful questions of what accuracy a decision requires and whether existing data can answer a question. It offers a framework and initial empirical evidence; reviewed findings show off-the-shelf estimates can be unreliable, while same-category grounding can improve estimates for adjacent attributes.

  • Problem

    The yes-or-no question of whether LLMs can replace human respondents leaves unresolved which accuracy standards decisions require and which questions existing data can inform.

    Source: §5 · p. 26

  • Method

    The paper develops a conceptual framework for those questions and presents initial empirical evidence.

    Source: §5 · p. 26

  • Results

    Off-the-shelf LLM responses can be unreliable, while same-category human-data grounding can improve estimates for a single adjacent attribute but does not transfer across categories or recover heterogeneity.

    Source: §5 · p. 26

  • Takeaways & Limitations

    Researchers should ask what accuracy a decision requires and whether questions fall within the informational reach of data already in hand.

    Source: §5 · p. 26

    The paper describes synthetic-data accuracy in marketing research as uncertain, with researchers often unable to tell when a given result is trustworthy.

    Source: §1 · p. 6

Abstract

from arXiv · show

Debate over synthetic data in marketing research has polarized between claims that large language models (LLMs) make human respondents obsolete and calls to avoid them entirely. We argue that both positions obscure the more useful question: not whether synthetic respondents work, but when. Building on Brand, Israeli, and Ngwe (2026), we make three contributions. First, we distinguish three types of synthetic data (ungrounded LLM responses, segment-level personas, and individual-level digital twins) and map each to the decisions it can support. Second, we develop a taxonomy of four families of accuracy measures and suggest that the wide range of reported twin accuracy, from near-perfect to near-chance, largely reflects differences in what is being measured rather than in method quality. Aggregate measures often perform well even when little information is supplied to the LLM, and can mask a complete absence of respondent-level differentiation. Third, we introduce the forgotten question problem, in which a question is omitted from a fielded study, as a setting for twin-based augmentation of existing data. We propose an ex-ante answerability diagnostic that requires no ground truth: the R^2 of a random forest predicting twin outputs from the data used to construct the twins. Across 108 attitude questions from a nationally representative survey (N = 3,063), screening at R^2 above 0.7 raises the mean twin-human individual-level correlation by 15% and reduces the share of poorly answered questions from 25.9% to 4.3%. Embedding similarity and experienced-researcher judgment provide correlated but weaker screens.

1. Introduction

Synthetic data should be evaluated by when and how it is used, rather than treated as an all-or-nothing substitute for human respondents. The introduction distinguishes forms of synthetic data, reviews mixed evidence on LLM performance, and highlights the promise and limits of grounding models in human survey data.

1. Introduction

  • The debate’s yes-or-no framing obscures the more useful question of which decisions synthetic data can support and when it should be used selectively.The authors note that businesses may value the 10–20% of study results that are genuinely new insights or surprises, not only predictable findings.Source: §1 · p. 2
  • The paper distinguishes ungrounded LLM responses, segment-level personas, and individual-level digital twins as three increasingly granular forms of synthetic data.Ungrounded responses are the most aggregated; individual twins are based on and validated against individual humans.Source: §1 · p. 4
  • Ungrounded responses may help with real-time business decisions made with little data and with early-stage or qualitative research.They can complement, augment, or replace such decision-making.Source: §1 · p. 4
  • Segment-level personas offer finer testing but require a human segmentation study and are difficult to validate against individuals; twins offer individual-level data for aggregation and heterogeneity.Personas can support simulated claims testing across segments and qualitative communication; twin performance varies with how they are built and what they are asked.Source: §1 · pp. 4–5
  • GPT’s out-of-the-box willingness-to-pay estimates sometimes missed the mark, including a threefold fluoride overestimate, wrong-signed preferences, and substantial variation across model versions.Without a human benchmark, researchers could not know in advance which estimates to trust.Source: §1 · p. 5
  • Fine-tuning on same-category human survey data improved estimates for withheld attributes, but laptop data did not transfer to tablets and did not recover demographic heterogeneity.Toothpaste data corrected willingness-to-pay signs for novel flavors; laptop data brought the projector estimate closer to the human benchmark. The paper identifies within-category extrapolation as the clearest promise and asks how adjacent a new feature must be.Source: §1 · pp. 6–7

3. How Do We Measure Digital Twins’ "Accuracy"?

Reported digital-twin accuracy ranges widely because studies measure different targets, not necessarily because their methods differ in quality. The section organizes those measures into four families and connects them to the decisions they can support.

3. How Do We Measure Digital Twins’ "Accuracy"?

  • Published accuracy discrepancies largely reflect differences in what is measured rather than differences in method quality, motivating a four-family taxonomy tied to research decisions.The targets range from population-level summaries to respondent-specific predictions.Source: §3 · pp. 7–8

3.1 A taxonomy of accuracy measures

  • Family 1 correlates question-level summaries across items, while Family 2 compares aggregate human and synthetic response distributions question by question; neither requires matching respondents and twins.Family 2 includes means, dispersion, distributional distances, and aggregate estimates such as willingness-to-pay.Source: §3.1 · p. 8
  • Family 3 compares each respondent’s answer with their own twin’s answer for each question, measuring matched individual-level accuracy.Measures can include continuous-response error or discrete-response hit rates.Source: §3.1 · pp. 8–9
  • Family 4a correlates each person’s true and twin answers across questions, whereas Family 4b correlates human and twin answers across people within each question.Family 4a can remain high from good aggregate responses without heterogeneity; Family 4b tests whether twins sort people correctly, even with shifted levels.Source: §3.1 · p. 9
  • Individual-level accuracy (Families 3 and 4b) dominates aggregate measures because correct individual responses determine every aggregate, whereas matching aggregates alone says nothing about individual accuracy.Family 4a accuracy can come from what the model knows about the population, not from respondent-specific information.Source: §3.1 · p. 10

3.2 Aspects to consider across accuracy measures

Accuracy measures can look strong when population-level structure or easy baselines do much of the work. Interpreting them requires appropriate floors and ceilings, attention to leakage and respondent heterogeneity, and clarity about how close the question is to the twin’s grounding data.

3.2 Aspects to consider across accuracy measures

  • A cross-question correlation can approach 1 even when every synthetic respondent receives the same answer, and correlation can conceal magnitude errors.One study reports treatment-effect correlation of 0.85 despite predictions roughly twice the true effect sizes, while within-experiment-arm correlations are 0.39.Source: §3.2 · pp. 10–11
  • On bounded scales, constant or modal-answer predictions can produce high hit rates without individuation, so compare against an information-free floor.Balanced accuracy gives each response option equal weight; a modal-only predictor scores 1/K, the same as uniform random guessing.Source: §3.2 · p. 11
  • 0.748 record accuracy, 0.734 with no individual information, and 0.746 with 14 demographic variables show how little respondent-level information may change the headline statistic.The demographics-only result is statistically indistinguishable from the full persona.Source: §3.2 · p. 11
  • Normalizing against test–retest reliability alone can make 65.7% raw accuracy appear as 82.6% relative accuracy, while hiding distance from the floor.Reporting both floor and ceiling corrections preserves both reference points.Source: §3.2 · p. 12
  • Holdout scores may overstate performance on new questions when familiar scales or survey items occur in model training data.The passage flags greater leakage concern for aggregate or across-question measures than for individual measures.Source: §3.2 · p. 12
  • Twins are systematically under-dispersed and accuracy varies across respondents, with higher accuracy for more educated, higher-income, and politically moderate people.Mean accuracy does not reveal either pattern; heterogeneity distributions should also be reported.Source: §3.2 · p. 12
  • Accuracy is a property of the twin–question pair: a question’s distance from the grounding data helps determine whether its answer is informative.A wrong-person twin compared with the correct-person twin estimates the contribution of person-specific information.Source: §3.2 · pp. 12–13

3.3 Matching the measure to the decision: what industry actually needs

The appropriate accuracy standard depends on the decision: population reads, heterogeneity claims, and individual-level uses demand different measures. A validation result is meaningful only for the tier and estimand it actually tests.

3.3 Matching the measure to the decision: what industry actually needs

  • Three commercial-use tiers match accuracy requirements to the decision, rather than asking for a single measure of whether twins are accurate.The tiers cover population-level reads, heterogeneity, and individual-level uses.Source: §3.3 · p. 13

Tier 1: Population-level reads

  • Population-level reads require Family 2 validity against matched human benchmarks, but aggregate users should check dispersion as well as means.A share simulator using compressed preferences can misforecast even when average willingness-to-pay is correct.Source: Tier 1: Population-level reads · p. 13

Tier 2: Heterogeneity

  • Segmentation, targeting, positioning, product-line design, and differentiated pricing require Family 4b sorting plus Family 2 comparisons within segments.Differential accuracy across respondent groups is particularly relevant to these uses.Source: Tier 2: Heterogeneity · p. 13

Tier 3: Individual-level uses

  • Personalization, respondent-level imputation, and longitudinal simulation require Family 3 and/or Family 4b measures.Individual-level measures determine whether twins can be trusted beyond benchmarked questions, extended to segments, or used to replace human sampling.Source: Tier 3: Individual-level uses · p. 14
  • When matched individual data exist, all four accuracy families can be computed from the same data at essentially no extra cost.A lower-tier measure validates only its own estimand and can mislead when used to support individual- or segment-level claims.Source: Tier 3: Individual-level uses · p. 14

3.4 Recommendations

The recommendations call for decision-matched measurement, explicit baselines, and validation that reflects the distance between questions and grounding data. Published accuracy levels often differ because they measure different targets and use different baselines.

3.4 Recommendations

  • State the decision and estimand first, report every supported accuracy family, and do not use a coarser family to justify finer individual- or segment-level claims.Report levels and sorting together when the data support both.Source: §3.4 · p. 14
  • Compare accuracy with information-free floors and test–retest ceilings, and use a wrong-person twin comparison to assess respondent-specific information.For multiple-choice items, the recommendations call for balanced accuracy.Source: §3.4 · pp. 14–15
  • Validate on novel stimuli, state the question’s distance from the twin’s training data, and report respondent-group heterogeneity and aggregate dispersion.The recommendations also caution against conventional standard errors for Tier 1 and Tier 2 synthetic estimates.Source: §3.4 · p. 15
  • 90%+ accuracy often measures aggregate item responses, ~75% often reflects a floor-inflated statistic, and weaker scores often test individuals against proper baselines on novel questions.The authors ask future academic and commercial work to state which bar was met and which decisions it supports.Source: §3.4 · p. 15

4.1 The forgotten question: Using LLMs to augment existing studies

A forgotten question is an important item omitted from a study, discovered after fieldwork when re-fielding is costly and may reduce comparability. The section considers using twins grounded in respondents’ existing survey records to answer that single missing question, and asks when such answers are dependable.

4.1 The forgotten question: Using LLMs to augment existing studies

  • Re-fielding to answer an omitted question is costly and slow, risks panel fatigue or attrition, and can create comparability problems between waves.The forgotten question may be an attribute left out of a conjoint design or a choice driver noticed after reviewing early results.Source: §4.1 · pp. 15–16
  • Prior findings report unreliable off-the-shelf LLM estimates, better alignment when grounded in same-category human data, and worse alignment when grounding transfers across categories.BIN found that grounding corrected some withheld-attribute WTP estimates, while laptop data applied to tablets worsened alignment.Source: §4.1 · p. 16
  • For each respondent, the twin can use the complete existing survey record to answer one additional question about the study’s population and category.This extends the human study by a single increment rather than simulating an entire study.Source: §4.1 · p. 16
  • The key questions are whether twins can accurately augment a study with one related question and which forgotten questions are answerable from its data.Questions close to the available information may be supported, while questions too far from it may elicit generic-prior answers that look plausible in aggregate but lack respondent-specific information.Source: §4.1 · pp. 16–17
  • Because a forgotten question has no human benchmark, researchers need an ex-ante diagnostic to identify when twin answers may fail.The section proposes and empirically tests a statistical approach to that diagnostic.Source: §4.1 · p. 17

4.2 Screening questions using the twins’ own fit

The section proposes screening twin answers by testing whether respondent-level inputs explain their variation. Its random-forest R² diagnostic needs no human answers to the forgotten question, but indicates answerability rather than correctness.

4.2 Screening questions using the twins’ own fit

  • A random forest predicts each twin’s forgotten-question answer from the respondent’s full existing survey record, yielding out-of-bag R² and variable importances.The forest accommodates nonlinear relationships; alternative models can also implement the approach.Source: §4.2 · pp. 17–18
  • The diagnostic uses twin inputs and outputs alone, so it can be computed without ground-truth responses to the forgotten question.This differs from hybrid approaches that use human labels for the target question.Source: §4.2 · p. 18
  • R² measures how much variation in twin predictions is systematically explained by respondent data: high values indicate data-dependent answers, while low values suggest reliance on general knowledge.A high R² is necessary for respondent-specific answers, not proof that the answers are correct.Source: §4.2 · p. 18
  • R² is a necessary but not sufficient condition for trust because a twin may use respondent data yet still predict the outcome incorrectly.Variable importances can also indicate which existing questions help answer the additional question.Source: §4.2 · p. 18
  • The screen is a go/no-go decision about whether to ask a question of the twin, not a choice between estimators; general knowledge may suffice for aggregate goals or low-variance questions.For binary or multiple-choice questions, the proposed alternative is a classification forest scored with a prevalence-robust measure such as out-of-bag balanced accuracy.Source: §4.2 · pp. 17–19

4.3 Empirical application

The empirical application examines how well a random-forest R² fit measure, embedding similarity, and experienced-researcher judgment screen forgotten survey questions for accurate digital-twin answers. It focuses on identifying when twin-based extrapolation is suitable and when new human data are needed.

4.3 Empirical application

  • Across 108 held-out questions, adding same-category attitudes raised twin-human correlation from r = 0.08 to r = 0.57 and lowered individual MAE from 2.01 to 1.46, but aggregate MAE rose from 0.39 to 0.48.The study emphasizes individual-level accuracy because its use case depends on respondent-specific information, not just topline estimates.Source: §4.3 · p. 20, Table 1 · p. 21
  • With demographics plus attitudes, mean random-forest R² was 0.65 and correlated with individual-level correlation at r = 0.61, aggregate MAE at r = −0.22, and individual MAE at r = −0.37.With demographics alone, average R² was 0.14; two high-R², low-correlation outliers concerned safety and diversity, equity, and inclusion.Source: §4.3 · p. 21

4.3.1 Using the R² metric as a screen

  • With the R² screen at > 0.711, 43% of candidate questions are retained; average correlation rises to 0.65, and the share of poorly answered questions falls to 4.3%.Source: §4.3.1 · p. 22

4.3.2 Replacing the R² screening with Question-Embedding Similarity

  • Maximal embedding similarity correlated with twin-human correlation at r = 0.53 and random-forest R² at r = 0.31; a >0.75 screen retained 38.9% of questions, with correlation 0.63 and 14.3% poorly answered.Embedding gains were smaller than R² screening except for individual MAE; unlike R², similarity uses question wording alone and indicates question-level answerability rather than respondent-level differentiation.Source: §4.3.2 · p. 24, Table 3 · p. 24

4.3.3 Replacing the R² screening with practitioner judgment

  • Sixteen TRC Insights marketing researchers rated 18 forgotten questions, and their average inter-rater correlation across questions was 0.816.Researchers saw each question and the twin’s demographics plus five remaining category-specific attitude answers, then rated confidence in recovering respondents’ answers on a 0–100 scale.Source: §4.3.3 · p. 25
  • Across 18 questions, expert confidence correlated with twins’ realized accuracy at r = 0.77 (p < 0.001) and with random forest R² at r = 0.87 (p < 0.001).The authors interpret this as evidence that human and statistical screens largely agree on which questions are answerable.Source: §4.3.3 · p. 25
  • Confidence thresholds of 55–65 retain 61%–83% of candidate questions, raise mean twin–human correlation from 0.60 to 0.65–0.67, and reduce poorly answered questions from 27.8% to 7%–13%.The estimates are imprecise because they are based on 18 questions and should be treated with caution.Source: §4.3.3 · p. 25, Table 4 · p. 25
  • For forgotten questions, firms can screen answerability with experienced researchers, the R² diagnostic, or both, using twins for questions that pass and flagging failures for new human data.This preserves human data first and LLM extrapolation second, while offering a lower-cost, faster alternative to re-fielding for screened-in questions.Source: §4.3.3 · p. 26

5. General Discussion

The useful question is which accuracy standard a decision requires and whether that question falls within the data already available. The discussion presents synthetic data as most trustworthy when the question is close to the twin's grounding data, while emphasizing that human respondents remain necessary.

5. General Discussion

  • Human data from the same population and category can improve accuracy on a single adjacent attribute, but does not transfer across categories or recover heterogeneity.The authors identify small distance between the question and the twin's grounding data as the relevant condition for trust.Source: §5 · p. 26

5.1 Beyond the Forgotten Question: The Forgotten Step

  • The forgotten-question setting concerns an item omitted from an already-fielded study, and the authors extend the same logic to forgotten preparatory steps.Those steps can restore learning around the final study without replacing it.Source: §5.1 · pp. 26–27
  • Commercial studies often omit preparatory steps such as qualitative exploration, pretesting, and data-derived attribute selection, even though those steps would improve the final study.Budget and time constraints usually leave commercial research with one step, occasionally two.Source: §5.1 · p. 27
  • Synthetic data can support low-stakes, high-iteration pretesting, including screening claims, bounding price ranges, checking attributes, and pressure-testing questionnaires before a human study.Directionally useful pretests need not be precise because the subsequent human study can catch errors before they reach business decisions.Source: §5.1 · p. 27

5.2 Heterogeneity and Segment-Level Uses

  • Fine-tuned models can approximate population means while failing to recover differences across income, gender, and political groups, sometimes inverting them.The discussion warns that differential accuracy can be especially problematic for targeting or positioning, where segment comparisons matter.Source: §5.2 · p. 28
  • In the demographics-only condition, mean individual-level twin-human correlation was 0.08, no question exceeded 0.5, and aggregate MAE was 0.39, slightly better than with demographics plus attitudes.The contrast shows that an acceptable topline can coexist with little respondent-level differentiation.Source: §5.2 · p. 28
  • Recovering heterogeneity requires individual-level grounding data; the authors say there is no shortcut from a population prior to individual or segment differences.They caution against relying on segment-level personas for segment-level decisions when individual differences are at issue.Source: §5.2 · p. 28

5.3 The Role of Human Respondents

  • Digital twins compress what people have already revealed and cannot originate preferences or provide an independent channel for new preferences to reach the system.The authors therefore argue that human respondents must remain involved, with people supplying surprises that neither researchers nor models knew before the study.Source: §5.3 · p. 29

A1. LLM Specification

The appendix specifies respondent-by-question digital-twin predictions using GPT-5.4-mini, with either demographic inputs alone or demographics plus other same-category attitude answers.

A1. LLM Specification

  • GPT-5.4-mini predicted each respondent's answers to 108 attitude questions one at a time, using that respondent's answers to other survey questions as prompt input.The input condition was either demographics alone or demographics plus other attitudes from the same category.Source: A1. LLM Specification · p. 32
  • Each prediction used low/minimal effort, a single call with no memory, and produced an answer within the requested 1–10 range.Limited testing at higher effort showed little improvement.Source: A1. LLM Specification · p. 32

A2. Sample prompt for the digital twin predictions

The sample prompt asks the model to answer a survey item as the respondent, with prior questions and answers supplied as context.

A2. Sample prompt for the digital twin predictions

A3. Random Forest Specification

The random forest specification uses default settings and out-of-bag validation to calculate R^2 for twin outputs.

A3. Random Forest Specification

  • The forest used 500 trees, mtry = p/3, sampling with replacement, and a sample size of 0.632 times the number of rows.The remaining rows were used for out-of-bag validation.Source: A3. Random Forest Specification · p. 32
  • The default terminal-node size was 5, and the random seed was left at its default setting.Source: A3. Random Forest Specification · p. 33
  • Out-of-bag R^2 was calculated cumulatively across trees, with each observation predicted only by trees for which it was held out.The authors tested several random seeds and report consistent R^2 values.Source: A3. Random Forest Specification · p. 33
Loading 2609.13995v2…