Source-linked AI summary
When Persona Simulations Are Informative: Graph-Structured Signals for Pluralistic Opinion Sensing
Taehyeon An, Jaehyeong Park, Donghyuk Shin
TL;DR
Synthetic survey responses may vary without being meaningfully grounded in their personas. This paper introduces PCI, a graph-based diagnostic, and finds that a PCI-selected 10% subset improves latent construct recovery over response-stability and random selection.
Problem
The paper addresses how to distinguish persona-grounded response variation from arbitrary variation caused by model priors or sampling noise.
Method
PCI models personas as a similarity graph and uses Local Moran’s I to identify concordant response deviations among semantically similar personas without construct labels.
Results
CFI reaches .925 for the PCI-selected 10% subset, versus .877 for response stability and .880 for random selection.
Takeaways & Limitations
PCI’s observed construct-recovery gains support local spatial coherence as an internal signal for screening synthetic respondents.
Takeaways & Limitations
Internal construct recovery does not guarantee external population validity, so PCI should be combined with external calibration for representative polling.
Abstract
from arXiv · showhide
Persona-conditioned large language models (LLMs) are increasingly used to simulate survey responses across diverse domains. However, apparent response variation can reflect unconditioned model priors or token sampling noise rather than systematic persona conditioning. We argue that persona-conditioned variation is informative when semantically similar personas exhibit concordant response shifts. To operationalize this principle, we introduce Persona-Conditioned Informativeness (PCI), an unsupervised diagnostic metric that measures whether semantically similar personas deviate in concordant directions relative to item-level sample baselines. By modeling personas as a similarity graph, PCI uses Local Moran's I to quantify local spatial coherence and extract compact persona subsets without using construct labels. To evaluate PCI without external human benchmarks, we test its ability to recover established latent value structure using the 57-item Portrait Values Questionnaire-Revised (PVQ-RR). Confirmatory factor analysis (CFA) shows that a PCI-selected 10% subset substantially improves overall construct recovery relative to response-stability and random selection. These findings support PCI as a principled internal diagnostic for screening synthetic respondents in survey pipelines.
1 Introduction
Persona-conditioned LLM responses may appear varied without reflecting systematic persona effects, making coherence among semantically similar personas a central diagnostic target. PCI addresses this need by measuring concordant item-level deviations on a persona similarity graph and using the signal to screen synthetic respondents.
- Problem: Plausible response variation is not necessarily informative because LLMs can generate differentiated answers without systematic persona influence.The key question is whether persona conditioning produces structured response shifts rather than merely varied outputs.
- Motivation: Persona profiles are multidimensional, while individual survey items engage only particular profile aspects, motivating coherence among semantically similar personas.Relevant profile cues should produce concordant response shifts, whereas unrelated traits may provide little consistent guidance.
- Limitations: Existing aggregate-alignment and repeated-run consistency evaluations can miss unconditioned individual responses, item-specific persona effects, and settings without external human data.These limitations motivate an internal diagnostic that does not depend on human benchmarks.
- Method: PCI models personas as a similarity graph and uses Local Moran’s I to measure concordant deviations from each item’s sample-wide response baseline.It treats survey responses as graph signals and aggregates local spatial coherence across items.
- Result: ΔCFI = +.045, p = .002, supporting local spatial coherence as an internal signal for screening synthetic respondents.The result is reported for the PCI-based evaluation described in the supplied passages.
2 Persona-Conditioned Informativeness
PCI treats persona-conditioned responses as informative when personas vary around item baselines and semantically proximate personas show concordant deviations. It implements this principle with profile-based similarity graphs, local spatial coherence, and top-score subset selection.
- 2 Persona-Conditioned Informativeness: PCI rewards responses only when persona-specific variation around the item-level baseline coincides with concordant deviations among semantically proximate personas.These conditions distinguish shared conditioning from invariant consensus and isolated sampling variance.
- 2 Persona-Conditioned Informativeness: Complete narrative profiles are embedded as dense representations, then connected in a weighted cosine-similarity k-nearest-neighbor graph.Profiles include demographics, socioeconomic status, family structure, occupation, cultural identity, and preferences.
- 2 Persona-Conditioned Informativeness: Item-level mean-centering removes shared sample-wide response levels, leaving centered deviations that capture between-persona variation around each baseline.Monte Carlo means across generation trials reduce stochastic token variance before this centering step.
- 2 Persona-Conditioned Informativeness: Local Moran’s I measures whether each persona’s centered deviation is corroborated by semantically similar neighbors, with positive values indicating concordant local clustering.High–high and low–low neighborhoods are concordant, whereas high–low and low–high neighborhoods are discordant.
- 2 Persona-Conditioned Informativeness: PCI_p = max(I_pi, 0) discards discordant or isolated deviations, and the aggregated score selects the top 10% (n = 148) as the primary subset.The non-negative truncation conservatively requires neighborhood support across items.
3 Construct-Recovery Evaluation
The evaluation tests whether PCI-selected PVQ-RR responses recover established construct structure without using construct information during selection. Across global and construct-level metrics, PCI outperforms response-stability and random selection, with robustness across subset fractions and graph densities.
- Evaluation setup: CFA modeled 19 correlated value factors plus an orthogonal common response factor, evaluating relative construct recovery through IoQ_v=|Corr(C_v,η_v)| under an identical structural model.The common response factor controls for scale acquiescence.
- Sensitivity analyses: CFI remained .924–.925 between 30% and 50%-NN, stayed .910–.915 up to 70%-NN, and persona rankings were highly consistent (Spearman ρ∈[.961, .990]).These sensitivity results indicate robustness across graph-density specifications.
- Overall construct recovery: CFI was .925 for the 10% PCI subset versus .877 for response stability and .880 for the random-baseline mean, with PCI significantly exceeding random selection (adjusted p=.010).PCI also reduced SRMR from .106 to .086 (adjusted p=.010).
- Construct-level recovery: PCI achieved the best or tied-best IoQ for 16 of 19 constructs, averaging .893 versus .857 for random selection and .856 for response stability.Gains were particularly pronounced for power-dominance (.622 to .758) and security-personal (.699 to .811).
- Sensitivity analyses: PCI maintained superior CFI from 10% through 80% subset fractions and peaked at .927 with a 20% fraction (n=296).The result supports improved recovery across a range of selection budgets.
4 Discussion and Conclusion
PCI is presented as an unsupervised internal diagnostic for detecting locally aligned response deviations among semantically proximate personas without construct labels. The discussion proposes a lightweight screening workflow while emphasizing that internal construct recovery does not establish external population validity.
- Contribution: PCI quantifies local alignment among semantically proximate personas’ item-level response deviations without requiring construct labels.Observed construct-recovery gains support local spatial coherence as an internal signal of structured response variation.
- Practical workflow: PCI supports a three-stage workflow: auditing generations, selecting compact informative cohorts, and triaging costly human validation.The workflow targets weak persona-aligned structure, cost-effective exploratory surveys, and focused validation of items or subgroups.
- Practical workflow: PCI remains computationally lightweight because it is closed-form given fixed generations and graph construction, without requiring model training.
- Limitations: Internal construct recovery diagnoses coherent profile–response structure but does not guarantee external population validity, so external calibration remains essential for representative polling.The primary graph construction embeds full narrative profiles into a single global metric space, motivating further research.