PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Raw Transcripts: Structured Persona Extraction for LLM-Based Digital Twins

Iris Ye, Tianze Deng, Ozan Candogan

arXiv:2608.20344v1cs.CLcs.CY

TL;DR

The paper asks whether LLM-based digital-twin accuracy depends on how persona information is organized rather than how much information is supplied. It compares fixed and automatically discovered persona structures, finding that BDE helps on homogeneous tasks while task-specific discovery restores gains across heterogeneous studies. The main limitation is the small evaluation panels and uncertain informativeness of calibration data.

  • Problem

    Prior compression results suggest information volume is not the binding constraint, leaving open how persona information should be organized before simulation.

  • Method

    The paper compares unstructured summaries with a theory-grounded BDE representation and an LLM pipeline that iteratively discovers task-specific structures and extraction prompts.

  • Results

    Across 13 iteration-eligible Mega-Study sub-studies, auto-discovered personas improve accuracy by +1.91pp over raw transcripts, while fixed BDE improves Twin-2K-500 accuracy by +1.91pp over raw transcripts.

  • Takeaways & Limitations

    Persona structure should be matched to task families: try fixed BDE for homogeneous tasks and use per-task discovery for heterogeneous behavioral primitives.

  • Takeaways & Limitations

    The experiments use small paired panels, and the paper does not yet establish when calibration-50 data is informative enough to select a useful structure.

Abstract

from arXiv · show

LLM-based "digital twins" aim to simulate how an individual would behavein new environments or respond to novel questions, given some representation of that individual's prior responses. A common approach constructs this representation from survey transcripts or summaries responses. Prior work shows that compressing long transcripts into shorter LLM-generated summaries does not significantly reduce predictive accuracy, suggesting that information volume is not the primary bottleneck. In this work, we argue that the key limitation is instead structural:how persona information is organized before being provided to thesimulator model. We study this by comparing unstructured summaries with structured persona representations. First, we introduce a hand-craftedschema (BDE: Background, Decision procedure, Evaluation), grounded in consumer-behavior theory, and show that it improves predictive accuracy over raw transcripts by +1.91 percentage points on a homogeneous benchmark (Twin-2K-500), with similar gains on gpt-5.4-mini and Qwen3-8B as robustness checks. However, this fixed structure does not generalizeacross more heterogeneous tasks, where performance is statistically indistinguishable from the raw transcript baseline. To address this limitation, we propose an automatic structure-discovery pipeline in which an LLM iteratively proposes and refines task-specific persona structures and extraction prompts. On a benchmark of 13 diverse sub-studies, this approach restores performance, improving mean accuracy by +1.91 percentage points over the raw transcript baseline and eliminating significant losses observed with the fixed schema. Overall, our results suggest that the main constraint in LLM-based digital twins is not how much information is provided, but how it is structured -- and that the optimal structure depends on the task.

1 Introduction

The paper argues that persona structure, rather than information volume, is the key open design question for LLM-based digital twins. It introduces BDE and automatic task-specific structure discovery, finding that fixed structure helps homogeneous tasks but may not generalize.

  • 1 Introduction: Compressed summaries can match raw transcripts, leaving how prior information is organized before simulation as the open question.The paper frames this as a structural rather than informational bottleneck.
  • 1 Introduction: BDE separates persona information into identity, reasoning, and preference sub-profiles.The structure is named Background, Decision procedure, and Evaluation.
  • 1 Introduction: The same fixed BDE structure essentially ties the raw transcript on the heterogeneous Mega-Study aggregate.The Mega-Study contains 19 heterogeneous sub-studies.
  • 1 Introduction: Auto-discovery searches task-specific structures and lifts aggregate accuracy by +1.91pp over raw transcripts.The pipeline produces per-sub-study structures and extraction prompts, with additional gains over BDE and unstructured summaries.
  • 1 Introduction: The paper will release BDE prompts, the auto-discovery implementation, and 19 per-sub-study structures.These artifacts are intended as drop-in baselines for future persona-side prompt design.

2 Related Work

Related work motivates persona structure through research on digital twins, prompt architecture, behavioral theory, and internal persona axes. These strands support separating identity, reasoning, and preferences while adapting structure to task demands.

  • 2 Related Work: Twin-2K-500 established per-respondent prediction from rich survey transcripts, while Mega-Study broadened evaluation across 19 diverse sub-studies.The related work positions this paper within individual-level LLM simulation.
  • 2 Related Work: Prompt architecture can create model-consistent artifacts, while under- and over-specified prompts can induce omitted-variable bias and focalism.These findings motivate studying input form, not only input content.
  • 2 Related Work: Consumer-behavior theory distinguishes identity, reasoning procedure, and preferences as separate signals relevant to behavioral outcomes.The paper grounds these axes in established theories including the Engel–Kollat–Blackwell model, Behavioral Reasoning Theory, and preference research.
  • 2 Related Work: A free-form summary leaves these behavioral axes entangled, whereas BDE separates them upstream of the simulator.This provides the behavioral rationale for the fixed BDE representation.
  • 2 Related Work: Persona-axis research suggests that structured multi-axis representations may activate combinations of internalized persona axes relevant to each task.The paper interprets BDE as a fixed recipe and auto-discovery as a per-task search over axis combinations.

3 Prerequisites and Experiment Design

The experiments compare raw transcripts, unstructured summaries, fixed BDE personas, and automatically discovered structures while holding the simulator evaluation framework fixed. BDE partitions persona evidence into three profiles, whereas auto-discovery iteratively refines task-specific structures.

  • 3 Prerequisites and Experiment Design: Twin-2K-500 provides 500 input questions, 88 holdout questions, and 17 prediction tasks for evaluating persona-context representations.The canonical baseline uses the raw question–answer transcript verbatim.
  • 3 Prerequisites and Experiment Design: Mega-Study supplies 19 pre-registered sub-studies spanning diverse decision domains for testing generalization.Each sub-study draws its own persona panel and uses range-normalized accuracy.
  • 3 Prerequisites and Experiment Design: BDE partitions each persona into Background, Decision procedure, and Evaluation profiles using input-side transcript evidence and downstream prediction questions.The profiles encode who the respondent is, how they reason, and what they want or would pay for.
  • 3 Prerequisites and Experiment Design: The BDE persona replaces the raw transcript in the simulator and is scored against holdout responses.Extraction is applied once per persona before simulator evaluation.
  • 3 Prerequisites and Experiment Design: Auto-discovery generates an initial structure and applies four refinement rounds before selecting one of five candidate rounds.The initial structure uses sub-study constructs and raw questions; later updates operate at both structure layers.

4 BDE Structure on Twin-2K-500

On Twin-2K-500’s 17 homogeneous prediction tasks, the study compares BDE, unstructured summaries, and raw transcripts using paired contrasts on the same 50 personas. BDE improves accuracy over both alternatives, while the same fixed structure does not transfer to the heterogeneous Mega-Study suite.

  • 4.1 Setup and statistical framing: The evaluation compares BDE with an unstructured summary to isolate structure, and with the raw transcript to measure structure plus compression.Each contrast uses the same paired personas; the raw-versus-unstructured comparison anchors compression alone.
  • 4.2 Findings on Twin-2K-500: Compression alone is roughly raw-equivalent overall (−0.58pp) and for pricing (+0.40pp), but mildly reduces cognitive-bias accuracy (−1.30pp).Table 1 reports the unstructured-summary contrast against the raw-transcript baseline.
  • 4.2 Findings on Twin-2K-500: +2.49pp overall, +2.25pp on cognitive bias, and +2.75pp on pricing: BDE significantly outperforms the unstructured summary on all three metrics.The pricing contrast is the largest absolute gain.
  • 4.2 Findings on Twin-2K-500: +1.91pp overall, +0.95pp on cognitive bias, and +3.15pp on pricing: BDE significantly improves over the raw transcript on all three metrics.This joint contrast combines compression and BDE structure relative to the raw transcript.
  • 4.3 Findings on the Mega-Study: Transferring BDE to 19 heterogeneous Mega-Study sub-studies produces an aggregate tie with raw transcripts and significant losses in three sub-studies.The Twin-2K-500 gain therefore does not transfer to the heterogeneous task suite.

5 Auto-discovery Structure on Mega-Study

The auto-discovery pipeline searches for task-specific persona structures using locked calibration and evaluation splits across 13 iteration-eligible Mega-Study sub-studies. It improves aggregate accuracy over raw, BDE, and unstructured baselines, but gains vary substantially across tasks.

  • Setup: The pipeline locks development/evaluation personas and calibration/holdout questions before prompt generation, reserving HOLDOUTs × EVALs for headline evaluation.The meta-extractor receives only construct-level calibration accuracy from the CALIBRATIONs × DEVELOPMENTs cell.
  • Setup: 13 iteration-eligible sub-studies were analyzed after excluding six zero-shot sub-studies whose question sets lacked calibration signal.The excluded sub-studies had question sets too small to support iterative calibration.
  • Iteration loop: The five-round loop initializes an LLM-generated structure, then revises sub-profile composition and transcript allocation using construct-level calibration accuracy.Rounds 1–4 refine both Layer 1 skeleton choices and Layer 2 evidence allocation without per-persona, per-item, or holdout feedback.
  • Findings: +1.91pp over raw transcripts, +2.22pp over BDE, and +1.40pp over unstructured summaries in the locked HOLDOUTs × EVALs aggregate.The aggregate uses Table 4’s 13-study row convention and compares all methods on the same cells.
  • Findings: Ten of thirteen sub-studies had point estimates within ±3pp of raw, so the aggregate lift was driven by a smaller set of larger wins.With n=30 evaluation personas and uneven holdout sizes, near-zero contrasts remain statistically ambiguous.
  • Findings: Three sub-studies improved by approximately +3pp or more against all three baselines, while significant losses against raw or BDE were absent at the sub-study level.The largest reported gains were consumer_minimalism, privacy, and digital_certification; one significant loss remained against the unstructured baseline.

6 Conclusion

The paper finds that persona structure affects digital-twin prediction, but fixed structures generalize poorly from homogeneous to heterogeneous tasks. Task-specific structure discovery restores aggregate gains across the heterogeneous benchmark, supporting task-dependent organization of persona information.

  • Conclusion: +1.91pp over raw transcripts for BDE on Twin-2K-500, while the same fixed structure essentially ties raw on the heterogeneous Mega-Study aggregate.BDE also incurs significant sub-study-level losses in three Mega-Study cases.
  • Conclusion: +1.91pp over raw, +2.22pp over BDE, and +1.40pp over unstructured summaries for auto-discovery on 13 Mega-Study sub-studies.No significant sub-study-level losses occurred against raw or BDE, although one remained against the unstructured baseline.
  • Conclusion: Persona structure is task-dependent: useful organization, rather than information volume alone, is identified as the relevant bottleneck.The conclusion follows from the contrasting fixed-schema and per-task auto-discovery results.
  • Practical takeaway: BDE is recommended as a first option for homogeneous task families with a behaviorally grounded partition, whereas heterogeneous tasks favor per-task structure discovery.The paper describes fixed allocation as potentially flattening across heterogeneous behavioral primitives.
  • Limitations: The experiments use small paired panels, calibration-based round selection whose informativeness is uncertain, limited simulator-family replication, and comparative rather than mechanistic evidence.The auto-discovery evaluation uses n=30 paired personas per sub-study, and broader cross-architecture replication remains open.

A Appendix: Datasets and evaluation metrics

The appendix documents datasets, scoring, aggregation, and the extracted BDE persona representation used in the experiments. It also records implementation and reproducibility details for the evaluation pipeline.

  • Evaluation metrics: Accuracy is computed per item from the normalized absolute difference between twin and respondent responses, then averaged within relevant cells and across persona panels.The Mega-Study formulation is equivalent when items within a question type share a response range.
  • Datasets: Mega-Study samples 50 personas separately for each of 19 sub-studies, with nearly disjoint samples across sub-studies.The mean pairwise overlap is 1.9 personas and the union contains 695 unique personas.
  • Evaluation metrics: Mega-Study holdout sets range from 1 to approximately 60 or more items and include multiple response formats and item-type mixes.The same range-normalized accuracy formula is used across Mega-Study comparisons.
  • Aggregation: Mega-Study aggregate mean accuracy is an unweighted mean over study–dependent-variable rows, with the headline aggregate restricted to locked HOLDOUT × EVALUATION cells.The 13-study headline contains 11 single-row studies and two studies contributing one MC and one Matrix row.
  • BDE representation: BDE personas are represented as concatenated Background, Decision procedure, and Evaluation files totaling roughly 250 lines before the test question.The appendix illustrates demographic, effort-regulation, and risk-preference excerpts alongside cognitive-test information.

C Appendix: Robustness across LLMs

Cross-model replications support the robustness of structured persona extraction, especially BDE’s advantage over unstructured summaries. Effects against raw transcripts are more model-dependent, while calibration-50 prompt effects transfer directionally across simulators.

  • BDE robustness: +1.27∗pp on gpt-5.4-mini and +2.61∗∗∗pp on Qwen3-8B: BDE improves overall accuracy over unstructured summaries.These replication contrasts reproduce the primary nano result in both direction and significance.
  • Raw-transcript contrasts: +2.43∗∗∗pp overall on gpt-5.4-mini, but +1.04pp ns on Qwen3-8B: BDE’s end-to-end gain over raw transcripts varies by simulator.On gpt-5.4-mini, the cognitive-bias contrast is +1.76∗∗∗pp; on Qwen3-8B, the gain concentrates there at +2.40∗∗∗pp.
  • Metric variation: Per-metric effects are model-specific: gpt-5.4-mini is directionally positive on every metric, whereas Qwen3-8B’s +5.90∗∗∗pp pricing effect against unstructured summaries does not transfer against raw transcripts.The overall BDE-versus-unstructured contrast remains consistent across both replication models.
  • Calibration-50 transfer: +1.01pp on gpt-5.4-mini: calibration-50 prompt effects transfer directionally from gpt-5.4-nano, but with attenuated magnitude.Calibration-50 reuses the nano-selected extraction prompts unchanged; the corresponding contrasts are +0.04pp versus structured and +0.05pp versus unstructured summaries.
  • Calibration-50 transfer: Calibration-50 prompt effects transfer directionally to both Qwen3-8B and gpt-5.4-mini when evaluated on the same locked cells.The reruns use paired-persona bootstrap confidence intervals with B=10,000 on 30 evaluation personas.

D Appendix: Iteration pipeline implementation and supplementary results

The appendix documents the iteration pipeline, its validation and cross-round refinement mechanisms, supplementary robustness evaluations, and deployment implications. It also identifies simulator sampling non-determinism as a limitation and excludes zero-shot sub-studies from trajectory-based iteration analysis.

  • Implementation: The iteration pipeline uses a fail-closed 9-rule validator before saving each meta-extractor proposal as the next-round extraction prompt.Invalid proposals are rejected and re-asked up to two times, with fallback to the prior round after a third failure.
  • Implementation: From round 2 onward, the meta-extractor receives prior DIFF_LOG blocks, an already-addressed-constructs table, and calibration trends to guide refinement.The template forbids re-fixing addressed constructs unless their trajectories regress and forbids reverting fixes without supporting trend evidence.
  • Limitation: Simulator sampling is non-deterministic because reasoning-effort calls use fixed default sampling, producing non-trivial round-to-round variance even for byte-identical prompts.The study controls this only through locked splits and the calibration-50 tie-break to the earliest round.
  • Supplementary results: Zero-shot sub-studies are excluded from per-round iteration analysis because their prompts remain byte-identical across rounds, leaving movement attributable to simulator sampling noise.Selecting a best round on this subset would inflate the aggregate.
  • Artifacts and cost: The released pipeline includes final per-sub-study prompts, round-specific prompts, a meta-extractor template, a validator, and paired-bootstrap logs.Deployment requires one extractor pass per persona–sub-study cell; iterative refinement is upstream tooling rather than deployment cost.

E Appendix: Selection-rule comparison

Calibration-50 is a practical round-selection rule that recovers much of the oracle headroom while avoiding headline-metric leakage. Using only 20 development personas is noisier and can select regressed late rounds.

  • Selection rules: Calibration-50 selects the round with highest mean calibration accuracy across all 50 personas, breaking near-ties in favor of the earliest round.The rule is the paper default.
  • Selection rules: The oracle rule selects the highest headline accuracy on HOLDOUT×EVAL, but it peeks at the headline cell and is therefore not deployable.It is reported only as a power upper bound.
  • Results: 69.45pp: calibration-50 recovers approximately 57% of the oracle’s lift over raw, compared with the oracle’s 70.91pp upper-bound headline and +3.37pp lift.Calibration-50’s lift is +1.91pp over raw.
  • Results: Calibration-20 falls below the unstructured baseline because its 20-person development signal is noisier and can select late rounds whose holdout performance regressed.The appendix identifies this as development-calibration overfit; calibration-50’s full-persona coverage and earliest-round tie-break guard against it.
  • Results: Calibration-50 keeps round 0 on 8/13 sub-studies, compared with 5/13 for calibration-20 and 4/13 for the oracle.The remaining calibration-20 and oracle selections shift toward rounds 1 or 2.

F Appendix: Examples of selected-round structures

Auto-discovered structures vary substantially across Mega-Study sub-studies, typically containing four or five sub-profiles rather than reproducing BDE’s labels. Examples show both targeted gains and cases where iteration does not generalize to holdout performance.

  • Structure patterns: 9 of 19 structures have 4 sub-profiles, 8 have 5, and 2 have 6, concentrating structure size at 4–5 sub-profiles.The two six-profile structures reached the validator cap.
  • Structure patterns: BDE’s literal labels never reappear among the 19 structures; reasoning and evaluation axes are instead authored as sub-study-specific concepts.Identity is represented through varying demographic, political, trait, or ideology dimensions.
  • Examples: Consumer_minimalism is the lone significant win against both baselines, growing from four to five sub-profiles by adding a cross-domain consistency axis.The added axis addressed insufficient coverage of the voluntary-versus-forced minimalism disambiguator.
  • Examples: Accuracy_nudges grows from four to five sub-profiles, but its gains over both baselines are not significant at n=30.The contrast against structured is slightly negative.
  • Examples: Obedient_twins reaches six sub-profiles, yet its selected round trends below raw because calibration and evaluation trajectories diverged.This illustrates that calibration-50 is leakage-clean but does not guarantee holdout generalization.

G.2 Pricing items (3 of the 40-item willingness-to-pay block)

The pricing block presents binary purchase decisions for branded products while varying product category, brand salience, and price deviation from typical shelf price. It includes ordinary-price items and an intentionally extreme overpricing probe, within a study that raises privacy and targeting risks for more accurate persona simulation.

  • Pricing items elicit binary purchase decisions for branded products across product category, brand salience, and deviation from typical shelf price.
  • The three examples include Tylenol at $2.19, Land O Lakes butter at $7.39, and Goya ham at $53.98, with the latter several times typical shelf price.The Goya item probes whether the simulator rejects an extreme offer.
  • More accurate persona simulation may support sample-efficient behavioral measurement but also creates risks of privacy-invasive profiling, manipulative personalization, and targeting vulnerable individuals.The paper recommends consent, limits on sensitive-attribute inference, and privacy and fairness evaluation in consequential deployments.
  • Experiments are inference-only against hosted APIs, making token usage the dominant cost.The stated assumptions include approximately 32×10^3 input tokens for a raw transcript and 4×10^3 for a structured persona.
Loading 2608.20344v1…