PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Psychological Dynamics in Generative Agents

Sumer S. Vaid, Ashley V. Whillans

arXiv:2610.04246v1cs.HCcs.AI

TL;DR

LLMs are increasingly used as computational replicas, but benchmarking whether they reproduce psychological experience over time remains difficult. This paper evaluates stable personas exposed to evolving events using published norms and a two-dimensional psychometric benchmark. The models often recover established relation directions, yet consistently fail to reproduce within-person correlation and temporal variance.

  • Problem

    Although LLMs increasingly simulate human behavior, quantifying whether they reproduce the richness of lived psychological experience over time remains difficult.

    Source: §1 · p. 1

  • Method

    The benchmark follows stable personas through evolving events and evaluates longitudinal trajectories against published norms on internal measurement structure and established psychological relations.

    Source: §1 · p. 2, §3.1 · p. 4, §4 · p. 5

  • Results

    LLMs achieve 84.3% mean direction agreement across meta-analytic targets, but consistently fail to reproduce within-person correlation and temporal variance.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Validated within-person dynamics are a necessary precondition for robust causal inference with computational replicas.

    Source: §1 · p. 2, §6 · p. 8

    Longitudinal ground truth remains a harder problem and lies beyond an all-silico evaluation.

    Source: §5.2 · p. 8

Abstract

from arXiv · show

Large language models (LLMs) are increasingly deployed to simulate human behavior, acting as computational replicas of human subjects. Yet the lived psychological experience of humans is difficult to benchmark, particularly as it unfolds over time. We introduce a psychometric benchmark for computational replicas: personas that carry a fixed identity through an evolving sequence of events. Built entirely from published norms and meta-analytic effects, the benchmark scores two dimensions of psychological realism. The first, internal validity, quantifies whether generated trajectories reproduce the internal structure of repeated human measurement: distributions, the between- versus within-person variance partition, temporal dependence, and range. The second, external validity, quantifies whether replicas recover established trait, state, and indicator relations. Across 36 open-weight and proprietary LLMs from nine developers (1B-671B parameters), most recover the direction of established relations (84.3% mean agreement) and the variance partition (26 of 34), yet the within-person correlation and distributional structure elude recovery. Internal validity is independent of scale and capability: a mid-size open LLM (Gemma-3-27B) strikes the best trade-off between the two dimensions. The benchmark is a precondition for using computational replicas in causal inference across domains (e.g., marketing, healthcare), and identifies within-person grounding as the central challenge ahead.

1 Introduction

Evaluating LLMs as human computational replicas remains difficult, especially when psychological experience unfolds over time rather than in isolated snapshots. This paper introduces a benchmark that follows stable personas through evolving events and evaluates longitudinal trajectories against published norms and psychological relations.

1 Introduction

  • Scientific standards for quantifying computational replicas’ psychological realism have received limited empirical attention despite growing use of LLMs to model human behavior.Task-specific benchmarks do not readily quantify how well systems recreate the richness of lived experience.Source: §1 · p. 1
  • Static, single-query evaluations miss whether personas maintain realistic psychological patterns as context accumulates over time.The paper argues that reliable simulation requires reproducing how thinking, feeling, and behavior change across interlaced events.Source: §1 · p. 1
  • The benchmark gives replicas stable identities, exposes them to evolving events, and measures psychological states at each timestep.This yields continuous trajectories rather than static snapshots.Source: §1 · p. 2
  • The psychometric benchmark assesses longitudinal state trajectories in synthetic populations against internal measurement dynamics and established psychological relations.It evaluates 300 synthetic employees across six workdays and 36 LLMs from nine developers, spanning 1B–671B parameters.Source: §1 · p. 2

2 Related Work

Prior work evaluates synthetic behavior largely through static outputs, while evidence on persona consistency, alignment-related diversity loss, and long-term stability points to the need for longitudinal measurement. The section situates this benchmark as a response to that methodological gap.

2.1 The Limits of Cross-Sectional Evaluation

  • Cross-sectional benchmarks can validate demographic matching, but matching static distributions leaves temporal continuity and stable behavioral architecture unresolved.Prior critiques note that single-trial elicitation can leave analytic flexibility unconstrained; survey consistency and one-shot persona studies likewise leave continuity open.Source: §2.1 · p. 2

2.2 Alignment Compression and Statistical Priors

  • Instruction tuning and safety alignment are associated with reduced behavioral diversity, shifting generated responses toward normative consensus.The paper describes this as an alignment-simulation trade-off: consensus-question performance may improve while simulation of diverse edge cases degrades.Source: §2.2 · p. 3

2.3 Architectural Bottlenecks Over Time

  • Over extended interactions, personas can drift toward average, homogeneous identities, while proposed stabilizing interventions remain localized or stop short of certifying temporal coherence.Prior work attributes some decay to transformer working-memory limits; prompt engineering and multi-turn reinforcement learning are described as patches rather than structural solutions.Source: §2.3 · p. 3

2.4 The Necessity of Longitudinal Measurement

  • Longitudinal applications and multi-agent theories assume personas can integrate experience and update over time, but this temporal architecture remains unverified.Deployed systems also change across silent LLM updates, so longitudinal standards must distinguish intended development from artifactual movement.Source: §2.4 · pp. 3–4

3 Creating the Benchmark

The benchmark seeds synthetic personas and workplace states from published human norms, evaluates them on shared evolving narratives, and scores both repeated-measurement structure and established psychological relations.

3.1 Population Initialization

  • Personas’ traits, state baselines, and between- versus within-person variance are initialized from published norms, meta-analytic relations, and experience-sampling intraclass correlations.The benchmark covers four workplace states and 300 personas across six simulated days; contextual indicators are seeded while target state columns remain unpopulated, making recovered indicator-state relations emergent rather than seeded.Source: §3.1 · p. 4

3.2 Persona Generation

  • Every evaluated LLM rates the same daily narratives, producing six state vectors for each of 300 personas.Identity portraits and event narratives are generated once by a fixed event model; identical narratives and a uniform prompt structure make rating differences attributable to the evaluated LLM rather than the events.Source: §3.2 · pp. 4–5

3.3 Two Dimensions of Psychological Realism

  • Internal validity tests whether trajectories match human repeated-measurement properties, while external validity tests established trait–state and indicator–state relations.Internal criteria include state means and spreads, variance partition, variability, cross-state correlations, temporal dependence, dimensionality, and scale range; two criteria are adjusted because the design limits identification.Source: §3.3 · p. 5

4 Experimental Setup

The experiment applies a uniform benchmark to a broad set of LLMs and evaluates external directional agreement alongside internal trajectory fidelity. The roster and metrics define the comparison scope.

4 Experimental Setup

  • The study analyzes 34 complete longitudinal corpora from a 36-model roster spanning nine developers and 1B–671B parameters.The cohort includes open-weight and proprietary models, dense parameter ladders, mixture-of-experts systems, and specialized reasoning engines.Source: §4 · p. 5
  • All benchmarked LLMs process the same fixed identity portraits and daily narratives under a uniform prompt architecture.The generation and rating pipelines run concurrently and can resume after halts or errors.Source: §4 · p. 5
  • External convergence is measured by the share of simulated relations matching established algebraic signs and by signed rank correlations.Internal validity uses equivalence verdicts and continuous distances, including absolute log-ratio error between synthetic and target human within-person standard deviations; lower absolute error indicates closer alignment.Source: §4 · p. 5

5 Results

Across the evaluated models, established relation directions are often recovered, but within-person correlations are compressed and internal validity does not reliably improve with model scale. Cross-cultural measurement structures remain consistent when personas are initialized with demographic priors.

5.1 The Divergence of Scale and Internal Validity

  • Gemma-2-27B has the lowest within-person variance error (0.043), narrowly ahead of the most mathematically capable frontier LLM (0.061), while external validity stays roughly flat across scale.The smallest Gemma architecture has the largest clean-roster variance error (0.427), despite 74.6% pooled directional agreement.Source: §5.1 · p. 6, Figure 2 · p. 7
  • 26 of 34 populations meet the between- versus within-person variance partition and range criteria, but few meet distributional-shape, cross-state-correlation, or full multilevel-eigenstructure criteria.The best population clears 3 of 6 internal-validity levels; the six-day autoregressive coefficient is non-identifiable, so temporal evidence is directional only.Source: §5.1 · p. 6

5.2 Compression of the Within-Person Correlation

  • 84.3% mean direction agreement contrasts with pooled within-person correlations near 0.05, versus published human effects of 0.20 to 0.70.Direction and rank are recovered, but the within-person signal remains near the sampling noise floor across rating conditions.Source: §5.2 · pp. 6–7

5.3 Cross-Cultural Invariance and Demographic Simulation

  • Across six cultures, cross-state structures depart from the global baseline by mean absolute differences of 0.014 to 0.038, and up to 75% of evaluated group differences match published directions.Precise demographic priors preserve cross-state correlation patterns while allowing cultural distinctions; the smaller Gemma-3-4B differentiates cultures more strongly than Gemma-3-27B.Source: §5.3 · p. 8

6 Discussion

The evaluated LLMs recover many established psychological relation directions, but this success coexists with compressed within-person variance and correlations. The discussion argues that temporal coherence must be validated before computational replicas are used for robust causal inference.

6 Discussion

  • 84.3% mean direction agreement across meta-analytic targets supports external validity, but does not resolve the systematic failure to reproduce within-person correlation and temporal variance.The paper identifies variance compression as a consistent bias, and relates it to instruction tuning and optimization toward normative consensus.Source: §6 · p. 8
  • Temporal coherence is a necessary precondition for robust causal inference with computational replicas, because distributional realism and directional accuracy can coexist with incorrect causal effect sizes.Applied longitudinal frameworks are described as relying on an unvalidated assumption of continuous stability.Source: §6 · p. 8

7 Conclusion

The conclusion frames longitudinal coherence as a measurement problem and calls for stability checks before population simulations over time. PsyDyBench releases its validation materials and corpora with a script to regenerate reported outputs.

7 Conclusion

  • Moving from static personas to longitudinal replicas exposes a gap between directional and quantitative fidelity when LLMs rate extended experience records.The authors present longitudinal coherence as a rigorous, auditable measurement problem and call for stability verification before simulating populations over time.Source: §7 · pp. 8–9

Reproducibility Statement

  • PsyDyBench releases the validation battery, generation pipeline, scoring code, generated corpora for 36 models, result tables, provenance manifests, and random seeds.A single script regenerates every table, figure, and reported number from the released corpora without API access.Source: Reproducibility Statement · p. 9

A The Within-Person Correlation: Why It Compresses and How to Recover It

The benchmark recovers the direction of within-person relations, but their correlation remains near zero. Internal and external validity also vary largely independently across models.

A The Within-Person Correlation: Why It Compresses and How to Recover It

  • Within-person relations have the correct direction, yet their correlation remains near zero—the work’s central puzzle.The appendix distinguishes routes available to all-silico evaluation from those requiring longitudinal human data.Source: A The Within-Person Correlation: Why… · p. 14
  • The appendix frames near-zero within-person correlation as a central puzzle despite recovery of relation directions.It separates all-silico evaluation routes from approaches requiring longitudinal human data.Source: A The Within-Person Correlation: Why… · p. 14
  • Across models, internal and external validity are largely independent (r ≈0.17).The figure caption reports that gains in internal validity leave external validity unchanged.Source: Figure 3 · p. 15

A.1 The safe-deviation tendency

The safe-deviation tendency describes how generated ratings preserve the direction of daily effects while shrinking their magnitude through several reinforcing mechanisms.

A.1 The safe-deviation tendency

  • A model’s end-of-day rating acts more like an average than a single response, smoothing within-person variation; single-rating reliability is 0.97.Within-person correlation is 0.059 at temperature 0.8 and 0.060 at temperature 0.0.Source: §A.1 · p. 15
  • Modal-answer alignment and supplied trait baselines pull ratings toward moderate values and the persona’s prior.Removing the baseline anchor raises sign agreement from 71% to 77% and rank correlation from 0.38 to 0.46, while within-person correlation holds steady.Source: §A.1 · p. 15
  • Two successive mappings—from seeded indicators to narrative, then narrative to rating—smooth the daily signal toward averages.Together, the mechanisms yield effects with the correct direction but diminished magnitude.Source: §A.1 · p. 15

A.2 Capability and within-person recovery

With prompts and narratives held fixed, lower-capability models show stronger within-person rank signals, while within-person correlation remains near the sampling floor.

A.2 Capability and within-person recovery

  • Within-person rank correlation rises from 0.38 for gpt-4.1 to 0.45 for gpt-4.1-mini and 0.48 for gpt-4.1-nano; the smallest model reaches 75% sign agreement.The comparison re-rates shared narratives while holding prompt and data fixed.Source: §A.2 · p. 15
  • Within-person correlation stays between 0.023 and 0.028 across the three gpt-4.1 models.The passage describes this range as near the sampling floor.Source: §A.2 · p. 15
  • The authors interpret the pattern as lighter alignment preserving day-to-day variation, while construct variance governs within-person correlation.Capability changes rank-signal strength but does not lift correlation from its near-floor range.Source: §A.2 · p. 15

A.3 Construct variance and within-person recovery

Within-person recovery depends on available construct variance, and literature-only prompt tuning produces small held-out gains. The appendix also specifies the benchmark’s scope, seeding, criteria, rating procedure, and model comparison.

A.3 Construct variance and within-person recovery

  • With more balanced between- and within-person variance, pooled |r| reaches 0.058 for exhaustion, 0.075 for fatigue, and 0.069 for flow, versus 0.023 for trait-saturated states.The three constructs have ICCs of 0.34, 0.40, and 0.48; results use 8 anchored pairs per state and are presented as a directional demonstration.Source: §A.3 · p. 16

A.4 In-silico achievability of the internal-validity targets

  • Literature-only prompt tuning raises held-out composite scores from 0.693 to 0.702 for gpt-4.1-mini and from 0.660 to 0.673 for gpt-4.1-nano.The objective uses published norms, and optimization uses 30 training personas with 30 held aside.Source: §A.4 · p. 16

A.5 A principled bound and the routes that remain

  • The all-silico target is matching within-person variance, range, and direction to published effects; correlation is bounded by construct variance.Methods conditioned on measured individual histories address realized-trajectory prediction rather than the all-silico benchmark.Source: §A.5 · p. 16

B Seeding construction

  • Personas are seeded from published Big Five norms and meta-analytic intercorrelations, with state baselines linked linearly to standardized traits.Published intraclass correlations split each state’s total variance; target state columns remain unpopulated so indicator–state relations must emerge during generation.Source: B Seeding construction · pp. 16–17

C Internal validity criteria

  • Internal validity uses equivalence tests for distributions, variance partition, cross-state structure, temporal dependence, latent dimensionality, and range.The AR(1) result is directional given six-day non-identifiability; the affect-specific L5 check is outside scope.Source: C Internal validity criteria · p. 17

D Rating prompt

  • Every model receives the same untuned prompt, asking for first-person end-of-day self-reports of actual internal states.The prompt combines persona identity, daily narrative, and scale guidance; outputs include ratings for four named states.Source: D Rating prompt · p. 17

E Model leaderboard

  • External-validity measures stay high and roughly flat as internal validity varies across the leaderboard.Table 1 ranks models by within-person variance error, with trait–state direction agreement and rank correlation alongside it.Source: E Model leaderboard · p. 17, Table 1 · p. 18
Loading 2610.04246v1…