PaperlayerView exact revision on arXiv

Source-linked AI summary

How Far Do Persona Effects Generalize in Language Models?

Yufan Zhou, Yuxuan Liu, Enze Ma, Lyumanshan Ye, Zhongqi Yue, Robin De Croon, Yucheng Jin, Katrien Verbert, Zhao Wang

arXiv:2609.32758v1cs.CLcs.AI

TL;DR

Persona prompts produce answers used to study human behavior, but it is unclear whether their effects generalize to new questions, transfer across models and prompts, or correspond to real populations. The paper tests prediction and transfer through fitted persona-effect gains, alongside a separate human-fidelity comparison. Effects can be predictable without being portable, and target refitting or calibration can outperform simple reuse in the tested settings.

  • Problem

    Prior evaluations examine agreement, responsiveness, diversity, and prompt sensitivity, but do not establish what a source relationship adds to prediction in a new target.

    Source: §1 · p. 1

  • Method

    The authors measure attribute-driven changes in answer log odds, fit gains between checkpoints or prompts, and compare their predictions on new questions and changed settings.

    Source: §1 · p. 2, §2 · p. 2, §2.2 · p. 3

  • Results

    Separate attribute and task gains improve prediction significantly in OLMo-3 and Qwen2.5, but OLMo-3 refitting beats borrowed patterns and prompt-transfer updates outperform reuse strategies in median at 64 questions per attribute.

    Source: §1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    Predictable persona effects need not transfer, so source relationships should be tested for added predictive value beyond target calibration and regularization.

    Source: §1 · p. 2, §8 · p. 9

    The evidence is limited to fixed releases, constructed forced-choice questions, and held-out questions rather than held-out people; human evaluation uses historical U.S. surveys.

    Source: A SCOPE AND LIMITATIONS · p. 13

Abstract

from arXiv · show

Persona prompts ask language models to answer as particular kinds of people. We test whether relationships learned from these effects predict responses to new questions and remain useful across models and prompts. Across 57 attributes, three behavioral domains, and seven pairs of open 7 to 9B checkpoints, persona effects can be predictable without being portable. Separate attribute and task gains improve prediction beyond shared scaling significantly in OLMo-3 and Qwen2.5, with the most robust evidence in OLMo-3. In that model, target refitting significantly outperforms gains borrowed from each of the other six pairs. Across model transfers, borrowed gains with one amplitude underperform shared scaling in most directions; allowing two target parameters removes the significant losses but yields no significant benefit over target shared scaling. After rewording, refitting significantly outperforms reuse with one amplitude in all six tested pairs, while changes of examples or country context often preserve reuse value. In the tested prompt transfers, regularized updates outperform both reuse strategies in median at 64 target questions per attribute. A separate survey comparison finds that selecting the more responsive checkpoint can worsen human fit; responsiveness is confounded with training status, and temperature calibration largely removes this cost but not errors in group ordering. Within the tested gain representation, apparent transfer can come from target calibration; source relationships must add predictive value beyond calibration and regularization. Code and data are available at https://github.com/thzva/persona-gain

1 INTRODUCTION

Persona prompts can produce answers used to study human behavior, but it remains unclear whether their effects form generalizable regularities, survive changes in setting, or correspond to real populations. This paper tests prediction on new questions, transfer across models and prompts, and human fidelity as separate questions.

1 INTRODUCTION

  • Persona prompts generate answers associated with particular kinds of people and are used to pilot surveys, replicate experiments, and populate social simulations.The open question is whether those answer changes correspond to generalizable behavioral regularities and real populations.Source: §1 · p. 1
  • Prior evaluations characterize survey agreement, attribute responsiveness, population diversity, and prompt sensitivity, but not the predictive contribution of a source relationship in a new target.Target-data improvements may reflect calibration rather than the borrowed pattern.Source: §1 · p. 1
  • The study compares shared scaling, source reuse, target calibration, and target refitting to test whether source relationships add predictive value beyond target adaptation.It covers 57 attributes, three behavioral domains, and seven matched checkpoint pairs of open 7 to 9B models.Source: §1 · p. 2
  • Separate attribute and task gains improve prediction significantly in OLMo-3 and Qwen2.5, while the strongest transfer tests show that refitting can outperform borrowed gains and responsiveness alone does not ensure human fit.The introduction also reports that refitting beats reuse with one amplitude after rewording, while other prompt changes often preserve reuse value.Source: §1 · p. 2
  • Persona effects can be predictable without being portable, so reuse must be tested on the target rather than inferred from similarity between fitted patterns.The figure distinguishes tests on new questions and prompt or profile changes from a separate human survey evaluation using different attributes.Source: §1 · p. 2, Figure 1 · p. 2

2 A PREDICTIVE FRAMEWORK FOR PERSONA EFFECTS

The framework measures how changing a persona attribute shifts answer preferences, then tests whether fitted gains predict unseen questions and transfer across settings. It compares shared scaling, source reuse, target refitting, and regularized updating on common target data.

2 A PREDICTIVE FRAMEWORK FOR PERSONA EFFECTS

  • The framework asks how an attribute changes an answer and whether fitted gains predict new questions or remain useful when context changes.A gain is a fitted multiplier describing how training changes the response.Source: §2 · p. 2

2.1 MEASURING HOW AN ATTRIBUTE CHANGES AN ANSWER

  • Persona effects are measured by changing exactly one attribute value between prompts for the same question and comparing the resulting answer log odds.The evaluation spans 57 attributes and fits gains on 181 questions before testing on 45 newly written questions.Source: §2.1 · pp. 2–3

2.2 LEARNING HOW TRAINING CHANGES ATTRIBUTE EFFECTS

  • Training changes are modeled by gains that may vary by attribute and task, with prediction on new questions testing whether they improve on one shared multiplier.Positive ΔQ2 means the varying-gain model predicts better; the score is not a mean-centered R² or accuracy.Source: §2.2 · p. 3

2.3 TESTING WHETHER A LEARNED RELATIONSHIP CARRIES OVER

  • Transfer tests retain each target’s Base effects and compare borrowed source gains, target refitting, and regularized updating against shared scaling on the same target questions.A two-parameter reuse variant separates overall responsiveness from attribute differences; Figure 2 contrasts rescaling, penalized updating, and refitting while keeping models fixed.Source: §2.3 · pp. 3–4, Figure 2 · p. 3

3 TRAINING RESHAPES PERSONA EFFECTS PREDICTABLY

Training changes persona effects unevenly across attributes and tasks, and some of that variation predicts new questions beyond shared scaling. The evidence is most robust in OLMo-3, while tested DPO steps are better described by shared scaling than Base-to-SFT steps.

3.1 TRAINING DOES NOT SIMPLY TURN PERSONA EFFECTS UP OR DOWN

  • Training can raise overall persona responsiveness while shrinking particular effects: OLMo-3 SFT amplifies 157 of 171 attribute-task combinations, yet gender contrasts shrink in all three tasks.Figure 3 summarizes category-level magnitude ratios and Base-to-trained agreement across all 57 attributes.Source: §3.1 · p. 4, Figure 3 · p. 4

3.2 PREDICTION IMPROVES MOST RELIABLY IN OLMO-3

  • Gains fitted on training questions predict new questions better than shared scaling by point estimate in six of seven pairs, with the most robust evidence in OLMo-3.Qwen2.5 also passes the strictest test; robustness support varies elsewhere.Source: §3.2 · pp. 4–5, Table 1 · p. 5

3.3 SHARED SCALING FITS THE TESTED DPO STEPS BETTER

  • Across three trajectories, shared scaling leaves 63 to 97% of no-change prediction error during Base-to-SFT, versus 22 to 31% during SFT-to-DPO.The passage cautions that smaller changes may be easier to approximate by scaling; OLMo-3 provides a clearer contrast because its two stages change effects by similar amounts.Source: §3.3 · p. 5, Table 1 · p. 5

4 PREDICTABLE IS NOT PORTABLE

Persona gains often predict new questions within a setting but do not reliably transfer across models or prompts. Rewording weakens reuse, while target updates—especially regularized updates with enough questions—can outperform reuse.

4.1 ACROSS MODELS: BORROWED GAINS ADD LITTLE REUSABLE VALUE

  • Across 42 directed transfers, one-amplitude reuse predicted worse than target shared scaling in 40 directions, significantly in 29; with two parameters and Qwen2.5 in float32, no direction differed significantly.For OLMo-3, refitting significantly beats all six borrowed patterns. Figure 4's bottom row is a target refit reference, not an upper bound.Source: §4.1 · pp. 5–6, Figure 4 · p. 6

4.2 ACROSS PROMPTS: REWORDING CARRIES A COST, OTHER CHANGES NEED NOT

  • After wording changes, refitting significantly beats one-amplitude reuse in all six pairs, while changing only examples preserves reuse gains over shared scaling in four of six pairs.The example-change result uses patterns fitted on two source templates and holds for either one amplitude or two target parameters.Source: §4.2 · p. 6, Figure 5 · p. 7

4.3 WHEN REUSE FAILS: UPDATE, AND AT WHAT COST

  • At 64 target questions per attribute, regularized updates beat one-amplitude reuse at every tested budget and nearly match refitting from 32 questions onward.The source prior shows no consistent advantage over a shared-gain prior; the frozen-template study prospectively confirmed refitting's advantage in OLMo-3 and Tülu.Source: §4.3 · p. 7, Figure 6 · p. 7

5 COMPOSITION CHANGES WHAT AN ATTRIBUTE DOES

The paper tests whether isolated-attribute effects remain stable when other persona information is added. Training shifts relative cue influence differently across models, and full profiles retain only a small fraction of isolated trait effects.

5.1 TRAINING CHANGES RELATIVE CUE INFLUENCE DIFFERENTLY ACROSS MODELS

  • Training shifts the relative country effect downward in Tülu, Qwen2.5, and Qwen3, upward in Qwen3.5, and leaves OLMo-3 and Llama-3.1 unresolved.The effect is context dependent and model specific; the results do not support a single mechanism.Source: §5.1 · p. 8, Figure 7 · p. 8

5.2 PROFILES CHANGE MORE THAN EFFECT MAGNITUDE

  • In Base models, full profiles retain only 2–10% of isolated trait-effect magnitude in bfloat16 and 3% for Qwen2.5 in float32.These small signals warrant caution about numerical precision, and the tests do not distinguish semantic interactions from prompt-length or insertion-position effects.Source: §5.2 · p. 8
  • For Qwen2.5 float32 trait insertions, refitting improves on shared scaling by 12.7 Q2 points, but replacement tests show no significant refitting advantage.The insertion and replacement tests hold the rest of a DeepPersona profile fixed.Source: §5.2 · p. 8

6 A BOUNDARY: RESPONSIVE IS NOT FAITHFUL

The paper separately evaluates whether persona responsiveness tracks human survey fit. Selecting the more responsive checkpoint can worsen fit, while temperature calibration reduces that selection cost but does not resolve group-ordering errors.

6 A BOUNDARY: RESPONSIVE IS NOT FAITHFUL

  • In six of eight checkpoint comparisons, the more persona-sensitive checkpoint is the worse simulator, with up to 1.41 nats of evaluation loss in Qwen2.5.The more sensitive checkpoint is always the trained one, so this is a boundary case for selection by sensitivity, not evidence that sensitivity independently predicts poor fit.Source: §6 · p. 8
  • One temperature per checkpoint reduces selection regret to at most 0.005 nats in all eight comparisons.Calibration on 32 questions comes within 0.002 nats of using all 84, averaged over subsets.Source: §6 · p. 8

7 RELATED WORK

Related work covers persona-based human simulation, attribute composition, and prompt robustness and calibration. This paper positions its contribution as testing what source relationships add to prediction in new targets beyond calibration and regularization.

7 RELATED WORK

  • Prior work uses conditioned models to reproduce aggregate survey patterns and classic experiments, while documenting uneven demographic alignment and answer-order effects.The cited studies include Argyle et al. (2023), Aher et al. (2023), Santurkar et al. (2023), and Dominguez-Olmedo et al. (2024).Source: §7 · p. 9
  • Research on persona composition finds that attribute relevance, backstories, and combinations can shape simulation quality and produce task-dependent interactions or collapse.The paper's controlled edits test whether isolated-attribute relationships predict behavior within richer personas.Source: §7 · p. 9
  • Prompt studies document sensitivity to role format and phrasing, while related work distinguishes steerability from distributional coverage and alignment.This paper tests what source relationships add beyond target calibration and regularization on the same target questions.Source: §7 · p. 9

8 DISCUSSION AND CONCLUSION

Persona effects can predict new responses without transferring reliably. The paper recommends testing source information against target calibration and regularization on held-out questions.

8 DISCUSSION AND CONCLUSION

  • The evidence covers forced-choice log odds from seven open 7–9B checkpoint pairs and concerns multiplicative gains, which cannot represent effects created post-training where Base effects are absent.Transfer through other representations is untested.Source: §8 · p. 9
  • In tested prompt transfers, regularized updates outperform both reuse strategies in median at 64 target questions per attribute.The comparison assumes access to the required checkpoints; model-transfer budgets remain untested.Source: §8 · p. 9
  • The practical test is whether source information adds predictive value beyond target calibration and regularization.The paper cautions that apparent transfer can arise from target calibration.Source: §8 · p. 9

A SCOPE AND LIMITATIONS

The study evaluates persona-effect prediction and transfer using fixed model releases, constructed questions, and forced-choice A/B readouts. Its scope is bounded by shared question pools, question-based rather than person-based holdouts, and the tested gain representation.

A SCOPE AND LIMITATIONS

  • The experiments use fixed model releases and constructed questions, measure first-token A/B log odds, and do not identify the mechanism behind measured changes.First-token probabilities can differ from generated answers; only the human selection study includes a sampled-answer check.Source: A SCOPE AND LIMITATIONS · p. 13

B MATERIALS AND MODELS

  • The evaluation covers seven behavioral model pairs, but pairs share a question pool and Tülu and Llama share a pretrained backbone, so pair comparisons are not independent replications.The tests hold out questions rather than people; human evaluation uses historical U.S. surveys, leaving new identities, open-ended generation, other languages, and non-U.S. populations for future testing.Source: B MATERIALS AND MODELS · p. 13, A SCOPE AND LIMITATIONS · p. 13
  • The study represents 57 attributes in 12 categories through 287 statements and evaluates on 45 independent questions written by a separate model-based writer without access to results.Each value uses a fixed sentence template; no statement or question was pretested with human respondents.Source: B MATERIALS AND MODELS · pp. 13, 15

C.1 READOUT, SUMMARIES, AND FITS

  • Persona effects are measured as changes in answer log odds when an attribute value changes, and gains describe how effects scale between checkpoints or prompts.The outcome is the log odds of the two answer tokens; the normalized A/B probability is derived from those odds.Source: §C.1 · p. 16
  • Weighted contrasts balance attribute categories and tasks, while shared, attribute, task, and cell gains fit different levels of variation.Question pairs share observations, so uncertainty procedures resample whole questions.Source: §C.1 · pp. 16–17

C.2 PREDICTORS

  • Reuse ranges from applying source gains without target fitting to fitting a target amplitude or two parameters; refitting instead estimates new cell gains.Flexible reuse contains shared scaling, and regularized updates interpolate between refitting and reuse.Source: §C.2 · p. 17

C.3 SCALE INVARIANCE

  • With fixed weights and splits, refitting gains makes held-out Q2 invariant to positive global rescaling of source and target contrasts.This does not license rescaling a target while keeping an unadjusted source coefficient; fixed penalties, nonuniform temperatures, and nonlinear probability transforms need separate analysis.Source: §C.3 · p. 17

C.4 READING Q2 AND SIMILARITY

  • Separate gains modestly reduce held-out error in the reported models, while high uncentered gain correlation can conceal opposing attribute differences.On new questions, OLMo-3’s weighted RMSE falls from 1.054 with shared scaling to 1.010 with attribute and task gains; a dominant common component can make oppositely structured patterns appear highly correlated.Source: §C.4 · p. 17

D PREDICTIVE STRUCTURE AND TRAINING STAGES

Separate attribute and task gains improve prediction beyond shared scaling in OLMo-3 and Qwen2.5 when fitting uncertainty and correction are included. Yet source gains often fail to add value beyond target scaling, while prompt transfer depends on the change and the update strategy.

D PREDICTIVE STRUCTURE AND TRAINING STAGES

D.1 TRAINING STAGES

  • After DPO, OLMo-3’s residual after shared scaling halves despite a response-magnitude increase of 1.80, so the data support neither exact conservation nor a scale-only mechanism.The reported DPO-step ratios are 0.27 for OLMo-3 and 0.22 for Tülu.Source: §D.1 · p. 19, Table 9 · p. 20

E REUSE ACROSS MODELS

  • One-amplitude cross-model reuse predicts worse than shared scaling in 40 of 42 directions, significantly in 29, and never significantly better.Two-parameter reuse removes significant losses but significantly beats shared scaling in only one direction; refitting significantly beats flexible reuse in all six directions into OLMo-3.Source: E REUSE ACROSS MODELS · pp. 19–20, Table 10 · p. 21

E.1 WHERE TARGET STRUCTURE IS ESTABLISHED

  • Where target structure is established, OLMo-3 refitting significantly beats all six borrowed patterns, showing that weak target signal does not explain those failures.All 12 incoming transfers to OLMo-3 and Qwen2.5 fall below shared scaling by point estimate, ten significantly.Source: §E.1 · p. 20, Table 12 · p. 22

E.2 IS THE LOSS AN ARTIFACT OF ESTIMATION NOISE OR ATTRIBUTE LABELS?

  • Real source attribute correspondence beats permutation means in 34 of 42 directions, but only seven are significant and most significant cases still lose to shared scaling.The tested noise-injection analysis attributes only a median 8% of the observed reuse gap to source estimation noise, under its stated assumptions.Source: §E.2 · pp. 21–22

E.3 TRANSFERRING PERSONA EFFECTS DIRECTLY, WITHOUT A BASE CHECKPOINT

  • Without a target Base checkpoint, direct reuse of another model’s post-trained effects is significantly worse than target Base shared scaling in 40 of 42 directions.The exception is Tülu into Llama-3.1, whose post-trainings share a base; the reverse direction does not show the same result.Source: §E.3 · p. 22, Table 14 · p. 23

F REUSE ACROSS PROMPTS

  • After rewording, refitting beats adapted reuse in all six tested pairs; example-only changes preserve significant reuse value in Qwen3.5 and Tülu.For joint wording and example changes, refitting significantly beats shared scaling in four models, while Qwen3.5’s structure beyond shared scaling remains unresolved.Source: F REUSE ACROSS PROMPTS · p. 23, Table 15 · p. 24

F.2 A DEPLOYED MODEL WITHOUT A BASE REFERENCE

  • For a deployed model, refitting beats unchanged reuse after rewording in all six models, while uncentered source–target gain correlations of 0.97–0.99 fall to 0.15–0.63 after mean removal.Country-context reuse beats target shared scaling in all five tested models, though only OLMo-3 clears the one-point margin.Source: §F.2 · p. 25, §F.3 · p. 26, §F.4 · p. 26

G UPDATING UNDER A MEASUREMENT BUDGET

Regularized target updates offer a useful middle ground between borrowed gains and full refitting under limited measurement budgets. In the tested prompt settings, their median performance surpasses reuse strategies as the target-question budget grows.

G UPDATING UNDER A MEASUREMENT BUDGET

  • The source-shrunk update beats adapted reuse in ten of twelve settings, but shows no detected advantage over a shared-shrunk update or full refitting.The comparison separates source-pattern value from regularization toward a common gain.Source: G UPDATING UNDER A MEASUREMENT BUDGET · p. 27, Table 23 · p. 28
  • The source-shrunk update has a positive median advantage over adapted reuse at every tested budget in all twelve settings and trails refitting by at most 0.4 points from 32 questions onward.The shared-shrunk update performs similarly and is often better at small budgets.Source: G UPDATING UNDER A MEASUREMENT BUDGET · pp. 27–28
  • On the frozen questionnaire template, both prospective pairs pass the primary comparisons and show structure beyond shared scaling, but the descriptive budget recipe is met by Tülu and not OLMo-3.Later model results are extensions rather than independent prospective replications.Source: G UPDATING UNDER A MEASUREMENT BUDGET · pp. 28–29, Table 26 · p. 29
  • At 64 questions per attribute, the regularized update exceeds flexible reuse in median in all 12 settings.At 32 questions per attribute it does so in 11 of 12 settings.Source: G UPDATING UNDER A MEASUREMENT BUDGET · p. 28, Table 25 · p. 29
  • Broad attribute coverage loses at small budgets and overtakes deeper measurement of a category-stratified half inventory at a median of 16–64 questions per attribute, depending on model.The median crossover is 16 questions in OLMo-3, Tülu, and Llama-3.1; 32 in Qwen3.5; and 64 in Qwen2.5 and Qwen3.Source: G UPDATING UNDER A MEASUREMENT BUDGET · p. 29

H ROBUSTNESS AND NUMERICAL PRECISION

Robustness checks largely preserve the core pattern: OLMo-3’s structure advantage persists, while borrowed gains do not reliably improve on shared scaling. Float32 reruns and controlled preference tests qualify which comparisons remain supported.

H ROBUSTNESS AND NUMERICAL PRECISION

  • 33 of 42 transfers are significantly negative with averaged answer orders, and 28 of 42 are significantly negative on independent questions; none are significantly positive in either check.With each checkpoint’s chat interface, 29 of 42 are significantly negative and none positive; OLMo-3 retains a 3.13-point structure estimate versus 3.10 originally, while Qwen2.5’s interval includes zero.Source: H ROBUSTNESS AND NUMERICAL PRECISION · p. 31
  • Flexible reuse never significantly beats shared scaling in the chat, averaged-order, or independent-question checks, with median gains below 0.1 points in every condition.Corrected upper bounds are below one point in 37, 39, and 36 of 42 directions, respectively.Source: H ROBUSTNESS AND NUMERICAL PRECISION · p. 31, Table 29 · p. 32

H.1 INFLUENCE OF THE MOST RESPONSIVE ATTRIBUTES

  • After dropping the six most responsive attributes, Qwen2.5’s estimate remains similar at 1.05 versus 1.23 points, though its interval includes zero; OLMo-3 remains significant and Tülu becomes significant.The dropped attributes were mostly value and attitude fields.Source: §H.1 · p. 31, Table 30 · p. 32

H.2 NUMERICAL CHECKS AND THE FLOAT32 RERUN

  • In Qwen2.5’s float32 core rerun, new-question structure is 1.28 points, and refitting still significantly beats flexible reuse in the transfer direction into OLMo-3.No direction shows a significant flexible-reuse gain; the upper bound for Qwen2.5 into OLMo-3 rises to 1.001, leaving 39 rather than 40 of 42 bounds below one point.Source: §H.2 · p. 32, Table 31 · p. 33
  • In the profile study, float32 changes several Qwen2.5 comparison outcomes: insertion refitting advantages remain significant, but source-only transfer loses significance in both arms.The other five profile pairs remain in bfloat16 with batch-size checks, and the Qwen2.5 precision discrepancy is not treated as a noise floor for them.Source: §H.2 · p. 33, Table 32 · p. 33

I.1 COUNTRY CUES COMBINED WITH PREFERENCES

  • Training lowers the country-effect relevance statistic C in Tülu, Qwen2.5, and Qwen3, raises it in Qwen3.5, and produces no significant change in OLMo-3 or Llama-3.1.Unrelated preferences still move choices; a matched non-demographic-cue control leaves a demographic-specific competition mechanism unresolved.Source: §I.1 · p. 34, Table 33 · p. 34

I.2 FULL PROFILES

In full profiles, isolated-trait gains do not uniformly transfer: refitting benefits selected insertion settings, while profile effects are often small and do not establish cue competition.

I.2 FULL PROFILES

  • No replacement setting shows a significant refitting advantage, and these profile tests do not extend to unseen people because cross-validation holds identities fixed.Neither attenuation ratios nor weaker profile contrasts alone identify cue competition, length effects, or suppression.Source: §I.2 · p. 34, Table 34 · p. 35, Table 35 · p. 35
  • 12.9% and 9.0%: refitting reduces shared-scaling error for inserted traits in Qwen2.5 and Qwen3, respectively, while reductions are at most 4.4% elsewhere.Profile contrasts are small in most pairs, with post-trained RMS of 0.12 to 0.35 log odds; they approach one log odd only in Qwen2.5 and Qwen3.Source: §I.2 · pp. 34–35
  • Refitting significantly beats flexible reuse for inserted traits in Qwen2.5, Llama-3.1, Qwen3, and Qwen3.5; flexible reuse beats shared scaling in Qwen2.5 and Qwen3.No replacement comparison is significant.Source: §I.2 · p. 35

J HUMAN SURVEYS

The survey analyses separate persona responsiveness from human fit, test group-direction recovery, and examine whether calibration or output interventions change those evaluations. Their scope is historical group distributions and controlled survey settings, not individual prediction.

J HUMAN SURVEYS

  • More responsive checkpoint selection is confounded with post-training: it selects the post-trained checkpoint in all eight comparisons, while development-set human loss selects Base in seven comparisons with positive regret.Calibration removes the regret, indicating the cost lies mainly in post-trained outputs’ probability scale, but does not establish accurate simulation.Source: Table 38 · p. 37, J HUMAN SURVEYS · pp. 38–39
  • Calibrated post-trained checkpoints reach weighted cross entropy of 0.661 to 0.680 nats, closing only 12 to 29% of the gap between a 50/50 prediction and human distributions.Source: J HUMAN SURVEYS · p. 38, Table 39 · p. 38
  • Models recover some group directions, but their recovery rates remain far below the 95% human-half replication rate; post-training changes recovery significantly only in Llama-3.1.Across the other six families, post-training changes of −3.5 to +7.4 points are unresolved.Source: J HUMAN SURVEYS · pp. 38–39, Table 40 · p. 39, Table 41 · p. 39
  • 32 human-labeled questions bring all eleven evaluated checkpoints within 0.002 nats of calibration on all 84 development questions; reusing temperature across waves costs at most 0.0007 nats in the four main families.Reusing Base temperature after training adds about 0.043 nats for Qwen2.5 and 0.040 for Llama; OLMo and Tülu remain inconclusive.Source: Figure 11 · p. 37, J HUMAN SURVEYS · p. 39

J.1 PERSONA BENEFIT AND AN OUTPUT INTERVENTION

  • In the output intervention, all fourteen 99.64% intervals lie within ±0.005 nats, a tolerance equal to 33–86% of the measured persona benefits.The intervals condition on fitted source gains and fixed human targets; the intervention acts on outputs, not an internal neural pathway.Source: §J.1 · p. 40

K RELATED WORK IN MORE DETAIL

  • Related work distinguishes this paper’s predictive question—what source relationships add after target recalibration—from studies of prompt sensitivity, simulation, attribute selection, or internal persona directions.The paper’s profile edits ask how surrounding information changes one attribute’s effect; attenuating that effect alone does not make a full profile a worse simulator.Source: K RELATED WORK IN MORE DETAIL · pp. 40–41

ETHICS STATEMENT

  • Survey targets are historical group response distributions, not fixed identity properties or predictions about individuals; the experiments do not justify replacing human participation in consequential decisions.Source: ETHICS STATEMENT · p. 42

REPRODUCIBILITY STATEMENT

  • The repository provides frozen designs, question pools, cached core-experiment log odds, numerical results, analysis scripts, and result-to-file mappings; model weights and raw survey records are not redistributed.Source: REPRODUCIBILITY STATEMENT · p. 42
Loading 2609.32758v1…