PaperlayerView exact revision on arXiv

Source-linked AI summary

Post-training makes large language models less human-like

Marcel Binz, Elif Akata, Abdullah Almaatouq, Mohammed Alsobay, Oleksii Ariasov, Franziska Brändle, David Broska, Jason W. Burton, Nuno Busch, Frederick Callaway, Vanessa Cheung, Brian Christian, Julian Coda-Forno, Can Demircan, Vittoria Dentella, Maria K. Eckstein, Noémi Éltető, Michael Franke, Thomas L. Griffiths, Fritz Günther, Susanne Haridi, Sebastian Hellmann, Stefan Herytash, Linus Hof, Eleanor Holton, Isabelle Hoxha, Zak Hussain, Akshay Jagadish, Elif Kara, Valentin Kriegmair, Evelina Leivada, Li Ji-An, Tobias Ludwig, Maximilian Maier, Marcelo G. Mattar, Marvin Mathony, Alireza Modirshanechi, Robin Na, Mariia Nadverniuk, Antonios Nasioulas, Surabhi S. Nath, Helen Niemeyer, Kate Nussenbaum, Sebastian Olschewski, Thorsten Pachur, Stefano Palminteri, Aliona Petrenco, Camille V. Phaneuf-Hadd, Angelo Pirrone, Manuel Rausch, Laura Raveling, Shashank Reddy, Milena Rmus, Evan M. Russek, Tankred Saanum, Kai Sandbrink, Louis Schiekiera, Johannes A. Schubert, Luca M. Schulze Buschoff, Nishad Singhi, Leah H. Somerville, Mikhail S. Spektor, Xin Sui, Christopher Summerfield, Mirko Thalmann, Anna I. Thoma, Taisiia Tikhomirova, Vuong Truong, Polina Tsvilodub, Konstantinos Voudouris, Kristin Witte, Shuchen Wu, Dirk U. Wulff, Hua-Dong Xiong, Songlin Xu, Lance Ying, Xinyu Zhang, Jian-Qiao Zhu, Eric Schulz

arXiv:2605.07632v2cs.CLcs.AIcs.LG

TL;DR

It is unclear how behavioral alignment changes across model generations and whether persona-induction helps model individual behavior. Using Psych-201 to evaluate alignment, the study finds that post-training reduces alignment broadly and that the gap widens in newer generations even as base models improve.

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used as surrogates for human participants, but it remains unclear which models best capture human behavior and why. To address this, we introduce Psych-201, a novel dataset that enables us to measure behavioral alignment at scale. We find that post-training -- the stage that turns base models into useful assistants -- consistently reduces alignment with human behavior across model families, sizes, and objectives. Moreover, this misalignment widens in newer model generations even as base models continue to improve. Finally, we find that persona-induction -- a popular technique for eliciting human-like behavior by conditioning models on participant-specific information -- does not improve predictions at the level of individuals. Taken together, our results suggest that the very processes that are currently employed to turn LLMs into useful assistants also make them less accurate models of human behavior.

Psych-201

This section uses Psych-201 to evaluate behavioral alignment between language models and human responses across model families, sizes, post-training objectives, and generations. It examines how post-training relates to alignment with human behavior.

Psych-201

  • Psych-201 is a transcript-based dataset with 208,021 participants, 25,906,599 responses, and hundreds of experiments, 3.5× larger than Psych-101 and 13× larger than the average mega-study.It also covers broader experimental paradigms and participant demographics than Psych-101.Source: Psych-201 · p. 5, Figure 1 · p. 6

Mapping behavioral alignment across experimental domains

No benefit of persona-induction

  • Persona-induction had only a minor metadata benefit, ranging from d=−0.02 for Llama3.2-3B-Instruct to d=0.02 for Llama3.1-70B; this held for base and instruction-tuned models and developmental experiments.Source: No benefit of persona-induction · p. 10

Discussion

  • Post-training reduces behavioral alignment across model families, sizes, and objectives, while misalignment grows in newer generations despite continued improvement in base models.Source: Discussion · p. 11
  • Psych-201, assembled through an open collaboration across behavioral sciences, provides a resource of unprecedented scale for evaluating behavioral alignment.Source: Discussion · pp. 11–12
  • The findings extend prior evidence that instruction-tuned models produce less human-like text [36] to behavioral responses across a broad range of post-training objectives.Source: Discussion · pp. 11–12
  • Existing methods proposed to mitigate post-training capability losses on common benchmarks do not extend to behavioral alignment with humans.Source: Discussion · p. 12
  • Average d=0.28, SEM=0.10: Centaur, fine-tuned on Psych-201 tasks previously included in Psych-101, improved behavioral alignment on held-out novel tasks.Source: Discussion · p. 12
  • Future methods should preserve base models’ behavioral fidelity while capturing human response stochasticity and individual variability; Psych-201’s participant-level metadata may support this goal.Source: Discussion · p. 12

Materials and Methods

The methods describe Psych-201 data collection, evaluation across three model families, and the metrics used to assess behavioral alignment. Supplementary figures compare chat-template prompting and post-training objectives, while the final subsections identify dataset and code availability.

Data collection

  • Researchers submitted datasets through an open call; prompts preserved trial-by-trial participant-session histories, and 10% of participants (up to 100) were held out for testing before a lightweight review.The review checked for obvious formatting or implementation bugs.Source: Data collection · p. 28

Large language models

  • Evaluations covered Qwen3, Llama3.X, and Olmo3.X using Hugging Face Transformers in bfloat16, with no prompt template because that setting performed best.This setting was also used for models that provide a template; default model-specific templates were evaluated separately.Source: Large language models · p. 28

Evaluation metrics

  • Average negative log-likelihood over human-response spans was the primary metric; post-training misalignment was quantified as mean Cohen’s d between base and post-trained models.For multi-token responses, token log-likelihoods were summed before averaging across responses.Source: Evaluation metrics · p. 28

Data availability

Code availability

  • Default chat templates yield worse human-response negative log-likelihood than no template for Qwen3, Olmo3.X, and Llama3.X, with the largest gap in Qwen3.The figure compares the models with and without their default chat templates.Source: Figure S1 · p. 30, Code availability · p. 30
  • Across objectives, families, and sizes, post-trained models show lower alignment than their base counterparts on Psych-201’s discrete-choice subset.The figure compares accuracy when predicting human responses.Source: Figure S2 · p. 31, Code availability · p. 31

Study List (excluding Psych-101)

This section catalogs studies excluding Psych-101 across learning, choice, reasoning, language, memory, perception, and social decision-making, organized by topic rather than presented as findings.

Reinforcement learning

  • Reinforcement-learning studies cover bandits, exploration, reward and reversal learning, two-step tasks, and learning from outcomes.Examples include a cross-cultural bandit task, a two-step task, and learning from good versus bad outcomes.Source: Study List (excluding Psych-101) · pp. 31, 32, 34

Other behavioral domains

  • Other entries span reasoning, psycholinguistics, memory, psychophysics, categorization, cognitive control, and social or moral judgment.Examples include causal structure learning, pragmatic inference, recognition memory, brightness discrimination, and the Moral Machine dilemmas.Source: Study List (excluding Psych-101) · pp. 31–33

Choice

  • Risky and intertemporal choice studies examine lottery decisions, information sampling, sequential risk, and delay discounting.The list also includes social influence on risk-taking and risky choice with feedback.Source: Study List (excluding Psych-101) · pp. 32–33

Study List (excluding Psych-101)

Loading 2605.07632v2…