Source-linked AI summary
Alignment Makes Language Models Normative, Not Descriptive
Eilam Shapira, Moshe Tennenholtz, Roi Reichart
TL;DR
Whether alignment helps language models predict human behavior, rather than merely produce preferred responses, remains unclear. The paper compares base and aligned models across strategic and simpler decisions, finding a strong boundary-condition reversal. Alignment helps where human behavior follows normative predictions but hurts prediction in history-dependent strategic interactions.
Problem
Prior behavioral modeling nearly always uses aligned models, leaving open whether alignment preserves fidelity to observed human strategic behavior.
Method
The paper compares 120 same-provider base–aligned model pairs on 10,050 human decisions in multi-round strategic games and boundary-condition tasks.
Results
Base models outperform aligned models 9.7:1 in multi-round games, while aligned models win 4.1:1 in one-shot games and 2.2:1 on lotteries.
Takeaways & Limitations
Alignment creates a normative–descriptive trade-off: aligned models suit normative or human-use settings, whereas base models better represent observed behavior in interactive strategic settings.
Takeaways & Limitations
The analysis uses only binary or ternary decisions and open-weight models, so extension to continuous actions and closed-source models remains open.
Abstract
from arXiv · showhide
Post-training alignment optimizes language models to match human preference signals, but this objective is not equivalent to modeling observed human behavior. We compare 120 base-aligned model pairs on more than 10,000 real human decisions in multi-round strategic games - bargaining, persuasion, negotiation, and repeated matrix games. In these settings, base models outperform their aligned counterparts in predicting human choices by nearly 10:1, robustly across model families, prompt formulations, and game configurations. This pattern reverses, however, in settings where human behavior is more likely to follow normative predictions: aligned models dominate on one-shot textbook games across all 12 types tested and on non-strategic lottery choices - and even within the multi-round games themselves, at round one, before interaction history develops. This boundary-condition pattern suggests that alignment induces a normative bias: it improves prediction when human behavior is relatively well captured by normative solutions, but hurts prediction in multi-round strategic settings, where behavior is shaped by descriptive dynamics such as reciprocity, retaliation, and history-dependent adaptation. These results reveal a fundamental trade-off between optimizing models for human use and using them as proxies for human behavior.
1 Introduction
The paper tests whether alignment improves models as proxies for human behavior, comparing base and aligned models across strategic and simpler decision settings. Base models predict human choices much better in multi-round strategic games, while aligned models perform better where normative predictions may suffice.
- Motivation and approach: Base models can predict observed behavior better than aligned models even though aligned models are commonly used as behavioral proxies.Prior work largely treated alignment as neutral or beneficial for behavioral prediction.
- Interpretation: Alignment may improve behavioral prediction where human choices follow normative theory but create a normative bias in history-dependent strategic behavior.Strategic behavior can involve bluffing, retaliation, reciprocity, and deviations from approved patterns.
- Motivation and approach: 120 same-provider base–aligned model pairs are evaluated on 10,050 real human decisions across bargaining, persuasion, negotiation, and repeated matrix games.Native formats are used: standard text completion for base models and chat-templated input for aligned models.
- Main result: 9.7:1 is the base-model advantage in multi-round games, with 213 vs. 22 wins and p < 10−40.The effect holds across all 23 model families, 10 prompt formulations, and game configuration parameters.
- Boundary conditions: Aligned models reverse the advantage in one-shot matrix games and binary lotteries, winning 4.1:1 and 2.2:1, respectively.The one-shot result is consistent across all 12 game types.
2 Related Work
Prior work uses aligned language models to reproduce human findings and study strategic behavior, but has not systematically tested whether pre-alignment models predict human choices better. This paper addresses that comparison while distinguishing behavioral prediction from strategic play.
- Behavioral modeling: Most prior LLM behavioral-modeling work uses aligned models and implicitly treats alignment as neutral for behavioral fidelity.Earlier studies examined experimental findings, subgroup opinions, and treatment effects.
- The alignment tax: Distributional narrowing and preference collapse provide theoretical reasons alignment may reduce coverage of the full range of human behavior.Prior work measured reduced diversity and benchmark or generation-quality costs, not behavioral prediction fidelity.
- Strategic games: Playing games and predicting human game choices are distinct tasks because equilibrium behavior can differ from observed human behavior.This paper predicts human choice distributions from token probabilities rather than evaluating generated play.
- Contribution: The paper fills an open gap with the first systematic base-versus-aligned comparison across 120 same-provider pairs and four game families.This extends prior LLM prediction work that used only aligned models.
3 Experimental Setup
The experiments compare base and aligned models on interactive strategic games, one-shot matrix games, and binary lotteries. Predictions are extracted deterministically from next-token probabilities and evaluated against human decision distributions.
- Datasets and games: The study covers bargaining, persuasion, negotiation, repeated Prisoner’s Dilemma, and repeated Battle of the Sexes games with varying information, complexity, and interaction length.The GLEE games involve human participants interacting with LLM opponents, while repeated matrix-game participants face pre-computed strategies derived from GPT-4.
- Datasets and games: 10,050 human decisions per model are evaluated across the four main game families, producing over 2.4 million total predictions.Negotiation decisions include three possible actions, while the other listed families use two decision tokens.
- Prediction method: For each decision point, a single forward pass extracts log-probabilities for the available decision tokens from the dialogue history.The method uses no generation or sampling and applies to both base and aligned models.
- Prediction method: The normalized paccept probability is computed over the family’s decision tokens, excluding probability mass assigned to non-decision tokens.Two-token families include bargaining, persuasion, and matrix games; negotiation has three tokens.
- Controls: Four prompt variants cross model type with native or cross-applied formats to separate model type from prompt-format effects.The main comparison uses native formats; additional variants apply chat formatting to base models or plain formatting to aligned models.
- Evaluation: Pearson correlation between predicted probabilities and ground-truth human behavior is the primary evaluation metric.The main games contain unique dialogue histories at each decision point, allowing decision-level correlation.
4 Results
Base models consistently outperform aligned models in multi-round strategic prediction, and the advantage survives prompt and configuration controls. The advantage reverses at round one and in simpler one-shot settings, while scaling with model size.
- Main comparison: Base models win 213 of 235 valid comparisons across four game families, a 9.7:1 advantage that is significant in every family.The seven largest model families also show majority base wins in every family.
- Prompt-format controls: Base models still win 5.0:1 with identical plain-text prompts and 5.3:1 with the aligned model’s chat template.These controls place the advantage in model weights rather than prompt format.
- Configuration robustness: The base advantage holds across every tested game-configuration value, becoming stronger in persuasion when the seller knows product quality.The informed condition yields a 14.5:1 advantage versus 2.3:1 when the seller is uninformed.
- Round-by-round dynamics: Aligned models win in round one, but base models dominate from round two onward in bargaining, negotiation, and persuasion.The transition links the base advantage to accumulated interaction history rather than game structure alone.
- Size scaling: The base advantage increases with model scale in bargaining, negotiation, and matrix games.Bargaining rises from +0.22 below 3B parameters to +0.36 at 14B or more.
5 Boundary Conditions
The base-model advantage reverses in one-shot games and lotteries, where aligned models better match human choices and normative predictions. This reversal is also consistent with the advantage being tied to accumulated interaction history rather than strategic structure alone.
- Size scaling: Figure 2 shows a positive base-minus-aligned correlation difference across every model-size bin, with the advantage growing as models scale.The figure reports 95% bootstrap confidence intervals from 5,000 resamples.
- One-shot games: Aligned models win 57 comparisons to base models’ 14 in one-shot games, a 4.1:1 advantage significant across all 12 game types.The benchmark contains 2,416 procedurally generated games and approximately 93,000 aggregated human decisions.
- Normative alignment: Aligned models are closer to Nash equilibrium than base models, with mean r = 0.41 versus 0.28 and aligned models closer in 59 of 76 filtered pairs.Human aggregate choices correlate with Nash equilibrium predictions at r = 0.616.
- Lotteries: Aligned models win 62:28 on non-strategic lotteries, a 2.2:1 advantage with p = 2.19 × 10^-4.The lottery dataset contains 1,001 binary choice problems evaluated with 90 same-provider pairs.
6 Discussion and Conclusion
The experiments identify a normative bias in aligned models: alignment helps predict behavior that follows normative theory but hurts prediction of history-dependent strategic behavior. This creates practical and scientific consequences for choosing models as behavioral proxies, while leaving open questions about generality and mechanism.
- Discussion: Alignment improves prediction in one-shot and non-strategic settings but degrades prediction in multi-round strategic interactions where behavior diverges from norms.The paper contrasts normative patterns with bluffing, retaliation, and history-dependent deviations from cooperation.
- Interpretation: Selective aligned-model wins argue against uniform capability degradation: alignment shifts expressed behavioral patterns while preserving relevant knowledge.Aligned models win selectively on one-shot games and lotteries rather than underperforming across all settings.
- Mechanism: KL-regularized reward maximization concentrates probability on annotator-approved modes, suppressing behavioral tails associated with reciprocity, retaliation, and reputation dynamics.The paper presents this distributional narrowing as behavioral evidence for preference collapse and its effect on predictive fidelity.
- Implications: Researchers using aligned models as simulated participants may measure normative behavior rather than actual human behavior when norms and behavior diverge.This methodological risk extends to simulations of voters, consumers, and social-media users.
- Open questions: The study leaves open whether opponent modeling, history integration, or trajectory novelty drives the multi-round base advantage, and whether findings generalize to auctions or coalition formation.Testing extreme scale could also clarify whether the normative shift persists or diminishes as models become more capable.
- Conclusion: Current alignment methods may impose a normative–descriptive trade-off because a single annotator-preference reward cannot preserve the full distribution of human behavior.The paper frames base-versus-aligned selection as a substantive modeling assumption rather than a formatting decision.
Limitations
The study’s evidence is constrained by its datasets, action spaces, model availability, and boundary-condition comparison. Prompt-format failures also limit which base-model comparisons remain valid.
- GLEE participants played against LLM opponents rather than other humans, although uncertainty about opponent identity may mitigate altered behavior.
- The analysis covers only binary or ternary decisions, leaving continuous action spaces untested.
- All 120 model pairs are open-weight, so closed-source models without available base versions are excluded.
- The one-shot boundary condition uses a different dataset, though round-one results within the multi-round games provide convergent evidence.
- Unstructured prompt variants reduce decision-token mass below 10% or leave too few valid pairs after filtering.
C Filtering Criteria and Sensitivity
The analysis applies pair-level mass and predictive-correlation filters independently within each game family, and sensitivity checks show the base-model advantage is robust to threshold choices.
- Filtering Criteria: The mass filter excludes a family when either model’s average decision-token probability mass falls below 0.8.This prevents unreliable normalized probabilities from entering comparisons.
- Filtering Criteria: The correlation filter excludes a pair only when both models correlate below 0.3 with binary human decisions.Pairs are retained when at least one model exceeds the threshold.
- Filtering Criteria: Filters are applied independently by game family, so exclusion from one family does not remove a pair from every analysis.
- Sensitivity Analysis: Base models win the majority of comparisons in every sensitivity-analysis cell, with p < 0.05 across threshold choices.The selected thresholds are mass ≥0.8 and correlation ≥0.3.
D Per-Pair Prediction Results
The appendix reports decision-token mass and Pearson correlation with human decisions for every same-provider pair across the main game families and boundary-condition datasets.
- Tables 10–11 list average decision-token mass and Pearson correlation for each same-provider pair across six datasets.The six datasets comprise four main game families and two boundary-condition datasets.
E Game Configuration Robustness
The game-configuration analysis reports base-versus-aligned win counts across parameter values, finding a consistent base advantage across configurations and families except for one bargaining condition.
- Tables 13–16 report base-versus-aligned win counts for every parameter value in each game family.They also report valid pairs after filtering and pairs excluded by the filters.
- The base advantage is consistent across all parameter values and game families, with one stated exception in bargaining with discounting.
Appendix Tables
The appendix catalogs the model pairs, prompt variants, prediction-result tables, game configurations, and sensitivity analyses used to evaluate base versus aligned models.
- Model pairs: Table 3 lists all base–aligned pairs, grouped alphabetically by family and sorted by parameter count within each family.
- Prompt variants: Table 4 lists 14 prompt variants, each modifying the reference JSON completion format and marking whether sufficient valid pairs remain after filtering.
- Sensitivity analyses: Tables 5–9 provide sensitivity analyses across mass and minimum-correlation thresholds for bargaining, persuasion, negotiation, matrix games, and all four families overall.
- Prediction results: Tables 10 and 11 report per-pair prediction results for strategic games, one-shot matrix games, and binary lotteries.
- Game configurations: Table 12 specifies game configuration parameters for each GLEE family.
- Configuration analyses: Tables 13–16 compare base and aligned wins across configuration parameters and repeated-game round phases, reporting valid pairs, exclusions, and one-sided binomial p-values.