Using Reinforcement Learning to Train Large Language Models to Explain Human Decisions

1 INTRODUCTION

• Neural models have improved predictive accuracy but often provide limited insight into underlying cognitive mechanisms.The authors argue that matching behavior alone offers limited explanatory power for understanding cognition.Source: §1 · p. 1
• LLM-generated chain-of-thought (CoT) is treated as a verbalized account of cognitive mechanisms that researchers can examine for meaningful explanations.The proposed approach aims to pair human-behavior prediction with interpretable natural-language accounts.Source: §1 · p. 1
• The study tests reinforcement learning (RL) with outcome-based rewards for generating CoTs and predictions of human risky choices, alongside SFT comparisons.Human choice proportions serve as the reward ground truth.Source: §1 · pp. 1–2
• RL produces sensible CoTs with predictive accuracy comparable to SFT-based methods, while its CoTs adapt to synthetic-data structure and weaken with a weaker backbone.When training data come from an expected-value maximizer, the generated reasoning reflects that structure.Source: §1 · p. 2

Leveraging LLMs to model human cognition

• LLMs can process broader, often natural-language stimuli than neural models built around abstract task representations (Zhu et al., 2025a).Recent work found that SFT can improve LLM prediction of human behavioral data.Source: §2 · p. 2
• This work aims to elicit verbal theories of human behavior with RL, extending beyond LLMs’ prior use for predicting behavior (Zhu et al., 2025b; Binz & Schulz, 2023).The example contrasts natural-language inputs with structured numerical features used by traditional and neural-network-based models.Source: §2 · p. 2

Automated discovery of cognitive models using LLMs

• A related line prompts LLMs to generate executable symbolic programs fitted to behavioral data, but typically relies on in-context learning rather than fine-tuning during search (Rmus et al., 2025; Wong et al., 2023).Code-based discovered models are interpretable, with examples involving heuristic decisions and multiarmed bandits (Rmus et al., 2025).Source: §2 · p. 2

3 METHOD

• Three post-training strategies—SFT, Centaur-style SFT, and GRPO-based RL—fine-tune identical LoRA modules on choices13k human risky-choice data.The dataset is split 90/10 by unique problem identifier, with the same fixed split used across methods and evaluation.Source: §3 · pp. 2–3
• The shared backbone is Qwen-2.5-7B-Instruct, with LoRA rank and alpha of 32 and modules applied to all linear layers.The setup has approximately 80.74M trainable parameters, or 1.05% of the 7B model.Source: §3 · p. 3
• SFT trains on JSON-formatted choice percentages rounded to integers, while Centaur-style SFT masks loss to bracketed human-data tokens and predicts aggregated responses.Standard SFT uses autoregressive next-token prediction; Centaur-style SFT applies the masked training at the problem level.Source: §3 · p. 3
• RL generates 12 candidate completions per risky-choice problem and rewards coherent option probabilities according to their absolute difference from the human Option B proportion.The outcome reward reaches 1 when the predicted Option B probability equals the observed human proportion; the predictions must sum to 1 and each lie between 0 and 1.Source: §3 · p. 4
• A format reward adds 0.25 for exactly one JSON output and another 0.25 when the prediction follows reasoning tokens.The maximum format reward is 0.5.Source: §3 · p. 4
• The GRPO advantage is each completion’s reward minus its group-average reward, and its token-level clipped objective includes a KL penalty against a reference policy.The clipping range is [0.8, 1.28], and the KL penalty coefficient β is 10^-4; RL differs from Centaur-style SFT by weighting policy updates with outcome rewards on predicted numerical values rather than tokenized outputs.Source: §3 · p. 4
• Figure 2 compares training- and test-set MSE across epochs for SFT, Centaur-style SFT, and RL, and shows final-checkpoint test MSE with error bars.The caption reports the lowest training and test MSE epochs for each method; final-checkpoint error bars are ±1 standard error across risky-choice problems.Source: Figure 2 · p. 5

Learning curves

• RL reduces prediction error faster per training example, but generates more completions and incurs substantially higher computational costs than SFT and Centaur-style SFT.All three methods show decreasing MSE on training and test sets; RL’s correctness and format rewards improve, and the format reward reaches 0.5 within the first few steps.Source: §4 · pp. 5–6, Figure 3 · p. 5
• Test-set MSEs were .0144 (SE = .0006) for SFT, .0155 (SE = .0006) for Centaur-style SFT, and .0148 (SE = .0006) for RL, with no significant differences among methods.The Mixture of Theories model achieved MSE .0113 and the neurally augmented prospect theory model .0204; both use problem features rather than text.Source: §4 · p. 6

Analyzing Chains-of-Thought

• On a held-out problem, RL changes both reasoning and predictions while preserving expected-value calculations and risk-preference considerations and amplifying references to psychological factors and biases.Figure 4 compares the model before and after RL training.Source: §4 · p. 6, Figure 4 · p. 6
• The analysis segments itemized reasoning into thoughts, embeds them in two dimensions, and identifies nine clusters labeled by summaries of centroid thoughts.Source: Figure 5 · p. 7, §4 · p. 7
• The RL model’s reasoning includes accurate expected-value calculations, coarse comparisons, psychological influences, risk preferences, and predictions conditioned partly on the expected-value difference.Source: §4 · p. 7
• Supplementary evaluations, CoT swapping, and tests of SFT-generated reasoning suggest that RL improves CoT quality, which in turn contributes to prediction accuracy.Source: §4 · p. 7

Cognitive mechanisms identified in CoT

• Expected-value computation and risk aversion each account for approximately 29% to 36% of thoughts; the model also invokes loss aversion, certainty effects, and other biases.Occasional mechanisms include risk seeking, probability weighting, framing effects, and ambiguity aversion.Source: §4 · pp. 7–8, Figure 6 · p. 8
• The model’s emphasis on expected values and risk aversion suggests that rational characterizations may merit greater attention alongside theories focused on deviations from rationality.Source: §4 · p. 8

Data control

• With synthetic rewards from an expected-value maximization model, RL quickly focuses reasoning on expected-value computation and comparison, adapting its strategies to the training data.Risk aversion still appears occasionally, which the authors suggest may reflect residual prompt cues about human risky choice.Source: §5 · p. 8

Model control

• With the smaller Gemma-2-2B-Instruct backbone, RL performs worse than both SFT methods on test MSE and fails to compute and compare expected values in its reasoning.SFT and Centaur-style SFT still predict human behavior at a level comparable to the stronger Qwen model, although without explanations.Source: §5 · pp. 8–9

RLVR, RLHF, and RLAIF

• RL with outcome-based rewards elicits chains of thought that act as verbal theories, but requires more computation than supervised fine-tuning.Expert-rated reinforcement learning would require large-scale judgments of both explanations and predictions, which the authors say are not currently available.Source: §6 · p. 9

Training predictive models and then explaining their predictions

• Unlike predict-then-explain approaches, RL simultaneously optimizes predictions and explanations and avoids direct explainability techniques that can be brittle for feature attribution.The discussion describes predict-then-explain as a promising alternative.Source: §6 · p. 9
• LLM-proposed hypotheses automate the search for explanatory signals but depend heavily on the base model’s capacity; human-proposed hypotheses may sometimes be more creative.Traditional explainability relies on features proposed by human researchers.Source: §6 · p. 9

Limitations and future research

• Under the elicitation hypothesis, RL is unlikely to generate entirely novel theories absent from the backbone, though it may apply embedded ideas from other disciplines to explain psychological phenomena.The authors give prospect theory as an example of a theory RL alone would likely not discover if it were absent from pretraining.Source: §6 · p. 10
• RL and supervised fine-tuning appear to have different generalization patterns, but the conditions favoring each method and how to combine them remain open questions.Source: §6 · p. 10

Conclusion

• RL-post-trained reasoning LLMs can generate meaningful explanations of human risky choices, but their effectiveness depends strongly on backbone capabilities.The authors present the approach as promising for other cognitive-modeling domains.Source: §6 · p. 10

• Princeton University’s IRB approved the human experiment, and all participants provided informed consent before taking part.Participants evaluated model completions on risky-choice problems; the authors anticipate no additional ethical concerns.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• The authors report complete descriptions of training methods and associated hyperparameter values, with training conducted using Hugging Face and vLLM packages.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 PROMPTS FOR THE SFT AND CENTAUR-STYLE SFT MODELS

• The SFT and Centaur-style SFT prompt asks for final JSON estimates of the proportions choosing options A and B, summing to 100.Source: §A.1 · p. 14

A.2 PROMPTS FOR THE RL MODEL

• The RL prompt adds a request to explain reasoning step by step before providing the same final JSON estimates.Source: §A.2 · p. 14

A.3 EXAMPLE PROMPTS FOR RISKY OPTIONS

• Risky-choice prompts describe options in natural language, with examples pairing a 50% chance of $2 and a 50% chance of $0 against a sure $1.Source: §A.3 · p. 14

B COGNITIVE MECHANISMS IDENTIFIED BY QWEN-2.5-7B-INSTRUCT AFTER RL TRAINING

• To identify mechanisms, the authors used GPT-4.1 to map thought atoms from RL-model chains of thought to standard psychological effects or cognitive biases.The prompt instructed GPT-4.1 to return only the most relevant established concepts as a JSON list.Source: B COGNITIVE MECHANISMS IDENTIFIED BY… · p. 14
• The section also lists additional mechanisms identified in the reasoning traces beyond those summarized in Table 1.The additional labels include concepts such as anchoring, risk neutrality, and probability weighting.Source: B COGNITIVE MECHANISMS IDENTIFIED BY… · pp. 14–15
• Table 1 reports frequencies of mechanisms identified in final-checkpoint RL-model reasoning, including only mechanisms found in at least 10 risky-choice problems.Source: B COGNITIVE MECHANISMS IDENTIFIED BY… · p. 14, Table 1 · p. 15

C.1 HUMAN EVALUATION OF COT

• RL CoTs were selected as more reasonable in 61.5% of trials (SE = 5.2%), above the 50% chance level.Twenty participants each evaluated 10 randomly sampled held-out problems, choosing between anonymized backbone and RL CoTs without seeing final predictions.Source: §C.1 · p. 16, Figure 7 · p. 16

C.2 USING GPT-4.1 TO JUDGE COT

• GPT-4.1 ratings rose modestly at first, stabilized, then declined slightly, suggesting judged CoT quality largely plateaued after the first RL epoch.The ratings used a 0-to-100 scale.Source: §C.2 · p. 17, Figure 8 · p. 17

C.3 SWAPPING COT BETWEEN BACKBONE AND RL MODELS

• Using the RL model’s CoT significantly reduced backbone prediction error, while using the backbone’s CoT significantly increased RL prediction error.The comparison used mean squared error on the test set; Table 2 describes the original- and swapped-CoT conditions.Source: §C.3 · p. 17, Table 2 · p. 18

C.4 TESTING THE COTS GENERATED BY SFT AND CENTAUR MODELS

• Neither SFT-generated nor Centaur-generated CoTs improved backbone predictions: their MSEs were 0.0785 (SE = 0.0023) and 0.0728 (SE = 0.0019), respectively.Both were comparable to the backbone using its original CoT; SFT and Centaur CoT frequencies showed no noticeable changes across epochs, though expected-value usage trended upward for SFT and downward for Centaur.Source: §C.4 · p. 18, Figure 9 · p. 18

D.1 LEARNING CURVES

• In the synthetic-data control, the Qwen RL model’s performance gradually improved over training steps.The control replaced human choice data with synthetic choices generated by an expected-value maximization model.Source: §D.1 · p. 19, Figure 10 · p. 19

D.2 COGNITIVE MECHANISMS IDENTIFIED BY QWEN-2.5-7B-INSTRUCT DURING THE DATA CONTROL EXPERIMENT

• With synthetic data from an expected-value maximizer, expected-value computation became the dominant mechanism in the RL model’s CoT reasoning.The control retained the Qwen-2.5-7B-Instruct backbone and used the main experiment’s RL hyperparameters; the learning curve gradually improved.Source: §D.2 · p. 19, §D.1 · p. 19, Figure 10 · p. 19

D.3 ADDITIONAL DATA CONTROL EXPERIMENTS

• On the Complexity Aversion Dataset, correctness reward rose toward 0.97, and complexity-aversion thoughts rose from around rank 16 to around rank 6 during RL training.The synthetic choice rates probabilistically favored the option with fewer outcomes, regardless of probabilities or payoffs; CoTs also retained some risky-choice framing, likely due to the prompt.Source: §D.3 · pp. 19–20, Figure 11 · p. 20
• On the Random Dataset, the model collapsed to predicting 50% choice rates and produced nonsensical CoTs, often multilingual gibberish.The Random Dataset used randomly generated choice rates sampled from U[0, 1].Source: §D.3 · pp. 19–20

E.1 LEARNING CURVES

• Test-set MSE was 0.0162 (SE = 0.0006) for SFT and 0.0163 (SE = 0.0007) for Centaur-style SFT, versus 0.0526 (SE = 0.0016) for RL with Gemma-2-2B-Instruct.SFT and Centaur-style SFT did not differ significantly; RL had significantly higher MSE.Source: §E.1 · p. 20

E.2 COGNITIVE MECHANISMS IDENTIFIED BY GEMMA-2-2B-INSTRUCT

• The Gemma RL CoTs lacked expected-value calculation and comparison, which the authors suggest likely contributed to the model’s poor risky-choice predictions.The model performed worse than both SFT-based methods using the same backbone and RL using the stronger Qwen model.Source: §E.2 · p. 20, Figure 14 · p. 21, Figure 12 · p. 21

F ADDITIONAL DISCUSSION

• Original GRPO on choices13k caused mode collapse; removing advantage normalization addressed the issue, and one minus absolute prediction error yielded the most stable tested reward.The authors suggest the dataset’s modal 50% human choice proportion may have contributed to collapse; the selected reward had smoother KL trajectories and smaller reward standard deviations than the other tested rewards.Source: F ADDITIONAL DISCUSSION · p. 22

G IMPLEMENTATION DETAILS

• RL training took 80 H100 GPU hours for 7B models and 40 for 2B models, compared with approximately 5 and 3 A100 hours per SFT training, respectively.Pilot and control RL runs together used approximately 2,500 H100 GPU hours.Source: G IMPLEMENTATION DETAILS · pp. 22–23
• On a single A100 GPU using vLLM, complete choices13k inference across all checkpoints took approximately 5 hours for SFT/Centaur-style SFT versus roughly 30 for RL, and 3 versus 20 hours, respectively.Source: G IMPLEMENTATION DETAILS · p. 23