Finetuning LLMs for Human Behavior Prediction in Social Science Experiments

1 Introduction

• Prior LLM simulations can distort opinion distributions, overestimate experimental effect sizes by 2 to 10 times, mispredict effect directions 10–32% of the time, and flatten demographic variation.These errors limit the viability of LLM simulations for social science experiments; prior fine-tuning work had demonstrated generalization on specific tasks such as cognitive science and public opinion.Source: §1 · pp. 1–2
• 30% for LLaMa3-8B and 26% for Qwen2.5-14B: fine-tuning improves alignment with human response distributions in unseen studies relative to the base models.The evaluation compares supervised fine-tuning, reasoning-trace augmentation, and contrastive preference optimization with prompting baselines; supervised fine-tuning improves distributional alignment, while contrastive preference optimization achieves the best individual-response accuracy.Source: §1 · p. 2
• SOCSCI210 comprises 2.9 million individual responses from more than 400,000 participants across 210 studies, with demographic reporting and coverage across multiple disciplines.The dataset standardizes study data into text describing demographic profiles, experimental questions, and recorded responses.Source: §1 · p. 2
• SOCRATES-LLAMA-8B and SOCRATES-QWEN-14B align 12.1% and 13.2% better with human response distributions than GPT-4o, respectively.The paper reports these gains for outcomes under diverse experimental conditions and says they reflect 26+% gains relative to the models’ base versions.Source: Figure 1 · p. 1, §1 · p. 2
• Fine-tuning on as little as 10% of an experiment’s data reduces prediction error by 13% on unseen participants; training on subsets of conditions or outcomes boosts generalization to unseen conditions by 71% and unseen outcomes by 49%.These results demonstrate generalization across participants, experimental conditions, and outcome questions.Source: §1 · p. 2

2 Related Work

• Existing behavioral datasets do not yet capture the full diversity of social science disciplines at a granular, individual level.Prior work includes aggregate public-opinion datasets and individual decision-making datasets spanning cognitive experiments, text reactions, web actions, and risky choices.Source: §2 · pp. 2–3
• SOCSCI210 expands topic coverage while retaining individual-level responses, a combination the paper identifies as enabling a shared behavioral model across social science contexts.The projected question embeddings are described as showing broader topic diversity than SubPop and Psych101.Source: §2 · p. 3, Figure 2 · p. 3
• Prior fine-tuning methods include supervised learning and preference-based approaches such as RLHF, DPO, and SimPO, which use preferred and dispreferred response pairs.The passage also describes reasoning-model training and distillation from teacher-generated reasoning traces.Source: §2 · p. 3
• Prior behavior-prediction fine-tuning includes direct SFT, reasoning-augmented SFT, and GRPO; this work comparatively evaluates SFT and DPO.The paper contrasts the scenarios in which SFT and DPO are most effective.Source: §2 · p. 3

3.1 Task Description

• Studies use ordinal or binary outcome questions, and the dataset restricts evaluation to these response types with minimal data-size reduction.The studies can use between-subject or within-subject designs; participants’ original condition assignments are retained rather than randomized again.Source: §3.1 · p. 4
• The model predicts an individual response from persona attributes, an experimental condition, and an outcome question.Each participant contributes responses associated with condition–outcome stimuli.Source: §3.1 · pp. 3–4

3.2 Evaluation

• Individual-response accuracy is normalized using the response-scale bounds for each condition and outcome, then averaged across studies.The bounds are the minimum and maximum values in the ground-truth responses.Source: §3.2 · p. 4
• Because the same demographic may respond differently to identical stimuli, the response function is treated as a distribution, and distributional alignment is prioritized over accuracy alone.Experiments use outcome-response distributions under conditions to derive end statistics.Source: §3.2 · pp. 4–5
• For each condition–outcome pair, predicted and empirical response distributions are standardized to [0, 1] and compared using Wasserstein distance.The distance estimates distribution shape and mean; lower values indicate better alignment.Source: §3.2 · p. 5

3.3 Dataset Construction

• The studies come from TESS, a repository of peer-reviewed, nationally representative, high-powered experiments across social-science disciplines.SOCSCI210 studies had a mean of 1,907 and median of 1,954.5 participants.Source: §3.3 · p. 5
• An automated data-construction agent parses source studies into persona, stimulus, and response records, successfully reconstructing 210 studies.Stimuli are tagged by condition and outcome question.Source: §3.3 · p. 5
• SOCSCI210 contains 2.9 million responses from 400,491 participants across 1,197 outcomes, 1,194 conditions, and 5,998 unique stimuli.The dataset spans multiple disciplines and includes five times as many individuals as prior human-behavior finetuning work.Source: §3.3 · p. 5

Finetuning Methods

• The finetuning comparison includes supervised finetuning (SFT), SFT with oracle reasoning traces, and contrastive preference tuning via DPO.The oracle traces explain human decisions from a social scientist’s perspective and are generated with GPT-4o-mini due to practical cost constraints.Source: §4 · p. 5

Supervised Finetuning (SFT)

• SFT prompts combine a persona, condition, and outcome question, and training minimizes the negative log-likelihood of the ground-truth response.The objective is cross-entropy over the observed response.Source: §4 · p. 5

Augmentation with Oracle Reasoning Traces

• Oracle reasoning traces are added to the target output as explicit rationales for the corresponding human response.GPT-4o-mini generates the traces from the prompt and response.Source: §4 · p. 5

Contrastive Finetuning via Preference Optimization

• DPO training contrasts responses by varying persona, condition, or outcome; demographic pairs hold condition and outcome fixed and prefer the response matching the focal persona.The contrasting persona is sampled to have a different response.Source: §4 · p. 5

5.1 Training Configurations

• The experiments train LLaMA3-8B-Instruct and Qwen2.5-14B-Instruct as representative open-source base models of different sizes.Training details are provided in §C.Source: §5.1 · p. 6

5.2 Baselines

• Baselines include direct, reasoning-trace, and few-shot prompting; the study also uses empirical-best and uniform-guess bounds and reports relative change versus a baseline.Positive relative change indicates improvement and negative change indicates regression.Source: §5.2 · p. 6

5.3 Generalization Across Unseen Studies

• SFT beats GPT-4o by 12.1% (LLaMA3-8B) and 13.2% (Qwen2.5-14B), with respective fine-tuning gains of 30.1% and 26.3%; Qwen’s best score is 0.151.The empirical-best bound for distribution alignment is 0.125.Source: §5.3 · pp. 6–7, Table 2 · p. 7

5.4 Generalization to Unseen Conditions and Outcomes

• 71% relative improvement in response-distribution estimation on held-out conditions; unseen-outcome distribution distance improves 49% versus the LLaMA3-8B base.SFT also raises individual accuracy from 71.0% to 74.2%; generalization across unseen conditions tends to exceed generalization across unseen outcomes.Source: §5.4 · pp. 7–8, Table 3 · p. 8

5.5 Generalization to Unseen Participants

• With 10% participant data, contrastive DPO raises seen-study accuracy from 71% to 75%; learning saturates around 10%, while SFT better aligns distributions.In unseen studies, DPO also surpasses GPT-4o, while reasoning traces provide little benefit.Source: §5.5 · p. 8, Figure 4 · p. 9

5.6 Demographic Bias and Parity

• Fine-tuning improves distribution alignment across demographic categories by 28.5% on average and reduces demographic parity difference by 10.6%.Parity difference is the absolute gap between the highest- and lowest-performing demographic subgroups.Source: §5.6 · p. 9

6 Conclusion

• Socrates-Qwen-14B produces predictions 13% more distributionally aligned with human responses than GPT-4o.The authors recommend that researchers fine-tune on existing datasets to yield more accurate and useful simulations.Source: §6 · p. 9
• SOCSCI210 comprises 2.9 million individual responses from more than 400,000 participants across 210 social science experiments.The paper presents fine-tuning as a way to create a general-use behavioral model for predicting individual responses.Source: §6 · p. 9
• The authors release SOCSCI210, the SOCRATES models, and fine-tuning code as a foundation for a unified behavioral prediction engine across disciplines.Source: §6 · p. 9

Limitations

• Participant-data sweeps plateau quickly, likely because the models have relatively small parameter counts; the authors anticipate larger models could improve performance.Source: Limitations · p. 9
• Reasoning traces come from GPT-4o-mini, and SFT on those traces does not always improve performance on this task.The authors suggest investigating traces from more powerful reasoning models.Source: Limitations · p. 9
• Training uses SOCSCI210, which covers representative U.S. samples and closed-form questions; generalization to non-U.S. populations and open-form questions is unevaluated.Source: Limitations · p. 9

• All materials were collected and processed under the respective data, checkpoint, and API usage policies.Source: Ethics Statement · p. 10
• The dataset draws on publicly available, peer-reviewed NSF TESS experiments that comply with established ethical and privacy standards.Source: Ethics Statement · p. 10
• Because stimuli may be contentious and models may produce incorrect or unsafe content, users are advised to verify outputs before real-world deployment.The authors also caution that improved accuracy could lead users to become overconfident.Source: Ethics Statement · p. 10

A Data Reconstruction Agent

• The agent uses each paper’s context, data files, codebook, and stimuli to identify conditions and outcome questions, then parses and merges records.Its generate-and-test cycle iterates until code reconstructs the dataset.Source: A Data Reconstruction Agent · p. 12
• The pipeline standardizes source files before reconstruction, including converting data files to CSV and PDFs or DOCX files to text.Source: A Data Reconstruction Agent · p. 12
• 210 of 321 deduplicated unique TESS studies were successfully reconstructed by the agent.The initial scrape pulled 443 projects, with nested subprojects inflating that count.Source: A Data Reconstruction Agent · p. 12
• A study is counted as successfully scraped only when parsing code executes without errors and produces non-empty row-level outputs.Studies are skipped when required stimulus and outcome-question conditions are not met; some failures reflected the agent’s difficulty generating code for long-context inputs.Source: A Data Reconstruction Agent · p. 12

B Discussion of Metric Evaluation: Accuracy vs. Distributional Alignment

• Individual-response accuracy can reward a midpoint prediction that improves scores while poorly representing the underlying response distribution.For a demographic group with varied answers, matching their probabilities can yield low single-example accuracy.Source: B Discussion of Metric Evaluation:… · p. 13
• GPT-4o predictions had σ = 0.154, compared with σ = 0.195 for SOCRATES-LLAMA-8B and σ = 0.192 for human responses.The authors suggest GPT-4o’s higher accuracy may reflect its narrower clustering around mid-scale values.Source: B Discussion of Metric Evaluation:… · p. 13
• In 30% of 40 evaluation studies, always predicting the response-scale mean outperformed sampling from the ground-truth distribution on accuracy.The authors characterize mean prediction as meaningless and ground-truth sampling as perfect distribution prediction.Source: B Discussion of Metric Evaluation:… · p. 13

C Implementation Details

• Finetuning used one epoch, a global batch size of 256, cosine learning-rate scheduling, and a 0.05 warm-up ratio.SFT used a learning rate of 1e−05 and DPO 1e−06; training took roughly 4–24 hours.Source: C Implementation Details · p. 13

D Prompt Templates For Prediction

• Direct prompts pair a respondent’s demographic profile with the stimuli, while few-shot prompts add similar-question or participant-profile-and-answer examples.Source: D Prompt Templates For Prediction · p. 14

E Reasoning Trace Generation Prompts

• GPT-4o-mini generates oracle reasoning traces as prospective behavioral-science explanations, constrained to avoid answer leakage and remain within five sentences.Source: E Reasoning Trace Generation Prompts · p. 15

F Demographic Subset Results

• The table’s listed fields include attribute category, finetuned LLaMA-8B, and improvement percentages.Source: F Demographic Subset Results · p. 16
• Table 4 reports prediction improvement in Wasserstein distance across demographic categories.Source: F Demographic Subset Results · p. 15, Table 4 · p. 16