SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors

1 INTRODUCTION

• Prior evaluations are fragmented across narrow tasks, sometimes producing contradictory findings that impede broader conclusions about when and how LLM simulations succeed.The field lacks a unified framework for assessing simulation success or how to train better simulators.Source: §1 · p. 2
• SIMBENCH unifies 20 diverse datasets spanning moral dilemmas, economic games, and psychological assessments across a global participant pool.The benchmark supports comparison of simulation fidelity across models, tasks, and populations.Source: §1 · p. 2
• Top LLMs score 40.80/100; fidelity scales log-linearly with model size but does not improve with increased inference-time compute.Instruction tuning improves performance on low-entropy consensus questions but harms it on high-entropy pluralistic questions; models struggle most with religious and ideological groups, while simulation ability correlates most strongly with MMLU-Pro (r = 0.939).Source: §1 · p. 2
• The authors frame group-level response distributions as a foundational proxy, while noting that static distributions do not capture interactive and complex human dynamics.They position SIMBENCH as infrastructure for making progress measurable and simulation research more systematic.Source: §1 · p. 2

2.1 DATA CURATION

• Curators selected large, redistributable, single-turn datasets with quantitative response formats and English questions or validated translations, balancing novelty, established benchmarks, demographic richness, and task diversity.The process yielded 20 datasets and exceptions such as Jester and Choices13k, which were retained for task diversity despite lacking demographic data; the benchmark is modular for future additions.Source: §2.1 · pp. 2–3

2.2 BENCHMARK PROPERTIES

• SIMBENCH covers decision-making, self-assessment, judgment, and problem-solving tasks, drawing participants from more than 130 countries across six continents.Anglosphere West samples make up 27.9% of the data, and 8 of 20 datasets use representative sampling techniques.Source: §2.2 · p. 3

2.3 UNIFYING SIMBENCH DATASET FORMATS

• SIMBENCH standardizes questions as multiple choice, maps continuous responses to bins, uses English versions, and aggregates responses into group-level probability distributions.Post-stratification weights are applied when applicable; the harmonization combines raw-response datasets with already aggregated statistics.Source: §2.3 · pp. 3–4
• The benchmark creates default targets from all participants and specific-group targets from shared demographic attributes, yielding 10,930,271 question-group targets.The authors note that zero-shot distributional prediction largely mitigates the inherent risk of training-data contamination.Source: §2.3 · p. 4

2.4 SIMBENCH SPLITS

• SimBenchPop contains 7,167 cases covering all questions in all 20 datasets, while SimBenchGrouped contains 6,343 cases from five large surveys.SimBenchPop evaluates broad and diverse populations; SimBenchGrouped selects questions with demographic response variation to test simulation of narrower participant groups.Source: §2.4 · p. 4

EXPERIMENTAL SETUP

• The evaluation covers 45 commercial and open-weight, base and instruction-tuned LLMs ranging from 0.5B to 405B parameters.Source: §3 · p. 4
• The setup does not assess intersectional demographic groups because available sample sizes would make their response distributions too noisy for reliable evaluation.Simulation ability may also be correlated with prompt language, a direction the authors encourage future work to examine.Source: §3 · p. 4
• Base models provide response distributions through first-token probabilities, while instruction-tuned models are prompted to verbalize option probabilities.The authors report that verbalized distributions substantially and consistently outperform direct token probabilities for instruction-tuned models, so each model class uses its suitable elicitation method.Source: §3 · p. 5
• SIMBENCH Score S derives from total variation distance and measures a model prediction's improvement over a uniform baseline relative to human ground truth.Scores are normalized using each dataset's mean baseline distance for stability across varying entropy; the reported model score averages test cases, with each dataset capped at 500 questions.Source: §3 · p. 5

4.1 RQ1: GENERAL SIMULATION ABILITY OF LLMS

• 40.80/100 (Claude-3.7-Sonnet) is the highest overall SIMBENCH score across 20 datasets, but LLMs remain far from consistently reliable simulators.The authors report that the best model closes roughly 40% of the gap between a uniform response distribution and human ground truth.Source: Table 1 · p. 5, §4.1 · pp. 5–6

4.2 RQ2: IMPACT OF LLM CHARACTERISTICS ON SIMULATION ABILITY

• SIMBENCH S rises log-linearly with model size, with diminishing returns, while increased inference-time compute offers no meaningful benefit across tested models.Instruction-tuned models outperform base counterparts above 10B parameters, but OLMo-2 base models outperform tuned variants at 7B and 13B.Source: Figure 2 · p. 6, §4.2 · pp. 6–7

4.3 RQ3: IMPACT OF TASK SELECTION ON SIMULATION FIDELITY

• Fidelity varies by dataset: models do better on stated-opinion surveys than behavioral choices and struggle on counter-normative traits or beliefs, sometimes below uniform.GPT-4.1 is a notable exception, scoring 61.9 on OSPsychRWAS.Source: Figure 3 · p. 7, §4.3 · p. 7

4.4 RQ4: THE ALIGNMENT-SIMULATION TRADEOFF

• Instruction tuning's gain falls as human-response entropy rises (r = −0.942), helping consensus questions by up to 40 S points but becoming harmful at highest plurality.The benefit crosses a point of no improvement around an entropy of 0.8.Source: Figure 4 · p. 8, §4.4 · p. 8
• Mediation estimates +6.46 direct and −1.74 entropy-mediated effects; general instruction tuning (S = 16.56) and Centaur cognitive tuning (S = 8.54) improve Llama-3.1-70B via opposing mechanisms.The authors present these as distinct paths to improving simulation.Source: §4.4 · p. 8

4.5 RQ5: SIMULATION ABILITY ACROSS PARTICIPANT GROUPS

• For SimBenchGrouped, questions were selected for highest variance across demographic groups; all evaluated models showed negative mean ∆S (−1.27 for DeepSeek-V3-0324 to −4.61 for Claude-3.7-Sonnet-4000), with 95% CIs excluding zero, likely an upper bound on challenges.Conditioning on Religiosity/Practice caused the largest decrease in simulation accuracy (∆S = −9.91).Source: §4.5 · p. 9

4.6 RQ6: SIMULATION ABILITY VS. GENERAL CAPABILITIES

• SIMBENCH correlates most strongly with MMLU-Pro (r = 0.94), versus general helpfulness (Chatbot Arena ELO, r = 0.71), instruction following (IF-Eval, r = 0.79), and OTIS AIME (r = 0.48).The authors posit that simulation ability is rooted in broad, knowledge-intensive reasoning.Source: §4.6 · p. 9

• Evidence on LLM simulation fidelity is mixed, including reports of homogenized group representations, hyper-rationality, and deterministic rather than distributional predictions.Source: §5 · p. 10
• Earlier studies generally examined individual-level simulation with little demographic conditioning and only one or two models in narrow contexts.Source: §5 · p. 10
• SimBench instead evaluates group-level simulation across domains with demographic conditioning and standardized distributional metrics based on Total Variation Distance.Source: §5 · p. 10

• SimBench unifies 20 diverse datasets to provide infrastructure for robustly evaluating and comparing LLM simulators.Source: §6 · p. 10
• Across 45 LLMs, simulation ability remains limited, scales log-linearly with model size, trades off with standard alignment, varies across demographic groups, and correlates with knowledge-intensive reasoning.Source: §6 · p. 10
• Future work should expand evaluation to interactive and open-ended simulations, develop distribution-preserving alignment, and investigate causal mechanisms linking other capabilities to simulation performance.Source: §6 · p. 10

• Because state-of-the-art models have limited simulation ability, the authors caution against relying on them for tasks where downstream harm is possible.Source: RESPONSIBLE USE AND ACKNOWLEDGMENT OF… · p. 11
• Replacing human participation with algorithmic approximations risks disadvantaging under-represented or marginalized communities by removing their opportunity to shape decisions.Source: RESPONSIBLE USE AND ACKNOWLEDGMENT OF… · p. 11
• The authors caution that higher benchmark performance should not be mistaken for genuine understanding of complex human behavior.Source: RESPONSIBLE USE AND ACKNOWLEDGMENT OF… · p. 11
• The benchmark contains aggregated, non-reversible group-level distributions rather than raw individual participant data, which the authors say protects participant privacy.Source: DATA PROVENANCE AND TRANSFORMATIVE USE · p. 11
• Sample-size limits prevent adequate assessment of intersectional identities, and single-demographic conditioning may miss nuances of individual behavior.Source: SCOPE OF REPRESENTATION AND… · p. 11

• The benchmark focuses on decision-making, self-assessment, judgment, and problem-solving, excluding embodied or multi-turn social dynamics.Source: A LIMITATIONS · p. 20
• The combined sample cannot fully represent any population; geographic gaps, uneven demographic coverage, and opt-in or crowdsourced samples may limit generalizability and precision.Source: A LIMITATIONS · p. 20
• Static datasets capture behavior at particular times, so the benchmark cannot yet assess evolving preferences, opinion shifts, or behavioral adaptation.Source: A LIMITATIONS · p. 20
• Multiple-choice, single-answer, single-turn tasks enable standardized comparisons but exclude multi-step decisions and interactive social dynamics.Source: A LIMITATIONS · p. 20
• Contamination cannot be definitively ruled out without full training corpora, though the authors believe it is unlikely to significantly affect current findings; structured source files and subgroup-distribution prediction further mitigate the risk.Source: A LIMITATIONS · pp. 20–21
• An S-score of 60 on likely training-exposed OpinionQA data and consistent scaling across datasets are presented as evidence against memorization artifacts.Source: A LIMITATIONS · pp. 20–21

B DATASET CURATION AND EXCLUSION RATIONALE

• Strict inclusion criteria excluded candidate datasets that were not redistributable or did not support a multiple-choice benchmark.The review considered prominent social and behavioral science datasets; exclusions commonly reflected licensing restrictions, complex experimental designs, or non-standard response formats.Source: B DATASET CURATION AND EXCLUSION… · p. 21, Table 3 · p. 21

C SIMBENCHPOP AND SIMBENCHGROUPED SAMPLING DETAILS

• SimBenchPop samples up to 500 questions per dataset, while SimBenchGrouped selects questions with demographic variation from five large surveys.The splits contain 7,167 and 6,343 test cases, respectively; Pop aggregates responses across participants, while Grouped creates cases for specific demographic groups.Source: C SIMBENCHPOP AND SIMBENCHGROUPED… · pp. 21–22, Table 4 · p. 22

D IMPLEMENTATION DETAILS

• Base models are evaluated using next-token probabilities, whereas instruction-tuned models are queried for verbalized probability distributions through APIs.For instruction-tuned outputs, failed JSON parsing triggers retries at higher temperature, and non-summing probabilities are normalized; the evaluation also includes models from several families.Source: D IMPLEMENTATION DETAILS · pp. 22–23

E VALIDATION OF ELICITATION METHOD

• Verbalized distributions consistently outperform direct token probabilities for every tested instruction-tuned model, while base models perform reasonably with token probabilities.For many instruction-tuned models, token-probability scores fall far below zero, consistent with poorly calibrated raw logits after instruction tuning.Source: Figure 5 · p. 24, E VALIDATION OF ELICITATION METHOD · p. 24

F METRIC ROBUSTNESS CHECK

• TVD and JSD rankings correlate strongly (Pearson r = 0.92), while TVD and rank correlation show a moderate negative correlation (r = −0.57).The authors report that model comparisons remain consistent across these statistical measures.Source: F METRIC ROBUSTNESS CHECK · p. 25, Table 6 · p. 25

G FULL SIMBENCH RESULTS (RQ1)

• The appendix reports SIMBENCH scores for all 45 evaluated models and scaling-law plots for the full model set.The supplied passages identify Table 7 and Figure 6 as the results displays but do not give their cell values or plotted estimates.Source: G FULL SIMBENCH RESULTS (RQ1) · p. 25, Table 7 · p. 26, Figure 6 · p. 27

H STATISTICAL ANALYSIS OF MODEL PERFORMANCE

• The statistical analysis includes confidence intervals, pairwise significance tests, and within-family tests of scaling trends.These analyses assess overall scores and model-size patterns.Source: H STATISTICAL ANALYSIS OF MODEL… · p. 25, §H.1 · p. 27

H.1 OVERALL MODEL RANKING AND SIGNIFICANCE

• Table 8 reports 95% confidence intervals and adjacent-rank two-sample t-tests, confirming a statistically robust hierarchy among evaluated models.The table covers the top eight models.Source: Table 8 · p. 27, §H.1 · p. 27

H.2 WITHIN-FAMILY PERFORMANCE ANALYSIS

• For most model families, larger models show statistically significant improvements in simulation ability.The authors used one-way ANOVA followed by post-hoc Tukey HSD pairwise comparisons; the reported pattern supports scaling trends in Figure 2.Source: §H.2 · p. 27, Table 9 · p. 28

I.1 OBSERVATIONAL ANALYSIS

• Base models are generally more accurate on high-entropy questions, whereas instruction-tuned models are generally more accurate on low-entropy questions.Figure 7 relates normalized entropy of human responses to TVD for question-model pairs in SimBenchPop; lower TVD indicates better performance.Source: §I.1 · p. 28

I.2 REGRESSION ANALYSIS

• The OLS regression predicts question-level TVD and includes dataset and model fixed effects, with an instruction-status-by-human-entropy interaction.The interaction tests whether the entropy–fidelity relationship differs for instruction-tuned models.Source: §I.2 · p. 28, Table 11 · p. 30
• For each one-unit increase in normalized entropy, TVD decreases by 0.2451 for base models and increases by 0.0997 for instruction-tuned models (both p < 0.001).The model explains approximately 17% of TVD variance (R2 = 0.168); the opposite signs persist after controlling for datasets and models.Source: Table 10 · p. 29, §I.2 · p. 31

I.3 CAUSAL MEDIATION ANALYSIS: DECOMPOSING THE DUAL EFFECTS OF INSTRUCTION TUNING

• The mediation analysis separates instruction tuning’s effect on SIMBENCH performance into direct and output-entropy-mediated pathways.It uses two OLS models and calculates the indirect effect as the product of path coefficients.Source: §I.3 · pp. 31–32
• Instruction tuning has a total effect of +4.72 SIMBENCH points, comprising a +6.46-point direct effect and a −1.74-point indirect effect through reduced prediction entropy.The reported paths are statistically significant; the direct effect is associated with improved instruction and persona following, while the indirect effect reflects lower-entropy, mode-seeking behavior.Source: §I.3 · pp. 32–33
• The authors conclude that the direct benefit currently outweighs entropy suppression on average and suggest distribution-preserving alignment as a future direction.They propose retaining instruction tuning’s direct benefits while mitigating its reduction of distributional diversity.Source: §I.3 · pp. 32–33

J DETAILED CORRELATION ANALYSIS (RQ6)

• The analysis covers five external capability benchmarks and only evaluated models with reliably sourced external-benchmark scores.The benchmark data came from model developers’ technical reports, the Open LLM Leaderboard, and Vals AI.Source: J DETAILED CORRELATION ANALYSIS (RQ6) · p. 33
• Figure 8 presents individual scatter plots for the correlations across the five external benchmarks.The analysis uses the subset of evaluated models with reliably sourced external benchmark performance data.Source: J DETAILED CORRELATION ANALYSIS (RQ6) · p. 33
• Table 12 summarizes Pearson correlation coefficients between SIMBENCH scores and external capability benchmarks.The table covers models evaluated in the study; the supplied passage does not report coefficient values.Source: Table 12 · p. 33, J DETAILED CORRELATION ANALYSIS (RQ6) · p. 33

K CASE STUDY: DETAILED ANALYSIS OF CENTAUR

• Instruction-tuned models score highest on low-entropy tasks, while Minitaur and Centaur performance progressively improves as human-response entropy increases.Figure 9 compares the binned SIMBENCH Score across normalized human response entropy for Llama-3.1 8B and 70B models.Source: K CASE STUDY: DETAILED ANALYSIS OF… · p. 33, Figure 9 · p. 35
• Instruction tuning sharpens prompt-following and consensus responses, whereas specialist cognitive tuning aligns model representations with human-choice patterns and preserves diverse outcomes.The analysis characterizes instruction tuning as mode-seeking and behavioral-data tuning as supporting mass-covering behavior.Source: K CASE STUDY: DETAILED ANALYSIS OF… · p. 33
• Because the tuning paradigms target different mechanisms and their benefits are not mutually exclusive, the authors suggest hybrid models as a possible path to improved simulation.Such models would combine general capability with alignment to the nuances of human behavior.Source: K CASE STUDY: DETAILED ANALYSIS OF… · p. 33

L.10 OSPSYCHMGKT

• The OpenPsych datasets assess general knowledge, Machiavellianism, and authoritarian tendencies; MGKT questions are expanded into five binary-choice items apiece.The MACH-IV measures endorsement of manipulation over morality, while the Right-Wing Authoritarianism Scale assesses submission to authority, outgroup aggression, and conventionalism.Source: §L.10 · p. 40, §L.11 · p. 41, §L.12 · p. 41

L.13 INTERNATIONAL SOCIAL SURVEY PROGRAMME (ISSP)

• The analysis includes only the five most recent ISSP surveys, collected from 2017 to 2021, comprising 1,688 questions with an average of 7,074 responses per question.Source: §L.13 · p. 42

L.13.4 ISSP 2020 ENVIRONMENT IV

• The items ask about environmental protest or demonstration participation in the last five years and frequency of internet searches for information about anxiety, stress, or similar problems during the past 12 months.Source: §L.13.4 · p. 44, §L.13.5 · p. 44

L.14 LATINOBARÓMETRO

• The study applied 19,205 interviews in 18 countries; samples were 1,000 representative cases in five Central American countries and the Dominican Republic, and 1,200 elsewhere.It included 155 questions, with an average of 18,083 responses per question.Source: §L.14 · pp. 44–45

L.15 GLOBALOPINIONQA

• GlobalOpinionQA repurposes World Values Survey and Pew Global Attitudes Survey items on social and political opinions to assess alignment with sociodemographic groups.It contains 2,556 questions, with country as its sole reported grouping variable.Source: §L.15 · p. 45

L.16 DIVERSITY IN CONVERSATIONAL AI EVALUATION FOR SAFETY (DICES) - 990

• DICES provides 990 multi-turn adversarial conversations with safety judgments, capturing variation in human perceptions of conversational AI safety.The judgments concern the last model response and were provided by 173 raters balanced by gender and country across the U.S. and India.Source: §L.16 · pp. 45–46

L.17 NUMBERGAME

• NumberGame contains anonymized judgments from a numerical generalization task inspired by Tenenbaum’s “number game” experiment, reflecting rule-based and similarity-based strategies; it includes 25,499 questions and 575 U.S. participants.Source: §L.17 · p. 46

L.18 CONSPIRACYCORR

• The ConspiracyCorr dataset measures individual endorsement of 11 widely circulated conspiracy beliefs.It includes respondents from 20 countries and participant groupings by country, age, gender, and education.Source: §L.18 · p. 47

L.19 MORALMACHINE

• Moral Machine records choices in life-and-death autonomous-vehicle dilemmas, drawing on 492,921 volunteer participants worldwide.Its dilemmas ask participants to choose between outcomes affecting pedestrians or passengers.Source: §L.19 · pp. 47–48

L.20 TRUST IN SCIENCE AND SCIENCE-RELATED POPULISM (TISP)

• TISP covers perceptions of science, its societal and political role, climate change, and science communication across 68 countries.Its 71,922 participants are grouped by eight characteristics, including age, education, political alignment, religion, residence, and income.Source: §L.20 · p. 48

M ADDITIONAL RELATED WORK

• SIMBENCH spans 20 tasks beyond surveys and enables consistent cross-domain and cross-model comparison, unlike OpinionQA and GlobalOpinionQA’s narrower survey focus.The paper frames simulation as a fundamental capability to evaluate systematically, rather than only a demographic-viewpoint representation challenge.Source: M ADDITIONAL RELATED WORK · p. 49

N LLM USAGE

• The authors used LLMs to refine drafted prose and Cursor and GitHub Copilot to assist with writing and debugging Python analysis scripts.They manually edited the prose to ensure accuracy and alignment with their intended meaning.Source: N LLM USAGE · p. 49

O DETAILED DEMOGRAPHIC BREAKDOWN

• Table 13 details country- and region-level simulation-target counts for SIMBENCHPOP and SIMBENCHGROUPED.The table reports a grand-total row of 6,009, 6,343, and 12,352.Source: O DETAILED DEMOGRAPHIC BREAKDOWN · pp. 49, 52, Table 13 · p. 49