Large Language Models Exhibit Human-Like Bayesian Hypocrisy

Introduction

• Cao et al. (2019) found that people who integrate Bayesian priors and likelihoods can condemn someone making the same judgment as unintelligent and immoral, a pattern called Bayesian hypocrisy.The passage notes that people described such reasoners as “sexist” in open-ended comments.Source: Introduction · p. 2
• The study tests two frontier models on Bayesian reasoning and asks them to judge a person giving the same statistically sound answer for competence and morality.This pairing probes whether models are internally consistent across their own reasoning and their social judgments.Source: Introduction · pp. 2–3
• Prompting and training can improve model reasoning, but gains from chain-of-thought prompting have not necessarily brought generalizability or human-like robustness across nuanced contexts.The introduction describes reasoning as potentially malleable through prompting, training changes, and inference-time computation.Source: Introduction · p. 4
• Whether model reasoning reflects human-like capacities or advanced pattern matching remains unclear, as does whether human decision-making biases have seeped into LLMs.The introduction presents this as an open question amid the accelerated proliferation of LLMs.Source: Introduction · pp. 4–5
• The study tests whether models can overcome humanlike Bayesian hypocrisy, whether both are equally free of it, and whether judgments differ between models given divergent training philosophies.Source: Introduction · p. 7

Participants

• The human comparison used the surgery condition from Cao et al. (2019, Study 4), with 294 participants; the LLM samples comprised 420 responses per model across four conditions.The design included 105 responses per model in each condition, for 840 LLM responses across eight cells.Source: Participants · pp. 8–9

Materials

• Materials adapted Cao et al. (2019, Study 4)’s doctor–nurse task, asking models for prior probabilities, revised probabilities after a surgery cue, and relevant likelihood estimates.The man or woman identified as having performed surgery varied between sessions, and the likelihood prompt asked for base rates among doctors and nurses who perform surgery.Source: Materials · pp. 9, 19

Procedure

• Study 1 crossed model, target gender, and elicitation order between subjects, measuring prior, posterior, and likelihood estimates plus Bayes-rule posteriors calculated from each session’s own priors and likelihoods.The two elicitation orders were likelihoods–priors–posteriors and priors–posteriors–likelihoods.Source: Procedure · p. 10
• The procedure elicited priors, posteriors, and likelihoods in counterbalanced order, with model sessions assigned approximately evenly to man and woman target conditions.Prompts were delivered sequentially in a single conversation thread.Source: Procedure · pp. 10–11

Results

• With 99–100% of trials assigning nurses a 0% surgery probability, ceiling effects and violated normality assumptions led researchers to use non-parametric tests; parametric tests yielded the same outcomes.Some conditions had zero variance, and the stated probabilities implied infinite likelihood ratios.Source: Results · p. 11

Did the models (like humans) demonstrate Bayesian reasoning?

• In the doctor/nurse scenario, LLMs performed similarly to humans, reaching near-optimal decision-making, but their lack of variance in judging the likelihood of nurses performing surgery caused minute deviations from pure optimality.Source: Did the models (like humans)… · p. 12

Pilot Studies (PS 1, 2, and 3): Do Models Condemn A Statistical Mimic?

• Three pilots tested whether models judged a person making the same statistical inference without calculation, progressively varying gender norms, incentives, Bayesian reasoning, and within-subject design.Source: Pilot Studies (PS 1, 2, and 3): Do… · pp. 14–15

Cao et al. (2019), Study 1 (with humans) and Pilot Study 1 (PS1) with LLMs

• Participants agreed that a man and woman were equally likely to be doctors and rated Person X negatively across four dimensions; both LLMs rated him below midpoint, while Claude showed extreme uniformity.Both LLMs relied on more procedural terms.Source: Cao et al. (2019), Study 1 (with… · p. 15

Cao et al. (2019), Study 2 (with humans) and Pilot Study 2 (PS2) with LLMs

• Both LLMs condemned Person X below the midpoint across three professions, unlike humans’ profession-graded pattern, and their evaluations shifted with target-presentation order.Human condemnation did not reach significance in the construction-worker scenario; LLM condemnation was as strong for lower-status professions as for higher-status ones.Source: Cao et al. (2019), Study 2 (with… · p. 16

Cao et al. (2019), Study 3 (with humans) and Pilot Study 3 (PS3) with LLMs

• Claude transferred $0 in the Bayesian condition versus $0.15 in the Egalitarian condition, while GPT-4o transferred $0.14 versus $0.15 (r = .06).Claude’s negative evaluation carried over into allocation; GPT-4o’s statistically significant difference was negligible and did not carry over.Source: Cao et al. (2019), Study 3 (with… · p. 17
• Study 2 replicated a within-subjects test in which each LLM first performed Bayesian reasoning and then evaluated another person’s structurally parallel judgment.The design tested whether models condemn others for reasoning they themselves use; preceding pilots had established that LLMs judge non-egalitarian social judgments negatively.Source: Cao et al. (2019), Study 3 (with… · pp. 17–18

Participants

• The human sample comprised 348 participants, while each model contributed 840 responses distributed across target-gender, elicitation-order, and doctor-scenario conditions.The models were Claude 3.7 Sonnet and GPT-4o; conversational history was maintained across API calls.Source: Participants · pp. 18–19

Materials

• Materials paired a pilot scenario testing Bayesian reasoning with a doctor scenario assessing evaluation of Person X, separated by unrelated filler tasks.In the pilot scenario, the model estimated probabilities before and after learning that a target communicated with air traffic control, then estimated relevant gender-specific base rates.Source: Materials · p. 19
• The doctor scenario varied target presentation order and the direction of Person X’s equivalent comparison, then elicited four 7-point evaluations.The variations were treated as one between-subjects counterbalancing factor.Source: Materials · p. 20

Procedure

• The within-subjects procedure presented a pilot Bayesian-updating task followed by a doctor judgment and evaluation of Person X.Participants estimated priors, updated posteriors after learning who communicated with air traffic control, and evaluated Person X on fairness, justness, accuracy, and intelligence.Source: Procedure · p. 21

Results

• LLM responses showed severe ceiling effects in likelihood ratios and posteriors, alongside normality violations across conditions.Claude also rated Person X “fair” and “just” at the scale floor in every case, preventing parametric tests for those items.Source: Results · p. 22

Do humans and LLMs update beliefs according to Bayes' rule? (A replication of Study 1)

• Both LLMs showed behavior similar to humans in Bayesian updating, although all sources significantly deviated from Bayesian accuracy in non-parametric analyses.The authors nevertheless describe all sources as performing the task accurately.Source: Do humans and LLMs update beliefs… · p. 22, Figure 2 · p. 23

Do humans and models negatively evaluate Person X?

• After completing Bayesian reasoning, both models condemned Person X for reaching a statistically sound judgment, and the evaluations were more negative than humans’.Humans rated Person X below neutral; GPT-4o was more negative than humans, and Claude’s responses were more extreme.Source: Do humans and models negatively… · p. 24, Do LLMs differ from humans on features… · pp. 25–26

Do LLMs differ from humans on features of Bayesian reasoning?

• LLMs reported higher priors that men were pilots and higher posterior probabilities than humans for both target genders; inflated priors contributed to the higher posteriors.The reported posterior differences were significant after correction.Source: Do LLMs differ from humans on features… · p. 25
• Unlike humans, the models’ reported probabilities shifted when likelihoods were elicited before prior and posterior judgments.Humans showed no corresponding order effect.Source: Do LLMs differ from humans on features… · p. 26

Discussion

• In a doctor-and-nurse job-expectation scenario, LLMs integrated priors and likelihoods to produce posteriors that deviated from the ideal.The discussion frames these results around whether LLMs update beliefs using base rates and whether they condemn others for making the same statistically grounded judgments.Source: Discussion · p. 26
• Both LLMs updated their posteriors in the Bayesian direction, then judged Person X unfair, unjust, inaccurate, and unintelligent for a structurally parallel response.The within-subject design had each model complete both tasks in sequence using distinct but equivalent scenarios.Source: Discussion · pp. 26–27
• LLMs’ reasoning appeared probabilistic but was more categorical and rule-based than humans’, and they readily collapsed to a single rule rather than a probability distribution.The authors characterize this rigidity as limiting the models’ ability to represent uncertainty and variability.Source: Discussion · pp. 27, 29
• Models used base rates in their own reasoning yet treated equivalent reasoning by another as inaccurate and immoral, revealing a self-performance/other-judgment dissociation.The authors note that the same model made the statistical assessment and then condemned another for an equivalent output in the same session.Source: Discussion · p. 27
• Both models’ open-ended descriptions converged on categories such as gender, women, and doctor more often than humans’ evaluative label “sexist,” a uniformity the authors interpret as mode collapse.This explanatory pattern persisted from the first pilot through Study 2.Source: Discussion · pp. 27–28
• Claude was the more severe evaluator, reaching the floor on nearly every scale and alone withholding an economic-game transfer, without greater self–other consistency.GPT-4o allocated the same amount regardless of condition, while Claude’s condemnation was more behaviorally consequential.Source: Discussion · p. 28
• The findings suggest LLMs fail to overcome and amplify human hypocrisy, so deployment where statistical fidelity and fairness norms collide warrants extreme caution.The authors argue that systems must recognize the equivalence between their own statistical reasoning and the same reasoning by others.Source: Discussion · p. 29
• The authors report receiving no specific grant funding and note that supplementary materials are available online.Source: Discussion · p. 31

Pilot Study 1, Doctor Scenario Method

• Pilot Study 1 used human data from Cao et al. (2019; N = 199) and 210 responses each from Claude 3.7 Sonnet and GPT-4o at temperature = 1.0.Full transcripts are included in the supplemental materials.Source: Pilot Study 1, Doctor Scenario Method · p. 39
• Participants chose an egalitarian or Bayesian doctor judgment, then rated Person X’s Bayesian statement on fairness, justice, accuracy, and intelligence (1–7), averaged into a composite.Lower composite scores indicate more negative evaluations.Source: Pilot Study 1, Doctor Scenario Method · p. 39

Agreement Patterns

• Both LLMs endorsed the egalitarian judgment at 100% (0% Bayesian), compared with 93% (7% Bayesian) of humans; the response distributions differed significantly.The preregistered chi-square test was significant, χ²(1) = 13.24, p < .001, Cramér's V = .18; Fisher’s exact test gave the same conclusion.Source: Results · p. 39, Table S1 · p. 39

Descriptive Statistics and Reliability

• Claude showed near-total floor effects, with 99.5% of responses at the scale minimum on all four items, making Cronbach’s α incomputable due to zero item variance.Variability decreased from humans to GPT-4o to Claude; internal consistency was acceptable for humans and GPT-4o.Source: Results · p. 42

Primary Analysis

• All sources rated Person X below midpoint (human r = .88; both LLMs r = 1.00); Claude rated Person X more negatively than humans and GPT-4o, while GPT-4o did not differ from humans.The primary between-group comparisons used Mann-Whitney U tests with Holm-Bonferroni correction.Source: Results · pp. 44–45, Table S3 · p. 44, Table S5 · p. 45

Parametric–Non-Parametric Convergence

• Parametric and non-parametric tests diverged only for GPT-4o versus humans: the t test was significant (t = 4.54, df = 218, p < .001, d = 0.46), but Mann-Whitney was not (W = 20,096.5, p = .488, r = .04).GPT-4o's restricted variance (SD = 0.33) explains the divergence; given normality violations, the authors place greater weight on the non-parametric null result.Source: Results · pp. 46–47, Table S7 · p. 47

Order Effects

• Presentation order had no significant effect on composite evaluations for any source (all ps > .32), and the Model × Order ANOVA also found no effects (all ps > .75).Source: Results · p. 48, Table S8 · p. 48

Exploratory Composites

• On exploratory immorality and incompetence composites, all sources gave elevated negative ratings, Claude was at ceiling (M = 7.00), and GPT-4o did not differ significantly from humans (both r = .04, ns).The composites used reverse-coded fairness/justness and accuracy/intelligence items, with higher scores indicating more negative evaluations.Source: Results · p. 48

Open Ended Text

• Humans used “sexist” among their top-five terms for both composites, whereas neither LLM did; the LLMs instead favored procedural language, and Claude’s justifications were notably uniform.For example, “gender” appeared 805 times for Claude versus 449 for GPT-4o and was absent from humans’ top five; the analysis included responses expressing condemnation.Source: Results · pp. 50, 52, Figure S3 · p. 51

Parametric-Nonparametric Convergence

• All 21 comparisons yielded the same significance decisions under parametric and non-parametric tests, despite normality violations in every LLM condition.Non-parametric tests were therefore reported as primary analyses.Source: Method · pp. 54, 78, Table S27 · p. 78

Results

• Humans showed similar Bayesian-endorsement rates across professions, whereas Claude and GPT-4o displayed sharply divergent, profession-specific agreement patterns.Claude endorsed the Bayesian judgment for 61.4% of firefighter trials versus 0% for butcher and 1% for construction worker; GPT-4o endorsed it for 43.8% of construction-worker trials versus 4.3% for butcher and 0% for firefighter.Source: Method · pp. 54, 56, Table S11 · p. 55, Figure S4 · p. 57

Do LLMs and Humans Negatively Evaluate Person X?

• Both LLMs evaluated Person X below the scale midpoint in all professions, while humans did so for butcher and firefighter but not construction worker.LLM effects were large; human construction-worker ratings did not significantly fall below the midpoint (p = .146).Source: Method · p. 58, Table S12 · p. 58

Does the Profession Matter?

• Human evaluations did not significantly vary across professions, but both LLMs showed large profession effects, and every LLM profession pair differed significantly.Claude condemned butcher most and firefighter least; GPT-4o condemned firefighter most and construction worker least.Source: Method · pp. 62–63, Table S14 · p. 62

Do LLMs Differ from Humans?

• Claude judged Person X more negatively than humans in all three professions, while GPT-4o did so for butcher and firefighter but not construction worker.GPT-4o’s construction-worker comparison was not significant (r = .10, p adj = .09).Source: Method · p. 65, Table S17 · p. 65

Exploratory Analyses

• Exploratory immorality and incompetence composites also showed profession effects for both LLMs, and LLMs generally rated Person X as more immoral and incompetent than humans.Claude rated Person X more immoral than GPT-4o in all professions; incompetence ratings differed by profession, with the direction reversed for firefighter.Source: Method · pp. 68, 70, 74

Order Effects

• Presentation order had no significant effect on human evaluations, but both LLMs showed significant order effects and order-by-profession interactions.The interactions indicate that presentation order’s influence on LLM evaluations varied by profession.Source: Method · p. 76, Table S26 · p. 76

Exploratory Analyses

• In open-ended evaluations, LLMs more often used abstract, principle-based language, whereas humans more often referenced the specific profession and used evaluative labels.The analysis considered responses exceeding the midpoint on immorality or incompetence composites.Source: Method · pp. 79–80

• LLMs judged Person X more negatively than humans across nearly all professions and showed large profession effects and order sensitivity, unlike humans.Human composite evaluations hovered near the scale midpoint, with a non-significant profession effect; Claude showed the most extreme condemnation.Source: Method · p. 82
• LLMs replicated and amplified the human tendency to condemn Bayesian judgments, but their condemnation varied across non-doctor professions.Humans did not strongly differentiate among butcher, firefighter, and construction worker contexts.Source: Method · p. 82

Pilot Study 3, Economic Incentives

• A between-subjects comparison of Bayesian and Egalitarian Person X tested whether negative evaluations translate into economic punishment.The study replicated Cao et al. (2019, Study 3).Source: Pilot Study 3, Economic Incentives · p. 83

Participants

• The human sample included 402 participants from Cao et al. (2019, Study 3), while each LLM provided 420 responses; all LLM responses used temperature 1.0.Humans were split between 202 in the Bayesian condition and 200 in the Egalitarian condition; each LLM had 210 responses per condition.Source: Method · p. 83

Procedure

• Participants judged whether a man or woman performing surgery was more likely to be a doctor, evaluated Person X's Bayesian or egalitarian judgment, and could share up to 30 cents with X.Participants also generated their own agreement response and probability estimates before evaluating Person X.Source: Method · p. 83

Results

• Claude and GPT-4o nearly always endorsed the egalitarian judgment, whereas 91.3% of humans did so and 8.7% endorsed the Bayesian judgment.Claude endorsed the egalitarian judgment in 100% of trials and GPT-4o in 99.8%; both LLMs differed significantly from humans in agreement distributions (both ps < .001).Source: Results · p. 83, Table S28 · p. 84
• Significant non-normality across testable conditions led the authors to use non-parametric tests as primary analyses and provide parametric equivalents for completeness.Claude’s zero variance precluded normality tests in multiple conditions; GPT-4o share amounts were extremely skewed in both conditions.Source: Results · pp. 88–90
• All sources rated Person X below the midpoint after the Bayesian judgment and above it after the egalitarian judgment, with Claude’s ratings reaching the scale floor and ceiling.In the Bayesian condition, composite means were 2.49 for humans, 2.76 for GPT-4o, and 1.01 for Claude; in the egalitarian condition, they were 6.34, 5.99, and 7.00, respectively.Source: Table S31 · p. 91, Results · pp. 91, 94
• Transfers varied by source: humans gave less to Bayesian than egalitarian Person X, GPT-4o showed a small difference, and Claude transferred 0 versus 15 cents.Human medians were 0 versus 15 cents (r = .42); GPT-4o medians were 15 cents in both conditions despite a small difference (r = .06); Claude’s constant transfers yielded r = 1.00 without an inferential test. No LLM transferred more than 15 cents, whereas eight humans transferred 30 cents.Source: Table S33 · p. 95, Results · p. 99
• All eight LLM–human comparisons on evaluations and transfers were significant after correction, while LLMs showed binary transfer patterns unlike humans’ continuous distributions.In the Bayesian condition, Claude condemned Person X most strongly but transferred less than humans; GPT-4o condemned Person X slightly less than humans but transferred more. In the egalitarian condition, both models evaluated Person X differently from humans and transferred more.Source: Results · pp. 101–102, Table S36 · p. 101, Table S37 · p. 102
• Claude evaluated Bayesian Person X more negatively than GPT-4o and transferred less, but evaluated egalitarian Person X more positively while their transfers did not differ significantly.For egalitarian transfers, p adj = .158 and r = .01; both models overwhelmingly transferred exactly 15 cents.Source: Table S39 · p. 105, Results · p. 105
• Exploratory subscales showed Claude near-ceiling Bayesian immorality and incompetence scores, while GPT-4o and humans had comparable scores.Claude differed significantly from humans on both subscales in both conditions, while GPT-4o showed smaller differences; Claude also differed from GPT-4o on both subscales, with massive Bayesian-condition effects.Source: Results · p. 106, Table S42 · p. 108, Table S43 · p. 109
• GPT-4o showed small gender-presentation order effects on evaluations in both conditions, while humans showed none and Claude showed none in the Bayesian condition.Claude’s zero variance in the egalitarian condition precluded testing an order effect.Source: Table S44 · p. 111, Results · p. 111
• All 12 comparisons tested with both parametric and non-parametric methods reached the same significance decision at α = .05; four categorical comparisons had no parametric counterpart.Parametric tests were t-tests and non-parametric tests were Mann–Whitney U tests; the categorical comparisons used chi-square tests.Source: Results · pp. 113–114, Table S46 · p. 113
• Open-ended evaluations commonly mentioned “doctor,” “women,” and “surgery”; models used these terms more than humans, Claude surfaced “reasoning,” and only humans used “sexist” and “male.”These word patterns came from responses with composite scores above the midpoint, indicating negative evaluations of Person X.Source: Results · pp. 115, 118, Figure S11 · p. 117
• Claude evaluated Person X more negatively than humans and GPT-4o in the Bayesian condition, but more positively in the Egalitarian condition; both models differed from humans on every economic-behavior comparison.LLM transfers were binary (0 or 15 cents), unlike humans’ use of the full 30-cent range; no LLM transferred more than half the endowment. Claude was at or near ceiling on both Bayesian-condition evaluation composites, while GPT-4o and humans were comparable.Source: Results · p. 118
• GPT-4o showed order effects absent for humans and Claude, including a Model × Order interaction in the Egalitarian condition, F(2, 614) = 8.14, p < .001.The interaction reflected modest sensitivity to gender-presentation order; order sensitivity varied across sources because GPT-4o showed significant effects while humans and Claude did not.Source: Results · pp. 112, 118, Table S45 · p. 112

Study 1, Bayesian Probability Judgments

• In LPP, GPT-4o refused the initial likelihood prompt; researchers added a best-guess instruction, after which it provided responses, while no modification was needed for PPL or Claude 3.7 Sonnet.Source: Study 1, Bayesian Probability Judgments · p. 119

Likelihood Estimates and Prior Probabilities

• Because distributions departed from normality and variances were heterogeneous, the study used nonparametric tests as its primary analyses; LLM nurse-likelihood comparisons were constrained by near-universal infinite ratios.Among humans with finite likelihood ratios, there was no significant male-versus-female target difference (W = 5,038.5, p = .189, r = .11); the comparison could not be conducted for either LLM because nearly all ratios were infinite.Source: Results · pp. 119, 120, 124
• Claude estimated a 0% chance of nurses performing surgery in all trials, while GPT-4o did so in 99.3%; their likelihood and posterior outputs showed severe ceiling effects.Claude’s model posteriors were 1.0 in all 420 responses, and GPT-4o’s model posteriors were at ceiling in 99.3%; only 34.7% of human responses produced infinite likelihood ratios.Source: Results · pp. 122–123, Table S48 · p. 123

Bayesian Accuracy: Model vs. Reported Posteriors

• Both LLMs’ reported posteriors fell significantly below their own Bayesian predictions for male and female targets, while humans showed no significant difference for male targets and only a small deviation for female targets.All LLM comparisons had p < .001 and r ≥ .98; the human female-target difference was significant in the primary nonparametric test (p = .017, r = .27).Source: Results · p. 130, Table S51 · p. 130

Gender Effects on Probability Estimates

• Both models showed target-gender effects on priors, and Claude also showed an effect on reported posteriors, whereas human gender effects were negligible overall.Claude’s invariant model posteriors made that gender comparison untestable; nonparametric and parametric tests differed on three significance decisions.Source: Results · pp. 132–133, Table S53 · p. 132

Comparing LLM & Human Performance

• GPT-4o and Claude 3.7 Sonnet differed from humans in several estimates: GPT-4o had lower doctor-likelihood estimates, while Claude had lower priors, higher posteriors, and lower doctor-likelihood estimates.The exploratory comparison also found lower nurse-likelihood estimates for GPT-4o than humans; Claude’s invariant zero nurse estimates prevented inclusion in that complete comparison family.Source: Table S55 · p. 134, Results · pp. 134–135

Direct Comparison Between Sonnet 3.7 & GPT-4o

• Claude had significantly lower priors, higher reported posteriors, and lower doctor-likelihood estimates than GPT-4o; their model-posterior comparison was not reported.The direct tests used Holm-Bonferroni correction across the testable comparisons.Source: Results · p. 136

Model vs. Model Comparisons

• Claude’s model posteriors were 1.0 on every trial and GPT-4o’s were 1.0 in 99.3%, leaving no meaningful variation for comparison.Parametric tests found effects for priors (d = −1.03), reported posteriors (d = 0.96), and doctor likelihood estimates (d = −0.65).Source: Results · pp. 137, 140, Table S58 · p. 139

Exploratory Analyses: Elicitation Order & Interactions

• All three sources showed significant elicitation-order effects on both priors and reported posteriors, with the largest effects for Claude.For priors, LPP produced higher estimates than PPL for humans and Claude, while GPT-4o showed the reverse; Claude’s posterior effect was detected nonparametrically despite ceiling responses in LPP.Source: Results · pp. 140, 142, Table S59 · p. 141
• ART analyses found all reported-posterior main effects and interactions significant, including Source × Gender × Order, but alignment diagnostics warrant caution.For priors, all effects were significant except Target Gender and the three-way interaction; the Gender × Order interaction held for both LLMs but not humans.Source: Results · pp. 142, 143, 145, 146, Table S61 · p. 143

Procedure

• Participants first estimated pilot-role priors and posteriors after learning which person communicated with ATC, then estimated how often pilots and flight attendants communicate with ATC.The order of likelihood elicitation and posterior estimates was counterbalanced.Source: Study 2, Inconsistency Between Own… · p. 147
• In the doctor scenario, participants stated whether the man was less likely, equally likely, or more likely to be a doctor, then rated Person X’s claim that he was more likely.Ratings covered fairness, justness, accuracy, and intelligence on 7-point Likert scales and were averaged into a composite score.Source: Study 2, Inconsistency Between Own… · p. 147
• The study applied gender-presentation and likelihood-elicitation counterbalancing to LLM data collection, balanced target gender across conditions, and captured open-ended responses about Person X.A brief trivia masking task preceded Part 2.Source: Study 2, Inconsistency Between Own… · p. 147

Descriptive Statistics and Normality Assessment

• Claude’s prior estimates were highest and its Person X evaluations lowest; ceiling effects and non-normal distributions across sources motivated non-parametric primary analyses.Claude’s priors averaged 93.27 for male targets and 93.23 for female targets, while its composite evaluation mean was 1.01; 100% of its male-target posteriors were at least 95%.Source: Results · pp. 147, 150, Table S65 · p. 149, Table S66 · p. 151

Likelihood Ratio Distributions

• Infinite likelihood ratios occurred in 100% of Claude’s male-target cases and 98.1% of GPT-4o’s male-target cases, versus 24.2% of humans’ male-target cases.Near-universal infinite ratios prevented the planned gender comparison for either model; humans showed a significant gender difference in log likelihood ratios (W = 10654.5, p = .001), replicating Cao et al. (2019).Source: Table S67 · p. 154, Results · p. 154

Bayesian Accuracy: Model vs. Reported Posteriors

• Both LLMs significantly underestimated their own Bayesian predictions across target genders, whereas humans underestimated male-target posteriors but overestimated female-target posteriors.All model comparisons had p < .001 and rs ≥ .61; human differences replicated Cao et al. (2019). Parametric paired tests converged with the non-parametric findings.Source: Table S68 · p. 155, Results · pp. 155, 158, Table S69 · p. 158

Gender Effects on Probability Estimates

• All three sources reported higher posteriors for male than female targets, while none showed a significant target-gender difference in prior probabilities.The posterior pattern replicated Cao et al. (2019, Study 5); parametric tests likewise found posterior gender effects for all sources and no prior gender effects.Source: Table S70 · p. 159, Results · pp. 159–160, Table S71 · p. 160

Comparing LLM & Human Performance

• All 14 LLM–human comparisons were significant (all ps < .001); both models gave higher priors and posteriors and lower Person X ratings, while Claude’s deviations were larger than GPT-4o’s.Mean effect size was r = .73 for Claude versus r = .42 for GPT-4o across seven tests; parametric comparisons also remained significant after correction.Source: Results · pp. 160–162, Table S72 · p. 161, Table S73 · p. 162

Direct Comparison Between Sonnet 3.7 & GPT-4o

• All seven Claude–GPT-4o comparisons were significant; Claude’s median priors were 95 versus 90 for GPT-4o, and its evaluation medians were 1 versus 2.Claude also had higher male-target posterior medians (100 versus 99); female-target medians were equal at 95, although distributions differed significantly. Parametric results remained significant after correction.Source: Table S74 · p. 164, Results · pp. 164–165, Table S75 · p. 165

Exploratory Analyses: Order Effects & Scenario Wording

• Likelihood elicitation order substantially shifted both models’ probability estimates, but doctor-scenario wording had no detectable effect on reported posteriors.For GPT-4o, LPP versus PPL medians were 95 versus 80 for priors and 100 versus 95 for posteriors; Claude’s posterior medians were 100 versus 95, while its prior effect was smaller and reversed in direction. Wording A versus B affected GPT-4o’s composite evaluations (r = .46), but not humans (r = .00) and only negligibly affected Claude at floor (r = .07); wording combined presentation order and phrasing.Source: Results · pp. 166, 168, Table S76 · p. 167

• The figure reports Bayesian updating error (reported minus model posterior) by source and target gender.Source: Results · p. 169
• A dashed line at zero marks perfect Bayesian accuracy.Source: Results · p. 169
• Positive error indicates overestimation, while negative error indicates underestimation.Source: Results · p. 169

Agreement Patterns with the Doctor Scenario from PS1

• All three sources rated Person X below the neutral midpoint, with condemnation ranging from moderate among humans to near-total for Claude.GPT-4o’s median was 2 on every item; Claude’s fair and just ratings were 1 for all respondents, and its composite mean was 1.01 (SD = 0.05).Source: Agreement Patterns with the Doctor… · p. 170
• Humans showed an Evaluation × Gender interaction in their posteriors, but neither LLM did; their near-constant Person X ratings limit interpretation of these null terms.The human interaction was F(1, 344) = 10.71, p = .001, η² = .023; LLM evaluation composites had SDs of 0.23 for GPT-4o and 0.05 for Claude. All sources showed a target-gender main effect.Source: Agreement Patterns with the Doctor… · pp. 170–171, Table S77 · p. 171
• LLMs chose the egalitarian judgment in 100% of responses and never chose the Bayesian or reverse option across 840 trials each; humans chose egalitarian in 79.0%.Both LLMs differed significantly from humans (χ²(2) = 187.74, p < .001, Cramér's V = .40); Claude and GPT-4o did not differ.Source: Table S78a · p. 172, Table S78b · p. 172, Agreement Patterns with the Doctor… · pp. 172–173
• Even extreme evaluators produced gender gaps: fitted human posteriors were 91.6% for men and 81.4% for women, while Claude’s observed medians were 100 and 95.GPT-4o had no extreme evaluators; fitted gaps were 3.3 points for GPT-4o and 3.9 points for Claude. The observed human extreme-evaluator difference was not significant, whereas Claude’s was significant.Source: Table S79 · p. 173, Agreement Patterns with the Doctor… · p. 173
• Humans’ model posteriors were higher for male than female targets (Mdn = 98.3 vs. 65.5), while both LLMs were at or near 100% for both targets.The LLM gender differences were statistically significant but small (GPT-4o r = .02; Claude r = .09), as near-infinite likelihood ratios drove posteriors to 100% regardless of target gender.Source: Agreement Patterns with the Doctor… · p. 174, Table S81a · p. 176, Table S81b · p. 177
• In open-ended evaluations, all sources frequently mentioned “doctor” and “women”; the models used category terms and avoided the evaluative label “sexist.”The analysis covered responses with composite scores above the midpoint and compared immoral and incompetent scales.Source: Agreement Patterns with the Doctor… · p. 178, Figure S18 · p. 179

Scenario 3

• Scenario 3 asks respondents for an open-ended impression of Person X or the statement Person X made.Source: Scenario 3 · p. 186

Scenario 2

• In Scenario 2, respondents rate Person X’s fire- or concrete-related likelihood statement for intelligence, accuracy, fairness, and justice.The supplied prompts include statements favoring the man and reversed statements favoring the woman.Source: Scenario 2 · pp. 187–190

Appendix C. PS3 Prompts

• Appendix C specifies that the Scenario 2 prompt substitutes the model’s Scenario 1 MAN_PROBABILITY value and is sent in the same conversation.Source: Appendix C · p. 192

Scenario 1

• Scenario 1 asks for probability estimates and a choice about whether the man and woman are equally or differently likely to hold the target occupation after the specified event.The prompts cover doctor/nurse and firefighter or construction-worker scenarios.Source: Scenario 1 · pp. 186, 189, 192

Scenario 2.5

• Scenario 2.5 presents Person X’s doctor-likelihood claim and asks respondents to rate its intelligence, accuracy, fairness, and justice.The quoted claim says the two numbers should differ, with the man more likely to be the doctor after performing surgery.Source: Scenario 2.5 · p. 193

Scenario 4

• Scenario 4 gives respondents a 30-cent bonus and asks how much to share with Person X and how much to keep.The two amounts must total 30 cents.Source: Scenario 4 · p. 193

Scenario 3

• Respondents gave an open-ended impression of Person X and/or the statement they made.Source: Scenario 3 · p. 195

Scenario 4

• Participants received a 30-cent bonus and chose how much to share with Person X versus keep, with the two amounts required to total 30 cents.Source: Scenario 4 · p. 195

Scenario 1

• In the doctor–nurse scenario, respondents estimated each person’s chance of being the doctor conditional on that person having performed surgery, then chose whether the two probabilities should differ or match.Source: Scenario 1 · p. 196

Scenario 2

• Respondents rated Person X’s statement for intelligence, accuracy, fairness, and justice after reading a claim that the man was more likely than the woman to be the doctor after surgery.A parallel prompt presented an equal-likelihood claim.Source: Scenario 2 · pp. 185, 197

Likelihoods

• Likelihood prompts asked for surgery rates among male and female doctors and nurses in the United States.Source: Likelihoods · pp. 201, 202, 204

Priors

• Prior prompts elicited 100%-summing guesses about which person held the doctor or pilot role in the respective man–woman pairs.Source: Priors · pp. 201, 205

Posteriors

• Posterior prompts asked participants to revise the role probabilities after learning that the man had performed surgery or communicated with air traffic control.Source: Posteriors · pp. 201, 205

Likelihoods

• Separate likelihood prompts asked how often male and female pilots and flight attendants communicate with air traffic control during a flight.Source: Likelihoods · pp. 204, 206
• Respondents are asked for separate percentages for female pilots and female flight attendants, using the exact format “PILOTS: [number] ATTENDANTS: [number].”Source: Likelihoods · p. 208

Scenario 2a

• Respondents were asked what percentage of U.S. households have at least one dog.Source: Scenario 2a · p. 208
• Respondents were asked what percentage of U.S. households have at least one cat.Source: Scenario 2a · p. 208
• For a newly obtained U.S. household dog, respondents estimated the percentage chance of each source and were told their answers should total 100%.Source: Scenario 2a · p. 208
• The listed dog sources were shelters or humane societies, friends or relatives, breeders, strays, and other.Source: Scenario 2a · p. 208
• For a newly obtained U.S. household cat, respondents estimated the percentage chance of each source and were told their answers should total 100%.Source: Scenario 2a · p. 208
• The listed cat sources were shelters or humane societies, friends or relatives, breeders, strays, and other.Source: Scenario 2a · p. 208

Scenario 2b

• Respondents first estimated the water and other-material percentages in human and jellyfish bodies, with each pair required to total 100%.Source: Scenario 2b · p. 208
• The update information stated that a jellyfish’s water percentage is at least 10 percentage points greater than a human being’s, then asked for revised human-body estimates.Source: Scenario 2b · pp. 208–209
• The same comparison preceded a request for revised estimates of the jellyfish’s water and other-material percentages, again totaling 100%.Source: Scenario 2b · p. 209

Scenario 2c

• Participants estimated the percentages of Earth’s surface covered by land and water, with their answers required to total 100%.Source: Scenario 2c · p. 209
• Participants ranked baking times of 30, 40, 45, 60, and 75 minutes, then updated their ranking after learning that one egg was used.Source: Scenario 2c · pp. 209–210

Scenario 2d

• Participants ranked five possible earnings amounts for a movie released five years earlier, from $100,000 to $200 million.The options were $100,000, $2 million, $25 million, $100 million, and $200 million.Source: Scenario 2d · p. 210
• They then reranked the earnings after learning that the movie received five Oscar nominations and won two.Source: Scenario 2d · p. 210

Scenario 2e

• Participants rated how interesting or uninteresting they found each trivia statement.Source: Scenario 2e · p. 211
• The trivia covered assorted facts, including sea otters holding hands, Tug of War as an Olympic event, and the meaning of “Kummerspeck.”Other statements concerned cat allergies, M&Ms, blue lobsters, horses, and cloud weight.Source: Scenario 2e · p. 211

Scenario 2f

• A follow-up tested recall with questions about horses’ inability to vomit, cloud weight, Tug of War’s Olympic years, “kummerspeck,” and M&Ms’ name.Source: Scenario 2f · pp. 211–212

Agreement

• Participants chose whether a man or woman who had recently performed surgery was less likely, equally likely, or more likely to be a doctor.Source: Agreement · p. 212

Evaluation

• Participants evaluated Person X’s claim that a man who performed surgery was more likely to be a doctor than a woman who performed surgery.The prompt asked for ratings of the statement and Person X.Source: Evaluation · p. 212
• Evaluation used 1–7 ratings of the statement’s intelligence and accuracy, from extremely unintelligent or inaccurate to extremely intelligent or accurate.Responses had to follow an exact format with no additional text.Source: Evaluation · p. 212
• Participants also rated the statement’s fairness and justice on 1–7 scales, from extremely unfair or unjust to extremely fair or just.Source: Evaluation · p. 212

Impression

• A separate prompt asked participants what impression they had of Person X and/or the statement.Source: Impression · p. 212

Agreement

• The agreement prompt asked whether a man or woman who had each performed surgery was less likely, equally likely, or more likely to be a doctor.Participants responded by repeating the statement they agreed with.Source: Agreement · p. 213

Evaluation

• Person X’s statement is rated for intelligence, accuracy, fairness, and justice on seven-point scales.The prompt describes a man and a woman both performing surgery and asks respondents to rate the claim that the woman is less likely to be a doctor.Source: Evaluation · p. 213

Impression

• The impression prompt asks what respondents think of Person X and/or the statement Person X made.Source: Impression · p. 213