PaperlayerView exact revision on arXiv

Source-linked AI summary

Confident, Not Wiser: The Dunning-Kruger Effect in Human-AI Interaction

Daniela Fernandes, Michelle Rausch, Agnes Mercedes Kloft, Daniel Buschek, Robin Welsch

arXiv:2609.31095v1cs.HC

TL;DR

Because AI performance varies and is not directly visible, how AI access relates to users’ metacognitive judgments remains unclear. The study compares Human alone and Human+AI across reasoning tasks using an AI benchmark and multiple forms of self-assessment; Human+AI performs better, but self-assessment does not track performance correspondingly.

  • Problem

    Because AI performance varies and is not directly visible, how access to AI relates to users’ metacognitive judgments remains unclear.

    Source: §1 · p. 2

  • Method

    Across four reasoning task types, the study benchmarked the AI model on the same items and collected global, block, and item-confidence judgments.

    Source: §1 · p. 2, §3.3 · p. 7

  • Results

    Human+AI scored higher, but self-estimates tracked performance weakly; average bias was similar, while pooled confidence discrimination was lower and the observed Dunning-Kruger contrast larger.

    Source: §5 · p. 22, §6 · p. 24

  • Takeaways & Limitations

    Self-assessment of joint work is a distinct design target: systems should help users evaluate where the partnership succeeds and where answers need checking.

    Source: §6 · p. 25

    The exploratory, non-preregistered study recruited separate groups without random assignment, so group contrasts do not identify causal effects of AI assistance.

    Source: §5.5 · p. 24

Abstract

from arXiv · show

AI assistance can improve performance without improving self-assessment. We report a study (N=366) comparing Human alone and Human+AI performance on reasoning tasks, for which the AI model is benchmarked on the same items. Participants estimated global and block performance and rated confidence in their answers. Human+AI achieved higher scores, but self-estimates tracked performance weakly. Average overestimation was similar across groups, covering individual errors. Across tasks, confidence distinguished correct from incorrect answers less accurately in the Human+AI group, while within-task differences remained uncertain. The Dunning-Kruger pattern was found in both groups, with a larger observed contrast in Human+AI. Controls for score noise reduced but did not eliminate the pattern, with the controlled group difference remaining inconclusive. An extended computational account describes global and block estimates. Our findings distinguish performance augmentation from metacognitive augmentation and motivate interfaces that support verification, communicate task-specific AI model performance, and help users evaluate the quality of their joint work rather than produce answers.

1 Introduction

AI assistance changes both the performance users judge and the evidence available for judging it, while the relationship between AI access and metacognition remains unclear. This study examines performance, self-assessment, and the Dunning-Kruger pattern across reasoning tasks.

1 Introduction

  • Prior research reports overestimation of both personal performance and AI gains; the Dunning-Kruger pattern can also arise from regression to the mean and measurement error.The pattern describes low performers overestimating and high performers underestimating their performance.Source: §1 · p. 2
  • Fernandes et al. found AI assistance raised reasoning performance while participants overestimated performance, but whether their computational conclusion generalizes across tasks remained unresolved.This study extends that work across four reasoning task types using global estimates, block estimates, and item confidence.Source: §1 · p. 2
  • The exploratory study compared separately recruited Human-alone and Human+AI groups across four reasoning tasks, benchmarking the AI model on the same items and collecting global, block, and answer-level judgments.The study was not preregistered.Source: §1 · p. 2
  • Human+AI outperformed Human alone, but self-estimates tracked performance weakly, and the Dunning-Kruger pattern remained in both groups after score-noise controls.The controlled difference between groups was inconclusive.Source: §1 · p. 2
  • The paper examines metacognitive accuracy and sensitivity together, applies controls for Dunning-Kruger measurement artifacts, and derives design implications for assessing joint and task-specific AI performance.It distinguishes pooled from within-task confidence discrimination.Source: §1 · p. 3

2 Related Work

Related work frames the study around two strands: separating judgments of overall performance from judgments of individual answers, and testing whether Dunning-Kruger patterns survive statistical controls. It also motivates examining how AI assistance changes cues for self-assessment and how estimates of the human-AI pair compare with measured performance.

2 Related Work

  • The review draws on metacognition and the Dunning-Kruger effect, including statistical robustness checks and computational accounts of the effect.Its metacognition strand separates judgments of total performance from judgments of individual answers.Source: §2 · p. 3

2.1 Metacognition in Human-AI Interaction

  • Signed calibration error compares perceived with actual performance, whereas metacognitive sensitivity measures how well item confidence separates correct from incorrect answers.Sensitivity is commonly measured with type-2 ROC area, where .5 is chance and 1 is perfect discrimination.Source: §2.1 · p. 3
  • Informative confidence can support a Human+AI pair outperforming both members under suitable error-dependence conditions, but AI assistance may make familiar self-assessment cues misleading.Prior work links effortless assistance and fluent explanations with misplaced credit or inflated understanding.Source: §2.1 · p. 3
  • Prior work leaves AI-performance judgments and expected Human+AI synergy understudied; this study compares beliefs about the self, AI, and the pair with observed performance and a same-item AI benchmark.Letting users compare their correctness with the AI model has been reported to improve pair decisions.Source: §2.1 · p. 4

2.2 The Dunning-Kruger Effect

  • Dunning-Kruger patterns may reflect regression to the mean, measurement noise, task difficulty, or bounded estimates, so the review motivates applying multiple controls together.The study applies a grouping control, measurement-error correction, and diagnostics, while considering block difficulty.Source: §2.2 · p. 4
  • The review contrasts an inference-from-noisy-attempts account with a bias-and-noise reporting account, then extends the latter to block estimates and decomposes overestimation.The prior human-AI finding left the Dunning-Kruger status unresolved; the study tests alternatives across four tasks with score-noise controls and a same-item AI benchmark.Source: §2.2 · p. 4

3 Method

The study compared reasoning and self-assessment in separately recruited Human-alone and Human+AI groups, using a same-item AI benchmark. Participants completed four task blocks and reported performance estimates and answer confidence at multiple levels.

3.1 Participants

  • The final sample comprised 366 participants: 187 in Human alone and 179 in Human+AI, recruited through separate calls and assigned by which group call they answered first.The study used continuous exclusions and replacement recruitment to reach its target in both groups.Source: §3.1 · p. 5

3.2 Study design and interface

  • The Human+AI group used GPT-5.6 Luna alongside the 40 items, while Human-alone participants saw the same items without chat; the chat panel was hidden during self-assessments.Human+AI participants had to send at least one message before submitting each answer.Source: §3.2 · p. 6, Figure 2 · p. 6

3.3 Tasks and measures

  • Participants completed four randomized ten-item blocks—matrix reasoning, mental rotation, syllogisms, and letter-string analogies—and rated confidence after each answer.Performance was scored out of 40 overall and out of 10 per task; estimates were collected before and after the battery and after each block.Source: §3.3 · pp. 6–7

3.4 Procedure

  • After consent and setup, participants completed instructions and a practice trial for each block, then the task items, post-task assessments, questionnaires, and leaderboard.The AI chat history persisted across items and blocks; sessions lasted a median of 77 minutes.Source: §3.4 · p. 7

3.5 AI Model Benchmarking

  • The AI-alone reference was GPT-5.6 Luna’s mean performance across 100 chat-parity runs on the same 40 items, but the benchmark only approximated the live interaction context.The benchmark omitted instructions and practice context and reset history at each block; unflagged benchmark replies had not been audited.Source: §3.5 · p. 7

3.6 Data Analysis

  • The analysis compared performance, estimates, signed errors, and confidence by group using one-sample, paired, and Welch tests, with 95% parametric intervals and specified corrections.Within-block reference tests were exploratory and unadjusted.Source: §3.6 · p. 9
  • Dunning-Kruger patterns were assessed with quartile contrasts and estimate–score slopes, alongside disjoint-score checks, split-half instrumental-variable correction, and sensitivity analyses.The analyses included 4,000 participant bootstrap resamples for nonlinear contrasts and related estimates.Source: §3.6 · pp. 9–10
  • Total-only and extended bounded-binomial models jointly described achieved scores and estimates, with the extension adding block-level estimates and person-specific reporting terms.Model checks were in-sample rather than held-out validation, and the binomial likelihood did not recover unreported estimates beyond the maximum.Source: §3.6 · p. 10

4 Results

The results examine how Human+AI and Human-alone performance relates to self-assessment, confidence, and observed Dunning–Kruger patterns, alongside associations with AI model performance and computational characterizations of global and block reports. They also describe limits in model fit and interpretation.

4.1 Human-AI Joint Performance

  • Human+AI scored 7.22 points above Human alone (95% CI [5.99, 8.46], p<.001), but its +0.65-point advantage over AI model alone was inconclusive (95% CI [−0.27, 1.57], p=.168), supporting augmentation, not overall synergy.The Human+AI–AI comparison did not establish synergy or equivalence.Source: §4.1 · p. 10
  • By block, only matrix reasoning exceeded both components (+1.48 vs Human alone; +0.52 vs AI model), while Human+AI beat Human alone on syllogisms (+3.88) and letter strings (+1.68), but not mental rotation (+0.17).Against AI model alone, Human+AI was higher on mental rotation (+1.28) and lower on letter strings (−0.73) and syllogisms (−0.42).Source: §4.1 · pp. 10–11, Figure 4 · p. 11

4.1.1 Knowing the self, the AI model, and the pair.

  • Human+AI participants’ AI-model estimates weakly tracked block performance (slope 0.324; mean absolute error 2.15), while pair estimates exceeded achieved scores by 0.71 points.Human+AI participants overestimated the AI benchmark by 0.55 points; Human alone participants’ AI-model estimates had a slope of 0.029 and mean overestimation of 1.63 points per block.Source: §4.1.1 · p. 12
  • Human alone participants expected the pair to exceed its better estimated component by 0.152 points per block, whereas the pair averaged 0.11 below its better measured component.Pair estimates followed AI-model estimates more than self-estimates among Human alone participants; among Human+AI participants, the associations with self and AI estimates were not distinguishable. Human+AI participants also reported higher overall-performance and reasoning-ability percentiles than Human alone participants.Source: §4.1.1 · pp. 13–14

4.2.1 Metacognitive accuracy.

  • Both groups overestimated whole-battery performance similarly (+2.02 Human+AI; +2.05 Human alone), with no group difference (−0.04, 95% CI [−1.63, +1.56]); mean absolute errors were 6.15 and 6.26 points.Both groups lowered their estimates after the battery; Human alone corrected its estimate 1.81 points more than Human+AI.Source: §4.2.1 · p. 14

4.2.2 Metacognitive sensitivity.

  • Pooled AUROC was .649 for Human alone and .564 for Human+AI (difference −.085), while mean item confidence was higher with AI (0.78 vs 0.66); within-block AUROC differences were uncertain.Sensitivity was descriptively lower with AI in three blocks, but not syllogisms, where Human+AI scores were near ceiling and few incorrect responses remained; the pooled contrast is not performance-independent.Source: §4.2.2 · pp. 14–15, §4.2.3 · p. 15

4.3.1 The quartile pattern.

  • The lowest-versus-highest-quartile overestimation contrast was 13.58 points in Human+AI and 8.87 in Human alone, a 4.71-point larger observed contrast with AI.The Human+AI group showed established overestimation in Q1 and underestimation in Q4; Human alone Q4 was not distinguishable from zero. The decrease also appeared within blocks when ranking used other items, though it was smaller.Source: §4.3.1 · pp. 16–17

4.3.2 Controls for regression to the mean and measurement error.

  • With disjoint items controlling shared-score regression to the mean, Q1–Q4 contrasts remained 8.87 points in Human+AI and 6.49 in Human alone; the controlled group difference was inconclusive.Measurement-error-corrected estimate-tracking slopes were 0.538 for Human alone and 0.071 for Human+AI. Constant-bias, constant-noise simulations reproduced about 29% and 40% of the observed contrasts, respectively, rejecting those specific accounts but not every statistical explanation.Source: §4.3.1 · p. 16, §4.3.2 · p. 18

4.3.3 Computational models of score self-assessment.

  • The fitted report-shift medians were 0.32 in Human alone and 6.26 in Human+AI; compression medians were 2.10 and 8.24, respectively.Other posterior medians (95% credible intervals) were shared person-offset variation τ = 0.85 [0.66, 1.15] and 3.82 [3.04, 4.76], and global-report shift c = 0.06 [−0.03, 0.15] and −0.62 [−1.06, −0.23]. These describe the latent report scale; the reference slopes were 0.47 [0.36, 0.58] and 0.09 [0.07, 0.11], and are not the observed-score or IV slopes. The global-report shift is not a pure aggregation effect.Source: §4.3.3 · p. 19
  • The global estimate minus summed block estimates was −0.84 points in Human+AI and +0.42 in Human alone, with a −1.26-point group difference (95% CI [−2.15, −0.37]).The Human+AI discrepancy was significant, t(178) = −2.50, p = .013, dz = −0.19; the Human-alone discrepancy was uncertain, t(186) = 1.38, p = .170, dz = 0.10. The group difference was Welch’s t(359.59) = −2.78, p = .006, g = −0.29.Source: §4.3.3 · p. 19
  • The extended specification more closely followed observed quartile contrasts than the total-only specification, but these were conditional in-sample checks rather than out-of-sample comparisons.Human+AI mental-rotation reports reached 10/10 for 33% of participants versus a posterior predictive median of 17% (95% interval [12, 22]%); for matrix reasoning, the corresponding values were 28% versus 16% %. The model did not fully reproduce ceiling mass or all block means and spreads; alternative specifications retained distributional misfit, and the low-performance part of the Human+AI mean curve was specification-sensitive.Source: §4.3.3 · pp. 19–20, Figure 9 · p. 20, §4.3.4 · p. 20

4.3.4 Item-level accuracy and confidence.

  • Human+AI accuracy rose 5.06 points out of 40 per +1 SD in AI item performance (95% CI [3.42, 6.96]), while confidence changed 0.04 [−0.67, 0.75]; Human-alone estimates were 1.55 [−1.25, 4.21] and −0.08 [−1.09, 0.93].These are descriptive accuracy-versus-confidence contrasts, not a direct statistical test of their difference; accuracy contrasts set random effects to zero and are not population-marginal effects.Source: §4.3.4 · p. 20, Figure 10 · p. 21

4.3.5 Confidence and aggregation.

  • The confidence component’s slope was −0.633 points per performance point in Human alone and −0.850 in Human+AI, while aggregation slopes showed no clear association with performance.Confidence error was calculated from summed item-confidence probabilities minus achieved score; aggregation was the whole-battery estimate minus that implied total, a relation between reports rather than an observed cognitive operation. The component slopes sum to overall overestimation slopes of −0.539 and −0.923, respectively; subtracting achieved score contributes mechanically to the confidence-component slope.Source: §4.3.5 · p. 21

5 Discussion

The discussion finds a gap between Human+AI performance and people’s ability to assess the partnership: average bias can conceal performance-dependent errors, and confidence is less discriminative across tasks. It considers evaluation and interface implications alongside limits on causal and broader interpretation.

5.1 Assessing a partnership, not only oneself

  • Human+AI participants scored higher than Human alone but judged performance less consistently across score levels; overall synergy beyond the AI model alone was not established.Human+AI outperformed the AI model alone on matrix reasoning, but not conclusively overall.Source: §5.1 · p. 22, §5 · p. 22
  • Human+AI participants’ estimates of AI performance were accurate on average but weakly tracked performance across blocks (coefficient 0.324), while Human-alone participants often expected the pair to exceed both components.24.6% of Human-alone reports placed the pair above both parts.Source: §5.1 · p. 22

5.2 Average bias conceals the distribution of self-assessment errors

  • Both groups’ mean absolute errors were about six points despite signed mean errors of about two, while the larger observed DKE contrast in Human+AI had an uncertain controlled group difference.The DKE pattern persisted within each group under disjoint-score controls; score-noise controls reduced but did not eliminate it.Source: §5.2 · pp. 22–23

5.3 Monitoring across tasks and within answers

  • Human+AI confidence distinguished correct from incorrect answers less effectively overall, but the difference was substantially smaller within the same task block.About 80% of the correct–incorrect pairs underlying the pooled measure crossed blocks.Source: §5.3 · p. 23

5.4 Implications for HAI evaluation and design

  • Evaluation should measure distributional estimation errors, absolute error, and pooled and within-task discrimination, not only average performance and signed bias.The same-item benchmark and repeated block estimates help locate where beliefs and performance diverge.Source: §5.4 · p. 23, §5.3 · p. 23
  • Interfaces could use task-specific feedback, verification prompts, or explain-back interventions, but their timing should be tested against task risk, confidence, effort, and interruption.Lower-performing users may perceive less need for feedback, while blanket confidence reduction could worsen calibration among higher-performing users.Source: §5.4 · pp. 23–24

5.5 Limitations

  • Because groups were recruited separately without random assignment and only four fixed task types were studied, group contrasts are not causal and broader reasoning-task generalization is limited.The task battery does not isolate why self-assessment tracking differs.Source: §5.5 · p. 24
  • Specified score-noise controls and bounded, error-prone reports do not uniquely establish a metacognitive deficit or psychological mechanism; the benchmark and interaction measures also differ from live use and omit verification behavior.The AI benchmark omitted instruction and practice context, while participants used one model in one session without accuracy feedback.Source: §5.5 · p. 24

6 Conclusion

The conclusion emphasizes that better Human+AI performance does not ensure better self-assessment. It calls for design that helps users judge joint work, while the included item-generation material describes designed difficulty levels rather than empirically equated ones.

6 Conclusion

  • Human+AI participants performed better than Human alone while judging that performance less consistently across score levels.Similar average bias concealed a larger observed DKE contrast and lower pooled confidence discrimination.Source: §6 · p. 24
  • HAI design should help users assess where a partnership succeeds and where answers need checking, alongside helping them produce answers.Source: §6 · p. 25

A.1 Item Difficulty

  • The administered battery’s difficulty labels are generator-designed levels, not empirical difficulty equivalences across tasks.The battery composition is summarized in Table 5.Source: §A.1 · p. 29, Table 5 · p. 29

A.2 Matrix reasoning

The included material describes reasoning-item construction and study procedures across several task families. It details task-specific design choices and constraints, along with the interface, questionnaire, and attention-check procedures.

A.2 Matrix reasoning

  • Matrix items vary relation rules across attributes and difficulty levels, with rendering and deduplication constraints intended to prevent visually ambiguous options.Element size is held constant across counts, and shared geometry keeps shapes pixel-identical between question and options.Source: §A.2 · pp. 29–30, Table 6 · p. 29

A.3 Mental rotation

  • Mental-rotation items cross depth versus picture-plane conditions with four difficulty levels, while the rotation axis switches with angle and is not separable from angular disparity.The figures use fixed geometry and camera direction, with shading reassigned after rotation.Source: Table 7 · p. 30, §A.3 · p. 30

A.4 Syllogistic reasoning

  • Syllogism items use two-premise arguments with three response categories, and difficulty tiers reflect one, two, or three permitted mental models.Three-question scenarios use term-disjoint triples, though shared narrative and interaction context can still correlate response errors.Source: §A.4 · p. 30

A.5 Letter-string analogies

  • Letter-string items combine single or ordered-pair transformations, with difficulty axes based on step size, sequence length, and grouping; alphabet changes co-vary with these axes.The verifier checks intended answers against enumerated rule hypotheses, but content-addressed rules may remain undetected and two underdetermined items accept alternate answers.Source: §A.5 · p. 31

B.1 Additional procedure and interface details

  • The Human+AI interface logged consultation and response behavior, required at least one AI message per item, and allowed participants to answer before consulting the model.The application used logged buttons to transfer item text or relevant stimulus images into chat.Source: §B.1 · p. 32

B.2 Questionnaire sections and attention checks

  • Post-task questionnaires covered participant background, AI strategy and interaction, task feedback, and attention checks; none of the 366 analyzed participants failed two or more checks.315 passed all three checks and 51 failed exactly one.Source: §B.2 · p. 32

C AI model benchmarking harness

The appendix documents the repeated-run harness behind the AI-alone reference. It describes its scoring procedure and clarifies that repeated runs sample the same fixed battery rather than independent items.

C AI model benchmarking harness

  • The AI-alone reference contains 4,000 scored item records from 100 complete runs on the same 40 items, with chat-parity prompts and rescored correctness used for the primary benchmark.Correcting the confirmed extraction error changes the benchmark mean from 27.71 to 27.72/40; scoring all twelve flagged records correct gives 27.83/40.Source: C AI model benchmarking harness · p. 32
  • Repeated runs are realizations on a fixed battery, not independent item samples, and the benchmark omits instruction and practice context and resets history between blocks.Source: C AI model benchmarking harness · pp. 32–33

D Additional analysis specifications

The additional specifications describe the frequentist regression procedures and exploratory Bayesian equivalence checks used in the analyses.

D Additional analysis specifications

  • Paired contrasts use mean individual differences divided by their standard deviation, with participant-clustered CR1 covariance and t inference using G−1 degrees of freedom.An endpoint sensitivity excludes rows where any report is 10.Source: D Additional analysis specifications · p. 33
  • Bayesian equivalence checks estimate the posterior probability that the standardized contrast lies within |d| < .30, using .95 as the practical-equivalence criterion.The margin was selected post hoc; .10 and .20 margins and a wider effect prior were sensitivity checks.Source: D Additional analysis specifications · p. 33
  • The Bayesian models are approximate mean comparisons, not models of endpoint mass or Bayes factors for exact equality.Primary fits used four chains of 20,000 iterations, including 4,000 warmup iterations per chain.Source: D Additional analysis specifications · p. 33

E Supplementary results and robustness

This supplementary section provides engagement summaries, correlations, and robustness analyses that support the main Results.

E Supplementary results and robustness

E.1 Engagement and correlations

The reported correlations cover estimation error and confidence measures, but estimation-error correlations share an achieved-score term and therefore are not independent evidence for the DKE.

E.1 Engagement and correlations

  • Correlations involving estimation error share an achieved-score term, so they are not independent evidence for the DKE.Source: §E.1 · p. 33
  • Table 9 reports correlations among metacognitive measures, including estimate-minus-performance error, the confidence gap between correct and incorrect items, and mean item confidence.Source: Table 9 · p. 33
  • Table 9 marks significance at p < .05, p < .01, and p < .001.Source: Table 9 · p. 33

E.2 Robustness procedures and sensitivities

The robustness analyses control for score noise and measurement error, while examining residual spread and sensitivity to modeling choices. Controlled DKE contrasts remained positive in both groups, but the between-group difference was inconclusive.

E.2 Robustness procedures and sensitivities

  • The split-score control ranks and evaluates performance on separate item portions; the grouping-only variant still shares items with its outcome.The naive analysis ranks and evaluates on all 40 items; the grouping-only variant ranks on 20 odd items and evaluates on all 40.Source: §E.2 · p. 34
  • The specified no-ability-dependent-bias null produced quartile differences of +3.91 (Human+AI) and +3.55 (Human alone), below observed contrasts of +13.58 and +8.87.No simulation reached either observed value (both p < .001); the simulated differences were about 29% and 40% of the observed group contrasts, respectively.Source: §E.2 · p. 34
  • The split-half IV analysis uses one item half as an instrument for performance on the other to correct measurement error, under instrument assumptions.This adapts Feld et al., who used prior GPA as an instrument for exam performance.Source: §E.2 · p. 34
  • Residual-spread slopes were −0.117 (Human+AI, p = .036) and −0.155 (Human alone, p = .001), indicating less estimate scatter among higher performers in both groups.The relation could not be distinguished from linear in either group: p = .902 for Human+AI and p = .129 for Human alone.Source: §E.2 · p. 34
  • The constant-noise bounded-response check yielded lower-tail p = .009 for Human+AI and p < .001 for Human alone.This plug-in test does not integrate uncertainty in fitted parameters or observed ability scores.Source: §E.2 · p. 34
  • Fixed-split residual-spread slopes were −0.117 (Human+AI) and −0.116 (Human alone), with 95% CIs [−0.227, −0.008] and [−0.201, −0.023].Adjusting for session duration and log median response time gave slopes of −0.164 and −0.130, respectively.Source: §E.2 · p. 35
  • 8.80 (Human+AI) and 5.96 (Human alone) were the scenario-preserving strict-control Q1–Q4 contrasts; the group difference was 2.84 points (95% CI [−2.16, 6.82]).Measurement-error-corrected tracking slopes were 0.104 (Human+AI) and 0.544 (Human alone).Source: §E.2 · p. 35
  • The 60 random item splits balanced task blocks but did not preserve shared syllogism scenarios; their ranges are exploratory sensitivity summaries, not participant-sampling confidence intervals.They are distinct from the 1,500-draw null-simulation reference intervals.Source: §E.2 · p. 35

E.3 Pooled item-level interaction sensitivities

The pooled item-level sensitivity analyses found a strong accuracy interaction but no confidence interaction; the separate tests do not establish that the accuracy and confidence slopes differ.

E.3 Pooled item-level interaction sensitivities

  • 0.957 logits per SD was the estimated accuracy interaction (Wald CI [0.730, 1.184], z=8.26, p<.001).The estimate came from supplemental pooled specifications with item-specific group contrasts.Source: §E.3 · p. 35
  • −0.0007 was the estimated confidence interaction (CI [−0.0131, 0.0117], z=−0.12, p=.908).The confidence specification had a singular random-effects fit and is treated as a sensitivity analysis.Source: §E.3 · p. 35
  • Separate interaction tests do not establish a difference between accuracy and confidence slopes.Uncertainty is conditional on the 100-run benchmark.Source: §E.3 · p. 35

F.1 Computational fit checks and priors

The extended specification reports conditional Q1–Q4 overestimation contrasts for both groups, with checks based on in-sample data rather than held-out model comparisons.

F.1 Computational fit checks and priors

  • 12.45 points in Human+AI and 8.18 in Human alone were the extended specification’s conditional expected Q1–Q4 overestimation contrasts.The corresponding total-only contrasts were 8.24 points and 4.85 points, respectively.Source: §F.1 · p. 35
  • 13.58 points in Human+AI and 8.87 in Human alone were the observed contrasts using those same quartiles.These observed contrasts accompany conditional model estimates, not held-out model comparisons.Source: §F.1 · pp. 35–36
  • The checks are conditional and in-sample, not held-out model comparisons.Figure-linked summaries use 2,000 posterior draws; WAMBS contrast summaries use all retained draws.Source: §F.1 · p. 36

F.2 Retrospective WAMBS review

The retrospective WAMBS review found stable score-level predictions and equivalence conclusions under tested sampling and prior changes, but some ceiling misfit and limits on the conclusions remain.

F.2 Retrospective WAMBS review

  • The completed WAMBS review found no substantive change in reviewed score-level predictions or equivalence conclusions under longer sampling and tested priors.Correlated latent parameters shifted together, while some predictive ceiling misfit remained.Source: §F.2 · p. 36
  • Across 50 fits, maximum R̂ was 1.00089, minimum bulk/tail ESS were 8,221/13,006, and minimum E-BFMI was .767, with no divergences or maximum-treedepth hits.Doubling sampling changed computational contrast medians by at most .0055 points and interval endpoints by .0167; some decimal-level precision targets remained unmet without changing a substantive comparison.Source: §F.2 · p. 36
  • Across computational priors, extended conditional quartile contrasts ranged from 12.39 to 12.48 points in Human+AI and 8.17 to 8.20 in Human alone.The primary b–σ correlations were .914 for the extended specification and .995 for the total-only specification; parameter trade-offs leave similar score predictions.Source: §F.2 · p. 37
  • All tested settings supported practical equivalence at the post-hoc |d| < .30 margin for mental rotation, signed error, absolute error, and Human alone’s two percentile judgments versus 50.This is equivalence of mean contrasts, not accurate individual judgment.Source: §F.2 · p. 37
  • Total performance versus the AI model benchmark and within-block AUROC remained inconclusive.The reported probabilities were P=.932–.937 for total performance and P=.881–.886 for within-block AUROC.Source: §F.2 · p. 37
  • Stable conditional quartile contrasts do not replace the disjoint-score test, whose between-group interval includes zero.The reviewed settings preserve the main conclusions without establishing a larger controlled DKE or a unique latent mechanism.Source: §F.2 · p. 37
  • Primary predictions still underrepresent some Human+AI ceiling reports; alternative mean/likelihood specifications were not adopted and remain outside this review.The review supports computational stability, not the adequacy of every report-distribution assumption.Source: §F.2 · p. 37

Related papers

Loading 2609.31095v1…