PaperlayerView exact revision on arXiv

Source-linked AI summary

Thinking effort aligns between humans and reasoning models in abductive reasoning

Henry Arthur

arXiv:2609.01867v1cs.CLcs.AI

TL;DR

Whether model reasoning traces align with human effort remains a question in cognitive modeling. This study compares human reaction times with LRM reasoning-token costs in abductive forced choice, finding similar item success and failure and stronger alignment under stochastic decoding averaged across runs.

  • Problem

    How closely model reasoning effort aligns with human effort remains an open question in cognitive modeling.

    Source: §1 · p. 1

  • Method

    The study compares human reaction times with reasoning-token counts on a commonsense abductive forced-choice task and tests decoding strategies across multiple models.

    Source: §2.5 · p. 4, §3 · p. 5, §5 · p. 8

  • Results

    Human and reasoning-model effort showed significant alignment and similar item-level successes and failures, while stochastic decoding averaged across runs strengthened alignment; GPT-OSS-120B explained R2 = .303 of residualized item-level reaction-time variance.

    Source: §4 · pp. 7–8, §5 · p. 8

  • Takeaways & Limitations

    The findings support chain-of-thought length as a proxy for cognitive effort and suggest within-task alignment may be stronger than previously reported.

    Source: §5 · p. 8

    The authors cannot rule out that models encountered ART items during pretraining, which could inflate accuracy through familiarity rather than reasoning.

    Source: Limitations · p. 9

Abstract

from arXiv · show

A major question in cognitive modeling concerns the behavioral alignment between large language models and humans across linguistic and non-linguistic tasks. Unlike standard LLMs, large reasoning models (LRMs) are optimized with reinforcement learning from verifiable rewards, encouraging correct solutions to reasoning tasks rather than preference-aligned responses. Recent work (de Varda et al., 2025) investigates the cost of thinking in humans and LRMs by comparing human reaction times with model reasoning traces across a range of reasoning tasks. We isolate this alignment by turning to abductive reasoning: unlike deductive tasks, its difficulty cannot be inferred from formal structure and offers no shortcuts a model could exploit to mimic effort without genuine search, providing firmer ground for empirical claims of shared effort. We find further evidence of alignment between LRM and human reasoning effort, as well as evidence that models and humans tend to make similar errors. Finally, we show that decoding methods that let models explore multiple reasoning paths increase alignment in reasoning cost between humans and LRMs across the three models tested.

1 Introduction

This paper asks how closely model reasoning effort aligns with human cognition, against a background of uncertainty about whether language models generalize beyond linguistic competence. It uses abductive reasoning, where difficulty depends on searching for explanations rather than formal cues, to examine that alignment.

1 Introduction

  • Language models’ relation to human cognition remains contested, including whether linguistic competence underlies thinking and whether models generalize to reasoning beyond language.LRMs use reinforcement learning from verifiable rewards to optimize correct answers, and their reasoning-token traces can serve as a proxy for computational effort; however, the relation between emitted traces and underlying computation remains unclear.Source: §1 · pp. 1–2

1.1 Related Work

  • Prior work found alignment between LRM reasoning-trace token counts and human reaction times across reasoning tasks, while debate continues about whether that alignment implies algorithmic correspondence or validates chain-of-thought as an effort measure.The paper frames alignment as an empirical phenomenon, not a mechanistic thesis, and tests multiple decoding strategies and effort levels in response to measurement concerns.Source: §1.1 · p. 2

1.2 Abductive Reasoning

  • Abduction tests effort alignment because its underdetermined evidence requires searching among explanations, leaving no formal-structure shortcut for mimicking item difficulty.Unlike syllogistic and deductive tasks, abductive difficulty cannot be recovered from formal structure alone.Source: §1.2 · p. 2

1.3 Abduction in LLMs

  • Earlier studies report that language models often underperform humans on abduction, across syllogistic, commonsense, and medical-diagnosis tasks.The literature also reports that explicit inference-to-the-best-explanation criteria can improve model-based evaluation of commonsense explanations.Source: §1.3 · p. 2

1.4 Abductive Reasoning Dataset

  • The study operationalizes abduction as choosing which of two candidate hypotheses best explains two observations, using items from an adversarially filtered commonsense dataset.The source dataset uses story beginnings and endings as observations and selects hypothesis pairs that are difficult to distinguish.Source: §1.4 · p. 3

1.5 Decoding

  • Temperature-based sampling can explore alternative reasoning paths and reduce repetitive traces, motivating tests of whether multi-run decoding strengthens human–model effort alignment.The paper adapts de Varda et al.’s method to abductive inference and reports temperature-based sampling aggregated across runs as strengthening human–model correlations.Source: §1.5 · p. 3

2 Methods

The study measured human choices and reaction times on a forced-choice abductive task, then compared them with reasoning-model outputs and token costs. Its methods cover participant recruitment, task allocation and procedure, item selection, and model evaluation under greedy and sampled decoding.

2 Methods

  • Participants chose which of two hypotheses best explained two observations in an online forced-choice experiment and were instructed to respond quickly while remaining accurate.The task used 160 items from the Abductive Natural Language Inference Dataset.Source: §2 · p. 3

2.1 Participants

  • The final human sample comprised 120 participants after seven of 127 recruits were excluded for implausibly fast or unusually long responses.Participants were native English speakers residing in the United States; mean age was 33.0 years (SD = 6.6).Source: §2.1 · p. 3

2.2 Design

  • Participants were split across four blocks of 40 unique scenarios, with no problems repeated between blocks.Block sizes were 28, 31, 32, and 29 participants.Source: §2.2 · p. 3

2.3 Procedure

  • The English-language survey used randomized Q/P response keys, comprehension checks, practice items, attention trials, and unlimited response time.A fixation cross appeared for 200 ms between trials; mean completion time was 9.2 minutes (SD = 4.0).Source: §2.3 · pp. 3–4

2.4 Materials

  • The researchers sampled 162 ART test items, retaining 160 main items and two practice items after matching hypothesis lengths and removing duplicate observation pairs.The test items had undergone adversarial filtering to make the competing hypotheses harder to distinguish.Source: §2.4 · p. 4

2.5 Model evaluation

  • The evaluation included seven LRMs and DeepSeek V3 as a non-reasoning baseline, using greedy decoding and model reasoning traces to measure token cost.The prompt mirrored the human instructions; DeepSeek V3 received an appended step-by-step cue, and its total generated tokens measured effort.Source: §2.5 · p. 4, Figure 2 · p. 4
  • For GPT-OSS-20B, Qwen3-32B, and R1, the researchers also tested temperatures 0.6, 1, and 2 across repeated runs and assessed convergence of partial correlations with human reaction times.They used bootstrap resampling and a cross-check based on convergence of log-transformed token counts to estimate the required run counts.Source: §2.5 · pp. 4–5

3 Results

Across abductive items, model reasoning-token counts aligned with human reaction times, and humans and models tended to succeed or fail on similar items. Averaging stochastic runs at recommended temperatures strengthened reasoning-cost alignment for all three tested models.

3 Results

  • The analysis relates per-item reasoning-token counts to mean human reaction times using partial correlations that control for prompt length.The authors report that alternate response subsets did not yield significantly different correlations; using prompt word count instead of token count also left results unchanged except for GPT-OSS-120B, whose token-based estimate was marginally higher.Source: §3 · p. 5

3.1 Greedy Decoding Correlations

  • Under greedy decoding, every model’s reasoning-token count correlated significantly with human reaction time after controlling for prompt length.GPT-OSS-120B had the strongest alignment (r = 0.552, p < .001), while DeepSeek-V3, the non-reasoning model, had the weakest (r = 0.197, p = .012).Source: §3.1 · p. 5, §3 · p. 5, Figure 4 · p. 6

3.2 Greedy Error Alignment

  • Human and model accuracy tracked item difficulty: human accuracy was 48.7% when at least 75% of models failed and 84.6% when at least 75% agreed on the correct answer.Across items, model-consensus accuracy correlated with human accuracy at r = 0.668 (p < .001), controlling for prompt length.Source: §3.2 · pp. 5–6

3.3 Stochastic Decoding Correlations

  • Stochastic averaging at recommended temperatures increased partial correlations with human reaction times for all three models, significantly for GPT-OSS-20B and Qwen3-32B.GPT-OSS-20B rose from r = 0.41 to r = 0.55; Qwen3-32B rose from r = 0.23 to r = 0.32; R1 rose from r = 0.32 to r = 0.37, not significantly.Source: §3.3 · p. 6, Figure 5 · p. 7

3.4 Stochastic Error Alignment

  • Token-count variance across stochastic runs did not predict human reaction time beyond the per-item mean token count.The reported partial correlations ranged from −.12 to +.08, with all ps > .12.Source: §3.4 · p. 6
  • At recommended temperatures, model-human accuracy alignment varied by model: GPT-OSS-20B peaked at r = 0.51 at temperature 1, while R1 peaked at r = 0.53 at temperature 0 and Qwen3 at r = 0.63 at temperature 0.6.All reported correlations had p < .001; across models, recommended temperatures produced the strongest alignment across accuracy and reaction-time measures.Source: §3.4 · p. 7, Figure 5 · p. 7

4 Discussion

The discussion argues that abductive reasoning offers a firmer test of shared effort than deductive tasks, where formal structure may reveal difficulty. It interprets the observed alignment as substantial but bounded by measurement noise, and reports that recommended-temperature sampling improves alignment while higher temperatures are less stable.

4 Discussion

  • The greedy ensemble’s mean partial correlation was r = 0.42, close to the syllogism result (r = 0.43) and above relational reasoning (r = 0.27) in de Varda et al. (2025).The discussion notes that abductive forced-choice reasoning, syllogisms, and relational reasoning involve structured, grounded scenarios.Source: §4 · p. 7
  • Abductive tasks provide stronger evidence of shared effort because their underdetermined evidence prevents models from using formal structure as a shortcut to apparent difficulty.In contrast, deductive task difficulty can be partly recovered from formal structure.Source: §4 · p. 7
  • All tested LRMs outperformed the non-reasoning V3 model, consistent with reasoning traces capturing effort beyond ordinary next-token generation.The paper characterizes this as an empirical alignment finding, not a mechanistic claim about brain-model correspondence.Source: §4 · p. 7
  • GPT-OSS-120B explains 35% of the variance in human reaction times that is explainable given the estimated noise ceiling.The noise ceiling was estimated from bootstrapped split-half reliability of human reaction times, corrected using Spearman-Brown (r = .861).Source: §4 · p. 7
  • Averaging stochastic runs reduces noise from idiosyncratic samples, analogous to pooling human reaction times across participants.The authors recommend quantifying convergence across runs, especially because higher temperatures introduce more variance and require more runs for stable correlations.Source: §4 · p. 8, Figure 6 · p. 8
  • At temp = 2, correlations were less stable, took longer to converge, and remained lower than recommended settings such as temp = 0.6; recommended temperature produced the strongest accuracy and reaction-time alignment.Source: §4 · p. 8
  • Model size did not predict alignment: GPT-OSS-20B showed stronger alignment than the much larger DeepSeek-R1 and Kimi-K2 models.GPT-OSS-20B has 20B parameters, compared with 671B for DeepSeek-R1 and 1T for Kimi-K2.Source: §4 · p. 8
  • Greedy decoding’s weaker alignment appears to reflect single-run noise attenuation rather than extreme token-count outliers.Log transformation limited outlier leverage, and removing the worst outliers did not meaningfully change partial r.Source: §4 · p. 8

5 Conclusion

In a commonsense abductive task, human and reasoning-model effort aligned, and they succeeded and failed on similar items. Stochastic decoding further strengthened alignment when results were averaged across runs.

5 Conclusion

  • Human thinking effort significantly correlated with all tested reasoning models, and humans and models succeeded and failed on similar abductive items.Abduction difficulty cannot be inferred from formal structure alone.Source: §5 · p. 8
  • Stochastic decoding, averaged over many runs, revealed stronger alignment between human and model thinking effort.This suggests within-task alignment in de Varda et al. (2025) may be stronger than originally reported.Source: §5 · p. 8
  • Stochastic decoding may reflect a cognitive process more similar to humans than greedy decoding.The authors also propose treating hyperparameters as experimental variables, analogously to controlling environmental conditions in human experiments.Source: §5 · p. 8

6 Future work

Future work could test stochastic sampling across more reasoning models, examine additional sampling hyperparameters, and apply the method to other abductive tasks and reasoning measures.

6 Future work

  • Testing stochastic sampling across more reasoning models could clarify what drives the differential increase in alignment.The authors also identify top_p and top_k, kept at default values here, as hyperparameters for future study.Source: §6 · p. 9
  • Adapting the method to long-form detective tasks could test abductive reasoning that requires deeper reasoning and connecting clues across context.The authors describe these as examples of other abductive reasoning datasets.Source: §6 · p. 9
  • Future measures could examine CoT traces beyond linguistic output, including continuous latent space, or mechanistic properties such as token-revision counts in deep layers.Source: §6 · p. 9

Limitations

The limitations concern the tested task, model instructions and possible item familiarity, as well as the default reasoning-effort setting used in the main analyses.

Limitations

  • Effort increased correctness on the hardest problem tercile, but main analyses used default reasoning-effort settings rather than difficulty-tuned settings.The authors suggest alignment may be stronger with datasets calibrated to avoid ceiling effects.Source: Limitations · p. 9
  • The study tests inference to the best explanation, one abductive reasoning type that does not capture all forms or differences in hypothesis selection and generation.Source: Limitations · p. 9
  • Although prompts were made as close as possible to human directions, prompt sensitivity means instruction variation could affect results.The passage cites Zhao et al. (2021) and Sclar et al. (2024) on prompt sensitivity.Source: Limitations · p. 9
  • The authors cannot rule out ART-item exposure during pretraining, which could inflate accuracy through familiarity rather than genuine reasoning.Shared error patterns suggest ground-truth labels were not simply memorized, but do not eliminate this possibility.Source: Limitations · p. 9

Ethical Considerations

The authors frame alignment as evidence of shared computational constraints, not equivalence between humans and models, and report participant privacy and consent safeguards.

Ethical Considerations

  • Observed alignment does not imply equivalence between models and humans or justify anthropomorphizing them or increasing trust in their outputs.The authors say their focus is evidence of shared computational constraints relevant to cognitive modeling.Source: Ethical Considerations · p. 9
  • No personally identifiable information was collected, and participants gave informed consent for use of anonymized accuracy and reaction-time data.Participants were recruited and compensated through Prolific.Source: Ethical Considerations · p. 9

Data Availability

The appendix provides the code and data location, discusses the scope of the DeepSeek-V3 comparison, and cautions that model size may not explain alignment differences on commonsense tasks.

Data Availability

A Frequently asked questions

  • The authors suggest harder abductive problems may distinguish reasoning models from standard LLMs more clearly than commonsense scenarios represented in pretraining.They note DeepSeek-V3 had the weakest alignment tested, despite no statistically significant difference from DeepSeek-R1.Source: §A.1 · p. 12

A.2 Why does alignment not scale with model size?

  • The authors suggest that training dynamics and activated-parameter counts may matter more than raw model size for alignment on this task.They caution that commonsense reasoning may be a poor setting for detecting scale effects because models can draw on pretrained world knowledge.Source: §A.2 · p. 12

A.3 Do the results change under different reasoning-effort parameters?

The appendix reports that GPT-OSS-20B's token–RT alignment is similar at low and high reasoning effort, with an accuracy advantage for high effort on the hardest items. It also lists model specifications and identifies tables covering hyperparameters, correlations, and greedy-decoding accuracy.

A.3 Do the results change under different reasoning-effort parameters?

  • GPT-OSS-20B token–RT alignment held at low and high effort settings, with low r = +.34 and high r = +.30 (both p < .001).The difference was not significant by Steiger’s test (p = .60); the models were tested across all 160 items.Source: §A.3 · p. 12
  • 69.2% vs. 53.8% accuracy on the hardest tercile favored high effort over low effort, while aggregate accuracy was similar at 82.5% vs. 86.3%.The hardest-tercile comparison used n = 52 and had McNemar exact p = .022; the effort-by-human-accuracy interaction was β = −3.71, p = .016.Source: §A.3 · p. 13

B Model Specifications

  • The specifications describe DeepSeek-R1 and several listed models as Mixture-of-Experts, while Qwen3-32B is dense and has a dedicated thinking mode that can be toggled.DeepSeek-R1 uses reinforcement learning for correct answers on verifiable tasks; the remaining listed specifications include model parameter counts.Source: B Model Specifications · p. 13

C Hyperparameters

  • Table 1 catalogs hyperparameters by model and marks provider defaults and values not specified in code or by providers.Source: Table 1 · p. 13

D Pearson and Spearman Correlations

  • Table 2 reports partial Pearson and Spearman correlations for models under greedy and stochastic decoding.Source: Table 2 · p. 13

E Human vs Model Accuracy

  • Table 3 covers abductive-task accuracy under greedy decoding, with model Wilson-score and human bootstrap 95% confidence intervals.The table uses n = 160 and 2,000 human bootstrap resamples.Source: Table 3 · p. 13

F Chain-of-Thought Convergence

The appendix estimates how many stochastic reasoning runs are needed to meet a standard-error threshold for log token counts and compares that estimate with bootstrap convergence. It also describes the ordering used to display GPT-OSS-20B token counts under stochastic decoding.

F Chain-of-Thought Convergence

  • The analytical estimate k* is the run count needed to bring the standard error below threshold θ in log-token space.Table 4 reports median per-item SD of log thinking tokens and k*, with a dagger marking an exhausted pool.Source: F Chain-of-Thought Convergence · p. 13, Table 4 · p. 14
  • The analytical run-count estimates ranged from 5 to 18, and partial correlations at the estimated and full run counts did not differ significantly.The two convergence criteria agreed on the ordering of models and temperatures.Source: F Chain-of-Thought Convergence · p. 13
  • At temperature 2.0, estimated run counts exceeded the available pool for GPT-OSS-20B and Qwen3-32B, though bootstrap convergence was found for Qwen3-32B at 24 runs.For GPT-OSS-20B, the estimate corroborated the original finding of non-convergence.Source: F Chain-of-Thought Convergence · p. 13

G Stochastic Attenuation

  • Figure 7 plots GPT-OSS-20B thinking tokens per item on a log scale, ordering items by stochastic mean token count from low to high difficulty.Source: Figure 7 · p. 14

Related papers

Loading 2609.01867v1…