PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Reasoning Capacity of AI Models and How to Quantify It

Santosh Kumar Radha, Oktay Goktas

arXiv:2501.13833v1cs.AIcs.CLcs.ITmath.IT

TL;DR

Accuracy and benchmark performance alone provide incomplete evidence about whether language models genuinely reason. The paper applies phenomenological models to controlled positional perturbations, finding that logical deduction remains challenging and apparent success often involves memorization and pattern matching.

  • Problem

    Accuracy-centered benchmarks provide incomplete insight into whether model success reflects logical deduction or other cognitive processes.

    Source: §I · p. 2

  • Method

    The paper uses controlled positional perturbations with a Probabilistic Mixture Model to decompose strategies and an Information-Theoretic Consistency analysis to quantify uncertainty.

    Source: §I · pp. 2–3

  • Results

    Position-invariant logical deduction remains challenging, and apparent success often relies on combinations of memorization and pattern matching.

    Source: §I · p. 3, §V · p. 18

  • Takeaways & Limitations

    Aggregate accuracy can overstate reasoning ability, while strategy distributions offer quantitative criteria for evaluating reliability in deployment scenarios.

    Source: §V · p. 18

Abstract

from arXiv · show

Recent advances in Large Language Models (LLMs) have intensified the debate surrounding the fundamental nature of their reasoning capabilities. While achieving high performance on benchmarks such as GPQA and MMLU, these models exhibit limitations in more complex reasoning tasks, highlighting the need for more rigorous evaluation methodologies. We propose a novel phenomenological approach that goes beyond traditional accuracy metrics to probe the underlying mechanisms of model behavior, establishing a framework that could broadly impact how we analyze and understand AI systems. Using positional bias in multiple-choice reasoning tasks as a case study, we demonstrate how systematic perturbations can reveal fundamental aspects of model decision-making. To analyze these behaviors, we develop two complementary phenomenological models: a Probabilistic Mixture Model (PMM) that decomposes model responses into reasoning, memorization, and guessing components and an Information-Theoretic Consistency (ITC) analysis that quantifies the relationship between model confidence and strategy selection. Through controlled experiments on reasoning benchmarks, we show that true reasoning remains challenging for current models, with apparent success often relying on sophisticated combinations of memorization and pattern matching rather than genuine logical deduction. More fundamentally, we demonstrate that accuracy alone often overstates a model's reasoning abilities, as model behavior can be characterized through underlying mechanisms in the phase space of cognitive strategies, revealing how models dynamically balance different approaches when responding to queries. This framework enables quantitative criteria for real-world deployments, allowing applications to specify reliability thresholds based on strategy distributions rather than aggregate performance metrics.

I. INTRODUCTION

Accuracy-based evaluation does not reveal whether language models solve reasoning tasks through logical deduction or other strategies. The paper proposes a phenomenological framework using controlled perturbations to characterize observable behavior and distinguish those strategies.

I. INTRODUCTION

  • Existing prompting and self-consistency approaches structure model inputs or outputs but do not directly reveal underlying decision processes.Reported gains may reflect better exploitation of model priors rather than stronger logical deduction.Source: §I · p. 2
  • Benchmark accuracy and F1 scores offer limited insight into strategy changes under task complexity or input perturbations.The paper argues that aggregate performance metrics treat behavior as a black box and miss the interplay between cognitive strategies.Source: §I · p. 2
  • The proposed framework combines a Probabilistic Mixture Model (PMM) for decomposing strategies with Information-Theoretic Consistency (ITC) analysis of decision uncertainty.The PMM distinguishes reasoning, memorization, and guessing; ITC examines uncertainty across cognitive modes.Source: §I · pp. 2–3
  • The framework quantifies observable behavior under controlled conditions but is not intended as a complete description of model internals.Its predictions can be tested under systematic perturbations, with positional bias serving as the primary experimental probe.Source: §I · p. 3
  • Positional perturbations reveal that position-invariant logical deduction remains challenging, while apparent success can rely on combinations of memorization and pattern matching.The introduction frames positional bias as a probe of strategy selection and reports relationships between strategy choice and prediction uncertainty.Source: §I · p. 3

II. RELATED WORK: BENCHMARKS, BIAS, AND REASONING IN LANGUAGE MODELS

Prior work questions whether benchmark success reflects robust reasoning or reliance on dataset patterns, heuristics, and structural cues. This paper uses positional bias as a diagnostic probe and develops phenomenological models to quantify behavior under controlled perturbations.

Benchmarks and reasoning

  • GSM-Symbolic studies report performance declines after slight phrasing changes or irrelevant numerical adjustments despite unchanged underlying reasoning [9].Related work presents these findings as evidence of surface-level memorization and heuristic reliance.Source: §II · pp. 3–4

Positional bias

  • Unlike approaches that mitigate positional bias, this work manipulates input positions to probe dependencies that may shape reasoning-task predictions [33] [35].Prior studies describe positional effects across tasks and associate them with causal attention and positional embeddings [29] [30].Source: §II · p. 4

Phenomenological modeling

  • Phenomenological analyses of neural networks have examined training dynamics and emergent behavior, while systematic analysis of reasoning models remains sparse [37].The paper positions its models as capturing observable behavior under controlled perturbations and predicting strategy selection.Source: §II · p. 4

III. MODELS AND FRAMEWORK

The paper begins its framework analysis with positional effects in GPT-4o-mini on GPQA. This focused model-benchmark study is intended to inform a broader investigation of reasoning behavior.

III. MODELS AND FRAMEWORK

  • The initial positional-effects analysis focuses on GPT-4o-mini’s performance on the GPQA benchmark.The paper describes GPQA as a multiple-choice benchmark probing logical and analytical thinking.Source: §III · p. 4
  • The focused study is intended to show how positional information influences reasoning behavior in one model-benchmark pair.The paper notes that positional-bias effects can vary with model architecture, parameters, and dataset characteristics.Source: §III · p. 4
  • The GPQA analysis establishes a methodological foundation for the paper’s subsequent broader investigation of reasoning.Source: §III · p. 4

A. Positional Bias Analysis

The analysis uses GPQA multiple-choice questions to examine how position randomization affects model behavior. It tracks accuracy across answer positions and combines average accuracy with position-dependent variance to characterize performance and positional bias.

A. Positional Bias Analysis

  • GPQA questions have four answer positions, A–D, with exactly one correct option among each question’s four choices.The analysis treats the model as a black box that generates a probability distribution over answer positions.Source: A. Positional Bias Analysis · p. 4
  • The order parameter θ controls randomization: θ = 0 keeps correct answers fixed, θ = 1 fully randomizes them, and intermediate values randomize that fraction of questions.The experiments also compare fixed and randomized presentations at θ = 0.5.Source: A. Positional Bias Analysis · p. 5
  • The difficulty map Ψ(q, θ) = (μ(q, θ), σ²(q, θ)) pairs position-averaged accuracy with cross-position variance; high μ and low σ² indicate consistent reasoning ability.High variance suggests stronger positional dependence, regardless of average accuracy.Source: A. Positional Bias Analysis · p. 5

B. Probabilistic Mixture Model

The Probabilistic Mixture Model represents responses as a mixture of memorization, reasoning, and guessing strategies. It uses accuracy differences across memorized and other positions to estimate strategy shares; the reported decomposition assigns the largest share to memorization.

B. Probabilistic Mixture Model

  • The model represents each response as a mixture of memorization, reasoning, and guessing probabilities that sum to one, with pure guessing assigned probability 1/|O|.Memorization depends on answer position, while reasoning and guessing are modeled as position-independent success probabilities.Source: B. Probabilistic Mixture Model · pp. 5–6
  • The framework assumes perfect reasoning and random guessing as an ideal baseline, explicitly distinguishing that benchmark from a claim that real-world reasoning is perfect.Under the baseline, reasoning always yields the correct answer regardless of its position.Source: B. Probabilistic Mixture Model · p. 6
  • The accuracy difference between memorized and other positions isolates memorization probability; a larger difference indicates stronger memorization behavior.The passage illustrates a 0.35 accuracy difference as evidence suggestive of positional bias.Source: B. Probabilistic Mixture Model · pp. 6–7
  • 47% memorization, 26% reasoning, and 27% guessing are the reported strategy shares, with memorization contributing the largest share.The analysis presents these shares as a decomposition of the model’s decision-making process.Source: B. Probabilistic Mixture Model · p. 7
  • Combining strategy shares with position-averaged accuracy and positional variance supports analysis across question types and difficulty levels.The passage describes this combination as enabling a comprehensive analysis of model behavior.Source: B. Probabilistic Mixture Model · p. 7

C. Information-Theoretic Consistency Analysis

The Information-Theoretic Consistency analysis measures prediction uncertainty with Shannon entropy and compares it with an ideal entropy–accuracy frontier. Deviations from that frontier indicate overconfidence or underconfidence, complementing positional accuracy and variance measures.

C. Information-Theoretic Consistency Analysis

  • Shannon entropy measures uncertainty in the model’s probability distribution over answer options; lower entropy indicates more concentrated probabilities and higher confidence.The analysis uses this measure alongside accuracy to assess how confidence relates to correctness.Source: C. Information-Theoretic Consistency… · p. 7
  • For four-option GPQA questions, maximum entropy is 2 bits at random-guessing accuracy 0.25, while the zero-accuracy limit is about 1.58 bits.The ideal frontier distributes probability c to the correct option and the remainder uniformly among the k−1 incorrect options, producing an asymmetric entropy profile.Source: C. Information-Theoretic Consistency… · pp. 7–8, Figure 2 · p. 7
  • Empirical points below the ideal frontier indicate overconfidence, while points above it indicate underconfidence.Combined with position-averaged accuracy and positional variance, frontier deviations help assess how position dependence relates to prediction uncertainty.Source: C. Information-Theoretic Consistency… · p. 8

IV. RESULTS AND DISCUSSION

In GPT-4o-mini experiments on GPQA-Diamond, incorrect-answer choices varied with the correct answer’s position, with position D selected less often as a distractor. The experiments varied answer placement while holding question content constant.

IV. RESULTS AND DISCUSSION

  • Experiments used GPT-4o-mini on 198 GPQA-Diamond questions spanning physics, mathematics, and computer science, varying correct-answer placement while preserving question content.This design was intended to isolate position-dependent effects.Source: §IV · p. 8
  • Each question received 100 trials across answer-position configurations, using direct answer elicitation without Chain-of-Thought or Iteration-of-Thought prompts.The authors identify testing reasoning-enhancing prompts as future work.Source: §IV · p. 9
  • Position D had incorrect-choice selection probabilities below 0.15, versus about 0.2–0.25 for other positions, indicating asymmetric wrong-answer distributions.The analysis also found a subtle preference for adjacent incorrect choices when the correct answer was at A or B; this pattern diminished toward D.Source: Figure 3 · p. 8, §IV · p. 9

A. Position-Dependent Accuracy Under Randomization as order-parameter

Controlled randomization reveals position-dependent accuracy patterns, including a persistent disadvantage for position D. A two-dimensional accuracy–variance map further separates consistent performance from position-sensitive success and difficulty.

A. Position-Dependent Accuracy Under Randomization as order-parameter

  • Under inclusive randomization, position-specific accuracies converge toward the random-chance level of 0.25; under exclusive randomization, the accuracy curves intersect at θ = 0.5.Inclusive randomization allows the correct answer at any position; exclusive randomization prevents it from occupying the original position.Source: A. Position-Dependent Accuracy Under… · p. 10
  • At θ = 1, position D’s inclusive–exclusive accuracy difference reaches −0.04, while positions A, B, and C remain within |∆µo(θ)| < 0.01.The D difference decreases monotonically as randomization increases, indicating a persistent disadvantage when correct answers are prevented from appearing there.Source: Figure 5 · p. 10, A. Position-Dependent Accuracy Under… · p. 10
  • In both randomization protocols, variance follows an inverted U-shape, peaking at intermediate θ as behavior transitions between position-dependent and randomized modes.The difference between inclusive and exclusive accuracy is used to assess associations between specific positions and correct answers.Source: A. Position-Dependent Accuracy Under… · p. 10
  • The accuracy–variance map’s high-µ, low-σ² region indicates consistent success across answer positions, while high-µ, high-σ² questions show position-dependent success.The latter pattern can inflate apparent reasoning capability; µ(q) is position-averaged accuracy and σ²(q) is positional variance.Source: Figure 6 · p. 11, A. Position-Dependent Accuracy Under… · p. 11
  • About 23% of questions fall in the low-µ, low-σ² region, representing consistent, position-independent difficulty; low-µ, high-σ² questions combine poor performance with positional sensitivity.The authors interpret the former as conceptual difficulties and the latter as reasoning breakdown with conflicting positional cues.Source: A. Position-Dependent Accuracy Under… · p. 11

B. Decomposing Language Model Decisions: Reasoning, Memorization, or Guessing?

The probabilistic mixture model decomposes answers into reasoning, memorization, and guessing, using position-sensitive performance to estimate each strategy. Its fit is strong for medium- and high-accuracy questions but weaker where guessing and heuristics dominate, while randomization reveals position-specific strategy dynamics.

Strategy decomposition

  • The model infers memorization from position-specific performance and derives reasoning and guessing probabilities from the remaining behavior.Position-invariant performance is treated as reasoning, while strong position-specific preference indicates memorization.Source: B. Decomposing Language Model… · p. 11

Model validation

  • For αobserved > 0.4, predicted and observed accuracies closely agree; deviations grow below 0.4, where the linear decomposition omits higher-order effects.The authors identify positional-reasoning cross-terms, non-uniform distractor selection, and partial understanding as possible sources of low-accuracy deviations.Source: Figure 7 · p. 12, B. Decomposing Language Model… · p. 12

Strategy distributions

  • Questions cluster along simplex edges, and the absent PR–PG edge indicates memorization contributes even when reasoning or guessing dominates.Interior cases are sparse and tend to have PM > PR > PG; poorest model agreement occurs near PR ≈ 0 with memorization and guessing contributing.Source: Figure 8 · p. 12, B. Decomposing Language Model… · p. 13

Randomization and strategy evolution

  • As positional randomization θ increases, reasoning contributions grow across answer positions, consistent with a shift toward position-independent strategies.Positions A, B, and C begin with similar balanced but memorization- and guessing-dominated distributions; position D instead starts with more guessing and less memorization.Source: B. Decomposing Language Model… · p. 13

Position-specific behavior

  • At position D, increasing randomization reduces guessing and raises memorization, without producing the reasoning gains seen as a general trend across positions.For this model and GPQA-like questions, D-position answers involve less memorization than other positions but more guessing rather than enhanced reasoning.Source: B. Decomposing Language Model… · p. 13

Strategy-space dynamics

  • Positions A–C show coherent probability flows, whereas position D has vortices and critical points where small input perturbations can shift strategy substantially.The authors connect these patterns to performance degradation when key information appears at sequence endings.Source: B. Decomposing Language Model… · p. 14
  • The flow analysis interpolates strategy probabilities over the simplex using a conservative dynamical system, with symplectic interpolation applied to 198 GPQA questions.The resulting stream plots distinguish flow patterns for fixed correct-answer positions.Source: B. Decomposing Language Model… · p. 14

Attractors

  • Stable attractors emerge around PR ≈ 0.4–0.5 and PM ≈ 0.3–0.4, marking configurations that remain consistent under positional perturbations.The paper relates these attractors to balanced pattern-matching and reasoning configurations, while noting that their origin in architecture or training remains an open question.Source: B. Decomposing Language Model… · pp. 14–15

C. Information-Theoretic Frontier of Model Confidence

The information-theoretic analysis compares empirical accuracy–entropy pairs with a theoretical calibration frontier. Across observed questions, entropy falls below the frontier, especially at low to moderate accuracy, indicating overly concentrated probabilities relative to performance.

Calibration frontier

  • All observed questions fall below the theoretical entropy–accuracy frontier, indicating under-dispersion of probability mass relative to ideal calibration.The frontier represents the maximum entropy achievable at each accuracy for a four-option multiple-choice system.Source: Figure 10 · p. 15, C. Information-Theoretic Frontier of… · p. 15

Accuracy-dependent uncertainty

  • For α < 0.7, entropy is typically 0.2–0.5 bits below Hideal(α); the gap narrows for α > 0.7.The paper interprets the persistent gap on challenging questions as a limitation in expressing appropriate uncertainty.Source: C. Information-Theoretic Frontier of… · pp. 15–16

Interpretation

  • The authors suggest that excessive reliance on learned positional heuristics may contribute to overconfidence when those heuristics have limited predictive value.They connect this proposed mechanism to the positional biases discussed earlier in the paper.Source: C. Information-Theoretic Frontier of… · p. 16

D. Geometric Characterization of Strategy-Uncertainty Interplay

The strategy–uncertainty analysis maps accuracy and entropy across reasoning, memorization, and guessing mixtures. It finds that accuracy, confidence, and strategy composition vary systematically, with reasoning associated with high accuracy and low entropy, while mixed strategies show more complex uncertainty.

Strategy contributions by entropy

  • Low entropy (H < 0.5) coincides with PR ≈ 0.8, while strategies contribute roughly equally near H ≈ 0.75.Above that uncertainty threshold, the authors describe a shift from reasoning toward increasingly heuristic approaches.Source: D. Geometric Characterization of… · pp. 16–17

Deployment criteria

  • The authors propose strategy and entropy thresholds for deployment, including PR > 0.7 with H < 0.5 bits for high-consequence applications.They also suggest that applications may set different tolerances for memorization according to their requirements.Source: D. Geometric Characterization of… · p. 17

Accuracy across strategy space

  • For α > 0.9, accuracy concentrates near pure reasoning; memorization-dominated regions show a moderate 0.45 < α < 0.6 plateau.Accuracy rises along the gradient from guessing toward reasoning-dominated regions.Source: D. Geometric Characterization of… · pp. 16–17

Entropy across strategy space

  • Minimum entropy concentrates near pure reasoning, but mixed-strategy regions contain multiple local maxima and higher-entropy pockets.The paper associates uncertainty in memorization-dominated regions with positional cues that provide conflicting evidence.Source: D. Geometric Characterization of… · p. 17

Correlation analysis

  • Reasoning probability correlates positively with accuracy (ρα,PR = 0.912) and negatively with entropy; guessing correlates negatively with accuracy (ρα,PG = −0.767).Memorization has a moderate positive correlation with entropy; its correlation with accuracy is not statistically significant (p ≈ 0.9).Source: D. Geometric Characterization of… · p. 18, Table I · p. 18

V. CONCLUSION

The conclusion presents PMM and ITC as a phenomenological framework for analyzing language-model reasoning beyond aggregate accuracy. It argues that benchmark scores can overstate reasoning because apparent success often involves memorization and pattern matching, and that strategy distributions can inform deployment criteria.

V. CONCLUSION

  • Non-zero memorization components persist even in reasoning-dominated regions, suggesting position-invariant logical deduction remains challenging for current models.The conclusion also reports stable attractors and conserved flow structures in strategy space.Source: §V · p. 18
  • Aggregate accuracy can overstate reasoning ability because it does not account for memorization, guessing, positional effects, or mixed strategies.The conclusion characterizes high-reasoning, low-entropy behavior as occupying a relatively small part of the strategy space.Source: §V · p. 18
  • The framework supports setting deployment thresholds on strategy distributions, with examples ranging from moderate memorization tolerance to stricter medical-use criteria.The paper gives PR > 0.7 and H < 0.5 bits as an example threshold for medical applications.Source: §V · p. 18

Related papers

Loading 2501.13833v1…