PaperlayerView exact revision on arXiv

Source-linked AI summary

Cognitive Profiling of LRMs' Reasoning Traces Using Bloom's Taxonomy

Maria-Eleni Zoumpoulidi, Georgios Paraskevopoulos, Alexandros Potamianos

arXiv:2608.23205v1cs.AIcs.CL

TL;DR

Reasoning traces offer a way to study how Large Reasoning Models think at the level of individual steps, but these cognitive patterns remain underexplored. The paper annotates reasoning steps with Bloom’s Taxonomy and finds shared and task-dependent thinking profiles, with Bloom-based features correlating with correctness.

  • Problem

    The paper addresses limited understanding of the cognitive patterns expressed in Large Reasoning Models’ reasoning traces across models and tasks.

  • Method

    The authors generate, segment, and automatically annotate reasoning traces with six Bloom’s Taxonomy levels, then analyze them across models, datasets, and tasks.

  • Results

    LRMs show a shared remember–understand–apply–evaluate arc in mathematics, task-dependent profiles with back-loaded verification, and Bloom-based features that correlate with correctness.

  • Takeaways & Limitations

    Bloom-level annotations provide fine-grained insights into LRM thinking patterns and suggest potential for improving reasoning.

  • Takeaways & Limitations

    The study does not explicitly enforce identified structural patterns to improve reasoning and should be extended to a broader, more diverse set of tasks.

Abstract

from arXiv · show

Large Reasoning Models (LRMs) have revolutionized reasoning in LLMs, and the increasing public availability of reasoning traces creates valuable opportunities to study model behavior not only at the surface level but also at the granularity of individual reasoning steps. However, understanding the types of thinking employed during reasoning - which offers critical insights into models' reasoning patterns and enables actionable applications - remains underexplored. To address this gap, we introduce a framework for automatic annotation of reasoning steps through the lens of Bloom's Taxonomy, which classifies thinking into six cognitive levels, such as Remembering, Applying and Evaluating. Using this framework, we perform a large-scale analysis across models and datasets, revealing both similarities and differences in thinking patterns across models and tasks. Moreover, we demonstrate that thinking-type information derived from reasoning traces correlates with correctness, paving the way for improved reasoning. Our findings establish a fine-grained framework for analyzing thinking patterns in LRMs and provide actionable insights for enhancing reasoning quality.

1 Introduction

The paper introduces a human-validated framework that segments and annotates LRM reasoning traces using Bloom’s six cognitive levels. It reveals shared and task-dependent thinking patterns, cognitive compression, difficulty-related shifts, and links Bloom-level features to correctness.

  • The framework automatically segments reasoning traces and annotates each step with one of Bloom’s six cognitive levels, including Remembering, Applying, and Evaluating.The pipeline generates chain-of-thought traces, segments them into reasoning steps, and applies human-validated automated annotation.
  • The study evaluates whether internal and output traces differ, how profiles vary across mathematical and non-mathematical tasks, and whether Bloom-level features characterize correctness.These research questions motivate the analysis of trace type, problem difficulty, task diversity, and cognitive transitions in correct versus incorrect reasoning.
  • Its cognitive-function perspective distinguishes operations such as restating, paraphrasing, inference, and recalling facts more finely than process-oriented taxonomies.For example, restating maps to Remembering while paraphrasing or inferring maps to Understanding, and recalling a theorem remains Remembering rather than Analyzing.
  • Across models, datasets, and tasks, reasoning shows shared remember–understand–apply–evaluate arcs, model-specific traits, output-trace compression, difficulty-related shifts toward analysis, and task-dependent profiles with back-loaded verification.These findings establish distinct thinking profiles across models and tasks and identify patterns associated with reasoning difficulty.

2 Related Work

Related work spans the evolution from chain-of-thought prompting and structured reasoning variants to fine-grained analyses of LRM reasoning traces, alongside growing applications of Bloom’s taxonomy in NLP. These studies motivate examining the cognitive types expressed in individual reasoning steps.

  • 2 Related Work: Early LLM research elicited step-by-step reasoning through chain-of-thought prompting, followed by Tree of Thoughts and diverse-path consistency methods (Wei et al. (2022), Kojima et al. (2022), Yao et al., 2023, Wang et al., 2023).These approaches encouraged explicit solution generation, traversed tree-like reasoning states, or sampled multiple reasoning paths before selecting a consistent answer.
  • 2 Related Work: Recent LRM research moves beyond accuracy and surface statistics toward step-level analyses of reasoning traces, partly motivated by overthinking, where longer reasoning may not improve correctness (Kumar et al. (2026)).This shift treats individual reasoning steps as an object of study rather than evaluating only final outcomes.
  • 2 Related Work: Taxonomies and trace studies report recurring reasoning patterns across models while also revealing model differences, including in DeepSeek-R1 and code-generation reasoning (Marjanovic et al. (2026); Halim et al. (2025)).Marjanovic et al. identify a consistent heartbeat of reasoning steps across reasoning models, whereas Halim et al. find both human-like performance patterns and notable cross-model differences.
  • 2 Related Work: Bloom’s taxonomy has been used to design mathematical-reasoning prompts, map benchmark coverage, and structure LLM-generated explanations (Zoumpoulidi et al. (2025b); Huber and Niklaus (2025); Zoumpoulidi et al. (2025a)).Prior work also identifies imbalances across cognitive levels in widely used benchmarks.

3 Preliminaries: Bloom’s Taxonomy

Bloom’s Taxonomy organizes thinking into six levels of increasing complexity, providing the framework’s cognitive hierarchy. The revised taxonomy includes processes such as remembering, understanding, applying, and creating.

  • 3 Preliminaries: Bloom’s Taxonomy: Bloom’s Taxonomy defines a six-level hierarchy of thinking with increasing cognitive complexity.The taxonomy is presented in the revised form associated with Anderson and Krathwohl (2001).
  • 3 Preliminaries: Bloom’s Taxonomy: Remembering involves retrieving, recognizing, and recalling relevant knowledge from long-term memory.
  • 3 Preliminaries: Bloom’s Taxonomy: Understanding involves constructing meaning by interpreting, exemplifying, classifying, summarizing, inferring, comparing, and explaining information.
  • 3 Preliminaries: Bloom’s Taxonomy: Applying means carrying out or using a procedure through executing or implementing it.
  • 3 Preliminaries: Bloom’s Taxonomy: Creating involves combining elements into a coherent or functional whole or reorganizing them into a new pattern or structure.The associated processes include generating, planning, and producing.

4 Our Approach

The approach analyzes LRM reasoning patterns through Bloom’s Taxonomy using a pipeline for CoT generation, step-level segmentation, annotation, and subsequent analysis. Experiments span three math datasets and seven reasoning models.

  • Framework: The pipeline generates zero-shot CoT solutions, segments reasoning traces into cognitive steps, labels each step with Bloom’s Taxonomy, and analyzes the resulting patterns.It stores both internal reasoning traces and final responses for each problem.
  • Framework: Llama-3.3-70B-Instruct automatically segments traces, assigns Bloom levels, and provides justifications, allowing thinking processes spanning multiple sentences to be annotated as single steps.Segmentation quality was additionally assessed through qualitative inspection of 50 samples, while evaluation focused on labeling quality.
  • Experimental Setup: The study queries seven models—Qwen3-30B-A3B-Thinking-2507, Qwen3-4B-Thinking-2507, DeepSeek-R1, three R1 distillations, and Phi-4-reasoning—using provider-recommended default settings.The settings are intended to avoid introducing artifacts or distortions.

5 Experiments

Experiments reveal distinct cognitive profiles across models, traces, outputs, datasets, and tasks, including an emergent Remember–Understand–Apply–Evaluate progression. Applying generally dominates, while model families, reasoning format, and task difficulty shift the balance of cognitive levels.

  • Thinking-type distributions: Applying dominates across models at 26.9%–44.2%, whereas Creating remains below 1.0%; Understanding and Analyzing occupy stable middle bands.Understanding ranges from 17.7%–24.0%, and Analyzing from 10.1%–14.3%.
  • Model differences: Model profiles differ: Qwen3 has the highest Evaluating shares at 25.6% and 23.1%, while Phi-4-Reasoning has the highest Remembering at 28.4% and lowest Analyzing at 10.1%.DeepSeek-R1 distillation shifts toward Applying and away from Evaluating, but the smaller R1-Distill-Qwen-1.5B more closely resembles its teacher than larger distilled variants.
  • Temporal dynamics: Across almost all models, temporal trajectories follow an emergent Remember–Understand–Apply–Evaluate arc, with Applying plateauing around 0.4–0.6 in the trace middle.Remembering and Understanding generally decay early, Applying dominates the middle, and Evaluating rises toward the end; Phi-4-Reasoning and some distilled variants diverge from these patterns.
  • Reasoning formats: Figure 4 shows internal reasoning traces distribute cognition broadly, whereas final outputs compress it into Remembering and Applying, with Evaluating falling from 18.8% to 5.4%.Outputs contain 27.8% Remembering and 44.9% Applying, while internal traces contain 33.9% Applying, 21.0% Understanding, 18.8% Evaluating, and 12.7% Analyzing.
  • Datasets and tasks: Across GSM8K, GSM-hard, and MATH500, the overall cognitive arc persists while Applying’s plateau declines from approximately 0.47 to 0.40 and then decays most steeply.The experiments also compare pooled mathematical reasoning with Formal Fallacies and Hyperbaton, spanning mathematical, logical, and linguistic tasks.

6 Conclusions

The paper introduces an automatic framework for annotating LRM reasoning steps with Bloom’s Taxonomy. Across models, datasets, and tasks, it reveals shared and task-dependent cognitive patterns, including difficulty-related shifts and consistent back-loaded verification.

  • The framework automatically annotates LRM reasoning steps using Bloom’s Taxonomy and supports analysis across models and datasets.
  • In the Bloom-based correctness model, positive coefficients are associated with correct solutions and negative coefficients with incorrect solutions.
  • Mathematical reasoning follows a shared remember–understand–apply–evaluate arc, while models retain distinct cognitive traits.
  • Internal reasoning traces are cognitively broader than output CoTs, which compress reasoning into recall and execution.
  • As mathematical difficulty increases, cognitive effort shifts from execution toward analysis.
  • Cognitive profiles depend on the task, although back-loaded verification remains consistent across tasks.

Limitations

The method provides informative and actionable insights, especially into the relationship between correctness and reasoning depth, but it does not yet enforce structural patterns to improve reasoning performance or cover a broad, diverse task set.

  • Limitations: The method does not explicitly enforce or shape reasoning-depth patterns to improve reasoning performance.The authors identify this as a direction for future work.
  • Limitations: The study should be extended to a broader and more diverse set of tasks for a more comprehensive analysis.

A Prompts

This section specifies the prompts for chain-of-thought generation and for automatically segmenting reasoning traces into cognitive steps assigned Bloom’s Taxonomy levels with justifications. It also identifies tables containing few-shot annotation examples for mathematical datasets and selected BBH tasks.

  • The framework uses prompts for CoT generation on mathematical datasets and BBH tasks, with the mathematical prompts in Table 4 and BBH prompts in Table 7.
  • A separate prompt automatically segments reasoning traces into discrete cognitive steps and assigns each step a Bloom’s Taxonomy level with a justification, as specified in Table 5.
  • Few-shot annotation examples for mathematical datasets appear in Table 6, while BBH examples cover formal fallacies in Table 8 and hyperbaton in Table 9.

B A closer look at the outlier Phi-4-Reasoning

The analysis investigates Phi-4-Reasoning’s divergent thinking profile by testing the role of its system prompt and examining traces qualitatively. It identifies terminal repetition of intended answers as a striking pattern and discusses plausible training-related explanations.

  • System-prompt analysis: The analysis reruns experiments without the developer-provided system prompt to test whether it enforces Phi-4-Reasoning’s divergent thinking behavior.
  • Qualitative analysis: Qualitative inspection finds that Phi-4-Reasoning frequently ends traces by repeatedly restating its intended final answer, including “The answer is 540.”This terminal-repetition pattern appears under both system-prompt settings and is consistent with the absence of outcome-based reinforcement learning, which would encourage cleaner termination.
  • Training-related explanations: Plausible explanations include training with a different teacher model, o3-mini, SFT-only optimization without outcome-based RL, and a heavily curated, knowledge-dense corpus (Abdin et al., 2025).

C Example Outputs across Diverse Tasks

The section presents example outputs from the annotation framework across diverse tasks to demonstrate task-specific differences in output characteristics.

  • C Example Outputs across Diverse Tasks: The annotation framework is illustrated through example outputs spanning diverse tasks.These examples are intended to demonstrate differences in characteristics across tasks.
  • C Example Outputs across Diverse Tasks: Tables 11, 12, and 13 provide the cross-task examples used for this comparison.
  • C Example Outputs across Diverse Tasks: The section focuses on how annotated outputs differ across tasks rather than reporting aggregate performance results.

D Human Annotation Details

Human annotation used independently completed labels from an author and a non-author annotator, both applying the same interface prompt after project-specific orientation. The prompt operationalized Bloom’s Taxonomy through segmented reasoning examples covering mathematical and BBH tasks.

  • Prompt Materials: The annotation materials included prompt templates and few-shot examples for mathematical datasets and BBH tasks.BBH materials included one-shot examples for Formal Fallacies and Hyperbaton, while mathematical materials included few-shot examples and annotated samples.
  • Annotation Process: The author and non-author annotator independently annotated reasoning traces using the same prompt through a dedicated user interface.The non-author annotator received project objectives, task requirements, and general background before annotation.
  • Mathematical Annotation Examples: Mathematical examples segment reasoning into steps labeled with Bloom levels and provide rationales for Remember, Understand, Analyze, Apply, and Evaluate.The examples illustrate arithmetic computation, problem decomposition, condition restatement, and correctness checking.
  • BBH Annotation Examples: A BBH Formal Fallacies example labels counterexample construction as CREATE, illustrating how the scheme captures higher-level reasoning beyond arithmetic procedures.The example constructs a counterexample showing that a conclusion makes a broader claim than its premise supports.
Loading 2608.23205v1…