Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction

Chenguang Wang; Ming Li; Xinyue Zeng; Zhuochun Li; Hong Jiao; Tianyi Zhou; Dawei Zhou

Source-linked AI summary

Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction

Chenguang Wang, Ming Li, Xinyue Zeng, Zhuochun Li, Hong Jiao, Tianyi Zhou, Dawei Zhou

arXiv:2606.28186v3Jun 26, 2026cs.CLcs.AIcs.CYcs.LG

TL;DR

Human item-difficulty prediction lacks scalable, interpretable evidence about the cognitive processes underlying difficulty. Epi2Diff structures LRM reasoning traces into cognitive episodes and combines their dynamics with item semantics, outperforming broad classification and regression baselines across four benchmarks.

Problem
Existing item-difficulty methods primarily use item-level representations, while cognitive demands underlying difficulty remain difficult to observe and model at scale.
Method
Epi2Diff converts reasoning traces into cognitively grounded episode sequences, derives process features, and combines them with semantic item representations for difficulty prediction.
Results
Across four benchmarks, Epi2Diff outperforms broad classification and regression baselines; harder items elicit more implementation-centered and iterative reasoning rather than merely longer traces.
Takeaways & Limitations
Cognitively grounded abstractions of reasoning traces provide scalable, interpretable proxy evidence for modeling human item difficulty.
Takeaways & Limitations
Reasoning traces are proxy evidence rather than direct observations of human cognition, and predictions may be unreliable outside evaluated domains.

Abstract

from arXiv · show

Predicting human item difficulty is central to educational assessment, where reliable estimates support fairness and effective test construction. Existing methods often depend on costly human calibration or item-level textual representations, providing limited evidence about the cognitive processes that make items difficult. We argue that difficulty should be viewed not only as a property of item text, but also as an observable consequence of the problem-solving burden an item induces. Large Reasoning Models (LRMs) offer scalable process evidence through reasoning traces, but such evidence must be structured to support interpretable modeling. To this end, we introduce Epi2Diff (Episode to Difficulty), a framework that maps LRM reasoning traces into cognitively grounded episode sequences. These episodes group trace segments into functional problem-solving states, enabling difficulty to be modeled through reasoning scale, effort allocation, and state transitions. Epi2Diff extracts compact episode-dynamic features and combines them with semantic item representations for human difficulty prediction. Experiments on four real-world human difficulty datasets show that Epi2Diff consistently outperforms strong baselines, including fine-tuned small language models, LLM in-context learning, and supervised LLM adaptation. On SAT-derived classification benchmarks, Epi2Diff achieves an 8.1% average relative gain over supervised LLM fine-tuning baselines. Further analyses show that harder items induce more effortful, iterative, and implementation-centered episode dynamics, rather than merely longer responses. These results demonstrate that cognitive episodes in LRM reasoning traces provide a predictive and interpretable process representation for human item difficulty, offering a new lens for educational measurement with reasoning models.

1 Introduction

Epi2Diff reframes human item difficulty as an observable consequence of item-induced problem-solving burden, using structured cognitive episodes from model-generated reasoning traces alongside semantic item representations. Across four real-world assessment benchmarks, it improves prediction while revealing that harder items elicit more effortful, implementation-centered, and iterative reasoning dynamics rather than merely longer traces.

Motivation: Existing item-difficulty methods primarily use interpretable feature-based approaches or end-to-end fine-tuning, creating a need to balance accuracy and interpretability.Feature-based methods depend on representation quality, while fine-tuning methods can be less interpretable.
Problem: Human item difficulty is commonly modeled as a single item-level outcome from text, metadata, annotations, embeddings, or other representations rather than problem-solving burden.The cognitive demands underlying difficulty have historically been difficult to observe and model at scale.
Approach: Epi2Diff converts model-generated reasoning traces into sentence-level cognitive episode sequences and combines episode-process features with semantic item representations for prediction.Its process representation captures reasoning length, episode distribution, and transitions, providing a structured and interpretable view of reasoning dynamics.
Results: Across four real-world assessment benchmarks, Epi2Diff consistently outperforms strong baselines in classification and regression.Further analyses show that harder items elicit more effortful, implementation-centered, and iterative reasoning dynamics rather than merely longer traces.

2 Method

Epi2Diff converts reasoning traces into an eight-episode sequence representation and combines process dynamics with item semantics for difficulty prediction. It uses automatic sentence-level episode labeling, feature groups capturing reasoning scale, effort allocation, and transitions, and aggregation across prompting profiles.

Episode taxonomy: The method uses an eight-way episode taxonomy: Read, Analyze, Plan, Implement, Explore, Verify, Monitor, and Answer.The Answer episode captures final answer commitment separately from preceding verification and monitoring.
Process representation: The episode-induced process representation combines length, episode-distribution, and transition features derived from labeled episode sequences.Length features measure total, thinking-part, and answer-part token counts; distribution features preserve absolute and relative episode allocation.
Process representation: Transition counts form an episode transition matrix that captures local sequential organization unavailable from token totals alone.The matrix is flattened into a 64-dimensional vector and concatenated with length, episode-count, and episode-ratio features.
Joint representation: The joint representation concatenates a Sentence-BERT item embedding with the episode-induced process representation to integrate item content and elicited reasoning burden.The semantic component encodes what the item is about, while the process component captures how it is solved.
Profile aggregation: Reasoning processes are elicited under direct, weak, medium, and strong prompting conditions and averaged across profiles.These prompting profiles are not interpreted as validated simulations of human students or ability levels.

3 Experiments and Results

Across four educational benchmarks and classification/regression metrics, Epi2Diff achieves the best overall performance, while ablations show that profile-diverse trace aggregation and combined semantic-process representations drive its strongest SAT Math results.

Main results: Epi2Diff achieves the best performance across all four test sets and every reported metric.The results indicate that structured representations from model-generated reasoning traces provide informative signals for predicting item difficulty.
Main results: Reasoning-process features yield especially clear gains on SAT Math while also improving SAT Reading & Writing, Cambridge, and USMLE performance.The findings support combining process features with semantic item representations across multi-step, text-based, classification, and regression settings.
Experimental setup: Four benchmarks span SAT Math, SAT Reading & Writing, Cambridge, and USMLE, covering classification and regression evaluation.Accuracy and macro-F1 are reported for classification, while RMSE and R2 are reported for regression.
Aggregation ablation: 0.730 accuracy and 0.728 macro-F1 are achieved by Both (Role), exceeding 0.707/0.708 for source-level Both aggregation.Profile-based aggregation across both trace sources outperforms the stronger single-source or source-level alternatives under the full representation.
Aggregation ablation: 0.728 macro-F1 versus 0.714 for direct replication shows that profiled aggregation remains stronger under a matched rollout budget.This comparison suggests the gain reflects structured profile diversity rather than simply more repeated generations.
Representation ablation: The full Sem. + FFea. representation consistently performs best, while Sem. Only trails it by 7.4–12.1 macro-F1 points.Length, episode-distribution, and transition feature families each contribute, and their full combination is strongest across aggregation settings.

4 Interpreting Item Difficulty through Episode-induced Features

In SAT Math, harder items exhibit a structured, implementation-centered reasoning signature rather than merely longer responses. Episode composition and transition dynamics provide complementary predictive evidence alongside semantic and length features.

Key Finding 1: Difficulty Is Associated with a Structured Process Signature: Harder items involve more intermediate work, implementation-centered effort, and stronger Analyze→Implement transitions.Implement→Implement and Implement→Analyze transitions also increase with fitted difficulty, indicating extended and iterative local execution.
Key Finding 1: Difficulty Is Associated with a Structured Process Signature: Difficulty reflects reasoning scale, effort allocation, and process flow—not simply longer final responses.Trace length and think proportion show stronger positive signals than final answer length, while Read, Plan, and Verify ratios are negative.
Key Finding 2: Episode Dynamics Provide Predictive Process Evidence: Episode composition and transition dynamics contribute complementary process evidence for difficulty prediction.Semantic features describe item content, length captures reasoning scale, episode composition captures effort allocation, and transitions capture local reasoning flow.
Key Finding 2: Episode Dynamics Provide Predictive Process Evidence: 0.136 in permutation importance is achieved by Epi.+Tran. in Both (Role), exceeding Sem. at 0.084 and Len. at 0.042.Under SHAP importance, Epi.+Tran. receives 1.024, compared with 0.526 for Len., while semantic features remain an important anchor.
Key Finding 2: Episode Dynamics Provide Predictive Process Evidence: The full representation performs best, and removing any process feature family weakens the model.Importance and ablation results indicate that Epi2Diff combines semantic, length, episode-composition, and transition evidence.

5 Conclusion

Epi2Diff combines semantic item representations with episode-level features from model-generated reasoning traces to capture complementary signals of problem-solving burden. Across four benchmarks, it outperforms broad classification and regression baselines, while SAT Math difficulty relates to implementation-centered reasoning and stronger analysis-toimplementation transitions rather than trace length alone.

5 Conclusion: Epi2Diff combines semantic item representations with episode-level features extracted from model-generated reasoning traces.The framework models reasoning scale, effort allocation, and local transitions alongside item content.
5 Conclusion: Across four benchmarks, Epi2Diff outperforms a broad range of classification and regression baselines.
5 Conclusion: On SAT Math, higher difficulty is associated with more implementation-centered reasoning and stronger analysis-toimplementation transitions, rather than simply longer traces.

Ethics and Broader Impact … A.1 Item Difficulty Prediction

The paper uses existing assessment data without collecting new human-subject or identifying information, while positioning interpretable difficulty prediction as decision support requiring expert oversight. Its related work situates the approach among costly response-data calibration, variable expert judgment, and increasingly strong transformer-based representations.

Ethics and Broader Impact: The study uses publicly available or previously released assessment data and collects no new human-subject data, private student records, or personally identifying information.
Ethics and Broader Impact: Automated difficulty predictions should not be the sole basis for test construction, item approval, placement, or licensure decisions.Deployment should include expert review, target-population validation, subgroup fairness analysis, and distribution-shift monitoring.
Ethics and Broader Impact: Predictions may reflect biases in the source datasets, requiring safeguards when they are deployed.
A.1 Item Difficulty Prediction: Response-data calibration under CTT or IRT remains central for calibrated assessment but is costly for newly authored items because field testing is typically required before operationalization.
A.1 Item Difficulty Prediction: Expert judgment offers a lower-cost alternative, but prior work reports limited or variable agreement between expert ratings and empirical measures.
A.1 Item Difficulty Prediction: Transformer-based representations have become strong baselines for item difficulty prediction, outperforming traditional feature-based, TF-IDF, and Word2Vec baselines in reported studies.BERT features have also been used in a multi-task generalized linear model to jump-start item-parameter estimates for adaptive language testing.

A.2 LLM-Based Methods for Item Difficulty Prediction · A.3 Reasoning Traces as Process Signals · B Additional Details

Prior work uses LLM outputs as auxiliary signals for human item difficulty, but these signals rarely provide structured, interpretable evidence of solving processes. Reasoning-trace research instead abstracts intermediate computation into functional episodes and emphasizes effort allocation over response length alone.

A.2 LLM-Based Methods for Item Difficulty Prediction: LLM-based methods use answers, answer variability, simulated responses, rationales, or direct estimates from item content to predict difficulty.These approaches include zero-shot answers and variability for multiple-choice difficulty and response-time prediction, plus simulated medical-candidate responses.
A.2 LLM-Based Methods for Item Difficulty Prediction: LLM outputs provide useful auxiliary cues, but do not uniformly outperform strong specialized baselines or offer structured, interpretable evidence aligned with human psychometric difficulty.Their extracted signals are generally treated as predictive features rather than representations of solving processes.
A.2 LLM-Based Methods for Item Difficulty Prediction: Broader LLM difficulty studies examine easy-to-hard generalization, cross-difficulty transfer, and benchmarks scaled by required reasoning length.This literature shows that difficulty affects model behavior and evaluation but does not focus on empirically calibrated human item difficulty.
A.3 Reasoning Traces as Process Signals: Reasoning traces expose intermediate computation and may reveal self-reflection, verification, and strategy adaptation during inference.Chain-of-thought prompting made intermediate steps explicit, while newer reasoning-oriented models produce longer, human-readable traces.
A.3 Reasoning Traces as Process Signals: Schoenfeld’s Episode Theory decomposes reasoning-model solutions into interpretable functional episodes.This operationalizes structured abstractions for analyzing model-generated reasoning traces.
A.3 Reasoning Traces as Process Signals: Reasoning efficiency and trace-quality studies indicate that effective reasoning depends on effort allocation across episodes rather than response length alone.The passage supports treating reasoning structure and effort distribution as distinct from simple trace length.

B.1 Evaluation Metrics … C.1 Transition Feature Design

The paper evaluates classification and regression with standard task-specific metrics, fixed dataset splits, and feature-group attribution analyses. Its transition ablation supports raw adjacent bigram counts as the default because alternatives add redundancy and sparsity without consistent gains.

B.1 Evaluation Metrics: Classification performance is evaluated with accuracy and macro-F1, whose class-wise scores are averaged uniformly across classes.Macro-F1 uses per-class F1 values derived from precision and recall.
B.1 Evaluation Metrics: Regression performance is reported with RMSE and R2, where lower RMSE indicates smaller error and higher R2 indicates better fit.
B.2 Dataset Splits: Cambridge, SAT Math, and SAT Reading & Writing use stratified 80/20 train–test splits with seed 42, while USMLE uses its official split.Table 3 reports partition sizes for the three newly split datasets; USMLE is excluded from that table.
B.3 Feature-Group Importance: Feature-group contributions are measured on held-out data using group-wise permutation importance and SHAP importance.Permutation importance jointly shuffles a feature group, preserving within-group dependencies, while SHAP aggregates mean absolute contributions.
B.3 Feature-Group Importance: The importance analyses cover semantic, length, episode-composition, transition, and combined episode-plus-transition feature groups.For SAT Math, permutation importance uses macro-F1 as the evaluation score, and larger values indicate greater performance degradation after shuffling.
C Supplemental Ablation Study: The supplemental transition ablation compares count, global-ratio, unique-count, raw-trigram, and deduplicated-trigram parameterizations across reasoning-trace sources and aggregation settings.Base combines semantic representation with length and episode-composition features; Dedup. collapses consecutive repeated episode labels before trigram computation.
C.1 Transition Feature Design: Raw adjacent bigram counts remain the default because alternative transition parameterizations do not produce more consistent gains in the SAT Math setting.Concatenated alternatives can remain competitive but do not consistently surpass the default design, so mixed representations are not adopted.
C.1 Transition Feature Design: With eight episode labels, trigram features expand the transition space from 8^2 = 64 bigram features to 8^3 = 512 trigram features, increasing sparsity and noise risk.Higher-order or reformulated transitions partially re-encode local-order information, increasing dimensionality and redundancy when concatenated.

C.2 Additional Feature-Importance and Directional-Effect Analyses

Additional analyses show that episode-composition and transition features provide substantial predictive signal beyond semantics and length, while directional effects associate greater difficulty with intermediate reasoning and execution-centered episode dynamics.

Feature-group importance: Under Both (Role), Epi.+Tran. reaches 0.136 in permutation importance, exceeding Sem. at 0.084 and Len. at 0.042.Episode-composition and transition features contribute nonzero importance across settings and can match or exceed semantic features.
Feature-group importance: Under Both (Role), Epi.+Tran. has mean absolute SHAP importance of 1.024, compared with Len. at 0.526.This supports using episode organization alongside item semantics and reasoning length.
Directional effects: Total trace length and think length have the largest positive effects, while final answer length has a smaller positive effect.Higher predicted difficulty is associated mainly with additional intermediate reasoning rather than merely longer final responses.
Directional effects: Implement has the largest positive count effect, its ratio is positive, and Verify, Read, and Plan ratios are negative.The pattern indicates that difficulty shifts the reasoning budget toward concrete solution execution rather than uniformly longer traces.
Transition effects: The strongest positive transition effect is Analyze→Implement, followed by Implement→Implement and Implement→Analyze.Additional positive effects for Plan→Implement, Analyze→Analyze, and Verify→Implement indicate movement into execution, sustained execution, and local implementation refinement.

C.3 Additional Ablation Results

Additional ablations compare Epi2Diff with alternative feature configurations, including LLM-extracted item-text features, combined semantic and extracted features, and reasoning-trace embeddings. Overall, Epi2Diff remains competitive against these alternatives.

Feature Configuration Comparisons: Table 7 compares Epi2Diff with three alternatives: LLM-extracted item-text features, combined item semantics and extracted features, and reasoning-trace embeddings.The alternatives use either item-text features, combined item and extracted representations, or trace-derived embeddings.
Overall Result: Epi2Diff remains competitive across the additional feature-configuration comparisons.The passage reports this as the overall result of the additional experiments.

C.4 Matched Rollout-Count Control for Role-Based Aggregation · C.5 Representative Item-level Episode Profiles

Matched rollout controls show that profile-based aggregation, rather than additional direct-prompt samples alone, improves difficulty prediction. Representative traces also exhibit systematic episode-dynamic differences as item difficulty increases, including greater reasoning length and implementation-oriented allocation.

C.4 Matched Rollout-Count Control for Role-Based Aggregation: C.4 Matched Rollout-Count Control for Role-Based Aggregation: Direct Replication matches Profiled Aggregation’s four reasoning-model rollouts, but only Profiled Aggregation uses proficiency-conditioned solver profiles.The control separates the effect of rollout quantity from structured profile conditioning.
C.4 Matched Rollout-Count Control for Role-Based Aggregation: C.4 Matched Rollout-Count Control for Role-Based Aggregation: For QwQ, Direct Replication increases macro-F1 from 0.681 to 0.683, whereas Profiled Aggregation reaches 0.697.The small replication gain does not account for the larger profile-aggregation improvement.
C.4 Matched Rollout-Count Control for Role-Based Aggregation: C.4 Matched Rollout-Count Control for Role-Based Aggregation: The results attribute the gain to structured variation from proficiency-conditioned profiles rather than repeated direct-prompt sampling alone.Profile-based aggregation provides a more robust characterization of item difficulty.
C.4 Matched Rollout-Count Control for Role-Based Aggregation: C.4 Matched Rollout-Count Control for Role-Based Aggregation: In the combined trace-source setting, Profiled Aggregation remains best, achieving 0.730 ACC and 0.728 macro-F1.Direct Replication improves over Single Direct but remains below Profiled Aggregation.
C.5 Representative Item-level Episode Profiles: C.5 Representative Item-level Episode Profiles: As SAT Math difficulty increases, total reasoning length and Implement share increase, while Verify and Read shares decrease.Analyze share also increases with difficulty.
C.5 Representative Item-level Episode Profiles: C.5 Representative Item-level Episode Profiles: The Hard example allocates substantial early reasoning budget to Analyze, whereas the Easy example remains more fragmented across monitoring, reading, planning, and execution/checking episodes.The comparison uses collapsed episode prefixes derived from Qwen3-32B traces.

D Additional Analysis Visualizations

The appendix presents role-wise visualizations of episode-induced process representations for SAT Math, using Qwen3-32B reasoning traces across four solver profiles. Each visualization combines length summaries, episode-ratio deltas, and transition-delta heatmaps across difficulty groups.

Role-wise visualizations: The visualizations cover direct, weak, medium, and strong solver roles on SAT Math using Qwen3-32B generated reasoning traces.Each role receives a corresponding visualization of the episode-induced process representation.
Visualization components: Each figure combines length summaries, episode-ratio deltas, and transition-delta heatmaps between Easy, Medium, and Hard items.These three views provide complementary summaries of episode dynamics across difficulty groups.