PaperlayerView exact revision on arXiv

Source-linked AI summary

The CoT Encyclopedia: Analyzing, Predicting, and Controlling how a Reasoning Model will Think

Seongyun Lee, Seungone Kim, Minju Seo, Yongrae Jo, Dongyoung Go, Hyeonbin Hwang, Jinho Park, Xiang Yue, Sean Welleck, Graham Neubig, Moontae Lee, Minjoon Seo

arXiv:2505.10185v1cs.CLcs.AI

TL;DR

Long CoTs can support reasoning, but the strategies models use and how they vary remain poorly understood. The CoT Encyclopedia extracts and organizes strategies from model outputs, then uses them to analyze and guide reasoning. Strategy guidance improved performance, while training format affected reasoning patterns more than data domain.

  • Problem

    How reasoning strategies vary across models and tasks, and whether they can be systematically controlled, remains underexplored.

    Source: §1 · p. 1

  • Method

    The CoT Encyclopedia extracts strategy criteria from model-generated CoTs, embeds and clusters them, creates contrastive rubrics, and classifies responses against those rubrics.

    Source: §1 · p. 2

  • Results

    Across five benchmarks, strategy guidance improved performance by 2.5–8.3% in three reasoning models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Training format affected reasoning patterns more than domain, and interpolating weights between format-trained models produced smooth strategy transitions without fine-tuning.

    Source: §1 · p. 3

    The reported comparisons cover six criteria across problem-solving benchmarks, with Cohen’s d up to 0.44, and safety benchmarks, with Cohen’s d up to 0.50.

    Source: D Limitations · pp. 20–21

Abstract

from arXiv · show

Long chain-of-thought (CoT) is an essential ingredient in effective usage of modern large language models, but our understanding of the reasoning strategies underlying these capabilities remains limited. While some prior works have attempted to categorize CoTs using predefined strategy types, such approaches are constrained by human intuition and fail to capture the full diversity of model behaviors. In this work, we introduce the CoT Encyclopedia, a bottom-up framework for analyzing and steering model reasoning. Our method automatically extracts diverse reasoning criteria from model-generated CoTs, embeds them into a semantic space, clusters them into representative categories, and derives contrastive rubrics to interpret reasoning behavior. Human evaluations show that this framework produces more interpretable and comprehensive analyses than existing methods. Moreover, we demonstrate that this understanding enables performance gains: we can predict which strategy a model is likely to use and guide it toward more effective alternatives. Finally, we provide practical insights, such as that training data format (e.g., free-form vs. multiple-choice) has a far greater impact on reasoning behavior than data domain, underscoring the importance of format-aware model design.

1 Introduction

The paper addresses limited understanding of the strategies used in long, multi-strategy CoTs and introduces a bottom-up framework to analyze and control them. It reports gains from strategy guidance and finds training format affects reasoning patterns more than domain.

1 Introduction

  • Prior analysis relied on fixed strategy categories, while clustering-based studies focused largely on short or medium CoTs rather than longer, multi-strategy traces.The paper frames its questions around strategy diversity, differences across models and tasks, and whether strategies can be systematically controlled to improve performance.Source: §1 · pp. 1–2
  • The CoT Encyclopedia extracts free-form strategy explanations, embeds and clusters them, then creates contrastive rubrics for classifying new CoTs.Human evaluations found the method reasonable in 92–97% of cases, versus 51% for prior top-down labels [5].Source: §1 · p. 2
  • Across five benchmarks, strategy guidance improved performance by 2.5–8.3% in three reasoning models.The approach predicts a model’s likely strategy, estimates correctness by strategy using Bayes’ rule, then prompts for the most promising strategy.Source: §1 · p. 2
  • Training format had a larger effect on reasoning patterns than data domain: Cohen’s d stayed below 0.2 for domain, while format effects reached 1.5.Multiple-choice-trained models tended toward concise, breadth-first-like responses; free-form-trained models favored longer, verification-heavy, depth-first-like chains.Source: §1 · pp. 2–3

2 Related Work

Prior work has developed methods both for scaling test-time reasoning and for analyzing reasoning strategies. These approaches include structured thought generation and analyses inspired by recurring patterns and cognitive behavior.

2.1 Scaling test-time compute through long chain-of-thought generation

  • Test-time reasoning methods include recursive CoT, adaptive multi-pass inference, self-consistency, Tree-of-Thoughts, and least-to-most prompting.These approaches add inference steps, refine answers iteratively, or structure reasoning paths for complex tasks.Source: §2.1 · p. 3

2.2 Analyzing reasoning strategies of models

  • Strategy-analysis studies examine recurring thought patterns, cognitive behavior, sudden insight, structured thinking, strategic examples, and targeted data generation.The cited work spans identifying patterns linked to accurate outcomes, relating behavior to psychology, and addressing specific reasoning failures.Source: §2.2 · p. 3

3 The COT ENCYCLOPEDIA

The CoT Encyclopedia builds a data-driven taxonomy from model-generated reasoning criteria and uses it to profile responses. Its criteria capture model differences and support adaptive analysis across tasks.

3.1 A framework for taxonomizing reasoning strategies

  • The five-stage framework extracts contrastive criteria from CoTs, embeds and clusters them, generates rubrics, then classifies responses into interpretable strategy profiles.It begins with LLM-assisted brainstorming; hierarchical clustering compresses semantically similar criteria, and rubric-based classifications are synthesized into natural-language reports.Source: §3.1 · pp. 3–4

3.2 COT ENCYCLOPEDIA enables sharper reasoning strategy classification

  • The Encyclopedia criteria revealed more substantial model differences than predefined cognitive behaviors, with many significant tests and effect sizes up to 0.4.Human annotators judged fine- and high-level criteria reasonable in most cases and found the resulting analyses more coherent than the baseline.Source: §3.2 · p. 5, Figure 3 · p. 5

3.3 COT ENCYCLOPEDIA enables adaptive analysis across diverse tasks

  • Different benchmarks yielded different criteria, including User Understanding for instruction following and ethical dimensions for safety tasks.Safety-specific examples include preventive versus risk-engaging precedence and censorship versus open discussion; the framework generates criteria suited to benchmark and model differences.Source: §3.3 · p. 5

4 Enhancing model helpfulness and safety via optimal reasoning control

The section examines whether optimal reasoning strategies can improve helpfulness and safety, and whether they can be predicted for individual questions. It reports benefits from strategy-guided prompting and a relationship between question similarity and strategy similarity.

4.1 Exploring optimal reasoning strategies for helpfulness and harmlessness

  • Optimal-strategy instructions improve accuracy and safety across benchmarks, including GPQA-Diamond accuracy rising from 72.7% to 79.1%.XSTest safety rises from 91.1% to 94.0%, and WildGuard safety from 89.1% to 92.9%.Source: Figure 4 · p. 6, §4.1 · p. 6

4.2 Similar inputs, Similar thoughts: How models approach related problems

  • More similar questions tend to have more similar reasoning strategies, while dissimilar questions are associated with greater strategy variation.This suggests that strategies for unseen questions may be predicted using strategies from similar encountered questions.Source: §4.2 · p. 6, Figure 5 · p. 7

4.3 Predicting question-specific optimal reasoning strategies

  • Classifiers predict question-specific optimal strategies from inputs and use them to guide responses, improving performance on initially incorrect questions across in-domain and cross-domain settings.The reported gains are 81.0%, 92.3%, and 80.7% on problem-solving tasks and 95.5% and 95.4% on safety tasks.Source: §4.3 · p. 7

5 Analyzing pathways to reasoning strategies: Data selection and interpolation

The section investigates how training data format and domain relate to reasoning strategies, then examines model merging as a way to interpolate between strategies. Its findings distinguish format effects from domain effects and show strategy transitions during merging.

5.1 Format matters more than domain in shaping reasoning strategies

  • Across three benchmarks and six reasoning criteria, training-format differences shift reasoning strategies more than domain differences.The comparison controls for content and format where appropriate, including synthetically converting NuminaMath to multiple-choice form.Source: §5.1 · pp. 7–8, Figure 6 · p. 8

5.2 Impact of training data format on model reasoning behavior

  • Multiple-choice-trained models produce concise, structured responses, whereas free-form-trained models are more verbose and often repeat filler words.Free-form models generate over 4.6 times as many “wait” tokens per answer (8.76 vs. 1.89); the styles are associated with breadth-first-like and depth-first-like reasoning, respectively.Source: §5.2 · pp. 8–9, Figure 7 · p. 8

5.3 Interpolating desired reasoning strategies through model merging

  • Linear weight merging interpolates reasoning strategies between multiple-choice- and free-form-trained models, with transition patterns varying by criterion and benchmark.MATH-500 shows especially dramatic transitions; its Bottom-Up analytical perspective falls from nearly 85% to approximately 15% as merging shifts from MC to FF.Source: §5.3 · p. 9, Figure 8 · p. 9

6 Conclusion

The conclusion presents the CoT Encyclopedia as a bottom-up framework for identifying and steering LongCoT reasoning strategies. It summarizes findings on strategy-guided performance, prediction, training format, and model merging.

6 Conclusion

  • The CoT Encyclopedia identifies strategies from model outputs through bottom-up clustering and supports steering toward strategies associated with better task performance.The conclusion also reports predicting strategies from input questions, stronger effects from training format than domain, and interpolation through weight merging without additional training.Source: §6 · pp. 9–10

A.1 COT ENCYCLOPEDIA details

  • The implementation extracts 4,057 fine-grained criteria, clusters them into six groups using silhouette scores, and selects medoid criteria as high-level criteria.It generates contrastive-pattern rubrics and reports CoT patterns using those rubrics.Source: §A.1 · p. 13

A.2 Statistical test details

  • The statistical analysis uses chi-squared tests for categorical pattern distributions and Cohen’s d to quantify distribution differences.The passage describes approximate effect-size reference points of 0.2, 0.5, and 0.8 for small, medium, and large effects.Source: §A.2 · p. 13

A.3 Human evaluation details

The human evaluation checks whether people judge the framework’s generated criteria and response analyses to be reasonable.

A.3 Human evaluation details

  • Four evaluators assess whether generated criteria and response analyses are plausible, sensible, relevant, and reasonable.The assessment covers both predefined and high-level criteria.Source: §A.3 · p. 13
  • Each evaluation consists of four binary questions about criteria plausibility and the quality of response analyses.Questions address detailed criteria, high-level summaries, and analyses expressed using either criterion level.Source: §A.3 · p. 13
  • The final human-evaluation metric is calculated as the ratio of yes/no responses for each question.Evaluators use Argilla as the evaluation platform.Source: §A.3 · p. 13

A.4 Benchmark evaluation details

The evaluation measures answer accuracy on problem-solving benchmarks and safe-response ratios on safety benchmarks.

A.4 Benchmark evaluation details

  • Accuracy is measured by parsing predicted values from generated LongCoT and verifying correctness with Math-Verify.MMLU-Redux, GPQA-Diamond, and MATH-500 are evaluated using generation even for multiple-choice datasets.Source: §A.4 · pp. 13–14
  • Safety is measured as the proportion of responses that LLaMA-Guard-3 8B classifies as safe.The evaluator receives each question and model response, then labels the response safe or unsafe.Source: §A.4 · p. 14

A.5 Training details

The appendix details GRPO-based RLVR training and implementation, then reports benchmark-specific criteria, representative-selection checks, and model-size comparisons.

A.5 Training details

  • GRPO trains the policy by comparing multiple outputs for each input, using relative-reward advantages and KL regularization without a separate value function.Training uses NuminaMath in free-form and synthetic five-choice formats, plus multiple-choice knowledge and commonsense datasets; multiple-choice standardizes verifiable rewards across domains.Source: §A.5 · p. 14

A.6 Computing resources

  • The 7B RLVR training run used eight H100 80GB GPUs for 576 GPU hours on 100,000 samples.Inference used sixteen A100 40GB GPUs for 384 GPU hours to process 3,698 samples.Source: §A.6 · p. 15

B.1 Fine-grained benchmark-specific criteria analysis

  • Arena-Hard yields criteria distinct from GPQA-Diamond, MMLU-Redux, and MATH-500, emphasizing user-intent interpretation rather than problem correctness.The analysis uses hierarchical clustering to derive six criteria from the original problem-solving benchmarks.Source: §B.1 · p. 15

B.2 Ablation study on representative embedding selection

  • Alternative cluster representatives largely preserve medoid-based results, with differences limited to the clarification-approach criterion.Silhouette-based and density-based selection yield “computation style” and “clarity on steps,” respectively; other aspects remain identical.Source: §B.2 · p. 15

B.3 Consistency of reasoning strategies across model sizes within the same family

  • Across Distill-R1 1.5B, 7B, and 32B, reasoning-strategy distributions are similar, with most pairwise Cohen’s d values below 0.1.The comparison covers six criteria across GPQA-Diamond, MMLU-Redux, and MATH-500.Source: §B.3 · p. 15

B.4 Extending reasoning strategy analysis to non-reasoning models

Analysis extends the CoT Encyclopedia to non-reasoning models and evaluates strategy patterns, prediction, and instruction effects across problem-solving and safety benchmarks. Results show benchmark-specific patterns and better outcomes when strategies are tailored to questions.

B.4 Extending reasoning strategy analysis to non-reasoning models

  • Broad, top_down, and immediate patterns consistently have higher probabilities of correct responses across problem-solving benchmarks.Safety-benchmark patterns with higher probabilities of safe responses include Moral Evaluation, Benevolent, Preventive, and Non-Directive.Source: Figure 9 · p. 16, Figure 10 · p. 16
  • Non-reasoning models yield criteria resembling those from reasoning models, while criteria differ between problem-solving and safety benchmarks and between the two Qwen models.The analysis spans GPQA-Diamond, MMLU-Redux, MATH-500, XSTest, and WildGuard; the authors attribute criteria more to the target benchmark than model output.Source: §B.4 · p. 16
  • Question-specific optimal strategy instructions produce the highest performance across five benchmarks, while unoptimal instructions sometimes underperform giving no instructions.Optimal instructions more than double safe-response ratios versus unoptimal instructions on XSTest and WildGuard.Source: Figure 11 · p. 17
  • Benevolent reasoning achieves safety scores of 84.0% in XSTest and 81.0% in WildGuard, while Malicious and Risk engaging patterns score lowest.Preventive, Moral, and Non-directive patterns also show relatively high safe-response ratios.Source: Figure 12 · p. 17

C Broader Impact

  • The authors identify controllability, taxonomy-informed curriculum design, and data-format diversity as implications for model development and deployment.They suggest these may support interpretable, debuggable, safety-aligned systems and interactive explanation or tutoring applications.Source: C Broader Impact · p. 17

B.4 Extending reasoning strategy analysis to non-reasoning models

  • Trained strategy classifiers outperform prompting-based prediction across benchmarks, with in-domain accuracy of 70.3–84.3% and cross-domain accuracy of 66.2–80.1%.Zero-shot and few-shot prompting perform only slightly above random chance (50%).Source: Figure 13 · p. 18

D Limitations

The limitations section qualifies the evidence base and presents robustness comparisons, prompt templates, and qualitative analyses of the framework.

D Limitations

  • Classification relies on GPT-4o evaluation, experiments cover three benchmarks and three model families, and guided gains depend on models following stylistic instructions.The authors call for broader task and model coverage, including scientific reasoning, code generation, multimodal tasks, smaller models, and multilingual models.Source: D Limitations · p. 18
  • Conventional cognitive-behavior categories show minimal differences between model pairs, whereas the Encyclopedia identifies numerous significant strategy differences.For problem-solving benchmarks, the Encyclopedia reports effect sizes up to Cohen’s d 0.44.Source: Table 1 · p. 18, D Limitations · p. 20
  • On problem-solving benchmarks, hierarchical clustering with PCA projections shows that medoid-, silhouette-, frequency-, and density-based representative selection yields consistent overall structure, with only minor variations in representative embedding positions.Across GPQA-Diamond, MMLU-Redux, and MATH-500, domain comparisons found only 4 of 18 significant differences and Cohen’s d mostly below 0.15; format differences reached 1.58 on MATH-500.Source: Figure 15 · p. 24, Table 10 · p. 26, Table 11 · p. 27

E Prompts

  • The prompts structure criterion extraction, contrastive rubric generation, response classification, multiple-choice assessment, and strategy steering.Steering instructions ask models to apply a selected strategy throughout problem solving.Source: Figure 19 · p. 30, Figure 20 · p. 31, Figure 21 · p. 32

F Qualitative Analyses

  • Qualitative analyses examine intermediate artifacts and compare final strategy analyses with predefined criteria, alongside responses from free-form- and multiple-choice-trained models.The analyses use GPQA-Diamond and Arena Hard examples, respectively.Source: F Qualitative Analyses · p. 34

F.1 Comparative Analysis of CoT Encyclopedia Outputs and Pre-defined Criteria

The physics comparison labels one response evidence-driven under CoT Encyclopedia criteria and backtracking under predefined criteria; the chemistry comparison likewise contrasts the frameworks’ labels for a multi-step problem.

F.1 Comparative Analysis of CoT Encyclopedia Outputs and Pre-defined Criteria

  • The physics table contrasts an evidence-driven CoT Encyclopedia assessment with a predefined-criteria assessment of backtracking.The sampled report emphasizes considering multiple solution routes and checking conclusions, while the predefined report emphasizes revisiting earlier reasoning.Source: Table 14 · p. 34, §F.1 · pp. 35–36
  • The physics sample reaches option B using both the Fresnel-drag expression and relativistic velocity addition.The report says the two approaches simplify to the same expression.Source: §F.1 · p. 35
  • The chemistry criteria also distinguish conceptual insight from formula application, cross-verification from single-source verification, and analytic derivation from plug-and-chug.Other listed contrasts include problem fragmentation versus holistic approach and qualitative versus quantitative interpretation.Source: §F.1 · p. 35
  • For the multi-step chemistry problem, CoT Encyclopedia criteria identify iterative clarification, while predefined criteria focus on backtracking.The listed fine-grained criteria include problem decomposition, data interpretation, functional-group identification, alternatives, verification, and iterative refinement.Source: Table 15 · p. 36, §F.1 · p. 37

F.2 Divergent reasoning strategies from Multiple-Choice vs. Free-Form Training Data

The section compares reasoning on shared tasks across models trained on multiple-choice and free-form data. The reported styles differ in structure, hypothesis handling, and verbosity.

F.2 Divergent reasoning strategies from Multiple-Choice vs. Free-Form Training Data

  • On the identical Arena-Hard problems, multiple-choice models decompose tasks, generate hypotheses, and restart, while free-form models pursue one hypothesis with continuous verification.The table presents this contrast as evidence that training format affects reasoning strategies.Source: Table 16 · p. 37
  • The toy-fairness setup assigns toys using a binary string and asks whether the value difference stays within the maximum toy value for every non-decreasing price sequence.Source: §F.2 · p. 38
  • In the toy-fairness example, the free-form response is lengthy and convoluted, while the multiple-choice analysis is more structured and explores multiple hypotheses.The task defines fairness over every non-decreasing sequence of toy values.Source: §F.2 · pp. 38–39
  • For technical troubleshooting, the multiple-choice model gives structured steps and actionable recommendations, whereas the free-form model repeatedly restates observations.Source: Table 17 · p. 40

Related papers

Loading 2505.10185v1…