PaperlayerView exact revision on arXiv

Source-linked AI summary

ThinkPatterns-21k: A Systematic Study on the Impact of Thinking Patterns in LLMs

Pengcheng Wen, Jiaming Ji, Chi-Min Chan, Juntao Dai, Donghai Hong, Yaodong Yang, Sirui Han, Yike Guo

arXiv:2503.12918v2cs.CL

TL;DR

The paper addresses limited systematic evidence on how thinking patterns affect LLM performance across model sizes. It introduces ThinkPatterns-21k and evaluates models using several internal thought patterns; smaller and larger models show different preferences, while monologue is effective across sizes.

  • Problem

    Evidence remains limited on how different thinking patterns perform across model sizes, and training data often lacks thinking processes and systematic pattern annotations.

    Source: §1 · pp. 1–2

  • Method

    The paper builds ThinkPatterns-21k with 21k instruction-response pairs augmented with five thought patterns and evaluates them across models sized 3B–32B.

    Source: §1 · p. 2

  • Results

    Smaller models (<30B parameters) benefit from most structured patterns, while 32B models perform better with unstructured thinking; monologue is effective across most sizes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings indicate that thinking-pattern effectiveness varies with model size, and the released dataset, checkpoints, and training logs provide research resources.

    Source: §5 · p. 8

    The dataset is exclusively English, potentially limiting applicability to multilingual scenarios and reasoning patterns in other languages and cultures.

    Source: §5 · p. 8

Abstract

from arXiv · show

Large language models (LLMs) have demonstrated enhanced performance through the \textit{Thinking then Responding} paradigm, where models generate internal thoughts before final responses (aka, System 2 thinking). However, existing research lacks a systematic understanding of the mechanisms underlying how thinking patterns affect performance across model sizes. In this work, we conduct a comprehensive analysis of the impact of various thinking types on model performance and introduce ThinkPatterns-21k, a curated dataset comprising 21k instruction-response pairs (QA) collected from existing instruction-following datasets with five thinking types. For each pair, we augment it with five distinct internal thinking patterns: one unstructured thinking (monologue) and four structured variants (decomposition, self-ask, self-debate and self-critic), while maintaining the same instruction and response. Through extensive evaluation across different model sizes (3B-32B parameters), we have two key findings: (1) smaller models (<30B parameters) can benefit from most of structured thinking patterns, while larger models (32B) with structured thinking like decomposition would degrade performance and (2) unstructured monologue demonstrates broad effectiveness across different model sizes. Finally, we released all of our datasets, checkpoints, training logs of diverse thinking patterns to reproducibility, aiming to facilitate further research in this direction.

1 Introduction

The introduction identifies limited evidence on how thinking patterns perform across model sizes and presents a study and dataset designed to examine that relationship. Its reported findings distinguish smaller models, which benefit from most structured patterns, from 32B models, while monologue is effective across sizes.

1 Introduction

  • Existing training data often omits thinking processes and systematic pattern annotations, while the relative effectiveness of patterns across model sizes remains poorly understood.Prior synthetic-data work focused on unstructured patterns, despite structured prompting approaches such as self-ask, debate, and divide-and-conquer.Source: §1 · pp. 1–2
  • The study compares thinking patterns across model sizes using Arena Hard and Alpaca Eval 2, varying internal thought while preserving each instruction and response.The evaluated range is 3B–32B parameters.Source: §1 · p. 2
  • ThinkPatterns-21k contains 21k instruction-response pairs, each augmented with five thinking patterns: one unstructured and four structured variants.The patterns are monologue, decomposition, self-ask, self-debate, and self-critic.Source: §1 · p. 2
  • Smaller models (<30B parameters) benefit from most structured thinking patterns, while 32B models perform better with unstructured thinking; monologue is effective across most sizes.The study evaluates models from 3B to 32B parameters.Source: §1 · p. 2

2 Related Work

Related work covers prompt-based reasoning frameworks, training-data synthesis, and test-time scaling. It notes that structured thinking and its interaction with model size remain underexplored, motivating comparison of several internal thought types.

Prompt Engineering for Reasoning

Model Training for Enhanced Reasoning

  • Data-synthesis and training approaches include Monte Carlo Tree Search, generated demonstrations, and distillation, but have not thoroughly investigated structured thinking or its relationship to model size.The passage cites Qin et al. (2024) and Guan et al. (2025) for search, Chen et al. (2024) for demonstrations, and multiple works for distillation.Source: §2 · p. 2

2 Related Work

  • The paper’s five thought types contrast vanilla responses with monologue, decomposition, self-ask, self-debate, and self-critic reasoning pathways.The figure describes these as distinct reasoning pathways and problem-solving strategies.Source: Figure 2 · p. 3

Test Time Scaling for LLMs

  • Test-time scaling uses either a single extended chain of thought with reflection and backtracking or repeated sampling with selection, while this work focuses on structural variants of single extended chains.Repeated-sampling examples include Best-of-N and search-guided methods such as MCTS.Source: §2 · p. 3

3 Dataset

ThinkPatterns-21k pairs 21,000 instruction-response examples with five internal thinking patterns while keeping each instruction and response fixed. The section defines the patterns and describes their construction and differing token-length and turn-count profiles.

3 Dataset

  • The dataset contains 21,000 instruction-response pairs, each augmented with five thinking patterns while preserving its original instruction and response.The pairs were sampled from English data in Infinite Instruct, with task-category balance maintained.Source: §3 · p. 3, §3.2 · p. 5

3.1 Thinking patterns in set T

  • The five patterns are monologue, decomposition, self-ask, self-debate, and self-critic, spanning free-form thought, structured problem solving, questioning, debate, and revision.Decomposition breaks problems into components; self-ask generates and answers sub-questions; self-debate uses opposing perspectives within one model; self-critic drafts, evaluates, and refines an answer.Source: §3 · p. 3, §3.1 · pp. 3–5

3.2 Dataset Construction Process

  • GPT-4o generated the internal thoughts using pattern-specific prompts, and three annotators spot-checked samples for reasoning coherence and relevance to instructions.The checks were sampling-based and aimed to retain basic quality and instruction-response relationships.Source: §3.2 · p. 5
  • Self-ask averages 36.51 turns and 1,393.09 tokens, while self-debate averages 24.38 turns and the highest token count, 1,486.97.Monologue averages 1,300.50 tokens in one turn; decomposition averages 1,218.28 tokens across five fixed steps; self-critic averages 1,445.12 tokens across two steps.Source: §3.2 · pp. 5–6
  • The authors characterize self-debate as comprehensive exploration, self-ask as progressive reasoning, decomposition as structured analysis, monologue as direct response, and self-critic as reflective thinking.These are proposed complementary roles of the patterns, not performance comparisons.Source: §3.2 · p. 6

4 Experiment about Thinking Patterns

The experiments compare thinking patterns across Qwen2.5 models from 3B to 32B parameters on two open-ended benchmarks. Thinking patterns generally outperform the instruction-response baseline, but their relative effectiveness varies with model size and pattern.

4.1 Experiment Setup

  • The evaluation uses Qwen2.5 models from 3B to 32B on AlpacaEval 2 and Arena-Hard, with LLM-as-judge comparisons against benchmark baselines.The benchmarks contain 805 and 500 instructions, respectively.Source: §4.1 · p. 6

4.2 Main Results Analysis

  • On AlpacaEval2, all thinking patterns exceed the 14B baseline’s 29.53% length control win rate, with the best reaching 44.68%.The reported performance gap over baseline widens as model size increases.Source: §4.2 · p. 7
  • On Arena-Hard, decomposition is effective for smaller models but shows diminishing returns as model size increases, ranking lowest among 32B patterns at a 50.80% win rate.Source: §4.2 · p. 8
  • Effectiveness varies by size: 3B–7B models improve comparably with structured and unstructured patterns, while monologue has a clear advantage at 32B.The 14B model is described as a point where all patterns provide substantial gains.Source: §4.2 · p. 8
  • Self-critic, a two-stage generation-and-evaluation pattern, remains stable across sizes and is competitive at 32B, with a 71.40% Arena-Hard win rate.It reaches a 36.42% length control win rate with 7B on AlpacaEval 2 and occasionally achieves the best performance.Source: §4.2 · p. 8

5 Conclusion

The conclusion reports that structured thinking can benefit models below 30B parameters, while 32B models perform better with unstructured approaches; monologue is effective across most sizes. The authors also identify English-only dataset construction as a scope limitation.

5 Conclusion

  • The study finds that models below 30B can benefit from structured thinking, while 32B models perform better with unstructured approaches; monologue is effective across most sizes.The authors report releasing ThinkPatterns-21k, checkpoints, and training logs for further research.Source: §5 · p. 8
  • Because the dataset is exclusively English, it may not capture reasoning patterns specific to other languages and cultures and may limit multilingual applicability.The authors suggest extending the dataset to multiple languages and broader reasoning scenarios.Source: §5 · p. 8

A Appendix

The appendix provides annotation prompt templates and examples illustrating monologue, decomposition, self-ask, self-debate, and self-critic before final answers. These examples include an Africa safari-destination prompt showing iterative comparison and feedback alongside a broad destination overview.

A Appendix

  • The appendix includes annotation prompt templates for monologue, decomposition, self-ask, self-debate, and self-critic thinking.It also includes a figure captioned as showing the distribution of instruction categories in the dataset.Source: Figure 8 · p. 11, Table 5 · p. 12, Table 6 · p. 13
  • A monologue example reflects on African safari criteria and destinations before presenting a final list of options.The example discusses the Serengeti, Maasai Mara, Kruger, Okavango Delta, and Etosha, among other destinations.Source: A Appendix · p. 17, Table 10
  • A decomposition example organizes the Bohemian-minority question into scoping, component identification, subproblem analysis, connection mapping, integration, and validation.The components include discrimination, political marginalization, cultural suppression, religious intolerance, and economic deprivation.Source: A Appendix · p. 19, Table 11
  • A self-ask example uses an iterative dialogue to assess safari destinations by wildlife variety, tourist infrastructure, and sustainability concerns.It discusses the Serengeti, Maasai Mara, and Kruger, then considers visitor limits, eco-friendly practices, conservation, and responsible tourism.Source: A Appendix · pp. 21–22

Self-ask

  • The self-ask example weighs the Serengeti, Okavango Delta, and Kruger National Park by attractions and drawbacks, then recommends among them according to traveler priorities and sustainable tourism.It contrasts wildlife spectacles, remoteness and access, and infrastructure, and emphasizes tailoring choices to travelers’ needs.Source: A Appendix · pp. 24–25

Self-debate

  • The self-debate example critiques a draft safari list for lacking detail, visit timing, activities, conservation context, and a conclusion, then presents a revised destination overview.The revision describes ten destinations and highlights varied wildlife, landscapes, and safari experiences.Source: A Appendix · pp. 26–27

Self-critic

  • The appendix identifies Table 14 as a concrete example of self-critic thinking, but the supplied passage gives no table contents to summarize.Source: Table 14
Loading 2503.12918v2…