PaperlayerView exact revision on arXiv

Source-linked AI summary

Parallel Thinking for Large Language Model with Path Diversity

Jiaru Qian, Yihong Dong, Xue Jiang, Chun-Wah Hsu, Zhi Jin, Ge Li

arXiv:2610.29501v1cs.AI

TL;DR

Parallel-thinking RL can produce redundant reasoning paths, limiting the distinct hypotheses it is meant to maintain. DPT adds prefix-based diversity rewards and a two-stage training curriculum, and the paper reports that it outperforms prior parallel-thinking baselines across mathematical, coding, and knowledge reasoning benchmarks. The authors identify path diversity as a key component of effective parallel thinking.

  • Problem

    Outcome-reward RL optimizes answer correctness without directly rewarding diversity, while measuring path diversity simply and effectively remains unresolved.

    Source: §1 · p. 2

  • Method

    DPT uses a token-level metric to penalize similarity in early reasoning prefixes and combines it with a two-stage curriculum balancing correctness and exploration.

    Source: §1 · p. 2, §6 · p. 8

  • Results

    DPT consistently outperforms the best-performing baseline on mathematical reasoning, coding, and knowledge reasoning benchmarks.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The results establish path diversity as a key component and design principle for effective parallel thinking in LLMs.

    Source: §1 · p. 2, §6 · p. 8

    The study covers text-based reasoning tasks with verifiable answers, uses a small set of fixed path counts, and leaves multimodal, interactive, and adaptive path allocation for future work.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Parallel thinking has emerged as a promising paradigm for enhancing the reasoning capabilities of Large Language Models (LLMs) by exploring multiple reasoning paths simultaneously. However, we identify that parallel thinking via Reinforcement Learning (RL) typically suffers from the mode collapse of reasoning paths, where parallel thinking trajectories degenerate into similar ones. This homogeneity undermines the fundamental motivation of parallel thinking, that is, maintaining multiple hypotheses concurrently to prevent premature loss into a suboptimal solution. In this paper, we propose Diverse Parallel Thinking (DPT), a simple yet effective RL approach for parallel thinking designed to encourage path diversity. DPT rewards the model to develop distinct reasoning strategies by penalizing the prefix similarity of parallel thinking paths, thereby enhancing a more comprehensive exploration of the reasoning space. We conduct various experiments and analyses across multiple reasoning benchmarks to demonstrate the effectiveness of DPT. Results show that DPT successfully outperforms previous parallel thinking methods, achieving state-of-the-art performance over baselines. We validate path diversity as a key element to improve parallel thinking. Our code is available at https://github.com/JiaruQian/DPT.

1 Introduction

DPT addresses reasoning-path collapse in parallel-thinking RL by explicitly rewarding diversity, with experiments identifying path diversity as a key design principle. Its prefix-based metric and training framework target distinct early reasoning strategies while maintaining task correctness.

1 Introduction

  • Outcome-reward RL can leave parallel paths highly redundant because it rewards correct answers without directly rewarding path diversity.Repeatedly covering the same approach limits exploration; the goal is to discourage redundancy while preserving correctness.Source: §1 · p. 1
  • DPT makes path diversity an intrinsic RL objective, aiming to maintain concurrent hypotheses and prevent premature convergence on suboptimal solutions.The paper presents this as a response to the difficulty of measuring diversity and balancing it against correctness.Source: §1 · p. 2
  • DPT adds a token-level diversity metric focused on early path prefixes to steer parallel reasoning toward distinct strategies.The framework is designed for integration into existing RL pipelines without architectural modifications.Source: §1 · p. 2
  • DPT consistently outperforms state-of-the-art parallel-thinking baselines across reasoning benchmarks and LLMs.The paper identifies path diversity as the primary driver of these gains.Source: §1 · p. 2
  • The paper theoretically attributes path collapse to outcome-supervised RL exploitation, which drives paths toward identical trajectories.Its contributions also describe prefix-based diversity metrics as requiring no additional computational overhead.Source: §1 · p. 2

2 Motivation

The motivation section combines empirical and theoretical evidence that outcome-supervised RL can collapse parallel reasoning paths. It argues that diversity should be explicitly encouraged at the prefix level, where early strategic decisions are made.

2 Motivation

  • The two example paths use the same vertex-alignment insight despite phrasing differences, illustrating semantic overlap rather than distinct strategies.The paper characterizes this as high-reward paraphrasing instead of exploration of disjoint solution spaces.Source: §2 · p. 2
  • 61.81% of sampled path pairs received judge similarity scores ≥0.9, indicating substantial overlap in reproduced Parallel-R1 rollouts.The assessment uses GPT-5-mini to judge approach, structure, technique, style, and strategy; its reliability depends on the evaluation protocol.Source: §2 · p. 2, Figure 2 · p. 3
  • Outcome-supervised RL rewards the aggregated answer and is agnostic to path diversity, while a shared policy reinforces the safest reliable reasoning pattern across path slots.This exploitation dynamic concentrates the path distribution on one pattern and can collapse all paths together.Source: §2 · p. 3
  • The analysis identifies explicit prefix-level redundancy penalties as a design principle because early reasoning decisions shape path diversity.Diversity should be an optimization constraint, not merely an emergent byproduct.Source: §2 · p. 3

3 Related Work

Prior work spans rollout diversity methods and approaches to parallel reasoning, but the latter still face limits in generating genuinely distinct paths. DPT combines format learning with diversity-aware RL to address that gap.

3 Related Work

4 Diversity Parallel Thinking

DPT adds a diversity-aware reward to parallel-thinking RL to discourage redundant paths while maintaining reasoning accuracy. The section defines parallel thinking and describes its cold-start curriculum and prefix-based diversity training.

4.1 Parallel Thinking Formulation

  • Parallel thinking generates multiple reasoning paths within one trajectory, which can be integrated before the main reasoning chain resumes.Paths may be isolated by structural attention masking or may attend to earlier paths, depending on the attention mode.Source: §4.1 · p. 4

4.2 Training Algorithm

  • DPT uses a two-stage curriculum: supervised fine-tuning on simpler problems precedes reinforcement learning on challenging tasks.The curriculum addresses the cold-start problem in parallel thinking.Source: §4.2 · p. 5

4.2.1 Cold Start Initialization

  • Because pretrained models have not encountered the parallel-thinking control tags, supervised fine-tuning teaches the format before RL exploration.The examples are generated from the GSM8K training set and filtered for valid tag structure.Source: §4.2.1 · p. 5

4 Diversity Parallel Thinking

  • DPT combines correctness and diversity rewards to penalize mode collapse among the parallel paths in a trajectory.The diversity reward is applied alongside the correctness reward as part of the training objective.Source: §4 · p. 4, §4.2.2 · p. 5

4.2.2 Reinforcement Learning

  • DPT rewards differing prefixes among paths, using the first n content tokens as an inexpensive proxy for diversity rather than a sufficient measure of semantic strategy.Whole-path similarity can confuse differences in length or word choice with strategic diversity; the RL curriculum applies the full reward for 2 of every 10 steps and correctness reward alone for the other 8.Source: §4.2.2 · pp. 5–6

5 Experiments

DPT outperforms prior parallel-thinking baselines across mathematical and additional reasoning benchmarks while producing less-overlapping paths. Further analyses examine path counts, test-time sampling, and adaptive use of parallel thinking.

5 Experiments

  • Experiments train with GRPO on DAPO and evaluate mathematical, coding, and knowledge-reasoning tasks against nine baselines.The listed baselines include Direct, GRPO, LTE, TTRL, EVOL-RL, and Parallel-R1 variants.Source: §5 · p. 6

5.1 Experimental Results

  • DPT (Unseen) achieves the best overall performance, outperforming Parallel-R1 (Unseen) on every reported mathematical benchmark and metric, including a 6.2-point AIME24 pass@16 gain.DPT (Unseen) also outperforms DPT (Seen) on average.Source: §5.1 · p. 6, Table 1 · p. 7
  • DPT outperforms GRPO and Parallel-R1 on HumanEval, ARC-c, and MMLU-Pro, extending its reported advantages to coding, science and commonsense, and knowledge-intensive reasoning.The gains are most pronounced on code generation and knowledge-intensive reasoning, while DPT also leads on ARC-c.Source: §5.1 · p. 7, Table 2 · p. 7
  • DPT produces less-overlapping paths across four mathematical benchmarks, with higher whole-path Jaccard distance and favorable judge assessments across five diversity dimensions.The reward acts on prefixes; the reported whole-path distances indicate that early divergence accompanies broader differences in continuations.Source: §5.1 · p. 7, Figure 4 · p. 7

5.2 Further Analysis

  • With K = 2, pass@1 is best on all three tested benchmarks, while K = 4 yields higher pass@16 on AIME25 and AMC23 and remains competitive on AIME24.The normalized pairwise diversity reward extends to more paths without changing its definition.Source: §5.2 · p. 8, Table 4 · p. 8
  • DPT maintains the strongest performance at every sampling budget and exceeds Parallel-R1 by 6.1 percentage points at AIME25 pass@256.Parallel-R1’s initial advantage over GRPO diminishes at larger budgets.Source: §5.2 · p. 8, Table 5 · p. 8
  • During RL training, parallel thinking becomes more selective from near-universal use after cold start, while accuracy continues improving.DPT encourages distinct paths when branching, while training adapts how often the model invokes parallel thinking.Source: §5.2 · p. 8, Figure 5 · p. 8

6 Conclusion

The conclusion presents path mode collapse as a problem for parallel thinking and proposes DPT as an RL approach to encourage diversity while maintaining reasoning accuracy.

6 Conclusion

  • DPT combines a token-level metric for early reasoning divergence with a two-stage curriculum balancing correctness and exploration, and outperforms the best-performing baseline across mathematics, coding, and knowledge benchmarks.The authors identify path diversity as a key component of effective parallel thinking.Source: §6 · p. 8

Limitations

The study is limited to text-based reasoning with verifiable answers and fixed path counts within each configuration.

Limitations

  • Evaluation covers text-based reasoning with verifiable answers; path count is fixed per configuration, leaving multimodal, interactive, and adaptive path allocation for future work.The authors propose choosing path counts according to problem difficulty and inference budget as a future direction.Source: Limitations · p. 9

Ethical Considerations

The paper reports no substantial additional ethical, societal, or environmental risks beyond those generally associated with LLM training and inference, alongside implementation, benchmark, and baseline details.

Ethical Considerations

  • The authors identify no substantial additional ethical, societal, or environmental risks beyond those generally associated with LLM training and inference.This statement is limited to the evaluated research setting.Source: Ethical Considerations · p. 9

A Implementation Details

  • Main experiments use 8 A100 GPUs with Qwen-3-4B-Base, cold-start SFT, and GRPO on DAPO; the main configuration uses K = 2 and n = 1.Diversity and format rewards are applied only during the last two steps of each ten-step RL cycle.Source: A Implementation Details · p. 11

B Evaluation Benchmarks

  • Evaluation includes AIME24, AIME25, AMC23, and MATH, with pass@1 and pass@16 reported for AIME and AMC benchmarks.The benchmark set also includes HumanEval, ARC-c, and MMLU-Pro for coding and knowledge-reasoning evaluation.Source: B Evaluation Benchmarks · p. 11

D Ablation on Diversity Metric Design

Across four mathematical reasoning benchmarks, Prefix-1 exact gives the strongest average accuracy, though longer-prefix settings lead on individual benchmarks. DPT’s advantage also grows with larger test-time sample budgets, consistent with reduced path redundancy.

D Ablation on Diversity Metric Design

  • Prefix-4 similarity leads on AIME25 pass@16, Prefix-4 exact on AMC23, and Prefix-2 exact on MATH, while Prefix-1 exact gives the strongest AIME24 results.Prefix-1 exact’s improvements across the remaining benchmarks yield the best overall average, rather than optimizing for one benchmark.Source: D Ablation on Diversity Metric Design · pp. 12–13
  • 51.27% is the best overall average accuracy, achieved by Prefix-1 exact, which improves every reported accuracy metric over Parallel-R1 without diversity reward.The ablation selects n = 1 with exact matching for the main configuration based on its strongest aggregate performance across the suite.Source: D Ablation on Diversity Metric Design · pp. 12–13, Table 6 · p. 13
  • The reported AIME24 results improve across the remaining benchmarks, yielding the best overall average of 51.27%; Prefix-2 exact has the highest parallel-thinking usage on AIME25 and AIME24.Effective path exploration, rather than branching frequency alone, is important for overall reasoning performance.Source: D Ablation on Diversity Metric Design · p. 13, Table 7 · p. 13

E Interpreting Test-Time Scaling

  • From k = 64 to k = 256, DPT’s pass@k rises from 49.2 to 57.0, versus 47.9 to 50.9 for Parallel-R1.DPT remains ahead of both baselines at every larger sampling budget, a pattern the authors connect to reduced redundancy and broader solution coverage.Source: E Interpreting Test-Time Scaling · p. 13

F Inference Cost and Response Length

DPT combines stronger reasoning results with lower measured inference costs and shorter responses in the evaluated parallel-thinking setting. Its training proxy encourages early path divergence while retaining correctness as the primary task signal.

F Inference Cost and Response Length

  • 4.9% fewer tokens per problem, 4.5% lower latency, and 4.2% fewer FLOPs accompany DPT’s stronger reasoning performance relative to Parallel-R1.Tokens per solved problem are also 5.5% lower; the cost analysis uses checkpoint-175 validation trajectories.Source: F Inference Cost and Response Length · p. 13, Table 8 · p. 14
  • DPT produces the shortest median responses and reaches the 3,000-token cap least often, consistent with a leaner inference profile.The authors report that DPT maintains the strongest reasoning results in the evaluated parallel setting while generating shorter responses and using fewer tokens.Source: F Inference Cost and Response Length · pp. 13–14, Table 9 · p. 14

G Prefix Divergence as a Training Proxy

  • DPT rewards differing early tokens as a local symmetry-breaking signal, giving parallel paths room to develop different continuations while leaving most tokens governed by task reward.Early tokens condition subsequent autoregressive generation, and correctness remains the primary training signal.Source: G Prefix Divergence as a Training Proxy · p. 14
  • The default n = 1 encourages different initial decisions, while larger prefixes extend the exploration window and similarity scoring gives graded credit for differing positions.Diversity and format rewards apply in the last two steps of each ten-step RL cycle; the other eight steps use correctness alone.Source: G Prefix Divergence as a Training Proxy · p. 14

H Lexical Overlap and Semantic Evaluation

The analyses report greater whole-path lexical distance for DPT and judge-based evidence that its paths differ in reasoning organization as well as surface form. A case study contrasts DPT’s algebraic and geometric approaches with Parallel-R1’s shared strategy and an earlier calculation error.

H Lexical Overlap and Semantic Evaluation

  • Whole-path Jaccard distance is a diagnostic rather than the training reward, and DPT shows substantially larger distances across all four benchmarks.The controlled comparison reports a higher average score for the prefix formulation than for whole-path Jaccard and Parallel-R1 without diversity regularization.Source: H Lexical Overlap and Semantic… · p. 14
  • GPT-5-mini evaluations favor DPT across solution approach, reasoning structure, mathematical technique, expression style, and strategy diversity.The judgments complement lexical results with evidence of differences in reasoning organization, not just surface form.Source: H Lexical Overlap and Semantic… · p. 15

I Case Studies

  • In the complex-number case, DPT’s two paths use algebraic and geometric viewpoints, whereas Parallel-R1’s paths share a strategy and do not correct an earlier calculation error.The figures present the DPT and Parallel-R1 case studies, respectively.Source: I Case Studies · p. 15, Figure 7 · p. 15, Figure 8 · p. 15

Related papers

Loading 2610.29501v1…