PaperlayerView exact revision on arXiv

Source-linked AI summary

Parallel-R1: Towards Parallel Thinking via Reinforcement Learning

Tong Zheng, Hongming Zhang, Wenhao Yu, Xiaoyang Wang, Runpeng Dai, Rui Liu, Huiwen Bao, Chengsong Huang, Heng Huang, Dong Yu

arXiv:2509.07980v2cs.CL

TL;DR

Existing SFT-based approaches to parallel thinking rely on pre-generated trajectories, while RL faces a cold-start challenge because parallel-thinking behavior is not already present in models. Parallel-R1 uses a progressive curriculum to teach the format on easier tasks before RL on harder problems; it reports consistent accuracy gains and a mid-training exploration scaffold that raises final performance ceilings.

  • Problem

    SFT-based parallel-thinking methods depend on costly pre-generated data and can constrain models to imitation, while high-quality parallel-thinking data for complex problems is rare.

    Source: §3.3 · p. 5

  • Method

    Parallel-R1 uses simple-task cold-start data to teach parallel-thinking format, then applies RL to generalize the behavior to harder mathematical problems.

    Source: §3.1 · p. 4, §3.4 · p. 6

  • Results

    Parallel-R1 achieves consistent accuracy improvements on challenging mathematical benchmarks, and its temporary mid-training exploration scaffold unlocks higher final performance ceilings, reaching 25.6% peak accuracy on AIME25.

    Source: §1 · p. 3, §5 · p. 13

  • Takeaways & Limitations

    The model’s use of parallel thinking shifts from early computational exploration to late multi-perspective verification during training.

    Source: §5 · p. 13

Abstract

from arXiv · show

Parallel thinking has emerged as a novel approach for enhancing the reasoning capabilities of large language models (LLMs) by exploring multiple reasoning paths concurrently. However, activating such capabilities through training remains challenging, as existing methods predominantly rely on supervised fine-tuning (SFT) over synthetic data, which encourages teacher-forced imitation rather than exploration and generalization. Different from them, we propose \textbf{Parallel-R1}, the first reinforcement learning (RL) framework that enables parallel thinking behaviors for complex real-world reasoning tasks. Our framework employs a progressive curriculum that explicitly addresses the cold-start problem in training parallel thinking with RL. We first use SFT on prompt-generated trajectories from easier tasks to instill the parallel thinking ability, then transition to RL to explore and generalize this skill on harder problems. Experiments on various math benchmarks, including MATH, AMC23, and AIME, show that Parallel-R1 successfully instills parallel thinking, leading to 8.4% accuracy improvements over the sequential thinking model trained directly on challenging tasks with RL. Further analysis reveals a clear shift in the model's thinking behavior: at an early stage, it uses parallel thinking as an exploration strategy, while in a later stage, it uses the same capability for multi-perspective verification. Most significantly, we validate parallel thinking as a \textbf{mid-training exploration scaffold}, where this temporary exploratory phase unlocks a higher performance ceiling after RL, yielding a 42.9% improvement over the baseline on AIME25. Our model, data, and code will be open-source at https://github.com/zhengkid/Parallel-R1.

1 Introduction

Parallel thinking can support divergent exploration before synthesis, but current training approaches often imitate pre-generated trajectories and generalize poorly. Parallel-R1 uses a progressive curriculum to teach the behavior on easier problems before RL training on harder tasks, and analyzes how its role changes during training.

1 Introduction

  • SFT-based approaches imitate pre-generated reasoning trajectories, depend on complex and costly data pipelines, and can limit generalization of parallel-thinking strategies.The introduction notes that RL also faces cold-start and reward-design challenges, including balancing final correctness against unnecessary or forgotten parallel thinking.Source: §1 · pp. 1–2
  • The model’s parallel thinking shifts from early computational exploration to later multi-perspective verification, and a temporary mid-training exploration phase can unlock higher final performance ceilings.The framework reports a peak accuracy of 25.6% on AIME25.Source: §1 · p. 3
  • Parallel-R1 first uses SFT on easier problems to teach parallel-thinking formats, then uses RL on harder tasks to explore and generalize the behavior.The initial trajectories are generated through simple prompting, and the curriculum is designed to address RL’s cold-start problem.Source: Figure 1 · p. 2, §1 · p. 3
  • The framework investigates alternating accuracy and parallel-thinking rewards, reporting a better balance of performance and consistent behavior than either reward type alone.The contributions also report comparisons across causal and structured model variants.Source: §1 · p. 3

2 Related Work

Prior parallel-thinking methods include fixed-schedule branching and verifier-guided search, while newer training approaches have focused on lossless conversion or toy tasks. The paper positions RL as a route to adaptive parallel thinking and notes that standard RLVR methods do not themselves instill that capability.

2.1 Parallel Thinking

  • Prior parallel-thinking approaches use fixed branching schedules or hand-crafted verifier-guided search, while newer SFT or RL work has focused on lossless conversion or toy tasks such as Countdown (Pan et al., 2025; Yang et al., 2025b).The paper argues that RL can support adaptive behaviors beyond lossless transformation.Source: §2.1 · pp. 3–4

2.2 Improving Reasoning via RLVR

  • Although RLVR has demonstrated effectiveness across domains, most approaches retain sequential reasoning and do not instill parallel-thinking capabilities in LLMs.The passage characterizes this as an unresolved challenge for standard RLVR methods.Source: §2.2 · p. 4

3 Learning Parallel Thinking via Reinforcement Learning

The section motivates RL as an alternative to SFT pipelines that depend on costly pre-generated data and can constrain models to imitation. Parallel-R1 uses easy tasks for cold-start data, then applies RL to harder problems, with an optional architectural setting that isolates parallel paths.

3. Learning Parallel Thinking via Reinforcement Learning

  • SFT-based training depends on the quality of pre-generated data, requiring costly pipelines for challenging problems and potentially limiting acquisition of generalizable reasoning skills.The paper introduces RL as an alternative to these constraints.Source: §3.1 · p. 4
  • Parallel-R1 generates cold-start data on simple tasks to teach parallel-thinking format, then uses RL to generalize the behavior to harder mathematical problems.The framework aims to avoid complex data pipelines for the final challenging problems.Source: §3.1 · p. 4
  • The framework studies RL both without architectural changes and with changes to attention masks and position IDs that prevent cross-attention between parallel paths.The latter setting enforces structural independence between reasoning paths.Source: §3.1 · p. 4

3.2 Formulation of Parallel Thinking Behaviors

Parallel thinking is formulated as an exploration phase followed by a summary phase, which can recur during reasoning. Control tags mark the parallel block, separate paths, and aggregated summary within an autoregressive inference process.

3.2 Formulation of Parallel Thinking Behaviors

  • At a critical reasoning step, the model pauses its main chain and launches N independent trajectories in parallel to explore possible solutions.The formulation labels this phase Exploration.Source: §3.2 · p. 5
  • After exploration, the model aggregates outcomes, distills insights, resolves conflicts, and resumes the main reasoning chain with a summary.The formulation labels this phase Summary and allows the two phases to repeat as needed.Source: §3.2 · p. 5
  • At inference, <Parallel> triggers concurrent <Path> threads, whose outputs are combined in a <Summary> block before generation continues.The inference workflow remains autoregressive in the main reasoning process.Source: §3.2 · p. 5

3.3 The Simple and Scalable Data Pipeline for Parallel Thinking

Simple prompting can produce high-quality parallel-thinking data for easier math but not challenging problems, motivating a scalable easy-task data pipeline that teaches format before RL.

3.3 The Simple and Scalable Data Pipeline for Parallel Thinking

  • 83.6% of GSM8K problems yielded valid parallel-thinking traces, versus 0.0% of DAPO problems for a powerful model.The contrast motivates generating cold-start data on easier problems rather than relying on prompted traces for challenging tasks.Source: §3.3 · p. 5
  • Existing pipelines for challenging problems are computationally intensive and limited in scalability, while simple prompting works well on GSM8K but struggles on DAPO.Table 1 compares data quality generated under identical prompts and sampling settings; it supports the stated difficulty contrast but provides no cell values here.Source: §3.3 · p. 5, Table 1 · p. 6
  • The cold-start dataset teaches the parallel-thinking format rather than how to solve target tasks, so RL can strengthen and elicit the behavior.The structured variant also uses a Parallel Thinking Format Check to filter examples for strict format adherence.Source: §3.3 · pp. 5–6

3.4 Eliciting Parallel Thinking via Reinforcement Learning in Causal Models

Parallel-R1 uses a progressive curriculum to move from format learning on easy math to RL on general math, where parallel-thinking ability is trained to generalize. Its causal-model variant uses GRPO and a multi-turn process of generation, parallel exploration, and summarization.

3.4 Eliciting Parallel Thinking via Reinforcement Learning in Causal Models

  • The causal-model framework uses a scalable easy-task data pipeline as a cold start, then applies RL to generalize parallel thinking to harder math.The approach does not modify the model architecture.Source: §3.4 · p. 6

3.4.1 Reinforcement Learning Algorithms

  • GRPO samples candidate responses and uses their rewards in the policy objective, while rollouts alternate between generation, parallel paths, and summarization.The model can repeat the parallel-exploration and summarization phases as reasoning proceeds.Source: §3.4.1 · p. 6

3.4.2 The Training Recipe and Reward Modeling

  • Training proceeds from SFT on Parallel-GSM8K to easy-math RL rewarding both correctness and parallel use, then general-math RL using accuracy reward alone.The cold-start set uses approximately 7k GSM8K training samples; general-math training uses DAPO and produces Parallel-Seen variants.Source: §3.4.2 · pp. 6–7

3.5 Eliciting Parallel Thinking via Reinforcement Learning in Structure Models

Parallel-Seen does not isolate reasoning paths, leaving potential cross-path information leakage and gradient interference. Parallel-Unseen adds attention and position-encoding constraints for path isolation, alongside a redesigned training recipe.

3.5 Eliciting Parallel Thinking via Reinforcement Learning in Structure Models

  • Parallel-Seen lacks explicit path isolation, so hidden representations can leak across paths and gradients can interfere during training.Parallel-Unseen introduces structural inductive biases to enforce isolation.Source: §3.5 · p. 7

3.5.1 Structured Attention Mechanism

  • Path-window masking limits each path to its own tokens and shared context, while disjoint position indices separate paths and the summary block can integrate their insights.Figure 2 depicts the distinct visibility regions of paths and the summary block.Source: §3.5.1 · p. 7, Figure 2 · p. 7

3.5.2 The Training Recipe and Reward Modeling

  • Because easy-to-hard mask generalization proved poor, Parallel-Unseen removes stage-one RL and compares accuracy-only training with alternating reward schedules.The reported attribution is to poor generalization of attention masks from easy to hard math.Source: §3.5.2 · p. 8
  • In the alternating schedule, 80% of steps use accuracy reward and 20% use tiered rewards: +1.2 for correct parallel outputs, +1.0 for correct nonparallel outputs, and -1.0 otherwise.The schedule reintroduces a calibrated incentive for parallel use without letting it dominate training.Source: §3.5.2 · p. 8

4 Experiments

Experiments compare Parallel-R1 with GRPO baselines, test training and reward-design choices, and track how parallel thinking changes during RL. The analyses also examine a two-stage curriculum that uses parallel reasoning for exploration before optimizing accuracy.

4.1 Experimental Setups

  • The experiments use Qwen-3-4B-Base and evaluate AIME’24, AIME’25, AMC’23, and MATH, with mean@16 for the first three and mean@1 for MATH.They also report pass@16 for AIME’24, AIME’25, and AMC’23; training includes cold-start SFT and later RL stages.Source: §4.1 · p. 8

4.2 Main Results

  • 48.9 average score: Parallel-R1-Seen consistently outperforms the baselines, while SFT alone scores 31.7 and direct GRPO scores 45.1.Adding RL on easier data to the baseline yields only a marginal average improvement, 45.3 versus 45.1.Source: Table 2 · p. 9, §4.2 · p. 9

4.3.1 Ablation on Training Approach

  • Removing stage-one GSM8K RL lowers causal-variant performance by 2.3% on average, whereas adding it lowers the structured variant by 8.6% on average.The authors attribute the structured variant’s decline to poor transfer of attention masks from easier to harder math, while the causal variant benefits from bootstrapping adaptive parallel use.Source: §4.3.1 · p. 10

4.3.2 Ablation on Parallel Thinking Prompt

  • Removing the parallel-thinking prompt degrades performance by up to 1.8% on average.The authors interpret detailed training instructions as helping the model understand the reasoning process rather than merely memorize output patterns.Source: §4.3.2 · p. 10

4.3.3 Ablation Studies on Reward Modeling: How to Effectively Stimulate Parallel Thinking

  • Alternating accuracy and parallel rewards achieves a parallel ratio of 63.0 and an AIME25 score of 19.0; accuracy-only yields a 13.6 ratio, while parallel-only reaches 80.3 but harms benchmark performance.The accuracy-only configuration scores 69.7 on AMC; parallel-only scores 59.4 on AMC23 and 81.7 on MATH.Source: §4.3.3 · pp. 10–11

4.4 Evolution of Parallel Thinking Behavior During RL Training

  • As RL progresses, the <Parallel> block moves later in the reasoning sequence, shifting from early computational exploration toward late multi-perspective verification.The authors interpret this as a more conservative strategy under final-answer-dominated rewards: the model first derives a high-confidence solution, then uses parallel reasoning to verify it.Source: §4.4 · p. 11

4.5 Extra Bonus: Parallel Thinking as a Mid-Training Exploration Strategy for RL Training

  • In a two-stage curriculum, alternating accuracy and parallel rewards promote exploration for 200 steps before accuracy-only RL; AIME25 accuracy peaks at 25.6% as parallel use declines.The authors interpret the initial forced exploration as a scaffold for reaching a more effective policy region, from which the model learns a final policy.Source: Figure 4 · p. 12, §4.5 · pp. 12–13

5 Conclusion

Parallel-R1 improves accuracy on challenging mathematical benchmarks and its learning dynamics shift from exploratory parallel thinking toward verification. The analysis also validates temporary parallel exploration as a mid-training scaffold for higher final performance ceilings.

5 Conclusion

  • The framework achieves consistent accuracy improvements on several challenging mathematical reasoning benchmarks compared with strong baselines.Its progressive curriculum separates learning of format, behavior, and core reasoning into distinct stages.Source: §5 · p. 13
  • The model shifts from early computational exploration to late-stage multi-perspective verification as it learns a risk-averse strategy.Source: §5 · p. 13
  • A temporary forced-exploration phase serves as a mid-training scaffold that can unlock higher final performance ceilings after RL training.Source: §5 · p. 13

A Prompts

The prompts direct models to reason step by step, introduce independent parallel paths when useful, and summarize their combined conclusions. A separate format check validates tag structure.

A Prompts

  • The response must end with a line beginning “Final Answer:” followed by the final answer.Source: A Prompts · p. 17
  • The prompt instructs the model to solve step by step and insert a parallel block when a step benefits from multiple perspectives or independent reasoning.Source: A Prompts · p. 17
  • Each parallel block requires at least two independent paths, followed by a concise summary of their conclusions.Paths use <Path> tags; the summary follows </Parallel> in <Summary> tags.Source: A Prompts · p. 17
  • The prompt says to repeat the process adaptively as needed throughout the reasoning chain.Source: A Prompts · p. 17
  • The format-check algorithm returns whether the trace is well formed by checking matching opening and closing tags and rejecting unmatched tags.Source: A Prompts · p. 18

B Case Studies

The case studies illustrate parallel exploration of distinct algebraic methods and late-stage verification through parallel thinking.

B Case Studies

  • Figure 5 shows two distinct algebraic methods explored in parallel for a number theory problem, both converging on the correct answer.Source: Figure 5 · p. 19
  • Figure 6 presents a qualitative case study of late-stage verification through parallel thinking.Source: Figure 6 · p. 20

Related papers

Loading 2509.07980v2…