PaperlayerView exact revision on arXiv

Source-linked AI summary

Native Parallel Reasoner: Reasoning in Parallelism via Self-Distilled Reinforcement Learning

Tong Wu, Yang Liu, Jun Bai, Zixia Jia, Shuyi Zhang, Ziyong Lin, Yanting Wang, Song-Chun Zhu, Zilong Zheng

arXiv:2512.07461v3cs.CL

TL;DR

NPR addresses limitations in native parallel reasoning for agentic language models with progressive self-distilled training and parallel-aware reinforcement learning. Across eight reasoning benchmarks, it reports consistent performance gains, up to 24.5%, alongside genuine parallel execution and faster inference.

  • Problem

    Native parallel reasoning for agentic language models remains limited by incompatible systems, inefficient hand-crafted branching, and reliance on teacher-generated supervision.

    Source: §1 · p. 1

  • Method

    NPR progressively induces parallel formats, stabilizes them with self-distilled parallel training, then uses PAPO to optimize branching policies within parallel execution graphs.

    Source: §1 · p. 2

  • Results

    Across eight reasoning benchmarks, NPR reports consistent performance improvements, up to 24.5%, and 100% genuine parallel execution in evaluated test cases.

    Source: §1 · p. 2

  • Takeaways & Limitations

    NPR produces genuinely parallel reasoning policies rather than simulated or scripted ones, without relying on external teacher models.

    Source: §5 · p. 8

    The text-based format checker misses rare corner cases.

    Source: §2.4 · p. 5

Abstract

from arXiv · show

We introduce Native Parallel Reasoner (NPR), a teacher-free framework that enables Large Language Models (LLMs) to self-evolve genuine parallel reasoning capabilities. NPR transforms the model from sequential emulation to native parallel cognition through three key innovations: 1) a self-distilled progressive training paradigm that transitions from ``cold-start'' format discovery to strict topological constraints without external supervision; 2) a novel Parallel-Aware Policy Optimization (PAPO) algorithm that optimizes branching policies directly within the execution graph, allowing the model to learn adaptive decomposition via trial and error; and 3) a robust NPR Engine that refactors memory management and flow control of SGLang to enable stable, large-scale parallel RL training. Across eight reasoning benchmarks, NPR trained on Qwen3-4B achieves performance gains of up to 24.5% and inference speedups up to 4.6x. Unlike prior baselines that often fall back to autoregressive decoding, NPR demonstrates 100% genuine parallel execution, establishing a new standard for self-evolving, efficient, and scalable agentic reasoning.

1. Introduction

NPR addresses the gap between sequential reasoning and native parallel reasoning with self-distilled progressive training and parallel-aware reinforcement learning. Across eight reasoning benchmarks, it reports performance gains, inference speedups, and genuine parallel execution.

1. Introduction

  • The work targets a gap in agentic language modeling: existing systems struggle with native branching, efficient shared-state use, and teacher-free parallel reasoning.The introduction argues that sequential reasoning is insufficient for the trajectory breadth needed in agentic AI.Source: §1 · p. 1
  • NPR progresses from format discovery with reinforcement learning, through self-distilled parallel warmup that imposes topological constraints, to native-parallel reinforcement learning.The first stage yields simulated parallelism; the second converts it to real parallel execution.Source: §1 · p. 2
  • Parallel-Aware Policy Optimization optimizes branching policies within the execution graph so the model can learn adaptive decomposition through trial and error.This stage uses collision-free parallel rollouts.Source: §1 · p. 2
  • The NPR Engine redesigns memory management and flow control to support stable, large-scale parallel reinforcement-learning rollouts.The paper identifies memory leaks, incorrect parallel-token calculations, and runtime failures as issues addressed by the engine.Source: §1 · p. 2
  • Across eight reasoning benchmarks, NPR achieves performance improvements up to 24.5% and inference speedup up to 4.6×.The reported speedup is over autoregressive decoding.Source: §1 · p. 2
  • NPR performs 100% genuine parallel reasoning, while previous baselines use autoregressive fallback on 30%+ of test cases.The paper reports no hidden autoregressive fallbacks or pseudo-parallel behavior in NPR's evaluated test cases.Source: §1 · p. 2
  • Self-distilled datasets outperform previous teacher-generated trajectories by an average of 10.1 points.NPR's parallel methods also outperform direct sequential RL baselines such as DAPO.Source: §1 · p. 2

2. Native Parallel Reasoner

NPR develops native parallel reasoning through a three-stage curriculum that induces a structured format, stabilizes it with self-distilled supervision, and optimizes parallel reasoning directly with reinforcement learning. Its NPR Engine supports stable large-scale parallel rollouts.

2. Native Parallel Reasoner

  • NPR progresses from reinforcement-learning-based format induction to SFT on self-distilled trajectories, then to parallel-aware reinforcement learning for native parallel reasoning.NPR-ZERO induces the structured format, NPR-BETA stabilizes it, and the final stage directly optimizes parallel reasoning.Source: §2 · pp. 2–3

2.1. Preliminaries

  • Parallel reasoning factorizes a sample over a dependency graph, allowing steps without mutual dependencies to be processed concurrently.Pa(st) denotes the parent steps on which step st directly depends.Source: §2.1 · p. 3

2.2. Stage 1: Format-follow Reinforcement Learning

  • NPR-ZERO uses DAPO with format and answer-accuracy rewards to induce structured parallel outputs without paired supervision or external teachers, then self-distills its generations.Format-check failures receive a penalty in (0.0, -2.0]; format-valid correct and incorrect answers receive +1.0 and -1.0, respectively.Source: §2.2 · p. 4

2.3. Stage 2: Rejection Sampling and Parallel Warmup

  • The warmup stage samples candidate reasoning trajectories, filters them with lightweight constraints, and uses accepted traces for SFT initialization of parallel RL.Parallel attention masks and positional encoding support multiple reasoning paths in one forward pass and permit KV-cache reuse.Source: §2.3 · p. 4

2.4. Stage 3: Native-parallel RL

  • Native-Parallel RL uses the NPR Engine for parallel rollouts, filters malformed schemas before optimization, and normalizes advantages at batch level after format violations are removed.The method also preserves gradients on special tokens that control parallel branching and merging.Source: §2.4 · p. 5

2.5. Engineering Enhancement: NPR Engine

  • The NPR Engine improves determinism, memory stability, and correctness under large-scale parallel RL workloads, with its remedies reported as essential to preventing off-policy artifacts and stabilizing optimization.The engine addresses cache reuse and memory corruption, aggregate token-budget undercounting, and illegal parallel schemas.Source: §2.5 · pp. 5–6

3. Experiments

The experiments evaluate NPR’s training stages across reasoning benchmarks and compare its overall performance with sequential and parallel baselines. Results separately examine self-distilled data, parallel supervision, and parallel reinforcement learning.

3.1. Training Setup.

  • Training uses a fixed 8,000-example subset of ORZ across all three stages and Qwen3-4B-Instruct-2507 or non-thinking Qwen3-4B models.The authors omit the thinking-mode model because it cannot be trained with standard supervised fine-tuning.Source: §3.1 · p. 6

3.2. Evaluation Setup

  • Accuracy is measured with avg@8 on four smaller benchmarks and avg@1 on four larger or more heterogeneous benchmarks.The smaller set is AIME24, AIME25, HMMT25, and AMC23; the larger or heterogeneous set is OlympiadBench, Minerva-Math, ZebraLogic, and MATH500.Source: §3.2 · p. 6

3.3. Overall Reasoning Performance.

  • NPR gains substantially across all benchmarks over strong Qwen3-4B baselines and consistently outperforms Multiverse-32B and Multiverse-4B.Table 2 summarizes performance across eight reasoning benchmarks.Source: §3.3 · p. 6, Table 2 · p. 7
  • Replacing the Multiverse training corpus with NPR’s self-distilled dataset raises the average score from 50.1 to 59.0 (+8.9).The reported dataset substitutions also improve AIME24 from 46.7 to 50.8 (+4.1), ZebraLogic from 60.2 to 76.1 (+15.9), AMC23 from 75.0 to 85.9 (+10.9), and MATH500 from 81.6 to 91.6 (+10.0). The authors attribute the difference to greater accuracy and diversity of candidate solutions and limited parallel-pattern coverage in the Multiverse corpus.Source: §3.3 · pp. 6–7
  • Parallel SFT raises overall performance from 58.2 to 59.0 (+0.8), with reported gains on AIME25, OlympiadBench, HMMT25, and ZebraLogic.The authors report minor regressions on a few benchmarks and contrast parallel trajectories with the step-dependency priors of sequential SFT.Source: §3.3 · p. 7
  • Parallel RL raises the average score from 62.0 to 65.0 (+3.0) over sequential RL, with gains across the reported benchmarks.Reported gains include AIME24 from 57.1 to 63.3 (+6.2), HMMT25 from 26.3 to 30.8 (+4.5), and Minerva-Math from 38.2 to 43.0 (+4.8).Source: §3.3 · p. 7

4. Analyses and Discussion

The analyses report inference speedups, compare parallel-reasoning trigger rates across datasets, and trace performance through NPR’s training stages. They find uniform parallel triggering for NPR and a gradual evolution toward final performance above the autoregressive baseline.

4.1. Inference Acceleration While Improving Effectiveness

  • NPR achieves the best efficiency across five benchmarks, with speedups of 4.6× on AIME25, 4.1× on HMMT25, and 2.9× on AMC23.The authors report that NPR consistently outperforms Multiverse by 1.3×–2.4× and also exceeds autoregressive baselines; speedups are larger on harder problems.Source: §4.1 · p. 7, Table 3 · p. 8

4.2. Parallel Reasoning Trigger Analysis

  • NPR has a uniform 100.0% parallel rate across eight datasets, while Multiverse-32B’s rate varies across datasets.The authors note lower Multiverse performance on logic-intensive ZebraLogic than on several math contest datasets.Source: §4.2 · p. 8, Table 4 · p. 8

4.3. Evolution Dynamics Towards NPR

  • Naively enforcing parallel format severely reduces AIME25 performance, from 47.5 to 17.5, while the three-stage process ends above the autoregressive baseline.The reported evolution includes a small transient dip during format stabilization; Figure 4 presents evaluation dynamics on AIME 2025.Source: §4.3 · p. 8, Figure 4 · p. 8

5. Conclusion

NPR combines self-distilled parallel SFT with agentic parallel RL to produce genuine parallel reasoning without external teacher models. Across eight reasoning benchmarks, it improves performance, accelerates inference, and shows adaptive parallelism in case studies.

5. Conclusion

  • NPR produces genuinely parallel reasoning policies and shows consistent improvements over Multiverse datasets, autoregressive training, and direct RL.The analysis also reports meaningful inference acceleration, stronger test-time scalability, and no pseudo-parallel behavior.Source: §5 · p. 8
  • Stages 1 and 2 share an initialization, and Stage 1 supplies data to Stage 2.Source: §5 · p. 8
  • The authors conclude that native parallel reasoning is a promising direction for more general and scalable intelligence.Source: §5 · p. 9

A. Algorithms

The algorithms construct an attention mask and position IDs from tagged parallel plans, isolating parallel reasoning steps while preserving causal attention.

B. Algorithms

  • The attention-mask procedure begins with a causal mask and removes cross-attention between distinct parallel steps.The resulting mask maps allowed positions to 0 and disallowed positions to −∞.Source: A. Algorithms · p. 12
  • Parallel steps are isolated from one another in both directions in the attention mask.Source: A. Algorithms · p. 12
  • The positional-ID procedure builds positions from the token sequence and tag tokens using a structure stack.It adjusts positions within guideline blocks and aligns parallel steps using the maximum step length.Source: A. Algorithms · p. 12

B. Related Work

Prior parallel-reasoning approaches include independent-path selection, hand-designed search, imitation from stronger models, and reinforcement learning. The section contrasts their constraints and describes the shift from coarse outcome rewards toward process-aware and verifiable feedback.

B. Related Work

  • Best-of-N and Self-Consistency select among independent paths but are not end-to-end optimized.Tree-of-Thought, Graph-of-Thought, and MCTS explore trees but rely on hand-designed structures and external verifiers.Source: B. Related Work · p. 12
  • SFT approaches such as Multiverse (Yang et al., 2025b), ParaThinker, and SSFT imitate parallel paths from stronger models, limiting discovery of novel patterns.RL approaches APR and Parallel-R1 (Zheng et al., 2025b) are described as toy-task-limited or reliant on supervised bootstrap data.Source: B. Related Work · p. 13
  • Reasoning RL progressed from outcome-level rewards to process-aware feedback and then verifiable rewards, which offer more objective and reproducible signals for verifiable tasks.Process-level feedback can be subjective, costly to annotate, and unstable when intermediate signals are ambiguous or unverifiable.Source: B. Related Work · p. 13

C. Test-time Scalability

NPR raises best@8 scores over SFT for both Non-thinking and Instruct Qwen3-4B backbones, with reported gains varying by task and starting backbone strength.

C. Test-time Scalability

D. Qualitative Case Studies

The case studies show adaptive parallel exploration and reflective verification across correct examples. They illustrate both independent strategy search and parallel subproblem decomposition, while identifying planning and overlong chains as sources of error.

D. Qualitative Case Studies

  • More challenging problems trigger broader exploration, while easier prompts use fewer focused branches; a final block can cross-check or refine candidates.Source: D. Qualitative Case Studies · p. 13
  • Parallel reasoning explores independent strategies or decomposes a problem into subproblems; errors typically stem from a flawed high-level plan or unnecessarily long branches.Source: D. Qualitative Case Studies · p. 14
  • The examples depict subproblem decomposition followed by reflective reasoning, and independent solution methods with reflection within a step.Source: Table 6 · p. 14, Table 7 · p. 15
  • 106° is the reported ∠AHB result, with three lines of reasoning leading to the same answer.Source: D. Qualitative Case Studies · p. 15

Related papers

Loading 2512.07461v3…