PaperlayerView exact revision on arXiv

Source-linked AI summary

Planned Test-Time Scaling with Coordinated Reasoning Paths

Xueqing Wu, Langxing Bai, Hritik Bansal, Po-Nien Kung, Shuo Li, Hao Liu, Nanyun Peng, Kai-Wei Chang

arXiv:2609.27374v1cs.CLcs.AI

TL;DR

Independent repeated sampling can spend additional inference compute on redundant reasoning paths. PTTS coordinates branches through planner-generated outlines and executor-produced solutions; across five mathematical reasoning benchmarks, its variants improve pass@64 over repeated sampling by up to 13.4 points.

  • Problem

    Independent repeated sampling may cluster attempts around similar high-probability solutions, limiting coverage and the benefit of additional inference compute.

    Source: §1 · p. 2, §3.1 · p. 3

  • Method

    PTTS uses a planner to generate branch-specific outlines and an executor to produce solutions conditioned on those outlines, with zero-shot and pass@k-trained planner variants.

    Source: §1 · p. 2, §3.3 · p. 6

  • Results

    Across five mathematical reasoning benchmarks, PTTS-RL improves pass@64 over repeated sampling by up to 13.4 points.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Greater outline diversity is associated with greater solution diversity and higher pass@4, while PTTS-RL directs diversity toward branches with correct answers.

    Source: §4.2 · p. 7

    The theoretical mode-selection analysis assumes an imperfect fixed executor, noisy category observations, a dominant mode, and a complementary mode that helps on a worst-case category.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Test-time scaling with parallel branches is widely adopted to improve performance on challenging reasoning tasks. The predominant approach, repeated sampling, draws branches independently from a single policy, which can produce redundant attempts and thereby limit the gains from additional inference compute. To address this limitation, we propose Planned Test-Time Scaling (PTTS), which replaces independent sampling with a coordinated joint policy: a planner generates a solution outline for each branch, steering the branches toward distinct reasoning paths, and an executor produces a full solution conditioned on each outline. Formally, we show that PTTS strictly generalizes repeated sampling and, in a stylized setting, provably promotes coverage of complementary reasoning modes and yields better pass@k scaling. We instantiate PTTS on top of strong reasoning models, keeping them fixed as executors while replacing repeated sampling with PTTS inference to further enhance test-time scaling. Concretely, we develop two variants: PTTS-ZS prompts a model to jointly generate outlines for all branches in a single autoregressive pass, while PTTS-RL directly optimizes the planner against the pass@k reward using truncated execution rollouts for efficient training and a sharper reward signal. Across five mathematical reasoning benchmarks with Qwen3-1.7B and 4B, PTTS-ZS improves pass@64 over repeated sampling by up to 6.7 points, while PTTS-RL further increases the gain to up to 13.4 points. Further analysis indicates that broader coverage of distinct reasoning paths contributes to these gains. Overall, PTTS provides a general framework for improving test-time scaling by coordinating reasoning branches, with zero-shot and trainable instantiations that yield substantial performance gains.

1 INTRODUCTION

Repeated sampling can waste parallel inference compute on redundant reasoning paths, while PTTS coordinates branches through jointly planned outlines. Its zero-shot and trained variants improve pass@k across mathematical reasoning benchmarks.

1 INTRODUCTION

  • Independent branches can concentrate on high-probability reasoning modes, repeating attempts and limiting the value of additional inference compute.The introduction also notes that RL may improve pass@1 while narrowing the policy distribution and degrading pass@k.Source: §1 · p. 2
  • Prior coordinated-planning methods use handcrafted pipelines, whereas PTTS formalizes planning as a joint policy that can be directly optimized.Examples include iterative concept elicitation and over-generation followed by plan clustering.Source: §1 · p. 2
  • PTTS replaces independent sampling with a planner-executor joint policy that assigns solution outlines to branches.The planner steers branches toward distinct reasoning paths, and the executor generates a full solution from each outline.Source: §1 · pp. 2–3
  • PTTS strictly generalizes repeated sampling, and in a stylized setting pass@k optimization promotes coverage of complementary reasoning modes.The authors report that broader reasoning-space coverage contributes to the observed gains.Source: §1 · pp. 2–3
  • PTTS-ZS jointly generates all branch outlines in one autoregressive pass, while PTTS-RL optimizes the planner directly for pass@k with a fixed executor.PTTS-RL truncates executor reasoning during training to reduce rollout cost and sharpen planner credit assignment.Source: §1 · p. 2
  • PTTS-RL improves pass@64 over repeated sampling by up to 13.4 points across five mathematical reasoning benchmarks.PTTS-ZS gains up to 6.7 points, and PTTS-RL further increases the gain.Source: §1 · p. 2

2 RELATED WORK

Related work addresses redundant test-time attempts through inference-time coordination and addresses mode collapse through diversity-oriented RL. PTTS is positioned as a formal, trainable approach that complements marginal-policy improvements.

2 RELATED WORK

  • Repeated sampling can make added attempts redundant when reduced diversity concentrates branches on similar solutions, particularly after RLVR narrows the policy distribution.The passage notes that this can limit coverage and harm pass@k.Source: §2 · p. 3
  • Inference-time approaches coordinate branches with plans, clustering, or query perturbations, but the cited methods lack a unified end-to-end optimizable framework.The passage also distinguishes planner-training work focused mostly on synthetic or coding tasks and lacking a joint formulation or theoretical perspective.Source: §2 · p. 3
  • Diversity-oriented RL improves the marginal policy through pass@k objectives, entropy, semantic diversity, risk-aware objectives, or exploration curricula.The paper characterizes these marginal-policy methods as complementary to PTTS, which can be combined with them.Source: §2 · p. 3

3 PLANNED TEST-TIME SCALING

PTTS replaces independent attempts with coordinated branches by factoring test-time computation into planning and execution. The section analyzes why coordination can improve pass@k, then describes zero-shot and RL-based instantiations.

3.1 FROM REPEATED SAMPLING TO PTTS

  • PTTS factors the joint policy into a planner that produces k outlines and an executor that generates each solution conditioned on its corresponding outline.The outlines steer attempts toward distinct reasoning paths, making branch coordination tractable.Source: §3.1 · p. 4

3.2 WHY PTTS IMPROVES PASS@k

  • PTTS strictly generalizes repeated sampling, so its optimal achievable pass@k is no lower than repeated sampling’s.An empty or constant outline for every branch reduces PTTS to repeated sampling.Source: §3.2 · p. 4
  • With fixed per-attempt success probability p, ideally disjoint PTTS success events yield pass@k = min(kp, 1), exceeding independent sampling for p in (0, 1) and k > 1.Repeated sampling instead yields pass@k = 1 − (1 − p)^k; the comparison illustrates how reducing redundancy can improve joint success.Source: §3.2 · p. 4
  • In the stylized mode-selection setting, pass@1 optimization concentrates on the dominant mode, while PTTS-RL allocates slots to non-dominant modes beyond a finite k* threshold.Under the stated conditions, PTTS-RL drives residual error toward zero exponentially; GRPO can plateau when the dominant mode fails on the worst-case category.Source: §3.2 · pp. 4–5

3.3 INSTANTIATION

  • PTTS-ZS uses a base model to generate k outlines in one pass and a strong reasoning model as executor; PTTS-RL trains the planner with GRPO against pass@k.PTTS-RL truncates executor outputs during training to reduce compute and limit reward from executor rethinking that could mask a poor outline.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

Across five mathematical benchmarks, the experiments compare PTTS variants with repeated and guided sampling, examine reasoning-path diversity, and test design choices. PTTS-RL leads the main comparisons, while analyses report diversity patterns and identify effective branching and execution settings.

4.1 SETUP

  • Experiments cover five benchmarks and Qwen3 1.7B/4B, compare repeated and Guided Sampling, and evaluate pass@k to 64; PTTS-RL planners train against pass@4.For larger total budgets, the procedure is repeated with branching factor k = 4 until K executor solutions are collected.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS AND DISCUSSIONS

  • PTTS-RL raises pass@64 over repeated sampling by up to 13.4 points (1.7B) and 6.7 points (4B), while pass@32 exceeds repeated-sampling pass@64 at both scales.The figures report average pass@64 gains of 6.4 points for 1.7B and 3.6 points for 4B; PTTS-ZS also consistently beats repeated sampling and PTTS-RL beats both baselines.Source: Figure 4 · p. 7, §4.2 · p. 7, Figure 5 · p. 8
  • Outline diversity correlates with solution diversity (ρ = 0.57) and pass@4 (ρ = 0.59), while PTTS-RL has 15.9% useful diversity despite PTTS-ZS leading overall diversity.Solution diversity also correlates positively with pass@4 (ρ = 0.32).Source: §4.2 · p. 7
  • Qwen3-1.7B and 4B executors follow their outlines at rates of 70% and 71%, respectively, but may pivot to alternatives when an approach fails.The authors note that such pivots may dilute outline influence as reasoning traces lengthen and affect the training signal.Source: §4.2 · p. 8

4.3 ABLATION STUDIES

  • Pass@64 peaks at branching factor k = 4 for Qwen3-1.7B and k = 8 for Qwen3-4B; larger k yields no consistent gains and raises PTTS-RL training cost.The main experiments therefore use k = 4.Source: §4.3 · p. 8, Figure 7 · p. 9
  • With a 10k-token inference budget, matching the 10k training execution budget lowers average pass@64 by 7.8 points and throughput by 27% versus the 4k peak.The 4k budget allows outlines to be executed while truncating unguided rethinking; an overly strict 2k budget truncates valid reasoning.Source: §4.3 · pp. 8–9, Figure 8 · p. 9
  • A 1.7B planner can guide a Qwen3-4B executor, matching 4B planners despite a train-test mismatch, while a 4B planner adds no benefit with a 1.7B executor.The results suggest training a smaller planner and scaling the executor at inference time.Source: §4.3 · p. 9, Table 1 · p. 9
  • With an e3-1.7B executor, PTTS-ZS and PTTS-RL beat repeated sampling by 4.4 and 3.3 points, respectively.These results show PTTS gains alongside an executor trained for diversity-aware exploration.Source: §4.3 · p. 9

5 CONCLUSION

PTTS coordinates reasoning branches through planning and execution, and its zero-shot and trained variants improve test-time scaling in the reported evaluations.

5 CONCLUSION

  • Figure 9 compares pass@k for repeated sampling, PTTS-ZS, and PTTS-RL using e3-1.7B as executor.The caption identifies the plotted methods and executor but gives no specific outcomes.Source: Figure 9 · p. 9
  • PTTS coordinates branches with a planner that produces outlines and an executor that generates solutions conditioned on them.The framework replaces independent repeated sampling with a coordinated joint policy.Source: §5 · p. 9
  • Across five mathematical datasets, PTTS-ZS and PTTS-RL consistently yield gains of up to 13.4 points over independent repeated sampling.The conclusion reports theoretical and empirical advantages for PTTS.Source: §5 · p. 9

A PROOFS OF PROPOSITIONS

The proof analysis contrasts pass@1-trained repeated sampling with direct pass@k allocation, showing mode collapse for the former and diversification for PTTS-RL under stated assumptions.

A PROOFS OF PROPOSITIONS

  • The analysis assumes finite data-category and reasoning-mode spaces, conditionally independent attempt outcomes, and imperfect observation of the true category.Its results also use mode dominance and a complementary mode for the dominant mode’s worst category.Source: A PROOFS OF PROPOSITIONS · p. 12
  • Under mode dominance, the unique pass@1-optimal marginal policy selects only the dominant mode, so GRPO-RS uses it for every attempt.The proposition states this result under Assumption 1.Source: A PROOFS OF PROPOSITIONS · p. 13
  • Under Assumptions 1–2, PTTS-RL assigns attempts to non-dominant modes beyond a finite budget threshold.The proof establishes that the pure allocation is strictly suboptimal from the threshold onward.Source: A PROOFS OF PROPOSITIONS · pp. 13–14
  • The threshold is sufficient rather than exact, and hedging pays off sooner when the worst category is more probable or more fixable.The threshold decreases as the posterior probability of that category or the complementarity gap increases.Source: A PROOFS OF PROPOSITIONS · p. 14
  • Under Assumptions 1–2, PTTS-RL error vanishes exponentially with a strictly better exponent, while GRPO-RS can plateau at the worst category’s posterior mass.When the dominant mode cannot solve that category, repeated sampling replays the failure; allocating a constant fraction to a complementary mode improves the error exponent.Source: A PROOFS OF PROPOSITIONS · pp. 14–15
  • PTTS-RL achieves at least the pass@k of GRPO-RS because its feasible allocations include placing every attempt on the dominant mode.The proof identifies that pure allocation with GRPO-RS’s pass@k.Source: A PROOFS OF PROPOSITIONS · p. 15

A.1 CLOSED-FORM ANALYSIS OF A SYNTHETIC EXAMPLE

The synthetic example gives closed-form comparisons in a blind two-category, two-mode setting: repeated sampling favors one mode, while PTTS-RL eventually splits its budget.

A.1 CLOSED-FORM ANALYSIS OF A SYNTHETIC EXAMPLE

  • The example uses a blind policy with prior probability 0.75 for c1, and each mode succeeds at 0.40 on one category and 0.02 on the other.The two modes are each highly effective on exactly one category.Source: §A.1 · p. 15
  • The dominant mode z1 yields rapid pass@k saturation on c1 problems and slower improvement on c2 problems under repeated sampling.The cited discussion attributes the slower improvement to the 0.98^k term.Source: §A.1 · pp. 15–16
  • PTTS-RL starts allocating slots to z2 at k = 4, with the optimal allocation placing approximately k/2 − 1.12 slots on z2.The allocation is rounded to an integer in this symmetric instance.Source: §A.1 · p. 16
  • With an approximately even split, PTTS-RL’s residual error decays at approximately 0.77 per attempt, faster than the repeated-sampling rate described in the example.The comparison is between the mixed allocation’s per-attempt rate and the 0.98^k term for repeated sampling.Source: §A.1 · p. 16

B TRAINING DETAILS

The training details specify PTTS-RL’s model setup, training data, compute, and reported checkpoint.

B TRAINING DETAILS

C.1 OUTLINE ADHERENCE ANALYSIS

The analysis uses GPT-5-mini to score how closely executor responses follow planner outlines.

C.1 OUTLINE ADHERENCE ANALYSIS

  • GPT-5-mini judges whether each response’s primary reasoning path implements the outline’s strategy, decomposition, or perspective, then normalizes its five-point score to [0, 1].The judge receives the problem, outline, and executor response.Source: §C.1 · p. 16

C.2 DIVERSITY ANALYSIS BY CLUSTERING

The paper measures outline and solution diversity by clustering responses according to their underlying mathematical or logical concepts.

C.2 DIVERSITY ANALYSIS BY CLUSTERING

  • The procedure extracts each outline’s or solution’s pivotal concept, then groups concepts that represent the same underlying mathematical mechanism.Concept extraction and clustering are applied separately to planner outlines and executor solutions.Source: §C.2 · pp. 16–17

C.3 DIVERSITY METRICS

The appendix defines diversity metrics, describes a rank-averaging correlation test, and supplies qualitative examples and prompts for planning, execution, and analysis.

C.3 DIVERSITY METRICS

  • Overall diversity counts the distinct clusters among k responses, with larger counts indicating broader coverage of reasoning approaches.Responses are assigned to clusters before diversity is measured.Source: §C.3 · p. 17
  • Useful diversity measures how distinct the reasoning approaches are among correct responses, normalized by the number of correct responses.It is low when correct responses concentrate on the same approach.Source: §C.3 · p. 17

C.4 RANKED-AVERAGED CORRELATION TEST

  • The correlation test rank-aligns groups within each problem, averages variables at each rank across problems, then computes Spearman correlation.It examines outline diversity versus response diversity, outline diversity versus pass@k, and response diversity versus pass@k.Source: §C.4 · p. 17

D QUALITATIVE EXAMPLES

E PROMPTS

  • The planner prompt requests distinct start-to-finish outlines without solving; the executor prompt then asks for a stepwise solution following an outline.Source: E PROMPTS · pp. 19–20
  • The outline-following rubric scores adherence to the outlined reasoning path separately from correctness, so a correct solution using another method can score low.Source: E PROMPTS · p. 20
  • Analysis prompts extract one pivotal concept from each response and cluster concepts by shared mathematical or logical mechanism, regardless of response correctness.Source: E PROMPTS · pp. 21–22

Related papers

Loading 2609.27374v1…