Source-linked AI summary
$V_1$: Unifying Generation and Self-Verification for Parallel Reasoners
Harman Singh, Xiuyu Li, Kusha Sareen, Monishwaran Maheswaran, Sijun Tan, Xiaoxia Wu, Junxiong Wang, Alpay Ariyak, Qingyang Wu, Samir Khaki, Rishabh Tiwari, Long Lian, Yucheng Lu, Boyi Li, Alane Suhr, Ben Athiwaratkun, Kurt Keutzer
TL;DR
Parallel reasoning requires reliable verification to identify correct solutions among sampled candidates. V1 unifies generation and pairwise self-verification through uncertainty-guided inference and joint RL training, reporting stronger and more efficient test-time scaling across code and math benchmarks.
Problem
Existing pointwise verification lacks comparative calibration, while self-aggregation can cause diversity collapse and discard correct solutions.
Method
V1 combines uncertainty-guided Swiss tournament pairwise verification with online RL co-training of one model as generator and pairwise self-verifier.
Results
V1-Infer improves Pass@1 by up to 10% over pointwise verification, while V1-PairRL achieves 7–9% test-time scaling gains and up to 8.7% higher base Pass@1 than standard RL.
Takeaways & Limitations
Pairwise verification and unified training provide a framework for effective parallel reasoning with stronger self-verification and more efficient test-time scaling.
Takeaways & Limitations
The paper identifies calibration collapse in pointwise verification and diversity collapse in self-aggregation as boundaries affecting existing approaches.
Abstract
from arXiv · showhide
Test-time scaling for complex reasoning tasks shows that leveraging inference-time compute, by methods such as independently sampling and aggregating multiple solutions, results in significantly better task outcomes. However, a critical bottleneck is verification: sampling is only effective if correct solutions can be reliably identified among candidates. While existing approaches typically evaluate candidates independently via scalar scoring, we demonstrate that models are substantially stronger at pairwise self-verification. Leveraging this insight, we introduce $V_1$, a framework that unifies generation and verification through efficient pairwise ranking. $V_1$ comprises two components: $V_1$-Infer, an uncertainty-guided algorithm using a tournament-based ranking that dynamically allocates self-verification compute to candidate pairs whose relative correctness is most uncertain; and $V_1$-PairRL, an RL framework that jointly trains a single model as both generator and pairwise self-verifier, ensuring the verifier adapts to the generator's evolving distribution. On code generation (LiveCodeBench, CodeContests, SWE-Bench) and math reasoning (AIME, HMMT) benchmarks, $V_1$-Infer improves Pass@1 by up to $10%$ over pointwise verification and outperforms recent test-time scaling methods while being significantly more efficient. Furthermore, $V_1$-PairRL achieves $7$--$9%$ test-time scaling gains over standard RL and pointwise joint training, and improves base Pass@1 by up to 8.7% over standard RL in a code-generation setting.
1 Introduction
Parallel reasoning scales inference by sampling multiple solution paths, but its effectiveness depends on reliably identifying correct candidates. V1 addresses this bottleneck by combining pairwise verification with unified generation-and-verification training.
- Parallel reasoning samples multiple independent chains of thought, then aggregates them to select a final solution.
- Existing self-verification can fail when models cannot reliably identify the correct solution among candidates.
- V1 unifies inference-time pairwise verification with reinforcement learning that jointly trains generation and self-verification.
- V1-Infer uses Swiss-system tournament refinement to allocate verification compute toward uncertain candidate pairs.
- V1-PairRL co-trains one model as generator and pairwise self-verifier using an online, co-evolving objective.
- Up to 10% Pass@1 improvement over pointwise verification and 7–9% test-time scaling gains over standard RL and pointwise co-training were reported across code and math benchmarks.
2 Related Work
Prior test-time scaling methods generate diverse solutions through parallel reasoning, then select or aggregate them, but self-verification and aggregation introduce distinct limitations. V1 focuses on pairwise self-verification to address these gaps in parallel reasoning.
- Parallel scaling generates multiple reasoning paths, while final solutions are obtained through self-selection or self-aggregation.
- Many existing approaches use external ground-truth signals, such as majority voting for math or executable feedback for code.
- Pointwise self-verification is reported to be biased toward accepting incorrect solutions.
- Self-aggregation can suffer diversity collapse and discard correct solutions during refinement.
- Prior generative-verifier and co-training approaches often use separate verifier or aggregator models and may incur additional costs.
3 Limitations of Current Self-Verification and Aggregation Approaches
Pointwise self-verification lacks a globally comparable scale, while recursive self-aggregation can reduce solution diversity and discard correct candidates. These limitations motivate pairwise verification as a diversity-preserving alternative.
- Pointwise verification lacks comparative references, causing calibration problems and potentially over-scoring plausible but incorrect solutions.
- Pairwise judgments provide relative comparisons and yield higher top-1 self-ranking accuracy than pointwise scoring.
- Self-aggregation combines parallel solutions but can monotonically reduce Pass@N as aggregation steps increase.
- Recursive self-aggregation may discard or degrade correct outlier solutions during refinement.
- Pairwise self-verification is presented as a better-calibrated, diversity-preserving alternative to pointwise verification and self-aggregation.
- Majority voting is less general because it applies only when objective ground-truth answers are available, such as in math.
4 Improving the Self-Verification Capability of LLMs using V1-Infer
V1-Infer improves self-verification by ranking candidate solutions through uncertainty-guided pairwise comparisons rather than isolated scores. Its budgeted tournament procedure improves accuracy over pointwise verification and Recursive Self-Aggregation while concentrating comparisons on informative candidates.
- Method: V1-Infer uses weighted pairwise comparisons and a two-phase budget strategy to rank candidates while avoiding exhaustive O(N^2) verification.Pairwise ratings use confidence-weighted outcomes; topology coverage anchors candidates before Swiss refinement focuses on uncertain near-ties.
- Method: V1-Infer first enforces minimum comparison coverage, then refines rankings by pairing nearby candidates with uncertain scores.The coverage phase reduces path dependence, while Swiss refinement prioritizes unseen pairs with small score gaps.
- Results: +7.3%: On CodeContests, GPT-OSS-20B improves from 66.06% to 73.33% with V1-Infer versus pointwise verification.Other reported gains include +8.6% and +10.0% for GPT-OSS-20B on LiveCodeBench-v5 and HMMT, respectively.
- Results: 45%: At a compute budget of 64 model calls, pairwise verification reaches 45% average accuracy on code-generation benchmarks versus approximately 33% for pointwise verification.The comparison uses Qwen3-4B-Instruct-2507 in a compute-matched setting.
- Results: 76% Pass@1: V1-Infer reaches this accuracy on LiveCodeBench-v6 with N=16 and only 48 verification calls, exceeding RSA's maximum accuracy with fewer calls.The method's uncertainty-guided Swiss refinement concentrates calls on informative pairs.
- Analysis and Ablations: +23.7%: On hard LCB-v6 problems, pairwise verification with a 3x budget raises accuracy from 40.2% Pass@1 to 63.9%.The reported gain is larger than the improvement on medium-difficulty problems, while easy problems already reach 99.3% Pass@1.
5 V1-PairRL: Improving Self-Verification via Unified RL Training
V1-PairRL unifies generation and pairwise self-verification in an online, co-evolving RL framework so verification data remains aligned with the generator’s current distribution. Across code benchmarks, this training improves test-time scaling and base generation quality over pointwise or standard RL baselines.
- Framework: V1-PairRL trains one LLM jointly for solution generation and pairwise verification using shared online rollouts.JGen optimizes correct reasoning paths, while JPairVerif optimizes pairwise ranking judgments on solutions generated by the current policy.
- Training objective: The unified objective uses the same G generated solutions both for generation rewards and for forming verification pairs.This keeps verifier training data in-distribution as the policy evolves.
- Reward design: A sparsity threshold rewards verifier scores only when they are within 0.2 of ground-truth correctness, discouraging safe middle-ground predictions.Scores near 0 or 1 receive positive reward, while indiscriminate 0.5 outputs do not.
- Reward hacking: Strict pairing prevents the empty-solution loop by triggering verification training only when pairs contain at least one suitable solution.Otherwise, the verifier could benefit from judging two incorrect outputs while the generator degrades toward trivially incorrect responses.
- Test-time scaling: +6.5% Pass@1 on LiveCodeBench-v5: V1-PairRL reaches 53.9% versus 47.4% for V1-PointRL at 2x budget.The corresponding gains are +6.8% on LiveCodeBench-v6 and +7.3% on CodeContests, with accuracy increasing as verification budget grows.
- Comparison with RL: +8.9% on CodeContests: V1-PairRL outperforms the RL baseline when both use V1-Infer with identical 2x verification budgets.The gains are +3.6% on LiveCodeBench-v5 and +1.9% on LiveCodeBench-v6.
- Base generation quality: +8.7% Pass@1 on CodeContests: V1-PairRL improves over the RL baseline without test-time scaling at N=16.It also improves Pass@1 by +2.9% on LiveCodeBench-v5 and +2.7% on LiveCodeBench-v6.
- Ablation: Co-evolving training consistently outperforms non-co-evolving training across all evaluated benchmarks.Pairwise verification therefore benefits from training on samples produced online by the model’s evolving generator rather than offline base-model samples.
6 Conclusion
The paper concludes that pairwise comparison is a stronger primitive for self-verification in parallel reasoning than absolute scoring. V1 combines uncertainty-guided tournament refinement with unified generation-and-verification training to improve scalable parallel reasoning.
- Conclusion: V1 is grounded in pairwise comparison as a more robust verification primitive than absolute scoring.The framework targets self-verification in parallel reasoning.
- V1-Infer: V1-Infer dynamically allocates verification compute to ambiguous candidate pairs through tournament-based refinement.It outperforms pointwise verification and prior aggregation techniques in the reported experiments.
- V1-PairRL: V1-PairRL jointly trains one model for generation and pairwise self-verification, improving test-time scaling over standard RL and pointwise baselines.The training is presented as a post-training complement to V1-Infer.
- Takeaway: Unified generation and pairwise verification provides a framework for both RL training and scalable parallel reasoning.This conclusion follows the paper’s combined inference and training results.
B V1-PointRL Training Details
V1-PointRL is a budget-matched baseline that co-trains generation with independent pointwise scoring rather than pairwise comparison. At inference, it selects the highest-scoring solution among independently evaluated candidates.
- Baseline: V1-PointRL uses the same co-evolving training setup as V1-PairRL but scores each solution independently.The verification objective, prompt, and reward computation distinguish it from pairwise training.
- Verification reward: The pointwise verifier outputs a confidence score from 1 to 10 for one solution and normalizes it to v ∈ [0, 1].The ground-truth correctness label is y ∈ {0, 1}.
- Stability: The sparsity threshold prevents safe-bet collapse, where the verifier always outputs v = 0.5.Without the threshold, solution-generation performance collapses and training becomes unstable.
- Inference: At inference, V1-PointRL generates N solutions, scores them independently, and selects the solution with the highest pointwise score.This provides the pointwise comparison baseline for pairwise verification.
C.1 Inference Sampling Parameters
The inference experiments use fixed sampling and model-generation settings, varying candidate count and verification budget across experiments. Trained-model results average three random seeds, while base-model inference runs once.
- Inference framework: SGLang provides batched inference for all inference experiments.The implementation choice is held across the reported inference studies.
- Sampling: Sampling uses temperature 0.6 for code, 1.0 for math, and top-p 0.95 throughout.These settings encourage exploration of the solution space.
- Candidate count: Most experiments generate N ∈ {8, 16} candidate solutions.Budget-matched comparisons additionally use N = 32 for pointwise verification against pairwise N = 16 at 3x budget.
- Verification budget: V1-Infer evaluates verification budgets of 1x, 2x, and 3x the number of base solutions.Full results across all budgets are reported in the appendix figures.
- Experimental runs: Base-model inference experiments run once, whereas trained-model results average three runs with different random seeds.Trained-model budget exploration is limited to 1x and 2x because of computational cost.
- Swiss refinement: Swiss refinement uses dmin = 2, window size h = 8, confidence floor τ = 0.1, and shared parameters across benchmarks and models.The parameters are not tuned to any specific benchmark.
C.2 Training Hyperparameters
The appendix specifies the procedures, prompts, and evaluation settings used for V1’s pairwise and pointwise verification experiments.
- Algorithm specification: V1’s detailed algorithm maintains estimated solution quality, comparison degrees, and a history of previously compared pairs.The state records µ_i, d_i, and H for ranking and comparison coverage.
- Algorithm specification: SwissPairs allocates remaining verification budget to informative near-ties, while CoveragePairs ensures minimum comparison degree among candidates.Together, the procedures support uncertainty-focused ranking with O(N)-scale pairwise verification.
- Verification prompts: Pairwise verification prompts models to rate two code or math solutions on a 1–10 scale, with higher ratings determining the winner and equal ratings producing ties.The prompts require separate ratings for solutions A and B and compare their correctness.
- Verification prompts: Pointwise verification prompts independently rate code or math solutions on a 1–10 correctness scale.The code prompt emphasizes robustness, while the math prompt grades the final answer.
G.1 Example 1: django__django-11049: Pairwise Correct, Pointwise Wrong
Across code-repair examples, pairwise comparison often selects patches that address the underlying issue more completely, though its ranking can also favor an overly complex incorrect patch.
- DurationField format: Pairwise ranking selected the correct Django DurationField patch, whereas pointwise ranking selected a patch that only added form help text.The correct patch also fixed the model error-message format string.
- Draggable legend pickling: Pairwise ranking selected the root-class DraggableBase fix for pickling, while pointwise ranking selected a Legend-only fix that missed draggable annotations.The parent-class change handles the unpickleable canvas reference for both cases.
- PostScript empty text: Pairwise comparison exposed an indentation error in the pointwise-selected PostScript patch, despite both patches attempting the same guard.The intended fix nests stream.append under if curr_stream.
- Q-Exists combination: Pairwise verification can fail when a more complex patch appears more thorough, selecting a regression-inducing change over a minimal correct fix.In the Q-Exists example, the correct patch wraps Expression operands in Q(), while the selected alternative modifies multiple files and breaks deconstruct() handling.
- Pipeline length: Both methods selected correct implementations of Pipeline.__len__, showing that different candidate placements can be equally valid.The alternatives differed only in method placement and docstring presence.
H.1 Example 1: Binary String Trade (1/16 correct): Pairwise Correct, Pointwise Wrong
Two coding examples show pairwise verification improving selection when pointwise scores collapse distinct candidates into ties or reward superficially plausible solutions.
- Binary String Trade: The correct binary-string solution enumerates trades over blocks of an augmented string, whereas the pointwise-selected solution mishandles augmented boundaries.The task augments the input with 1s at both ends before evaluating trades.
- Binary String Trade: Pairwise verification found the only correct binary-string solution among 16 candidates, while pointwise selected an incorrect solution with a perfect score.The correct candidate received 8/10 pointwise but the highest pairwise µ score of 1.000.
- Substring Character Check: Pairwise ranking rejected a subtle off-by-one solution in the substring task, while pointwise gave that incorrect candidate 10/10 alongside most correct solutions.The bug checks s[i+k-1] instead of the first character after the substring, s[i+k].
H.3 Example 3: Group Element Assignment (4/16 correct): Pairwise Correct, Pointwise Wrong
In group element assignment, pairwise ranking selects a correct candidate where pointwise scoring saturates and favors an incorrect brute-force solution. The contrast highlights pairwise comparison's ability to distinguish algorithmic efficiency.
- The task requires assigning each group a valid element of smallest index, or −1 when no element divides the group size.
- 4 of 16 candidates are correct, and pairwise ranking selects correct idx=12 while pointwise ranking selects incorrect idx=0.Pointwise scoring assigns 10/10 to 12 of 16 candidates, including 9 incorrect solutions; correct idx=8 receives 3/10.
- The correct solution enumerates divisors in O(√g) with hash-map lookup, whereas the incorrect solution uses an O(|groups| × |elements|) nested loop that exceeds time limits.Both approaches produce identical outputs on small examples, obscuring the efficiency difference during pointwise inspection.
- Pairwise comparison recognizes the more efficient algorithm by directly contrasting the two otherwise similar solutions.The example presents this as a score-saturation failure in which pointwise scoring cannot distinguish algorithmic efficiency.
H.5 Example 5: Three-Subarray Distinct Count (2/16 correct): Pairwise Correct, Pointwise Wrong
In the three-subarray distinct-count problem, pairwise ranking places both correct solutions near the top while pointwise scoring favors an incorrect solution. The example shows pairwise verification is less susceptible to surface-level judgments about code quality.
- The task requires splitting a sequence into three non-empty contiguous subarrays to maximize the total number of distinct integers, with 3 ≤ N ≤ 3 × 10^5.
- 2 of 16 candidates are correct, and pairwise ranking selects correct idx=3 while pointwise ranking selects incorrect idx=10.Pointwise scores the correct candidates 4/10 and 3/10, but scores incorrect idx=10 6/10.
- The incorrect candidate contains an off-by-one error in sliding-window handling that miscounts middle-segment distinct elements when repeated values are encountered.
- Pairwise ranking places both correct solutions in the top five, with µ = 0.946 and 0.773, while placing incorrect idx=10 at rank 10.The correct solutions appear less optimized because they use straightforward O(N^2) enumeration, which likely lowers their pointwise scores.