Source-linked AI summary
Reasoning Arena: Trace Tournaments When Verifiable Rewards Fall Short
Han Zhou, Adam X. Yang, Laurence Aitchison, Anna Korhonen, Albert Q. Jiang
TL;DR
RLVR provides no group-relative learning signal when all sampled traces receive identical rewards, despite potentially differing reasoning quality. Reasoning Arena routes these groups to head-to-head trace tournaments and aggregates sparse comparisons with Bradley-Terry estimation. It consistently improves performance over RLVR while accelerating training and reducing generation compute.
Problem
RLVR’s group-relative advantages collapse to zero for non-diverse reward groups, wasting traces whose reasoning quality may differ.
Method
Reasoning Arena adaptively routes non-diverse groups to judge-based trace tournaments and fits Bradley-Terry rewards on incomplete comparison graphs.
Results
7.6 points: REASONING ARENA-Live reaches 53.9 average performance versus 46.3 for standard RLVR across the reported evaluation tasks.
Takeaways & Limitations
The framework converts otherwise discarded non-diverse traces into usable relative reward signals while preserving exact verifier supervision for diverse groups.
Takeaways & Limitations
The prevalence of non-diverse groups depends on the training-data mixture and the base policy’s capabilities.
Abstract
from arXiv · showhide
Reinforcement learning with verifiable rewards (RLVR) has become a leading paradigm for improving the reasoning ability of large language models through outcome-based supervision. However, verifiable rewards frequently become uninformative at the group level: when all sampled traces of a given prompt receive identical rewards, group-relative advantage estimation provides no gradient signal, even though the traces may differ substantially in reasoning quality. We propose Reasoning Arena, an adaptive training framework that routes such non-diverse reward groups to a judge system instead of discarding them. Beyond examining the final answer, Reasoning Arena constructs trace tournaments, where reasoning traces are compared head-to-head to expose finer-grained preferences within the group, converting reasoning quality into rich relative reward signals. To make reward estimation efficient, rather than exhaustively comparing every pair, each new trace is evaluated against a small, dynamically updated pool of previously generated traces as anchors to efficiently establish a relative ranking. We then fit a Bradley-Terry model on the incomplete comparison graph, enabling scalable RL integration without quadratic pairwise comparisons. Empirical results demonstrate that Reasoning Arena consistently outperforms the RLVR baseline by 7.6% on average in competition mathematics and coding benchmarks. By converting otherwise wasted zero-advantage samples into useful gradient updates, our method accelerates training by 27% to 41%, saving nearly 50% of generation compute, and substantially improves overall reasoning performance.
1 Introduction
RLVR can waste generated reasoning traces when every trace in a prompt group receives the same verifier outcome. Reasoning Arena routes these non-diverse groups to trace tournaments while retaining verifiable rewards for diverse groups.
- Motivation: Identical rewards make group-relative advantages zero even when traces differ in reasoning quality, soundness, or conciseness.Such groups contribute no gradient, wasting the compute used to generate their traces.
- Motivation: Prior approaches mainly filter easy or difficult prompts, but discarding non-diverse groups forfeits fine-grained signals in their trajectories.Entropy-guided shaping instead relies on intrinsic token distributions and cannot distinguish rigorous proofs from hallucinated derivations.
- Approach: Reasoning Arena adaptively routes non-diverse groups to an LLM-judge tournament while preserving verifier rewards for groups with informative diversity.The judge compares reasoning traces head-to-head to recover relative preferences.
- Results: The framework improves math and code reasoning while also improving training efficiency, out-of-domain generalization, and utilization of zero-advantage samples.These samples would otherwise be discarded or produce no effective update.
2 Related Work
Related work addresses RLVR’s zero-variance groups through filtering, intrinsic reward shaping, or surrogate rewards, while judge-based methods provide richer but potentially biased supervision. Reasoning Arena combines verifier and external judge rewards adaptively and uses tournament-style comparisons for non-diverse groups.
- RLVR: Group-relative RLVR methods inherit a zero-variance failure mode when all traces in a group receive the same verifier outcome.Dynamic sampling methods address this by filtering fully correct or fully incorrect groups.
- RLVR: Entropy-guided methods recover signals from zero-variance prompts, but their policy-intrinsic rewards cannot distinguish rigorous proofs from hallucinated derivations.REASONING ARENA instead introduces an external judge reward applied adaptively.
- LLM-as-a-Judge: LLM-as-a-judge methods evolved from pointwise rubric scoring toward pairwise elicitation and tournament-based ranking.Pointwise judging is sensitive to prompt design and surface-level bias.
3 Preliminary
Group-relative RLVR scales token updates using advantages computed from within-group verifier rewards. When binary rewards lack diversity, the variance and every trace advantage collapse to zero, so the generated group contributes no reward-driven gradient.
- RLVR Setup: RLVR samples N reasoning traces per prompt and assigns each a binary verifiable reward from a rule-based verifier.CISPO clips importance-sampling weights rather than policy-gradient terms.
- RLVR Setup: The group-relative trace advantage scales every token-level update for its corresponding trace.The clipped importance-sampling weight controls the token-level update magnitude.
- Zero-Variance Failure: A group produces a learning signal only when at least one trace has a non-zero advantage.Without such an advantage, the group contributes no meaningful gradient.
- Zero-Variance Failure: Reward diversity is defined through within-group reward variance, with binary non-diverse groups consisting entirely of correct or entirely of incorrect traces.These are the all-correct and all-incorrect cases.
- Zero-Variance Failure: D(G) = 0 iff σ_G = 0 iff A_i = 0 for every trace i.The equivalence formalizes the collapse from non-diverse rewards to zero advantages.
- Zero-Variance Failure: Non-diverse groups contribute no reward-driven policy gradient regardless of differences in their reasoning trajectories.Discarding their already-generated traces forces additional sampling to refill asynchronous training batches, increasing cost and latency.
4 Reasoning Arena
Reasoning Arena routes reward-diverse groups through standard RLVR and sends non-diverse groups to trace tournaments, where efficient pairwise comparisons and Bradley-Terry aggregation produce relative rewards.
- Adaptive group routing: Reward-diverse groups retain verifier rewards, while non-diverse groups are routed to judge-based rewards because the verifier otherwise provides zero gradient.The router applies judge intervention only where within-group reward variance is absent.
- Trace tournaments: Trace tournaments compare reasoning trajectories pairwise because identical final-answer rewards cannot distinguish their underlying quality.Pairwise judging is preferred over isolated scoring because absolute scores are difficult to calibrate and vulnerable to superficial heuristics.
- Trace tournaments: Judge verdicts become soft pairwise outcomes, with ties mapped to 1/2 and wins or losses controlled by γ.The soft-margin parameter satisfies γ ∈(1/2, 1], interpolating between uninformative ties and deterministic wins.
- Tournament calibration: Order randomization and mirrored outcomes enforce antisymmetry, reducing position bias and supporting unbiased win-rate aggregation.Each observed match is augmented with its reversed outcome, satisfying oij + oji = 1.
- Live tournament: Full round-robin judging costs O(N^2), so the live tournament compares each arriving trace with current best, worst, and median anchors, reducing complexity to O(N).The dynamically updated anchors provide informative comparisons while traces arrive asynchronously.
- Bradley-Terry aggregation: Bradley-Terry fitting estimates latent trace strengths from the incomplete comparison graph and normalizes them into scalar rewards.L2 regularization stabilizes estimates on sparse or disconnected graphs, after which fitted strengths are min-max normalized to [0, 1].
5 Experiments
Experiments show that adaptive routing with trace tournaments improves reasoning performance, preserves out-of-domain gains, and increases training and sample efficiency over verifier-only and pointwise judging baselines.
- Performance over training: REASONING ARENA and REASONING ARENA-Live improve steadily during training, while verifier-only RLVR often plateaus or regresses after early training.The methods also preserve gains on GPQA-Diamond and out-of-distribution LiveCodeBench.
- Main results: REASONING ARENA-Live achieves an average score of 53.9, outperforming standard RLVR by 7.6 points and improving AIME 2026 by 12.9 points.Adaptive routing consistently improves across the evaluated tasks.
- Efficiency: With group size N = 8, the live opponent strategy uses at most 18 judge calls versus 28 for a full tournament.The live strategy improves the overall performance average while reducing comparison cost.
- Efficiency: Adaptive routing reduces wall-clock step time by 27% to 41% and saves nearly 50% of generations per step.It targets non-diverse groups, converting otherwise wasted rollouts into tournament signals.
- Adaptive routing: REASONING ARENA improves sample utilization to nearly 100% while providing denser reward signals than tournament-only training.Uniform tournament rewards improve over RLVR, but adaptive routing better preserves exact-verifier reliability on diverse groups.
- Tournaments vs. pointwise scoring: Full trace tournaments improve the average score from 51.3 to 53.5 compared with adaptive pointwise scoring.Pairwise comparisons anchor evaluations across traces sharing identical verifiable outcomes.
6 Conclusion
REASONING ARENA addresses non-diverse reward groups in RLVR by routing them to trace tournaments and aggregating sparse comparisons, improving reasoning-task performance over standard baselines.
- 6 Conclusion: REASONING ARENA routes non-diverse groups to head-to-head trace tournaments instead of discarding their reasoning traces.An LLM judge extracts relative preferences from intermediate reasoning steps.
- 6 Conclusion: Live opponent selection and Bradley-Terry aggregation infer consistent rewards from sparse comparison graphs without full quadratic evaluation.The design supports asynchronous RL training at lower comparison cost.
- 6 Conclusion: REASONING ARENA consistently outperforms standard RLVR and pointwise judge baselines across mathematical and code reasoning tasks.The conclusion attributes this improvement to converting zero-advantage samples into informative gradient updates.
A Limitations and Future Work
The framework is broadly applicable but remains sensitive to data and policy-model conditions, trace length constraints, and the additional resources required to serve judges.
- A Limitations and Future Work: The prevalence of non-diverse reward groups depends on the underlying data mixture and the base capabilities of the policy model.The framework reduces the need for perfectly difficulty-matched datasets but does not remove this dependence.
- A Limitations and Future Work: Complete reasoning trajectories can exceed 160,000 input tokens per pair under asynchronous RL latency and off-policy constraints.The current system therefore relies on concise traces generated within the policy model’s response trunk.
- A Limitations and Future Work: Future work could incorporate truncated reasoning tokens and extend evaluation to tool-use agents with intermediate tool calls and planning steps.These extensions are presented as potential ways to broaden judge-based trace evaluation.
- A Limitations and Future Work: REASONING ARENA accelerates training and sampling while requiring additional GPU resources or API compute for judge models.This creates a trade-off among runtime acceleration, generation efficiency, and total resource allocation.
B Implementation Details
The implementation trains an 8B policy model with online RL, uses large language models as judges, and evaluates across mathematics, graduate-level question answering, and code benchmarks.
- Models and Datasets: Ministral-3-8B-Instruct-2512 serves as the policy model trained with online RL.The models and datasets are released for research use under their respective licenses.
- Models and Datasets: DeepSeekMath-V2 serves as the LLM judge, while Qwen3-235B-A22B and Qwen3.5-122B-A10B support judge-family ablations.The judge models are used for pairwise tournament and pointwise scoring or alternative-backbone studies.
- Models and Datasets: Evaluation covers AIME 2024–2026, BeyondAIME, GPQA-Diamond, and LiveCodeBench.These datasets span competition mathematics, broader mathematics, graduate-level question answering, and code generation.
- Setups: Training uses FP8 quantization, a maximum sequence length of 81,920, learning rate 4e-7, 1500 optimization steps, and group size N = 8.Each minibatch contains 480 RL-sampled rollouts with 50 warm-up steps.
- Setups: REASONING ARENA-Live uses at most 18 judge inferences per non-diverse group with three opponents, compared with 28 inferences for the stated alternative.Later traces compare against three anchors, and the judge uses a 32,768-token context window with the ground-truth answer.
C A Demo of Trace Tournament
The trace-tournament demo compares reasoning quality beyond final correctness, favoring complete derivations among correct answers and structured partial reasoning among incorrect ones.
- All-correct non-diverse reward group: For f(x) = ln(x^2 + 1), the demonstrated solution identifies preimages of 0, 1, and 2 and counts 9 valid domains.The domain must contain 0 and at least one point from each of the two symmetric preimage pairs.
- All-correct non-diverse reward group: Response A is judged better because it explicitly restricts the domain to the relevant preimages, making its correct answer more rigorous than Response B’s.Both responses obtain 9, but only Response A states why extra domain points would introduce unwanted range values.
- Trace tournament mechanism: The demo illustrates that tournaments can distinguish rigorous reasoning from logical gaps and structured attempts from unsupported guesses despite identical verifier outcomes.This supplies intermediate-step feedback where exact verification would provide no distinction within all-correct or all-incorrect groups.
- All-incorrect non-diverse reward group: For the ten-question scoring problem, Response B derives a 2n pattern and answers 20, while Response A gives an unsupported guess of 41.The comparison selects B because it is closer to the correct answer, 27, and provides more reasoning despite remaining incorrect.
D Prompt Template
The prompt templates support pointwise scoring and pairwise comparison of mathematical solution responses. They require evaluators to assess correctness and reasoning, use reference answers where available, and produce constrained final verdicts.
- Prompt Variants: The prompt collection includes tournament, pointwise rubric, and adaptive pointwise components for judging reasoning responses.These named components distinguish pairwise preference evaluation from pointwise scoring procedures.
- Pairwise Comparison: Pairwise evaluation compares two solution attempts using their final answers and reasoning quality to select A, B, or Tie.The prompt supplies separate response slots and requires the final verdict to use exactly one permitted label.
- Evaluation Inputs: Reference-answer fields provide the correct answer for evaluators to use when judging solution responses.The same reference-answer placeholder appears in both pointwise and pairwise prompt components.
- Pointwise Evaluation: Pointwise evaluation asks whether a mathematical response provides a correct answer supported by a valid proof or derivation.The rubric emphasizes rigorous reasoning alongside the final numerical or symbolic answer.
- Pointwise Evaluation: Scores of 1, 0.5, and 0 distinguish complete correctness, minor omissions or errors, and fatal errors or nonresponsive solutions.The highest score requires all steps to be executed properly and clearly; the lowest covers severe omissions that undermine validity.
- Output Constraints: The templates impose strict output formatting, including required opening or concluding phrases and a 500-word analysis limit.One evaluator prompt requires a detailed analysis followed by an exact boxed score, while the pairwise prompt requires an exact boxed verdict.