Source-linked AI summary
ConsensusBench: Benchmark of Consensus Nodes for LLM Reasoning via Outcome Reward Densifying
Shi-Qi Yan, Chao-Hong Tan, Qian Chen, Wen Wang, Xiangang Li, Zhen-Hua Ling
TL;DR
Long-chain reasoning exposes a limitation of outcome-only reinforcement learning: final-answer rewards provide sparse supervision about intermediate progress. The paper introduces ConsensusBench and ConsensusPR, which derive verifiable process signals from clustered intermediate conclusions, and reports consistent gains across reasoning benchmarks while noting limitations in current formulation and data construction.
Problem
Outcome-only RLVR provides sparse supervision for intermediate reasoning in long chains, while PRM-style alternatives require costly annotations and subjective step-quality judgments.
Method
ConsensusBench filters correct trajectories from N rollouts, clusters semantically equivalent intermediate conclusions into consensus nodes, and integrates their rule-based rewards into GRPO-style training through ConsensusPR.
Results
ConsensusPR consistently improves performance over vanilla GRPO and DAPO across benchmarks, including MATH-500 accuracy rising from 58.4% to 66.6% with DAPO on Qwen3-4B-Base.
Takeaways & Limitations
Consensus nodes provide verifiable intermediate targets and the benchmark evaluates reasoning with Final Answer Accuracy (Acc), Node Coverage Rate (NCR), and Tokens per Node (TPN).
Takeaways & Limitations
The framework does not fully exploit consensus nodes, does not use a standalone process-reward algorithm, and may inherit bias from closed-source models used for data construction.
Abstract
from arXiv · showhide
Reinforcement learning (RL) has become one of the primary paradigms for reasoning enhancement of large language models (LLMs). In particular, Group Relative Policy Optimization (GRPO) and related algorithms have demonstrated strong performance with outcome-level rewards. However, these methods depend solely on the final answer, without feedback regarding which intermediate steps contribute to success or failure. As task complexity and reasoning trajectory length increase, such sparse final-answer rewards become increasingly insufficient. To address this limitation, we introduce ConsensusBench, a novel dataset designed to provide rule-based process-level signals. We posit that a correct final answer relies on a small set of intermediate conclusions throughout the reasoning process, which can be seen as a verifiable sub-outcome. We identify these sub-outcomes by filtering correct trajectories from N rollouts and clustering semantically equivalent intermediate statements. We call these clustered statements as Consensus Nodes. By integrating a rule-based process reward derived from these nodes into GRPO-style algorithms, we develop a new reinforcement learning signal named ConsensusPR. It directly reduces the reward sparsity of outcome reward across long reasoning trajectories. To facilitate systematic process-level evaluation, we introduce three metrics to our benchmark: Final Answer Accuracy (Acc), Node Coverage Rate (NCR), and Tokens per Node (TPN). Experiments across AIME 2024, AIME 2025, GSM8K, MATH-500, and our ConsensusBench demonstrate that the proposed method consistently surpasses GRPO-style approaches, highlighting the practical value of consensus nodes in guiding reasoning.
1 Introduction
Outcome-only rewards are scalable and verifiable, but long reasoning chains expose severe credit-assignment mismatches. ConsensusBench and ConsensusPR address this gap with verifiable intermediate consensus nodes and dense process-level signals.
- Motivation: Outcome rewards provide scalable, low-noise supervision based on final-answer correctness, but give little information about intermediate progress in long reasoning chains.A single scalar reward distributes credit uniformly across the trajectory.
- Motivation: Reward mismatch includes nearly correct trajectories, format errors, and other cases where final-answer scoring misrepresents reasoning quality.The paper frames these mismatches through a four-type case study using correct and incorrect reasoning spans.
- Prior limitations: Process Reward Models mitigate credit assignment but require costly annotation and training, rely on subjective quality judgments, and remain vulnerable to distribution shift and reward hacking.PRMs score intermediate steps rather than only whether the trajectory reaches a correct destination.
- Proposed approach: ConsensusBench identifies verifiable intermediate outcomes by retaining correct rollouts and clustering semantically equivalent conclusions across N sampled trajectories.These consensus nodes represent checkpoints such as resolved subproblems, verified constraints, partitioned cases, or applied lemmas.
- Proposed approach: ConsensusPR integrates node-derived process rewards into existing GRPO and RLVR pipelines while retaining fine-grained signals, efficiency, and generalization.The paper presents the benchmark and reward as balancing outcome-only and process-based supervision.
2 Related Work
Prior work centers on two complementary approaches: trajectory-level outcome rewards for efficient optimization and process-level scorers for evaluating intermediate reasoning steps.
- Outcome-based reinforcement learning: GRPO, DAPO, and GSPO use a single binary final-answer signal to train reasoning models efficiently on mathematics and code.These methods are contrasted with critic-based approaches such as PPO.
- Process reward models and benchmarks: Process Reward Models score individual reasoning steps and can support verifier-guided decoding or step-level policy optimization.PRM800K provides human-annotated mathematical reasoning labels for training learned process scorers.
3 CONSENSUSBENCH
CONSENSUSBENCH represents correct reasoning through consensus nodes: verifiable intermediate conclusions frequently reached across valid rollouts. It constructs these nodes from multiple correct solutions and evaluates both final answers and intermediate reasoning behavior.
- Design Principle: CONSENSUSBENCH uses frequently reached intermediate conclusions from valid solutions as verifiable process-reward targets.The benchmark avoids requiring a single canonical reasoning path.
- Data Construction: The construction pipeline samples correct rollouts, clusters semantically equivalent conclusions, and refines nodes for verifiability.Refinement removes trivial restatements while prioritizing nodes containing numbers and formulas.
- Dataset Statistics: CONSENSUSBENCH contains 5,577 problems averaging 6.80 consensus nodes per problem.Its test split has 1,707 samples averaging 4.88 nodes, while the training split has 3,870 samples averaging 7.64 nodes.
- Evaluation Metrics: The benchmark reports Final Answer Accuracy (Acc), Node Coverage Rate (NCR), and Tokens per Node (TPN) to assess outcomes and reasoning behavior.NCR measures attained annotated nodes, while TPN measures the average tokens needed to reach the next node.
- Evaluation Metrics: TPN complements Acc and NCR by measuring reasoning efficiency rather than serving as a direct quality score.Very low TPN indicates efficient reasoning, whereas very high TPN may indicate invalid reasoning or unnecessary trials.
4 ConsensusPR
ConsensusPR converts consensus nodes into fine-grained process rewards while retaining alignment with outcome rewards. It integrates these signals into GRPO-style training through several reward aggregation strategies and curriculum learning.
- Motivation: Consensus nodes provide temporally densified process rewards aligned with the outcome reward and integrated into GRPO-style training.The approach is intended to preserve lightweight computation while supplying intermediate feedback.
- Integration: The method adds node-level signals directly to the standard GRPO scalar reward without staged scheduling or complex training mechanisms.This minimal coupling preserves the original reward structure.
- Process Reward: ConsensusPR evaluates whether rollout text matches each consensus node using rule-based normalization and soft matching.The resulting Node Coverage Ratio (NCR) serves directly as the process-reward score.
- Reward Formulations: Linear integration combines Acc and NCR with weight α, while outcome-priority integration gives maximum score to correct final answers and otherwise falls back to node matching.The linear formulation treats the final answer as a higher-weight special node.
- Reward Formulations: Curriculum learning first trains exclusively on node-matching rewards before switching after the first epoch toward process and final-answer reasoning.The first stage emphasizes intermediate reasoning quality without requiring a final answer.
5 Experiments
Experiments evaluate ConsensusPR across multiple model scales, reasoning benchmarks, process metrics, task difficulties, and reward configurations. ConsensusPR generally improves challenging-task accuracy and process coverage, with gains increasing for harder and longer reasoning tasks.
- Experimental Setup: ConsensusPR is evaluated with Qwen3-1.7B, Qwen3-4B, and Qwen3-8B models in GRPO and DAPO, replacing only the reward computation module.The optimization pipeline remains otherwise unmodified.
- Main Results: ConsensusPR consistently improves over vanilla GRPO and DAPO; with Qwen3-4B and DAPO, ConsensusPRCL raises MATH-500 accuracy from 58.4% to 66.6%.On Qwen3-8B, the same configuration reaches 91.2% on GSM8K, 77.6% on MATH-500, and 89.7% on CONSENSUSBENCH accuracy.
- Main Results: Reward aggregation creates different optimization biases: ConsensusPRCL favors challenging-benchmark accuracy, while linear and max variants often achieve higher node coverage.With Qwen3-4B and DAPO, ConsensusPRlinear reaches the highest NCR at 78.4%, while trailing ConsensusPRCL by 2.3% on CONSENSUSBENCH accuracy.
- Difficulty Analysis: ConsensusPR gains grow with task difficulty: improvements are marginal on GSM8K but substantial on MATH and AIME as reasoning trajectories become longer.The difficulty analysis divides MATH-500 into easy, medium, and hard groups and supports the same pattern.
- Scaling Analysis: Scaling from Qwen3-4B to Qwen3-8B increases accuracy by 11.0% on MATH-500 and 2.5% on CONSENSUSBENCH under identical training configurations.The 8B model nevertheless performs worse than the 4B model on GSM8K, which the authors attribute to constrained generalization at this scale.
- Process Metrics: Consensus-based rewards increase reasoning length relative to vanilla GRPO and DAPO, while reinforcing stepwise reasoning through node-aware feedback.Table 4 reports Tokens per Node across models; the baseline methods generally produce the shortest reasoning lengths.
- Efficiency Analysis: ConsensusBench performance is relatively insensitive to α when α avoids the degenerate extremes of 0.0 and 1.0.The analysis also observes task-complexity-dependent differences in the balance between process and outcome rewards.
6 Conclusion
The paper argues that outcome-only RLVR is insufficient for long-chain reasoning and introduces consensus nodes as verifiable intermediate conclusions. CONSENSUSBENCH collects these nodes by clustering agreement among correct rollouts to support process rewards.
- Motivation: Outcome-only RLVR provides sparse supervision for long-chain reasoning because a binary final-answer signal offers limited fine-grained feedback.The conclusion frames this limitation as the motivation for consensus nodes.
- Contribution: Consensus nodes are intermediate conclusions frequently reached by correct trajectories and are collected by clustering reasoning agreement across N rollouts.CONSENSUSBENCH annotates these verifiable sub-outcomes for process reward.
Limitations
The benchmark introduces consensus-node evaluation but acknowledges that the current framework does not fully exploit these nodes and may inherit bias from closed-source models used in construction.
- The current framework does not devise a standalone process-reward algorithm, instead integrating process rewards directly into the outcome signal.
- Closed-source models used for data construction create a potential bias risk in node extraction.
- CONSENSUSBENCH differs from single-path process benchmarks by annotating high-consensus sub-outcomes reached by diverse correct rollouts.
C Experiments Details
The experiments use a standardized step-by-step question-answering prompt that requires the final answer to appear on a separate line after an explicit Answer marker.
- C.1 Prompt: The prompt instructs models to solve each math problem step by step.
- C.1 Prompt: The final response must end with an Answer line containing the problem’s answer.
C.2 Training Details
Training uses consistent GRPO and DAPO configurations, with selected length and batch parameters adjusted for memory constraints and rule-based normalization applied to mathematical expressions.
- C.2 Training Details: GRPO and DAPO experiments use train_batch_size=256 with maximum prompt and response lengths specified for DAPO.
- C.2 Training Details: The study retains default verl configurations for most parameters and adjusts response length and batch size to prevent out-of-memory errors.
- C.2 Training Details: Table 6 reports pass@1, pass@16, NCR, and TPN results across reasoning benchmarks for Qwen3-1.7B-Base.
- C.2 Training Details: Rule-based normalization extracts mathematical expressions and standardizes notation, fractions, multiplication symbols, and variable names.
D.1 Performance of Main LLMs
Existing models achieve high final-answer accuracy on CONSENSUSBENCH, but their node coverage and tokens-per-node metrics vary substantially, revealing differences in process quality and reasoning strategy.
- D.1 Performance of Main LLMs: 93.3% to 95.1% final-answer accuracy contrasts with NCR values ranging from 67.9% to 86.3% across existing models.
- D.1 Performance of Main LLMs: Variation in TPN scores indicates that models use different reasoning strategies.
- D.1 Performance of Main LLMs: Human annotation validation includes annotation consistency and recall rate metrics.
D.2 Human Annotation Validation
The human evaluation targets the extraction of key consensus nodes from open-ended mathematical reasoning, where generic agreement measures are limited. Automatic extraction reached 62.2% agreement with human annotations and 90.5% recall, with most discrepancies attributed to coarser human annotation.
- Annotation setting: Consensus-node annotations require task-specific evaluation because the steps are open-ended, free-form reasoning statements unsuitable for generic agreement measures.The annotation setting involves extracting nodes from multiple lengthy mathematical chains of thought.
- Human validation: 62.2% agreement with human annotations and 90.5% recall indicate that automatic extraction captured most human-identified nodes.The reported discrepancy pattern is attributed mainly to humans omitting finer-grained annotations rather than to incorrect extraction results.
- Scaling study: Performance gains were less pronounced for Qwen3-1.7B than for 4B and 8B models in the scaling study.The authors attribute this disparity to limited emergent capabilities and difficulty jointly improving process reward and result reward.