Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning

• Existing RL approaches rely heavily on human-annotated labels or pretrained reward models, which are costly or may suffer from bias and distribution mismatch.Source: §1 · p. 1
• Prior self-rewarding methods reward final answers while overlooking intermediate states, leaving them vulnerable to reward hacking through superficial shortcuts.The passage gives always answering “0” as an example of a shortcut that can fool majority voting.Source: §1 · pp. 1–2
• CoVo treats reasoning trajectories as sequences of intermediate states and compares their model-likelihood distances to candidate final answers.Source: §1 · p. 2
• Correct responses tend to show higher consistency and lower volatility: states converge toward their own answer with little deviation toward alternatives.Source: §1 · p. 2, Figure 1 · p. 2
• CoVo combines consistency and volatility as intrinsic rewards, uses robust vector-space aggregation, and adds a curiosity bonus for diverse exploration.The aggregation is described as reducing sensitivity to outlier trajectories.Source: §1 · p. 2
• CoVo performs comparably to or sometimes surpasses supervised RL with verifiable answers across math-specific and general-purpose benchmarks.Source: §1 · p. 2

• A trajectory consists of discrete steps and a final answer; each intermediate state combines the prompt with the steps generated before it.Source: §2 · p. 3
• The distance matrix records each trajectory state's distance to each distinct sampled final answer, with its first column representing distance to that trajectory's own answer.The matrix supports pattern analysis, and prompts with identical sampled answers are excluded because they typically lack meaningful learning signals.Source: §2 · p. 3
• The RL formulation updates a policy using a reward function and policy-gradient methods; Reinforce++ is the paper's backbone algorithm.Its loss combines clipped updates with normalized advantage values.Source: §2 · p. 3

3.1 Empirical Analysis: Trajectory Patterns of Different Samples

• Correct trajectories approach and consistently favor their final answer early, while incorrect ones fluctuate and converge later; normalized distance below 1 means a state is closest to its own answer.The empirical analysis uses MATH and MMLU and compares trajectories across varying numbers of intermediate states.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2 Trajectory Features: Consistency and Volatility

• Consistency measures the share of intermediate states favoring their own answer, while volatility tracks the last deviation from that answer as a fraction of trajectory states.Higher consistency and lower volatility characterize correct trajectories relative to incorrect ones.Source: §3.2 · pp. 4–5, §3.3 · p. 5

3.3.1 Intrinsic Reward for Self-Evaluation

• CoVo groups sampled trajectories by final answer for relative rewards, then geometrically aggregates consistency and volatility to reduce sensitivity to outliers.The vector reward preserves the features’ interplay and their monotonic behavior as consistency and volatility vary.Source: §3.3.1 · p. 5

3.3.2 Curiosity Reward for Self-Exploration

• A curiosity reward encourages exploration of uncertain reasoning transitions, with a KL-divergence penalty limiting dominance by very low-probability tokens; total reward adds curiosity to the intrinsic reward.The motivation is that declining model diversity can make comparisons across distinct answers difficult.Source: §3.3.2 · pp. 5–6

4.1 Environmental Setups

• The evaluation uses Pass@1 accuracy at temperature 0 across benchmark datasets, with correctness checked against ground truth using Math-Verify.The experiments cover three models under 10B parameters and compare against prompting, supervised RL, and self-rewarding RL baselines.Source: §4.1 · p. 6

4.2 Main Results

• 51.2% accuracy on MATH benchmarks for CoVo with Llama3.2-3B-Instruct is on par with GRPO (51.8%), while Qwen2.5-3B-Instruct improves from 65.4% to 68.2% on MATH and outperforms RLOO and GRPO.Across most cases, CoVo is on par with or exceeds supervised RL baselines, including with Qwen2.5-7B-Instruct.Source: §4.2 · pp. 6–7, Table 1 · p. 7
• Despite training only on mathematical reasoning data, CoVo shows comparable performance gains across commonsense and science benchmarks relative to other strong baselines.The authors describe this as transfer to unseen general problem sets.Source: §4.2 · p. 7
• CoVo’s reasoning paths appear more dispersed than GRPO’s concentrated clusters, consistent with the method’s aim to encourage diverse exploration.The visualization encodes responses with Sentence Transformers and reduces them to two dimensions using UMAP.Source: Figure 3 · p. 7, §4.2 · p. 8

4.3 Analysis

• Correct and incorrect responses show clear differences in consistency and volatility across four reasoning datasets, supporting these features as intrinsic indicators of trajectory correctness.The section reports statistical data in Table 2 and visualizes the distributions in Figure 4.Source: §4.3 · p. 8, Table 2 · p. 8, Figure 4 · p. 8
• CoVo’s reward accuracy remains consistently high and its training is more stable against potential reward hacking than majority-voting-based reward signals.The comparison includes TTRL and EMPO, which use majority voting and answer probabilities for reward assignment.Source: §4.3 · p. 8, Figure 5 · p. 8

4.4 Ablation Study

• Across almost all reasoning tasks, combining intrinsic and curiosity rewards outperforms either signal alone.The ablation compares five reward configurations across reasoning domains.Source: §4.4 · p. 9, Table 3 · p. 9
• Vectorial aggregation of the intrinsic reward yields superior accuracy on mathematical and scientific reasoning tasks and is described as less susceptible to sample noise.The authors connect this advantage to tasks requiring substantial arithmetic and analytical reasoning.Source: §4.4 · p. 9

5.1 Rethinking Majority Voting in RL Optimization

• If the most likely response is wrong, majority-vote RL converges toward it and causes reward hacking as training proceeds.The analysis attributes this to optimizing toward the modal answer, which can diverge from the true answer distribution when model priors are misaligned.Source: §5.1 · p. 9

5.2 Theoretical Foundations of Self-Rewarding Reinforcement Learning

• CoVo’s self-rewarding objective can be viewed as approximate variational inference over latent reasoning trajectories, grounding rewards in intermediate states and final answers.The stated bound relates log answer likelihood to expected reward assignment and KL regularization.Source: §5.2 · p. 10

5.3 Complexity and Convergence of CoVo

• Computing trajectory distance matrices costs O(NTK) per batch for N paths, T reasoning steps, and K distinct answer candidates.Each trajectory’s T-by-K matrix requires O(TK) computations.Source: §5.3 · p. 10
• Under the stated ranking, misclassification, and KL assumptions, CoVo has a convergence-time upper bound, and trajectories with higher initial likelihood converge faster.The passage relates this inverse dependence on initial likelihood to RL amplifying existing reasoning capabilities rather than instilling entirely new behaviors.Source: §5.3 · p. 10

RL for Reasoning

• Earlier RL work used PPO to align models with human preferences, while later reasoning-focused methods developed reward models and RL algorithms; external labels and reward models constrain scalability.The passage contrasts these approaches with deriving reward signals from the policy itself.Source: §6 · p. 10

Self-Rewarding LLM

• Self-rewarding methods include model-as-judge prompting [54] [48] [50], ensembles of internal reward models [46] [63], and intermediate-step rewards [62] [64].Prompt-generated criteria and rewards can be resource-intensive, so these approaches are typically used in offline iterative DPO [30].Source: §6 · pp. 10–11
• TTRL [68] and EMPO [61] compute online rewards from sampled final answers, whereas CoVo uses reasoning-trajectory patterns to seek more robust reward signals.The passage notes that relying solely on final-answer distributions can lead to reward hacking.Source: §6 · p. 11

7 Conclusion

• CoVo performs on par with or better than rule-based RL methods and shows stability compared with other self-rewarding methods.The conclusion describes its intrinsic trajectory reward and curiosity reward as supporting diverse exploration without external signals.Source: §7 · p. 11
• CoVo’s generalizability to visual language models and effectiveness in semi-supervised settings with limited labels remain to be explored.The authors identify both as future research directions.Source: §7 · p. 11

A.1 Environmental Setups

• Evaluation uses Pass@1 accuracy at temperature 0 and assesses reasoning diversity with ACS, Entropy, and Self-BLEU.The appendix also lists evaluation prompts and training hyperparameters in Tables 4 and 5, and reports experiments on 8×A100-80GB GPUs.Source: §A.1 · p. 17, Table 4 · p. 17, Table 5 · p. 17

A.2 Additional Results

• CoVo’s reasoning-path diversity exceeds that of rule-based RL methods and self-rewarding baselines across the reported metrics.The comparison is summarized in Table 6.Source: §A.2 · p. 17, Table 6 · p. 18

A.3 Discussion

• Reward calculation occurs during prefilling and is significantly faster than decoding, making it more efficient than self-rewarding methods that generate criteria and final rewards.The computational timing analysis is reported for a single trajectory; Table 7 covers computational resources.Source: §A.2 · pp. 17–18, Table 7 · p. 18

Q1: Why our methods have the potential to outperform rule-based RL?

• Rule-based RL rewards final-answer correctness without assessing reasoning quality, so incorrect paths that happen to reach correct answers may receive high rewards.CoVo instead evaluates correctness using intermediate reasoning states and emphasizes reasoning-process quality.Source: Q1: Why our methods have the potential… · p. 18

Q2: How does the method handle universally incorrect samples during RL?

• Completely wrong answers may receive zero advantage under CoVo when their intermediate states have low consistency, whereas majority voting can amplify dominant errors.Source: Q2: How does the method handle… · p. 18

B.1 Technical Setting and Notations

• The theoretical analysis simplifies optimization by using tabular policies and assuming a relatively small, non-dominant KL coefficient.It also uses a gradient-flow approximation with a relatively small learning rate to analyze optimization dynamics.Source: §B.1 · pp. 18–19

B.2 Useful Lemmas

• Under gradient flow, the appendix bounds output-probability growth by πθ(T)(y|x) ≤ πθ(0)(y|x) · exp(2T).Source: §B.2 · pp. 19–20

B.3.1 Monotonicity Analysis

• The intrinsic rewards are monotonic in the intended direction: higher consistency and lower volatility indicate a higher reward.The analysis compares linear and vector-space reward formulations across consistency and volatility increments.Source: §B.3 · p. 20, §B.3.1 · pp. 20–21

B.3.2 Robustness Analysis

• The vector-space intrinsic reward is more robust than the linear reward when consistency values contain abnormally large outliers.The appendix connects such outliers among incorrect trajectories with spurious reward assignments.Source: §B.3.2 · pp. 21, 23

B.4 Proof of Proposition 1

• Under the stated reward, the most frequently sampled answer’s probability converges to 1, so an incorrect sample-mode causes reward hacking.The policy can maximize reward by repeatedly producing that answer rather than the factually correct response.Source: §B.4 · pp. 23–24
• The proof’s reward selects the answer with the highest sample count, which is the condition behind the collapse result.It defines C(y) as the count of sampled responses equal to y.Source: §B.4 · p. 23

B.5 Proof of Proposition 2

• Optimizing the variational lower bound assigns rewards to latent reasoning states while regularizing their distribution toward a prior.The paper interprets this self-rewarding objective as approximate variational inference coupling answer generation and reasoning-trajectory inference.Source: §B.5 · p. 24

B.6 Proof of Proposition 3

• Under the stated assumptions, CoVo’s higher-consistency, lower-volatility outputs can reach a probability threshold that yields an expected real-reward increase of γ.The proof assumes the reward ranks yγ highest, has low misclassification probability for incorrect trajectories, and uses a sufficiently small KL coefficient.Source: §B.6 · pp. 25, 27

D.1 Training

• Open-Reasoner-Zero training data includes public math datasets and additional reasoning tasks synthesized programmatically.The listed sources include AIME through 2023, MATH, Numina-Math, Tulu3 MATH, and OpenR1-Math-220k.Source: §D.1 · p. 27
• The intrinsic-reward algorithm accumulates consistency and volatility across trajectory states, then aggregates them as vectors using cosine and sine components.The algorithm’s input is a group of trajectories sharing the same answer and their distance matrices.Source: §D.1 · p. 28
• The curiosity-reward algorithm processes successive intermediate reasoning states using token log probabilities and a KL-based term.It averages the accumulated curiosity reward over the number of intermediate states.Source: §D.1 · p. 28

D.2 Evaluation

• The mathematical evaluation set includes high-school and elementary problems, math competitions, and international Olympiad problems.It comprises MATH-500, GSM8K, AMC-233, and Olympiad Bench.Source: §D.2 · p. 28
• Commonsense evaluation uses MMLU-Pro’s broad subject coverage and CommonsenseQA’s everyday and social questions.MMLU-Pro is described as assessing knowledge and reasoning across fields; CommonsenseQA targets commonsense understanding.Source: §D.2 · p. 28
• GPQA evaluates scientific knowledge and reasoning through difficult multiple-choice questions written and validated by experts.Its questions span biology, physics, and chemistry.Source: §D.2 · p. 29
• The seven evaluation datasets cover mathematical, commonsense, and scientific reasoning domains.The passage lists four mathematical datasets, two commonsense datasets, and one science dataset.Source: §D.2 · pp. 28–29, Table 8 · p. 29

E Case study

• The midpoint-based solution gives x = 6, which violates the problem’s x > 7 condition and requires re-evaluation.The revised approach uses the parallelogram property that the vector from A to C equals the vector from B to D.Source: E Case study · p. 29
• The revised case-study approach uses the parallelogram vector property to determine the fourth vertex.The passage states that the vector from B to D is also (2, 1).Source: E Case study · pp. 29–30

F Broader Impact

• CoVo is presented as a potential aid in limited-label settings, with possible applications including scientific education, programming, and healthcare.The authors say its intrinsic reward may contribute to more coherent reasoning and improved interpretability and robustness.Source: F Broader Impact · p. 30