Source-linked AI summary
Learning to Self-Verify Makes Language Models Better Reasoners
Yuxin Chen, Yu Wang, Yi Zhang, Ziang Ye, Zhengzhou Cai, Yaorui Shi, Qi Gu, Hui Su, Xunliang Cai, Xiang Wang, An Zhang, Tat-Seng Chua
TL;DR
LLMs generate strong reasoning paths but remain weak at verifying their own answers, and generation improvements do not resolve this asymmetry. The paper studies this issue and trains models to self-verify, then integrates generation and verification through multi-task reinforcement learning. Self-verification improves generation performance, while integrated training consistently outperforms generation-only training.
Problem
LLMs often lack the ability to reliably verify their own outputs, and stronger generation ability does not naturally improve self-verification.
Method
The paper trains models to judge the correctness of their own solutions and formulates multi-task reinforcement learning with independent generation and self-verification objectives.
Results
Self-verification improves generation performance, while integrated generation-verification training consistently outperforms generation-only training.
Takeaways & Limitations
Self-verification functions as a useful training signal for improving generation and supporting more efficient reasoning.
Takeaways & Limitations
The experiments remain limited in model scale, combination strategies, and verification-task forms, and additional verification objectives incur computation costs.
Abstract
from arXiv · showhide
Recent large language models (LLMs) achieve strong performance in generating promising reasoning paths for complex tasks. However, despite powerful generation ability, LLMs remain weak at verifying their own answers, revealing a persistent capability asymmetry between generation and self-verification. In this work, we conduct an in-depth investigation of this asymmetry throughout training evolution and show that, even on the same task, improving generation does not lead to corresponding improvements in self-verification. Interestingly, we find that the reverse direction of this asymmetry behaves differently: learning to self-verify can effectively improve generation performance, achieving accuracy comparable to standard generation training while yielding more efficient and effective reasoning traces. Building on this observation, we further explore integrating self-verification into generation training by formulating a multi-task reinforcement learning framework, where generation and self-verification are optimized as two independent but complementary objectives. Extensive experiments across benchmarks and models demonstrate performance gains over generation-only training in both generation and verification capabilities.
1. Introduction
LLMs can generate strong reasoning paths yet remain weak at reliably verifying their own answers. The paper investigates this asymmetry and finds that self-verification can improve generation, motivating complementary multi-task training.
- LLMs often improve on complex reasoning tasks while remaining unable to reliably verify their own outputs.The paper frames this as a persistent asymmetry between generation and self-verification.
- Training solely for self-verification improves generation performance and can achieve comparable results to standard generation training.The resulting models also acquire stronger verification capability.
- Self-verification training reduces the number of tokens needed to solve the same problems and improves majority voting when verification results are incorporated.These findings indicate more efficient reasoning and effective test-time scaling.
- The paper formulates multi-task reinforcement learning that independently optimizes generation and self-verification as complementary objectives.The integrated strategies include self-verification pretraining and alternating generation-verification training.
- Experiments show that integrated training strategies consistently outperform generation-only training.The paper reports gains in both generation and verification capabilities.
- Training for better generation does not correspondingly improve self-verification, even on the same tasks.This pattern is observed throughout training evolution.
2. Preliminary
This section presents RLVR as reinforcement learning with automatically verifiable rewards, then distinguishes generation training from verification training under the same framework. It also introduces GRPO’s group-relative optimization and the self-verification framework’s data construction pipeline.
- RLVR: RLVR trains language models with automatically computable rewards from a rule-based verifier rather than human preference models.The verifier evaluates each output against a reference and returns a scalar reward.
- GRPO: GRPO updates the policy by comparing sampled candidates with group statistics using a clipped surrogate objective and normalized advantages.The method encourages generations performing above the group average and suppresses those with lower relative rewards.
- Self-Verification Framework: The self-verification framework builds training data from on-policy problem-solving trajectories, verifier labels, and post-processing including balancing, filtering, and diversity-aware sampling.The resulting data trains the model to judge the correctness of its own answers.
- Generation Training: In generation training, the model samples candidate solutions and optimizes directly for task-solving correctness against reference answers.The verifier’s reward reflects whether each generated answer matches the reference solution.
- Verification Training: In verification training, the model receives a query and candidate solution, then predicts a binary correctness judgment evaluated against its reference label.The model is optimized to assess provided solutions rather than solve the task itself.
3. Learning to Self-Verify
The paper investigates the asymmetry between generation and self-verification, finding that self-verification training can preserve or improve accuracy while producing more efficient reasoning and stronger verification. It evaluates the approach across mathematical benchmarks and models, then examines verification capability and test-time scaling.
- Self-Verification Framework: Self-verification training uses on-policy generated query–answer samples, rule-based correctness labels, balanced filtering, and a verification-only reward objective.The model is trained to judge whether provided answers are correct, without a generation reward.
- Experimental Setup: The study compares generation-only and self-verification-only training across six mathematical reasoning benchmarks and three Qwen2.5-Instruct models.Evaluation reports Acc@16 accuracy and average reasoning length in tokens.
- Main Results: Self-verification training achieves comparable or better accuracy than generation-only training, including 32.1% on OlympiadBench and 65.6% on Math500 for Qwen2.5-3B-Instruct.These results surpass the generation baseline by 4.7% and 6.0%, respectively.
- Main Results: Self-verification-trained models produce substantially shorter reasoning traces while maintaining comparable performance, using about 25% of generation-training tokens at 7B and roughly 60% at 3B.The paper attributes this efficiency to better error recognition and reduced redundant or fake verification.
- Analysis: Self-verification training strengthens direct verification and error correction, whereas generation training provides only marginal improvement over the base model in corrupted-reasoning evaluation.The verification-trained model substantially outperforms both the base and generation-trained models on error detection and correction.
- Analysis: At test time, verified scores for multiple candidate solutions are combined with majority voting, and this additional signal consistently improves performance.The procedure provides a self-verification-based scaling strategy beyond naive sampling or self-consistency.
4. Integrating Self-Verification into Training
Self-verification can serve as an effective training signal for generation, producing shorter reasoning traces and improving performance when integrated through decoupled objectives.
- Motivation: Self-verification-only training reaches generation performance comparable to pure generation training while producing significantly shorter outputs.These shorter outputs indicate more efficient reasoning traces.
- Framework: The framework formulates generation and self-verification as decoupled but complementary objectives in a multi-task reinforcement learning setup.Both objectives use the same RLVR framework, with training stages distinguished by their reward signals.
- Training Strategies: Stage-wise initialization first trains self-verification, then uses the resulting policy to initialize standard generation training.The verification-trained policy is optimized first with verification reward r_v, followed by generation reward r_g.
- Training Strategies: Alternating training switches between generation and self-verification phases, using preceding generation outputs to construct verification data.A self-verification phase is triggered after every n generation steps.
- Results: Across six benchmarks and three models, integrating self-verification consistently improves generation performance across most evaluated settings.Table 4 reports Acc@16 under four training strategies.
- Results: Verify-Alter raises Qwen2.5-1.5B-Instruct average accuracy from 20.2% to 22.7%, exceeding standard generation and mixed-objective training.On AMC23, accuracy improves by 5.9 points; on AIME, it rises from 0.8% to 4.2%.
5. Related Works
Related work establishes the importance of verification for reasoning and traces the field’s progression from standalone verifiers toward integrating verification signals into generator training.
- LLM Verifiers: Verifiers guide better generations and support test-time scaling through discriminative scores or generative textual judgments.Generative verifiers can produce textual judgments or reward signals.
- LLM Verifiers: Training methods for LLM verifiers have progressed from supervised fine-tuning to preference optimization and reinforcement learning with verifiable rewards.This progression follows advances in reasoning-oriented models.
- Verification in Generator Training: Recent approaches incorporate verification signals into generator training, including imitation learning from external-model correctness signals and joint optimization.Template-based imitation learning can shorten responses but may slightly degrade generation performance.
- Positioning: This work differs by showing that rewarding self-verification alone can produce comparable generation performance with better reasoning traces.It further treats generation and self-verification as decoupled but complementary objectives.
6. Conclusion
The paper identifies an asymmetric relationship between generation and self-verification: improving generation does not naturally improve verification, whereas self-verification training can improve generation.
- Findings: Improving generation does not naturally improve self-verification, even when both abilities are evaluated on the same task.This asymmetry motivates explicit investigation of the two capabilities during training.
- Findings: Learning to self-verify alone can significantly improve generation performance and produce more efficient and effective reasoning traces.The conclusion presents verification as a potentially powerful training signal rather than merely an auxiliary component.
- Future Work: Future work is needed on designed verification tasks, principled objective integration, and more efficient training.These directions are explicitly left beyond the scope of the current work.
Impact Statement
The authors argue that strengthening self-verification can change how language models reason and generate responses, including improving generation efficiency.
- Implications: Strengthening self-verification can improve both model reliability and generation efficiency.The impact statement connects these effects to the interaction among reasoning, verification, and generation.
Limitation
The framework has added computational costs and remains limited in model scale, verification formulations, task diversity, and domain coverage.
- Adding self-verification to generation training incurs extra inference and optimization costs.
- Experiments cover different parameter sizes but do not establish whether the findings extend to larger models.
- The study explores limited combinations of generation and self-verification, using only one form of verification task.
- The evidence focuses primarily on mathematical reasoning benchmarks, leaving generalization to other domains open.