Source-linked AI summary
RAVR: Reference-Answer-guided Variational Reasoning for Large Language Models
Tianqianjin Lin, Xi Zhao, Xingyao Zhang, Rujiao Long, Yi Xu, Zhuoren Jiang, Wenbo Su, Bo Zheng
TL;DR
RL can have difficulty sampling useful reasoning paths on tasks beyond a model’s competence. RAVR uses answer-conditioned reasoning as a variational surrogate for question-only reasoning, and experiments report improvements across general and math reasoning benchmarks.
Problem
RL depends on sampling useful completions with non-negligible probability, but tasks beyond a model’s competence can make high-utility paths difficult to obtain.
Source: §1 · p. 1Method
RAVR uses reasoning generated with a reference answer as a variational surrogate for question-only reasoning, optimizing utility while reducing the distributions’ KL divergence.
Source: §1 · p. 2Results
Across general and math reasoning benchmarks, RAVR outperforms state-of-the-art methods; Qwen3-1.7B trained on CrossThink-QA scores 40.91 on GPQA-Diamond, 5.56 points above DAPO.
Source: §1 · p. 2Takeaways & Limitations
RAVR’s analysis finds reduced hesitation, stronger conclusion consolidation, and more problem-specific reasoning strategies.
Source: §1 · p. 2The authors identify open-ended tasks as future work, where reference answers offer richer information and high-quality reasoning is typically harder to obtain from the model’s inherent capabilities alone.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Reinforcement learning (RL) can refine the reasoning abilities of large language models (LLMs), but critically depends on a key prerequisite: the LLM can already generate high-utility reasoning paths with non-negligible probability. For tasks beyond the LLM's current competence, such reasoning path can be hard to sample, and learning risks reinforcing familiar but suboptimal reasoning. We are motivated by the insight from cognitive science that Why is this the answer is often an easier question than What is the answer, as it avoids the heavy cognitive load of open-ended exploration, opting instead for explanatory reconstruction-systematically retracing the reasoning that links a question to its answer. We show that LLMs can similarly leverage answers to derive high-quality reasoning paths. We formalize this phenomenon and prove that conditioning on answer provably increases the expected utility of sampled reasoning paths, thereby transforming intractable problems into learnable ones. Building on this insight, we introduce RAVR (Reference-Answer-guided Variational Reasoning), an end-to-end framework that uses answer-conditioned reasoning as a variational surrogate for question-only reasoning. Experiments in both general and math domains demonstrate consistent improvements over strong baselines. We further analyze the reasoning behavior and find that RAVR reduces hesitation, strengthens conclusion consolidation, and promotes problem-specific strategies in reasoning.
1 Introduction
RL can struggle when useful reasoning paths are hard to sample, risking reinforcement of familiar but suboptimal completions. RAVR uses reference-answer-guided reasoning as a variational surrogate for question-only reasoning, with reported gains across general and math benchmarks.
1 Introduction
- RL optimization depends on sampling useful completions with non-negligible probability; beyond the model’s competence, training can reinforce familiar but suboptimal paths.Relative-advantage methods can reinforce weak completions when they outperform the other sampled completions.Source: §1 · p. 1
- Conditioning on the reference answer increases the likelihood of sound reasoning and decreases the likelihood of flawed reasoning, improving expected reasoning quality over sampled paths.The paper formalizes this effect and provides a proof.Source: §1 · p. 2
- RAVR treats answer-conditioned reasoning as a variational surrogate for question-only reasoning, optimizing expected reasoning utility while reducing the distributions’ KL divergence.The framework uses first-person think-aloud prompting, a prior-based utility baseline, and utility-weighted KL samples to align training with question-only inference.Source: §1 · p. 2
- On over 50% of 50 hard CrossThink-QA questions missed across eight attempts, Qwen3-1.7B generated logically coherent reasoning after receiving the answer.A GPT-5 judge assessed coherence without being told that the model had access to the answer.Source: Figure 1 · p. 2, §1 · p. 2
- Across general and math reasoning benchmarks, RAVR outperforms state-of-the-art methods; Qwen3-1.7B trained on CrossThink-QA scores 40.91 on GPQA-Diamond, 5.56 points above DAPO.The authors also report reduced hesitation, stronger conclusion consolidation, and more problem-specific reasoning strategies.Source: §1 · p. 2
2 RAVR: Reference-Answer guided Variational Reasoning
RAVR addresses the difficulty of sampling useful reasoning paths by using reference-answer-conditioned reasoning to guide learning of a question-only model. Its objective trains this posterior and prior together while using their divergence to regularize both.
2.1 Preliminaries and the Sampling Challenge
- RL can reinforce familiar but suboptimal completions when good ones are rarely sampled, and binary correctness yields no optimization signal if every sampled completion is incorrect.Group-relative reward normalization can give a weak completion positive advantage when it outperforms others.Source: §2.1 · p. 3
2.2 Motivation: Conditioning on reference answer amplifies good reasoning
- Conditioning on the reference answer reweights question-only reasoning paths by answer likelihood, increasing the sampling probability of above-average paths and raising expected utility.For a path set, posterior sampling is at least as likely as prior sampling when its mean utility is at least the overall mean.Source: §2.2 · p. 4
2.3 Reference-Answer-Conditioned Variational Optimization Objective
- RAVR treats answer-conditioned reasoning as an amortized posterior and variational surrogate for question-only reasoning, deriving an ELBO with an answer-likelihood term and a KL penalty.The likelihood term encourages paths that make the reference answer more likely; the KL term pulls the prior toward the posterior and regularizes the posterior against out-of-distribution paths.Source: §2.3 · p. 5
- A utility baseline compares posterior-path answer likelihood with its expectation under the question-only prior, while clipping negative rewards and length-normalizing likelihood stabilize and calibrate rewards.The method also weights KL samples by reward so learning can focus on paths with higher utility than the model's own.Source: §2.3 · p. 5
- RAVR realizes question-only and answer-conditioned distributions in one LLM by adjusting prompts, asking the posterior to derive a step-by-step explanation without revealing prior access to the answer.GRPO optimizes the first term, and the variational objective is jointly optimized with the raw objective.Source: §2.3 · pp. 5–6
3 Experiments
Experiments evaluate RAVR on general and math benchmarks, alongside analyses of reasoning behavior, learning dynamics, and component ablations. The reported results include strong cross-domain performance and improved sampling efficiency relative to larger GRPO rollout groups.
3.1 Performance of RAVR
- 44.75 overall average and 42.92 out-of-domain math score were reported for RAVR trained on CrossThink-QA; DeepMath training yielded a 45.02 overall average and 45.05 on out-of-domain general tasks.Table 1 averages GPQA-Diamond and MMLU-Pro for general tasks, and AIME24, AIME25, AMC23, and Minerva for math reasoning.Source: Table 1 · p. 6, §3.1 · p. 7
3.2 Reasoning Behavior of RAVR
- RAVR and GRPO are compared through discourse-marker frequencies as proxies for cognitive moves; Figure 3 plots frequent thinking words by average frequency per response.The supplied passage mentions fewer hesitation markers for RAVR but does not provide the remainder of the reported behavior findings.Source: §3.2 · p. 7, Figure 3 · p. 7
3.3 Learning Dynamics of RAVR
- With rollout group size 8, RAVR matched or exceeded GRPO at size 24, while its smoother learning curve indicated improved learning stability.The comparison covers GPQA-Diamond and MMLU-Pro; the authors interpret it as evidence of more efficient sampling of high-quality reasoning paths.Source: §3.3 · p. 7, Figure 4 · p. 8
- The posterior–prior KL divergence gradually decreased as prior reasoning utility improved, while the posterior retained a stable utility gain.The authors interpret the trend as transfer of high-utility reasoning to question-only inference and continued learning.Source: §3.3 · p. 7, Figure 5 · p. 8
3.4 Ablation Study of RAVR
- Ablations identify the utility objective, prior-based utility baseline, posterior instructions, answer prefix, and question-only prior objective as important design choices.Removing the utility objective made RAVR underperform GRPO across both datasets; removing KL reweighting raised variance, while the answer prefix stabilized early training.Source: §3.4 · p. 8
4 Related works
Related work uses reference answers to obtain reasoning traces, but the cited approaches rely on staged training or teacher-generated explanations. RAVR instead uses answers within an end-to-end framework that trains the model’s own reasoning.
Related approaches using answers
- STaR and its RL extension use reference answers to generate reasoning samples, but require interlaced generation and fine-tuning stages and judge traces by answer correctness.STaR first attempts problems unaided, then supplies the ground truth after failure; Zhou et al. (2025a) extends it to preference optimization.Source: §4 · pp. 8–9
Teacher-student distillation
- Teacher-student distillation produces detailed explanations from complete step-by-step solutions, but does not use the teacher LLM’s reasoning and requires a two-model, two-phase setup.The cited work is Cetin et al. (2025).Source: §4 · p. 9
RAVR in relation to prior work
- RAVR incorporates reference answers into end-to-end training of the LLM’s own problem-solving, and even a single word or phrase can suffice as the answer.This contrasts with approaches requiring a complete solution as input to generate explanations.Source: §4 · p. 9
5 Conclusion
The conclusion presents RAVR as an end-to-end variational reinforcement-learning framework for using reference answers to guide an LLM’s reasoning. It highlights training-stability strategies and identifies open-ended tasks as future work.
Conclusion
- RAVR uses reference answers to guide an LLM to reason about why an answer follows, supporting its learning of how to answer.The conclusion describes this as a novel variational reinforcement-learning objective.Source: §5 · p. 9
- The proposed stability strategies include a reasoning reward baseline, reward-based sample weighting in KL calculation, and an answer prefix.The conclusion characterizes these strategies as simple but effective for training stability and performance.Source: §5 · p. 9
- Future work will explore open-ended tasks, where reference answers offer richer information and high-quality reasoning is harder to obtain from the model’s inherent capabilities alone.Source: §5 · p. 9
Ethics Statement
The ethics statement reports no human subjects, sensitive personal data, or biased decision-making in this work. It identifies computational carbon footprint as the main potential negative impact and notes efforts to reduce it.
Ethics Statement
- The work does not involve human subjects or sensitive personal data.Source: Ethics Statement · p. 9
- The statement reports no biased decision-making in the work.Source: Ethics Statement · p. 9
- Computational resources for training and inference are identified as the main potential source of negative impact, with efficient techniques used to reduce environmental impact.Source: Ethics Statement · p. 9
Reproducibility Statement
The paper reports that its code, experimental settings, and datasets are available, and that mathematical results are accompanied by definitions and derivations.
Reproducibility Statement
- The implementation code and experimental settings, including hyperparameter configurations, are provided in the supplementary materials and Appendix A.5.The authors also state that the training and test datasets are publicly available, with sources cited in the main text.Source: Reproducibility Statement · p. 9
- The paper identifies publicly available code and datasets as reproducibility resources.The code is in the supplementary materials; dataset sources are cited in the main text.Source: Reproducibility Statement · p. 9
- Mathematical results are accompanied by precise definitions and complete derivations in Section 2.2 and Appendices A.3 and A.4.Source: Reproducibility Statement · p. 10
A.1 The Use of Large Language Models (LLMs)
The authors disclose LLM assistance for figure creation, manuscript editing, and code implementation, naming the tools used for each task.
A.1 The Use of Large Language Models (LLMs)
- Gemini-2.5-Pro with Nano Banana assisted in creating Figure 1.Source: §A.1 · p. 11
- GPT-5 was used for grammar checking, paraphrasing, and shortening sections of the manuscript.Source: §A.1 · p. 11
- VSCode Copilot powered by GPT-5 aided code implementation, particularly visualization-related scripts.Source: §A.1 · p. 11
A.2 Motivation Experiments
The motivation experiment contrasts question-only and reference-answer-guided reasoning on a renewable-energy capacity-factor question. In the reported case, answer access accompanied a more accurate explanation that judges deemed independent of the answer.
A.2 Motivation Experiments
- With only the question, the model chose hydroelectricity and incorrectly estimated geothermal after overlooking hydro’s dependence on water availability.With the reference answer, it discussed hydroelectric variability and geothermal’s steady operation under viable geological conditions, and gave more accurate estimates.Source: §A.2 · p. 11
- The authors report that answer access prompted fuller discussion of both the initially uncertain incorrect option and the correct option, rather than answer-conforming shortcuts.Source: §A.2 · p. 11
- The case asks which listed renewable source has the highest capacity factor; its reference answer is geothermal energy, option D.The experiment used Qwen3-1.7B with thinking mode and the officially recommended inference settings.Source: §A.2 · pp. 11–12
- GPT-5 judged the answer-guided reasoning correct and independent of the reference answer.The judgement says it correctly concludes that geothermal has the highest capacity factor and shows no signs of relying on the provided answer.Source: §A.2 · p. 14
A.3 Decomposition and Rewriting of Pr(Z | x, y∗)
This derivation relates the posterior probability of sampling a reasoning path in a set to its question-only probability and conditional utility. It uses a conditional-expectation identity to obtain a factorization.
A.3 Decomposition and Rewriting of Pr(Z | x, y∗)
- The notation defines x as the input problem, y∗ as the reference answer, Z as a subset of reasoning paths, and πθ(z | x) as the model’s path distribution.Source: §A.3 · p. 14
- The posterior over paths given the problem and reference answer assigns probability to the event of sampling a path in Z.Source: §A.3 · p. 14
- For a discrete path space, the derivation expresses conditional expected utility given membership in Z, multiplies by Pr(Z | x), and substitutes to obtain a factorization.The passage states the resulting relation as Pr(Z | x, y∗) = Pr(Z | x) · E[s(z) | z ∈ Z, x].Source: §A.3 · pp. 14–15
A.4 ELBO Derivation with the Amortized Answer-conditioned Posterior
The appendix derives an ELBO using an amortized answer-conditioned posterior and documents implementation settings, supplementary learning dynamics, and reasoning-behavior results across additional datasets.
A.4 ELBO Derivation with the Amortized Answer-conditioned Posterior
- The amortized answer-conditioned posterior changes the sampling distribution and yields an ELBO on log J (θ), with a KL estimator for the posterior-to-prior divergence.The derivation introduces πθ(z | x, y∗) to aid learning of πθ(z | x).Source: §A.4 · p. 15
A.5 Implementation Details
- Experiments use 32 NVIDIA H100 GPUs, specified rollout and evaluation settings, and 5,000 DeepMath-103K and 1,000 MMLU-pro training examples.The evaluation uses GPT-4.1-mini for correctness judgments and reports Avg4 for both general multiple-choice and math tasks.Source: §A.5 · p. 15
A.6 Supplementary Learning Dynamics
- Reference-answer-conditioned paths mention the phrase “reference answer” less often as training progresses, while question-only outputs maintain normal behavior without that phrase.Source: §A.6 · p. 15