PaperlayerView exact revision on arXiv

Source-linked AI summary

Coupled Variational Reinforcement Learning for Language Model General Reasoning

Xueru Wen, Jie Lou, Yanjiang Liu, Hongyu Lin, Ben He, Xianpei Han, Le Sun, Yaojie Lu, Debing Zhang

arXiv:2512.12576v3cs.CLcs.AI

TL;DR

Verifier-free RL uses answer probabilities as rewards, but question-only trace sampling can be inefficient and leave traces incoherent with answers. CoVRL couples prior and posterior sampling through a composite distribution and hybrid strategy. It reports improvements over the base model and strongest baseline across diverse reasoning benchmarks.

  • Problem

    Verifier-free RL uses probabilities of generating reference answers as rewards, but question-only trace sampling can be inefficient and incoherent with final answers.

    Source: §1 · p. 1

  • Method

    CoVRL frames reasoning training as variational optimization, coupling prior and posterior generation through a composite distribution and hybrid sampling.

    Source: §1 · p. 2

  • Results

    12.4% improvement over the base model and 2.3% improvement over the strongest baseline are reported across diverse benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CoVRL jointly optimizes question-only and answer-guided generation without external verifiers, balancing exploration with inference-time transferability.

    Source: §5 · p. 9

    Enhanced reasoning abilities could be misused to generate sophisticated misinformation.

    Source: Impact Statement · p. 9

Abstract

from arXiv · show

While reinforcement learning has achieved impressive progress in language model reasoning, it is constrained by the requirement for verifiable rewards. Recent verifier-free RL methods address this limitation by utilizing the probabilities that LLMs generate reference answers as reward signals. However, these approaches typically sample reasoning traces conditioned only on the question. This design decouples reasoning-trace sampling from answer information, leading to inefficient exploration and incoherence between traces and final answers. In this paper, we propose \textit{\b{Co}upled \b{V}ariational \b{R}einforcement \b{L}earning} (CoVRL), which bridges variational inference and reinforcement learning by coupling prior and posterior distributions through a hybrid sampling strategy. By constructing and optimizing a composite distribution that integrates these two distributions, CoVRL enables efficient exploration while preserving strong thought-answer coherence. Extensive experiments on mathematical and general reasoning benchmarks show that CoVRL improves performance by 12.4\% over the base model and achieves an additional 2.3\% improvement over state-of-the-art verifier-free RL baselines, providing a principled framework for enhancing the general reasoning capabilities of language models.

1. Introduction

Verifier-free RL avoids external reward checkers but question-only trace sampling can be inefficient and incoherent with final answers. CoVRL couples prior and posterior generation through variational optimization and hybrid sampling.

1. Introduction

  • Rule-based RL rewards require verifiable answers, limiting applicability where formal verification is challenging, such as chemistry.Using an LLM as verifier requires a strong verifier and risks reward hacking.Source: §1 · p. 1
  • Question-only trace sampling can be inefficient on difficult questions and produce reasoning that is incoherent with the final answer.Even correct reasoning may receive low rewards when its answer format differs from the ground truth.Source: §1 · p. 2
  • CoVRL frames reasoning training as variational optimization, coupling question-conditioned prior sampling with answer-conditioned posterior sampling.The prior reflects inference conditions, while the posterior provides answer guidance during training.Source: §1 · p. 2
  • Because direct sampling from the composite distribution is computationally complex, CoVRL randomly selects between generation modes for each example.It optimizes a variational lower bound with answer reconstruction and transferability regularization, using importance weighting across both modes.Source: §1 · p. 2
  • 12.4% improvement over the base model and 2.3% over the strongest baseline are reported across diverse benchmarks.The contribution summary reports these improvements without specifying a metric label.Source: §1 · p. 2

2. Methodology

The methodology frames reasoning training through latent-variable variational inference, then couples question-only prior and answer-conditioned posterior trace generation. It uses hybrid sampling and importance weighting to optimize the composite distribution, with separate mechanisms for reconstruction, KL regularization, and prompt formatting.

2.1. Preliminaries

  • GRPO can optimize verifiable rewards, but this limits reinforcement learning to domains with accessible verifiers; latent-variable variational inference offers an ELBO with reconstruction and inference terms.Source: §2.1 · pp. 2–3

2.2. Coupled Variational Reinforcement Learning

  • Prior-only sampling can waste exploration and mismatch answer formulations; posterior-only training risks inference gaps because reverse KL does not ensure prior-region coverage.Source: §2.2 · p. 3

2.3. Composite Distribution

  • The composite distribution token-wise mixes prior and posterior probabilities, coupling answer-guided prior optimization with inference alignment and avoiding posterior-only coverage issues.Equal weighting balances gradient contributions from both distributions.Source: §2.3 · p. 4

2.4. Hybrid Sampling and Coupled Optimization

  • Hybrid rollouts sample from the prior with probability α or posterior with 1−α, then use importance weighting to optimize the composite distribution with GRPO and the model’s answer probability as reward.NLL loss is restricted to valid formatted outputs with positive advantage.Source: §2.4 · pp. 4–5

2.5. Optimizing KL Divergence

  • KL regularization uses sampling-dependent estimators with Bregman control variates; soft-clipped importance ratios limit overflow, and maximum-length samples skip KL loss.The estimators remain non-negative for prior- or posterior-sampled traces.Source: §2.5 · pp. 5–6

2.6. Prompt Template and Tokenization

  • Prompt templates distinguish question-only prior generation from question-and-answer posterior generation, while separately encoding special tokens and using string stop sequences during generation.Source: §2.6 · p. 6

3. Experiments

CoVRL improves performance across benchmarks, while training analyses report stable optimization and reasoning gains that transfer across model sizes and domains.

3.1. Setup

  • Experiments finetune Qwen2.5-7B-Base directly on non-mathematical WebInstruct questions, comparing verifier-free baselines and evaluating general and mathematical reasoning benchmarks.The dataset receives no additional filtering.Source: §3.1 · p. 6

3.2. Main Results

  • 50.2% overall represents a 12.4% improvement over the base model, while CoVRL exceeds the strongest baseline by 2.3% on average.Table 1 reports Average@N, and its Overall column is the arithmetic mean across benchmark tasks.Source: §3.2 · pp. 6–7, Table 1 · p. 7

3.3. Training Dynamics

  • Figure 4 reports posterior reward consistently above prior reward while prior reward rises; response length increases, and NLL and KL losses decline.The authors interpret the trends as improved answer prediction and effective regularization alongside stable optimization.Source: Figure 4 · p. 7, §3.3 · p. 7

3.4. Hybrid Sampling Strategy

  • Figure 5 compares prior-sampling probabilities: α = 0.1 outperforms α = 0.9, but balanced sampling performs best overall.At α = 0.9 reasoning chains shorten; at α = 0.1 they lengthen, while training-inference mismatch limits performance relative to balanced sampling.Source: §3.4 · pp. 7–8, Figure 5 · p. 8

3.5. Training Model and Data

  • Across tested base models and sizes, CoVRL delivers consistent gains ranging from 8.6% to 14.0%.Experiments include Qwen2.5-base and Qwen3-base models from 7B to 14B parameters.Source: §3.5 · p. 8
  • Training on math-only data improves non-mathematical reasoning, and training on non-mathematical data improves mathematical reasoning.The results indicate transfer across the two training and evaluation domains.Source: §3.5 · p. 8

4. Related Work

Related work contrasts verifier-dependent reinforcement learning with verifier-free methods, then positions CoVRL as coupling prior and posterior sampling in one composite distribution.

Verifier-free Reinforcement Learning

  • RLVR requires robust verifiers, limiting broader reasoning-task applicability; LaTRO instead formulates reasoning as sampling from latent distributions (Chen et al., 2024).Source: §4 · p. 8
  • JLB, VeriFree, and RLPR use probability-based rewards but prior-based sampling is described as inefficient and misaligned; RAVR instead samples from posterior distributions.The cited works are JLB, VeriFree (Zhou et al., 2025a), RLPR (Yu et al., 2025b), and RAVR (Lin et al., 2025).Source: §4 · p. 9

Self-improving Language Models

  • CoVRL couples prior and posterior sampling, whereas Zhou et al. (2025b) use separate models, IWAE, and EM; CoVRL optimizes one composite distribution through online RL.Source: §4 · p. 9

5. Conclusion

CoVRL enhances language-model reasoning without external verifiers by jointly optimizing question-only and answer-guided generation through coupled prior and posterior distributions.

5. Conclusion

  • CoVRL targets language-model reasoning without external verifiers.Source: §5 · p. 9
  • Its composite distributions jointly optimize question-only and answer-guided generation.Source: §5 · p. 9
  • Hybrid sampling couples the distributions to balance efficient exploration with inference-time transferability.Source: §5 · p. 9

Impact Statement

The appendix develops KL estimators for CoVRL’s hybrid sampling, while the impact statement flags potential misuse and calls for safety mechanisms and ethical guidelines.

Impact Statement

  • The authors warn that enhanced reasoning could be misused to generate sophisticated misinformation and encourage safety mechanisms and ethical guidelines for deployment.Source: Impact Statement · p. 9

A. KL Divergence Estimator

A.1. Derivation via Control Variates and Bregman Divergence

  • For reverse and forward KL, zero-mean control variates reduce estimator variance, and the resulting forms are nonnegative by a Bregman-divergence argument.Source: §A.1 · p. 13

A.2. KL Estimators Under Hybrid Sampling

  • When r > 1 the posterior favors answer-aligned traces, whereas 0 < r < 1 indicates prior-favored exploratory paths; hybrid sampling uses both patterns for stable optimization.Source: Figure 6 · p. 14, §A.2 · p. 14

A.3. Relationship with On-Policy Distillation

CoVRL couples an inference-time prior with an answer-conditioned posterior, contrasting with static-teacher distillation and related verifier-free methods. Its asymmetric objective prioritizes the prior, while a symmetric JSD formulation is discussed as a future direction.

A.3. Relationship with On-Policy Distillation

  • OPD combines reinforcement learning on student-sampled trajectories with dense token-level teacher supervision, but prior work notes support mismatch and error accumulation on shifted prefixes.Source: §A.3 · pp. 14–15
  • Unlike OPD’s static external teacher, CoVRL couples an inference-time prior and answer-conditioned posterior so each shapes the other’s training distribution.Source: §A.3 · p. 15
  • CoVRL’s symmetric coupling resembles JSD regularization and is described as mitigating OPD’s unidirectional optimization pathologies, though standard JSD cannot replace its asymmetric KL objective.Source: §A.3 · p. 15
  • A symmetric JSD formulation is proposed for future study when performance, calibration, and continued evolution of prior and posterior are equally important.Source: §A.3 · pp. 15–16

B. Comparison Among Existing Approaches

C. Evaluation Implementation

The evaluation implementation standardizes multiple-choice datasets and uses response-parsing procedures tailored to question type.

C. Evaluation Implementation

D. Training Dynamics of Different Sampling Ratios

Sampling ratios shift prior and posterior reward scores in opposite directions, with corresponding changes in response length; the text states that balancing exploration and exploitation gives the best overall performance.

D. Training Dynamics of Different Sampling Ratios

E. Influence of Loss Components

The section tests CoVRL’s loss components and reward formulations, then evaluates it on reasoning models that already received reasoning-specific training. The results show sensitivity to KL regularization, relative resilience to NLL changes, and overall gains on both reasoning models.

E. Influence of Loss Components

  • 27.4% overall accuracy after reducing the KL coefficient to 0.1, alongside rising KL divergence and training instability.The authors attribute the degradation to insufficient regularization, distribution shift, and training-inference mismatch.Source: E. Influence of Loss Components · pp. 17–18

F. Improvement on Reasoning Model

  • 4.0% overall improvement for Qwen3-8B (55.9% to 59.9%) and 0.9% for Qwen3-14B (61.6% to 62.5%), despite mixed benchmark-level changes.Training response length was raised from 2048 to 4096 tokens to reduce clipping of reasoning traces.Source: Table 5 · p. 18, F. Improvement on Reasoning Model · p. 18

E. Influence of Loss Components

  • 44.7% performance after reducing the NLL coefficient to 0.1, a moderate decline that the authors attribute to overlap between NLL and RL objectives.They describe NLL loss as primarily training reasoning summarization and final-answer generation.Source: E. Influence of Loss Components · p. 18
  • 49.7% to 50.6% overall performance across reward formulations, with less than one percentage point of variation.The tested variants changed sequence-length normalization and whether reward signals used log-probabilities or raw probabilities.Source: E. Influence of Loss Components · p. 18

G. Computation Cost Analysis

CoVRL has relatively small end-to-end training-time overhead compared with GRPO in the reported setting. Rollout generation dominates step time, while on-policy computation can remove one source of overhead.

G. Computation Cost Analysis

  • Peak GPU memory remains largely unchanged in the reported setup using vLLM and a fixed micro-batch size via gradient accumulation.CoVRL optimizes both distributions during actor update and also computes old log-probabilities.Source: G. Computation Cost Analysis · pp. 18–19
  • 36.09 seconds per training step with CoVRL versus 30.32 seconds with GRPO, although the actor update cost approximately doubles.Rollout generation dominates total step time, limiting the end-to-end increase in the measured setup.Source: G. Computation Cost Analysis · pp. 18–19
  • Under a fully on-policy implementation, old log-probability computation can be merged with the actor-update forward pass, eliminating that overhead relative to standard GRPO.The reported measurement used Qwen2.5-7B-Base on 4 × 8 GPUs, global batch size 192, and maximum response length 2048.Source: G. Computation Cost Analysis · p. 19

H. Reasoning Process Evaluation

CoVRL training was evaluated for intermediate reasoning quality using multiple LLM judges on sampled MMLU-Pro questions. Judges reported gains across all four assessed dimensions, with the authors noting that this evaluation does not replace human expert annotation.

H. Reasoning Process Evaluation

  • CoVRL consistently improves factuality, validity, coherence, and utility scores across all evaluators.The evaluation used 100 randomly sampled MMLU-Pro questions and multiple open- and closed-source LLM judges, with scores on a 1–5 scale.Source: H. Reasoning Process Evaluation · p. 19
  • LLM-as-judge scores provide additional evidence about reasoning traces but cannot replace human expert annotation.Each judge scored traces before and after training, with eight queries per sample and mean and standard deviation reported.Source: H. Reasoning Process Evaluation · p. 19
Loading 2512.12576v3…