PaperlayerView exact revision on arXiv

Source-linked AI summary

RLVR Training of LLMs Does Not Improve Thinking Ability for General QA: Evaluation Method and a Simple Solution

Kaiyuan Li, Jing-Cheng Pang, Yang Yu

arXiv:2603.20799v1cs.CLcs.LG

TL;DR

The paper asks whether RLVR-driven reasoning gains transfer to general question answering, where the value of explicit thinking traces has not been thoroughly validated. It evaluates traces across models, finds their efficacy lower in GQA and direct RL often weak at improving thinking, then presents START, which improves thinking and answers across GQA benchmarks and RL algorithms.

  • Problem

    Whether RLVR gains on verifiable tasks transfer to GQA has not been thoroughly validated, leaving the value of thinking traces in general answering unclear.

    Source: §1 · p. 1

  • Method

    Cross-Generation evaluation tests thinking traces across responder models, and START trains thinking first while keeping final-answer generation fixed and rewarding the complete response.

    Source: §1 · pp. 1–2

  • Results

    Thinking traces are less effective in GQA than in reasoning tasks, direct RL often improves answers without substantially improving thinking, and START outperforms joint RL baselines across GQA datasets and algorithms.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings support explicit GQA training alongside verifiable-task training, and report START as a way to improve both thinking quality and final answers.

    Source: §1 · p. 2

    The observed emergence of meta-context patterns may not fully account for START’s final performance gains.

    Source: §4.4 · p. 10

Abstract

from arXiv · show

Reinforcement learning from verifiable rewards (RLVR) stimulates the thinking processes of large language models (LLMs), substantially enhancing their reasoning abilities on verifiable tasks. It is often assumed that similar gains should transfer to general question answering (GQA), but this assumption has not been thoroughly validated. To assess whether RLVR automatically improves LLM performance on GQA, we propose a Cross-Generation evaluation framework that measures the quality of intermediate reasoning by feeding the generated thinking context into LLMs of varying capabilities. Our evaluation leads to a discouraging finding: the efficacy of the thinking process on GQA tasks is markedly lower than on verifiable tasks, suggesting that explicit training on GQA remains necessary in addition to training on verifiable tasks. We further observe that direct RL training on GQA is less effective than RLVR. Our hypothesis is that, whereas verifiable tasks demand robust logical chains to obtain high rewards, GQA tasks often admit shortcuts to high rewards without cultivating high-quality thinking. To avoid possible shortcuts, we introduce a simple method, Separated Thinking And Response Training (START), which first trains only the thinking process, using rewards defined on the final answer. We show that START improves both the quality of thinking and the final answer across several GQA benchmarks and RL algorithms.

1 INTRODUCTION

The paper asks whether reasoning gains from RLVR transfer to general-purpose tasks, where the effectiveness of explicit thinking traces remains insufficiently understood. It finds that transfer to GQA is limited and introduces START to train thinking separately from response generation.

1 INTRODUCTION

  • The effectiveness of explicit thinking traces in open-ended question answering and instruction following remains inadequately understood.The paper situates this gap against the gains of thinking models trained with verifiable rewards on math and logic tasks.Source: §1 · p. 1
  • Cross-Generation Evaluation feeds thinking traces from source models to responder models to assess their intrinsic value across capabilities.The framework tests the premise that high-quality reasoning should benefit any responder.Source: §1 · p. 1
  • RLVR-trained traces boost reasoning performance, but their efficacy drops sharply on GQA, where answerer capability outweighs stronger thinking traces.The authors conclude that explicit GQA training remains necessary alongside verifiable-task training.Source: §1 · pp. 1–2
  • Direct RL improves final answers more than intermediate thinking, which the authors attribute to GQA shortcuts that can earn reward through answer optimization alone.Unlike verifiable tasks, GQA may not require a robust logical chain for high rewards; the shortcut discussion cites Turpin et al. (2023) and Singhal et al. (2024).Source: §1 · p. 2
  • START trains thinking while keeping final-answer generation fixed, and outperforms joint RL baselines across RL algorithms and GQA datasets.The authors report higher reward and answer quality, and observe richer meta-context in START-trained traces.Source: §1 · p. 2

2 THE THINKING STAGNATION OF RL ON GQA

The section examines whether thinking traces help GQA and whether direct RL improves them. Cross-generation evaluations show that thinking has less influence on GQA than answerer capacity, while direct RL often improves answers without substantially evolving thinking.

2 THE THINKING STAGNATION OF RL ON GQA

  • The section first evaluates thinking’s contribution in GQA, then tests whether direct GQA reinforcement learning evolves more effective traces or mainly improves answering.The direct-RL experiments continue the cross-generation protocol and report training reward and win rate against the Base model.Source: §2 · p. 2, §2.2 · pp. 3–4

2.1 UNCOVERING THE LIMITED EFFICACY OF THINKING IN GQA

  • Cross-Generation evaluation passes a source model’s thinking trace as fixed context to a responder model, separating trace quality from answerer capability.Experiments use Qwen3 scales of 1.7B, 4B, and 8B; GQA is evaluated on AlpacaEval 2.0 using length-controlled win rate against GPT-4-1106-preview.Source: §2.1 · pp. 2–3
  • 45.98% to 56.59% win rate from stronger thinking for a 1.7B answerer, while pairing 8B thinking with an 8B answerer reaches 67.73% on AlpacaEval.The results contrast with MATH, where upgrading thinking for a 1.7B answerer raises accuracy from 67.00% to 82.50%, while upgrading the answerer then yields only 83.50%.Source: Figure 1 · p. 3, §2.1 · p. 3

2.2 THE STAGNATION OF THINKING IN REINFORCEMENT LEARNING ON GQA

  • On ExpertQA and UltraFeedback, Tpost + Apre yields nearly negligible gains, while Tpre + Apost captures most post-training gains, sometimes matching or exceeding the full post-trained model.For Qwen3-1.7B on ExpertQA, Reward rises only from 0.1842 to 0.1849 (+0.0007) with Tpost + Apre.Source: §2.2 · p. 4, Table 1 · p. 4
  • Across Hunyuan-1.8B-Instruct and DeepSeek-R1-Distill-Qwen-1.5B, thinking-only reward gains are +6.88% and +11.38%, but answer-phase gains remain larger; on MATH, thinking drives most gains.In Setting A, Tpost + Apre gives a +19.29% reward increase, almost triple the contribution of evolved thinking alone.Source: §2.2 · p. 4, Table 2 · p. 5
  • The authors characterize direct RL on GQA as thinking stagnation: it primarily improves high-reward answers from existing traces rather than evolving better logical foundations.They attribute this pattern to weaker coupling between thinking and answering in general tasks, where rewards can be obtained without superior thinking.Source: §2.2 · p. 4

3 SEPARATED THINKING AND RESPONSE TRAINING (START)

START separates thinking from response generation during initial training, then jointly optimizes both. It formalizes the training setup as an MDP and is designed to integrate with standard RL algorithms.

3 SEPARATED THINKING AND RESPONSE TRAINING (START)

  • START introduces a two-phase optimization process that decouples cognitive thinking from response generation.The method follows from the thinking stagnation identified in the preceding analysis.Source: §3 · p. 5

3.1 PRELIMINARIES: RL FOR LLMS AS AN MDP

  • LLM generation is formalized as an MDP: states contain the instruction and generated tokens, actions are vocabulary tokens, transitions append tokens, and reward follows the complete sequence.The complete sequence consists of thinking and answer tokens.Source: §3.1 · p. 5

3.2 CONCEPTUAL SHIFT: FROM EXPLORATION SPACE TO ENVIRONMENT

  • START shifts answering from the policy’s exploration space into the environment, addressing weak coupling between thinking and answer quality in general tasks.With answering treated as a stationary transition, improving thinking traces becomes the policy’s route to higher rewards.Source: §3.2 · p. 6

3.3 PHASE I: THINKING EVOLUTION VIA GRADIENT MASKING

  • In Phase I, answer-token gradients are zeroed while the sampled answer still maps the thinking trace to a reward, making thinking the only route to improve the policy’s reward.Phase II removes the mask and jointly optimizes thinking and answering.Source: §3.3 · p. 6

3.4 PHASE II: JOINT OPTIMIZATION

  • After Phase I, Phase II removes the gradient mask and jointly fine-tunes thinking and answering so the answer can use the acquired thinking capabilities.The mask is set to 1 for tokens in both the thinking and answer sequences.Source: §3.4 · p. 6

3.5 INTEGRATED WITH STANDARD RL ALGORITHMS

  • START is presented as compatible with existing RL algorithms, using outcome-based reward models rather than requiring expensive step-level annotations.The method requires only a token-level loss mask and is described as compatible with frameworks such as DAPO and open-source GRPO implementations.Source: §3.5 · p. 7

4 EXPERIMENT

START separates thinking optimization from answer generation, and the experiments report gains in standalone thinking utility and final-model performance. The analysis also finds increased user-needs identification and structural planning in START-trained traces, while noting these patterns may not fully explain performance gains.

4.1 SETUP

  • Experiments use Qwen3-1.7B as the seed model, ArmoRM as the reward model, and GRPO and DAPO on ExpertQA and UltraFeedback.Baselines use the same seed model and RL algorithm without two-phase separation and are trained until convergence.Source: §4.1 · p. 7

4.2 MAIN RESULT

  • GRPO+START achieves a 68.15% win rate against vanilla GRPO when post-trained thinking traces are paired with the frozen base answering head.The authors attribute this result to masking answering gradients in Phase I.Source: §4.2 · p. 7, Table 3 · p. 7
  • In holistic post-trained evaluation, GRPO+START achieves a 59.24% win rate over vanilla GRPO and reaches approximately 0.220 reward.Its reward curve rises during Phase I and surges in Phase II, surpassing GRPO and GRPO-MA.Source: §4.2 · pp. 7–8, Figure 3 · p. 8
  • GRPO-MA reaches a 54.14% win rate in the Tpost + Apre setting, below START’s 68.15%, and ends at a lower reward plateau.The authors suggest multi-answer sampling reduces the number of unique thinking paths explored under a fixed sampling budget.Source: §4.2 · p. 8
  • Supplementary experiments with DAPO and UltraFeedback report significant START gains across different RL frameworks and data distributions.Table 4 presents results across datasets and algorithms, with win rates against the post-trained base algorithm.Source: §4.2 · p. 8, Table 4 · p. 9

4.3 ANALYSIS OF PHASE I: THINKING AS A FOUNDATION

  • Replacing base-model thinking traces with Phase I traces yields a +0.0046 reward increase and a 69.11% win rate.The authors report that the improved thinking quality persists when answering gradients are unmasked in Phase II.Source: §4.3 · p. 9, Table 5 · p. 9

4.4 EXPLORING THE EMERGENCE OF META-CONTEXT MODELING

  • START traces identify user needs in 90.45% of cases, compared with 54.78% for vanilla GRPO, and show structural planning in 92.36%.The authors caution that these patterns alone may not fully account for final performance gains.Source: §4.4 · p. 10, Table 6 · p. 10

5 RELATED WORK

Prior work describes RLVR’s advances on tasks with objective rewards, while whether those capabilities generalize to GQA remains unresolved. Research on thinking in general domains includes TPO, but the specific mechanics and efficacy of thinking on non-verifiable tasks remain underexplored.

5.1 THINKING MODELS AND RLVR

  • Thinking models generate a Chain-of-Thought before a final response, and the broader shift toward complex reasoning has accompanied RL’s rise.The passage cites Wei et al. (2022) and Wang et al. (2023) for Chain-of-Thought.Source: §5.1 · p. 10
  • RLVR has advanced reasoning on verifiable tasks such as mathematics, coding, and formal logic, but its transfer to GQA remains an open question.These tasks provide objective reward signals, such as answer correctness or code pass rates.Source: §5.1 · p. 10

5.2 LLMS IN GENERAL QA

  • Although GQA benchmarks use LLM-as-a-judge evaluation and thinking modes have entered general-purpose applications, the efficacy of thinking on non-verifiable tasks remains little studied.Prior work TPO trains models to think in general domains through iterative search and optimization without direct human thought supervision.Source: §5.2 · pp. 10–11

6 CONCLUSION

The paper finds that thinking traces are less effective in GQA than in reasoning tasks, and that direct RL on GQA often does not substantially evolve thinking. It proposes START and reports consistent effectiveness across multiple benchmarks and algorithms.

6 CONCLUSION

  • Thinking processes have significantly lower efficacy in GQA than in reasoning tasks.The paper reports this disparity using its Cross-Generation evaluation framework.Source: §6 · p. 11
  • Direct reinforcement learning on GQA often fails to stimulate substantive thinking evolution, which the paper hypothesizes is linked to loose coupling between thinking and answering.START isolates the thinking phase during initial training.Source: §6 · p. 11
  • Across multiple benchmarks and algorithms, the paper reports consistent effectiveness for START, which isolates initial thinking-phase training.Source: §6 · p. 11

A DATASETS AND HYPERPARAMETERS

The appendix section introduces an overview of the study’s datasets and the configurations used for evaluation and training.

A DATASETS AND HYPERPARAMETERS

A.1 MATH DATASET

The MATH setup focuses on the hardest problems and uses separate training and test samples. Structured reasoning is elicited with a step-by-step prompt, and answers are scored against ground truth.

A.1 MATH DATASET

  • The study uses Level 5 MATH problems, the dataset’s most challenging subset, to assess model thinking depth and avoid performance saturation.MATH spans algebra, geometry, and calculus and has five difficulty levels.Source: §A.1 · p. 14
  • The authors randomly sampled 2,000 problems for training and a separate 200 for testing.Source: §A.1 · p. 14
  • A step-by-step prompt elicits structured reasoning, while verl answer extraction and normalization score responses against ground-truth solutions.Source: §A.1 · p. 14

A.2 ALPACAEVAL 2.0 BENCHMARK

AlpacaEval 2.0 evaluates general question answering across 805 diverse prompts. The section reports length-controlled win rate and describes the evaluator and rare scoring failures.

A.2 ALPACAEVAL 2.0 BENCHMARK

  • AlpacaEval 2.0 simulates human preferences across 805 prompts spanning creative writing, coding assistance, and general information seeking.Source: §A.2 · p. 14
  • Length-controlled win rate is the main indicator, adjusting for response length to mitigate evaluators’ length bias.Source: §A.2 · p. 14
  • The local vLLM Llama 3 70B evaluator is described as highly ranked for human agreement; parsing or readable-score failures occurred in fewer than 0.4% of samples and did not statistically affect win rate.Source: §A.2 · p. 14

A.3 EXPERTQA DATASET

ExpertQA tests performance on complex domain-specific questions verified by experts. The authors use the official second-round compiled data, splitting it randomly into training and test sets.

A.3 EXPERTQA DATASET

  • ExpertQA covers academic and professional fields with complex, expert-verified questions requiring factual accuracy and nuanced reasoning.Source: §A.3 · p. 14
  • The evaluation uses the official anonymized compiled data from the second round of expert reviews.Source: §A.3 · p. 14
  • The authors randomly assign 90% of samples to training for optimizing thinking traces and responses, reserving 10% for testing.Source: §A.3 · p. 14

A.4 ULTRAFEEDBACK DATASET

The appendix details datasets and training configurations, then describes exploratory comparisons of thinking–answer coupling across MATH and ExpertQA using nested sampling and scoring.

A.4 ULTRAFEEDBACK DATASET

  • UltraFeedback provides diverse prompts and feedback on instruction-following, truthfulness, and honesty; the study samples 2,000 training and 200 test instances.The curated prompt set excludes excessively long inputs.Source: §A.4 · p. 15

A.5 SAMPLING PARAMETERS

  • Sampling configurations for three model series follow recommended family-specific settings, summarized in Table 7.Source: §A.5 · p. 15, Table 7 · p. 15

A.6 REWARD MODEL CONFIGURATION

  • Continuous-reward experiments use ArmoRM-Llama3-8B-v0.1’s overall score, aggregating attribute-specific rewards into one scalar.Source: §A.6 · p. 15

B REPRODUCTION DETAILS AND RL ENVIRONMENTS

  • The authors state that scripts, evaluation code, and configurations are included in supplementary material and that the codebase will be open-sourced upon acceptance.Source: B REPRODUCTION DETAILS AND RL… · p. 15

B.1 ENV

  • Experiments use a standardized verl Docker environment with optimized Transformers, vLLM, and FSDP components.Source: §B.1 · pp. 15–16

B.2 GRPO / B.3 DAPO / B.4 GRPO-MA

  • GRPO and DAPO configurations are specified in Tables 8 and 9; GRPO-MA uses two thinking traces and three answers per trace, keeping six samples per prompt.GRPO-MA modifies verl for two-tier generation and uses the same base RL hyperparameters as standard GRPO.Source: §B.2 · p. 16, Table 8 · p. 16, §B.3 · p. 16

C ANALYSIS OF THINKING-ANSWER COUPLING ACROSS DOMAINS

C.1 SAMPLING METHOD

  • For each instruction, the nested framework samples distinct thinking traces and multiple responses per trace, then scores each complete path.MATH paths receive binary correctness rewards, whereas ExpertQA paths receive continuous reward-model scores; the experiments use m = 8 and n = 8, yielding 64 trajectories per instruction.Source: §C.1 · p. 18

C.2 STRONG COUPLING IN REASONING TASKS

On MATH, a low minority outcome ratio reflects strong coupling between thinking traces and answer correctness, whereas ExpertQA shows loose coupling across models. The supplied examples illustrate answer-planning meta-context, and the prompts detect identity speculation and presentation planning.

C.2 STRONG COUPLING IN REASONING TASKS

  • The Minority Outcome Ratio γ measures stochasticity in answering for a fixed MATH thinking trace with binary correct-or-incorrect outcomes.The ratio is averaged over sampled thinking traces across the dataset.Source: §C.2 · p. 18
  • 0.0469 is the MATH minority outcome ratio, reflecting strong coupling between a thinking trace's logical integrity and answer correctness.The authors describe the answering phase as nearly deterministic once a logical path is established.Source: §C.2 · p. 18

C.3 LOOSE COUPLING IN GENERAL QA

  • Qwen3-1.7B's ExpertQA coupling-ratio distribution peaks near 0.65, while no evaluated model achieves universal strong coupling.A significant portion of each model's distribution remains in the loose-coupling zone, ρ < 1.0.Source: Figure 5 · p. 19, §C.3 · p. 19
  • In GQA, answer-quality variation often matches or exceeds thought-path variation; ρ < 1 denotes loose coupling, with style, structure, or language affecting output quality.Answering fluctuation measures variation for a fixed thought, while thinking fluctuation measures variation across thought paths.Source: §C.3 · p. 19

Base Model (Qwen3-1.7B)

Vanilla GRPO

  • Vanilla GRPO's example adds structural planning and audience needs, which the passage describes as improvements in meta-context.The shown planning emphasizes accessible language and a logical answer structure.Source: Vanilla GRPO · p. 20

E PROMPTS FOR META-CONTEXT IDENTIFICATION

  • One prompt flags thinking that speculates about a user's identity, role, hidden intent, motivation, or psychology.It asks for a binary 1-or-0 judgment.Source: E PROMPTS FOR META-CONTEXT… · p. 20
  • A separate prompt detects whether thinking mentions answer structure, content, tone, style, or level of detail.It also requests a binary 1-or-0 output.Source: E PROMPTS FOR META-CONTEXT… · p. 21

Related papers

Loading 2603.20799v1…