PaperlayerView exact revision on arXiv

Source-linked AI summary

LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewards

Guanzheng Chen, Michael Qizhe Shieh, Lidong Bing

arXiv:2603.02146v1cs.CL

TL;DR

Long-context RLVR struggles because outcome-only rewards provide insufficient guidance for retrieving relevant evidence. LongRLVR adds a verifiable context reward for grounding chunks, and it consistently outperforms outcome-only RLVR across models and benchmarks. The results support explicitly rewarding contextual grounding in long-context applications.

  • Problem

    Outcome-only RLVR rewards do not adequately guide contextual grounding, the process of identifying relevant information in extensive external contexts.

  • Method

    LongRLVR augments answer rewards with a verifiable context reward for selecting ground-truth evidence chunks before answer generation.

  • Results

    LongRLVR consistently and substantially outperforms naive RLVR across LLaMA and Qwen models and long-context benchmarks.

  • Takeaways & Limitations

    Explicitly rewarding the grounding process is an effective strategy for improving long-context reasoning capabilities.

  • Takeaways & Limitations

    The analysis assumes that correct solutions often require synthesizing a complete set of prerequisite evidence, although alternative supporting evidence or partial subsets may sometimes suffice.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capabilities of Large Language Models (LLMs) by optimizing them against factual outcomes. However, this paradigm falters in long-context scenarios, as its reliance on internal parametric knowledge is ill-suited for tasks requiring contextual grounding--the ability to find and reason over externally provided information. We identify a key reason for this failure: a reward based solely on the final answer is too sparse to effectively guide the model for identifying relevant evidence. We formally prove that the outcome-only reward leads to significant vanishing gradients for the context grounding process, rendering learning intractable. To overcome this bottleneck, we introduce LongRLVR to augment the sparse answer reward with a dense and verifiable context reward. This auxiliary signal directly incentivizes the model for selecting the correct grounding information, providing a robust learning gradient that solves the underlying optimization challenge. We validate our method on challenging long-context benchmarks using Qwen and LLaMA models. LongRLVR consistently and significantly outperforms the standard RLVR across all models and benchmarks, e.g., boosting a 14B model's scores on RULER-QA from 73.17 to 88.90 and on LongBench v2 from 39.8 to 46.5. Our work demonstrates that explicitly rewarding the grounding process is a critical and effective strategy for unlocking the full reasoning potential of LLMs in long-context applications. Our code is available at https://github.com/real-absolute-AI/LongRLVR.

1 INTRODUCTION

RLVR improves factual reasoning but struggles with long-context tasks that require retrieving and synthesizing external information. LongRLVR addresses this bottleneck by adding verifiable grounding rewards, enabling continued improvement in contextual recall and answer accuracy.

  • Motivation: RLVR primarily strengthens internal knowledge, limiting its effectiveness for long-context contextual grounding.Long-context reasoning requires accurately retrieving and synthesizing information from external documents.
  • Motivation: Outcome-only rewards cause contextual recall to stagnate during naive long-context RLVR training.The resulting retrieval plateau creates a ceiling for answer accuracy and halts learning progress on training rewards.
  • Approach: LongRLVR adds a verifiable context reward by having the model identify grounding chunk identifiers before producing the final answer.Predicted identifiers are compared with ground-truth chunks, directly rewarding relevant evidence extraction.
  • Approach: The added grounding signal allows contextual recall and answer accuracy to improve continuously throughout training.This addresses the sparse learning signal associated with outcome-only rewards.
  • Evaluation: LongRLVR is evaluated on LLaMA-3.1 and Qwen2.5 models across RULER, LongBench v2, and LongReason.The experiments compare the method with outcome-only RLVR using a synthetic long-context question-answering data pipeline annotated with grounding chunks.

2 METHOD

LongRLVR explicitly separates contextual grounding from answer generation and augments outcome-only rewards with verifiable signals for selecting relevant evidence. Its theory identifies vanishing grounding gradients, while its practical design uses Fβ-based rewards and grounded QA data.

  • Explicit grounding formulation: Long-context RLVR must distinguish selecting relevant evidence from synthesizing the final answer.The factorized policy models grounding and answering as separate stages, with the Answer Head conditioned on selected evidence.
  • Vanishing grounding gradient: Outcome-only rewards produce vanishing grounding gradients because each chunk receives useful credit only when prerequisite evidence is already selected.The activation event requiring other interacting evidence makes the learning signal extremely unlikely for challenging long-context tasks.
  • Verifiable context reward: LongRLVR adds a verifiable context reward that directly supervises the grounding head alongside the final-answer reward.The method gives positive feedback for selecting relevant chunks even when the complete evidence set is not recovered.
  • Verifiable context reward: The context reward supplies a dense component independent of the rare prerequisite-activation event, preventing the grounding gradient from vanishing.For positively associated chunk selections, cross-covariance terms can further strengthen the signal.
  • Practical reward design: LongRLVR uses an Fβ-score to balance precision and recall, with β allowing recall to receive greater weight, and modulates it with answer reward.The reward combines unconditional grounding feedback with a synergistic term tied to final-answer success.
  • Synthetic data generation: Training requires grounded QA tuples containing the context, question, evidence-chunk set, and answer.A synthetic pipeline produces high-fidelity examples with precise grounding annotations for direct supervision.

3 EXPERIMENTAL SETUP

The experiments train three model configurations with RLVR and compare LongRLVR against SFT, naive RLVR, and larger reference models on three long-context QA benchmarks.

  • Data and training: The training set contains 46K long-context question-answer pairs with explicit grounding annotations from book, arXiv, and code documents.Documents are filtered to 8K–64K tokens and answers are grounded in specific evidence segments.
  • Data and training: The trained models are LLaMA-3.1-8B, Qwen2.5-7B-1M, and Qwen2.5-14B-1M, using naive GRPO-based RLVR.Easy questions are excluded before training based on a Qwen3 judge rating.
  • Evaluation: All benchmark metrics are accuracy, with best overall, second-best overall, and within-model trained-method scores separately marked.Within each trained model, SFT, RLVR, and LongRLVR are compared directly.
  • Baselines: LongRLVR is compared with SFT and naive RLVR using the same synthetic training data and base models.The evaluation also includes larger open-source models and the specialized QwenLong-L1-32B baseline.
  • Benchmarks: The evaluation covers RULER-QA, LongBench v2, and LongReason, spanning synthetic and realistic long-context question answering.RULER-QA uses 32K, 64K, and 128K contexts, while LongBench v2 includes documents up to 128K tokens.

4 RESULTS AND ANALYSES

LongRLVR consistently outperforms SFT and naive RLVR across models and benchmarks, with reward-component analyses showing that grounding and answer rewards work best together.

  • Main results: LongRLVR consistently and substantially outperforms SFT and naive RLVR across LLaMA and Qwen models at 7B, 8B, and 14B scales.The comparison uses the same base models and identical training data.
  • Main results: 46.5 vs. 39.8 on LongBench v2 for Qwen2.5-14B-1M demonstrates a large LongRLVR gain over naive RLVR.Additional LongBench v2 comparisons are 38.6 vs. 32.4 for Qwen2.5-7B-1M and 36.2 vs. 32.4 for LLaMA-3.1-8B.
  • Comparison with larger models: 79.22 on LongReason for Qwen2.5-7B-1M exceeds 57.59 for LLaMA-3.1-70B and 72.76 for Qwen2.5-72B-YaRN.The Qwen2.5-14B-1M model also reaches 46.5 on LongBench v2, surpassing the 72B comparison model there.
  • Comparison with specialized models: 88.90 vs. 87.60 on RULER-QA and 78.42 vs. 77.50 on LongReason show Qwen2.5-14B-1M outperforming Qwen3-14B on both reported benchmarks.The comparison is made against a newer model with a more advanced backbone and post-training strategy.
  • Reward components: Answer-only training stagnates in contextual recall, context-only training eventually loses downstream accuracy, and LongRLVR achieves the best stable performance by combining both rewards.The analysis links recall behavior to LongBench v2 performance and training answer reward.
  • Hyperparameters: Figure 5 varies the context-reward blending factor η, F-score parameter β, and chunks per document for Qwen2.5-7B and LLaMA-3.1-8B.The study reports overall LongBench v2 performance under these hyperparameter changes.
  • Hyperparameters: Performance peaks at η = 0.1 and β = 2, while LongRLVR remains robust to the number of chunks per document.The reward ablation describes η as balancing unconditional and synergistic grounding signals, while β trades off recall and precision.
  • Data quality: Figure 4 evaluates rejection-sampling quality and data-filtering strategies on LongBench v2, with high-quality, challenging data reported as most effective.The ablation uses Qwen2.5-7B-1M-LongRLVR.

5 RELATED WORK

Related work covers outcome-based RLVR, long-context alignment, and agentic workflows for processing extended documents.

  • Reinforcement Learning with Verifiable Rewards: RLVR enhances LLM reasoning by rewarding deterministic, ground-truth outcomes such as passing unit tests or deriving correct solutions.The related-work discussion associates this approach with strong performance on self-contained reasoning tasks.
  • Long Context Alignment: Long-context alignment extends context windows and uses post-training to improve models’ ability to use information in long-context applications.The passage contrasts context-window extension with the need to unlock latent capabilities through alignment.
  • Long-Context LLM Agent: Long-context agent methods split documents into chunks, process them sequentially, and integrate information through multi-turn collaboration.These workflows avoid processing the entire context in a single LLM pass.

6 CONCLUSION

The paper identifies sparse outcome-only rewards as a grounding limitation in long-context RLVR and introduces LongRLVR, which combines answer and verifiable context rewards.

  • Conclusion: LongRLVR addresses RLVR’s inability to learn contextual grounding from sparse, outcome-only rewards in long-context scenarios.The paper names this limitation the “vanishing grounding gradient” problem.
  • Conclusion: LongRLVR augments the standard answer reward with a dense verifiable context reward that teaches models to identify and extract relevant evidence before answering.The conclusion attributes the method’s success to synergy between context and answer rewards.
  • Conclusion: LongRLVR substantially outperforms SFT and naive RLVR across multiple models and benchmarks.The conclusion presents these experiments as evidence for the method’s effectiveness in long-context reasoning.

REPRODUCIBILITY STATEMENT

The paper emphasizes reproducibility through formal proofs, detailed synthetic-data documentation, and reporting of implementation details.

  • Theoretical claims are formally proven in the appendix with step-by-step derivations for REINFORCE and GRPO estimators.The derivations cover both policy-gradient estimators.
  • The synthetic data generation pipeline is documented through Algorithm 1 and additional details on corpus sourcing, preprocessing, and quality control.
  • Implementation details include model specifications and training hyperparameters.The supplied passage indicates that implementation details are reported, though the sentence is truncated.

THE USE OF LARGE LANGUAGE MODELS (LLMS)

The appendix defines the grounding setup, reward assumptions, and policy-gradient tools used to analyze how outcome and context rewards train evidence selection.

  • Grounding setup: The grounding policy partitions the long context into chunks and selects them with a binary vector Z.Each chunk cj has a selection indicator zj that records whether it is selected.
  • Grounding setup: The policy uses a log-linear distribution over chunk subsets with logits, dependency potentials, and marginal selection probabilities.The independent Bernoulli model is included as the special case with zero interaction potential.
  • Reward assumptions: The ground-truth evidence set G contains essential chunks, and success requires selecting all of them.The sparse answer reward is modeled through a monotone function of selected ground-truth chunks.
  • Reward design: An additive context reward is combined with the answer reward to provide chunk-level supervision.The total reward is defined as rtotal = rans + rctx.
  • Gradient analysis: The analysis uses the REINFORCE score-function identity and its covariance form to characterize reward gradients.A baseline independent of zj preserves the covariance interpretation.

A.2 PROOF OF PROPOSITION 1: VANISHING GRADIENTS FOR OUTCOME-ONLY REWARDS

The proof shows that outcome-only rewards provide a weak grounding gradient because a chunk receives useful credit only when related evidence has already been selected.

  • Proposition 1: The gradient for an essential chunk is controlled by its activation event Ej, where the chunk has positive marginal reward given the other selected evidence.The activation event captures the prerequisite selections needed for the chunk to receive credit.
  • REINFORCE proof: The REINFORCE gradient equals Cov(rans, zj), linking outcome-reward learning to the association between answer reward and chunk selection.The constant reward component drops out, leaving the covariance with the grounding-dependent reward.
  • Proposition 1: The gradient is bounded by pj(1 − pj) δ̄j Prθ(Ej), so it becomes small when the activation event is unlikely.Monotonicity also ensures the gradient is non-negative under the stated assumptions.
  • GRPO proof: Under GRPO, the gradient inherits the same activation-probability scaling as the REINFORCE result.The group-relative baseline changes the estimator form but not the vanishing-gradient dependence.

A.3 PROOF OF PROPOSITION 2: NON-VANISHING GROUNDING SIGNAL

The proof shows that an additive context reward contributes a dense per-chunk gradient independent of the rare activation event that suppresses outcome-only learning.

  • Proposition 2: The total reward combines the answer reward with an additive context reward weighted over essential chunks.The context reward is linear in the chunk-selection indicators.
  • Gradient decomposition: The answer-only gradient remains bounded by pj(1 − pj) δ̄j Prθ(Ej), while the context term contributes αj pj(1 − pj) without Prθ(Ej).The added term therefore remains available even when the answer-only signal is negligible.
  • Gradient decomposition: Non-negative correlations among related chunks can further increase the context-reward gradient through cross-covariance terms.The result is stated for grounding policies whose related chunk selections are non-negatively correlated.
  • REINFORCE proof: Under REINFORCE, the total gradient decomposes into answer-reward and context-reward covariance contributions.The context component supplies the dense per-chunk signal independent of the rare activation event.
  • GRPO and special case: The same non-vanishing context term appears under GRPO, and independent equal-weight selections reduce the expression to the simpler main-text formula.Under independence, cross-covariances vanish and Var(zj) equals pj(1 − pj).

B DATA CURATION AND GENERATION DETAILS

LongRLVR’s grounded QA dataset is built from diverse long documents, structured into evidence-oriented segments, and filtered through multi-stage generation and quality control. The process yields well-grounded question-answer pairs after selecting and rating candidate examples.

  • The corpus combines book and arXiv documents from Long-Data-Collection with code repositories from StarCoder, retaining documents between 8K and 64K tokens.
  • Each document is partitioned into exactly 64 sentence-aware segments and embedded with BGE-M3 before clustering-based evidence identification.
  • For each document, the pipeline samples four semantic clusters and generates three candidate (Q, y, G) tuples per cluster.
  • Candidates are rated from 1 to 10 for clarity, correctness, and evidence relevance, then filtered by selecting cluster winners and discarding pairs rated below 9.
  • The final dataset contains 46K documents, each paired with one high-quality, well-grounded question-answer pair.
  • A training example may require synthesizing evidence from multiple chunks, such as a war’s outcome, its cause, and biological and environmental extinction causes.
Loading 2603.02146v1…