PaperlayerView exact revision on arXiv

Source-linked AI summary

LongR: Unleashing Long-Context Reasoning via Reinforcement Learning with Dense Utility Rewards

Bowen Ping, Zijun Chen, Yiyao Yu, Tingfeng Hui, Junchi Yan, Baobao Chang

arXiv:2602.05758v1cs.CL

TL;DR

Long-context reinforcement learning is limited by sparse outcome rewards that inadequately guide complex reasoning over extensive documents. LongR addresses this with a dynamic Think-and-Read mechanism and relative-information-gain rewards, achieving approximately 9% improvement on LongBench v2 and consistent gains across other benchmarks and RL algorithms.

  • Problem

    Sparse outcome-only rewards provide limited gains for long-context reinforcement learning, where reasoning must support rigorous deduction over extensive documents.

  • Method

    LongR interleaves reasoning with document consultation and uses relative information gain to provide dense contextual utility supervision.

  • Results

    Approximately 9% improvement is reported on LongBench v2, with consistent gains on RULER, InfiniteBench, and across DAPO, GSPO, and CISPO.

  • Takeaways & Limitations

    LongR provides a unified long-context reasoning framework that improves performance across multiple benchmarks and reinforcement learning algorithms.

  • Takeaways & Limitations

    Absolute-density rewards can be hacked through verbose or esoteric segments, while threshold-density rewards lack granularity for distinguishing moderately relevant text from precise evidence.

Abstract

from arXiv · show

Reinforcement Learning has emerged as a key driver for LLM reasoning. This capability is equally pivotal in long-context scenarios--such as long-dialogue understanding and structured data analysis, where the challenge extends beyond consuming tokens to performing rigorous deduction. While existing efforts focus on data synthesis or architectural changes, recent work points out that relying solely on sparse, outcome-only rewards yields limited gains, as such coarse signals are often insufficient to effectively guide the complex long-context reasoning. To address this, we propose LongR, a unified framework that enhances long-context performance by integrating a dynamic "Think-and-Read" mechanism, which interleaves reasoning with document consultation, with a contextual density reward based on relative information gain to quantify the utility of the relevant documents. Empirically, LongR achieves a 9% gain on LongBench v2 and consistent improvements on RULER and InfiniteBench, demonstrating robust efficiency in navigating extensive contexts. Furthermore, LongR consistently enhances performance across diverse RL algorithms (e.g., DAPO, GSPO). Finally, we conduct in-depth analyses to investigate the impact of reasoning chain length on efficiency and the model's robustness against distractors.

1. Introduction

LongR addresses the difficulty of rigorous reasoning over long contexts by interleaving document consultation with reasoning and providing dense contextual utility feedback. It reports gains on LongBench v2 and broader evaluations, while analyzing efficiency and distractor robustness.

  • Motivation: Long-context reasoning requires rigorous deduction over content, not merely processing vast numbers of input tokens.
  • Motivation: Prior approaches emphasize synthesized reasoning data or architectural changes, while sparse outcome rewards provide limited gains for long-context reinforcement learning.
  • Approach: LongR interleaves reasoning with document consultation in standard autoregressive generation, avoiding rigid document chunking and preserving information integrity.
  • Approach: LongR uses relative information gain to quantify retrieved-document utility and provide dense supervision that encourages high-value context use.
  • Results: Approximately 9% improvement is reported on LongBench v2, with further gains across RULER, InfiniteBench, and DAPO, GSPO, and CISPO.
  • Analysis: The paper analyzes reasoning-chain length for computational efficiency and evaluates LongR’s robustness against distractors.

2. Related Work

Related work adapts reinforcement learning to long-context reasoning through architectural changes, synthesized data, or auxiliary rewards. LongR distinguishes itself by avoiding rigid segmentation and learned reward-model co-evolution while using relative information gain for dense supervision.

  • Prior approaches: Long-context reinforcement learning research includes architectural innovations, synthesized reasoning data, and chunk-based supplementation of sparse outcome rewards.
  • Prior approaches: Pre-segmented chunks can constrain applicability by risking information integrity and requiring formats that diverge from natural user queries and mainstream post-training distributions.
  • LongR: LongR avoids rigid segmentation and uses relative information gain as an intrinsic dense reward for long-context reasoning.
  • Reward design: Learned reward models may introduce instability and computationally expensive reward-policy co-evolution, whereas LongR uses an information-based reward design.

3. Method

LongR formulates long-context question answering as reinforcement learning and combines supervised warm-up, interleaved document consultation, contextual utility rewards, and curriculum learning. Its dense reward addresses sparse outcome supervision by rewarding information gained from consulted context.

  • Problem Formulation: LongR formulates long-context question answering as an MDP in which the policy processes a query and lengthy document to generate an answer under a reference-model trust-region constraint.The reward evaluates reasoning-chain correctness and quality.
  • Supervised Fine-tuning (Cold Start): SFT warm-up initializes the policy from high-quality expert reasoning trajectories and enforces structural requirements needed for subsequent group-based exploration.The process injects domain knowledge while teaching the required output format.
  • Contextual Reward: LongR augments format and answer-correctness rewards with a contextual reward that quantifies the information utility of consulted document segments.The framework integrates contextual dense rewards with curriculum learning.
  • Intrinsic Grounding Reward via Information Gain: The Think-and-Read process identifies context segments explicitly consulted during reasoning, then measures their utility through verifier-based information gain.The utility is framed as uncertainty reduction when context helps explain a consulted segment.
  • Intrinsic Grounding Reward via Information Gain: Relative information gain normalizes contextual utility by prior uncertainty, producing a bounded reward that encourages context with greater relative explanatory power.The trajectory-level contextual reward averages scores across unique quotes and is zero when no quotes are generated.
  • Curriculum Learning: Curriculum learning progressively increases the maximum document length, beginning with shorter documents before reaching the target context length.The schedule can double the length limit across stages, such as 16K, 32K, and eventually Lmax.

4. Experiments

LongR is evaluated across multiple long-context benchmarks, reward designs, verifier sizes, and RL algorithms. Results show broad gains, while ablations identify relative information gain and a 4B verifier as effective choices.

  • Main Results: LongR achieves 36.23 on LongBench v2 for Qwen3-8B, improving over DAPO by +3.39 and SFT by +9.59.For Qwen3-4B, LongR reaches 35.44, outperforming DAPO by +5.03 and SFT by +9.79.
  • Main Results: LongR improves challenging LongBench v2 examples, with the 8B Hard split rising from 25.65 to 33.20 and the Long split from 23.35 to 39.12.The 4B Hard split rises from 28.10 to 35.61.
  • Long-context Benchmarks: LongR outperforms outcome-only DAPO on RULER, with 8B gains of +6.60 at 128K and +16.96 at 64K context lengths.The 4B model also leads in both settings, scoring 60.71 at 128K and 69.43 at 64K.
  • Long-context Benchmarks: LongR achieves the highest InfiniteBench average scores, 50.18 for 8B and 47.60 for 4B.It improves En.MC from 54.15 to 58.45 for 8B and En.QA from 12.68 to 21.85 for 4B.
  • Reward Design: Relative Information Gain yields superior reward-design performance, whereas outcome-only, absolute-density, and threshold-density variants have identified shortcomings.Outcome-only rewards are insufficient; absolute density is unbounded and complexity-sensitive; threshold density lacks granularity.
  • Verifier Scaling: A 4B verifier captures most verifier-scaling gains, improving Qwen3-8B from 34.59 to 36.23 over 1.7B while 32B reaches 36.83.The authors therefore adopt Qwen3-4B as the default verifier.
  • RL Algorithms: CISPO and GSPO improve over SFT across both model sizes, while their gains concentrate differently across length and difficulty splits.CISPO is strongest on the 8B Medium split at 38.61%, whereas GSPO leads on the 8B Hard and 4B Long splits.

5. Analysis

LongR improves long-context grounding without substantially increasing reasoning length or indiscriminately copying distractors. Its verifier-size gains are positive but diminish as the verifier grows.

  • Verifier analysis: Verifier-size comparisons use 1.7B, 4B, and 32B Qwen3 verifiers while holding other components fixed on LongBench v2.The corresponding reasoning-chain comparisons are reported for 8B and 4B models.
  • Evaluation setting: LongR is evaluated on RULER NIAH across context lengths from 8k to 128k, where one relevant fact is embedded in unrelated text.The benchmark separates relevant evidence from irrelevant noise, with accuracy reported in Figures 8 and 9.
  • Reasoning efficiency: LongR maintains an average token count comparable to the Outcome-only baseline, and sometimes slightly lower at 32k and 128k contexts.This argues against length inflation as the primary source of its gains.
  • Reward behavior: LongR uses a surgical extraction strategy because Relative Information Gain assigns negligible scores to irrelevant or parametric content.The reward therefore penalizes adding noise rather than rewarding indiscriminate copying.
  • Distractor robustness: LongR’s distractor usage profile remains nearly identical to Outcome-only across context lengths on the NIAH task.The model does not resort to indiscriminate quotation to accumulate dense rewards.

6. Conclusion

LongR combines interleaved reasoning and document consultation with a relative-information-gain reward to address sparse feedback in long-context reasoning.

  • Conclusion: LongR introduces a “Think-and-Read” pattern alongside a contextual dense reward based on relative information gain.The reward measures the utility of retrieved context.

A.1. Templates

The templates contrast outcome-only reinforcement learning with interleaved reasoning and document consultation. The latter requires structured reasoning paths supported by quoted evidence.

  • Outcome-only RL: The outcome-only RL template is specified by the system prompt shown in Figure 5.This template provides the comparison condition for the interleaved approach.
  • Interleaved reasoning and consultation: The interleaved template instructs the model to reason while consulting documents and to generate structured paths with quoted evidence.It operationalizes evidence seeking within the reasoning process.

A.2. Training and Evaluation Details

The experiments train Qwen3 models with a two-stage reinforcement-learning setup and evaluate them across three long-context benchmarks using controlled baselines and reproducible settings.

  • Training setup: Training uses Qwen3 models at 4B and 8B scales on 16 NVIDIA H20 141GB GPUs.The implementation uses LLaMA-Factory for supervised fine-tuning and verl with DAPO for reinforcement learning.
  • Curriculum learning: The curriculum limits context length to 16K in stage m = 0 and extends it to 32K in stage m = 1.Each stage comprises 100 steps with a constant learning rate of 1e-6.
  • Training data: The training data includes 1,000 synthesized samples for each 16K and 32K configuration, plus DocQA-RL-1.6K in the second stage.The additional dataset is incorporated during m = 1.
  • Reward configuration: The reward combines format validity, answer accuracy, and contextual density computed with a frozen Qwen3-4B verifier.The format and accuracy rewards are set to 1.0 and 2.0, respectively.
  • Evaluation: Evaluation covers LongBench v2, RULER, and InfiniteBench using Qwen3-Base, reproduced Qwen3-SFT, and Qwen3-SFT with DAPO baselines.Official scripts and fixed random seeds support controlled comparison.

A.3. Stage-wise Performance of Curriculum Learning

Curriculum learning improves performance progressively as context length increases, with gains reported for both Qwen3-8B and Qwen3-4B. The strongest reported improvements include the Hard and Long subsets for Qwen3-8B.

  • A.3. Stage-wise Performance of Curriculum Learning: Qwen3-8B Overall score rises from 26.64 (SFT) to 32.41 after 16K warm-up and 36.23 after extending to 32K.The stages are labeled m=0 and m=1, respectively.
  • A.3. Stage-wise Performance of Curriculum Learning: Qwen3-8B reaches scores of 33.20 on Hard and 39.12 on Long after curriculum learning.These are the largest reported gains among the listed subsets.
  • A.3. Stage-wise Performance of Curriculum Learning: Qwen3-4B Overall performance increases from 25.65 to 31.56 after Stage m=0 and then to 35.44 after Stage m=1.The passage reports balanced gains across Difficulty and Length splits.
Loading 2602.05758v1…