PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning

Xiaoyue Xu, Sikui Zhang, Xiaorong Wang, Xu Han, Chaojun Xiao

arXiv:2606.18831v1cs.CLcs.AI

TL;DR

Long-context RL research has focused largely on reward engineering despite scarce diverse training data. This paper proposes a data-centric recipe spanning three complementary task families and finds consistent gains across long-context benchmarks, with transfer to agentic tasks.

  • Problem

    Existing long-context RL work emphasizes reward engineering, while high-quality diverse training data remains scarce.

  • Method

    The paper constructs and curates training data across retrieval, multi-evidence synthesis, and reasoning, then evaluates it with minimal outcome-based RL.

  • Results

    +7.21, +3.17, and +6.41 points are gained across seven long-context benchmarks for the three Qwen3 models, with additional gains on GAIA and BrowseComp.

  • Takeaways & Limitations

    The data recipe provides a simple approach for improving long-context reasoning, with gains that transfer to agentic tasks.

  • Takeaways & Limitations

    Experiments cover only the Qwen3 family up to 30B-A3B, use inputs under 64K tokens, and rely mostly on synthetic datasets.

Abstract

from arXiv · show

Long-context reasoning is an essential capability for large language models, particularly when they are deployed as autonomous agents that must reason over lengthy trajectories. Reinforcement learning (RL) has recently emerged as a dominant paradigm for improving this ability, yet existing work largely focuses on reward engineering while diverse training data remains scarce. We revisit this problem from a data-centric perspective and show that a simple yet effective data recipe alone, paired with a minimal outcome-based GRPO setup, suffices to substantially improve long-context reasoning. Our recipe targets three complementary task families -- retrieval, multi-evidence synthesis, and reasoning -- for which we construct and curate eight datasets totaling ~14K examples. Experiments on three models (Qwen3-4B/8B/30B-A3B) yield average gains of +7.2/+3.2/+6.4 points across seven long-context benchmarks, surpassing prior RL training sets. We further demonstrate that these gains transfer to agentic tasks, where continuing RL training on an agent-tuned model with our data recipe improves GAIA by +4.8 and BrowseComp by +7.0 points. We will release our datasets to facilitate future research.

1 Introduction

The paper argues that diverse training data, rather than special reward engineering, is sufficient to improve long-context reasoning. Its recipe trains complementary retrieval, multi-evidence synthesis, and reasoning abilities, producing gains that transfer to agentic tasks.

  • Motivation: Long-context reasoning is essential for LLM agents handling lengthy inputs in web search, software engineering, and GUI automation.These settings contain scattered task-relevant evidence across web pages, code repositories, and action histories.
  • Motivation: Prior long-context RL work emphasizes auxiliary reward signals, while high-quality training data remains scarce and existing datasets are narrow or closed-source.The paper identifies reward design and limited data coverage as the dominant gaps in prior work.
  • Data recipe: The proposed data recipe jointly exercises Retrieval, Multi-evidence Synthesis, and Reasoning as complementary core abilities for long-context reasoning.Retrieval locates indirect evidence amid distractors, synthesis integrates evidence from different input regions, and reasoning performs complex mental operations.
  • Results: +7.2, +3.2, and +6.4 points are the average improvements achieved by Qwen3-4B-Thinking-2507, Qwen3-8B, and Qwen3-30B-A3B-Thinking-2507, respectively.The evaluation covers seven long-context benchmarks and uses no special reward engineering or staged RL training.
  • Agentic transfer: GAIA improves by 4.8 points and BrowseComp improves by 7.0 points when long-context RL continues on an agent-tuned model.The recipe also outperforms DocQA-RL-1.6K and KeyChain under the same RL setup.

2 Related Works

Related work improves long-context reasoning through post-training, with supervision curation as a central challenge. Long-context RL spans algorithm-centric methods that modify rewards or optimization and data-centric methods, while this work proposes a unified recipe covering three complementary abilities.

  • Long-context Reasoning via Post-training: Post-training work targets long-context reasoning, but curating diverse, high-quality long-context supervision remains a central challenge.Examples include LongAlign’s long instruction-following data, self-generated SFT examples, and long agent trajectories.
  • Reinforcement Learning in Long-context Scenarios: Long-context RL includes algorithm-centric approaches that improve reward signals or the optimization procedure.Reward-based methods add evidence-grounding or intermediate-step-quality signals to outcome-only RLVR rewards.
  • Our work: This work follows the data-centric line with a unified recipe exercising three complementary core abilities and producing consistent gains across long-context benchmarks.The passage contrasts this recipe with prior efforts without specifying their individual methods.

3 Methodology

The methodology combines a long-context data mixture spanning retrieval, multi-evidence synthesis, and reasoning with a minimal outcome-based GRPO setup. The datasets are curated to target specific long-context failure modes and retain moderately difficult examples for informative RL signals.

  • RL setup: Training uses Group Relative Policy Optimization with an outcome-based reward, and omits the KL regularization term.The methodology describes a minimal GRPO setup, following recent reasoning-RL practice for omitting KL regularization.
  • Training mixture: The training mixture targets Retrieval, Multi-evidence Synthesis, and Reasoning, with each category isolating a distinct ability essential to long-context reasoning.The methodology analyzes category-specific difficulties, constructs corresponding datasets, and concludes with an overall data recipe and curation process.
  • Retrieval: Retrieval tasks use FuzzyNeedle to eliminate keyword overlap and MultiNeedle to retrieve the K-th conversation among many near-duplicate conversations.FuzzyNeedle requires resolving an is-instance-of relation and rejecting distractors, while MultiNeedle requires distinguishing near-duplicate needles by occurrence order.
  • Multi-evidence Synthesis: Multi-evidence synthesis tasks address shortcut use and incomplete coverage through CrossEntity aggregation, WebSearch multi-hop chains, and MultiQuery evidence-collection requirements.CrossEntity answers require synthesizing derived attributes across entities; WebSearch requires tracing relation chains amid distractors; MultiQuery makes every question a required evidence piece.
  • Reasoning: Reasoning data adapts difficult short-context math problems into long narrative scenarios, distributing variables and numerical conditions across at least five story fragments.A second LLM verifies the rewrites and filters out unsolvable problems in LongMath.
  • Data curation: 14,069 examples remain after removing inputs above 64K tokens and retaining only questions whose four Qwen3-30B-A3B-Thinking-2507 rollouts are neither all-correct nor all-wrong.This postprocessing keeps rollout costs tractable and isolates moderate-difficulty problems that provide informative RL learning signals.

4 Experiments

Experiments evaluate the data recipe across seven long-context benchmarks, three Qwen3 models, and comparisons with two prior long-context RL datasets. The recipe improves performance consistently across models and benchmarks, with gains that extend to longer contexts than used during training.

  • Benchmarks: Seven benchmarks probe multi-hop QA and holistic long-context reasoning across real-world documents and challenging reasoning tasks.The evaluation suite includes LongBench v1 QA subsets, FRAMES, LongBench v2, and AA-LCR.
  • Main results: +7.2, +3.2, and +6.4 average score gains occur for Qwen3-4B-Thinking-2507, Qwen3-8B-128K, and Qwen3-30B-A3B-Thinking-2507, respectively.All seven benchmarks improve for the 4B and 30B-A3B models, while six of seven improve for Qwen3-8B-128K.
  • Comparison with prior long-context RL data: +2.66, +1.02, and +3.66 average-score advantages over KeyChain-15K occur on the 4B, 8B, and 30B-A3B models, respectively.The recipe achieves the highest average score on every model and delivers the largest gains on most benchmarks, especially reasoning-heavy benchmarks.
  • Generalization to Longer Context: +8.6 improvement over the base model appears in the 0–64K LongBench v2 bucket despite training being capped at 64K tokens.Positive gains are retained on longer input-length buckets, and evaluation extends to a maximum input length of 230K tokens.

5 Analysis

The analysis finds that task diversity and task balancing materially improve long-context RL, while the resulting skills transfer to long-horizon agentic tasks. It also evaluates a generic LLM-as-judge process reward without requiring additional evidence annotations.

  • Task categories: Reasoning alone yields the largest single-category gain, improving the average by +3.46, LongBench v2 by +4.13, and LongReason by +5.56.Every single category and pairwise combination improves the average and most individual benchmarks, supporting the proposed task categorization.
  • Task balancing: Removing task balancing decreases the average score from 57.58 to 56.05 (-1.53), indicating inter-task competition in multi-task long-context RL.Task balancing combines task-balanced sampling with task-level advantage normalization.
  • Reward design: The study tests a generic LLM-as-judge process reward that scores internal coherence and whether retrieved information is correctly synthesized into the final answer.Because the training mixture contains no evidence annotations, DeepSeek-V3.2 serves as the judge.
  • Agentic transfer: +4.8 GAIA Pass@3 and +7.0 BrowseComp points are achieved by the 50-step checkpoint over AgentCPM-Explore.The agent-tuned model is trained with the proposed long-context recipe and the same RL configuration as the main experiments.

6 Conclusion

The paper concludes that a simple data recipe combining retrieval, multi-evidence synthesis, and reasoning consistently improves long-context reasoning across seven benchmarks. These gains also transfer to agentic tasks, improving GAIA and BrowseComp through continued RL training.

  • The data recipe combines retrieval, multi-evidence synthesis, and reasoning to improve long-context RL across seven long-context reasoning benchmarks.
  • 60.63, 53.10, and 68.23 are the average scores for Qwen3-4B-Thinking-2507, -8B, and -30B-A3B-Thinking-2507, with gains of +7.21, +3.17, and +6.41 over base models.
  • +4.8 on GAIA and +7.0 on BrowseComp result from continuing RL training of an agent-tuned model with the data recipe.The conclusion identifies transfer of long-context gains to agentic tasks.

Limitations

The study’s validation is limited to the Qwen3 family through 30B-A3B-Thinking and RL inputs below 64K tokens. Its predominantly synthetic datasets may not fully reflect real-world long-context inputs.

  • Validation covers only Qwen3 models up to 30B-A3B-Thinking, excluding larger models and other model families because of computational constraints.
  • RL training is restricted to inputs under 64K tokens, leaving longer context lengths unvalidated.
  • Most training datasets are synthetic and may not fully match the distribution of real-world long-context inputs.The synthetic datasets nevertheless improved long-context reasoning in the reported experiments.

Ethics Statement

The work does not involve individual information or offensive content. Its training data is synthesized or drawn from public datasets, and released data will follow the sources’ original licenses.

  • Ethics Statement: The study uses no individual information or offensive content, and its training data is synthesized or sourced from publicly available datasets.Released data will follow the original licenses of its sources.

A Training Details

RL training uses Megatron-LM with SGLang rollouts on H100 GPUs, with experiment runtimes varying by model. CrossEntity and LongMath are scored with an LLM-as-judge, and the training mixture’s input-length distributions are documented.

  • Infrastructure: Megatron-LM is used as the training backend, SGLang for rollout, and all experiments run on H100 GPUs.These components define the implementation stack for the RL experiments.
  • Infrastructure: ∼22 hours, ∼50 hours, and ∼14 hours are required per experiment for Qwen3-4B-Thinking-2507, Qwen3-8B, and Qwen3-30B-A3B-Thinking-2507, respectively.The first two models use 100 steps and 16 GPUs; the 30B-A3B model uses 50 steps and 32 GPUs.
  • Evaluation: An LLM-as-judge scores CrossEntity and LongMath during training.The judge prompt is provided in Figure 4.
  • Data: The training mixture’s input-length distributions are summarized across its datasets.Figure 3 reports these distributions in characters.

B Evaluation Details. · C Agent Evaluation Details · C.1 Agent Evaluation Configurations

Evaluation uses DeepSeek-V3.2-based judging or rule-based answer comparison, with detailed HELMET scores reported by input length. Agent evaluation follows Agent-ToLeaP with web search, URL fetching, Python execution, a 240k-token context window, capped interaction outputs, sampled benchmarks, and real-time leakage filtering for GAIA.

  • B Evaluation Details.: DeepSeek-V3.2 serves as the LLM judge throughout all evaluations.For LongBench v1 QA and FRAMES, the reported score is the maximum of the LLM-judge score and substring exact match.
  • B Evaluation Details.: LongBench v2, LongReason, and GraphWalks use rule-based answer extraction and comparison.For DocFinQA and AA-LCR, the LLM judge assesses whether the prediction matches the gold answer.
  • B Evaluation Details: Detailed HELMET scores are reported at different input lengths in Table 7.The table compares Qwen3-4B-Thinking-2507 with the paper’s RL-trained variant.
  • C Agent Evaluation Details: Agent evaluation adopts the Agent-ToLeaP framework used by AgentCPM-Explore.The agent has Google Serp API web search, URL fetching with a DeepSeek-V3.2 browser processor, and a sandboxed Python executor.
  • C.1 Agent Evaluation Configurations: 240k-token context windows and 16,384 maximum output tokens per interaction are configured for agent evaluation.The 16,384-token output limit applies to both GAIA and BrowseComp, with SGLang as the inference engine.
  • C.1 Agent Evaluation Configurations: 127 GAIA text-only validation examples and a 100-example BrowseComp subset are evaluated.The BrowseComp subset is sampled from the full test set because multi-round agentic interactions are time-consuming.
  • C.1 Agent Evaluation Configurations: A real-time leakage filter removes search or fetch results containing both the original GAIA question substring and the gold answer verbatim.This addresses frequent online leakage of GAIA test cases and prevents search tools from directly retrieving leaked answers during evaluation.

C.2 Additional Benchmark Details · D Dataset Licenses

The additional benchmark details report per-benchmark and difficulty-stratified evaluations, showing especially large gains on harder GAIA and BrowseComp subsets. The paper also documents external dataset licenses and states that remaining self-constructed data will be released permissively.

  • C.2 Additional Benchmark Details: Average Pass@3 on GAIA and BrowseComp is reported in Table 9.The table provides the aggregate agentic benchmark results corresponding to the additional benchmark analysis.
  • C.2 Additional Benchmark Details: Per-benchmark long-context scores for AgentCPM-Explore and its RL-trained variants are reported in Table 8 alongside the averaged Avg. LC column from Table 5.These results provide benchmark-level detail beyond the aggregate long-context score.
  • C.2 Additional Benchmark Details: Pass@3 is stratified by official GAIA difficulty levels and consensus-based BrowseComp difficulty labels.BrowseComp labels use 3 models × 3 rounds = 9 trials per question: easy (6–9), med (3–5), hard (1–2), and vhard (0).
  • C.2 Additional Benchmark Details: An LLM-as-judge prompt is provided for AA-LCR and DocFinQA evaluation.The prompt is shown in Figure 6.
  • C.2 Additional Benchmark Details: 42.1 →52.6 on GAIA L3 and 28.6 →57.1 on BrowseComp Hard are the most pronounced gains for the 50-step checkpoint.The gains are concentrated on harder subsets.
  • D Dataset Licenses: Table 12 lists the licenses of all external datasets used in the work, and the training recipe uses only their official training splits.The remaining mixture datasets are constructed by the authors and will be released under a permissive license.
  • D Dataset Licenses: LoongRL KeyChain training data does not specify an explicit repository license, although its underlying sources are all under permissive licenses.This licensing status is noted separately from the external datasets listed in Table 12.
Loading 2606.18831v1…