PaperlayerView exact revision on arXiv

Source-linked AI summary

LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contexts

Siyuan Wang, Gaokai Zhang, Li Lyna Zhang, Ning Shang, Fan Yang, Dongyao Chen, Mao Yang

arXiv:2510.19363v2cs.CL

TL;DR

Long-context reasoning requires models to retrieve and reason over extensive external inputs, but advanced reasoning patterns and difficult verifiable RL data remain limited. LoongRL addresses this with KeyChain-synthesized tasks and RL, yielding stronger long-context QA and retrieval while preserving short-context capabilities.

  • Problem

    Advanced reasoning over long contexts remains underexplored, and difficult long-context RL data with reliably verifiable answers are scarce.

  • Method

    LoongRL uses KeyChain to hide questions behind UUID chains and distractor documents, then trains models with RL on these stepwise retrieval-and-reasoning tasks.

  • Results

    +23.5% and +21.1% absolute gains improve Qwen2.5-7B and 14B long-context multi-hop QA, while LoongRL-14B reaches 74.2 and models trained at 16K handle 128K tasks.

  • Takeaways & Limitations

    KeyChain RL induces a plan–retrieve–reason–recheck pattern that generalizes beyond training length, improves long-context retrieval, and preserves short-context reasoning.

Abstract

from arXiv · show

Reasoning over long contexts is essential for large language models. While reinforcement learning (RL) enhances short-context reasoning by inducing "Aha" moments in chain-of-thought, the advanced thinking patterns required for long-context reasoning remain largely unexplored, and high-difficulty RL data are scarce. In this paper, we introduce LoongRL, a data-driven RL method for advanced long-context reasoning. Central to LoongRL is KeyChain, a synthesis approach that transforms short multi-hop QA into high-difficulty long-context tasks by inserting UUID chains that hide the true question among large collections of distracting documents. Solving these tasks requires the model to trace the correct chain step-by-step, identify the true question, retrieve relevant facts and reason over them to answer correctly. RL training on KeyChain data induces an emergent plan-retrieve-reason-recheck reasoning pattern that generalizes far beyond training length. Models trained at 16K effectively solve 128K tasks without prohibitive full-length RL rollout costs. On Qwen2.5-7B and 14B, LoongRL substantially improves long-context multi-hop QA accuracy by +23.5% and +21.1% absolute gains. The resulting LoongRL-14B reaches a score of 74.2, rivaling much larger frontier models such as o3-mini (74.5) and DeepSeek-R1 (74.9). It also improves long-context retrieval, passes all 128K needle-in-a-haystack stress tests, and preserves short-context reasoning capabilities.

1 INTRODUCTION

LoongRL addresses the gap between retrieval and advanced reasoning over long contexts by using difficult synthetic RL tasks to induce transferable reasoning patterns. It improves long-context QA while retaining broader capabilities and extending from 16K training to 128K tasks.

  • 1 INTRODUCTION: Reasoning over long contexts remains difficult because existing RL mainly targets short inputs, while long-context models often excel at retrieval without deeply reasoning over documents.Effective training data must be difficult, retrieval-dependent, and reliably verifiable, but such data are scarce.
  • 1 INTRODUCTION: LoongRL uses KeyChain RL data to induce a plan–retrieve–reason–recheck pattern for long-context reasoning.KeyChain hides the true question among distracting documents, requiring stepwise chain tracing, retrieval, and reasoning.
  • 1 INTRODUCTION: +23.5% and +21.1% absolute accuracy gains improve Qwen2.5-7B and 14B on long-context multi-hop QA.LoongRL-14B reaches 74.2, near o3-mini at 74.5 and DeepSeek-R1 at 74.9.
  • 1 INTRODUCTION: Models trained at 16K generalize to 128K reasoning tasks without prohibitive full-length RL rollout costs.The approach also improves long-context retrieval and preserves short-context and general reasoning capabilities.

2 RELATED WORKS

Prior long-context reasoning work relies mainly on prompting, synthetic-data SFT, or longer-context RL, while advanced reasoning patterns remain underexplored. Existing synthesis methods lengthen inputs with distractors but do not address the deeper reasoning challenge.

  • 2 RELATED WORKS: Advanced long-context reasoning patterns remain limited compared with short-context reasoning methods that elicit self-reflection and “aha” moments.
  • 2 RELATED WORKS: Prompting is constrained by base-model reasoning capacity, while synthetic-data SFT can introduce noise or bias.
  • 2 RELATED WORKS: Existing long-context synthesis methods mainly pad questions with irrelevant documents or shuffle passages to increase context length.These approaches increase length but are presented as limited beyond simple context extension.

3 METHODOLOGY

LoongRL constructs difficult, verifiable KeyChain tasks from real multi-hop QA data and trains models with RL to reason through distractor-heavy contexts. Its curriculum and reward design support emergent structured reasoning while preserving broader capabilities.

  • 3.1 KEYCHAIN DATA CONSTRUCTION: KeyChain hides an original question behind UUID key-value chains, requiring chain tracing before long-context retrieval and reasoning.One chain resolves to the original question, while distracting chains resolve to alternative questions.
  • 3.1 KEYCHAIN DATA CONSTRUCTION: The dataset starts from real multi-hop QA and extends contexts with additional non-overlapping documents, producing approximately 16K-token distractor-heavy inputs.The source datasets are HotpotQA, MuSiQue, and 2WikiMultiHopQA.
  • 3.2 METHODOLOGY: KeyChain RL elicits a plan–retrieve–reason–recheck loop that learned at 16K generalizes to contexts up to 128K.The model plans subproblems, retrieves evidence, reasons over it, and re-checks retrieved content when uncertain.
  • 3.2.1 GROUP RELATIVE POLICY OPTIMIZATION FOR LONG-CONTEXT REASONING: A two-way substring exact-match verifier supplies binary rewards while tolerating valid answer variations and reducing reliance on LLM judging.The model outputs its final answer in boxed form for unambiguous extraction.

4 EXPERIMENTS

LoongRL achieves frontier-level long-context reasoning at smaller scales, generalizes from 16K training to 128K contexts, improves retrieval, and preserves short-context abilities. KeyChain data and multi-stage RL contribute to these gains.

  • Main results: 74.2 at 14B rivals o3-mini at 74.5 and DeepSeek-R1 at 74.9, while LoongRL-7B reaches 72.4 and surpasses QwenLong-L1-32B by 2.3 points.LoongRL improves Qwen2.5-7B-Instruct and Qwen2.5-14B-Instruct by +23.5% and +21.1%, respectively.
  • Main results: Training on 16K contexts generalizes to reasoning tasks up to 128K, where LoongRL maintains strong performance as other baselines degrade with longer contexts.The reported longer-context gains include +14.8% and +16.0% on NarrativeQA for the 7B and 14B models.
  • Main results: LoongRL preserves short-context capabilities, gaining +2.8% on MMLU at 7B and limiting IFEval declines to -0.3% and -2.6% at 7B and 14B.The approach also stably preserves the base models’ math ability.
  • Main results: LoongRL-7B achieves 100% retrieval accuracy across all Needle in a Haystack document depths, unlike the base model and other evaluated approaches.R1-Distill-7B cannot retrieve beyond 20K, and QwenLong-L1-32B does not fully pass the benchmark.
  • Ablation study: Long-context reasoning accuracy grows across the three RL stages while average response length steadily increases, supporting the multi-stage curriculum.
  • Ablation study: 72.4 with KeyChain data exceeds 66.2 from regular long-context QA under otherwise matched RL settings.Regular-data training lacks explicit planning and careful reasoning over retrieved information, producing a mixed reasoning-with-retrieval pattern.

5 CONCLUSION

LoongRL trains advanced long-context reasoning through KeyChain and induces a plan-retrieve-reason-recheck pattern that generalizes beyond training length. The resulting 14B model reaches 74.2 on long-context QA while preserving short-context capabilities and approaching larger frontier models.

  • 5 CONCLUSION: LoongRL trains models to develop a plan-retrieve-reason-recheck thinking pattern for advanced long-context reasoning.KeyChain transforms standard multi-hop questions into high-difficulty tasks.
  • 5 CONCLUSION: Models trained on 16K token contexts effectively solve tasks up to 128K tokens.
  • 5 CONCLUSION: 74.2: LoongRL-14B rivals much larger frontier models such as o3-mini and DeepSeek-R1 on long-context QA.
  • 5 CONCLUSION: LoongRL preserves short-context reasoning and retrieval capabilities while delivering long-context gains.

REPRODUCIBILITY STATEMENT

The paper supports reproducibility by documenting its training algorithm, hyperparameters, prompts, datasets, and supplementary implementation materials.

  • REPRODUCIBILITY STATEMENT: The paper provides GRPO algorithm and hyperparameter details in Sections 3.2.1 and 4.1.
  • REPRODUCIBILITY STATEMENT: The training prompt template is provided in Appendix A.2.
  • REPRODUCIBILITY STATEMENT: Supplementary materials include RL training code, KeyChain synthesis code, and representative synthesized-data samples.

A.1 USE OF LARGE LANGUAGE MODELS IN PAPER WRITING

The authors used LLMs as general-purpose tools for improving paper clarity and for evaluating an answer verifier in an ablation study.

  • A.1 USE OF LARGE LANGUAGE MODELS IN PAPER WRITING: LLMs were used to improve the clarity and readability of the paper.
  • A.1 USE OF LARGE LANGUAGE MODELS IN PAPER WRITING: DeepSeek-V3 served as the baseline in an LLM-as-a-judge evaluation of the answer verifier.
  • A.1 USE OF LARGE LANGUAGE MODELS IN PAPER WRITING: The reported LLM uses were limited to paper editing and answer-verifier evaluation.

A.2 TRAINING PROMPT TEMPLATE

The training prompt instructs the model to produce intermediate reasoning before its final answer, and the exact prompt format is documented in Figure 5.

  • A.2 TRAINING PROMPT TEMPLATE: The model generates intermediate reasoning inside <think> ... </think>.
  • A.2 TRAINING PROMPT TEMPLATE: The model provides its final answer inside \boxed{}.
  • A.2 TRAINING PROMPT TEMPLATE: Figure 5 documents the exact system prompt format used during training.

A.3 EXAMPLE OF KEYCHAIN-AUGMENTED TRAINING DATA

KeyChain augmentation structures long-context RL data around a hidden question and distracting documents, as illustrated by the training-data skeleton.

  • KeyChain augmentation hides the true question within a structured long-context training example.Figure 6 presents the skeleton of the augmented training data used for LoongRL.

A.4 COMPARISON BETWEEN TRACES OF KEYCHAIN VS. NON-KEYCHAIN MODELS

KeyChain-trained models produce more systematic reasoning traces than models trained without KeyChain data, which often shortcut retrieval and make errors.

  • KeyChain training elicits step-by-step reasoning and re-checking when ambiguity arises.Under identical hyperparameters, the KeyChain-augmented model traces the relevant information more systematically on the same question.
  • Non-KeyChain reasoning often shortcuts retrieval and begins reasoning prematurely, leading to errors.The comparison attributes the differing trace patterns to the training data while holding hyperparameters constant.

A.5 TRAJECTORY EXAMPLES ON LONG-CONTEXT RETRIEVAL

KeyChain data improves structured long-context retrieval, while LoongRL training shows staged gains and strong retrieval robustness across extended contexts.

  • KeyChain-trained models perform clearer, stepwise retrieval than non-KeyChain models on Ruler VT examples.Without KeyChain data, models can conflate intermediate variable assignments and propagate errors.
  • Training rewards and consistently correct predictions rise within each curriculum stage, reset at harder-task transitions, and increase again.The 7B and 14B schedules use different stage counts, with the 14B model skipping warm-up because of stronger base capabilities.
  • LoongRL models are evaluated on LongBench-v2 across difficulty, context length, and task type.Table 6 summarizes comparisons with baseline models under these groupings.
  • RULER evaluation uses distinct prompting and answer extraction procedures for models with versus without long-context reasoning abilities.Reasoning-capable models omit the completion suffix and generate explicit reasoning before answer extraction.
  • LoongRL-14B maintains strong needle-in-a-haystack retrieval accuracy across extended context lengths.Figure 9 reports the model’s robustness in extremely long-text retrieval settings.
Loading 2510.19363v2…