PaperlayerView exact revision on arXiv

Source-linked AI summary

GoLongRL: Capability-Oriented Long Context Reinforcement Learning with Multitask Alignment

Minxuan Lv, Tiehua Mei, Tanlong Du, Junmin Chen, Zhenpeng Su, Ziyang Chen, Ziqi Wang, Zhennan Wu, Ruotong Pan, jian Liang, Ruiming Tang, Han Li

arXiv:2605.19577v1cs.CL

TL;DR

Long-context post-training remains difficult because existing reinforcement-learning methods offer narrow task coverage and struggle with heterogeneous rewards. GoLongRL introduces a capability-oriented 23K-sample RLVR dataset spanning 9 task types and TMN-Reweight for multitask optimization, raising Qwen3-4B-Thinking’s benchmark average from 53.0 to 63.0 while preserving reported general capabilities.

  • Problem

    Effective long-context post-training remains a bottleneck, while existing methods provide narrow task coverage and inadequate reward formulations for practical requirements.

  • Method

    GoLongRL constructs 23K RLVR samples across 9 capability-oriented task types with task-specific metrics and combines task-level normalization with difficulty-adaptive reweighting.

  • Results

    53.0 to 63.0: On Qwen3-4B-Thinking, vanilla GRPO with the capability-oriented dataset raises the long-context average to 62.2, while TMN-Reweight raises it further.

  • Takeaways & Limitations

    The open dataset, construction pipeline, and training code support long-context capability improvement with heterogeneous rewards while reported general capabilities remain stable or improve.

  • Takeaways & Limitations

    Difficulty-reweighting effectiveness varies by model scale, with clearer gains at 4B and less settled evidence at larger scales; MoE-specific integration remains future work.

Abstract

from arXiv · show

We present GoLongRL, a fully open-source, capability-oriented post-training recipe for long-context reinforcement learning with verifiable rewards (RLVR). Existing long-context RL methods often treat data construction as a matter of designing increasingly complex retrieval paths, leading to homogeneous task coverage and reward formulations that inadequately reflect practical long-context requirements. Our work offers two contributions. (1) Capability-oriented data construction with full open release. We openly release a dataset of 23K RLVR samples, the complete construction pipeline, and all training code. Guided by a taxonomy of long-context capabilities, the dataset spans 9 task types, each paired with its natural evaluation metric. It comprises curated open-source samples from established corpora and synthetic samples whose QA pairs are generated from real source documents such as books, academic papers, and multi-turn dialogues. Under the same vanilla GRPO setup, our dataset alone outperforms the closed-source QwenLong-L1.5 dataset. Moreover, our Qwen3-30B-A3B model trained on this data delivers long-context performance comparable to DeepSeek-R1-0528 and Qwen3-235B-A22B-Thinking-2507, suggesting that broader coverage and greater reward diversity substantially benefit long-context capability improvement. (2) TMN-Reweight for heterogeneous multitask optimization. To address optimization challenges from heterogeneous rewards, we propose TMN-Reweight, which combines task-level mean normalization for cross-task reward scale alignment with difficulty-adaptive weighting for more reliable advantage estimation. TMN-Reweight further improves average performance over vanilla GRPO, with general capabilities preserved or improved across reported evaluations.

1. Introduction

GoLongRL addresses narrow task coverage and uniform rewards in long-context RL with a capability-oriented dataset spanning 9 task types and 23K RLVR samples. Its natural task metrics and TMN-Reweight improve long-context performance while preserving or improving general capabilities.

  • Motivation: Long-context reasoning supports multi-document analysis, deep research, RAG, and agentic workflows over tens or hundreds of thousands of tokens.These practical tasks require coherent understanding across very long inputs.
  • Limitations: Existing RL methods rely on complex retrieval paths that yield narrow task coverage, artificial difficulty structures, and overly uniform reward design.The cited examples include UUID chain tracking and chunk-based question answering.
  • Data: 23K RLVR samples across 9 task types use each task’s natural evaluation metric as its reward function.The dataset is designed around core long-context understanding capabilities and heterogeneous reward functions.
  • Results: 53.0→62.2 and 60.1→69.8 are the long-context average improvements for Qwen3-4B-Thinking and Qwen3-30B-A3B under the same GRPO setting.At both scales, vanilla GRPO on the capability-oriented dataset outperforms QwenLong-L1.5 trained with GRPO.
  • Algorithm: TMN-Reweight raises the 4B-model average to 63.0 and consistently improves over vanilla GRPO by addressing reward-scale inconsistency and difficulty-induced advantage bias.The method combines task-level reward normalization with difficulty-aware optimization, as described in the introduction’s contribution summary.
  • Generalization: Capability-oriented long-context RL preserves or improves general reasoning and memory, including transfer to unseen agentic-memory and long-term-dialogue-memory benchmarks.Reported evaluations include MMLU-Pro, AIME24/25, and GPQA.

2. Related Work

Related work applies RL to long-context understanding, develops GRPO variants for heterogeneous multitask rewards, and explores token-level weighting. GoLongRL builds on these directions while targeting advantage-estimation challenges in heterogeneous long-context RL.

  • Long-Context RL Training: Recent methods use reinforcement learning to improve long-context understanding beyond supervised fine-tuning, including compositional multi-hop training and denser reward designs.LoongRL transfers reasoning patterns from 16K training to 128K evaluation, while LongRLVR addresses exponential gradient attenuation from sparse final-answer rewards.
  • GRPO Variants for Multitask RL: RLVR uses rule-based metrics such as EM, F1, and NDCG, while GRPO estimates advantages from normalized within-prompt rewards.The supplied formulation is A_i = (r_i−μ_u)/σ_u.
  • Difficulty bias and its corrections: Dr. GRPO removes per-prompt standard-deviation normalization, whereas F-GRPO downweights easy prompts but relies on raw rewards for difficulty estimation.Dr. GRPO argues σ_u can inflate advantages for easy and hard prompts while suppressing medium-difficulty samples; the supplied F-GRPO discussion is truncated after noting reliability concerns.
  • Cross-task scale normalization: QwenLong-L1.5 normalizes advantages within each task type to reduce cross-task reward-scale differences.This provides a task-specific normalization strategy for multitask RL.
  • Token-Level Importance Weighting for Long-Context Training: Token-level long-context methods use context-aware denoising, revised perplexity, and informative-token weighting, conceptually aligning with response-level difficulty-adaptive reweighting.These approaches primarily operate in supervised settings and make position- and content-dependent weighting explicit.

3. Data for Capability-Oriented Long-Context RLVR

GoLongRL constructs a capability-oriented long-context RLVR dataset spanning nine capabilities, heterogeneous natural reward metrics, real-document sources, and context lengths up to 256K tokens. Its 22,965 samples combine curated and synthetic data, with task coverage and source design intended to broaden long-context training beyond repeated retrieval patterns.

  • Design principles: 9 core capabilities each receive a dedicated task, broadening training beyond repeated retrieval patterns through balanced capability coverage.The framework starts from required long-context capabilities and aligns task design with each capability.
  • Design principles: Natural evaluation metrics serve as task rewards, preserving graded or structured information that a single metric such as Exact Match would discard.Ranking tasks use NDCG because their outputs contain graded relevance judgments.
  • Dataset composition: 22,965 samples cover 9 tasks across context lengths from 0.1K to 256K tokens, with most samples concentrated between 4K and 256K.The dataset intentionally provides broad coverage across both task types and context lengths.
  • Data sources: Approximately 14K curated open-source samples and 9K synthetic samples form two complementary pools, with synthetic QA generated from real source documents.The sources include books, academic papers, legal filings, and financial reports, prioritizing natural document structure over template regularities.
  • Synthetic data construction: T2 is the primary large-scale synthesis target because multiple-choice QA supports reliable automatic generation and verification, while T1 adds approximately 4K exact-match needle-in-a-haystack calibration samples.More complex rewards, including token-level F1, structured IoU, and ranking-based NDCG, are harder to verify automatically.
  • Quality control: 31% of samples achieve a perfect pass rate under the 4B evaluator, 58% fall in the intermediate range, and 11% are never solved.Quality-insufficient samples with pass rates below 0.25 are discarded as likely label noise rather than genuine difficulty.

4. TMN-Reweight for Multitask Long-Context RL

TMN-Reweight addresses heterogeneous multitask long-context RL by separating cross-task reward-scale normalization from difficulty correction. It combines task-level normalization with difficulty-adaptive four-quadrant reweighting to produce more comparable gradient signals while preserving within-task difficulty information.

  • Motivation: 9 reward functions with different scales and variance profiles create optimization challenges for standard GRPO in multitask long-context training.The section identifies difficulty-induced advantage bias and cross-task reward-scale inconsistency as the two central defects.
  • Motivation: Difficulty-induced bias amplifies advantages for easy and hard prompts while compressing medium-difficulty prompts, which often provide the most informative learning signals.Dr. GRPO removes per-prompt standard-deviation normalization, but this alone leaves cross-task scale differences unresolved.
  • Motivation: Cross-task reward-scale inconsistency lets high-variance tasks generate disproportionately large gradients and dominate optimization after per-prompt normalization is removed.The relevant metrics include EM, F1, NDCG, and ROUGE-L, whose reward distributions differ in scale and variance profiles.
  • TMN-Reweight: TMN replaces per-prompt normalization with task-level normalization, reducing cross-task scale differences while preserving relative advantage magnitudes and within-task difficulty structure.The task denominator is shared across prompts within the same task, leaving difficulty correction for the second transformation.
  • TMN-Reweight: 0.54 is Dr. GRPO’s cross-task coefficient of variation, versus 0.34 for GRPO, showing that removing normalization increases disparity while per-prompt normalization only partially reduces it.Figure 5 evaluates normalized mean absolute advantage per task and uses lower coefficient of variation to indicate more uniform gradient magnitudes.
  • TMN-Reweight: TMN-Reweight combines task-level normalization with difficulty-adaptive four-quadrant reweighting, and the two components are empirically complementary.The reweighting addresses difficulty bias while avoiding symmetric scaling concerns for hard and easy prompts.

5. Experiments

Experiments evaluate GoLongRL across diverse long-context and general-capability benchmarks under controlled comparisons between vanilla GRPO and TMN-Reweight. TMN-Reweight improves aggregate long-context performance, preserves or enhances general capabilities, and transfers to longer contexts.

  • Training setup: The 23k-instance dataset is subsampled to 8k for Qwen3-4B-Thinking-2507, while Qwen3-30B-A3B-Thinking-2507 uses the full dataset under strictly on-policy training.All 8k-subset methods share identical training conditions, with IcePop used to mitigate train–inference mismatch in long-context settings.
  • Evaluation: Evaluation covers six long-context benchmarks spanning comprehension, retrieval, multi-hop reasoning, and related capabilities, plus MMLU-Pro, AIME24/25, GPQA-Diamond, BFCL-V4, and LongMemEval for general capability retention.The evaluation follows QwenLong-L1.5’s protocol for fair comparison.
  • Main results: 0.8 points: TMN-Reweight raises average performance from 62.2 to 63.0 over vanilla GRPO, with gains on CorpusQA (+4.5) and LBV2 (+1.6).Vanilla GRPO remains stronger on MRCR, scoring 67.5 versus 65.5 for TMN-Reweight.
  • Main results: TMN-Reweight achieves the best or second-best result on five of six sub-tasks, whereas vanilla GRPO leads retrieval-oriented MRCR with 67.5 and QwenLong-L1.5 scores 47.9.The results indicate broader robustness across heterogeneous long-context reasoning tasks rather than maximizing a single-task score.
  • Effect of α: α = 0.8 gives the best average performance, outperforming α = 1.0 (pure prompt-level, 61.5) and α = 0.0 (pure task-level, 61.3).The intermediate setting balances prompt-specific granularity with task-level stability; larger α can increase variance, while smaller α loses granularity.
  • General capability retention: +13.6 on LongMemEval: both 4B and 30B models improve dialogue memory, while the 4B model also gains on MMLU-Pro (+0.7), AIME24 (+1.6), AIME25 (+0.6), and GPQA-Diamond (+2.3).At longer contexts, the 30B model improves MRCR 128K–512K by +12.61, MRCR 512K–1M by +5.45, and CorpusQA 1M by +2.74.

6. Conclusion and Future Work

GoLongRL addresses narrow long-context data coverage and unstable heterogeneous-reward optimization through a capability-oriented framework with a 23K-sample, 9-task dataset and TMN-Reweight. Its open-source release preserves general capabilities, while future work targets scale-dependent reweighting, multi-document reasoning, and finer-grained context-aware RLVR training.

  • Conclusion: GoLongRL targets narrow data coverage and unstable multitask optimization under heterogeneous rewards with a capability-oriented long-context RL framework.
  • Conclusion: 23K samples across 9 task types combine curated open-source data with synthetic QA pairs generated from real documents.Both tracks use a four-phase construction pipeline with multi-stage quality control and iterative refinement.
  • Conclusion: The dataset, construction pipeline, and training code are open-sourced, while general-capability evaluations suggest no reasoning or memory degradation.
  • Future work: Difficulty reweighting shows clearer gains at 4B scales than at larger scales, motivating further study of scale-dependent optimization dynamics.
  • Future work: The remaining CorpusQA gap suggests targeted data supplementation for multi-document reasoning may be beneficial.
  • Future work: Combining context-aware token weighting with RLVR is proposed as a promising direction for more fine-grained training.

A. Gradient Analysis and Theoretical Motivation for TMN · A.1. Per-Prompt Gradient Norm Bound

The appendix derives TMN task-level normalization from REINFORCE-style policy gradients, showing that per-task gradient magnitude is governed primarily by within-prompt reward variance. Equalizing these magnitudes motivates dividing by the task-level standard deviation σ_task.

  • A. Gradient Analysis and Theoretical Motivation for TMN: TMN’s gradient analysis starts from the Dr. GRPO REINFORCE-style policy gradient to motivate task-level normalization.The derivation targets the σ_task denominator introduced in the main text.
  • A.1. Per-Prompt Gradient Norm Bound: In the analyzed GRPO variant, the per-prompt standard deviation is removed, yielding a REINFORCE-form policy gradient.This is the Dr. GRPO setting used for the bound.
  • A.1. Per-Prompt Gradient Norm Bound: For prompt u, μ_u denotes the expected sampled-response reward, while r(u, o) denotes the response reward.The notation defines the prompt, sampled response, reward, and expectation used in the derivation.
  • A.1. Per-Prompt Gradient Norm Bound: Applying Cauchy–Schwarz inside the expectation produces an upper bound on the per-prompt gradient magnitude.The inequality is the key step connecting the gradient expression to reward variation.
  • A.1. Per-Prompt Gradient Norm Bound: When log-probability gradient norms vary modestly, within-prompt reward variance becomes the dominant factor controlling per-prompt gradient magnitude.This conclusion assumes reward deviations vary more substantially than the log-probability gradient norm.
  • A.1. Per-Prompt Gradient Norm Bound: The within-prompt reward variance equals the square of the per-prompt standard deviation σ_u defined in the main text’s Eq. 1.Thus, variance-based gradient scaling corresponds directly to σ_u-based normalization.

A.2. Per-Task Gradient Magnitude … B. Data Generation Prompts

The appendix explains why heterogeneous reward variances create unequal per-task gradient contributions, motivates TMN-Reweight’s task-level normalization, contrasts it with GRPO’s prompt-level normalization, and documents the capability-oriented data-generation and verification prompts.

  • A.2. Per-Task Gradient Magnitude: Within-prompt reward variance governs per-task gradient magnitude, so continuous F1 and NDCG rewards can produce larger gradients than binary EM or Accuracy rewards.Without explicit normalization, higher-variance tasks can dominate joint optimization.
  • A.3. Deriving the Normalization Coefficient: TMN normalizes each task’s advantage by the expected gradient scale, using the TMN denominator as the normalization constant for equalizing per-task gradient contributions.The derivation identifies the denominator with the variance-based gradient bound.
  • A.3. Deriving the Normalization Coefficient: The normalization uses expected within-prompt reward variance rather than pooled variance, avoiding confusion between gradient-driving variation and between-prompt difficulty differences.Computing σ_task(i) from per-prompt variances preserves the difficulty structure used by TMN-Reweight’s second step.
  • A.4. Structural Relationship to GRPO: GRPO and TMN both normalize by standard deviation, but GRPO operates at prompt level while TMN applies a shared task-level denominator across each task.TMN preserves relative advantage magnitudes within tasks for subsequent difficulty reweighting.
  • A.4. Structural Relationship to GRPO: Removing prompt-level σ_u can help single-task uniform-reward settings but can degrade multitask optimization by exposing gradients to uncorrected cross-task scale differences.TMN addresses this multitask tension by retaining task-level normalization.
  • B. Data Generation Prompts: The data-generation appendix organizes prompts under a 9-task taxonomy, with subtasks specifying Full or Partial context, evaluation metric, and output format.Each prompt combines a primary capability definition with secondary task requirements, and generated samples are parsed, verified by Gemini-2.5-Pro, and multi-stage quality-calibrated.
  • B. Data Generation Prompts: The shared generation instruction asks the synthesis model to identify the input language, create three differentiated questions with answers and rationales, construct identifiers, and self-check compliance and accuracy.The prompt requires regeneration when any self-check criterion fails.
  • B. Data Generation Prompts: QA-pair verification rejects samples whose answers are ambiguous, distractors are invalid, content is hallucinated, task requirements are unmet, or language and format are inconsistent.Verification uses the full source document and generated QA pair before multi-stage verification.

B.5.6. T3-f: Comprehensive Error and Anomaly Sweep

T3-f evaluates comprehensive error and anomaly detection across the full document, with a focus on consistency and compliance checking. The task requires locating errors such as misspellings and uses F1 as its metric.

  • Comprehensive Error and Anomaly Sweep: T3-f locates all errors or anomalies, such as misspellings, across the full document.The context requirement is the full text.
  • Consistency and Compliance Checking: The primary task detects and locates contradictions, violations, or conflicts in documents.
  • Comprehensive Error and Anomaly Sweep: The full-text error-sweep task is evaluated with F1.
  • Output Format: The output begins with the "[Answer]" identifier, followed by each identified error on a separate line.

B.5.7. T3-g: Dependency-Aware Multi-Version Impact Analysis

This task analyzes dependency-aware impacts across multiple document or code versions by identifying methods or elements whose status changed. It evaluates outputs with F1 and requires exact method-signature identifiers in a specified format.

  • Task definition: The primary task compares changes across different text or code versions.It is framed as version and code diff analysis.
  • Task definition: The secondary task tracks dependency changes across versions and uses F1 as its full metric.The context requirement is full, with F1 specified as the evaluation metric.
  • Analysis instruction: The instruction is to identify all methods or elements that changed status between versions.This focuses the analysis on status changes rather than merely listing textual differences.
  • Output specification: Each changed method must be identified by a signature formatted as MethodName(ParameterTypes).The required identifier format uses the method name followed by parameter types in parentheses.

B.10.3. T8-c: Global Frequency Analysis

T8-c evaluates global frequency analysis by counting and sorting terms across the full text. The task requires descending frequency ordering and reports Pairwise Accuracy as its metric.

  • Global Frequency Analysis: The task counts and sorts terms by their frequency of occurrence across the full text.This is framed as aggregation and clustering of sorted results.
  • Global Frequency Analysis: The task’s context requirement is full, and its evaluation metric is Pairwise Accuracy.The task is identified as Global Frequency Analysis under the secondary task specification.
  • Global Frequency Analysis: Terms must be ordered in descending order by their frequency of appearance in the text.The instruction explicitly requires frequency-based descending sorting.

C. Evaluation Alignment with QwenLong-L1.5

The evaluation pipeline was aligned with QwenLong-L1.5 by reproducing its reported results, with small deviations across most benchmarks confirming protocol fidelity.

  • Protocol verification: The authors verified that their evaluation pipeline faithfully reproduces QwenLong-L1.5’s reported results for a fair comparison.Tables 9–11 report published scores alongside Δ = ours−published.
  • Reporting convention: Tables 9, 10, and 11 display published scores with Δ = ours−published as subscripts beside each value.Small deviations appear in gray, while larger deviations are highlighted in orange.
  • Protocol alignment: Across most benchmarks, deviations were small, confirming alignment between the evaluation protocol and the QwenLong-L1.5 paper.The reported comparison uses the published scores and their corresponding deviations.

D. Training Hyperparameters

Table 12 summarizes the key reinforcement-learning training hyperparameters for GoLongRL. The configurations are shared across both model scales.

  • Table 12 summarizes GoLongRL’s key reinforcement-learning training hyperparameters.
  • The reported training configurations are shared across the two model scales.
  • The training hyperparameters are presented as a consolidated configuration for GoLongRL.
Loading 2605.19577v1…