PaperlayerView exact revision on arXiv

Source-linked AI summary

LongAct: Harnessing Intrinsic Activation Patterns for Long-Context Reinforcement Learning

Bowen Ping, Zijun Chen, Tingfeng Hui, Qize Yu, Chenxuan Li, Junchi Yan, Baobao Chang

arXiv:2604.14922v1cs.LGcs.CL

TL;DR

Long-context RL has largely emphasized external supervision while leaving intrinsic latent features underused. LongAct identifies high-magnitude query/key activations and selectively updates their associated weights, reporting improved LongBench v2 performance, RULER generalization, and robustness across RL algorithms. The authors also report that high-magnitude features are critical to the observed improvements, while larger-model scaling remains unexplored.

  • Problem

    Long-context RL has largely overlooked features embedded in latent representations despite their relevance to complex reasoning.

  • Method

    LongAct uses intrinsic high-magnitude query/key activations to guide sparse reinforcement-learning updates of associated weights.

  • Results

    LongAct improves LongBench v2, generalizes on RULER, and boosts performance across GRPO, DAPO, and KL-Cov.

  • Takeaways & Limitations

    The reported analyses identify high-magnitude activations as critical features for LongAct’s long-context reasoning improvements.

  • Takeaways & Limitations

    Larger models could not be used for reinforcement learning because of computing-resource limits, leaving scaling effects for future work.

Abstract

from arXiv · show

Reinforcement Learning (RL) has emerged as a critical driver for enhancing the reasoning capabilities of Large Language Models (LLMs). While recent advancements have focused on reward engineering or data synthesis, few studies exploit the model's intrinsic representation characteristics to guide the training process. In this paper, we first observe the presence of high-magnitude activations within the query and key vectors when processing long contexts. Drawing inspiration from model quantization -- which establishes the criticality of such high-magnitude activations -- and the insight that long-context reasoning inherently exhibits a sparse structure, we hypothesize that these weights serve as the pivotal drivers for effective model optimization. Based on this insight, we propose LongAct, a strategy that shifts from uniform to saliency-guided sparse updates. By selectively updating only the weights associated with these significant activations, LongAct achieves an approximate 8% improvement on LongBench v2 and enhances generalization on the RULER benchmark. Furthermore, our method exhibits remarkable universality, consistently boosting performance across diverse RL algorithms such as GRPO and DAPO. Extensive ablation studies suggest that focusing on these salient features is key to unlocking long-context potential.

1 Introduction

LongAct addresses a gap in long-context RL by using intrinsic activation patterns rather than relying mainly on external supervision or curricula. It selectively updates weights linked to high-magnitude query/key features, improving long-context performance and generalization.

  • Long-context RL research has mainly optimized external rewards, synthesized reasoning data, or training curricula, while overlooking latent representation features.
  • LongAct uses sparse, high-magnitude activations in query and key representations as anchors for saliency-guided sparse reinforcement-learning updates.The method selectively updates associated weights instead of uniformly updating parameters.
  • 8% improvement on LongBench v2 is reported for LongAct, demonstrating gains from focusing updates on intrinsic activation patterns.
  • 4% gain on 128K RULER accompanies improved generalization across generic long-context tasks and performance gains across GRPO, DAPO, and KL-Cov.
  • Ablations report 36.73 on LongBench v2 when updating high-magnitude-associated weights versus 29.82 when updating low-magnitude-associated weights.
  • Case analysis reports immediate collapse and repetitive loops after disrupting high-magnitude activations, whereas neutralizing low-magnitude counterparts preserves reasoning coherence.

2 Related work

Prior long-context RL work emphasizes architectural changes, reward design, data synthesis, or context scaling. LongAct instead builds on evidence that high-magnitude hidden activations are important and maps them to sparse weight updates.

  • Long-context RL approaches have used architectural modifications, progressive context scaling, and carefully designed reward functions.
  • Activation-aware quantization studies preserve or quantize weights and KV-cache values according to high-magnitude activation distributions.
  • LongAct’s framework combines dynamic saliency-guided sparse updates with a standard group-based policy optimization loop.

3 Method

LongAct combines a long-context RL framework with saliency-guided sparse updates derived from high-magnitude query and key activations. It applies gradient masks to selected rows of WQ and WK while leaving other parameters fully updated and inference dense.

  • High-magnitude Activations: LongAct measures high-magnitude query and key activations in attention layers to identify critical parameters for updating.It computes activation magnitudes across the sequence dimension and applies the same analysis to Q and K.
  • Reinforcement Learning in Long-context Scenarios: The method incorporates long-context c into RL so the policy grounds relevant information before generating reasoning chains for question x.The framework extends standard language-model RL, which typically generates from a short question alone.
  • Training Framework: LongAct uses SFT to initialize a robust policy prior, then trains with rule-based format and answer rewards under a GRPO-based framework.The implementation employs DAPO for more stable training, while the reward assigns binary values for required reasoning tags and answer correctness.
  • Dynamic Saliency-guided Updates: The core update strategy restricts gradients to rows of WQ and WK associated with selected high-magnitude feature dimensions.Selection is performed within each attention head rather than globally, preserving the multihead structure through binary gradient masks.
  • Dynamic Saliency-guided Updates: Only WQ and WK receive sparse gradient masking; WV, WO, and MLP parameters receive standard full updates.The dynamic mask is computed on collapsed head-dimension tensors, making its overhead negligible relative to the full forward-backward pass.

4 Experimental Results

LongAct is evaluated across long-context benchmarks, model sizes, RL algorithms, activation-selection strategies, and short-context tasks. Results show consistent gains, including stronger performance on difficult and longer inputs.

  • LongAct reaches 36.73 on LongBench v2 for Qwen3-8B, improving over SFT w/ DAPO by +3.93 and the cold-start SFT model by +9.69.
  • LongAct improves LongBench v2 performance across difficulty and length splits, including 34.72 on the 8B Long split and 33.33 on the 4B Long split.
  • LongAct achieves the best RULER averages at both 128K and 64K context lengths across 8B and 4B models.
  • LongAct provides the best InfiniteBench average for both model sizes, with the 8B model reaching 49.39 and improving over SFT w/ DAPO by +1.28.
  • LongAct maintains stable gains across DAPO, GRPO, CLIP-conv, and KL-conv, while DAPO gives the best overall results on both model sizes.
  • Selecting massive values outperforms random selection and minimum values, while selecting 30% achieves the best overall sparsity setting.
  • LongAct also outperforms SFT and full-parameter DAPO on GSM8K, HumanEval, and TruthfulQA.

5 Analysis

Perturbation analysis indicates that high-magnitude activations are important for maintaining long-context reasoning. Removing them causes model collapse, whereas removing quieter activations preserves coherent reasoning.

  • Removing the bottom 30% of activations preserves logical coherence and correct answers in generated chain-of-thought reasoning.
  • Removing the top 30% of high-magnitude activations causes immediate model collapse and repetitive outputs such as “3333...”.
  • The contrast supports high-magnitude activations as pivotal for long-context reasoning and motivates using them to guide training updates.

6 Conclusion

LongAct leverages the model’s internal representations to enhance long-context reasoning.

  • LongAct is presented as a robust method that leverages internal representations to enhance long-context reasoning.

Limitations

The study could not evaluate LongAct with larger reinforcement-learning models because of computing-resource limitations.

  • Computing-resource constraints prevented reinforcement-learning experiments with larger models.The authors identify scaling effects to larger models as future work.

A Appendix

The appendix illustrates how LongAct maps selected dimensions from attention heads to projection-weight rows and updates only those rows.

  • With H = 2, D = 4, and λ = 0.3, the example selects k = 1 active dimension per head.The selection ratio determines the number of active dimensions per head.
  • The indexing mechanism maps each selected local head index to a global row of the projection weight W ∈R(H·D)×din.
  • Head 0 maps to global row 2, while Head 1 maps to global row 5.
  • Only rows {2, 5} of W are updated, leaving the remaining six of eight rows frozen.

B Discussion

The discussion links LongAct’s saliency selection to activation patterns, retrieval behavior, sparse RL updates, and comparisons with related methods. It also provides visual and ablation materials for examining these effects.

  • Activation analysis: Q and K activations show no distinct or consistent high-magnitude pattern across sequence positions, unlike the hidden-feature dimension.The analysis uses activation magnitudes measured by the ℓ2-norm.
  • Theoretical justification: Masking high-magnitude activations collapses retrieval to 0/503 correct, whereas masking low-magnitude activations preserves 108/503 correct.This motivates targeting high-magnitude channels as context-processing circuitry.
  • Comparison with MIGU: LongAct targets within-task intrinsic activation sparsity using ℓ2-norm saliency to filter noisy gradient updates; on LongBench v2 Hard, it improves by +7.63.
  • Visual analyses: Figures 3–5 visualize query, key, and value representation magnitudes for Qwen3-8B on RULER, while Figure 6 illustrates activation-disruption effects on a LongBench v2 example.Figure 6 contrasts disrupting the top 30% with disrupting the smallest 30% of activations.
  • Ablations: Table 7 reports ablations over the percentage of selected massive values, and Table 8 reports LongBench v2 accuracy after clamping selected activations to the global mean.
Loading 2604.14922v1…