Source-linked AI summary
Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVR
Jiakang Wang, Runze Liu, Fuzheng Zhang, Xiu Li, Guorui Zhou, Ling Pan
TL;DR
RLVR methods often apply uniform constraints despite differences between reasoning-related and knowledge-related tokens, while isolating token groups can conflict with autoregressive dependencies. Archer uses response-level entropy to assign differentiated clipping and KL constraints while jointly optimizing tokens. It outperforms GRPO across mathematical and coding benchmarks, and the paper identifies entropy-based role assignment and fixed binary constraints as limitations.
Problem
RLVR methods largely apply uniform constraints across tokens, despite their differing roles, while token-isolation approaches can disrupt autoregressive dependencies.
Source: §1 · pp. 1–2Method
Archer uses intra-response entropy quantiles to infer token roles and jointly optimizes tokens with role-conditioned clipping and KL constraints.
Source: §1 · p. 2, §4.1.2 · p. 5Results
Archer outperforms GRPO across benchmarks, with average gains of 5.6% for mathematical tasks and 3.0% for coding tasks.
Source: §5.2 · p. 7, §6 · p. 10Takeaways & Limitations
The results support coordinating token-type learning through entropy-aware constraints while preserving joint optimization.
Source: §6 · p. 10Entropy is a heuristic proxy for token roles, and the current method uses a binary partition with fixed clipping and KL coefficients.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Reinforcement Learning with Verifiable Rewards (RLVR) has become an effective post-training method for improving the reasoning abilities of Large Language Models (LLMs). However, existing methods mainly apply uniform optimization constraints across all tokens, ignoring their heterogeneous roles. Prior work shows that high-entropy tokens are closely tied to reasoning, while low-entropy tokens primarily encode factual knowledge, and recent approaches attempt to exploit this distinction by isolating token updates via masking or asynchronous training. We argue that such isolation breaks the sequential dependency structure of autoregressive generation, leading to suboptimal learning. To address this, we propose \textbf{Archer}, an entropy-aware RLVR framework with \textbf{dual-token constraints} that preserves joint optimization while modulating update strength across token types. Our method introduces response-level entropy normalization for stable token classification and applies differentiated clipping ranges and KL regularization to encourage exploration on reasoning tokens while preserving knowledge tokens. Experiments on mathematical reasoning and code generation benchmarks show that Archer consistently outperforms strong baselines across multiple model scales, improving both \textit{pass@1} and \textit{pass@K} performance. These results highlight the importance of respecting sequence-level dependencies when designing fine-grained RL optimization strategies for LLMs.
1 Introduction
RLVR improves LLM reasoning, but uniform token constraints overlook differences between reasoning-related and knowledge-related tokens. Archer preserves joint autoregressive optimization while adapting constraints to token roles.
1 Introduction
- Reasoning-critical tokens tend to have higher entropy, while knowledge-related tokens tend to have lower entropy and greater stability.The paper frames token-role differences as a question for RL optimization.Source: §1 · pp. 1–2
- Existing token-isolation approaches mask or asynchronously update token subsets, which the authors argue disrupts autoregressive dependencies and can impair credit assignment and learning efficiency.A token's optimization signal depends on its surrounding context.Source: §1 · p. 2
- Archer preserves joint optimization while adapting policy movement across tokens, using local clipping and global reference-policy KL anchors.It infers token roles zero-shot from intra-response entropy quantiles, without auxiliary networks.Source: §1 · p. 2
2 Related Work
Prior RL approaches use response-level signals or uniform token treatment, while token-level analyses connect entropy with reasoning-related behavior. Archer differentiates constraints without separating token groups.
2.1 Reinforcement Learning for Large Language Models
- GRPO and its extensions assign the same response-level advantage to every token, potentially leaving reasoning-critical steps with suboptimal learning signals.Process-based and unsupervised RL provide finer-grained rewards but still do not account for different token functions.Source: §2.1 · p. 2
2.2 Critical Token Analysis in RL for Reasoning
- Prior analyses associate high-entropy tokens with logical connectors, reasoning behaviors, or dominant RL updates; Archer instead trains token groups jointly with differentiated constraints.Earlier approaches isolate or separately train token subsets, whereas Archer avoids direct filtering or separation.Source: §2.2 · p. 3
3 Preliminaries
GRPO estimates advantages from multiple sampled responses rather than learning a value model, and its loss includes a KL penalty against a reference policy.
3 Preliminaries
- GRPO samples multiple responses and rewards for each prompt to estimate advantages without a learned value model.The supplied passages describe the group-based setup and begin the advantage calculation.Source: §3 · p. 3
- The GRPO objective weights KL divergence between the current policy and reference policy with coefficient β.Source: §3 · p. 3
4 Method
Archer identifies token roles using entropy and applies differentiated constraints while updating all tokens jointly. Its design uses response-level statistics, token-conditioned clipping and KL regularization, and expanded optimization regions for high-entropy tokens.
4 Method
- Archer combines entropy-based token identification with differentiated clipping and KL constraints in a joint optimization framework.The method section introduces entropy identification, response-level statistics, and token-level optimization constraints.Source: §4 · p. 3, §4.1.2 · p. 5
4.1 Intra-Response Entropy as a Zero-Shot Proxy for Token Roles
- High-entropy tokens are mainly reasoning-related, while low-entropy tokens are often factual or word-suffix tokens, supporting entropy as a proxy for token roles.The analysis considers 1,024 responses, with tokens retained when they appear more than 100 times.Source: Figure 1 · p. 3, §4.1 · p. 4
4.1.1 Response-Level Entropy Statistics
- Entropy varies across prompts and sampled responses, so batch-level thresholds can misclassify reasoning tokens; Archer computes entropy quantiles independently within each response.With an 80th-percentile threshold, only 4.34% of tokens may be labeled high-entropy in low-entropy responses.Source: §4.1.1 · p. 4, Figure 2 · p. 4
4.1.2 Dual-Anchor Token-Conditioned Trust Region
- Archer applies stricter clipping and stronger KL penalties to low-entropy knowledge tokens, and looser clipping and weaker KL penalties to high-entropy reasoning tokens.The local clipping anchor controls step-wise update magnitude, while the global reference-policy anchor controls drift during training.Source: §4.1.2 · p. 5
4.1.3 Visualization of RL Optimization Regions
- For high-entropy tokens, Archer extends clipping regions to add reward signals for positive advantages and stronger penalties for negative advantages.Figure 3 depicts the optimization regions for positive and negative advantages.Source: §4.1.3 · pp. 5–6, Figure 3 · p. 6
5 Experiments
Archer is evaluated on mathematical and coding benchmarks across multiple model scales, with results and ablations examining its performance and token-specific constraints.
5.1 Setup
- Experiments compare four base models with RLVR baselines on six math datasets and LiveCodeBench v5 and v6, using avg@K and pass@K metrics.The study uses 6,753 code problems and 51,800 math problems; training samples 16 rollouts per prompt.Source: §5.1 · pp. 6–7
5.2 Main Results
- Archer gains 12.3% average accuracy over the base model on AIME24 and AIME25, and beats GRPO by 5.6% on math and 3.0% on code.Against the base model, accuracy rises 12.7% on LiveCodeBench v5 and 13.0% on v6.Source: §5.2 · p. 7, Table 1 · p. 8, Table 2 · p. 8
- Archer achieves the highest average accuracy on mathematical reasoning and outperforms comparable coding models, while also showing advantages in pass@K.The comparison includes DeepScaleR-1.5B, DeepCoder-1.5B, and Nemotron-1.5B; Archer uses single-stage training and fewer GPU hours than the compared open-source models.Source: §5.2 · pp. 7–8
5.3.1 Impact of Different Entropy Thresholds
- ρ = 0.8 gives the best average performance, while ρ = 0.9 degrades results; stricter thresholds can classify reasoning-critical tokens as low-entropy and constrain their updates.Smaller ρ exposes more high-entropy tokens to exploratory updates.Source: §5.3.1 · pp. 8–9, Table 3 · p. 8
5.3.2 Impact of Different KL Weights
- Both removing low-entropy-token KL regularization and using an overly large weight hurt performance, through greater collapse risk or slower learning, respectively.The paper concludes that low-entropy-token KL regularization helps keep the policy near the base model, prevent collapse, and retain abilities.Source: §5.3.2 · p. 9, Table 4 · p. 9
5.3.3 Impact of Clip Ranges on Different Token Types
- Low-entropy clip thresholds strongly affect training and final performance, whereas high-entropy thresholds have less effect, with larger values mainly influencing later training.Higher low-entropy thresholds accelerate entropy decline and repetition growth; lower thresholds slow LiveCodeBench v5 improvement, while larger high-entropy thresholds encourage exploration.Source: §5.3.3 · p. 9, Table 5 · p. 9
6 Conclusion
The conclusion presents Archer as synchronized entropy-aware training that applies token-specific constraints, and reports improvements over the base model and existing state-of-the-art models. It also identifies limits in the token-role heuristic and binary constraint design.
6 Conclusion
- Archer updates all tokens simultaneously while varying regularization and clipping by token type, and experiments show improvements over the base model and existing state-of-the-art models.The paper frames this as coordinating learning across token types while balancing factual accuracy and logical reasoning.Source: §6 · p. 10
- Entropy is only a heuristic proxy for token roles, and the current method uses a binary partition with fixed clipping and KL coefficients.The authors suggest exploring continuous constraints or finer-grained token types.Source: §6 · p. 10
A.1 Dataset
The code and mathematics training corpora are assembled from curated sources and filtered for quality, redundancy, and evaluation overlap.
A.1.1 Code Domain
- The code corpus combines DeepCoder, CodeContests, and CodeForces, prioritizing the latter two when problems duplicate because they provide more test cases.The final code dataset contains 6,753 problems.Source: §A.1.1 · p. 15
- Code problems are filtered using test-case checks, model validation, deduplication, benchmark-overlap removal, and sampling-stability checks, then standardized for automated verification.Problems with fewer than five test cases or homogeneous warm-start outcomes are removed.Source: §A.1.1 · p. 15
A.1.2 Mathematics Domain
- The math corpus merges DeepScaleR, Skywork-OR1, and DAPO data with N-gram deduplication, then filters problems using model sampling and verification.The final corpus contains approximately 51,800 problems and is deduplicated against AIME24 and AIME25 to avoid benchmark contamination.Source: §A.1.2 · pp. 15–16
B Additional Experimental Results
The appendix includes a comparison of computational efficiency for Archer and baselines on 1.5B models.
B Additional Experimental Results
- Table 6 compares computational efficiency between Archer and baselines on 1.5B models.Source: Table 6 · p. 16
B.2 Training Dynamics
Archer outperforms GRPO with lower repetition on Qwen3-4B and Qwen3-8B training dynamics. The figures also track test performance and the effects of varying KL weights and token-specific clipping thresholds.
B.2 Training Dynamics
- Test curves report results for Qwen3-4B methods across six mathematical benchmarks.Source: Figure 6 · p. 17
- Figure 7 examines how varying KL weights relates to LiveCodeBench v5 performance, model entropy, and repetition ratio.Source: Figure 7 · p. 17
- Figure 8 examines how varying the low-entropy-token clip threshold relates to LiveCodeBench v5 performance, model entropy, and repetition ratio.Source: Figure 8 · p. 17
- Figure 9 examines how varying the high-entropy-token clip value relates to LiveCodeBench v5 performance, model entropy, and repetition ratio.Source: Figure 9 · p. 18
B.4 Mutual Enhancement Between Math RL and Code RL
Math RL and code RL each improve performance both in-domain and on out-of-domain benchmarks. The analysis associates gains with problem difficulty and improved reasoning, rather than new knowledge.
B.4 Mutual Enhancement Between Math RL and Code RL
- RL gains mainly reflect better reasoning and integration of existing abilities, not fundamentally new knowledge or basic skills such as arithmetic.Source: §B.4 · pp. 18–19
- RL-trained responses show clearer logical flow and improved structural coherence.Source: §B.4 · p. 18
- Models show increased attention to edge cases and boundary conditions, especially with Code RL.Source: §B.4 · p. 18
- RL-trained models integrate previous reasoning steps more accurately, whereas the base model can produce inconsistent final answers from incorrect intermediate reasoning.Source: §B.4 · p. 19