Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning

1 Introduction

• Advantage normalization with group-level mean and batch-level standard deviation plus token-level loss aggregation unlocks critic-free learning with vanilla PPO, surpassing mainstream algorithms with redundant components.The paper identifies these two techniques as a minimalist combination for unlocking critic-free policies.Source: §1 · p. 2
• Most RL techniques show strong sensitivity to model type, data distribution, reward mechanism, and hyperparameters in the paper’s isolated evaluations.The study reproduces widely used techniques on a shared open-source framework across datasets of varying difficulty, model sizes, and model types.Source: §1 · p. 2
• Group-level normalization is robust across reward settings, batch-level normalization improves stability under large-scale rewards, and group-level mean with batch-level standard deviation enables robust normalization.The introduction reports these as separate normalization findings.Source: §1 · p. 2
• Clip Higher promotes high-quality exploration for aligned models, while clipping upper bounds appear to follow a performance “scaling law” on small models.The reported scaling relationship is limited to small-sized models.Source: §1 · p. 2
• Token-level loss aggregation is effective on base models but offers limited improvement over sequence-level aggregation on aligned models.Source: §1 · p. 2
• Overlong filtering improves accuracy and clarity on short-to-medium reasoning tasks but provides limited benefit for long-tail reasoning.Source: §1 · p. 2

2.1 Proximal Policy Optimization (PPO)

• PPO stabilizes actor-critic policy-gradient learning with a clipped surrogate objective that limits changes between old and new policies.The objective weights policy probabilities by advantages, typically estimated with GAE, and uses a clipping hyperparameter to constrain updates.Source: §2.1 · p. 3

2.2 Group Relative Policy Optimization (GRPO)

• GRPO removes the critic, estimates advantages by normalizing rewards across responses to the same prompt, and adds policy-reference KL divergence directly to its loss.The group normalization emphasizes differences among candidate responses and can preserve gradient reliability under sparse rewards (Hu et al., 2020).Source: §2.2 · p. 3

2.3 Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO)

• DAPO (Yu et al., 2025) builds on group-normalized advantages with decoupled clipping, dynamic sample filtering, token-level loss aggregation, and reward shaping for overlong or truncated responses.It samples response groups for each question and optimizes a surrogate objective.Source: §2.3 · p. 3

2.4 Reinforcement Learning Techniques

• Practical RL techniques reduce variance or stabilize updates through alternative baselines, clipping strategies, reward or advantage normalization, and filtering of uninformative samples.Examples include group-mean baselines (Shao et al., 2024), other-sample gradient baselines, relaxed upper ratio clipping (Yu et al., 2025), and difficulty-based filtering (Yu et al., 2025; Zhang et al., 2025; Chu et al., 2025).Source: §2.4 · p. 4
• The taxonomy also covers token- versus sequence-level loss aggregation, auxiliary KL or SFT losses, and reward shaping; this work focuses on normalization, clipping, masking, and loss aggregation.Reward examples include length penalties, formatting rewards, and length-dependent accuracy rewards.Source: §2.4 · p. 4

3.1 Experimental Setup

• Experiments use PPO loss with REINFORCE-computed advantages in ROLL, a global batch size of 1024, 8 responses per prompt, and a maximum response length of 8192 tokens.The learning rate is 1e −6; generation uses top_p 0.99, top_k 100, and temperature 0.99.Source: §3.1 · p. 4
• The model comparison covers Qwen3-4B and Qwen3-8B in both base and aligned initializations.This design enables assessment of RL gains under different initialization conditions.Source: §3.1 · pp. 4–5
• Training uses open-source SimpleRL-Zoo-Data and DeepMath, filters excessive True/False-label examples, and samples 5,000 examples for each of the Easy, Medium, and Hard tiers.Easy data is drawn from SimpleRL-Zoo-Data-Easy; Medium uses the 5,000 easiest DeepMath-103k examples, while Hard sampling is proportional to assigned difficulty.Source: §3.1 · p. 5
• Evaluation spans six math datasets, from basic arithmetic to competition-level problems, including MATH-500, OlympiadBench, MinervaMath, AIME24-25, and AMC23 subsets.Source: §3.1 · p. 5

3.2 Baseline Results

• As training epochs increase, accuracy trajectories differ across difficulty tiers, and harder samples often elicit longer responses as models fit complex reasoning patterns.The study analyzes loss dynamics, accuracy trajectories, and generalization gaps across Easy, Medium, and Hard training sets.Source: §3.2 · p. 5
• Aligned models start with higher accuracy and longer responses, but further RL training improves accuracy by only roughly 2%.The authors interpret this as a slight improvement for aligned models that are already highly optimized.Source: §3.2 · p. 5

4.1 Normalization

• GRPO (Shao et al., 2024) and RLOO normalize advantages within prompt groups, whereas REINFORCE++ (Hu et al., 2025) uses batch-level normalization.The passage notes that batch-level normalization is motivated by concern that excessive within-prompt optimization can encourage reward hacking and hinder generalization when response diversity is low.Source: §4.1 · pp. 5–6

4.1.1 Impact of the standard deviation term in advantage normalization

• On easy data, removing the standard-deviation divisor prevents concentrated rewards from amplifying advantages and destabilizing gradients; on hard data, either normalization remains stable with similar efficiency.The experiments identify the standard-deviation term as the mechanism behind the instability under concentrated rewards.Source: §4.1.1 · p. 7, Figure 5 · p. 9

4.1.2 Reconstruct a robust normalization technique

• Using a group-level mean with a batch-level standard deviation provides stronger normalization, reducing gradient magnitudes and supporting more stable learning under biased, sparse rewards.The reported comparison finds a clear advantage for global rather than group-level standard-deviation calculation.Source: §4.1.2 · pp. 7–9, Figure 6 · p. 9

4.2 Clip-Higher

• Clip-Higher raises the clipping upper bound, giving low-probability tokens more opportunity to gain probability and mitigating entropy collapse.The section frames this as a response to standard clipping suppressing low-probability tokens and reducing exploration.Source: §4.2 · p. 10

4.2.1 In which settings should we clip higher

• Clip-Higher has model-dependent effects: it barely helps or can hurt base models, while aligned models retain entropy and improve evaluation as the upper bound rises.The authors attribute the limited base-model effect partly to a low clipping rate of approximately 0.003 and limited policy expressiveness.Source: §4.2.1 · pp. 10–11, Figure 8 · p. 11

4.2.2 Analyzing the effectiveness of Clip-Higher from a linguistic perspective

• Raising the upper bound from 0.2 to 0.28 significantly expands policy update space, shifts clipping away from discourse connectives toward high-frequency functional tokens, broadens discourse reasoning structures, and maintains core sentence stability.Source: §4.2.2 · pp. 12–13

4.2.3 How to set the upper bound for advantage clipping

• Performance appears to scale with the clipping upper bound on the small-sized model, but this relationship is absent on larger models.Source: §4.2.3 · p. 13
• For 4B models, performance improves as the advantage-clipping upper bound rises, peaking at 0.32; for 8B models, it does not improve progressively and is strongest at 0.28.The comparison concerns aligned models trained on medium data.Source: §4.2.3 · p. 13, Figure 11 · p. 13

4.3 Loss Aggregation

• Sequence-level aggregation, used by GRPO (Shao et al., 2024), weights each response equally, while token-level aggregation normalizes the summed loss by total token count to weight tokens equally.Equal response weighting reduces each token’s influence in longer responses and may bias optimization toward brevity (Liu et al., 2025a; Liu et al., 2025b; Yu et al., 2025).Source: §4.3 · pp. 13–14

4.3.1 Does token-level loss aggregation suit all settings?

• Token-level loss aggregation improves convergence, peak accuracy, and robustness for base models, whereas sequence-level aggregation generally performs better on aligned models.Equalizing token gradients may be unnecessary or detrimental for aligned models, whose reasoning is already strong and stable.Source: §4.3.1 · p. 14, Figure 12 · p. 15

4.4 Overlong Filtering

• Fixed generation-length limits can truncate multi-step reasoning before the final answer, falsely label coherent responses as negative, and contaminate training signals; overlong filtering masks rewards for excessively long responses.The analysis examines its effects across datasets and maximum generation lengths; cited prior work includes Chen et al. (2025), Team et al. (2025), Yu et al. (2025), and He et al. (2025b).Source: §4.4 · p. 14

4.4.1 When to use the overlong filtering

• Overlong filtering improves accuracy and response clarity on medium- and short-length reasoning tasks but has limited effectiveness on long-tail reasoning; benefits are substantial at 8k tokens and diminish at 20k.At 20k, filtered models tend to generate longer responses than the vanilla policy, whereas an 8k threshold produces shorter responses.Source: §4.4.1 · p. 15, Figure 13 · p. 16
• At 20k, overlong filtering more often removes repetitive or non-terminating outputs, while an 8k threshold can also mask extended but valid reasoning and encourage concise responses.Thus, filtering can remove pathological outputs when very long reasoning is expected, or reduce unnecessary verbosity when long-tail reasoning is not needed.Source: §4.4.1 · pp. 15–16
• Overlong filtering reduces repetitive, non-terminating samples during instruction-tuned RL training, helping models distinguish completed from truncated generations and avoid learning from truncated portions.The share of abnormal samples among overlong outputs otherwise increases during training, indicating degraded EOS modeling.Source: §4.4.1 · pp. 16–17, Figure 14 · p. 17

5 A simple combination: Lite PPO

• For 4B-Base and 8B-Base non-aligned models, advantage normalization and token-level loss aggregation are highly effective techniques.Advantage normalization uses group-level mean and batch-level standard deviation to shape sparse rewards into more robust guiding signals.Source: §5 · p. 18
• Lite PPO outperforms DAPO and GRPO on non-aligned models using critic-free vanilla PPO loss.DAPO combines group-level normalization, Clip-Higher, overlong reward shaping, token-level loss, and dynamic sampling.Source: §5 · p. 18
• On small models lacking basic reasoning ability, Lite PPO trends upward stably while other policies rapidly collapse after peaking, aided by normalization under homogeneous rewards.The cited homogeneous reward distributions occur in datasets with non-uniform reward levels (easy and hard).Source: §5 · p. 18
• On 8B-Base models with long-tail generation capabilities trained on the hard dataset, Lite PPO performs better by removing overlong filtering and using token-level loss aggregation.The passage attributes the gains to avoiding a restriction on complex long-tail outputs and to token-level loss efficiency on base models.Source: §5 · p. 18

6 Conclusion

• Lite PPO, combining only two core techniques, achieves superior performance over algorithms with redundant components.The finding challenges over-engineering and underscores contextual adaptability in technique selection.Source: §6 · p. 18
• A systematic, reproducible evaluation under a unified framework clarifies the mechanisms and applicability of prominent RL techniques across scenarios.The study disentangles normalization, clipping, and filtering to provide actionable technique-selection guidelines.Source: §6 · p. 18
• Because policy initialization consistently uses Qwen3, conclusions may vary across LLM families with different pre-training and architectures.The authors advocate greater disclosure of implementation details in future industry technical reports to bridge the academia-industry understanding gap.Source: §6 · p. 18

7 Future work

• They plan to monitor RL4LLM developments and distill emerging practices into evidence-based guidelines for academic and industrial practitioners.This is part of a sustained effort to guide the field along principled, empirically grounded trajectories.Source: §7 · p. 18
• They aim to use ROLL to consolidate RL algorithms and optimization strategies into a modular suite for flexible composition and benchmarking in consistent infrastructure.The suite is intended to provide comprehensive framework support.Source: §7 · pp. 18–19
• They will continue exploring streamlined RL algorithms that achieve strong empirical performance with minimal engineering overhead.The authors see this as advancing RL4LLM through algorithmic innovation and supporting robust, adaptable, broadly beneficial progress.Source: §7 · p. 19

A.1 Parameters

• The pipeline is implemented in ROLL, described as a user-friendly, efficient open-source reinforcement learning framework.Source: §A.1 · p. 22
• PPO uses 1 epoch, the reinforce advantage estimator, an initial KL coefficient of 0.0, and async generation level 1.Source: §A.1 · p. 22
• Training uses a 1.0e-6 learning rate, weight decay 0, batch size 4, gradient accumulation 32, 50 warmup steps, and 50 epochs.Source: §A.1 · p. 22
• Generation uses top_p 0.99, top_k 100, one beam, temperature 0.99, and 8 return sequences.Source: §A.1 · p. 22

A.2 Prompt

• The system prompt instructs the model to reason step by step and put its final answer within \boxed{}.Source: §A.2 · p. 22
• The prompt is intended to encourage visible reasoning and make the final answer explicit, improving output clarity and readability.Source: §A.2 · p. 22

B.1 Repeat Ratio

• For the aligned model, the filter traces backward from truncation to find repeated content and marks samples exceeding a predefined threshold as “no-stop repetition” anomalies.Source: §B.1 · p. 22
• The repeat ratio is the share of all overlong samples classified as repeated, used to quantify termination behavior at the current step.Source: §B.1 · pp. 22–23

B.2 Examples of Ostensible Positive Phenomena

• Weaker-capability models can continue generating aimlessly after correctly reasoning and answering, until they exceed the output-length limit.Figure 16 illustrates an ostensible positive case that does not terminate after the answer.Source: §B.2 · p. 23, Figure 16 · p. 23
• Rule-based evaluation gives these false positives a reward of 1, introducing noise into training.Source: §B.2 · p. 23
• Figure 16 shows an ostensible positive inference that cannot be terminated after the answer is given.Source: Figure 16 · p. 23

• Qwen3-8B-Base initialization yielded more competitive benchmark results across training-dataset difficulty levels; Figure 17 compares sample-level and token-level loss on medium and extremely hard datasets.Source: C Detailed Experimental Results · p. 23, Figure 17 · p. 23
• Accuracy over training iterations was evaluated across reward scales with batch-level normalization, using easy and medium training datasets.Source: C Detailed Experimental Results · p. 23, Figure 18 · p. 24

D Case Study of Clip Higher

• Figure 19 shows the same prompt with high clipping bounds of 0.20 and 0.28 to visualize Clip Higher’s trigger behavior.The top case uses 0.20; the bottom case uses 0.28.Source: Figure 19 · p. 24, D Case Study of Clip Higher · p. 24
• Figure 20 compares predicted probability distributions for Qwen3-8B-Base and Qwen3-8B under clipping upper bounds of 0.20 and 0.28.The base model is shown on the left and the aligned model on the right.Source: Figure 20 · p. 25, D Case Study of Clip Higher · p. 25