Source-linked AI summary
SIRI: Scaling Iterative Reinforcement Learning with Interleaved Compression
Haoming Wen, Yushi Bai, Juanzi Li, Jie Tang
TL;DR
Prior methods to reduce LRMs’ unnecessary reasoning can compromise performance. SIRI alternates compression and expansion of the reasoning budget during training; its reported results improve accuracy while reducing response length. The authors note that some comparison methods could not be evaluated on AIME25 and AMC because their checkpoints were unavailable.
Problem
Attempts to reduce LRMs’ unnecessary reasoning have often compromised performance, limiting progress toward efficient reasoning.
Source: §1 · p. 2Method
SIRI periodically adjusts the maximum rollout length, alternating compression to reduce overthinking with expansion to encourage exploration.
Source: §1 · p. 2Results
SIRI-high achieves the highest accuracy on all benchmarks, improving average performance by 33.6%; SIRI-low reduces response length by 43.1% and boosts average performance by 27.0% versus the original 1.5B model.
Source: §5.2 · p. 6Takeaways & Limitations
Across iterations, SIRI consistently improves model performance and token efficiency, while the upper performance threshold remains unknown.
Source: §6 · p. 9Unavailable checkpoints prevented evaluating the comparison methods on AIME25 and AMC.
Source: §5.2 · p. 7
Abstract
from arXiv · showhide
We introduce SIRI, Scaling Iterative Reinforcement Learning with Interleaved Compression, a simple yet effective RL approach for Large Reasoning Models (LRMs) that enables more efficient and accurate reasoning. Existing studies have observed repetitive thinking patterns in LRMs, and attempts to reduce them often come at the cost of performance. In this paper, we show that this trade-off can be overcome through a training regime that iteratively alternates between compressing and expanding the reasoning budget, by dynamically adjusting the maximum rollout length during training. The compression phase cuts the rollout length, forcing the model to make precise and valuable decisions within a limited context, which effectively reduces redundant tokens and increases reasoning density. The expansion phase then relaxes the length limit, providing space for the model to explore and plan in long-horizon settings. Remarkably, we find that after each compression-expansion cycle, the model's performance improves even as its output length decreases, steadily pushing it closer to the Pareto frontier in the performance-efficiency trade-off. Training on DeepSeek-R1-Distill-Qwen-1.5B, SIRI-low improves performance on AIME24 by 43.2% while reducing token usage by 46.9% after three iterations, and SIRI-high achieves the highest accuracy compared to all other methods (Figure 1). Our findings shed light on the potential of periodically oscillating the LRM's output truncation length during training to dynamically balance exploration and efficiency in reasoning, converging towards an optimal "sweet spot" between the two. Our models are publicly available.
1 INTRODUCTION
SIRI addresses the performance cost of prior attempts to reduce LRMs’ repetitive reasoning by alternating compression and expansion of the reasoning budget. On AIME24, its variants report higher pass@1 with fewer tokens than the initial pre-RL model.
1 INTRODUCTION
- Prior length-compression approaches reduce token use but degrade or stall performance, despite LRMs’ use of long reasoning traces to improve test-time reasoning.RL also increases output length and unnecessary reasoning, raising training and inference time.Source: §1 · p. 2
- SIRI periodically adjusts the maximum rollout length to alternate compression, which reduces overthinking, with expansion, which encourages further exploration of mature reasoning traces.The authors report steady accuracy gains despite shorter outputs and generalization across model sizes.Source: §1 · p. 2
- SIRI-high reaches 43.6% pass@1 on AIME24 using 10K average tokens, while SIRI-low reaches 40.4% using 7K tokens and improves 43% over the initial pre-RL model.Both variants were trained under the same 16K output limit.Source: §1 · p. 2
2 RELATED WORK
Related work addresses reasoning-token efficiency through trace-based fine-tuning, reward shaping, budget forcing, and iterative training. SIRI applies iterative on-policy RL to advance the performance-efficiency frontier for LRMs.
2.1 LENGTH COMPRESSION FOR LRMS
- Length-compression work uses precise traces, reward shaping, adaptive penalties, budget forcing, and end-of-thinking tokens to reduce unnecessary reasoning.The paper reports that these methods compromise performance relative to full-length RL.Source: §2.1 · pp. 2–3
2.2 ITERATIVE TRAINING
- Earlier iterative-training approaches generate or reuse responses across iterations for supervised fine-tuning or preference optimization, with reported modest gains on GSM8K for a combined method.The cited methods are described as off-policy and focused on simple reasoning problems.Source: §2.2 · p. 3
- SIRI extends iterative training to on-policy RL, which the authors use to advance LRMs’ performance-efficiency Pareto frontier.The paper contrasts this with prior iterative methods’ focus on off-policy training for simple reasoning.Source: §2.2 · p. 3
3 PRELIMINARIES
The preliminaries formulate mathematical reasoning as optimizing model parameters for answer reward under a response-length budget, then describe group-relative policy optimization and the adopted DAPO improvements.
3.1 PROBLEM FORMULATION
- The formulation samples a math question and response, scores the final predicted answer as correct or incorrect, and optimizes model parameters subject to a token budget.The response length is |y| and the budget is L.Source: §3.1 · p. 3
3.2 GROUP RELATIVE POLICY OPTIMIZATION
- GRPO samples multiple responses per question and estimates each response’s advantage from group-level rewards before calculating the policy loss.Figure 2 motivates SIRI’s repeated compression and expansion as a way to reduce overthinking while preserving performance and leaving room for exploration.Source: §3.2 · p. 3, Figure 2 · p. 4
- The method adopts DAPO’s decoupled clipping thresholds, with a larger upper threshold to encourage exploration, and removes the KL divergence.The paper attributes KL removal to post-trained reasoning models naturally diverging from the base model.Source: §3.2 · p. 4
4 SIRI: SCALING ITERATIVE REINFORCEMENT LEARNING WITH INTERLEAVED COMPRESSION
SIRI interleaves compression and expansion to pursue performance gains without sustained response-length growth, with compression required to recover performance before the next expansion. Its length-capping reward and scheduler design aim to preserve concise reasoning while allowing exploration at longer limits.
4.1 MOTIVATION
- DeepScaleR’s 8K stage improved performance despite sharply shorter responses, motivating interleaved compression and expansion; compression must restore performance before the next expansion.The motivation passage hypothesizes that interleaving can yield gains while keeping response lengths comparable across expansion stages.Source: §4.1 · p. 4
4.2 REWARD SHAPING
- The cap reward is 1 for a correct answer extractable from clip(y, L), otherwise 0; it works when groups include a short correct response and a longer correct or incorrect response.This group diversity lets policy updates favor short, correct responses and penalize longer responses, preserving dense reasoning while pruning inefficient or wrong patterns.Source: §4.2 · p. 4
- Adaptive length penalties are not adopted because their coefficients require complex tuning across length constraints and their training efficiency is worse than direct capping.Source: §4.2 · p. 5
4.3 LENGTH SCHEDULER
- Schedulers should prevent performance degradation during compression and let generation length plateau before the expansion phase ends.Source: §4.3 · p. 5
- Figure 3 illustrates scheduler curves over 640-step cycles; stair-cosine combines smooth transitions with exploration at Lmax and exploitation at Lmin, unlike cosine alone.The paper says cosine does not maintain the maximum and minimum limits, which may hinder exploration after expansion and performance recovery after compression.Source: §4.3 · p. 5, Figure 3 · p. 5
5 EXPERIMENTS
Experiments assess whether compression-expansion improves reasoning accuracy while pruning redundant tokens, how scheduler design affects that trade-off, and whether the approach applies across models. Results show gains in accuracy and token efficiency, with scheduler cycle length and shape affecting performance during compression and expansion.
5.1 EXPERIMENT SETUP
- Training uses 40K math questions and 1.5B/7B Qwen distills; evaluation covers AIME24, AIME25, AMC, and MATH500.The 1.5B and 7B models are evaluated with Pass@1 accuracy and average response token count.Source: §5.1 · p. 6
5.2 RESULTS
- SIRI-high achieves the highest accuracy across all benchmarks, improving performance by 33.6% on average; SIRI-low reduces response length by 43.1% and boosts performance by 27.0% on average versus the original 1.5B model.Table 1 compares AIME24, AIME25, MATH500, and AMC.Source: §5.2 · p. 6, Table 1 · p. 7
- Across compressed variants, Pass@1 rises 33.3%→38.1%→40.4% as average response length falls 8065→7266→7093 tokens.The results are for SIRI-Iter1/2/3-Compressed on AIME24.Source: §5.2 · p. 7, Figure 4 · p. 8
- Against DAPO-DeepScaleR-16K under similar training times, SIRI reaches comparable AIME24 performance but outperforms on AIME25 with substantially fewer tokens.The comparison concerns the 1.5B model; the DAPO schedule transitions from 8K to 16K at step 320.Source: §5.2 · p. 7, Figure 5 · p. 8
- Entropy decreases during compression and rises during expansion, remaining within a bounded range rather than collapsing as performance gains often accompany rising entropy.The authors report this analysis in Appendix A.2.Source: §5.2 · p. 8
5.3 ABLATIONS ON SCHEDULER DESIGN
- The 640-step cosine scheduler has a milder performance drop at the response-length minimum and sustains accuracy gains during its longer expansion phase, unlike the 320- and 480-step cycles.It also yields the largest length oscillation and highest compression ratio at the response-length minimum.Source: §5.3 · p. 8
5.2 RESULTS
- Compression suppresses “wait” backtracking-and-verification tokens, while expansion encourages them; the other sampled token types remain stable.The “wait” token pattern is consistent for correct and incorrect responses.Source: §5.2 · p. 8, Figure 6 · p. 9
5.3 ABLATIONS ON SCHEDULER DESIGN
- With a 480-step cycle, cosine mitigates compression-stage performance loss, while stair scheduling produces larger gains during expansion.Direct 8K compression with the stair scheduler causes a sharp performance drop; extended 16K expansion brings additional gains.Source: Figure 7 · p. 9, §5.3 · p. 9
6 CONCLUSION
SIRI iteratively alternates token-budget expansion and compression to improve LRM performance while pruning repetitive reasoning. The authors note that its upper performance threshold remains unknown and identify other reasoning tasks as a future direction.
6 CONCLUSION
- SIRI iteratively expands and compresses the token budget to enhance LRM performance while pruning repetitive reasoning traces.The authors describe the phases as encouraging exploration and consolidation in turn.Source: §6 · p. 9
- Experiments show consistent gains in model performance and token efficiency during each iteration.Source: §6 · p. 9
- The upper performance threshold remains unknown, with dataset size and algorithm efficiency offered as possible constraints.Applying SIRI to intensive-reasoning tasks such as code generation is identified as future work.Source: §6 · p. 9
A.1 RESULTS ON DEEPSEEK-R1-DISTILL-QWEN-7B
For DeepSeek-R1-Distill-Qwen-7B, SIRI reaches comparable AIME24 performance and outperforms DAPO-DeepScaleR-16K on AIME25, using fewer tokens on both benchmarks.
A.1 RESULTS ON DEEPSEEK-R1-DISTILL-QWEN-7B
- SIRI reaches comparable AIME24 performance to DAPO-DeepScaleR-16K while using fewer tokens on the 7B model.Source: §A.1 · p. 12
- SIRI outperforms DAPO-DeepScaleR-16K on AIME25 with fewer tokens for the 7B model.Source: §A.1 · p. 12
- DAPO-DeepScaleR-16K transitions from 8K to 16K context at step 360 in the training-dynamics comparison.Source: Figure 8 · p. 12
A.2 DETAILS ON TRAINING DYNAMICS: AN ENTROPY VIEW
The entropy view tracks changes during cosine-scheduler training and compares them with entropy patterns in non-iterative DAPO-DeepScaleR-16K training.
A.2 DETAILS ON TRAINING DYNAMICS: AN ENTROPY VIEW
- During SIRI compression, model entropy decreases; during expansion, it slowly increases without collapsing, remaining within a stable range.Source: §A.2 · p. 12, Figure 9 · p. 12
- DAPO-DeepScaleR-16K also shows a roughly cosine-shaped entropy curve during 16K-context training, despite its non-iterative schedule.Source: §A.2 · p. 13, Figure 10 · p. 13
- Performance increases as entropy rises, even while DAPO-DeepScaleR-16K response length plateaus after step 360.The authors suggest entropy bonuses or higher clipping during expansion as possible ways to further improve SIRI.Source: §A.2 · p. 13