PaperlayerView exact revision on arXiv

Source-linked AI summary

Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic Shuffle

Linghao Zhu, Yiran Guan, Dingkang Liang, Jianzhong Ju, Zhenbo Luo, Bin Qin, Jian Luan, Yuliang Liu, Xiang Bai

arXiv:2508.05612v6cs.LGcs.AI

TL;DR

Most RL approaches use static, uniform trajectory sampling even though learning-signal informativeness varies during training. Shuffle-R1 dynamically selects trajectory pairs and reshapes batches, outperforming representative algorithms across in-domain and out-of-domain tasks while reaching GRPO-matched accuracy in roughly 60% of its wall-clock time.

  • Problem

    Most RL approaches use static, uniform trajectory sampling despite changing signal informativeness, while most rollouts have near-zero advantages that weaken gradient signals.

    Source: §1 · p. 1, §3.2 · p. 4

  • Method

    Shuffle-R1 pairs trajectories with large advantage gaps and adaptively reshapes batches to emphasize informative samples and increase their gradient exposure.

    Source: §1 · p. 2, §3.3 · p. 4, §3.4 · p. 5

  • Results

    Shuffle-R1 outperforms representative algorithms on in-domain and out-of-domain tasks; at GRPO-matched accuracy, it requires roughly 60% of the wall-clock time.

    Source: §4.2 · p. 8, §5 · p. 10

  • Takeaways & Limitations

    The findings demonstrate the value of data-centric adaptive design for improving multimodal large-language-model RL training efficiency.

    Source: §5 · p. 10

    The project uses public academic datasets and open-source checkpoints and code, with no non-public or private data.

    Source: ETHICS STATEMENT · p. 11

Abstract

from arXiv · show

Reinforcement learning (RL) has emerged as an effective post-training paradigm for enhancing the reasoning capabilities of multimodal large language model (MLLM). However, current RL pipelines often suffer from training inefficiencies caused by two underexplored issues: Advantage Collapsing, where most advantages in a batch concentrate near zero, and Rollout Silencing, where the proportion of rollouts contributing non-zero gradients diminishes over time. These issues lead to suboptimal gradient updates and hinder long-term learning efficiency. To address these issues, we propose Shuffle-R1, a simple yet principled framework that improves RL fine-tuning efficiency by dynamically restructuring trajectory sampling and batch composition. It introduces (1) Pairwise Trajectory Sampling, which selects high-contrast trajectories with large advantages to improve gradient signal quality, and (2) Advantage-based Trajectory Shuffle, which increases exposure of valuable rollouts through informed batch reshuffling. Experiments across multiple reasoning benchmarks show that our framework consistently outperforms strong RL baselines with minimal overhead. These results highlight the importance of data-centric adaptations for more efficient RL training in MLLM.

1 INTRODUCTION

RL is increasingly used to strengthen reasoning in language and multimodal models, but static trajectory sampling can overlook how learning-signal value changes during training. Shuffle-R1 dynamically selects and reallocates trajectories to improve the signal and use of RL updates.

1 INTRODUCTION

  • Static RL sampling treats trajectories uniformly even though their informativeness varies and evolves during training.The introduction argues that ignoring this variation risks noisy trajectories overwhelming useful signals.Source: §1 · p. 1
  • Advantage Collapsing concentrates most advantages near zero, while Rollout Silencing reduces the share of rollouts contributing non-zero gradients.The paper links these patterns to weak or negligible updates and wasted computation, respectively.Source: §1 · p. 2, Figure 1 · p. 2
  • Shuffle-R1 substantially improves multimodal reasoning, surpasses GPT-4o and Claude-3.7 on MathVerse and MathVista, and matches GRPO with half the training steps.The reported comparisons are presented as evidence for dynamic data prioritization in RL fine-tuning.Source: §1 · p. 2
  • Shuffle-R1 pairs high-contrast trajectories with large advantage gaps and reshapes batches to emphasize informative trajectories.The introduction presents the two modules as addressing Advantage Collapsing and Rollout Silencing, respectively.Source: §1 · p. 2
  • The framework's dynamic data prioritization supports adaptive interaction between model and data during training.The pipeline comparison contrasts the dynamic paradigm with static and rule-based pre-filter paradigms.Source: §1 · p. 2, Figure 2 · p. 2
  • The paper reports experiments across model scales and in-domain and out-of-domain benchmarks as evidence of effectiveness and generalizability.Its stated conclusion is that RL post-training should reconsider which data receives updates, beyond reward design alone.Source: §1 · p. 2

2 RELATED WORK

Related work spans approaches to reasoning in large language models and efforts to apply or improve RL for multimodal models. Shuffle-R1 positions its contribution as dynamic, adaptive selection and resampling of queries and rollouts.

2.1 LARGE REASONING MODELS

2.2 REINFORCEMENT LEARNING FOR MLLM

3 METHOD

Shuffle-R1 combines pairwise trajectory selection with advantage-based batch reshuffling to address weak or diminishing RL training signals. Its method moves from policy-gradient preliminaries and problem analysis to these two modules.

3.1 PRELIMINARIES

  • Static RL can produce near-zero advantages and progressively fewer rollouts with non-trivial updates, motivating a more dynamic training framework.The method describes policy-gradient updates as using sampled responses, verifiable rewards, and estimated advantages; its clipping hyperparameter ϵ is intended to prevent training collapse.Source: §3.1 · pp. 3–4

3.2 PROBLEM ANALYSIS

  • Advantage Collapsing weakens gradient signals, while Rollout Silencing leaves fewer rollouts contributing non-zero gradients as training proceeds.Larger rollout sizes can partly mitigate Advantage Collapsing but increase computational overhead; simple queries converge early while difficult ones remain inaccurate, and standard pipelines use each rollout only once.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3 PAIRWISE TRAJECTORY SAMPLING

  • Pairwise Trajectory Sampling (PTS) forms high-to-low advantage pairs and retains the pairs with the largest advantage contrasts for gradient updates.It selects a top-k subset from the pairing set according to sampling ratio α, focusing updates on diverse, gradient-rich trajectories without increasing gradient-computation cost.Source: §3.3 · pp. 4–5

3.4 ADVANTAGE-BASED BATCH SHUFFLE

  • Advantage-based Batch Shuffle (ABS) reshapes PTS batches to increase the update frequency of high-value trajectories and reinforce them through repeated exposure.ABS weights pairs by the sum of their absolute advantages, resamples sub-batches, and preserves the original batch size and gradient-update paradigm.Source: §3.4 · pp. 5–6

4 EXPERIMENTS

Experiments evaluate Shuffle-R1 across model scales, training datasets, and visual reasoning benchmarks. The reported results show gains over RL baselines, improved training efficiency, and applicability to larger models and soft-reward tasks.

4.1 EXPERIMENTAL SETUP

  • Experiments use Geo3K and K12 for limited-data evaluation, then a 30k-sample Geo3K–MM-Eureka training set to assess scaling.Evaluation covers in-domain Geometry3K and MMK12 plus out-of-domain math reasoning, visual perception, and chart understanding benchmarks.Source: §4.1 · p. 6

4.2 MAIN RESULTS

  • On Geometry3K, our method’s 3B model reaches 47.88% accuracy, beating GRPO by 5.2% and DAPO by 2.7%; its 7B model reaches 55.89%, with gains of 3.3% and 1.4%, respectively.Source: §4.2 · pp. 6–7
  • At matched GRPO accuracy, Shuffle-R1 uses roughly half the training steps and about 60% of the wall-clock time, with total GPU time increasing 4%–7.7%.It also reaches comparable performance with as little as half the training steps and maintains high token utilization across training stages.Source: §4.2 · p. 8, Figure 7 · p. 8, Figure 6 · p. 8

4.3 ABLATION STUDY

  • On Geometry3K, PTS raises accuracy from 42.64% to 46.21%, and adding ABS contributes a further 1.67 points to reach 47.88%.The full setting also improves over GRPO on the math reasoning set and ChartQA; ABS shows improvement when applied to uniformly sampled trajectory pairs.Source: §4.3 · p. 9, Table 4 · p. 9
  • With the same 8-of-16 rollout budget, PTS consistently improves in-domain and out-of-domain performance, while one-way positive and negative sampling fall below GRPO.Unbiased random sampling yields only a minor improvement over baseline.Source: §4.3 · p. 9, Table 5 · p. 9
  • Sampling ratio α = 0.5 performs strongly, while α = 0.25 lags; performance improves as shuffle rounds increase to S = 8, then saturates.The authors associate aggressive pruning with reduced data diversity and report diminishing returns beyond eight shuffles.Source: §4.3 · pp. 9–10, Figure 8 · p. 9

4.4 EXTENSION EXPERIMENTAL RESULTS

  • Shuffle-R1 performs well on Qwen2.5-VL-32B and adapts to referring-expression comprehension trained with an IoU-based soft reward.The 32B experiment uses 30k samples for 50 steps; the REC experiment trains on 60K samples drawn from RefCOCO, RefCOCOg, and RefCOCO+.Source: §4.4 · p. 10, Table 6 · p. 10, Table 7 · p. 10

5 CONCLUSION

Shuffle-R1 is presented as a data-centric framework for improving MLLM reinforcement-learning efficiency. Its two modules accompany reported gains on in-domain and out-of-domain tasks.

5 CONCLUSION

  • Shuffle-R1 is designed to improve reinforcement-learning training efficiency for multimodal large language models.The framework combines Pairwise Trajectory Sampling with Advantage-based Batch Shuffle.Source: §5 · p. 10
  • The framework significantly outperforms representative algorithms and models on both in-domain and out-of-domain tasks.The conclusion frames these results as evidence for data-centric adaptive design.Source: §5 · p. 10
  • The authors identify data-centric adaptive design as valuable for reinforcement-learning-based reasoning enhancement.Source: §5 · p. 10

ETHICS STATEMENT

The authors describe their data, code, and model sources, report adherence to relevant licenses and ethics requirements, and document reproducibility and LLM use. The appendix also outlines the workflow and training prompt.

ETHICS STATEMENT

  • The project uses public academic datasets and open-source checkpoints and code, reports license compliance, and states that no private data was used.The authors also affirm compliance with the ICLR Code of Ethics.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

  • The paper reports base checkpoints, code, data, benchmarks, metrics, and core hyperparameters, with pseudocode and further materials in the appendices.Appendix D covers additional hyperparameters and infrastructure; Appendices E–G include prompts, data examples, and inference examples.Source: REPRODUCIBILITY STATEMENT · p. 11

LLM USAGE STATEMENT

  • LLMs were used for code checking and bug fixes, writing refinement, and evaluation tasks including answer extraction and correctness assessment.The authors state that the core research, design, implementation, data collection, and analysis were conducted independently without LLM reliance.Source: LLM USAGE STATEMENT · p. 11

A PSEUDO CODE

  • The pseudocode presents Shuffle-R1 as a workflow that generates rollouts, forms advantage-ranked pairs, retains selected pairs, then samples a shuffled batch for updates.Source: A PSEUDO CODE · p. 16

B PROMPT DESIGN

  • The training and evaluation prompts require reasoning inside <think> tags and the final answer inside \boxed{} notation.The prompt is inserted at the beginning of the user message while retaining the Qwen2.5-VL system prompt.Source: B PROMPT DESIGN · p. 16

C MORE EXPERIMENTAL RESULTS

Additional experiments report gains on detailed out-of-domain benchmarks, larger-scale training, and comparisons with more RL algorithms. Further analyses compare online batch shuffling with replay and examine hyperparameter choices.

Full Setting Comparison

  • On the full 30k-sample, 150-step Qwen2.5-VL-7B experiment, Shuffle-R1 gains consistently over strong baselines across all six evaluation benchmarks.Table 10 reports the full-setting comparison; the passage describes the gains as evidence of robustness and scalability.Source: C MORE EXPERIMENTAL RESULTS · p. 16, Table 10 · p. 17

Comparison with Prioritize Experience Replay

  • Replacing ABS with prioritized experience replay yields a later training-accuracy plateau and validation-accuracy drop, while ABS better mitigates Rollout Silencing.The authors suggest replay may overemphasize historical trajectories and contribute to suboptimal convergence.Source: C MORE EXPERIMENTAL RESULTS · p. 17, Figure 9 · p. 17

Detailed Impact of Hyperparameters

  • Performance improves as α rises from 0.25 to 0.5 and mildly decreases at 0.75; increasing S from 4 to 8 helps, while S = 16 reduces performance.The authors interpret these trends as support for appropriate repeated training on high-quality samples without affecting data diversity.Source: C MORE EXPERIMENTAL RESULTS · pp. 17–18, Table 12 · p. 18

D EXPERIMENT SETTINGS

The experiments combine format and accuracy rewards and use EasyR1 with task-specific training steps. They report training on eight 80G GPUs.

  • Format and accuracy rewards each assign 1 for a correct format or ground-truth answer and 0 otherwise.Source: D EXPERIMENT SETTINGS · p. 18
  • The final reward is a weighted sum of format and accuracy rewards, with format reward weighted less because formatting is easy to learn.Source: D EXPERIMENT SETTINGS · p. 18
  • Training used 80 steps for Geometry3K and K12 experiments and 150 steps for joint training on about 30k samples.Other unspecified hyperparameters remained at EasyR1 defaults.Source: D EXPERIMENT SETTINGS · p. 18
  • Experiments used EasyR1, a KL coefficient of 0.0, a frozen vision encoder, and eight 80G GPUs.Source: D EXPERIMENT SETTINGS · p. 18

E DATASET COLLECTION

Training uses Geometry3K and a 2.1k-sample subset of MMK12 called K12. Evaluation spans in-domain test sets and benchmarks covering math reasoning, visual perception, and chart understanding.

  • Geometry3K provides 2.1k training samples and 601 test samples, while K12 is a randomly selected 2.1k-sample MMK12 subset.Geometry3K contains real-world geometry problems with image-based conditions; MMK12 includes math reasoning samples and broader STEM test problems.Source: E DATASET COLLECTION · p. 19
  • Evaluation covers Geometry3K and K12 in-domain tests plus math reasoning, visual perception, and chart-understanding benchmarks.Math benchmarks are MathVerse, MathVision, WeMath, and MathVista; HallusionBench and ChartQA assess visual perception and chart understanding.Source: E DATASET COLLECTION · p. 19

F EVALUATION METRICS

Evaluation reports average pass@1 accuracy across repeated runs and uses Gemini to extract and verify answers on several benchmarks. Case studies describe improvements in geometry, chart reasoning, and perception, while the theoretical analysis models advantages as Gaussian and simplifies PTS as threshold-based selection.

F EVALUATION METRICS

  • Evaluation uses pass@1 accuracy averaged over eight runs at temperature 0.5, with the same prompt format and settings for reproduced comparisons.Gemini-2.0-Flash-001 extracts answers for MathVerse, MathVision, WeMath, MathVista, and ChartQA; WeMath uses loose-mode accuracy and MathVerse reports overall accuracy.Source: F EVALUATION METRICS · pp. 19–20

G CASE STUDY

  • In two geometry cases, the RL model corrects the base model’s visual misreading of angles and misuse of a geometric theorem.Source: G CASE STUDY · p. 20
  • In a chart case, the RL model correctly parses the chart and reasons to the answer despite the base model’s reasoning error.Source: G CASE STUDY · p. 20
  • On a HallusionBench case with a tiny inserted hen, the RL model identifies the image that the base model misses.Source: G CASE STUDY · p. 20

H THEORETICAL ANALYSIS OF SHUFFLE-R1

  • The theoretical analysis assumes advantages follow a Gaussian distribution and models PTS as retaining samples whose absolute advantages exceed a threshold.Within the retained set, trajectories are weighted in proportion to absolute advantage.Source: H THEORETICAL ANALYSIS OF SHUFFLE-R1 · pp. 20–21

H.1 INTUITIVE ANALYSIS ON TRAINING DYNAMICS

Under a Gaussian advantage assumption, the analysis argues that threshold selection and advantage-weighted sampling increase expected absolute advantage and gradient magnitude. It also identifies positive bias toward higher-advantage trajectories as a consequence of adaptive sampling.

H.1 INTUITIVE ANALYSIS ON TRAINING DYNAMICS

  • PTS truncation and sampling weighted by absolute advantage yield a greater expected absolute advantage than the original distribution.The argument decomposes the original expectation at a threshold and compares the retained high-magnitude samples with the full distribution.Source: §H.1 · p. 21
  • Because per-sample gradient norm scales with absolute advantage, the increased expectation yields larger expected gradient norms and update magnitudes under the stated independence assumption.Source: §H.1 · p. 21

H.2 BIAS UNDER SELECTIVE SAMPLING

  • Adaptive sampling introduces positive bias, systematically tilting gradient updates toward trajectories with higher advantages.Source: §H.2 · p. 21
Loading 2508.05612v6…