Source-linked AI summary
Stable-MM-R1: Anchoring Multimodal Reasoning Dynamics via Entropy-Guided Stratification
Yimeng Ye, Shuang Chen, Wenxuan Huang, Manyuan Zhang, Kaituo Feng, Zhangquan Chen, Jiayu Chen, Yucheng Zhou, Yicheng Xiao, Zhiyuan Feng, Tianyu Shi
TL;DR
Reasoning RL can be unstable, with rollout and group composition contributing to poor learning signals and entropy collapse. Stable-MM-R1 combines dynamic query mining with entropy- and reward-guided rollout stratification to organize optimization groups. The paper reports stronger average results than evaluated open-source 7B-scale baselines and healthy Path-Entropy trajectories, while noting added scaffolding and data-generation costs.
Problem
Reasoning RL can be brittle, and stochastic rollout and group quality can produce zero-signal groups and premature entropy collapse.
Source: §1 · pp. 1–2Method
Stable-MM-R1 combines PAQM, which filters queries using Pass@K and Path Entropy, with HSR, which organizes rollouts by Path Entropy and reward into structured optimization groups.
Source: §1 · p. 2, §3.2 · p. 4Results
Stable-MM-R1 outperforms evaluated open-source 7B-scale baselines on average across diverse benchmarks while maintaining healthy Path-Entropy trajectories that avoid collapse patterns observed in GRPO and DAPO.
Source: §1 · p. 2Takeaways & Limitations
Explicit contrast within the micro-batch via anchors and hard negatives can improve long-chain reasoning training without changing the underlying GRPO objective.
Source: §5 · p. 8The method relies on proprietary Seed-1.5-VL scaffolding, adds offline teacher-generation and query-resampling costs, and requires careful intra-step buffer management.
Source: Limitations · p. 8
Abstract
from arXiv · showhide
While Reinforcement Learning (RL) effectively incentivizes reasoning in Large Language Models, current pipelines are hindered by training instability and rapid entropy collapse. These limitations often stem from "Rollout Silencing" and low-quality gradient signals in standard sampling procedures. In this work, we propose a robust, data-centric framework to stabilize RL training. We first introduce Potential-Aware Query Mining (PAQM), which filters data dynamically to focus on the "Distillation Zone"---samples with high potential for capability elicitation. Furthermore, we present Hybrid Stratified Replay (HSR), a novel mechanism that restructures batches by stratifying rollouts based on Path Entropy, a rollout-level confidence proxy, and outcome reward. Within each optimization step, HSR reuses current-policy "Stability Anchors" and "Hard Negatives" to construct high-contrast optimization groups, then clears its buffers before the next step. This approach mitigates entropy collapse while improving the utilization of learning signals under limited compute. Our method outperforms strong baselines on complex reasoning tasks, offering a principled solution for stable and efficient RL fine-tuning.
1 Introduction
The introduction frames reasoning-RL instability as a problem of rollout and group quality, and presents PAQM and HSR as data-centric interventions. It reports that Stable-MM-R1 outperforms evaluated open-source 7B-scale baselines on average across diverse benchmarks.
1 Introduction
- Reasoning RL can suffer oscillating rewards, abrupt KL/entropy shifts, premature narrow-mode collapse, and degenerate groups with vanishing or destabilizing updates.Long-CoT settings intensify the problem because gradient signals are sparse and delayed.Source: §1 · p. 1
- The introduction asks whether controlling the data seen at each step can stabilize RL, since easy queries waste compute, impossible ones add noise, and mixed-quality groups yield unreliable advantages.This contrasts with objective-centric stabilization methods that use clipping or KL and entropy controls.Source: §1 · p. 2
- PAQM dynamically filters queries using Pass@K and Path Entropy to focus updates within a learnable “Distillation Zone.”The stated aim is to avoid spending compute on trivial or impossible queries.Source: §1 · p. 2
- HSR structures optimization groups with Stability Anchors and Hard Negatives when available, pairing them with exploration targets to reduce degenerate groups.The framework is designed to mitigate premature entropy collapse without changing the underlying RL objective.Source: §1 · p. 2
- Stable-MM-R1 outperforms evaluated open-source 7B-scale baselines on average across diverse benchmarks while maintaining healthy Path-Entropy trajectories.The reported trajectories avoid the collapse patterns observed in GRPO and DAPO.Source: §1 · p. 2
2 Related Works
Related work spans multimodal reasoning models, objective-centric RL stabilization, and data-centric optimization. Stable-MM-R1 positions Path Entropy and structured sample stratification as a data-curation approach to stability.
2.1 Multimodal Large Reasoning Models (MLRMs)
- SATORI-R1 and Retrv-R1 address specialized multimodal reasoning tasks, while this work focuses on stability of the underlying optimization process.The broader context includes verifiable-reward RL for multimodal models such as Qwen2.5-VL and LLaVA-OneVision.Source: §2.1 · p. 2
2.2 Entropy Collapse and Stabilization in RL
- Entropy collapse narrows a policy prematurely; prior stabilization approaches modify objectives through quantile baselines, token weighting, momentum reference models, or PI control.The overview figure depicts Stable-MM-R1’s alternative pipeline: query filtering followed by rollout stratification and hybrid group assembly.Source: §2.2 · pp. 2–3, Figure 1 · p. 3
2.3 Data-Centric Policy Optimization
- Data-centric approaches organize rollouts or samples, while Stable-MM-R1 adds Path Entropy as a confidence dimension and stratifies stability anchors with exploration targets.Related approaches include rollout silencing and contrastive batch organization, sample polarity, variance-aware sampling, and rollout response recomposition.Source: §2.3 · p. 3
3 Methodology
Stable-MM-R1 stabilizes multimodal reasoning post-training through dynamic query curation and rollout stratification followed by structured group assembly.
3.1 Phase 0: Potential-Aware Query Mining (PAQM)
- PAQM targets variable-success Distillation Zone queries; fully mastered queries are downsampled, while zero-success queries become diagnosis candidates.Source: §3.1 · p. 4
- Path Entropy is a sample-level confidence proxy, not Shannon entropy of the full predictive distribution or a direct estimate of epistemic uncertainty.Source: §3.1 · p. 4
- For impossible-zone queries, modality-decoupled diagnosis uses ground-truth captions to isolate failure, then adds visual decomposition hints for perceptual deficits or reasoning nodes for cognitive deficits.These hints reclaim roughly 30% of hard queries into the Distillation Zone; they are generated offline with Seed-1.5-VL and injected only during Phase 0 curriculum construction.Source: §3.1 · p. 4
3.2 Phase 1: Diagnostic Path-Entropy–Reward Stratification
- Batch-adaptive 10th- and 90th-percentile Path-Entropy cutoffs classify correct low-entropy anchors, incorrect low-entropy Hard Negatives, and correct high-entropy Epiphanies; high-entropy errors are excluded.Source: §3.2 · p. 4
3.3 Phase 2: Hybrid Group Assembly
- HSR splits groups evenly between a structured stream with 50% Epiphanies, 25% Stability Anchors, and 25% Hard Negatives, and uniform natural sampling; buffers use current-step rollouts and clear after gradients.The natural stream preserves on-policy coverage alongside the deliberately shifted structured stream.Source: §3.3 · p. 5
3.3.1 Theoretical Motivation: Degenerate Groups
- Randomly sampled groups can give every rollout the same reward, causing group-relative advantages to vanish; HSR includes successful and failed rollouts when available.The paper describes a fallback when a structured quota cannot be satisfied.Source: §3.3.1 · p. 5
4 Experiments
Stable-MM-R1 leads the evaluated open-source 7B models on average and maintains more stable late-training dynamics in this run. Ablations report gains from PAQM and HSR, while the evaluation uses a matched update budget but not identical preparation costs.
4.1 Experimental Setup
- The RL Prompt Corpus contains approximately 189k multimodal samples, and evaluation spans mathematical reasoning, visual perception, and chart understanding benchmarks.The corpus aggregates Geometry3K and a curated MM-Eureka subset, with ViRL39K added for visual-reasoning diversity; offline Seed-1.5-VL hints are injected into selected prompts during curriculum construction.Source: §4.1 · p. 5
- Experiments use Qwen2.5-VL-7B-Instruct, freeze the vision encoder, and update the LLM backbone across 400 steps with 16 rollouts per query for HSR.In-house methods match final group size and update count, but PAQM adds offline hint-cache and possible query-resampling costs.Source: §4.1 · p. 6
- Text MinHash LSH and image perceptual hashing filter near-duplicate evaluation overlap, but cannot rule out all semantic contamination.The text filter removes samples above 0.8 Jaccard similarity to evaluation queries.Source: §4.1 · p. 6
- GRPO and DAPO baseline scores are reported at their pre-collapse peaks because they frequently collapsed and degraded after step 250.Stable-MM-R1 maintained stability through the approximately 400-step schedule.Source: §4.1 · p. 6
4.2 Main Results
- 56.2% average score places Stable-MM-R1 9.7 points above Vision-G1 among evaluated open-source 7B models; MathVerse reaches 56.3%, 6.3 points above Vision-G1.On MathVision, the method also improves by 28.0 points over the Qwen2.5-VL base model and 20.2 points over the strongest evaluated open-source baseline; MathVista is 0.6 points below Vision-G1.Source: §4.2 · p. 6, Table 1 · p. 7
- Stable-MM-R1’s Path Entropy remains mostly around 0.35–0.60 nats after an initial decline, while DAPO falls to approximately 0.2 and GRPO fluctuates before a sharp decline.Stable-MM-R1’s reward stays near its late-training plateau through step 400; the reported trajectories indicate greater late-training stability in this run, not monotonic improvement.Source: §4.2 · pp. 6–7
4.3 Ablation Study
- Adding PAQM alone improves the average score by 3.1 points over GRPO.The comparison uses the hint-augmented curriculum with the standard GRPO objective.Source: §4.3 · p. 7, Table 2 · p. 7
- Adding HSR without PAQM improves the average score by 4.0 points over GRPO, while the combined method reaches the best ablation average of 65.8.Comparing + PAQM with Full Stable-MM-R1 isolates a further 2.0-point average gain from HSR under PAQM.Source: §4.3 · p. 7, Table 2 · p. 7
5 Conclusion
The conclusion attributes reasoning-RL instability in part to stochastic optimization-group composition and presents PAQM with HSR as a data-centric framework. In experiments, the full method mitigated entropy collapse, while explicit micro-batch contrast improved long-chain reasoning training without changing the GRPO objective.
5 Conclusion
- Stochastic optimization-group composition can contribute to instability in RL training for reasoning.The paper presents this as one contributing source, not the sole cause.Source: §5 · p. 7
- PAQM dynamically curates learnable queries, while HSR structures groups with stability anchors and contrastive negatives to mitigate entropy collapse in the experiments.These are the two components of the proposed data-centric framework.Source: §5 · pp. 7–8
- Explicit anchor–negative contrast within the micro-batch can improve long-chain reasoning training without changing the underlying GRPO objective.The paper frames this data-centric paradigm as complementary to other stabilization approaches.Source: §5 · p. 8
Limitations
The framework depends on a proprietary teacher for initial scaffolding and adds processing costs; its current focus is verifiable reasoning, with open-ended generation left for future work.
Limitations
- Initial scaffolding relies on proprietary Seed-1.5-VL, which may introduce distillation bias and accessibility constraints.The method also incurs offline teacher-generation and query-resampling costs, and requires careful intra-step buffer management.Source: Limitations · p. 8
- PAQM adds offline teacher-generation and query-resampling costs, while HSR requires careful intra-step buffer management.The comparisons match update count and final group size.Source: Limitations · p. 8
- The method focuses on verifiable reasoning; extending entropy-guided stratification to open-ended generation remains future work.Source: Limitations · p. 8
Ethical Considerations
The paper discusses risks from capability-based data filtering, teacher-generated hints, deployment of stabilized multimodal models, and computational costs, alongside mitigation steps and scope boundaries.
Bias in Data Selection
- PAQM could exclude data about dialects, cultural contexts, or visual concepts that the base model classifies as impossible.The authors use diverse STEM benchmarks and advise auditing discarded data in sociologically sensitive domains.Source: Ethical Considerations · pp. 8–9
Data Privacy and Content Safety
- The training sources are mainly public STEM benchmarks with relatively low PII risk; keyword filtering and random-sample audits were used to reduce privacy and content-safety risks.Source: Ethical Considerations · p. 9
Teacher-Student Alignment Risks
- Teacher-generated hints create a dependency on the teacher model’s safety alignment, and harmful instructions could be reinforced through high-reward gradients.The experiments focus on verifiable math and logic; open-ended generation would require safety-specific reward models.Source: Ethical Considerations · p. 9
Dual Use and Automation
- Stabilizing 7B-scale models lowers the barrier to deploying multimodal agents and may increase misuse potential in automated tasks.The models are research prototypes intended for academic evaluation, and the authors recommend rigorous red-teaming before real-world deployment.Source: Ethical Considerations · p. 9
Environmental Impact
- Experiments used 8× H100 GPUs, but the paper does not report a full energy or carbon accounting.The authors identify data-centric efficiency as an important direction for reducing post-training’s environmental cost.Source: Ethical Considerations · p. 9
A.1 Hyperparameter Sensitivity
The sensitivity analysis examines HSR’s stream split ratio and Path-Entropy percentile thresholds, finding a robust middle setting and instability at more extreme settings.
A.1 Hyperparameter Sensitivity
- The analysis varies HSR’s Structured/Natural stream split ratio and percentile thresholds.Source: §A.1 · p. 12
- A 50% Structured Stream ratio provides the optimal balance; ratios above 75% overfit the replay buffer, while ratios below 25% fail to prevent entropy collapse.Source: §A.1 · p. 12
- The 10th/90th percentile setting proved robust across examined benchmarks; 5th-percentile thresholds overly restrict the anchor pool, while 20th-percentile thresholds admit noisy anchors.Source: §A.1 · p. 12
A.2 Algorithmic Implementation Details
Stable-MM-R1 extends GRPO with a dynamic query curriculum and an intra-step stratified replay mechanism. The procedure assesses queries, partitions current-step rollouts by reward and Path Entropy, then combines structured and natural samples for optimization.
A.2 Algorithmic Implementation Details
- Each query uses 16 rollouts, and the algorithm requires an offline hint cache generated in advance by frozen proprietary teacher Seed-1.5-VL.The group size is G = N = 16.Source: §A.2 · pp. 12, 14
- PAQM estimates success from pilot rollouts and, when the empirical success rate is zero, retrieves a cached hint and regenerates rollouts to move selected impossible queries into the Distillation Zone.Source: §A.2 · p. 13
- HSR computes batch-adaptive 10th and 90th Path-Entropy percentiles, then separates high-confidence correct anchors, high-confidence incorrect hard negatives, and low-confidence correct epiphanies.Incorrect high-entropy confusion samples are filtered out.Source: §A.2 · p. 13
- The structured stream uses 50% of the group budget: 25% epiphanies, 12.5% anchors, and 12.5% hard negatives when available; unavailable quotas fall back to natural on-policy sampling.Source: §A.2 · p. 13
- The remaining 50% is sampled uniformly from valid samples excluding noise, retaining natural on-policy data alongside structured replay.The natural stream regularizes against overfitting to the structured stream’s artificial topology.Source: §A.2 · p. 13
- After each GRPO update, the anchor, hard-negative, and epiphany buffers are cleared, so optimization samples remain from the current policy without stale-data corrections.Source: §A.2 · pp. 13–14
A.3 Conditional Variance Analysis of Structured Groups
The analysis explains why standard GRPO can produce zero-signal groups and establishes a conditional variance guarantee for HSR’s structured groups, while framing structured sampling as a distribution shift rather than an unbiased estimator.
A.3.1 Degenerate Groups in Standard GRPO
- In binary-reward GRPO, groups with identical rewards have zero empirical reward spread, so their advantages vanish or become ill-conditioned; such groups become likelier as success rates approach either extreme.The analysis does not assert a non-zero update for every optimization step.Source: §A.3.1 · p. 14, §A.3 · p. 14
A.3.2 Conditional Variance of Structured Groups
- When both outcome classes are available, HSR’s structured group has strictly positive reward variance; if a quota cannot be met, the bound does not guarantee non-zero variance at every step.The unavailable allocation is released and filled through natural on-policy sampling.Source: §A.3.2 · p. 14
A.3.3 Distribution-Shift Trade-off
- HSR’s structured sampling emphasizes anchors, hard negatives, and high-Path-Entropy successes, deliberately shifting the empirical training distribution without claiming an unbiased estimator.Natural on-policy samples remain in each optimization step, and empirical training dynamics evaluate the stability trade-off.Source: §A.3.3 · p. 15